From 1fde38beaa0c3e66c340efc7cc0dc272d6254bb0 Mon Sep 17 00:00:00 2001 From: Magnus Hagander Date: Thu, 5 Apr 2018 21:57:26 +0200 Subject: Allow on-line enabling and disabling of data checksums This makes it possible to turn checksums on in a live cluster, without the previous need for dump/reload or logical replication (and to turn it off). Enabling checkusm starts a background process in the form of a launcher/worker combination that goes through the entire database and recalculates checksums on each and every page. Only when all pages have been checksummed are they fully enabled in the cluster. Any failure of the process will revert to checksums off and the process has to be started. This adds a new WAL record that indicates the state of checksums, so the process works across replicated clusters. Authors: Magnus Hagander and Daniel Gustafsson Review: Tomas Vondra, Michael Banck, Heikki Linnakangas, Andrey Borodin --- doc/src/sgml/func.sgml | 65 +++++++++++++++++ doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/initdb.sgml | 6 +- doc/src/sgml/ref/pg_verify_checksums.sgml | 112 ++++++++++++++++++++++++++++++ doc/src/sgml/reference.sgml | 1 + doc/src/sgml/wal.sgml | 81 +++++++++++++++++++++ 6 files changed, 263 insertions(+), 3 deletions(-) create mode 100644 doc/src/sgml/ref/pg_verify_checksums.sgml (limited to 'doc/src') diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 122f034f177..6257563eaad 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -19540,6 +19540,71 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); + + Data Checksum Functions + + + The functions shown in can + be used to enable or disable data checksums in a running cluster. + See for details. + + + + Checksum <acronym>SQL</acronym> Functions + + + + Function + Return Type + Description + + + + + + + pg_enable_data_checksums + + pg_enable_data_checksums(cost_delay int, cost_limit int) + + + void + + + + Initiates data checksums for the cluster. This will switch the data checksums mode + to in progress and start a background worker that will process + all data in the database and enable checksums for it. When all data pages have had + checksums enabled, the cluster will automatically switch to checksums + on. + + + If cost_delay and cost_limit are + specified, the speed of the process is throttled using the same principles as + Cost-based Vacuum Delay. + + + + + + + pg_disable_data_checksums + + pg_disable_data_checksums() + + + void + + + Disables data checksums for the cluster. + + + + +
+ + + Database Object Management Functions diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 4e01e5641cf..7cd6ee85dc9 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -211,6 +211,7 @@ Complete list of usable sgml source files in this directory. + diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml index 949b5a220f5..826dd91f729 100644 --- a/doc/src/sgml/ref/initdb.sgml +++ b/doc/src/sgml/ref/initdb.sgml @@ -195,9 +195,9 @@ PostgreSQL documentation Use checksums on data pages to help detect corruption by the I/O system that would otherwise be silent. Enabling checksums - may incur a noticeable performance penalty. This option can only - be set during initialization, and cannot be changed later. If - set, checksums are calculated for all objects, in all databases. + may incur a noticeable performance penalty. If set, checksums + are calculated for all objects, in all databases. See + for details. diff --git a/doc/src/sgml/ref/pg_verify_checksums.sgml b/doc/src/sgml/ref/pg_verify_checksums.sgml new file mode 100644 index 00000000000..463ecd5e1b3 --- /dev/null +++ b/doc/src/sgml/ref/pg_verify_checksums.sgml @@ -0,0 +1,112 @@ + + + + + pg_verify_checksums + + + + pg_verify_checksums + 1 + Application + + + + pg_verify_checksums + verify data checksums in an offline PostgreSQL database cluster + + + + + pg_verify_checksums + option + datadir + + + + + Description + + pg_verify_checksums verifies data checksums in a PostgreSQL + cluster. It must be run against a cluster that's offline. + + + + + Options + + + The following command-line options are available: + + + + + + + + Only validate checksums in the relation with specified relfilenode. + + + + + + + + + Force check even if checksums are disabled on cluster. + + + + + + + + + Enable debug output. Lists all checked blocks and their checksum. + + + + + + + + + + Print the pg_verify_checksums version and exit. + + + + + + + + + + Show help about pg_verify_checksums command line + arguments, and exit. + + + + + + + + + Notes + + Can only be run when the server is offline. + + + + + See Also + + + + + + + diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index ef2270c4673..78c214f1b08 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -284,6 +284,7 @@ &pgtestfsync; &pgtesttiming; &pgupgrade; + &pgVerifyChecksums; &pgwaldump; &postgres; &postmaster; diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index f4bc2d4161e..6249cb41361 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -230,6 +230,87 @@ + + Data checksums + + checksums + + + + Data pages are not checksum protected by default, but this can optionally be enabled for a cluster. + When enabled, each data page will be assigned a checksum that is updated when the page is + written and verified every time the page is read. Only data pages are protected by checksums, + internal data structures and temporary files are not. + + + + Checksums are normally enabled when the cluster is initialized using + initdb. They + can also be enabled or disabled at runtime. In all cases, checksums are enabled or disabled + at the full cluster level, and cannot be specified individually for databases or tables. + + + + The current state of checksums in the cluster can be verified by viewing the value + of the read-only configuration variable by + issuing the command SHOW data_checksums. + + + + When attempting to recover from corrupt data it may be necessary to bypass the checksum + protection in order to recover data. To do this, temporarily set the configuration parameter + . + + + + On-line enabling of checksums + + + Checksums can be enabled or disabled online, by calling the appropriate + functions. + Disabling of checksums takes effect immediately when the function is called. + + + + Enabling checksums will put the cluster in inprogress mode. + During this time, checksums will be written but not verified. In addition to + this, a background worker process is started that enables checksums on all + existing data in the cluster. Once this worker has completed processing all + databases in the cluster, the checksum mode will automatically switch to + on. + + + + The process will initially wait for all open transactions to finish before + it starts, so that it can be certain that there are no tables that have been + created inside a transaction that has not committed yet and thus would not + be visible to the process enabling checksums. It will also, for each database, + wait for all pre-existing temporary tables to get removed before it finishes. + If long-lived temporary tables are used in the application it may be necessary + to terminate these application connections to allow the process to complete. + Information about open transactions and connections with temporary tables is + written to log. + + + + If the cluster is stopped while in inprogress mode, for + any reason, then this process must be restarted manually. To do this, + re-execute the function pg_enable_data_checksums() + once the cluster has been restarted. It is not possible to resume the work, + the process has to start over and re-process the cluster. + + + + + Enabling checksums can cause significant I/O to the system, as most of the + database pages will need to be rewritten, and will be written both to the + data files and the WAL. + + + + + + Write-Ahead Logging (<acronym>WAL</acronym>) -- cgit v1.2.3