From 8c16ad3b43299695f203f9157a2b27c22b9ed634 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 6 Sep 2023 16:27:16 -0700 Subject: Allow using syncfs() in frontend utilities. This commit allows specifying a --sync-method in several frontend utilities that must synchronize many files to disk (initdb, pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). On Linux, users can specify "syncfs" to synchronize the relevant file systems instead of calling fsync() for every single file. In many cases, using syncfs() is much faster. As with recovery_init_sync_method, this new option comes with some caveats. The descriptions of these caveats have been moved to a new appendix section in the documentation. Co-authored-by: Justin Pryzby Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com --- doc/src/sgml/config.sgml | 12 +++--------- doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/ref/initdb.sgml | 22 ++++++++++++++++++++++ doc/src/sgml/ref/pg_basebackup.sgml | 25 +++++++++++++++++++++++++ doc/src/sgml/ref/pg_checksums.sgml | 22 ++++++++++++++++++++++ doc/src/sgml/ref/pg_dump.sgml | 21 +++++++++++++++++++++ doc/src/sgml/ref/pg_rewind.sgml | 22 ++++++++++++++++++++++ doc/src/sgml/ref/pgupgrade.sgml | 23 +++++++++++++++++++++++ doc/src/sgml/syncfs.sgml | 36 ++++++++++++++++++++++++++++++++++++ 10 files changed, 176 insertions(+), 9 deletions(-) create mode 100644 doc/src/sgml/syncfs.sgml (limited to 'doc/src') diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f0a50a5f9ad..6bc1b215db9 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -10511,15 +10511,9 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' On Linux, syncfs may be used instead, to ask the operating system to synchronize the whole file systems that contain the data directory, the WAL files and each tablespace (but not any other - file systems that may be reachable through symbolic links). This may - be a lot faster than the fsync setting, because it - doesn't need to open each file one by one. On the other hand, it may - be slower if a file system is shared by other applications that - modify a lot of files, since those files will also be written to disk. - Furthermore, on versions of Linux before 5.8, I/O errors encountered - while writing data to disk may not be reported to - PostgreSQL, and relevant error messages may - appear only in kernel logs. + file systems that may be reachable through symbolic links). See + for more information about using + syncfs(). This parameter can only be set in the diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index e3d94a62b3f..4c63a7e7689 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -183,6 +183,7 @@ + diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 2e271862fc1..f629524be07 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -294,6 +294,7 @@ break is not needed in a wider output rendering. &acronyms; &glossary; &color; + &syncfs; &obsolete; diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml index 22f1011781f..8a09c5c4388 100644 --- a/doc/src/sgml/ref/initdb.sgml +++ b/doc/src/sgml/ref/initdb.sgml @@ -365,6 +365,28 @@ PostgreSQL documentation + + + + + When set to fsync, which is the default, + initdb will recursively open and synchronize all + files in the data directory. The search for files will follow symbolic + links for the WAL directory and each configured tablespace. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file systems that contain the + data directory, the WAL files, and each tablespace. See + for more information about using + syncfs(). + + + This option has no effect when is used. + + + + diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index 79d3e657c32..d2b8ddd200c 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -594,6 +594,31 @@ PostgreSQL documentation + + + + + When set to fsync, which is the default, + pg_basebackup will recursively open and synchronize + all files in the backup directory. When the plain format is used, the + search for files will follow symbolic links for the WAL directory and + each configured tablespace. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file system that contains the + backup directory. When the plain format is used, + pg_basebackup will also synchronize the file systems + that contain the WAL files and each tablespace. See + for more information about using + syncfs(). + + + This option has no effect when is used. + + + + diff --git a/doc/src/sgml/ref/pg_checksums.sgml b/doc/src/sgml/ref/pg_checksums.sgml index a3d0b0f0a3d..7b44ba71cf9 100644 --- a/doc/src/sgml/ref/pg_checksums.sgml +++ b/doc/src/sgml/ref/pg_checksums.sgml @@ -139,6 +139,28 @@ PostgreSQL documentation + + + + + When set to fsync, which is the default, + pg_checksums will recursively open and synchronize + all files in the data directory. The search for files will follow + symbolic links for the WAL directory and each configured tablespace. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file systems that contain the + data directory, the WAL files, and each tablespace. See + for more information about using + syncfs(). + + + This option has no effect when is used. + + + + diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index a3cf0608f5b..c1e2220b3cb 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -1179,6 +1179,27 @@ PostgreSQL documentation + + + + + When set to fsync, which is the default, + pg_dump --format=directory will recursively open and + synchronize all files in the archive directory. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file system that contains the + archive directory. See for more information + about using syncfs(). + + + This option has no effect when is used or + is not set to directory. + + + + diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 15cddd086b7..80dff161682 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -284,6 +284,28 @@ PostgreSQL documentation + + + + + When set to fsync, which is the default, + pg_rewind will recursively open and synchronize all + files in the data directory. The search for files will follow symbolic + links for the WAL directory and each configured tablespace. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file systems that contain the + data directory, the WAL files, and each tablespace. See + for more information about using + syncfs(). + + + This option has no effect when is used. + + + + diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 7816b4c6859..bea0d1b93f9 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -190,6 +190,29 @@ PostgreSQL documentation variable PGSOCKETDIR + + + + + When set to fsync, which is the default, + pg_upgrade will recursively open and synchronize all + files in the upgraded cluster's data directory. The search for files + will follow symbolic links for the WAL directory and each configured + tablespace. + + + On Linux, syncfs may be used instead to ask the + operating system to synchronize the whole file systems that contain the + upgraded cluster's data directory, its WAL files, and each tablespace. + See for more information about using + syncfs(). + + + This option has no effect when is used. + + + + username username diff --git a/doc/src/sgml/syncfs.sgml b/doc/src/sgml/syncfs.sgml new file mode 100644 index 00000000000..00457d24579 --- /dev/null +++ b/doc/src/sgml/syncfs.sgml @@ -0,0 +1,36 @@ + + + + <function>syncfs()</function> Caveats + + + syncfs + + + + On Linux syncfs() may be specified for some + configuration parameters (e.g., + ), server applications (e.g., + pg_upgrade), and client applications (e.g., + pg_basebackup) that involve synchronizing many + files to disk. syncfs() is advantageous in many cases, + but there are some trade-offs to keep in mind. + + + + Since syncfs() instructs the operating system to + synchronize a whole file system, it typically requires many fewer system + calls than using fsync() to synchronize each file one by + one. Therefore, using syncfs() may be a lot faster than + using fsync(). However, it may be slower if a file + system is shared by other applications that modify a lot of files, since + those files will also be written to disk. + + + + Furthermore, on versions of Linux before 5.8, I/O errors encountered while + writing data to disk may not be reported to the calling program, and relevant + error messages may appear only in kernel logs. + + + -- cgit v1.2.3