summaryrefslogtreecommitdiff
path: root/src/bin/pg_basebackup/walmethods.c
AgeCommit message (Collapse)Author
2025-01-09Fix off_t overflow in pg_basebackup on Windows.Thomas Munro
walmethods.c used off_t to navigate around a pg_wal.tar file that could exceed 2GB, which doesn't work on Windows and would fail with misleading errors. Use pgoff_t instead. Back-patch to all supported branches. Author: Davinder Singh <davinder.singh@enterprisedb.com> Reported-by: Jakub Wartak <jakub.wartak@enterprisedb.com> Discussion: https://postgr.es/m/CAKZiRmyM4YnokK6Oenw5JKwAQ3rhP0YTz2T-tiw5dAQjGRXE3Q%40mail.gmail.com
2025-01-01Update copyright for 2025Bruce Momjian
Backpatch-through: 13
2024-11-06Remove unused #include's from bin .c filesPeter Eisentraut
as determined by IWYU Similar to commit dbbca2cf299, but for bin and some related files. Discussion: https://www.postgresql.org/message-id/flat/0df1d5b1-8ca8-4f84-93be-121081bde049%40eisentraut.org
2024-07-02Convert some extern variables to staticPeter Eisentraut
These probably should have been static all along, it was only forgotten out of sloppiness. Reviewed-by: Andres Freund <andres@anarazel.de> Discussion: https://www.postgresql.org/message-id/flat/e0a62134-83da-4ba4-8cdb-ceb0111c95ce@eisentraut.org
2024-01-03Update copyright for 2024Bruce Momjian
Reported-by: Michael Paquier Discussion: https://postgr.es/m/ZZKTDPxBBMt3C0J9@paquier.xyz Backpatch-through: 12
2023-10-31C comment: mention why no setting lasterrno in dir_existsfile()Bruce Momjian
Reported-by: Wei Sun Discussion: https://postgr.es/m/tencent_1276C08F98579CC19D8A4488C848A8411806@qq.com Backpatch-through: master
2023-08-07Improve const use in zlib-using codePeter Eisentraut
If we define ZLIB_CONST before including zlib.h, zlib augments some interfaces with const decorations. By doing that we can keep our own interfaces cleaner and can remove some unconstify calls. ZLIB_CONST was introduced in zlib 1.2.5.2 (17 Dec 2011). When compiling with older zlib releases, you might now get compiler warnings about discarding qualifiers. CentOS 6 has zlib 1.2.3, but in 8e278b6576, we removed support for the OpenSSL release in CentOS 6, so it seems ok to de-support the zlib release in CentOS 6 as well. Reviewed-by: Tristan Partin <tristan@neon.tech> Discussion: https://www.postgresql.org/message-id/flat/33462926-bb1e-7cc9-8d92-d86318e8ed1d%40eisentraut.org
2023-08-01Add and use symbolic constants for tar header offsets and file types.Robert Haas
Because symbolic constants in a header file are better than magic constants embedded in the code. Patch by me, reviewed by Tom Lane, Dagfinn Ilmari Mannsåker, and Tristan Partin. Discussion: http://postgr.es/m/CA+TgmoZNbLwhmCrNtkJAvi8FLkwFdMeVU3myV2HQQpA5bvbRZg@mail.gmail.com
2023-05-19Pre-beta mechanical code beautification.Tom Lane
Run pgindent, pgperltidy, and reformat-dat-files. This set of diffs is a bit larger than typical. We've updated to pg_bsd_indent 2.1.2, which properly indents variable declarations that have multi-line initialization expressions (the continuation lines are now indented one tab stop). We've also updated to perltidy version 20230309 and changed some of its settings, which reduces its desire to add whitespace to lines to make assignments etc. line up. Going forward, that should make for fewer random-seeming changes to existing code. Discussion: https://postgr.es/m/20230428092545.qfb3y5wcu4cm75ur@alvherre.pgsql
2023-05-08pg_basebackup: Restore lost translation markersPeter Eisentraut
The refactoring in ebfb814f7c lost some translation markers.
2023-03-06Revise pg_pwrite_zeros()Michael Paquier
The following changes are made to pg_write_zeros(), the API able to write series of zeros using vectored I/O: - Add of an "offset" parameter, to write the size from this position (the 'p' of "pwrite" seems to mean position, though POSIX does not outline ythat directly), hence the name of the routine is incorrect if it is not able to handle offsets. - Avoid memset() of "zbuffer" on every call. - Avoid initialization of the whole IOV array if not needed. - Group the trailing write() call with the main write() call, simplifying the function logic. Author: Andres Freund Reviewed-by: Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/20230215005525.mrrlmqrxzjzhaipl@awork3.anarazel.de
2023-01-02Update copyright for 2023Bruce Momjian
Backpatch-through: 11
2022-11-08Use pg_pwrite_zeros() in walmethods.cMichael Paquier
This change impacts pg_receivewal and pg_basebackup, for the pre-padding with zeros of all the new non-compressed WAL segments, so as the code is more robust on partial writes. This makes the code consistent with the backend (XLogFileInitInternal) when wal_init_zeros is enabled for the WAL segment initialization. Author: Bharath Rupireddy Reviewed-by: Nathan Bossart, Andres Freund, Thomas Munro, Michael Paquier Discussion: https://postgr.es/m/CALj2ACUq7nAb7=bJNbK3yYmp-SZhJcXFR_pLk8un6XgDzDF3OA@mail.gmail.com
2022-09-19walmethods.c/h: Make WalWriteMethod more object-oriented.Robert Haas
Normally when we use object-oriented programming techniques, we provide a pointer to an object and then some way of looking up the associated table of callbacks, but walmethods.c/h took the alternative approach of providing only a pointer to the table of callbacks and thus imposed the artificial restriction that there could only ever be one object of each type, so that the callbacks could find it via a global variable. That doesn't seem like the right idea, so revise the approach. Each callback which does not already have an argument of type Walfile * now takes a pointer to the relevant WalWriteMethod * so that these callbacks need not rely on there being only one object of each type. Freeing a WalWriteMethod is now performed via a callback provided for that purpose rather than requiring the caller to know which WAL method they want to free. Discussion: http://postgr.es/m/CA+TgmoZS0Kw98fOoAcGz8B9iDhdqB4Be4e=vDZaJZ5A-xMYBqA@mail.gmail.com
2022-09-19walmethods.c/h: Make Walfile a struct, rather than a void *Robert Haas
This makes the curent file position and pathname visible in a generic way, so we no longer need current_walfile_name global variable or the get_current_pos() method. Since that purported to be able to fail but never actually did, this also lets us get rid of some unnecessary error-handling code. One risk of this change is that the get_current_pos() method previously cleared the error indicator, and that will no longer happen with the new approach. I looked for a way that this could cause problems and did not find one. The previous code was confused about whether "Walfile" was the implementation-dependent structure representing a WAL file or whether it was a pointer to that stucture. Some of the code used it one way, and some in the other. The compiler tolerated that because void * is interchangeable with void **, but now that Walfile is a struct, it's necessary to be consistent. Hence, some references to "Walfile" have been converted to "Walfile *". Discussion: http://postgr.es/m/CA+TgmoZS0Kw98fOoAcGz8B9iDhdqB4Be4e=vDZaJZ5A-xMYBqA@mail.gmail.com
2022-09-14Fix incorrect value for "strategy" with deflateParams() in walmethods.cMichael Paquier
The zlib documentation mentions the values supported for the compression strategy, but this code has been using a hardcoded value of 0 rather than Z_DEFAULT_STRATEGY. This commit adjusts the code to use Z_DEFAULT_STRATEGY. Backpatch down to where this code has been added to ease the backport of any future patch touching this area. Reported-by: Tom Lane Discussion: https://postgr.es/m/1400032.1662217889@sss.pgh.pa.us Backpatch-through: 10
2022-07-16Replace many MemSet calls with struct initializationPeter Eisentraut
This replaces all MemSet() calls with struct initialization where that is easily and obviously possible. (For example, some cases have to worry about padding bits, so I left those.) (The same could be done with appropriate memset() calls, but this patch is part of an effort to phase out MemSet(), so it doesn't touch memset() calls.) Reviewed-by: Ranier Vilela <ranier.vf@gmail.com> Reviewed-by: Alvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://www.postgresql.org/message-id/9847b13c-b785-f4e2-75c3-12ec77a3b05c@enterprisedb.com
2022-07-03Remove redundant null pointer checks before pg_free()Peter Eisentraut
These are especially useless because the whole point of pg_free() was to do that very check before calling free(). pg_free() could be removed altogether, but I'm keeping it here to keep the API consistent. Discussion: https://www.postgresql.org/message-id/flat/dac5d2d0-98f5-94d9-8e69-46da2413593d%40enterprisedb.com
2022-04-18Handle compression level in pg_receivewal for LZ4Michael Paquier
The new option set of pg_receivewal introduced in 042a923 to control the compression method makes it now easy to pass down various options, including the compression level. The change to be able to do is simple, and requires one LZ4F_preferences_t fed to LZ4F_compressBegin(). Note that LZ4F_INIT_PREFERENCES could be used to initialize the contents of LZ4F_preferences_t as required by LZ4, but this is only available since v1.8.3. memset()'ing its structure to 0 is enough. Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-12Remove WalCompressionMethod in favor of pg_compress_algorithmMichael Paquier
The same structure, with the same set of elements (for none, lz4, gzip and zstd), exists in compression.h, so let's make use of the centralized version instead of duplicating things. Some of the variables used previously for WalCompressionMethod are renamed to stick better with the new structure and routine names. WalCompressionMethod was leading to some confusion in walmethods.c, as it was sometimes used to refer to some data unrelated to WAL. Reported-by: Robert Haas Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
2022-04-08Improve frontend error logging style.Tom Lane
Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
2022-03-15Change HAVE_LIBLZ4 and HAVE_LIBZSTD tests to USE_LZ4 and USE_ZSTD.Robert Haas
These tests were added recently, but older code tests USE_LZ4 rathr than HAVE_LIBLZ4, so let's follow the established precedent. It also seems more consistent with the intent of the configure tests, since I think that the USE_* symbols are intended to correspond to what the user requested, and the HAVE_* symbols to what configure found while probing. Discussion: http://postgr.es/m/CA+Tgmoap+hTD2-QNPJLH4tffeFE8MX5+xkbFKMU3FKBy=ZSNKA@mail.gmail.com
2022-01-23pg_basebackup: Skip a few more fsyncs if --no-sync is specified.Andres Freund
This is mostly interesting for running the regression tests on machines with slow / overloaded IO. Discussion: https://postgr.es/m/20220119041646.rhuo3youiqxqjmo2@alap3.anarazel.de
2022-01-07Update copyright for 2022Bruce Momjian
Backpatch-through: 10
2022-01-07Refactor tar method of walmethods.c to rely on the compression methodMichael Paquier
Since d62bcc8, the directory method of walmethods.c uses the compression method to determine which code path to take. The tar method, used by pg_basebackup --format=t, was inconsistent regarding that, as it relied on the compression level to check if no compression or gzip should be used. This commit makes the code more consistent as a whole in this file, making the tar logic use a compression method rather than assigning COMPRESSION_NONE that would be ignored. The options of pg_basebackup are planned to be reworked but we are not sure yet of the shape they should have as this has some dependency with the integration of the server-side compression for base backups, so this is left out for the moment. This change has as benefit to make easier the future integration of new compression methods for the tar method of walmethods.c, for the client-side compression. Reviewed-by: Georgios Kokolatos Discussion: https://postgr.es/m/Yb3GEgWwcu4wZDuA@paquier.xyz
2021-11-24Remove useless LZ4 system call on failure when writing file headerMichael Paquier
If an error occurs when writing the LZ4 file header, LZ4F_compressEnd() was called in the error code path of write(), followed by LZ4F_freeCompressionContext() to finish the cleanup. The code as-is was not broken, but the LZ4F_compressEnd() proves to not be necessary as there are no contents to flush at this stage, so remove it. Per gripe from Jeevan Ladhe and Robert Haas. Discussion: https://postgr.es/m/CAOgcT0PE33wbD7giAT1OSkNJt=p-vu8huq++qh=ny9O=SCP5aA@mail.gmail.com
2021-11-17Clean up error handling in pg_basebackup's walmethods.c.Tom Lane
The error handling here was a mess, as a result of a fundamentally bad design (relying on errno to keep its value much longer than is safe to assume) as well as a lot of just plain sloppiness, both as to noticing errors at all and as to reporting the correct errno. Moreover, the recent addition of LZ4 compression broke things completely, because liblz4 doesn't use errno to report errors. To improve matters, keep the error state in the DirectoryMethodData or TarMethodData struct, and add a string field so we can handle cases that don't set errno. (The tar methods already had a version of this, but it can be done more efficiently since all these cases use a constant error string.) Make the dir and tar methods handle errors in basically identical ways, which they didn't before. This requires copying errno into the state struct in a lot of places, which is a bit tedious, but it has the virtue that we can get rid of ad-hoc code to save and restore errno in a number of places ... not to mention that it fixes other places that should've saved/restored errno but neglected to. In passing, fix some pointlessly static buffers to be ordinary local variables. There remains an issue about exactly how to handle errors from fsync(), but that seems like material for its own patch. While the LZ4 problems are new, all the rest of this is fixes for old bugs, so backpatch to v10 where walmethods.c was introduced. Patch by me; thanks to Michael Paquier for review. Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us
2021-11-17Handle close() failures more robustly in pg_dump and pg_basebackup.Tom Lane
Coverity complained that applying get_gz_error after a failed gzclose, as we did in one place in pg_basebackup, is unsafe. I think it's right: it's entirely likely that the call is touching freed memory. Change that to inspect errno, as we do for other gzclose calls. Also, be careful to initialize errno to zero immediately before any gzclose() call where we care about the error status. (There are some calls where we don't, because we already failed at some previous step.) This ensures that we don't get a misleadingly irrelevant error code if gzclose() fails in a way that doesn't set errno. We could work harder at that, but it looks to me like all such cases are basically can't-happen if we're not misusing zlib, so it's not worth the extra notational cruft that would be required. Also, fix several places that simply failed to check for close-time errors at all, mostly at some remove from the close or gzclose itself; and one place that did check but didn't bother to report the errno. Back-patch to v12. These mistakes are older than that, but between the frontend logging API changes that happened in v12 and the fact that frontend code can't rely on %m before that, the patch would need substantial revision to work in older branches. It doesn't quite seem worth the trouble given the lack of related field complaints. Patch by me; thanks to Michael Paquier for review. Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us
2021-11-05Add support for LZ4 compression in pg_receivewalMichael Paquier
pg_receivewal gains a new option, --compression-method=lz4, available when the code is compiled with --with-lz4. Similarly to gzip, this gives the possibility to compress archived WAL segments with LZ4. This option is not compatible with --compress. The implementation uses LZ4 frames, and is compatible with simple lz4 commands. Like gzip, using --synchronous ensures that any data will be flushed to disk within the current .partial segment, so as it is possible to retrieve as much WAL data as possible even from a non-completed segment (this requires completing the partial file with zeros up to the WAL segment size supported by the backend after decompression, but this is the same as gzip). The calculation of the streaming start LSN is able to transparently find and check LZ4-compressed segments. Contrary to gzip where the uncompressed size is directly stored in the object read, the LZ4 chunk protocol does not store the uncompressed data by default. There is contentSize that can be used with LZ4 frames by that would not help if using an archive that includes segments compressed with the defaults of a "lz4" command, where this is not stored. So, this commit has taken the most extensible approach by decompressing the already-archived segment to check its uncompressed size, through a blank output buffer in chunks of 64kB (no actual performance difference noticed with 8kB, 16kB or 32kB, and the operation in itself is actually fast). Tests have been added to verify the creation and correctness of the generated LZ4 files. The latter is achieved by the use of command "lz4", if found in the environment. The tar-based WAL method in walmethods.c, used now only by pg_basebackup, does not know yet about LZ4. Its code could be extended for this purpose. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-11-04Rework compression options of pg_receivewalMichael Paquier
pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
2021-07-26Fix a couple of memory leaks in src/bin/pg_basebackup/Michael Paquier
These have been introduced by 7fbe0c8, and could happen for pg_basebackup and pg_receivewal. Per report from Coverity for the ones in walmethods.c, I have spotted the ones in receivelog.c after more review. Backpatch-through: 10
2021-07-20Fix some issues with WAL segment opening for pg_receivewal --compressMichael Paquier
The logic handling the opening of new WAL segments was fuzzy when using --compress if a partial, non-compressed, segment with the same base name existed in the repository storing those files. In this case, using --compress would cause the code to first check for the existence and the size of a non-compressed segment, followed by the opening of a new compressed, partial, segment. The code was accidentally working correctly on most platforms as the buildfarm has proved, except bowerbird where gzflush() could fail in this code path. It is wrong anyway to take the code path used pre-padding when creating a new partial, non-compressed, segment, so let's fix it. Note that this issue exists when users mix successive runs of pg_receivewal with or without compression, as discovered with the tests introduced by ffc9dda. While on it, this refactors the code so as code paths that need to know about the ".gz" suffix are down from four to one in walmethods.c, easing a bit the introduction of new compression methods. This addresses a second issue where log messages generated for an unexpected failure would not show the compressed segment name involved, which was confusing, printing instead the name of the non-compressed equivalent. Reported-by: Georgios Kokolatos Discussion: https://postgr.es/m/YPDLz2x3o1aX2wRh@paquier.xyz Backpatch-through: 10
2021-01-02Update copyright for 2021Bruce Momjian
Backpatch-through: 9.5
2020-06-15Assorted cleanup of tar-related code.Robert Haas
Introduce TAR_BLOCK_SIZE and replace many instances of 512 with the new constant. Introduce function tarPaddingBytesRequired and use it to replace numerous repetitions of (x + 511) & ~511. Add preprocessor guards against multiple inclusion to pgtar.h. Reformat the prototype for tarCreateHeader so it doesn't extend beyond 80 characters. Discussion: http://postgr.es/m/CA+TgmobWbfReO9-XFk8urR1K4wTNwqoHx_v56t7=T8KaiEoKNw@mail.gmail.com
2020-01-01Update copyrights for 2020Bruce Momjian
Backpatch-through: update all files in master, backpatch legal files through 9.4
2019-10-25Make the order of the header file includes consistent in non-backend modules.Amit Kapila
Similar to commit 7e735035f2, this commit makes the order of header file inclusion consistent for non-backend modules. In passing, fix the case where we were using angle brackets (<>) for the local module includes instead of quotes (""). Author: Vignesh C Reviewed-by: Amit Kapila Discussion: https://postgr.es/m/CALDaNm2Sznv8RR6Ex-iJO6xAdsxgWhCoETkaYX=+9DW3q0QCfA@mail.gmail.com
2019-07-29Handle fsync failures in pg_receivewal and pg_recvlogicalPeter Eisentraut
It is not safe to simply report an fsync error and continue. We must exit the program instead. Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Sehrope Sarkuni <sehrope@jackdb.com> Discussion: https://www.postgresql.org/message-id/flat/9b49fe44-8f3e-eca9-5914-29e9e99030bf@2ndquadrant.com
2019-04-01Unified logging system for command-line programsPeter Eisentraut
This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-02-13More unconstify usePeter Eisentraut
Replace casts whose only purpose is to cast away const with the unconstify() macro. Discussion: https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
2019-01-02Update copyright for 2019Bruce Momjian
Backpatch-through: certain files through 9.4
2018-09-01Avoid using potentially-under-aligned page buffers.Tom Lane
There's a project policy against using plain "char buf[BLCKSZ]" local or static variables as page buffers; preferred style is to palloc or malloc each buffer to ensure it is MAXALIGN'd. However, that policy's been ignored in an increasing number of places. We've apparently got away with it so far, probably because (a) relatively few people use platforms on which misalignment causes core dumps and/or (b) the variables chance to be sufficiently aligned anyway. But this is not something to rely on. Moreover, even if we don't get a core dump, we might be paying a lot of cycles for misaligned accesses. To fix, invent new union types PGAlignedBlock and PGAlignedXLogBlock that the compiler must allocate with sufficient alignment, and use those in place of plain char arrays. I used these types even for variables where there's no risk of a misaligned access, since ensuring proper alignment should make kernel data transfers faster. I also changed some places where we had been palloc'ing short-lived buffers, for coding style uniformity and to save palloc/pfree overhead. Since this seems to be a live portability hazard (despite the lack of field reports), back-patch to all supported versions. Patch by me; thanks to Michael Paquier for review. Discussion: https://postgr.es/m/1535618100.1286.3.camel@credativ.de
2018-08-05Reset properly errno before calling write()Michael Paquier
6cb3372 enforces errno to ENOSPC when less bytes than what is expected have been written when it is unset, though it forgot to properly reset errno before doing a system call to write(), causing errno to potentially come from a previous system call. Reported-by: Tom Lane Author: Michael Paquier Reviewed-by: Tom Lane Discussion: https://postgr.es/m/31797.1533326676@sss.pgh.pa.us
2018-06-26Correct handling of fsync failures with tar mode of walmethods.cMichael Paquier
This file has been missing the fact that it needs to report back to callers a proper failure on fsync calls. I have spotted the one in tar_finish() while Kuntal has spotted the one in tar_close(). Backpatch down to 10 where this code has been introduced. Reported by: Michael Paquier, Kuntal Ghosh Author: Michael Paquier Reviewed-by: Kuntal Ghosh, Magnus Hagander Discussion: https://postgr.es/m/20180625024356.GD1146@paquier.xyz
2018-06-25Address set of issues with errno handlingMichael Paquier
System calls mixed up in error code paths are causing two issues which several code paths have not correctly handled: 1) For write() calls, sometimes the system may return less bytes than what has been written without errno being set. Some paths were careful enough to consider that case, and assumed that errno should be set to ENOSPC, other calls missed that. 2) errno generated by a system call is overwritten by other system calls which may succeed once an error code path is taken, causing what is reported to the user to be incorrect. This patch uses the brute-force approach of correcting all those code paths. Some refactoring could happen in the future, but this is let as future work, which is not targeted for back-branches anyway. Author: Michael Paquier Reviewed-by: Ashutosh Sharma Discussion: https://postgr.es/m/20180622061535.GD5215@paquier.xyz
2018-04-07Refactor dir/file permissionsStephen Frost
Consolidate directory and file create permissions for tools which work with the PG data directory by adding a new module (common/file_perm.c) that contains variables (pg_file_create_mode, pg_dir_create_mode) and constants to initialize them (0600 for files and 0700 for directories). Convert mkdir() calls in the backend to MakePGDirectory() if the original call used default permissions (always the case for regular PG directories). Add tests to make sure permissions in PGDATA are set correctly by the tools which modify the PG data directory. Authors: David Steele <david@pgmasters.net>, Adam Brightwell <adam.brightwell@crunchydata.com> Reviewed-By: Michael Paquier, with discussion amongst many others. Discussion: https://postgr.es/m/ad346fe6-b23e-59f1-ecb7-0e08390ad629%40pgmasters.net
2018-01-02Update copyright for 2018Bruce Momjian
Backpatch-through: certain files through 9.3
2017-08-04Message style improvementsPeter Eisentraut
2017-08-02Add new files to nls.mk and add translation markersPeter Eisentraut
2017-06-30Update code comments for pg_xlog -> pg_walPeter Eisentraut
Author: Michael Paquier <michael.paquier@gmail.com>
2017-06-21Phase 3 of pgindent updates.Tom Lane
Don't move parenthesized lines to the left, even if that means they flow past the right margin. By default, BSD indent lines up statement continuation lines that are within parentheses so that they start just to the right of the preceding left parenthesis. However, traditionally, if that resulted in the continuation line extending to the right of the desired right margin, then indent would push it left just far enough to not overrun the margin, if it could do so without making the continuation line start to the left of the current statement indent. That makes for a weird mix of indentations unless one has been completely rigid about never violating the 80-column limit. This behavior has been pretty universally panned by Postgres developers. Hence, disable it with indent's new -lpl switch, so that parenthesized lines are always lined up with the preceding left paren. This patch is much less interesting than the first round of indent changes, but also bulkier, so I thought it best to separate the effects. Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us