summaryrefslogtreecommitdiff
path: root/src/bin/pg_upgrade
AgeCommit message (Collapse)Author
2017-09-22Revert "Add basic TAP test setup for pg_upgrade"Peter Eisentraut
This reverts commit f41e56c76e39f02bef7ba002c9de03d62b76de4d. The build farm client would run the pg_upgrade tests twice, once as part of the existing pg_upgrade check run and once as part of picking up all TAP tests by looking for "t" directories. Since the pg_upgrade tests are pretty slow, we will need a better solution or possibly a build farm client change before we can proceed with this.
2017-09-19Make WAL segment size configurable at initdb time.Andres Freund
For performance reasons a larger segment size than the default 16MB can be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier to write scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk less frequently. But at the same time large segment size are disadvantageous for smaller databases. So far the segment size had to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load. Therefore change it to a initdb time setting. This includes a breaking changes to the xlogreader.h API, which now requires the current segment size to be configured. For that and similar reasons a number of binaries had to be taught how to recognize the current segment size. Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion: https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
2017-09-19Add basic TAP test setup for pg_upgradePeter Eisentraut
The plan is to convert the current pg_upgrade test to the TAP framework. This commit just puts a basic TAP test in place so that we can see how the build farm behaves, since the build farm client has some special knowledge of the pg_upgrade tests. Author: Michael Paquier <michael.paquier@gmail.com>
2017-09-09pg_upgrade: Message style fixesPeter Eisentraut
2017-08-25pg_upgrade: Remove more dead codePeter Eisentraut
related to 6ce6a61840cc90172ad3da7bf303656132fa5fab Reported-by: Christoph Berg <myon@debian.org>
2017-08-24pg_upgrade: Remove dead codePeter Eisentraut
Remove code meant for upgrading to a particular version of PostgreSQL 9.0. Since pg_upgrade only supports upgrading to the current major version, this code is no longer useful.
2017-08-22pg_upgrade: Message translatability and style fixesPeter Eisentraut
2017-08-11pg_upgrade: Clarify one messagePeter Eisentraut
Reported-by: Dennis Björklund <db@zigo.dhs.org>
2017-08-07Translation updatesPeter Eisentraut
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git Source-Git-Hash: 1a0b5e655d7871506c2b1c7ba562c2de6b6a55de
2017-07-31Fix typoPeter Eisentraut
Author: Masahiko Sawada <sawada.mshk@gmail.com>
2017-07-14pg_upgrade i18n: Fix "%s server/cluster" wordingAlvaro Herrera
The original wording was impossible to translate correctly. Discussion: https://postgr.es/m/20170523002827.lzc2jkzh2gubclqb@alvherre.pgsql
2017-07-14Fix broken link-command-line ordering for libpgfeutils.Tom Lane
In the frontend Makefiles that pull in libpgfeutils, we'd generally done it like this: LDFLAGS += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) That method is badly broken, as seen in bug #14742 from Chris Ruprecht. The -L flag for src/fe_utils ends up being placed after whatever random -L flags are in LDFLAGS already. That puts us at risk of pulling in libpgfeutils.a from some previous installation rather than the freshly built one in src/fe_utils. Also, the lack of an "override" is hazardous if someone tries to specify some LDFLAGS on the make command line. The correct way to do it is like this: override LDFLAGS := -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(LDFLAGS) so that libpgfeutils, along with libpq, libpgport, and libpgcommon, are guaranteed to be pulled in from the build tree and not from any referenced system directory, because their -L flags will appear first. In some places we'd been even lazier and done it like this: LDFLAGS += -L$(top_builddir)/src/fe_utils -lpgfeutils -lpq which is subtly wrong in an additional way: on platforms where we can't restrict the symbols exported by libpq.so, it allows libpgfeutils to latch onto libpgport and libpgcommon symbols from libpq.so, rather than directly from those static libraries as intended. This carries hazards like those explained in the comments for the libpq_pgport macro. In addition to fixing the broken libpgfeutils usages, I tried to standardize on using $(libpq_pgport) like so: override LDFLAGS := $(libpq_pgport) $(LDFLAGS) even where libpgfeutils is not in the picture. This makes no difference right now but will hopefully discourage future mistakes of the same ilk. And it's more like the way we handle CPPFLAGS in libpq-using Makefiles. In passing, just for consistency, make pgbench include PTHREAD_LIBS the same way everyplace else does, ie just after LIBS rather than in some random place in the command line. This might have practical effect if there are -L switches in that macro on some platform. It looks to me like the MSVC build scripts are not affected by this error, but someone more familiar with them than I might want to double check. Back-patch to 9.6 where libpgfeutils was introduced. In 9.6, the hazard this error creates is that a reinstallation might link to the prior installation's copy of libpgfeutils.a and thereby fail to absorb a minor-version bug fix. Discussion: https://postgr.es/m/20170714125106.9231.13772@wrigleys.postgresql.org
2017-06-28Change pg_ctl to detect server-ready by watching status in postmaster.pid.Tom Lane
Traditionally, "pg_ctl start -w" has waited for the server to become ready to accept connections by attempting a connection once per second. That has the major problem that connection issues (for instance, a kernel packet filter blocking traffic) can't be reliably told apart from server startup issues, and the minor problem that if server startup isn't quick, we accumulate "the database system is starting up" spam in the server log. We've hacked around many of the possible connection issues, but it resulted in ugly and complicated code in pg_ctl.c. In commit c61559ec3, I changed the probe rate to every tenth of a second. That prompted Jeff Janes to complain that the log-spam problem had become much worse. In the ensuing discussion, Andres Freund pointed out that we could dispense with connection attempts altogether if the postmaster were changed to report its status in postmaster.pid, which "pg_ctl start" already relies on being able to read. This patch implements that, teaching postmaster.c to report a status string into the pidfile at the same state-change points already identified as being of interest for systemd status reporting (cf commit 7d17e683f). pg_ctl no longer needs to link with libpq at all; all its functions now depend on reading server files. In support of this, teach AddToDataDirLockFile() to allow addition of postmaster.pid lines in not-necessarily-sequential order. This is needed on Windows where the SHMEM_KEY line will never be written at all. We still have the restriction that we don't want to truncate the pidfile; document the reasons for that a bit better. Also, fix the pg_ctl TAP tests so they'll notice if "start -w" mode is broken --- before, they'd just wait out the sixty seconds until the loop gives up, and then report success anyway. (Yes, I found that out the hard way.) While at it, arrange for pg_ctl to not need to #include miscadmin.h; as a rather low-level backend header, requiring that to be compilable client-side is pretty dubious. This requires moving the #define's associated with the pidfile into a new header file, and moving PG_BACKEND_VERSIONSTR someplace else. For lack of a clearly better "someplace else", I put it into port.h, beside the declaration of find_other_exec(), since most users of that macro are passing the value to find_other_exec(). (initdb still depends on miscadmin.h, but at least pg_ctl and pg_upgrade no longer do.) In passing, fix main.c so that PG_BACKEND_VERSIONSTR actually defines the output of "postgres -V", which remarkably it had never done before. Discussion: https://postgr.es/m/CAMkU=1xJW8e+CTotojOMBd-yzUvD0e_JZu2xHo=MnuZ4__m7Pg@mail.gmail.com
2017-06-21Phase 3 of pgindent updates.Tom Lane
Don't move parenthesized lines to the left, even if that means they flow past the right margin. By default, BSD indent lines up statement continuation lines that are within parentheses so that they start just to the right of the preceding left parenthesis. However, traditionally, if that resulted in the continuation line extending to the right of the desired right margin, then indent would push it left just far enough to not overrun the margin, if it could do so without making the continuation line start to the left of the current statement indent. That makes for a weird mix of indentations unless one has been completely rigid about never violating the 80-column limit. This behavior has been pretty universally panned by Postgres developers. Hence, disable it with indent's new -lpl switch, so that parenthesized lines are always lined up with the preceding left paren. This patch is much less interesting than the first round of indent changes, but also bulkier, so I thought it best to separate the effects. Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21Phase 2 of pgindent updates.Tom Lane
Change pg_bsd_indent to follow upstream rules for placement of comments to the right of code, and remove pgindent hack that caused comments following #endif to not obey the general rule. Commit e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 wasn't actually using the published version of pg_bsd_indent, but a hacked-up version that tried to minimize the amount of movement of comments to the right of code. The situation of interest is where such a comment has to be moved to the right of its default placement at column 33 because there's code there. BSD indent has always moved right in units of tab stops in such cases --- but in the previous incarnation, indent was working in 8-space tab stops, while now it knows we use 4-space tabs. So the net result is that in about half the cases, such comments are placed one tab stop left of before. This is better all around: it leaves more room on the line for comment text, and it means that in such cases the comment uniformly starts at the next 4-space tab stop after the code, rather than sometimes one and sometimes two tabs after. Also, ensure that comments following #endif are indented the same as comments following other preprocessor commands such as #else. That inconsistency turns out to have been self-inflicted damage from a poorly-thought-through post-indent "fixup" in pgindent. This patch is much less interesting than the first round of indent changes, but also bulkier, so I thought it best to separate the effects. Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21Initial pgindent run with pg_bsd_indent version 2.0.Tom Lane
The new indent version includes numerous fixes thanks to Piotr Stefaniak. The main changes visible in this commit are: * Nicer formatting of function-pointer declarations. * No longer unexpectedly removes spaces in expressions using casts, sizeof, or offsetof. * No longer wants to add a space in "struct structname *varname", as well as some similar cases for const- or volatile-qualified pointers. * Declarations using PG_USED_FOR_ASSERTS_ONLY are formatted more nicely. * Fixes bug where comments following declarations were sometimes placed with no space separating them from the code. * Fixes some odd decisions for comments following case labels. * Fixes some cases where comments following code were indented to less than the expected column 33. On the less good side, it now tends to put more whitespace around typedef names that are not listed in typedefs.list. This might encourage us to put more effort into typedef name collection; it's not really a bug in indent itself. There are more changes coming after this round, having to do with comment indentation and alignment of lines appearing within parentheses. I wanted to limit the size of the diffs to something that could be reviewed without one's eyes completely glazing over, so it seemed better to split up the changes as much as practical. Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-21Final pgindent run with old pg_bsd_indent (version 1.3).Tom Lane
This is just to have a clean basis for comparison with the results of the new version (which will indeed end up reverting some of these changes...) Discussion: https://postgr.es/m/E1dAmxK-0006EE-1r@gemulon.postgresql.org Discussion: https://postgr.es/m/30527.1495162840@sss.pgh.pa.us
2017-06-20pg_upgrade: start/stop new server after pg_resetwalBruce Momjian
When commit 0f33a719fdbb5d8c43839ea0d2c90cd03e2af2d2 removed the instructions to start/stop the new cluster before running rsync, it was now possible for pg_resetwal/pg_resetxlog to leave the final WAL record at wal_level=minimum, preventing upgraded standby servers from reconnecting. This patch fixes that by having pg_upgrade unconditionally start/stop the new cluster after pg_resetwal/pg_resetxlog has run. Backpatch through 9.2 since, though the instructions were added in PG 9.5, they worked all the way back to 9.2. Discussion: https://postgr.es/m/20170620171844.GC24975@momjian.us Backpatch-through: 9.2
2017-06-08Fix bit-rot in pg_upgrade's test.sh, and improve documentation.Tom Lane
Doing a cross-version upgrade test with test.sh evidently hasn't been tested since circa 9.2, because the script lacked case branches for old-version servers newer than 9.1. Future-proof that a bit, and clean up breakage induced by our recent drop of V0 function call protocol (namely that oldstyle_length() isn't in the regression suite anymore). (This isn't enough to make the test work perfectly cleanly across versions, but at least it finishes and provides dump files that you can diff manually. One issue I didn't touch is that we might want to execute the "reindex_hash.sql" file in the new DB before dumping it, so that the hash indexes don't vanish from the dump.) Improve the TESTING doc file: put the tl;dr version at the top not the bottom, and bring its explanation of how to run a cross-version test up to speed, since the installcheck target isn't there and won't be resurrected. Improve the comment in the Makefile about why not. In passing, teach .gitignore and "make clean" about a couple more junk output files. Discussion: https://postgr.es/m/14058.1496892482@sss.pgh.pa.us
2017-05-19pg_upgrade: Handle hash index upgrades more smoothly.Robert Haas
Mark any old hash indexes as invalid so that they don't get used, and create a script to run REINDEX on all of them. Without this, we'd still try to use any upgraded hash indexes, but it would fail. Amit Kapila, reviewed by me. Per a suggestion from Tom Lane. Discussion: http://postgr.es/m/CAA4eK1Jidtagm7Q81q-WoegOVgkotv0OxvHOjFxcvFRP4X=mSw@mail.gmail.com
2017-05-17Post-PG 10 beta1 pgindent runBruce Momjian
perltidy run not included.
2017-05-15Translation updatesPeter Eisentraut
Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git Source-Git-Hash: 398beeef4921df0956f917becd7b5669d2a8a5c4
2017-05-05Fix wording in pg_upgrade docsMagnus Hagander
Author: Daniel Gustafsson
2017-04-10Run most pg_dump and pg_dumpall tests with --no-syncAndrew Dunstan
Commit 96a7128b made pg_dump and pg_dumpall sync their output by default. However, there's no great need for that in testing, and it could impose a performance penalty, so we add the --no-sync flag to most of the test cases. Michael Paquier
2017-03-17Rename "pg_clog" directory to "pg_xact".Robert Haas
Names containing the letters "log" sometimes confuse users into believing that only non-critical data is present. It is hoped this renaming will discourage ill-considered removals of transaction status data. Michael Paquier Discussion: http://postgr.es/m/CA+Tgmoa9xFQyjRZupbdEFuwUerFTvC6HjZq1ud6GYragGDFFgA@mail.gmail.com
2017-03-09Fix hard-coded relkind constants in assorted src/bin files.Tom Lane
Although it's reasonable to expect that most of these constants will never change, that does not make it good programming style to hard-code the value rather than using the RELKIND_FOO macros. Discussion: https://postgr.es/m/11145.1488931324@sss.pgh.pa.us
2017-03-01Move atooid() definition to a central placePeter Eisentraut
2017-02-25Remove useless duplicate inclusions of system header files.Tom Lane
c.h #includes a number of core libc header files, such as <stdio.h>. There's no point in re-including these after having read postgres.h, postgres_fe.h, or c.h; so remove code that did so. While at it, also fix some places that were ignoring our standard pattern of "include postgres[_fe].h, then system header files, then other Postgres header files". While there's not any great magic in doing it that way rather than system headers last, it's silly to have just a few files deviating from the general pattern. (But I didn't attempt to enforce this globally, only in files I was touching anyway.) I'd be the first to say that this is mostly compulsive neatnik-ism, but over time it might save enough compile cycles to be useful.
2017-02-15pg_upgrade: Fix problems caused by renaming pg_resetxlog.Robert Haas
Commit 85c11324cabaddcfaf3347df78555b30d27c5b5a renamed pg_resetxlog to pg_resetwal, but didn't make pg_upgrade smart enough to cope with the situation. Michael Paquier, per a complaint from Jeff Janes
2017-02-09Rename user-facing tools with "xlog" in the name to say "wal".Robert Haas
This means pg_receivexlog because pg_receivewal, pg_resetxlog becomes pg_resetwal, and pg_xlogdump becomes pg_waldump.
2017-01-25Change unknown-type literals to type text in SELECT and RETURNING lists.Tom Lane
Previously, we left such literals alone if the query or subquery had no properties forcing a type decision to be made (such as an ORDER BY or DISTINCT clause using that output column). This meant that "unknown" could be an exposed output column type, which has never been a great idea because it could result in strange failures later on. For example, an outer query that tried to do any operations on an unknown-type subquery output would generally fail with some weird error like "failed to find conversion function from unknown to text" or "could not determine which collation to use for string comparison". Also, if the case occurred in a CREATE VIEW's query then the view would have an unknown-type column, causing similar failures in queries trying to use the view. To fix, at the tail end of parse analysis of a query, forcibly convert any remaining "unknown" literals in its SELECT or RETURNING list to type text. However, provide a switch to suppress that, and use it in the cases of SELECT inside a set operation or INSERT command. In those cases we already had type resolution rules that make use of context information from outside the subquery proper, and we don't want to change that behavior. Also, change creation of an unknown-type column in a relation from a warning to a hard error. The error should be unreachable now in CREATE VIEW or CREATE MATVIEW, but it's still possible to explicitly say "unknown" in CREATE TABLE or CREATE (composite) TYPE. We want to forbid that because it's nothing but a foot-gun. This change creates a pg_upgrade failure case: a matview that contains an unknown-type column can't be pg_upgraded, because reparsing the matview's defining query will now decide that the column is of type text, which doesn't match the cstring-like storage that the old materialized column would actually have. Add a checking pass to detect that. While at it, we can detect tables or composite types that would fail, essentially for free. Those would fail safely anyway later on, but we might as well fail earlier. This patch is by me, but it owes something to previous investigations by Rahila Syed. Also thanks to Ashutosh Bapat and Michael Paquier for review. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com
2017-01-21Fix cross-shlib linking in temporary installs on HPUX 10.Tom Lane
Turns out this has been broken for years and we'd not noticed. The one case that was getting exercised in the buildfarm, or probably anywhere else, was postgres_fdw.sl's reference to libpq.sl; and it turns out that that was always going to libpq.sl in the actual installation directory not the temporary install. We'd not noticed because the buildfarm script does "make install" before it tests contrib. However, the recent addition of a logical-replication test to the core regression scripts resulted in trying to use libpqwalreceiver.sl before "make install" happens, and that failed for lack of finding libpq.sl, as shown by failures on buildfarm members gaur and pademelon. There are two changes needed to fix it: the magic environment variable to specify shlib search path at runtime is SHLIB_PATH not LD_LIBRARY_PATH, and the shlib link command needs to specify the +s switch else the library will not honor SHLIB_PATH. I'm not quite sure why buildfarm members anole and gharial (HPUX 11) didn't show the same failure. Consulting man pages on the web says that HPUX 11 honors both LD_LIBRARY_PATH and SHLIB_PATH, which would explain half of it, and the rather confusing wording I've been able to find suggests that +s might effectively be the default in HPUX 11. But it seems at least as likely that there's just a libpq.so installed in /usr/lib on that machine; as long as it's not too ancient, that would satisfy the test. In any case I do not think this patch will break HPUX 11. At the moment I don't see a need to back-patch this, since it only matters for testing purposes, not to mention that HPUX 10 is probably dead in the real world anyway.
2017-01-13pg_upgrade: Fix for changed pg_ctl default stop modePeter Eisentraut
In 9.5, the default pg_ctl stop mode was changed from "smart" to "fast". pg_upgrade still thought the default mode was "smart" and only specified the mode when "fast" was asked for. This results in using "fast" all the time. It's not clear what the effect in practice is, but fix it nonetheless to restore the previous behavior.
2017-01-03Update copyright via script for 2017Bruce Momjian
2016-11-13pg_upgrade: Upgrade sequence data via pg_dumpPeter Eisentraut
Previously, pg_upgrade migrated sequence data like tables by copying the on-disk file. This does not allow any changes in the on-disk format for sequences. It's simpler to just have pg_dump set the new sequence values as it normally does. To do that, create a hidden submode in pg_dump that dumps sequence data even when a schema-only dump is requested, and trigger that submode in binary upgrade mode. (This new submode could easily be exposed as a command-line option, but it has limited use outside of pg_dump and would probably cause some confusion, so we don't do that at this time.) Reviewed-by: Anastasia Lubennikova <a.lubennikova@postgrespro.ru> Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2016-11-07pg_upgrade: Add NLSPeter Eisentraut
Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
2016-11-07pg_rewing pg_upgrade: Fix translation markersPeter Eisentraut
In pg_log_v(), we need to translate the fmt before processing, not the formatted message afterwards.
2016-10-20Rename "pg_xlog" directory to "pg_wal".Robert Haas
"xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier
2016-10-03Enforce a specific order for probing library loadability in pg_upgrade.Tom Lane
pg_upgrade checks whether all the shared libraries used in the old cluster are also available in the new one by issuing LOAD for each library name. Previously, it cared not what order it did the LOADs in. Ideally it should not have to care, but currently the transform modules in contrib fail unless both the language and datatype modules they depend on are loaded first. A backend-side solution for that looks possible but probably not back-patchable, so as a stopgap measure, let's do the LOAD tests in order by library name length. That should fix the problem for reasonably-named transform modules, eg "hstore_plpython" will be loaded after both "hstore" and "plpython". (Yeah, it's a hack.) In a larger sense, having a predictable order of these probes is a good thing, since it will make upgrades predictably work or not work in the face of inter-library dependencies. Also, this patch replaces O(N^2) de-duplication logic with O(N log N) logic, which could matter in installations with very many databases. So I don't foresee reverting this even after we have a proper fix for the library-dependency problem. In passing, improve a couple of SQL queries used here. Per complaint from Andrew Dunstan that pg_upgrade'ing the transform contrib modules failed. Back-patch to 9.5 where transform modules were introduced. Discussion: <f7ac29f3-515c-2a44-21c5-ec925053265f@dunslane.net>
2016-09-30Improve error reporting in pg_upgrade's file copying/linking/rewriting.Tom Lane
The previous design for this had copyFile(), linkFile(), and rewriteVisibilityMap() returning strerror strings, with the caller producing one-size-fits-all error messages based on that. This made it impossible to produce messages that described the failures with any degree of precision, especially not short-read problems since those don't set errno at all. Since pg_upgrade has no intention of continuing after any error in this area, let's fix this by just letting these functions call pg_fatal() for themselves, making it easy for each point of failure to have a suitable error message. Taking this approach also allows dropping cleanup code that was unnecessary and was often rather sloppy about preserving errno. To not lose relevant info that was reported before, pass in the schema name and table name of the current table so that they can be included in the error reports. An additional problem was the use of getErrorText(), which was flat out wrong for all but a couple of call sites, because it unconditionally did "_dosmaperr(GetLastError())" on Windows. That's only appropriate when reporting an error from a Windows-native API, which only a couple of the callers were actually doing. Thus, even the reported strerror string would be unrelated to the actual failure in many cases on Windows. To fix, get rid of getErrorText() altogether, and just have call sites do strerror(errno) instead, since that's the way all the rest of our frontend programs do it. Add back the _dosmaperr() calls in the two places where that's actually appropriate. In passing, make assorted messages hew more closely to project style guidelines, notably by removing initial capitals in not-complete-sentence primary error messages. (I didn't make any effort to clean up places I didn't have another reason to touch, though.) Per discussion of a report from Thomas Kellerer. Back-patch to 9.6, but no further; given the relative infrequency of reports of problems here, it's not clear it's worth adapting the patch to older branches. Patch by me, but with credit to Alvaro Herrera for spotting the issue with getErrorText's misuse of _dosmaperr(). Discussion: <nsjrbh$8li$1@blaine.gmane.org>
2016-09-30Fix multiple portability issues in pg_upgrade's rewriteVisibilityMap().Tom Lane
This is new code in 9.6, and evidently we missed out testing it as thoroughly as it should have been. Bugs fixed here: 1. Use binary not text mode to open the files on Windows. Before, if the visibility map chanced to contain two bytes that looked like \r\n, Windows' read() would convert that to \n, which both corrupts the map data and causes the file to look shorter than it should. Unless you were *very* unlucky and had an exact multiple of 8K such occurrences in each VM file, this would cause pg_upgrade to report a failure, though with a rather obscure error message. 2. The code for copying rebuilt bytes into the output was simply wrong. It chanced to work okay on little-endian machines but would emit the bytes in the wrong order on big-endian, leading to silent corruption of the visibility map data. 3. The code was careless about alignment of the working buffers. Given all three of an alignment-picky architecture, a compiler that chooses to put the new_vmbuf[] local variable at an odd starting address, and a checksum-enabled database, pg_upgrade would dump core. Point one was reported by Thomas Kellerer, the other two detected by code-reading. Point two is much the nastiest of these issues from an impact standpoint, though fortunately it affects only a minority of users. The Windows issue will definitely bite people, but it seems quite unlikely that there would be undetected corruption from that. In addition, I failed to resist the temptation to do some minor cosmetic adjustments, mostly improving the comments. It would be a good idea to try to improve the error reporting here, but that seems like material for a separate patch. Discussion: <nsjrbh$8li$1@blaine.gmane.org>
2016-08-15Stamp HEAD as 10devel.Tom Lane
This is a good bit more complicated than the average new-version stamping commit, because it includes various adjustments in pursuit of changing from three-part to two-part version numbers. It's likely some further work will be needed around that change; but this is enough to get through the regression tests, at least in Unix builds. Peter Eisentraut and Tom Lane
2016-08-08Obstruct shell, SQL, and conninfo injection via database and role names.Noah Misch
Due to simplistic quoting and confusion of database names with conninfo strings, roles with the CREATEDB or CREATEROLE option could escalate to superuser privileges when a superuser next ran certain maintenance commands. The new coding rule for PQconnectdbParams() calls, documented at conninfo_array_parse(), is to pass expand_dbname=true and wrap literal database names in a trivial connection string. Escape zero-length values in appendConnStrVal(). Back-patch to 9.1 (all supported versions). Nathan Bossart, Michael Paquier, and Noah Misch. Reviewed by Peter Eisentraut. Reported by Nathan Bossart. Security: CVE-2016-5424
2016-07-28Register atexit hook only once in pg_upgrade.Tom Lane
start_postmaster() registered stop_postmaster_atexit as an atexit(3) callback each time through, although the obvious intention was to do so only once per program run. The extra registrations were harmless, so long as we didn't exceed ATEXIT_MAX, but still it's a bug. Artur Zakirov, with bikeshedding by Kyotaro Horiguchi and me Discussion: <d279e817-02b5-caa6-215f-cfb05dce109a@postgrespro.ru>
2016-06-16Reword bogus commentAlvaro Herrera
2016-06-16Remove unused prototypeAlvaro Herrera
Commit 6f56b41ac0cd7 removed function get_pg_database_relfilenode but left its prototype in place. Remove it.
2016-06-09pgindent run for 9.6Robert Haas
2016-06-06pg_upgrade: Don't overwrite existing files.Robert Haas
For historical reasons, copyFile and rewriteVisibilityMap took a force argument which was always passed as true, meaning that any existing file should be overwritten. However, it seems much safer to instead fail if a file we need to write already exists. While we're at it, remove the "force" argument altogether, since it was never passed as anything other than true (and now we would never pass it as anything other than false, if we kept it). Noted by Andres Freund during post-commit review of the patch that added rewriteVisibilityMap, commit 7087166a88fe0c04fc6636d0d6d6bea1737fc1fb, but this also changes the behavior when copying files without rewriting them. Patch by Masahiko Sawada.
2016-06-06pg_upgrade: Improve error checking in rewriteVisibilityMap.Robert Haas
In the old logic, if read() were to return an error, we'd silently stop rewriting the visibility map at that point in the file. That's safe, but reporting the error is better, so do that instead. Report by Andres Freund. Patch by Masahiko Sawada, with one correction by me.
2016-05-06Fix pg_upgrade to not fail when new-cluster TOAST rules differ from old.Tom Lane
This patch essentially reverts commit 4c6780fd17aa43ed, in favor of a much simpler solution for the case where the new cluster would choose to create a TOAST table but the old cluster doesn't have one: just don't create a TOAST table. The existing code failed in at least two different ways if the situation arose: (1) ALTER TABLE RESET didn't grab an exclusive lock, so that the lock sanity check in create_toast_table failed; (2) pg_upgrade did not provide a pg_type OID for the new toast table, so that the crosscheck in TypeCreate failed. While both these problems were introduced by later patches, they show that the hack being used to cause TOAST table creation is overwhelmingly fragile (and untested). I also note that before the TypeCreate crosscheck was added, the code would have resulted in assigning an indeterminate pg_type OID to the toast table, possibly causing a later OID conflict in that catalog; so that it didn't really work even when committed. If we simply don't create a TOAST table, there will only be a problem if the code tries to store a tuple that's wider than a page, and field compression isn't sufficient to get it under a page. Given that the TOAST creation threshold is intended to be about a quarter of a page, it's very hard to believe that cross-version differences in the do-we-need-a-toast- table heuristic could result in an observable problem. So let's just follow the old version's conclusion about whether a TOAST table is needed. (If we ever do change needs_toast_table() so much that this conclusion doesn't apply, we can devise a solution at that time, and hopefully do it in a less klugy way than 4c6780fd17aa43ed did.) Back-patch to 9.3, like the previous patch. Discussion: <8110.1462291671@sss.pgh.pa.us>