summaryrefslogtreecommitdiff
path: root/contrib/pg_upgrade
AgeCommit message (Collapse)Author
2013-03-22Allow I/O reliability checks using 16-bit checksumsSimon Riggs
Checksums are set immediately prior to flush out of shared buffers and checked when pages are read in again. Hint bit setting will require full page write when block is dirtied, which causes various infrastructure changes. Extensive comments, docs and README. WARNING message thrown if checksum fails on non-all zeroes page; ERROR thrown but can be disabled with ignore_checksum_failure = on. Feature enabled by an initdb option, since transition from option off to option on is long and complex and has not yet been implemented. Default is not to use checksums. Checksum used is WAL CRC-32 truncated to 16-bits. Simon Riggs, Jeff Davis, Greg Smith Wide input and assistance from many community members. Thank you.
2013-03-03Add a materialized view relations.Kevin Grittner
A materialized view has a rule just like a view and a heap and other physical properties like a table. The rule is only used to populate the table, references in queries refer to the materialized data. This is a minimal implementation, but should still be useful in many cases. Currently data is only populated "on demand" by the CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW statements. It is expected that future releases will add incremental updates with various timings, and that a more refined concept of defining what is "fresh" data will be developed. At some point it may even be possible to have queries use a materialized in place of references to underlying tables, but that requires the other above-mentioned features to be working first. Much of the documentation work by Robert Haas. Review by Noah Misch, Thom Brown, Robert Haas, Marko Tiikkaja Security review by KaiGai Kohei, with a decision on how best to implement sepgsql still pending.
2013-02-28Improve pg_upgrade commentary on multixact changeAlvaro Herrera
Per gripe from Andres Freund
2013-02-14pg_upgrade: conditionally create cluster delete scriptBruce Momjian
If users create tablespaces inside the old cluster directory, it is impossible for the delete script to delete _only_ the old cluster files, so don't create a script in that case, and issue a message to the user.
2013-02-14Fix pg_upgrade log file cleanup codeBruce Momjian
Recent pg_upgrade parallel improvements introduced a bug that prevented cleanup of per-database log files.
2013-02-12Create libpgcommon, and move pg_malloc et al to itAlvaro Herrera
libpgcommon is a new static library to allow sharing code among the various frontend programs and backend; this lets us eliminate duplicate implementations of common routines. We avoid libpgport, because that's intended as a place for porting issues; per discussion, it seems better to keep them separate. The first use case, and the only implemented by this patch, is pg_malloc and friends, which many frontend programs were already using. At the same time, we can use this to provide palloc emulation functions for the frontend; this way, some palloc-using files in the backend can also be used by the frontend cleanly. To do this, we change palloc() in the backend to be a function instead of a macro on top of MemoryContextAlloc(). This was previously believed to cause loss of performance, but this implementation has been tweaked by Tom and Andres so that on modern compilers it provides a slight improvement over the previous one. This lets us clean up some places that were already with localized hacks. Most of the pg_malloc/palloc changes in this patch were authored by Andres Freund. Zoltán Böszörményi also independently provided a form of that. libpgcommon infrastructure was authored by Álvaro.
2013-01-24pg_upgrade: detect stale postmaster.pid lock filesBruce Momjian
If the postmaster.pid lock file exists, try starting/stopping the cluster to check if the lock file is valid. Per request from Tom.
2013-01-24Use the catversion to distinguish old/new clustersAlvaro Herrera
This makes 9.3 -> 9.3 upgrades work when they cross the commit that added persistent multixacts; early 9.3 pg_controldata did not have the required oldestMultiXact line, and so would fail to upgrade. per Bruce Momjian
2013-01-24Don't require oldestMultixact if server doesn't have itAlvaro Herrera
2013-01-24pg_upgrade: report failed cluster nameBruce Momjian
When pg_upgrade can't find required pg_controldata information, report _which_ cluster is failing, with this message: The %s cluster lacks some required control information:
2013-01-23Improve concurrency of foreign key lockingAlvaro Herrera
This patch introduces two additional lock modes for tuples: "SELECT FOR KEY SHARE" and "SELECT FOR NO KEY UPDATE". These don't block each other, in contrast with already existing "SELECT FOR SHARE" and "SELECT FOR UPDATE". UPDATE commands that do not modify the values stored in the columns that are part of the key of the tuple now grab a SELECT FOR NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently with tuple locks of the FOR KEY SHARE variety. Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this means the concurrency improvement applies to them, which is the whole point of this patch. The added tuple lock semantics require some rejiggering of the multixact module, so that the locking level that each transaction is holding can be stored alongside its Xid. Also, multixacts now need to persist across server restarts and crashes, because they can now represent not only tuple locks, but also tuple updates. This means we need more careful tracking of lifetime of pg_multixact SLRU files; since they now persist longer, we require more infrastructure to figure out when they can be removed. pg_upgrade also needs to be careful to copy pg_multixact files over from the old server to the new, or at least part of multixact.c state, depending on the versions of the old and new servers. Tuple time qualification rules (HeapTupleSatisfies routines) need to be careful not to consider tuples with the "is multi" infomask bit set as being only locked; they might need to look up MultiXact values (i.e. possibly do pg_multixact I/O) to find out the Xid that updated a tuple, whereas they previously were assured to only use information readily available from the tuple header. This is considered acceptable, because the extra I/O would involve cases that would previously cause some commands to block waiting for concurrent transactions to finish. Another important change is the fact that locking tuples that have previously been updated causes the future versions to be marked as locked, too; this is essential for correctness of foreign key checks. This causes additional WAL-logging, also (there was previously a single WAL record for a locked tuple; now there are as many as updated copies of the tuple there exist.) With all this in place, contention related to tuples being checked by foreign key rules should be much reduced. As a bonus, the old behavior that a subtransaction grabbing a stronger tuple lock than the parent (sub)transaction held on a given tuple and later aborting caused the weaker lock to be lost, has been fixed. Many new spec files were added for isolation tester framework, to ensure overall behavior is sane. There's probably room for several more tests. There were several reviewers of this patch; in particular, Noah Misch and Andres Freund spent considerable time in it. Original idea for the patch came from Simon Riggs, after a problem report by Joel Jacobson. Most code is from me, with contributions from Marti Raudsepp, Alexander Shulgin, Noah Misch and Andres Freund. This patch was discussed in several pgsql-hackers threads; the most important start at the following message-ids: AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com 1290721684-sup-3951@alvh.no-ip.org 1294953201-sup-2099@alvh.no-ip.org 1320343602-sup-2290@alvh.no-ip.org 1339690386-sup-8927@alvh.no-ip.org 4FE5FF020200002500048A3D@gw.wicourts.gov 4FEAB90A0200002500048B7D@gw.wicourts.gov
2013-01-22pg_upgrade: remove --single-transaction usageBruce Momjian
With AtEOXact applied, --single-transaction makes pg_restore slower, and has the potential to require lock table configuration, so remove the argument. Per suggestion from Tom.
2013-01-18Improve pg_upgrade error reportBruce Momjian
If the cluster alignments don't match, output this suggestion: Likely one cluster is a 32-bit install, the other 64-bit
2013-01-12Extend and improve use of EXTRA_REGRESS_OPTS.Andrew Dunstan
This is now used by ecpg tests, and not clobbered by pg_upgrade tests. This change won't affect anything that doesn't set this environment variable, but will enable the buildfarm to control exactly what port regression test installs will be running on, and thus to detect possible rogue postmasters more easily. Backpatch to release 9.2 where EXTRA_REGRESS_OPTS was first used.
2013-01-09Allow parallel copy/link in pg_upgradeBruce Momjian
This patch implements parallel copying/linking of files by tablespace using the --jobs option in pg_upgrade.
2013-01-03Prevent creation of postmaster's TCP socket during pg_upgrade testing.Tom Lane
On non-Windows machines, we use the Unix socket for connections to test postmasters, so there is no need to create a TCP socket. Furthermore, doing so causes failures due to port conflicts if two builds are carried out concurrently on one machine. (If the builds are done in different chroots, which is standard practice at least in Red Hat distros, there is no risk of conflict on the Unix socket.) Suppressing the TCP socket by setting listen_addresses to empty has long been standard practice for pg_regress, and pg_upgrade knows about this too ... but pg_upgrade's test.sh didn't get the memo. Back-patch to 9.2, and also sync the 9.2 version of the script with HEAD as much as practical.
2013-01-02Adjust a few pg_upgrade functions to return void.Bruce Momjian
Adjust pg_upgrade page conversion functions (which are not used) to return void so transfer_all_new_dbs can return void.
2013-01-01Update copyrights for 2013Bruce Momjian
Fully update git head, and update back branches in ./COPYRIGHT and legal.sgml files.
2012-12-26Add pg_upgrade --jobs parameterBruce Momjian
Add pg_upgrade --jobs, which allows parallel dump/restore of databases, which improves performance.
2012-12-20Avoid using NAMEDATALEN in pg_upgradeBruce Momjian
Because the client encoding might not match the server encoding, pg_upgrade can't allocate NAMEDATALEN bytes for storage of database, relation, and namespace identifiers. Instead pg_strdup() the memory and free it. Also add C comment in initdb.c about safe NAMEDATALEN usage.
2012-12-20Add pg_upgrade comment about mismatch errorBruce Momjian
Add comment stating that constraint and index names must match.
2012-12-11Fix pg_upgrade for invalid indexesBruce Momjian
All versions of pg_upgrade upgraded invalid indexes caused by CREATE INDEX CONCURRENTLY failures and marked them as valid. The patch adds a check to all pg_upgrade versions and throws an error during upgrade or --check. Backpatch to 9.2, 9.1, 9.0. Patch slightly adjusted.
2012-12-10Fix pg_upgrade -O/-o optionsBruce Momjian
Fix previous commit that added synchronous_commit=off, but broke -O/-o due to missing space in argument passing. Backpatch to 9.2.
2012-12-07Improve pg_upgrade's status displayBruce Momjian
Pg_upgrade displays file names during copy and database names during dump/restore. Andrew Dunstan identified three bugs: * long file names were being truncated to 60 _leading_ characters, which often do not change for long file names * file names were truncated to 60 characters in log files * carriage returns were being output to log files This commit fixes these --- it prints 60 _trailing_ characters to the status display, and full path names without carriage returns to log files. It also suppresses status output to the log file unless verbose mode is used.
2012-12-04In pg_upgrade testing script, turn off command echo at the end so statusBruce Momjian
report is clearer.
2012-12-03Restore set -x in pg_upgrade/test.sh, so the user can see what is beingBruce Momjian
executed.
2012-12-03Add initdb --sync-only option to sync the data directory to durableBruce Momjian
storage. Have pg_upgrade use it, and enable server options fsync=off and full_page_writes=off. Document that users turning fsync from off to on should run initdb --sync-only. [ Previous commit was incorrectly applied as a git merge. ]
2012-12-03Revert initdb --sync-only patch that had incorrect commit messages.Bruce Momjian
2012-12-03dummy commitBruce Momjian
2012-12-03dummy commitBruce Momjian
2012-12-01In pg_upgrade, remove 'set -x' from test script.Bruce Momjian
2012-12-01Revert:Bruce Momjian
In pg_upgrade, remove pg_restore's --single-transaction option, as it throws errors in certain cases.
2012-12-01Remove pg_restore's --single-transaction option, as it throws errors inBruce Momjian
certain cases.
2012-11-30In pg_upgrade, improve status wording now that we have per-databaseBruce Momjian
status output for dump/restore.
2012-11-30In pg_upgrade, dump each database separately and useBruce Momjian
--single-transaction to restore each database schema. This yields performance improvements for databases with many tables. Also, remove split_old_dump() as it is no longer needed.
2012-11-30Clean environment for pg_upgrade test.Andrew Dunstan
This removes exisiting PG settings from the environment for pg_upgrade tests, just like pg_regress does.
2012-11-24In pg_upgrade, simplify function copy_file() by using pg_malloc() andBruce Momjian
centralizing error/shutdown code.
2012-11-24In pg_upgrade, fix a few place that used maloc/free rather thanBruce Momjian
pg_malloc/pg_free.
2012-11-19In pg_upgrade, report errno string if file existence check returns anBruce Momjian
error and errno != ENOENT.
2012-11-14In pg_upgrade, add third meaningless parameter to open().Bruce Momjian
2012-11-14In pg_upgrade, copy fsm, vm, and extent files by checking for fileBruce Momjian
existence via open(), rather than collecting a directory listing and looking up matching relfilenode files with sequential scans of the array. This speeds up pg_upgrade by 2x for a large number of tables, e.g. 16k. Per observation by Ants Aasma.
2012-11-13Mark pg_upgrade's free_db_and_rel_infos() as a static function.Bruce Momjian
2012-11-06In pg_upgrade, set synchronous_commit=off for the new cluster, toBruce Momjian
improve performance when restoring the schema from the old cluster. Backpatch to 9.2.
2012-10-18Use a more portable platform test.Andrew Dunstan
2012-10-10In pg_upgrade, issue proper error message when we can't open PG_VERSION.Bruce Momjian
Backpatch to 9.2.
2012-10-02In pg_upgrade, use full path name for analyze_new_cluster.sh script.Bruce Momjian
Backpatch to 9.2.
2012-10-02Work around unportable behavior of malloc(0) and realloc(NULL, 0).Tom Lane
On some platforms these functions return NULL, rather than the more common practice of returning a pointer to a zero-sized block of memory. Hack our various wrapper functions to hide the difference by substituting a size request of 1. This is probably not so important for the callers, who should never touch the block anyway if they asked for size 0 --- but it's important for the wrapper functions themselves, which mistakenly treated the NULL result as an out-of-memory failure. This broke at least pg_dump for the case of no user-defined aggregates, as per report from Matthew Carrington. Back-patch to 9.2 to fix the pg_dump issue. Given the lack of previous complaints, it seems likely that there is no live bug in previous releases, even though some of these functions were in place before that.
2012-10-02In pg_upgrade, improve error reporting when the number of relationBruce Momjian
objects does not match between the old and new clusters. Backpatch to 9.2.
2012-10-02Adjust pg_upgrade query so toast tables related to system catalog schemaBruce Momjian
entries are not dumped. This fixes an error caused by droping/recreating the information_schema, but other failures were also possible. Backpatch to 9.2.
2012-10-02In pg_upgrade, try to convert the locale names to canonical form beforeBruce Momjian
comparison; also report the old/new values if they don't match. Backpatch to 9.2.
generated by cgit v1.2.3 (git 2.46.0) at 2025-08-28 09:16:03 +0000