From 7f563c09f8901f6acd72cb8fba7b1bd3cf3aca8e Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 31 Mar 2018 19:52:01 -0700 Subject: Add amcheck verification of heap relations belonging to btree indexes. Add a new, optional, capability to bt_index_check() and bt_index_parent_check(): check that each heap tuple that should have an index entry does in fact have one. The extra checking is performed at the end of the existing nbtree checks. This is implemented by using a Bloom filter data structure. The implementation performs set membership tests within a callback (the same type of callback that each index AM registers for CREATE INDEX). The Bloom filter is populated during the initial index verification scan. Reusing the CREATE INDEX infrastructure allows the new verification option to automatically benefit from the heap consistency checks that CREATE INDEX already performs. CREATE INDEX does thorough sanity checking of HOT chains, so the new check actually manages to detect problems in heap-only tuples. Author: Peter Geoghegan Reviewed-By: Pavan Deolasee, Andres Freund Discussion: https://postgr.es/m/CAH2-Wzm5VmG7cu1N-H=nnS57wZThoSDQU+F5dewx3o84M+jY=g@mail.gmail.com --- doc/src/sgml/amcheck.sgml | 126 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 96 insertions(+), 30 deletions(-) (limited to 'doc/src') diff --git a/doc/src/sgml/amcheck.sgml b/doc/src/sgml/amcheck.sgml index 852e260c098..a712c86a10f 100644 --- a/doc/src/sgml/amcheck.sgml +++ b/doc/src/sgml/amcheck.sgml @@ -9,13 +9,13 @@ The amcheck module provides functions that allow you to - verify the logical consistency of the structure of indexes. If the + verify the logical consistency of the structure of relations. If the structure appears to be valid, no error is raised. The functions verify various invariants in the - structure of the representation of particular indexes. The + structure of the representation of particular relations. The correctness of the access method functions behind index scans and other important operations relies on these invariants always holding. For example, certain functions verify, among other things, @@ -44,7 +44,7 @@ - bt_index_check(index regclass) returns void + bt_index_check(index regclass, heapallindexed boolean) returns void bt_index_check @@ -55,7 +55,9 @@ bt_index_check tests that its target, a B-Tree index, respects a variety of invariants. Example usage: -test=# SELECT bt_index_check(c.oid), c.relname, c.relpages +test=# SELECT bt_index_check(index => c.oid, heapallindexed => i.indisunique) + c.relname, + c.relpages FROM pg_index i JOIN pg_opclass op ON i.indclass[0] = op.oid JOIN pg_am am ON op.opcmethod = am.oid @@ -83,9 +85,11 @@ ORDER BY c.relpages DESC LIMIT 10; This example shows a session that performs verification of every catalog index in the database test. Details of just - the 10 largest indexes verified are displayed. Since no error - is raised, all indexes tested appear to be logically consistent. - Naturally, this query could easily be changed to call + the 10 largest indexes verified are displayed. Verification of + the presence of heap tuples as index tuples is requested for + unique indexes only. Since no error is raised, all indexes + tested appear to be logically consistent. Naturally, this query + could easily be changed to call bt_index_check for every index in the database where verification is supported. @@ -95,10 +99,11 @@ ORDER BY c.relpages DESC LIMIT 10; is the same lock mode acquired on relations by simple SELECT statements. bt_index_check does not verify invariants - that span child/parent relationships, nor does it verify that - the target index is consistent with its heap relation. When a - routine, lightweight test for corruption is required in a live - production environment, using + that span child/parent relationships, but will verify the + presence of all heap tuples as index tuples within the index + when heapallindexed is + true. When a routine, lightweight test for + corruption is required in a live production environment, using bt_index_check often provides the best trade-off between thoroughness of verification and limiting the impact on application performance and availability. @@ -108,7 +113,7 @@ ORDER BY c.relpages DESC LIMIT 10; - bt_index_parent_check(index regclass) returns void + bt_index_parent_check(index regclass, heapallindexed boolean) returns void bt_index_parent_check @@ -117,19 +122,21 @@ ORDER BY c.relpages DESC LIMIT 10; bt_index_parent_check tests that its - target, a B-Tree index, respects a variety of invariants. The - checks performed by bt_index_parent_check - are a superset of the checks performed by - bt_index_check. + target, a B-Tree index, respects a variety of invariants. + Optionally, when the heapallindexed + argument is true, the function verifies the + presence of all heap tuples that should be found within the + index. The checks that can be performed by + bt_index_parent_check are a superset of the + checks that can be performed by bt_index_check. bt_index_parent_check can be thought of as a more thorough variant of bt_index_check: unlike bt_index_check, bt_index_parent_check also checks - invariants that span parent/child relationships. However, it - does not verify that the target index is consistent with its - heap relation. bt_index_parent_check - follows the general convention of raising an error if it finds a - logical inconsistency or other problem. + invariants that span parent/child relationships. + bt_index_parent_check follows the general + convention of raising an error if it finds a logical + inconsistency or other problem. A ShareLock is required on the target index by @@ -158,6 +165,47 @@ ORDER BY c.relpages DESC LIMIT 10; + + Optional <parameter>heapallindexed</parameter> verification + + When the heapallindexed argument to + verification functions is true, an additional + phase of verification is performed against the table associated with + the target index relation. This consists of a dummy + CREATE INDEX operation, which checks for the + presence of all hypothetical new index tuples against a temporary, + in-memory summarizing structure (this is built when needed during + the basic first phase of verification). The summarizing structure + fingerprints every tuple found within the target + index. The high level principle behind + heapallindexed verification is that a new + index that is equivalent to the existing, target index must only + have entries that can be found in the existing structure. + + + The additional heapallindexed phase adds + significant overhead: verification will typically take several times + longer. However, there is no change to the relation-level locks + acquired when heapallindexed verification is + performed. + + + The summarizing structure is bound in size by + maintenance_work_mem. In order to ensure that + there is no more than a 2% probability of failure to detect an + inconsistency for each heap tuple that should be represented in the + index, approximately 2 bytes of memory are needed per tuple. As + less memory is made available per tuple, the probability of missing + an inconsistency slowly increases. This approach limits the + overhead of verification significantly, while only slightly reducing + the probability of detecting a problem, especially for installations + where verification is treated as a routine maintenance task. Any + single absent or malformed tuple has a new opportunity to be + detected with each new verification attempt. + + + + Using <filename>amcheck</filename> effectively @@ -197,18 +245,31 @@ ORDER BY c.relpages DESC LIMIT 10; operating system locales and collations. + + + Structural inconsistencies between indexes and the heap relations + that are indexed (when heapallindexed + verification is performed). + + + There is no cross-checking of indexes against their heap relation + during normal operation. Symptoms of heap corruption can be subtle. + + Corruption caused by hypothetical undiscovered bugs in the - underlying PostgreSQL access method code or sort - code. + underlying PostgreSQL access method + code, sort code, or transaction management code. Automatic verification of the structural integrity of indexes plays a role in the general testing of new or proposed PostgreSQL features that could plausibly allow a - logical inconsistency to be introduced. One obvious testing - strategy is to call amcheck functions continuously + logical inconsistency to be introduced. Verification of table + structure and associated visibility and transaction status + information plays a similar role. One obvious testing strategy + is to call amcheck functions continuously when running the standard regression tests. See for details on running the tests. @@ -242,6 +303,12 @@ ORDER BY c.relpages DESC LIMIT 10; absolute protection against failures that result in memory corruption. + + When heapallindexed verification is + performed, there is generally a greatly increased chance of + detecting single-bit errors, since strict binary equality is + tested, and the indexed attributes within the heap are tested. + In general, amcheck can only prove the presence of @@ -253,11 +320,10 @@ ORDER BY c.relpages DESC LIMIT 10; Repairing corruption No error concerning corruption raised by amcheck should - ever be a false positive. In practice, amcheck is more - likely to find software bugs than problems with hardware. - amcheck raises errors in the event of conditions that, - by definition, should never happen, and so careful analysis of - amcheck errors is often required. + ever be a false positive. amcheck raises + errors in the event of conditions that, by definition, should never + happen, and so careful analysis of amcheck + errors is often required. There is no general method of repairing problems that -- cgit v1.2.3