index GIN GIN stands for Generalized Inverted Index. It is an index structure storing a set of (key, posting list) pairs, where a posting list is a set of rows in which the key occurs. Each indexed value can contain many keys, so the same row ID can appear in multiple posting lists. It is generalized in the sense that a GIN index does not need to be aware of the operation that it accelerates. Instead, it uses custom strategies defined for particular data types. One advantage of GIN is that it allows the development of custom data types with the appropriate access methods, by an expert in the domain of the data type, rather than a database expert. This is much the same advantage as using GiST. The GIN implementation in PostgreSQL is primarily maintained by Teodor Sigaev and Oleg Bartunov. There is more information about GIN on their website. The GIN interface has a high level of abstraction, requiring the access method implementer only to implement the semantics of the data type being accessed. The GIN layer itself takes care of concurrency, logging and searching the tree structure. All it takes to get a GIN access method working is to implement four (or five) user-defined methods, which define the behavior of keys in the tree and the relationships between keys, indexed values, and indexable queries. In short, GIN combines extensibility with generality, code reuse, and a clean interface. The four methods that an operator class for GIN must provide are: int compare(Datum a, Datum b) Compares keys (not indexed values!) and returns an integer less than zero, zero, or greater than zero, indicating whether the first key is less than, equal to, or greater than the second. Datum *extractValue(Datum inputValue, int32 *nkeys) Returns an array of keys given a value to be indexed. The number of returned keys must be stored into *nkeys. Datum *extractQuery(Datum query, int32 *nkeys, StrategyNumber n, bool **pmatch, Pointer **extra_data) Returns an array of keys given a value to be queried; that is, query is the value on the right-hand side of an indexable operator whose left-hand side is the indexed column. n is the strategy number of the operator within the operator class (see ). Often, extractQuery will need to consult n to determine the data type of query and the key values that need to be extracted. The number of returned keys must be stored into *nkeys. If the query contains no keys then extractQuery should store 0 or -1 into *nkeys, depending on the semantics of the operator. 0 means that every value matches the query and a full-index scan should be performed (but see ). -1 means that nothing can match the query, and so the index scan can be skipped entirely. pmatch is an output argument for use when partial match is supported. To use it, extractQuery must allocate an array of *nkeys booleans and store its address at *pmatch. Each element of the array should be set to TRUE if the corresponding key requires partial match, FALSE if not. If *pmatch is set to NULL then GIN assumes partial match is not required. The variable is initialized to NULL before call, so this argument can simply be ignored by operator classes that do not support partial match. extra_data is an output argument that allows extractQuery to pass additional data to the consistent and comparePartial methods. To use it, extractQuery must allocate an array of *nkeys Pointers and store its address at *extra_data, then store whatever it wants to into the individual pointers. The variable is initialized to NULL before call, so this argument can simply be ignored by operator classes that do not require extra data. If *extra_data is set, the whole array is passed to the consistent method, and the appropriate element to the comparePartial method. bool consistent(bool check[], StrategyNumber n, Datum query, int32 nkeys, Pointer extra_data[], bool *recheck) Returns TRUE if the indexed value satisfies the query operator with strategy number n (or might satisfy, if the recheck indication is returned). The check array has length nkeys, which is the same as the number of keys previously returned by extractQuery for this query datum. Each element of the check array is TRUE if the indexed value contains the corresponding query key, ie, if (check[i] == TRUE) the i-th key of the extractQuery result array is present in the indexed value. The original query datum (not the extracted key array!) is passed in case the consistent method needs to consult it. extra_data is the extra-data array returned by extractQuery, or NULL if none. On success, *recheck should be set to TRUE if the heap tuple needs to be rechecked against the query operator, or FALSE if the index test is exact. Optionally, an operator class for GIN can supply a fifth method: int comparePartial(Datum partial_key, Datum key, StrategyNumber n, Pointer extra_data) Compare a partial-match query to an index key. Returns an integer whose sign indicates the result: less than zero means the index key does not match the query, but the index scan should continue; zero means that the index key does match the query; greater than zero indicates that the index scan should stop because no more matches are possible. The strategy number n of the operator that generated the partial match query is provided, in case its semantics are needed to determine when to end the scan. Also, extra_data is the corresponding element of the extra-data array made by extractQuery, or NULL if none. To support partial match queries, an operator class must provide the comparePartial method, and its extractQuery method must set the pmatch parameter when a partial-match query is encountered. See for details. Internally, a GIN index contains a B-tree index constructed over keys, where each key is an element of the indexed value (a member of an array, for example) and where each tuple in a leaf page is either a pointer to a B-tree over heap pointers (PT, posting tree), or a list of heap pointers (PL, posting list) if the list is small enough. Updating a GIN index tends to be slow because of the intrinsic nature of inverted indexes: inserting or updating one heap row can cause many inserts into the index (one for each key extracted from the indexed value). As of PostgreSQL 8.4, GIN is capable of postponing much of this work by inserting new tuples into a temporary, unsorted list of pending entries. When the table is vacuumed, or if the pending list becomes too large (larger than ), the entries are moved to the main GIN data structure using the same bulk insert techniques used during initial index creation. This greatly improves GIN index update speed, even counting the additional vacuum overhead. Moreover the overhead can be done by a background process instead of in foreground query processing. The main disadvantage of this approach is that searches must scan the list of pending entries in addition to searching the regular index, and so a large list of pending entries will slow searches significantly. Another disadvantage is that, while most updates are fast, an update that causes the pending list to become too large will incur an immediate cleanup cycle and thus be much slower than other updates. Proper use of autovacuum can minimize both of these problems. If consistent response time is more important than update speed, use of pending entries can be disabled by turning off the FASTUPDATE storage parameter for a GIN index. See for details. GIN can support partial match queries, in which the query does not determine an exact match for one or more keys, but the possible matches fall within a reasonably narrow range of key values (within the key sorting order determined by the compare support method). The extractQuery method, instead of returning a key value to be matched exactly, returns a key value that is the lower bound of the range to be searched, and sets the pmatch flag true. The key range is then searched using the comparePartial method. comparePartial must return zero for an actual match, less than zero for a non-match that is still within the range to be searched, or greater than zero if the index key is past the range that could match. Create vs insert Insertion into a GIN index can be slow due to the likelihood of many keys being inserted for each value. So, for bulk insertions into a table it is advisable to drop the GIN index and recreate it after finishing bulk insertion. As of PostgreSQL 8.4, this advice is less necessary since delayed indexing is used (see for details). But for very large updates it may still be best to drop and recreate the index. Build time for a GIN index is very sensitive to the maintenance_work_mem setting; it doesn't pay to skimp on work memory during index creation. During a series of insertions into an existing GIN index that has FASTUPDATE enabled, the system will clean up the pending-entry list whenever it grows larger than work_mem. To avoid fluctuations in observed response time, it's desirable to have pending-list cleanup occur in the background (i.e., via autovacuum). Foreground cleanup operations can be avoided by increasing work_mem or making autovacuum more aggressive. However, enlarging work_mem means that if a foreground cleanup does occur, it will take even longer. The primary goal of developing GIN indexes was to create support for highly scalable, full-text search in PostgreSQL, and there are often situations when a full-text search returns a very large set of results. Moreover, this often happens when the query contains very frequent words, so that the large result set is not even useful. Since reading many tuples from the disk and sorting them could take a lot of time, this is unacceptable for production. (Note that the index search itself is very fast.) To facilitate controlled execution of such queries GIN has a configurable soft upper limit on the number of rows returned, the gin_fuzzy_search_limit configuration parameter. It is set to 0 (meaning no limit) by default. If a non-zero limit is set, then the returned set is a subset of the whole result set, chosen at random. Soft means that the actual number of returned results could differ slightly from the specified limit, depending on the query and the quality of the system's random number generator. GIN doesn't support full index scans. The reason for this is that extractValue is allowed to return zero keys, as for example might happen with an empty string or empty array. In such a case the indexed value will be unrepresented in the index. It is therefore impossible for GIN to guarantee that a scan of the index can find every row in the table. Because of this limitation, when extractQuery returns nkeys = 0 to indicate that all values match the query, GIN will emit an error. (If there are multiple ANDed indexable operators in the query, this happens only if they all return zero for nkeys.) It is possible for an operator class to circumvent the restriction against full index scan. To do that, extractValue must return at least one (possibly dummy) key for every indexed value, and extractQuery must convert an unrestricted search into a partial-match query that will scan the whole index. This is inefficient but might be necessary to avoid corner-case failures with operators such as LIKE or subset inclusion. GIN assumes that indexable operators are strict. This means that extractValue will not be called at all on a NULL value (so the value will go unindexed), and extractQuery will not be called on a NULL comparison value either (instead, the query is presumed to be unmatchable). A possibly more serious limitation is that GIN cannot handle NULL keys — for example, an array containing a NULL cannot be handled except by ignoring the NULL. The PostgreSQL source distribution includes GIN operator classes for tsvector and for one-dimensional arrays of all internal types. Prefix searching in tsvector is implemented using the GIN partial match feature. The following contrib modules also contain GIN operator classes: btree-gin B-Tree equivalent functionality for several data types hstore Module for storing (key, value) pairs intarray Enhanced support for int4[] pg_trgm Text similarity using trigram matching