Always commute strategy when preprocessing DESC keys.

A recently added nbtree preprocessing step failed to account for the
fact that DESC columns already had their B-Tree strategy number commuted
at this point in preprocessing.  As a result, preprocessing could output
a set of scan keys where one or more keys had the correct strategy
number, but used the wrong comparison routine.

To fix, make the faulty code path that looks up a more restrictive
replacement operator/comparison routine commute its requested inequality
strategy (while outputting the transformed strategy number as before).
This makes the final transformed scan key comport with the approach
preprocessing has always used to deal with DESC columns (which is
described by comments above _bt_fix_scankey_strategy).

Oversight in commit commit b3f1a13f, which made nbtree preprocessing
perform transformations on skip array inequalities that can reduce the
total number of index searches.

Author: Peter Geoghegan <pg@bowt.ie>
Reported-By: Natalya Aksman <natalya@timescale.com>
Discussion: https://postgr.es/m/19049-b7df801e71de41b2@postgresql.org
Backpatch-through: 18

1 files changed, 18 insertions, 6 deletions

diff --git a/src/backend/access/nbtree/nbtpreprocesskeys.c b/src/backend/access/nbtree/nbtpreprocesskeys.c
index 936b93f157a..07a3ff11a0b 100644
--- a/src/backend/access/nbtree/nbtpreprocesskeys.c
+++ b/src/backend/access/nbtree/nbtpreprocesskeys.c
@@ -1412,6 +1412,7 @@ _bt_skiparray_strat_decrement(IndexScanDesc scan, ScanKey arraysk,
 	Datum		orig_sk_argument = high_compare->sk_argument,
 				new_sk_argument;
 	bool		uflow;
+	int16		lookupstrat;
 
 	Assert(high_compare->sk_strategy == BTLessStrategyNumber);
 
@@ -1433,9 +1434,14 @@ _bt_skiparray_strat_decrement(IndexScanDesc scan, ScanKey arraysk,
 		return;
 	}
 
-	/* Look up <= operator (might fail) */
-	leop = get_opfamily_member(opfamily, opcintype, opcintype,
-							   BTLessEqualStrategyNumber);
+	/*
+	 * Look up <= operator (might fail), accounting for the fact that a
+	 * high_compare on a DESC column already had its strategy commuted
+	 */
+	lookupstrat = BTLessEqualStrategyNumber;
+	if (high_compare->sk_flags & SK_BT_DESC)
+		lookupstrat = BTGreaterEqualStrategyNumber; /* commute this too */
+	leop = get_opfamily_member(opfamily, opcintype, opcintype, lookupstrat);
 	if (!OidIsValid(leop))
 		return;
 	cmp_proc = get_opcode(leop);
@@ -1464,6 +1470,7 @@ _bt_skiparray_strat_increment(IndexScanDesc scan, ScanKey arraysk,
 	Datum		orig_sk_argument = low_compare->sk_argument,
 				new_sk_argument;
 	bool		oflow;
+	int16		lookupstrat;
 
 	Assert(low_compare->sk_strategy == BTGreaterStrategyNumber);
 
@@ -1485,9 +1492,14 @@ _bt_skiparray_strat_increment(IndexScanDesc scan, ScanKey arraysk,
 		return;
 	}
 
-	/* Look up >= operator (might fail) */
-	geop = get_opfamily_member(opfamily, opcintype, opcintype,
-							   BTGreaterEqualStrategyNumber);
+	/*
+	 * Look up >= operator (might fail), accounting for the fact that a
+	 * low_compare on a DESC column already had its strategy commuted
+	 */
+	lookupstrat = BTGreaterEqualStrategyNumber;
+	if (low_compare->sk_flags & SK_BT_DESC)
+		lookupstrat = BTLessEqualStrategyNumber; /* commute this too */
+	geop = get_opfamily_member(opfamily, opcintype, opcintype, lookupstrat);
 	if (!OidIsValid(geop))
 		return;
 	cmp_proc = get_opcode(geop);