user/sven/git.git/midx.c, branch v2.26.0-rc2

nth_packed_object_oid(): use customary integer return

2020-02-24T20:55:42Z

Our nth_packed_object_sha1() function returns NULL for error. So when we wrapped it with nth_packed_object_oid(), we kept the same semantics. But it's a bit funny, because the caller actually passes in an out parameter, and the pointer we return is just that same struct they passed to us (or NULL). It's not too terrible, but it does make the interface a little non-idiomatic. Let's switch to our usual "0 for success, negative for error" return value. Most callers either don't check it, or are trivially converted. The one that requires the biggest change is actually improved, as we can ditch an extra aliased pointer variable. Since we are changing the interface in a subtle way that the compiler wouldn't catch, let's also change the name to catch any topics in flight. We can drop the 'o' and make it nth_packed_object_id(). That's slightly shorter, but also less redundant since the 'o' stands for "object" already. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

midx: honor the MIDX_PROGRESS flag in midx_repack

2019-10-23T03:05:06Z

Update midx_repack to only display progress when the MIDX_PROGRESS flag is set. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

midx: honor the MIDX_PROGRESS flag in verify_midx_file

2019-10-23T03:05:05Z

Update verify_midx_file to only display progress when the MIDX_PROGRESS flag is set. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

midx: add progress to expire_midx_packs

2019-10-23T03:05:05Z

Add progress to expire_midx_packs. Progress is displayed when the MIDX_PROGRESS flag is set. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

midx: add progress to write_midx_file

2019-10-23T03:05:05Z

Add progress to write_midx_file. Progress is displayed when the MIDX_PROGRESS flag is set. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

midx: add MIDX_PROGRESS flag

2019-10-23T03:05:05Z

Add the MIDX_PROGRESS flag and update the write|verify|expire|repack functions in midx.h to accept a flags parameter. The MIDX_PROGRESS flag indicates whether the caller of the function would like progress information to be displayed. This patch only changes the method prototypes and does not change the functionality. The functionality change will be handled by a later patch. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

midx: switch to using the_hash_algo

2019-08-19T22:05:00Z

Instead of hard-coding the hash size, use the_hash_algo to look up the hash size at runtime. Remove the #define constant which was used to hold the hash length, since writing the expression with the_hash_algo provide enough documentary value on its own. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano

midx: implement midx_repack()

2019-06-11T17:34:40Z

To repack with a non-zero batch-size, first sort all pack-files by their modified time. Second, walk those pack-files from oldest to newest, compute their expected size, and add the packs to a list if they are smaller than the given batch-size. Stop when the total expected size is at least the batch size. If the batch size is zero, select all packs in the multi-pack-index. Finally, collect the objects from the multi-pack-index that are in the selected packs and send them to 'git pack-objects'. Write a new multi-pack-index that includes the new pack. Using a batch size of zero is very similar to a standard 'git repack' command, except that we do not delete the old packs and instead rely on the new multi-pack-index to prevent new processes from reading the old packs. This does not disrupt other Git processes that are currently reading the old packs based on the old multi-pack-index. While first designing a 'git multi-pack-index repack' operation, I started by collecting the batches based on the actual size of the objects instead of the size of the pack-files. This allows repacking a large pack-file that has very few referencd objects. However, this came at a significant cost of parsing pack-files instead of simply reading the multi-pack-index and getting the file information for the pack-files. The "expected size" version provides similar behavior, but could skip a pack-file if the average object size is much larger than the actual size of the referenced objects, or can create a large pack if the actual size of the referenced objects is larger than the expected size. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

multi-pack-index: prepare 'repack' subcommand

2019-06-11T17:34:40Z

In an environment where the multi-pack-index is useful, it is due to many pack-files and an inability to repack the object store into a single pack-file. However, it is likely that many of these pack-files are rather small, and could be repacked into a slightly larger pack-file without too much effort. It may also be important to ensure the object store is highly available and the repack operation does not interrupt concurrent git commands. Introduce a 'repack' subcommand to 'git multi-pack-index' that takes a '--batch-size' option. The subcommand will inspect the multi-pack-index for referenced pack-files whose size is smaller than the batch size, until collecting a list of pack-files whose sizes sum to larger than the batch size. Then, a new pack-file will be created containing the objects from those pack-files that are referenced by the multi-pack-index. The resulting pack is likely to actually be smaller than the batch size due to compression and the fact that there may be objects in the pack- files that have duplicate copies in other pack-files. The current change introduces the command-line arguments, and we add a test that ensures we parse these options properly. Since we specify a small batch size, we will guarantee that future implementations do not change the list of pack-files. In addition, we hard-code the modified times of the packs in the pack directory to ensure the list of packs sorted by modified time matches the order if sorted by size (ascending). This will be important in a future test. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

multi-pack-index: implement 'expire' subcommand

2019-06-11T17:34:40Z

The 'git multi-pack-index expire' subcommand looks at the existing mult-pack-index, counts the number of objects referenced in each pack-file, deletes the pack-fils with no referenced objects, and rewrites the multi-pack-index to no longer reference those packs. Refactor the write_midx_file() method to call write_midx_internal() which now takes an existing 'struct multi_pack_index' and a list of pack-files to drop (as specified by the names of their pack- indexes). As we write the new multi-pack-index, we drop those file names from the list of known pack-files. The expire_midx_packs() method removes the unreferenced pack-files after carefully closing the packs to avoid open handles. Test that a new pack-file that covers the contents of two other pack-files leads to those pack-files being deleted during the expire subcommand. Be sure to read the multi-pack-index to ensure it no longer references those packs. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano