user/sven/git.git/fsmonitor.c, branch v2.26.0-rc2

fsmonitor: handle version 2 of the hooks that will use opaque token

2020-01-13T22:58:43Z

Some file monitors like watchman will use something other than a timestamp to keep better track of what changes happen in between calls to query the fsmonitor. The clockid in watchman is a string. Now that the index is storing an opaque token for the last update the code needs to be updated to pass that opaque token to a verion 2 of the fsmonitor hook. Because there are repos that already have version 1 of the hook and we want them to continue to work when git is updated, we need to handle both version 1 and version 2 of the hook. In order to do that a config value is being added core.fsmonitorHookVersion to force what version of the hook should be used. When this is not set it will default to -1 and then the code will attempt to call version 2 of the hook first. If that fails it will fallback to trying version 1. Signed-off-by: Kevin Willford Signed-off-by: Junio C Hamano

fsmonitor: change last update timestamp on the index_state to opaque token

2020-01-13T22:58:43Z

Some file system monitors might not use or take a timestamp for processing and in the case of watchman could have race conditions with using a timestamp. Watchman uses something called a clockid that is used for race free queries to it. The clockid for watchman is simply a string. Change the fsmonitor_last_update from being a uint64_t to a char pointer so that any arbitrary data can be stored in it and passed back to the fsmonitor. Signed-off-by: Kevin Willford Signed-off-by: Junio C Hamano

unpack-trees: skip stat on fsmonitor-valid files

2019-11-21T03:48:18Z

The index might be aware that a file hasn't modified via fsmonitor, but unpack-trees did not pay attention to it and checked via ie_match_stat which can be inefficient on certain filesystems. This significantly slows down commands that run oneway_merge, like checkout and reset --hard. This patch makes oneway_merge check whether a file is considered unchanged through fsmonitor and skips ie_match_stat on it. unpack-trees also now correctly copies over fsmonitor validity state from the source index. Finally, for correctness, we force a refresh of fsmonitor state in tweak_fsmonitor. After this change, commands like stash (that use reset --hard internally) go from 8s or more to ~2s on a 250k file repository on a mac. Helped-by: Junio C Hamano Helped-by: Kevin Willford Signed-off-by: Utsav Shah Signed-off-by: Junio C Hamano

fsmonitor: do not compare bitmap size with size of split index

2019-11-13T01:30:26Z

3444ec2e ("fsmonitor: don't fill bitmap with entries to be removed", 2019-10-11) added a handful of sanity checks that make sure that a bit position in fsmonitor bitmap does not go beyond the end of the index. As each bit in the bitmap corresponds to a path in the index, this is the right check most of the time. Except for the case when we are in the split-index mode and looking at a delta index that is to be overlayed on the base index but before the base index has actually been merged in, namely in read_ and write_fsmonitor_extension(). In these codepaths, the entries in the split/delta index is typically a small subset of the entire set of paths (otherwise why would we be using split-index?), so the bitmap used by the fsmonitor is almost always larger than the number of entries in the partial index, and the incorrect comparison would trigger the BUG(). Reported-by: Utsav Shah Helped-by: Kevin Willford Helped-by: William Baker Signed-off-by: Junio C Hamano

fsmonitor: don't fill bitmap with entries to be removed

2019-10-12T01:16:11Z

While doing some testing with fsmonitor enabled I found that git commands would segfault after staging and unstaging an untracked file. Looking at the crash it appeared that fsmonitor_ewah_callback was attempting to adjust bits beyond the bounds of the index cache. Digging into how this could happen it became clear that the fsmonitor extension must have been written with more bits than there were entries in the index. The root cause ended up being that fill_fsmonitor_bitmap was populating fsmonitor_dirty with bits for all entries in the index, even those that had been marked for removal. To solve this problem fill_fsmonitor_bitmap has been updated to skip entries with the the CE_REMOVE flag set. With this change the bits written for the fsmonitor extension will be consistent with the index entries written by do_write_index. Additionally, BUG checks have been added to detect if the number of bits in fsmonitor_dirty should ever exceed the number of entries in the index again. Another option that was considered was moving the call to fill_fsmonitor_bitmap closer to where the index is written (and where the fsmonitor extension itself is written). However, that did not work as the fsmonitor_dirty bitmap must be filled before the index is split during writing. Signed-off-by: William Baker Signed-off-by: Junio C Hamano

Merge branch 'cb/fsmonitor-intfix'

2019-07-09T22:25:38Z

Variable type fix. * cb/fsmonitor-intfix: fsmonitor: avoid signed integer overflow / infinite loop

fsmonitor: avoid signed integer overflow / infinite loop

2019-06-18T01:14:29Z

883e248b8a ("fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.", 2017-09-22) uses an int in a loop that would wrap if index_state->cache_nr (unsigned) is bigger than INT_MAX Signed-off-by: Carlo Marcelo Arenas Belón Signed-off-by: Junio C Hamano

fsmonitor: force a refresh after the index was discarded

2019-05-08T03:03:48Z

With this change, the `index_state` struct becomes the new home for the flag that says whether the fsmonitor hook has been run, i.e. it is now per-index. It also gets re-set when the index is discarded, fixing the bug demonstrated by the "test_expect_failure" test added in the preceding commit. In that case fsmonitor-enabled Git would miss updates under certain circumstances, see that preceding commit for details. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

Merge branch 'jk/snprintf-truncation'

2018-05-30T12:51:28Z

Avoid unchecked snprintf() to make future code auditing easier. * jk/snprintf-truncation: fmt_with_err: add a comment that truncation is OK shorten_unambiguous_ref: use xsnprintf fsmonitor: use internal argv_array of struct child_process log_write_email_headers: use strbufs http: use strbufs instead of fixed buffers

fsmonitor: use internal argv_array of struct child_process

2018-05-21T00:58:56Z

Avoid magic array sizes and indexes by constructing the fsmonitor command line using the embedded argv_array of the child_process. The resulting code is shorter and easier to extend. Getting rid of the snprintf() calls is a bonus -- even though the buffers were big enough here to avoid truncation -- as it makes auditing the remaining callers easier. Inspired-by: Jeff King Signed-off-by: Rene Scharfe Signed-off-by: Junio C Hamano