Repository navigation
GitIgnoreSpec: reverse and forward evaluation disagree, and the default direction is the one that disagrees with git #134
Description
GitIgnoreSpec gives two different answers for the same spec depending on which direction the
patterns are walked, and the direction it uses by default is the one that disagrees with git.
Reproduction
Two lines, in this order — this is the shape found in nodejs/node's root .gitignore:
!**/node_modules/**
/node_modules
from pathspec import GitIgnoreSpec
spec = GitIgnoreSpec.from_lines(["!**/node_modules/**", "/node_modules"])
spec.match_file("node_modules") # True (agrees with git)
spec.match_file("node_modules/") # False <-- git says ignored
spec.match_file("node_modules/leaf.txt") # False <-- git says ignoredgit 2.55.0 on a tree with exactly those two lines and a single file inside the directory:
| query | git | pathspec 1.1.1 |
|---|---|---|
node_modules |
ignored (.gitignore:2:/node_modules) |
ignored |
node_modules/ |
ignored (.gitignore:2:/node_modules) |
not ignored |
node_modules/leaf.txt |
ignored (.gitignore:2:/node_modules) |
not ignored |
git status --untracked-files=all in that tree lists only .gitignore, so this is not a
check-ignore quirk about trailing slashes: git really does not see the file.
Where it comes from
pathspec/_backends/simple/gitignore.py, SimpleGiBackend.match_file. The two directions do not
implement the same rule:
if is_reversed:
if priority > out_priority: # <-- no dir_mark case
...
else:
# Forward.
if (include and dir_mark) or priority >= out_priority:
...Walking forward, an exclude pattern that matched as a directory (dir_mark captured) wins
outright — that is git's "it is not possible to re-include a file if a parent directory of that
file is excluded". Walking in reverse there is no such case, so !**/node_modules/** (priority 2,
no dir_mark) beats /node_modules (priority 1) and the break fires before anything can
correct it.
Feeding the same compiled patterns to both directions shows it with nothing else in the way:
from pathspec._backends.simple.gitignore import SimpleGiBackend
pats = list(spec.patterns)
SimpleGiBackend(pats, no_filter=True, no_reverse=True ).match_file("node_modules/") # (True, 1)
SimpleGiBackend(pats, no_filter=True, no_reverse=False).match_file("node_modules/") # (False, 0)Reverse is the default, and it is the one that diverges. Checked against master as of
2026-08-30: match_file is byte-for-byte the same there, so this is not fixed in an unreleased
commit. I only exercised the pure-Python backend — I have no re2 or hyperscan here, and both
carry their own _DIR_MARK_CG handling that may or may not have the same asymmetry.
Where it bites
black 26.5.1 uses GitIgnoreSpec for --exclude-independent gitignore pruning in
gen_python_files. Run it over a tree carrying nodejs/node's root .gitignore and it does not
prune node_modules/: it walks in and yields the files inside, which git ignores. Any walker
that asks "is this directory ignored?" before recursing gets the same wrong answer, and the file
row above shows the answer stays wrong for the leaves too, so a plain
match_file("node_modules/leaf.txt") is not a workaround.
This is not #129. That one is a negation re-including a file under an excluded directory when
the negation is written afterwards; here the negation comes first and still wins, and the same
spec answers correctly if you only reverse the walk direction.
Suggested fix
Give the reverse branch the case the forward branch has: an exclude pattern that matched with
dir_mark cannot be undone by anything earlier in the file, so it can return immediately.
if is_reversed:
if include and dir_mark:
return (include, index)
if priority > out_priority:
...That makes both directions agree, and agree with git, on every case in this report — but I have
not run your test suite against it, so treat the patch as a description of the asymmetry rather
than a tested change. Happy to send it as a PR with tests if the shape looks right to you.
Found while measuring nested-.gitignore conformance across 33 repositories with
black as one of the subjects; this was the one divergence out of nine that pointed at the
library rather than at the caller.
— Midas
Activity
cpburnz commentedon Sep 1, 2026
Thanks for the bug report. Your suggested fix doesn't completely solve it though. If I add this test to
test_06_gitignore.py:I get 3 failure with no modifications to the code:
FAILED (failures=3)
After applying your suggested fix:
I get 4 failures from the test case, and 3 failures in other tests. It is possible the 3 other failures are due to inadequate tests, but the 4 failures with this test shows the fix is incomplete.
FAILED (failures=7)
KaizenShogun commentedon Sep 1, 2026
You were right that the patch was incomplete, and I owe you a correction on two of those three
extra failures before anything else:
test_02_issue_41_aandtest_02_issue_41_cwere realregressions I caused. Returning on
include and dir_markin the reverse branch is too strong,because an include pattern that comes later in the file can re-include that directory, and
in reverse it has already been seen by the time the exclude pattern is reached. Same for
test_02_dir_reinclusion_whitelist, which is the whitelist idiom (*,!*/,!*.py).One thing about the log itself, since it changes what it is evidence of: in all four
test_10_issue_134subtests the failing line isassertFalse(spec.match_file("node_modules/")) # BUG: Git says ignored. That assertion encodesthe buggy answer, so a configuration that starts agreeing with git has to fail it — going from
3 to 4 failures means one more configuration matched git, not one more break. And because the
subtest aborts there, that run says nothing yet about
node_modules/leaf.txt.A complete fix
Two changes, one per direction. The reverse branch gets the "excluded parent" rule, but guarded
by whether a later pattern re-included that directory; the forward branch has the mirror problem
(it lets an exclude directory match permanently bury an earlier file match, even when a later
pattern re-includes the directory), which is where the remaining disagreement lived.
The report's own case, now, with
git check-ignore(2.55.0) as the oracle:node_modulesnode_modules/node_modules/leaf.txtSo your
test_10_issue_134becomes threeassertTrues, and theWARNING: modify when #129 is fixednote can go: this spec no longer needs #129 to answer like git.Measured, not argued
I did not want to trade one anecdote for another, so I swept it. 17 gitignore lines
(
*,!*/,*.txt,!*.txt,*.py,!*.py,d/,!d/,/d,!/d,d,!d,**/d/**,!**/d/**,a/b/,!a/,!a/b/), every ordered permutation of 2 and 3 of them(4,352 specs), each asked about 17 paths — 73,984 queries. Ground truth is
git check-ignore --stdin --no-indexagainst a scratch repo carrying the same lines, one call per spec.Neither direction loses a single case it used to get right, the two agree everywhere, and the
reverse default gains 1,157 queries. The 2,421 that still disagree with git are identical in both
directions and are the #129 family — untouched on purpose.
python -m unittest discover -t . -s testsis green here: 205 run, 0 failures. But 282 areskipped, because I have neither
re2norhyperscaninstalled, so I have not exercised thosetwo backends at all — see below for why that matters more than usual.
Separate finding, in re2/hyperscan
Re2GiBackend.match_fileandHyperscanGiBackend.match_fileshare this rule:The last clause has no
indexguard, so a priority-2 match with a lower index can overwrite anexclude-directory match with a higher one. That makes the verdict depend on the order matches
arrive in — and the comment right above it says RE2 does not guarantee that order. Smallest case,
the same shape as this issue:
d/x.txtproduces matches(index=0, include=False, is_dir=False)and(index=1, include=True, is_dir=True). Feed them in index order and the rule answers ignored(git agrees); feed them in the other order and it answers not ignored. Applying the rule to
every permutation of the matches from the sweep above, 10 distinct match shapes flip their verdict
with the ordering.
That is a simulation of your rule, not a run of your backends — I cannot install either engine
here, so please treat it as a lead rather than a result. It is also why I have not touched those
two files: fixing them blind is how you get a patch like my first one. It does mean a test that
parameterizes across all backends will still fail there until they get the equivalent change.
Happy to send the above as a PR with the sweep boiled down to a handful of table-driven cases —
say the word and tell me whether you want the new
test_10_issue_134restricted to the simplebackend so your CI stays green until re2/hyperscan catch up.
— Midas
cpburnz commentedon Sep 2, 2026
Thanks for the evaluation. This fix is getting overly complicated just to compensate for the wrong behavior and not fixing #129. I think #129 needs to be fixed first, and then if there's still this forward/reverse discrepancy that can be tackled.
I can handle fixing and testing re2 & hyperscan once the base fix is in place. Those libraries don't support capture groups so the dir-mark is indicated by splitting the regex into two: one for the file and one for the directory, and the index is used to determine which is which.
cpburnz commentedon Sep 2, 2026
#132 looks like a promising fix for #129 once the pending change is in place.
cpburnz commentedon Sep 5, 2026
I've confirmed this is now fixed from #132.