Repository navigation
GitIgnoreSpec: X/** reports the bare directory as ignored (5 cases), and the excluded-ancestor rule outlives its own re-inclusion (1 case) #137
Description
Following up on your request in #132.
Everything below is measured on master at 5f14318, against the same corpus I used there — 9,852 (.gitignore, path) cases from 43 real repositories, git check-ignore as the oracle, KaizenShogun/gitignore-conformance — with GitIgnoreSpec on all three backends. The six remaining divergences are two bugs, not six.
A. X/** reports the directory X itself as ignored — 5 of the 6, pre-existing
from pathspec.gitignore import GitIgnoreSpec
GitIgnoreSpec.from_lines(['d/**']).match_file('d/') # True
GitIgnoreSpec.from_lines(['d/*']).match_file('d/') # Falsegit ignores d in neither case. d/** excludes what is under d, not d itself, and that difference is exactly what makes a re-inclusion work under d/** and fail under d/:
.gitignore |
git stages d/keep.txt |
match_file('d/') |
match_file('d/keep.txt') |
|---|---|---|---|
d/** + !d/keep.txt |
yes | True | False |
d/ + !d/keep.txt |
no | True | True |
On the first row pathspec calls the directory ignored and the file inside it not ignored. A caller that prunes a walk with match_file(append_dir_sep(p)) — the idiom append_dir_sep() was added for in #65 — never reaches the file git would stage.
The mechanism is in the compiled regex. A trailing ** emits a bare '/' (spec.py:324), so d/** becomes ^d/, which the string d/ matches with nothing left over — and with no ps_d group it lands in the file accumulator, not the directory one:
'd/**' -> ^d/
'd/*' -> ^d/[^/]+(?:(?P<ps_d>/)|$)
'd/' -> ^(?:.+/)?d(?P<ps_d>/)
d/* is already right on the same question, which is what makes this look like an oversight rather than a decision. GitIgnoreBasicPattern has the same shape (basic.py:280).
Appending one character there — out_parts.append('/.') — takes the bench from 6 to 1 on all three backends. Of the 212 tests in tests/, three then fail, and all three are assertEqual on the literal regex text: test_03_child_double_asterisk, test_03_duplicate_leading_double_asterisk_edge_case, test_14_issue_81_a. The match_file assertions inside those same tests still hold, and **/api/** still compiles equal to **/**/api/**/**. I'm not putting that forward as the fix — it's a one-character probe that says where the five live. Happy to send it as a PR with those three assertions updated if you want it in that shape.
The five corpus rows, all the same shape: .settings/ under .settings/** (kubernetes), .superset/ under **/.superset/** (supabase), and tensorflow/lite/gen/, tensorflow/lite/tools/make/downloads/, tensorflow/lite/tools/make/gen/ under their /…/** (tensorflow).
B. The excluded-ancestor rule outlives the negation that re-includes the ancestor — 1 of the 6, new in #132
GitIgnoreSpec.from_lines(['.*', '!**/node_modules/**']).match_file(
'vendor/deps/npm/node_modules/.bin/x.txt') # True; git: not ignored93e0179 (master before the merge) answered False here on all three backends; from f404ca47 on, True. Reduced from nodejs/node's real .gitignore with git check-ignore arbitrating each step. Swap .bin for bin and the answer is right again, so it is the hidden ancestor doing it.
.* is the only pattern that produces a _DIR_MARK match, so it sets dir_include = True for the ancestor .bin. !**/node_modules/** compiles to ^(?:.+/)?node_modules/ — no dir mark — so it only ever reaches file_include, and if dir_include: wins. But that same negation also re-includes .bin itself: git names it as the deciding pattern for the bare ancestor path, and a canary dropped inside .bin stages. pathspec agrees, when you ask it directly:
s = GitIgnoreSpec.from_lines(['.*', '!**/node_modules/**'])
s.match_file('vendor/deps/npm/node_modules/.bin') # False — ancestor not excluded
s.match_file('vendor/deps/npm/node_modules/.bin/x.txt') # True — because that ancestor is excludedSo the ancestor rule fires on an ancestor the same spec says is not excluded. The directory accumulator is resolved over _DIR_MARK matches only, while whether an ancestor ends up excluded is decided by every pattern that matches it. I don't have a measured fix for this one — it looks like an ordering question inside the accumulator rather than a design problem, but I haven't got a patch I'd stand behind yet.
How git was asked, since half of this is about directories
The obvious ways to ask git about a directory are both traps, and they cost me a probe script each. check-ignore on d/ with the trailing slash matches d/*, because * happily matches the empty string — that is the question answering itself, not git ignoring anything. And !! d/ in status --ignored fires just as readily when a directory merely has nothing left to stage.
So directory verdicts here come from a real tree: check-ignore -v on the bare path, plus a canary probe — append !d/canary, create it, git add -A -n, and staged means the parent was never excluded, per the "not possible to re-include a file if a parent directory is excluded" rule in gitignore(5). That is the only thing I found that separates d/* from d/. Corpus cases were kept only where the independent ways agreed; disagreements were dropped and counted, never guessed at. git 2.55.0 throughout.
— Midas
Activity
KaizenShogun commentedon Sep 6, 2026
Two measurements that were sitting in my duplicate of this issue (#136, now closed) and belong here instead.
Before/after,
93e0179— the first parent of the merge commit, so the comparison isolates #132 and nothing else — against5f14318:Compared as sets rather than counts: 64 fixed, 1 new, 5 pre-existing on
simple(60 / 1 / 5 on re2 and hyperscan, whose pre-merge sets were already smaller). The same six paths survive on all three backends.The two halves land on different APIs, which may change what the fix should be:
match_tree_filesgets all five right. They only bite callers that run their ownos.walkand prune onmatch_file("some_dir/")— which is what black'sgen_python_filesdoes, and it's common enough that I built a second bench around that pattern. So that half may be a documentation matter (directory queries belong to the tree API) rather than a code one; your call.match_tree_filestoo, so it isn't the same kind of thing. With the file actually on disk in a real repo:git status --untracked-files=alllists it,match_tree_files(repo, negate=True)returns it on93e0179, and returns nothing on5f14318.Bench, corpus and the reduction scripts: https://github.com/KaizenShogun/gitignore-conformance — happy to run any candidate fix through it, same as with #132.
— Midas
KaizenShogun commentedon Sep 10, 2026
Part A is closed — #138 fixed it, and on
f9a833ethe bench is down to the single §B case on all three backends. I said in the comment above that I didn't have a patch for §B that I'd stand behind. Now I do, and the part worth your time is the trap I walked into on the way, because it turns the obvious fix into a false negative.The mechanism, stated more precisely than I had it.
dir_includeconflates two different facts: "some strict ancestor of this path is excluded" and "this path is an excluded directory". Only the first is git's rule. And the bucket is resolved among patterns that produce a_DIR_MARKmatch, while whether an ancestor ends up excluded is decided by every pattern that matches it —!**/node_modules/**compiles to a regex with no dir mark, so it can never displace.*from the directory bucket even though it re-includes the very ancestor.*excluded.So two changes. The directory bucket only accepts matches on a strict ancestor, and the excluded-ancestor rule only fires once the ancestor is confirmed excluded by the whole spec — ask about it as a directory query, outermost first, exactly the order git stops descending in.
The trap. A single pattern can match both a strict ancestor and the path itself, and the engine only hands back the leftmost match.
!*/compiles to an unanchored(?P<ps_d>/). Againstsub/d/the separators it matches areClassify with that one match and
!*/counts as an ancestor exclusion and never as the directory it actually re-includes, which breakstest_02_dir_reinclusion_whitelist— a test that predates #138, and git agrees with it (sub/d/stages a canary). My first patch cleared the bench to 0/9852 on all three backends and broke that test onsimple. Zero divergences bought with a regression the corpus can't see is exactly the trade I keep warning other people about, so: the simple backend now asks for every separator, not the first.That also corrects something I wrote above. I called §B "an ordering question inside the accumulator rather than a design problem". Wrong on the evidence — it isn't ordering, it's that the accumulator never receives the information it would need to order.
One thing to know before touching the re2/hyperscan sets. Splitting the directory variant there costs nothing; adding a third expression per pattern costs a factor of ~9 (re2: 1.80 → ~16 µs/check). It isn't the shape — a third expression that is an innocuous duplicate of one already in the set costs the same.
f'{base_regex}/?$'keeps it at two and the cost stays in the noise.Measured on
f9a833evs the patch,GitIgnoreSpec, all three backends, CPython 3.14.7, git 2.55.0 as the oracle:tests/match_tree_filesover the 43 repos materialisedThe sweep is 96 two-pattern specs (an excluding shape × a negation) over 13 queries, each verdict taken from a real repository —
check-ignore --stdinfor files, the canary probe for directories, sincecheck-ignore d/answers itself.Two things I am not claiming. The sweep still leaves 43 disagreements on
simpleand 18 on the other two; those are other shapes, untouched by this, and the patch doesn't move them either way. And onf9a833ethe three backends disagree with each other on 24 of the 1,248 —['*', '!d/k.txt']ond/k.txtisFalseon simple,Trueon re2 and hyperscan, and git saysTrue. That's a separate report, not this one; I'll open it once I've reduced it properly rather than dumping it here.Diff is two accumulators plus the regex split, no public API change. Happy to send it as a PR with tests and a changelog entry if the approach suits you — it touches all three backends, so I'd rather you look at the shape first than review a branch you'd want built differently.
— Midas
youdie006 commentedon Sep 14, 2026
Owning the §B half: #132 was mine, and the regression is real. Confirming it independently on
f0fb3f4, withpathspec/_backends/simple/gitignore.pyat md5f4a2df1c71502242d0ce5471ffb1b4b2:Git verdict from a real tree (
git init,git add -A, thengit ls-files), not fromcheck-ignore, for the reason you gave:check-ignoreanswers itself on this class. The file comes back tracked.Worth one line because it is the one thing I can add that you cannot: this is git 2.43.0, where your measurements are on 2.55.0. Same verdicts on every row, so the divergence is not version skew.
Your diagnosis reads correctly to me, and the sharp part is the bit that is easy to miss:
dir_includeis resolved among patterns that produce a_DIR_MARKmatch, while whether the ancestor actually ends up excluded is decided by every pattern that matches it.!**/node_modules/**compiles without a dir mark, so it cannot displace.*from the directory bucket even though it re-includes the very ancestor.*excluded. That is not an ordering bug inside the accumulator, as you said; the accumulator never receives the fact it would need.The
!*/trap is the part I would have walked into. Classifying on the leftmostsearch()match makes!*/an ancestor exclusion and never the directory it re-includes, andtest_02_dir_reinclusion_whitelistis right to catch that.I am not sending a competing patch. You found it, reduced it, and have the measurements; this is just the author of the regression confirming it and saying the shape looks right.
Disclosure: I used Claude (an AI assistant). The four rows and the git version above are from runs on my machine.
Do not apply the excluded directory rule to a re-included ancestor
KaizenShogun commentedon Sep 15, 2026
@youdie006 — your four rows, run against the branch in #144. Base is
f0fb3f4, the same tree your md5 identifies, and I checked that md5 before trusting the comparison. All three backends:f0fb3f4vendor/deps/npm/node_modules/.bin/x.txtvendor/deps/npm/node_modules/.bin.hiddenbuild/x.txtSame oracle as yours — real tree,
git init+git add -A+git ls-files, notcheck-ignore— on git 2.55.0, so your row-for-row agreement across 2.43.0 and 2.55.0 holds from this end too. Over the 9,852-query corpus the branch goes 1 → 0 onsimple,re2andhyperscan, compared as sets: the one it fixes is your row, and nothing else moves.test_02_dir_reinclusion_whitelist— the!*/trap — runs and passes; the suite goes 215 → 216. One line about that number, because it nearly fooled me: those tests only run withre2andhyperscanimportable. Without them 198 of them skip, and the OK at the bottom looks identical.— Midas
cpburnz commentedon Oct 7, 2026
Part B was fixed by #144.