Skip to content

fix(test): stop comparing rows that no walk could measure - #43

Merged
TheMeinerLP merged 1 commit into
mainfrom
fix/footprint-jvm-noise
Aug 3, 2026
Merged

fix(test): stop comparing rows that no walk could measure#43
TheMeinerLP merged 1 commit into
mainfrom
fix/footprint-jvm-noise

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Contributor

Why

ChunkFootprintTest failed three times on 2026-08-03 — in #39, in #40, and on main, where it stopped the snapshot from being published. Two different test methods, both at the same line, both with a count below zero:

FalcoChunk retains -2 objects of [B against 0 of DynamicChunk
FalcoChunk retains 0 objects of java.io.FileCleanable against -3 of DynamicChunk

The per-class figures are differences between two walks: everything the chunk and its instance retain, minus what the instance retains alone. A count below zero cannot be a property of the chunk — a set of objects that exist because the chunk exists has no negative cardinality. It means the instance grew between the two walks, and both classes caught doing it are the JVM's own bookkeeping: FileCleanable when a file handle is registered, byte[] when a string is built. Comparing such a row states nothing about either chunk.

Roughly three of ten CI runs, on ubuntu and on windows. Never reproduced locally in 8 attempts, including 3 pinned to two cores.

What changed

Rows with a negative count on either side are skipped, their bytes carried into the total so the per-class table still has to sum to what the footprint reports, and the classes are printed.

The cap of two is the part that matters. Without it the mechanism passes everything: inverting the condition so that every class counts as unmeasurable left the test green, because the byte correction absorbed the whole comparison. That is the failure mode this repository has paid for repeatedly — a check that cannot fail. With the cap, the same edit fails. Both observed failures carried exactly one such class.

What did not change

The expectations. 25 objects and 840 bytes are still asserted exactly, and no tolerance was introduced anywhere.

Checks that it still bites

Mutation Expected Result
exclusion without the cap, every class skipped green — the hole confirmed, cap added
same, with the cap red red
a real Object[4] field on FalcoChunk red red
reverted green green

What this does not do

It does not stop the JVM from allocating between the two walks — it stops that from being reported as a property of the chunk. The underlying non-determinism of a two-walk measurement is unchanged, and if a future failure names a class that is not JVM bookkeeping, this cap will surface it rather than hide it.

ChunkFootprintTest failed three times on 2026-08-03 -- twice in pull requests and
once on main, where it stopped the snapshot from being published. Two different
test methods, both at the same line, both with a count below zero:

  FalcoChunk retains -2 objects of [B against 0 of DynamicChunk
  FalcoChunk retains 0 objects of java.io.FileCleanable against -3 of DynamicChunk

The per class figures are differences between two walks: everything the chunk and
its instance retain, minus what the instance retains alone. A count below zero
cannot be a property of the chunk -- a set of objects that exist because the
chunk exists has no negative cardinality. It means the instance grew between the
two walks, and both classes seen doing it are the JVM's own bookkeeping:
FileCleanable when a file handle is registered, byte[] when a string is built.
Comparing such a row states nothing about either chunk.

Those rows are now skipped, their bytes carried into the total so the table still
has to sum to what the footprint reports, and the classes printed.

The exclusion is capped at two classes, and the cap is the part that matters.
Without it the mechanism passes everything: inverting the condition so that every
class counts as unmeasurable left the test green, because the byte correction
absorbed the whole comparison. With the cap the same edit fails. Skipping one or
two rows loses nothing; skipping many means the walk disagreed with itself, and
the answer to that is a failure rather than a comparison of what survived. Both
failures carried exactly one such class.

The expectations are untouched. 25 objects and 840 bytes are still asserted
exactly, and no tolerance was introduced anywhere. Checked by injecting a real
defect -- an Object[4] field on FalcoChunk -- which fails as it did before,
and reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGpJqdmh7ZNH487GLqPJmK
@TheMeinerLP
TheMeinerLP requested a review from a team as a code owner August 3, 2026 17:25
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Test results

  267 files    267 suites   10m 31s ⏱️
  894 tests   893 ✅ 1 💤 0 ❌
2 709 runs  2 707 ✅ 2 💤 0 ❌

Results for commit ff0cf63.

@TheMeinerLP
TheMeinerLP merged commit c4239bd into main Aug 3, 2026
8 checks passed
@TheMeinerLP
TheMeinerLP deleted the fix/footprint-jvm-noise branch August 3, 2026 17:35
@github-actions github-actions Bot mentioned this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant