Skip to content

Count unmatched trades on both sides in every grading tier - #919

Open
luisleo526 wants to merge 2 commits into
mainfrom
fix/grader-unmatched-gross
Open

luisleo526 wants to merge 2 commits into
mainfrom
fix/grader-unmatched-gross

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

Two commits:

  • Corpus grader (Known issue: the parity scoreboard counts unmatched trades net, which overstates 24 probes #836). The corpus grader now counts every unmatched gated trade on both sides in every grading tier, instead of netting unmatched trades, which overstated the match for some probes. This fixes the 1.14.0 known issue #836. The grader's tests cover the count rule, the metrics and the open-mark case; the CHANGELOG entries are under "Unreleased".
  • Changelog headings. The released 1.14.0 and 1.13.0 sections of CHANGELOG.md are now under their version and date. Only heading lines change.

Engine behaviour is unchanged: no product source changes.

Refs #836

…sion and date

The 1.14.0 entries were still under the heading "Unreleased" after v1.14.0
was tagged, and the 1.13.0 heading had no date. The first heading now reads
"1.14.0 — 2026-11-20" and the second "1.13.0 — 2026-11-18", the dates of the
commits the two release tags point to, in the form the older headings use.
Only these two heading lines change; no entry is edited or moved.
The corpus grader counted unmatched trades net: one engine-only and one
TradingView-only trade left the two list sizes equal, so a probe with a real
trade mismatch could still grade excellent, and strong grading and
fragmented-FIFO eligibility read the net difference of the list sizes.

analyze_strategy now takes ONE gross count: the unmatched TradingView trades
plus the unmatched engine trades of the gated pools (a zero-alignment input
counts both lists). The excellent gate, the strong tier and the fragmented-FIFO
eligibility all read it; the percentage is that count over the larger gated
pool (at least one). Thresholds, matching and every other gate are unchanged.
The net difference of list sizes stays in the report as net_count_abs_delta
next to gross_unmatched_count, and the printed count line shows both.

There is no exception for any strategy: verify_corpus.py --all exits 1 and
lists every strategy graded below Strong or missing an input, unless the
strategy declares an anomaly or an engine-only outcome.

The tests run real parsed tapes through analyze_strategy and main: balanced
unmatched trades on both sides, one-sided mismatches on each side, the
threshold boundary at and just below the strong limit, fragmented-FIFO
eligibility, zero-alignment inputs, and the aggregate and single-strategy exits.

check_corpus_parity.sh, README.md and docs/ci.md record the headline measured
on 2026-11-19 on corpus 93d956a2: excellent=303, strong=8, moderate=0,
anomaly=1 (excellent=311, strong=0 under the net count); all 312 pinned
strategy trade outputs still match the baseline hashes.

The two CHANGELOG entries sit under a new "Unreleased" heading above the 1.14.0
section; the 1.14.0 entry that called the correction planned for a later
release stays as that release shipped it.

Refs: #836

Documentation provenance: the verbatim originals of the README.md sentences
that this change removes or rewrites.

1. README.md:309 (removed; replaced by the historical regrade paragraph)
Known issue ([#836](#836)): the parity scoreboard counts unmatched trades net instead of gross, so a probe with one engine-only and one TradingView-only trade still grades excellent; counted gross, 24 of the 7,989 graded probes (7,959 excellent and 30 strong instead of 7,983 and 6) and 10 of the 312 open reference probes (301 excellent instead of 311) grade one tier lower, with no trade list changed, and the corrected grader is planned for a release after 1.14.0.

2. README.md:319 (Public board result cell rewritten; the whole table row follows)
| **Public** — [open corpus](https://github.com/pineforge-4pass/pineforge-corpus) | 312 reference strategies on BINANCE:ETHUSDT.P 15m, Apache-2.0, reproducible by anyone | this repository's sweep of the v1.0.1 library: **311 excellent + 1 declared anomaly**; the <!-- pf:scoreboard.corpusProbes\|int -->309<!-- /pf --> of them in the measured population: **<!-- pf:scoreboard.scopes.corpus.excellent\|int -->309<!-- /pf --> excellent** |

3. README.md:343 (excellent bullet rewritten: "the same number of trades" -> "zero unmatched gated trades on either side")
- **excellent** — the same number of trades, ≥ 99% of TradingView's trades matched, the distinct-entry identity check passed, entry and exit prices within 0.01% and PnL within 1% at the 90th percentile (trailing-stop scripts are graded on the *production* profile: exits within 0.02% and PnL within 1%, since a trail fill depends on TradingView's sub-bar path);

4. README.md:344 (strong bullet rewritten: "trade count within 6%" -> gross count rule)
- **strong** — ≥ 95% matched, trade count within 6%, entries within 0.1% and exits within 0.5% at p90;

from verify_corpus import MATCH_WINDOW_SECONDS, analyze_strategy
from test_verify_corpus_trail_profile import TRADE_COLUMNS, TRAIL_STRATEGY
import verify_corpus

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify code review View in Graphify

Review completed · 035368777224

Note

Merge confidence: 5/5 · Nothing blocking · 7 possible issues to check · 🟠 High impact
7 possible issues not confirmed

The verifier counts unmatched trades on both sides and reports the net difference separately.

Full summary

The verifier counts unmatched trades on both sides and reports the net difference separately. The parity script’s expected headline and tests are updated for the new grading behavior.

No confirmed issues. We found 7 possible issues we couldn't confirm; see below.

Not confirmed (7 possible issues)

Leads worth a look; we didn't double-check these. They don't change the score.

Not confirmed · The CLI-output regex in the zero-alignment test hard-codes a literal X where the count delta is expected, so the test can fail or pass for Inspect source

Detailed review · not confirmed

The test pins the printed count-delta line to the text (X; abs=2; net=0). The literal X looks like a placeholder, not a value the report prints. If the report's format differs, the test fails even when the count is right.

Lines 1061-1064 use assertRegex with the pattern Count delta:\s+200\.0000%\s+\(X; abs=2; net=0\). The same test separately checks abs=2; net=0 at line 1065, so the X part adds a brittle, unexplained constraint that the surrounding assertions do not need.

  • The format produced by _print_verification was not read, so it is unknown whether X is a real output token.

  • Found by the per-file sweep; not double-checked.

Not confirmed · No-alignment early return ignores the warmup/trim window that the main grading path applies to the count Inspect source

Detailed review · not confirmed

When nothing aligns and there are no range-end mark pairs, the count is computed over the whole untrimmed pools. The main path counts only interior trades when trim or warmup bounds are set, so the same strategy can get different count results depending on whether any trade happened to align.

Line 1229 computes count_abs_delta from tv_cmp and eng_cmp without applying _split_interior or the tv_gate/eng_gate filter used at lines 1289-1293 and 1300-1316. The early return then sets count_delta and count_ok from that untrimmed number at lines 1250-1254.

  • Whether trim_to_common_match_window can leave out-of-window trades in tv_cmp and eng_cmp when nothing matches was not checked.

  • This divergence was present before the PR too; the PR carries the same rule into the new count expression.

  • Found by the per-file sweep; not double-checked.

Not confirmed · The pinned parity headline moves 8 strategies from excellent to strong, but the grading change it depends on is not in this file Inspect source

Detailed review · not confirmed

The expected verifier headline changes from excellent=311, strong=0 to excellent=303, strong=8, and the PR description is empty. Nothing in this file implements the new unmatched-trade counting that the PR title describes, so the new pin cannot be checked here.

The comment at lines 90-95 says the excellent-only rule now requires zero gross unmatched TV and engine trades, and that the corpus was measured on 2026-11-19. Line 96 is the only code change in this diff, and the script only compares strings: it does not check grading. The PR title says unmatched trades are counted on both sides in every tier, which would explain the downgrade, but the verifier that implements that is not in this diff. The description is empty, so the rationale cannot be confirmed from the change itself.

  • scripts/verify_corpus.py was not provided, so it is unknown whether it prints the new headline

  • Whether issue #​836 and the 2026-11-19 measurement support the new counts was not verified

  • The sums in the new headline add to 312, which is consistent, but the per-tier values were not checked against any run

Not confirmed · An environment variable named EXPECTED_VERIFY silently replaces the pinned headline, even though the comment calls it pinned Inspect source

Detailed review · not confirmed

The line uses the ${EXPECTED_VERIFY:-...} form, so any exported value overrides the built-in pin. The comment says changes require measured evidence and that the pin is deliberate.

The default expansion ${VAR:-default} uses the environment value whenever it is non-empty. The header comment lists EXPECTED_VERIFY as a configurable environment variable (line 49), so the committed pin on line 96 is only a default. This predates the PR but is on a line the PR rewrote, and the new comment still describes the value as pinned.

  • Whether CI or the repository sets EXPECTED_VERIFY anywhere was not checked, because only this file was provided

  • Found by the per-file sweep; not double-checked.

Not confirmed · The new comment says the headline must be backed by evidence, but the script does not check that the measured corpus is the pinned one Inspect source

Detailed review · not confirmed

The comment cites measurement on corpus 93d956a2, but the script only checks that the checked-out corpus matches the gitlink from HEAD. Nothing ties the 93d956a2 reference to that gitlink.

The removed comment said moving the headline meant the corpus gitlink moved and the same commit carried the evidence. The new comment on lines 90-95 drops that link and names corpus 93d956a2 only in prose. Line 105 reads the gitlink and lines 111-118 check the checkout against it, but no check compares the gitlink with 93d956a2 or with any recorded measurement.

  • The actual gitlink value was not provided, so it is unknown whether it equals 93d956a2 today

  • Whether a separate review or CI step checks the corpus measurement was not inspected

  • Found by the per-file sweep; not double-checked.

Not confirmed · A test comment refers to a "previous grader", so the expectation for the 3-substitution case is undocumented and tied to an outside harness Inspect source

Detailed review · not confirmed

The comment at line 848 says the assertion was the first failure on a previous grader. That gives the reader no reason the 0.08 and moderate expectations are correct for this code.

Lines 847-858 pin exit_p90 to 0.08, pnl_p90 to 2/3, the observation count to 97, the label to moderate, and the CLI exit code to 1. The comment replaces the reason with a reference to a grader that the file does not describe.

  • The metric values were not recomputed from the code, so their correctness is unchecked here.

  • Reference checks do not establish safe deletion for dynamic or external consumers.

  • Found by the per-file sweep; not double-checked.

Not confirmed · gross_unmatched_count is a second name for count_abs_delta and can drift from it Inspect source

Detailed review · not confirmed

The new field gross_unmatched_count always holds the same value as count_abs_delta, so the report now carries two fields for one number.

The field comment says it is an alias of the canonical count_abs_delta. analyze_strategy sets it to count_abs_delta in both the early return (line 1253) and the main result (line 1558), and report_row exports it again (line 404).

  • Only the code shown in this file was checked; other report generators that read these fields were not inspected.

  • Reference checks do not establish safe deletion for dynamic or external consumers.

  • Found by the per-file sweep; not double-checked.

What this review covered
  • We didn't review 3 files that aren't source code, such as docs, config, images, or generated or third-party code.
  • Some lookups asked for files or lines that do not exist; the review was told and carried on.
  • 5 of 5 changed files inspected. This is scope, not a correctness guarantee.
Code graph and static checks · 345 functions depend on this change

Impact & health

Impact — 345 functions depend on the 328 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: analyze_strategy() — 16 callers, 23 callees
  • worse: _analyze_tapes() — 10 callers, 6 callees

Formal checks — 345 functions this change can affect weren't formally checked in this review. Formal checks are informational only.

Gate & verification

✅ PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 345 function(s) in the blast radius were not formally verified this run

Test selection

1465 of 1465 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • benchmarks/tests/test_aggregate.py — full-run-safety
  • benchmarks/tests/test_check_provenance.py — full-run-safety
  • benchmarks/tests/test_compare.py — full-run-safety
  • benchmarks/tests/test_public_names.py — full-run-safety
  • benchmarks/tests/test_run_all.py — full-run-safety
  • benchmarks/tests/test_run_strategy_feeds.py — full-run-safety
  • docker/pineforge_sink_finalize_test.py — full-run-safety
  • docker/pineforge_sink_replay_rq_test.py — full-run-safety
  • docker/pineforge_sink_replay_test.py — full-run-safety
  • docker/pineforge_sink_to_json_test.py — full-run-safety
  • docker/quota_columns_test.py — full-run-safety
  • docker/quota_convert_test.py — full-run-safety
  • docker/quota_finalize_test.py — full-run-safety
  • docker/replay_rq_engine_test.py — full-run-safety
  • docker/run_json_artifact_window_test.py — full-run-safety
  • docker/run_json_bin_test.py — full-run-safety
  • docker/run_json_bounded_transport_test.py — full-run-safety
  • docker/run_json_bulk_equity_fuzz_test.py — full-run-safety
  • docker/run_json_bulk_equity_test.py — full-run-safety
  • docker/run_json_codes_test.py — full-run-safety
  • docker/run_json_converter_metadata_test.py — full-run-safety
  • docker/run_json_converter_session_test.py — full-run-safety
  • docker/run_json_diagnostics_test.py — full-run-safety
  • docker/run_json_equity_scan_test.py — full-run-safety
  • docker/run_json_fused_test.py — full-run-safety
  • docker/run_json_harnesses_test.py — full-run-safety
  • docker/run_json_metrics_only_test.py — full-run-safety
  • docker/run_json_misc2_test.py — full-run-safety
  • docker/run_json_misc_test.py — full-run-safety
  • docker/run_json_pfbar_test.py — full-run-safety
  • docker/run_json_provenance_test.py — full-run-safety
  • docker/run_json_quota_supervisor_test.py — full-run-safety
  • docker/run_json_replay_test.py — full-run-safety
  • docker/run_json_selected_sink_test.py — full-run-safety
  • docker/run_json_sink_lexical_test.py — full-run-safety
  • docker/run_json_sink_summary_test.py — full-run-safety
  • docker/run_json_sink_test.py — full-run-safety
  • docker/run_json_sink_vectors_test.py — full-run-safety
  • docker/run_json_spool_work_test.py — full-run-safety
  • docker/run_json_stringio_test.py — full-run-safety
  • docker/run_json_symbol_feeds_test.py — full-run-safety
  • docker/run_json_syminfo_test.py — full-run-safety
  • docker/run_json_trade_scan_test.py — full-run-safety
  • docker/run_json_window_budget_test.py — full-run-safety
  • docker/run_json_wrapper_work_test.py — full-run-safety
  • include/pineforge/native_run_spec.hpp — full-run-safety
  • scripts/derive_corpus_feeds_self_test.py — full-run-safety
  • scripts/fingerprint_self_test.py — full-run-safety
  • scripts/run_corpus_self_test.py — full-run-safety
  • scripts/test_abi_receipt_skips.py — full-run-safety
  • … and 1415 more

⚠️ non-code file(s) changed (CHANGELOG.md, README.md, docs/ci.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)

⚠️ changed code file(s) with no mapped test (CHANGELOG.md, README.md, docs/ci.md, scripts/check_corpus_parity.sh) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Every flagged function

  • analyze_strategy() in scripts/verify_corpus.py:1165 · 16 callers, 23 callees: fans out to 23 callees (efferent coupling); 16 callers depend on it (afferent coupling)
  • _analyze_tapes() in scripts/test_verify_corpus_count_rule.py:103 · 10 callers, 6 callees: fans out to 6 callees (efferent coupling); 10 callers depend on it (afferent coupling)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant