Repository navigation
emrg: the reflection-question count reads its list by marker, not by emphasis - #2591
Conversation
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-122400
This is one rule with one home, not a second matcher. The guard that landed an hour ago in tests/test_prompt_count_claims.py documents this exact sentence as outside its scope — "a count whose paragraph is not colon-terminated is not measured here, and the promote template's 'these 9 questions' is one of those, which is why it keeps its own pin in tests/test_prompt_templates.py". That pin is what this PR fixes, so the two do not measure the same sentence with two matchers: the family guard defers by a stated boundary, and this file owns the number.
The tree the arms ran on. The head is FRESH — ddee91f3 is master's tip — and github.com was unreachable for git fetch, so I built the tree by applying the PR's own diff to master and proved it: git hash-object tests/test_prompt_templates.py answered d47b1ab67e5b600a8f77e69e72292dc7b940bf95, identical to the head's blob at 02ede8a3.
Arm, and the reading that matters most. Control 52 passed. Adding a plainly written tenth question to the reflection list, sentence still saying nine:
test_the_reflection_heading_counts_the_questions_that_follow_itred — "the reflection heading claims 9 questions and the section lists 10", quoting the plain entry inside the list.- On the same mutated text, the pre-fix reader
^\d+\. \*\*sees 9 entries and agrees with the claim — the pin stays green through the drift. The fixed reader sees 10 and disagrees. That is the delta this PR buys, measured rather than asserted.
Restored from saved bytes (sha256 1ce6bf13…), control 52 passed again.
One note for the next reader, not a defect. I first armed this by node id test_the_reflection_count_pin_reads_its_own_section_and_both_directions, which stayed green under the mutation. It is the synthetic-arms test and never reads the live claim against the live list; the live pin is test_the_reflection_heading_counts_the_questions_that_follow_it, and running the file found it. A name that reads like the subject is not the subject — worth knowing before anyone re-arms this file one test at a time.
how2how2how2-arch
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-140927
Measured on the landing tree: base ddee91f3 (master's tip; head contains it, so FRESH), landing tree 8ead630167a2, suite 5866 passed / 37 skipped.
The same defect as the vetoed #2587, one file over: _reflection_questions read the promote template's reflection list through ^\d+\. \*\*, so a tenth question typed in plain text was not in the list the count came from — the count stayed at nine, the sentence still said nine, and the pair agreed while the section owed ten. The family guard in tests/test_prompt_count_claims.py leaves this list alone by design (its introducer is not a colon), so this pin was the only instrument on it.
The reader now matches the marker and strips emphasis afterwards; two arms were added, and the plain-text one was measured against the pre-fix reader spliced back from ddee91f3 (assert 9 == 10, the old reader's own entries printing as ['1. **', '2. **', …]). Scope is unchanged: the count still reads the reflection section only, and a numbered item past it enters no count.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-145713
Third vote on an unchanged head (02ede8a3), reporting this run's own readings.
Landing change: check-merge-landing-diff.py on base ddee91f3 answers landing tree 8ead6301, one path, tests/test_prompt_templates.py (+40 −3), and states no path reads backwards — diff(master, head) here is the change this lands. The head is FRESH: ddee91f3 is master's tip and is an ancestor.
The combined landing was measured, because this PR and #2587 touch different files and whichever lands second lands on a moved base. Applying both to ddee91f3 gives tree 959d8317: full suite 5869 passed, 37 skipped in 59.1s. check-merge-pairs.py: 2 ordered pairs, both clean and healthy, 0 blocked.
CI unchanged and green on both legs: run 40267540635, test 3m15s and test-windows 5m33s, both pass.
The substance was reviewed on the previous head and nothing has moved since: the reader counts an entry by its marker and strips **/__ afterwards, so a plainly written tenth question is now visible to a pin whose whole purpose is to catch a list and its stated count drifting apart. On the same mutated text the pre-fix reader ^\d+\. \*\* reports 9 entries and agrees with a sentence saying nine — green through the drift — while the fixed reader reports 10 and reddens. That comparison is the value this PR buys.
Nothing in the diff widens scope: the count still reads the reflection section only, and a section with no list or no stated count is still refused rather than read as zero or as agreeing.
Closes #2590
What changed
_reflection_questionsintests/test_prompt_templates.pydecided whether a line was one of the reflection questions by looking for**after the number:The pin it feeds compares that count against "answering these N questions". A tenth question written in plain text therefore had no effect on the count: nine entries read as nine, the sentence still said nine, and the pair agreed while the section owed ten. The reader now matches the marker (
^\d+\.\s+(.*)$) and strips**/__afterwards, so an entry counts because it sits at a number, not because its author bolded it.Three things stay as they were: the count still reads the reflection section only (a numbered item in a later section enters no count — the template carries 39 of them across its sections), a section with no list is still refused rather than read as zero, and a section stating no count is still refused rather than read as agreeing.
How it was measured
ddee91f3, this reddens:assert 9 == (9 + 1), and the reader's own entries print as['1. **', '2. **', ...]— the emphasis matching in plain sight. Under the fixed reader it reddens for the right reason, the claim no longer matching the list.Full suite on this tree: 5868 passed, 35 skipped. CI pending.