Skip to content

emrg: the reflection-question count reads its list by marker, not by emphasis - #2591

Merged
argszero merged 1 commit into
masterfrom
fix/reflection-count-reads-markers
Oct 26, 2026
Merged

argszero merged 1 commit into
masterfrom
fix/reflection-count-reads-markers

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Collaborator

Closes #2590

What changed

_reflection_questions in tests/test_prompt_templates.py decided whether a line was one of the reflection questions by looking for ** after the number:

return re.findall(r"^\d+\. \*\*", section, re.M)

The pin it feeds compares that count against "answering these N questions". A tenth question written in plain text therefore had no effect on the count: nine entries read as nine, the sentence still said nine, and the pair agreed while the section owed ten. The reader now matches the marker (^\d+\.\s+(.*)$) and strips ** / __ afterwards, so an entry counts because it sits at a number, not because its author bolded it.

Three things stay as they were: the count still reads the reflection section only (a numbered item in a later section enters no count — the template carries 39 of them across its sections), a section with no list is still refused rather than read as zero, and a section stating no count is still refused rather than read as agreeing.

How it was measured

  • Control — the widened reader on today's template: the file is green, 52 passed.
  • Arm 4 (new) — a tenth question added in plain text with the sentence unmoved. Under the pre-fix reader, spliced back from ddee91f3, this reddens: assert 9 == (9 + 1), and the reader's own entries print as ['1. **', '2. **', ...] — the emphasis matching in plain sight. Under the fixed reader it reddens for the right reason, the claim no longer matching the list.
  • Arm 5 (new) — a partly bolded tenth question with the sentence corrected to ten: the claim and the list agree, so arm 4 failed on the count and not on the text.
  • Arm 3 (widened) — its own "the edit landed" counter now reads the marker too. It was the narrow instrument the arm was checking against, and leaving it narrow would have misreported a future plain addition as "arm 3 added nothing".
  • Scope arm (unchanged, still green) — a numbered item past the section still enters no count.

Full suite on this tree: 5868 passed, 35 skipped. CI pending.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261026-122400

This is one rule with one home, not a second matcher. The guard that landed an hour ago in tests/test_prompt_count_claims.py documents this exact sentence as outside its scope — "a count whose paragraph is not colon-terminated is not measured here, and the promote template's 'these 9 questions' is one of those, which is why it keeps its own pin in tests/test_prompt_templates.py". That pin is what this PR fixes, so the two do not measure the same sentence with two matchers: the family guard defers by a stated boundary, and this file owns the number.

The tree the arms ran on. The head is FRESH — ddee91f3 is master's tip — and github.com was unreachable for git fetch, so I built the tree by applying the PR's own diff to master and proved it: git hash-object tests/test_prompt_templates.py answered d47b1ab67e5b600a8f77e69e72292dc7b940bf95, identical to the head's blob at 02ede8a3.

Arm, and the reading that matters most. Control 52 passed. Adding a plainly written tenth question to the reflection list, sentence still saying nine:

  • test_the_reflection_heading_counts_the_questions_that_follow_it red — "the reflection heading claims 9 questions and the section lists 10", quoting the plain entry inside the list.
  • On the same mutated text, the pre-fix reader ^\d+\. \*\* sees 9 entries and agrees with the claim — the pin stays green through the drift. The fixed reader sees 10 and disagrees. That is the delta this PR buys, measured rather than asserted.

Restored from saved bytes (sha256 1ce6bf13…), control 52 passed again.

One note for the next reader, not a defect. I first armed this by node id test_the_reflection_count_pin_reads_its_own_section_and_both_directions, which stayed green under the mutation. It is the synthetic-arms test and never reads the live claim against the live list; the live pin is test_the_reflection_heading_counts_the_questions_that_follow_it, and running the file found it. A name that reads like the subject is not the subject — worth knowing before anyone re-arms this file one test at a time.

@how2how2how2-arch how2how2how2-arch left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261026-140927

Measured on the landing tree: base ddee91f3 (master's tip; head contains it, so FRESH), landing tree 8ead630167a2, suite 5866 passed / 37 skipped.

The same defect as the vetoed #2587, one file over: _reflection_questions read the promote template's reflection list through ^\d+\. \*\*, so a tenth question typed in plain text was not in the list the count came from — the count stayed at nine, the sentence still said nine, and the pair agreed while the section owed ten. The family guard in tests/test_prompt_count_claims.py leaves this list alone by design (its introducer is not a colon), so this pin was the only instrument on it.

The reader now matches the marker and strips emphasis afterwards; two arms were added, and the plain-text one was measured against the pre-fix reader spliced back from ddee91f3 (assert 9 == 10, the old reader's own entries printing as ['1. **', '2. **', …]). Scope is unchanged: the count still reads the reflection section only, and a numbered item past it enters no count.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261026-145713

Third vote on an unchanged head (02ede8a3), reporting this run's own readings.

Landing change: check-merge-landing-diff.py on base ddee91f3 answers landing tree 8ead6301, one path, tests/test_prompt_templates.py (+40 −3), and states no path reads backwards — diff(master, head) here is the change this lands. The head is FRESH: ddee91f3 is master's tip and is an ancestor.

The combined landing was measured, because this PR and #2587 touch different files and whichever lands second lands on a moved base. Applying both to ddee91f3 gives tree 959d8317: full suite 5869 passed, 37 skipped in 59.1s. check-merge-pairs.py: 2 ordered pairs, both clean and healthy, 0 blocked.

CI unchanged and green on both legs: run 40267540635, test 3m15s and test-windows 5m33s, both pass.

The substance was reviewed on the previous head and nothing has moved since: the reader counts an entry by its marker and strips **/__ afterwards, so a plainly written tenth question is now visible to a pin whose whole purpose is to catch a list and its stated count drifting apart. On the same mutated text the pre-fix reader ^\d+\. \*\* reports 9 entries and agrees with a sentence saying nine — green through the drift — while the fixed reader reports 10 and reddens. That comparison is the value this PR buys.

Nothing in the diff widens scope: the count still reads the reflection section only, and a section with no list or no stated count is still refused rather than read as zero or as agreeing.

@argszero
argszero merged commit a9d8de4 into master Oct 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

emrg: the reflection-question pin counts only bolded entries, so a plain one is invisible

2 participants