Skip to content

thread-briefs: ask the summarizer for status instead of deriving it - #10

Merged
trotterdylan merged 1 commit into
mainfrom
bb/thread-briefs-finished-threads-almost-never-read-thr_8txf3h94jw
Oct 5, 2026
Merged

trotterdylan merged 1 commit into
mainfrom
bb/thread-briefs-finished-threads-almost-never-read-thr_8txf3h94jw

Conversation

@trotterdylan

Copy link
Copy Markdown
Collaborator

Why

Finished threads almost never read Done. On the bb-dylan server, 11 of 21 Done threads were Done only because someone had pinned them by hand. Each of those 11 was checked against its transcript:

Cause Threads Fixed here?
An offer or optional check recorded as nextStep ("want me to file an issue?", "if you want the hash confirmed, run…") 4 Yes: status is asked for directly, and nextStep is descriptive only
nextStep/blockedOn carried over from the previous brief after the thread moved on 3 Yes: neither is fed back any more
A step only the user can take, done outside the thread (approve PR, merge a branch, rerun a job) 4 Partly: see results

Status used to be derived: nextStep and blockedOn both empty meant Done, while the prompt asked for "the single most concrete next action". This replaces the derivation instead of adding a fifth rule to it. #9 took the other route and should be closed in favour of this.

What changed

  • Status is asked for directly, as the stored enum, with one definition and one example each. The question is: assume the user does whatever the thread asks of them. Is the task then finished? A Done thread may still carry a suggestion in nextStep.
  • Shorter prompt. The four done-related rules and the "stage and nextStep must agree" rule are deleted. The whole prompt drops from 1100 words to about 850, and the done-related text from about 470 words to about 190.
  • No field-against-field overrides. reconcileStage is removed. The parser keeps only one hard guard, which holds whatever the model says: a non-empty blockedOn is never done. A status it can't read falls back to waiting-on-me.
  • Previous brief fed back now carries only title, goal, currentState and constraints.
  • modelStatus is stored on the row. Rows written before this change have no modelStatus and use the old derivation (legacyStatus) until their thread is next summarized.
  • eval/: export.ts freezes real threads from a server, and run.ts scores any checkout's prompt against the live model. Point --src at a worktree of main to compare. The fixtures are real transcripts, and this repo is public, so they are not committed.
  • README and SKILL.md "Stage and status" are rewritten to match.

Evaluation

Model: glm-5p3-flash on Fireworks (the production setting), temperature 0. Labels follow the definition above and were agreed with the user.

27 current threads (every briefed thread on bb-dylan except the one doing this work; 22 done, 5 waiting), previous brief fed back, 2 runs each:

Agreement False Done Missed Done
main 30/54 (56%) 1 23
this PR 45/54 (83%) 0 9

56 mid-thread snapshots. These are the same threads cut at points where the agent stopped and the user replied: 46 waiting, 10 done, labelled by hand, with 27 ambiguous cuts skipped. They exist to catch false Dones, which the set above has too few negatives to show:

Agreement False Done Missed Done
main 46/56 (82%) 2 8
this PR 50/56 (89%) 2 4

What still misses. The model still reads a step that is the user's alone ("merge the guidance branch", "run kploy image add") as Waiting on you about half the time. That is the safe direction, and the manual pin covers it. Pushing the prompt further toward Done on this small set would be tuning to these exact threads. The two false Dones in the snapshots are the same as on main: an agent offering options and stopping.

Archiving

A Done thread can now carry a step that is the user's, such as "approve PR #12". It is archived on the usual 48-hour clock. That is a deliberate choice, documented in both docs:

  • the step stays on the Done card for two days;
  • the ring greys after 24 hours, with "archiving soon" on hover;
  • a thread you un-archive by hand is never auto-archived again.

Rollout

  • Merging restarts the bb-dylan server within about 10 minutes, via the plugin pin sync. Merge at a quiet moment.
  • No one-time re-summary. Every old idle thread that would now read Done is already past 48 hours idle, so the next hourly sweep would archive them all at once. Old briefs change on their next real activity, or when someone presses Re-summarize.

Tests: npm test (392 passing) and npm run typecheck, both clean.

🤖 Generated with Claude Code

Done was a side effect of the model leaving nextStep and blockedOn blank,
while the same prompt asked for "the single most concrete next action". On
the bb-dylan server 11 of 21 Done threads were Done only by a hand pin. Read
against their transcripts, they failed because an offer or optional check
was recorded as the step, because nextStep/blockedOn were fed back from the
previous brief and outlived the turns that retired them, or because the step
left was the user's own.

- The prompt asks for status directly, as the stored enum, with one
  definition and one example each, framed as: assume the user does what the
  thread asks; is the task then finished? nextStep becomes descriptive.
- The four done-related rules and the stage/nextStep agreement rule are gone
  (prompt 1100 -> ~850 words), and so is reconcileStage. The parser keeps one
  hard guard: a non-empty blockedOn is never done. An unreadable status
  falls back to waiting-on-me.
- The previous brief fed back carries only title, goal, currentState and
  constraints.
- modelStatus is stored on the row; briefs written before it keep the old
  derivation (legacyStatus) until their thread is next summarized.
- eval/ exports real threads from a server and scores any checkout's prompt
  against the live model. Fixtures stay out of the repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trotterdylan
trotterdylan merged commit edbc4d9 into main Oct 5, 2026
6 checks passed
@trotterdylan
trotterdylan deleted the bb/thread-briefs-finished-threads-almost-never-read-thr_8txf3h94jw branch October 5, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant