Repository navigation
Phase 7: the extractor - #21
Merged
Merged
Conversation
…h fact Tasks 1 and 2 read the 19 misses of sample A that no stored fact answers. In 14 of them, the user stated the fact in passing, in a turn about another topic. In 5, the answer is a value that the assistant said. Task 2 found no reply at the token limit, no reply that fails to parse, and no near fact in "Facts already in memory". plan.md records the findings. The variant `detail` asks the extractor to read every turn, to keep a fact that the user states in passing, to keep each item of a list, and to write absolute dates. Each fact now carries the source `session:<hash>`, with 12 hex digits of the SHA-256 of the session id. The block shows sources to the reader, and an evidence session id holds `answer`, so the source does not show the id. `--no-fact-sources` runs the memory arm as phases 5 and 6 did. The diagnosis gives the session of each fact that holds the answer.
…imit The first runs with sources stopped on question a3332713. The extractor reply for one session reached the limit of 4,000 output tokens. In that session the user pastes a list of survey answers from other people. A reply with a schema that reaches its limit now comes back with `stopped` set, and the cache keeps it. The extractor writes no facts for that session, so a replay and the diagnosis skip the same session. Each result row and the result file count these sessions in `sessions_at_limit`. The marker calls of the diagnosis still stop the run at the limit.
On sample A, the baseline with sources scored 22 of 54 and `detail` scored 31. On sample B, `detail` scored 32, against 27 for the phase 6 baseline. The group with no stored fact for the answer went from 19 to 16 on sample A and from 16 to 14 on sample B. docs/benchmarks.md reports each run with its prompt, model, cost, and diagnosis. It also records that the marker of the diagnosis can miss stored facts. plan.md records the owner's acceptance of `detail` below the stop condition, that task 6 did not run, and the spend of the phase.
The README gave only the baseline scores of the memory arm. It now also gives the scores of the `detail` prompt on both samples. A replay of sample A with `--no-fact-sources` gave the same 54 answers and marks as phase 6, with no model call. docs/benchmarks.md records it.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19.
This PR adds a second extractor prompt for the memory arm,
detail, and a session source on each fact. The library does not change. All work is inbench/.How
bench/prompts.py: thedetailvariant. It tells the extractor to read every turn, to keep a fact that the user states in passing, to keep each item of a list, and to write absolute dates.baselineis unchanged.bench/e2e.py: each fact carriessession:<hash>, 12 hex digits of the SHA-256 of the session id. The block shows sources to the reader, and LongMemEval names evidence sessionsanswer_..., so the plain id would tell the reader which facts hold the answer.--no-fact-sourceskeeps the phase 5 and 6 behavior.bench/models.py: a reply with a schema that reaches its token limit comes back withstoppedset, and the cache keeps it. The extractor writes no facts for that session and counts it insessions_at_limit.bench/diagnose.py: names the session of each fact that holds the answer.Results (
docs/benchmarks.md)baseline, no sources (phase 6)baseline, with sourcesdetail, with sourcesdetailVerification
make verify, with new offline tests for the variant, the source, and a reply at the limit.--no-fact-sourcesgave the same 54 answers and marks as phase 6, with no model call.Notes
detailmet a stop condition: it cut the target group from 19 to 16, and the issue asked for 12 or fewer. The owner accepted it.docs/internal/plan.mdrecords the findings and decisions.detailon sample A, the facts exist but were not marked.detailstill drops values that the assistant said, in 7 single-session-assistant misses on sample A.