diff --git a/docs/internal/plan.md b/docs/internal/plan.md index 4fa0b84..1e49911 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -1,19 +1,23 @@ # Plan: the second workstream, memory accuracy Status: agreed, 2026-10-13. The owner agreed to each decision and to the target on -2026-10-13. Phase 7 closed on 2026-10-14 with PR #21. Phase 8 is open, as issue -#22. This file holds the open workstream, phases 8 to 13. Each phase becomes one GitHub +2026-10-13. Phase 8 closed on 2026-10-15 with PR #24. On 2026-10-16 the owner added +three phases after it: faster evaluation, the ranking with episode chunks, and a large +memory store. The later phases moved to 12 to 15. Phase 9 is open as issue #25, and +the brief of phase 10 is issue #26. This file holds the open workstream, +phases 9 to 15. Each phase becomes one GitHub issue and one pull request, worked in task order. When a phase closes, `roadmap.md` gets its paragraph, and this file keeps only the phases still open. The plan of phase 5 is in git history, at commit `4a6fb36`. The plan of phase 6, with the decisions taken during -its work, is at commit `8a7318d`. The plan of phase 7 is at commit `ab93da1`. +its work, is at commit `8a7318d`. The plan of phase 7 is at commit `ab93da1`. The plan +of phase 8, with the decisions of its task 1, is at commit `b5443ac`. ## Goal When this workstream closes, the memory arm meets the target below, and hotmemory is on PyPI. At the start of the workstream, the memory arm scored 19 of 54 on sample A (35.2 -percent). After phase 7, it scores 31 of 54 on sample A and 32 on sample B. The retrieval -arm scored 45 of 54 on sample A. +percent). After phase 8, it scores 45 of 54 on sample A and 45 on sample B, with +episodes after the facts. The retrieval arm scored 45 of 54 on sample A. The target (agreed) is a distance from the lowest published score of a memory layer with a GPT-4o reader: Zep at 71.2 percent (read, not reproduced). The published range goes up @@ -24,7 +28,7 @@ to 84.2 percent. | Pass | At most 20 points | At least 28 of 54 on each | At least 51.2 percent | | Goal | At most 10 points | At least 33 of 54 on each | At least 61.2 percent | -The workstream closes on the pass level, measured in phase 12 on all 500 questions. The +The workstream closes on the pass level, measured in phase 14 on all 500 questions. The sample counts guide the phases before it. A sample of 54 is uncertain by about 10 points, so the number on all 500 questions decides. @@ -39,7 +43,7 @@ writes, and does not try to reduce it. The roadmap lists the latency work under All numbers are from `docs/benchmarks.md`. The rows of sample A were observed on 2026-10-10, and the rows of sample B on 2026-10-14. The rows of phase 7 were observed on -2026-10-14. +2026-10-14, and the rows of phase 8 on 2026-10-15. | Item | Value | |---|---| @@ -59,6 +63,11 @@ All numbers are from `docs/benchmarks.md`. The rows of sample A were observed on | Phase 7, memory arm with `detail` and sources, sample B | 32 correct, 1,664 reader tokens at the median | | Phase 7 misses with `detail`, no stored fact held the answer | 16 of 23 on sample A, 14 of 22 on sample B | | Phase 7 misses with `detail`, the block held some needed facts | 6 of 23 on sample A, 7 of 22 on sample B | +| Phase 8, facts only, the baseline of phase 8 | 29 on sample A, 31 on sample B, 1,715 reader tokens at the median | +| Phase 8, `episodes="search"` at 64,000 characters | 45 on sample A, 45 on sample B, 12,849 and 12,676 reader tokens | +| Phase 8, `episodes="search"` at 32,000 characters | 43 on sample A, 5,715 reader tokens | +| Phase 8 misses with `search` at 64,000 characters | 9 on each sample: 5 with no stored fact, and an evidence session in the block in 8 | +| Phase 8 run time on sample A, 4 workers, from `duration_s` in the result files | 1,145 s for facts only, 1,507 s for `search` at 64,000 characters, 1,133 to 2,085 s over the seven runs | Four facts about the memory arm, read from `bench/e2e.py` and `src/hotmemory/contract.py`, shape the phases below: @@ -106,104 +115,69 @@ The owner agreed to each item on 2026-10-13. `docs/contracts.md`, and `docs/guarantees.md` in the same commit, and `MemoryStore` gets the same behavior as `HotdataStore`. -## Phase 8. Episodes behind facts +## Phase 9. Faster evaluation -Goal: `recall` can give the reader the source text of its facts. The retrieval arm shows -that whole sessions answer 45 of 54. This phase lets the memory contract use them, within -a budget. +Goal: a run that changes only `recall` or the reader takes minutes, not 20 minutes, and a +smoke run checks a change in a few minutes. This phase changes `bench/` only. -The target is the two largest groups of misses after phase 7. With `detail`, no stored fact -held the answer in 16 of 23 misses on sample A and 14 of 22 on sample B. The block held -some of the needed facts in 6 and 7. An episode holds the turns that the extractor -dropped, so it can answer both groups. - -This phase changes the library. The memory arm runs with `--extractor detail` and fact -sources, the configuration of phase 7. Its results are the baseline of this phase: 31 of -54 on sample A and 32 on sample B. - -What phase 7 left for this phase, read from `bench/e2e.py` and `src/hotmemory/contract.py`, -not observed: - -- Each benchmark fact carries the source `session:`, set by the extractor in - `bench/e2e.py`. `load` adds the id of its episode to the sources of each fact, and the id - is `/@`. If `capture` does the same, each fact gets a second, - longer source. -- The block shows the sources to the reader, and `capture` passes the block of `recall` to - the extractor as "Facts already in memory". A change to the sources changes both - prompts, so the extractor cache of phase 7 does not replay. Phase 7 observed this for - its own change. A run with a new source pays for the extractor again, at about the cost - of a run of phase 7. -- The key of the episode must not show the session id, for the reason of phase 7: - LongMemEval names each evidence session `answer_...`, and the reader sees the sources. +Phase 8 ran seven configurations on sample A. Each run captured the same memory again, +because the episodes value and the budget act after `capture`. A run took 1,133 to +2,085 seconds with 4 workers (observed, `duration_s` in the result files), and most of +that time was `capture`: about four engine round trips for each of about 2,700 sessions +(read from the code, not timed). +The diagnosis replays `capture` again. An internet outage stopped four runs, and each +lost every finished question, because a run writes its result file only at the end. Tasks: -1. Propose the surface in `docs/contracts.md` and stop for the owner's agreement. The - proposal below is a starting point: - - `capture` takes an optional `episode` key. With it, `capture` stores the text as an - episode under that key, and adds the episode id to the `sources` of each fact. This - is what `load` does for each chunk, applied to one text. - - `recall` takes an optional `episodes` argument with three values: none (today), the - episodes that the recalled facts cite, or the episodes that a search over episodes - ranks first. The block gives the facts first, then the episodes, each with its date. - One `budget` covers both, and the facts never lose their place to an episode. - - How a fact shows its episode. The proposal names the source string that the block - shows, and whether the extractor of the benchmark still sets `session:`. -2. Build the agreed surface in `Memory`, for both drivers. The conformance suite, the - ledger in `docs/guarantees.md`, the skill commands, and `CHANGELOG.md` follow. -3. The memory arm captures each session with an episode key, the hash of phase 7. Then it - recalls with each `episodes` value, at three budgets: 16,000, 32,000, and 64,000 - characters. The search inside `capture` does not change with these values, so each - run after the first replays `capture` from the extractor cache (read from source, not - observed). Only the reader and the judge pay for each value and budget. -4. `bench/diagnose.py` reports, for each miss, whether the block held an evidence session - as an episode. -5. Report accuracy against reader tokens for each value on sample A, then the best one - on sample B, with the diagnosis and the cost of each run. - -Decisions of task 1, agreed by the owner on 2026-10-15. `docs/contracts.md` holds the -surface. - -- `capture` takes an optional `episode` key. `remember` writes the facts, and then one - `put` writes the episode, as in `load`. The proposal said one writer, but a flush of - `HotdataStore` loads its tables at the same time, so the order of the two writes was - not fixed (read from source, not observed). If the episode is current with the same - content, `capture` calls no extractor, writes nothing, and returns no id. -- `recall` takes `episodes`: none, `"cited"`, or `"search"`. Both values read with the - `Store` methods of today. `"search"` uses the same `k` as the facts. -- The block gives the facts first, then a line `episodes:`, then each episode under a - header `## @ [observed: ]`. It holds whole episodes only, - and it stops before the first episode that does not fit. -- A source that is the id of a record in the same namespace shows as `@`. - So the run id of the benchmark is in no prompt, and a run after the first replays - `capture` from the cache. The blocks of documents that `load` stored get shorter. -- The benchmark extractor stops setting `session:`. The episode key is - `session-`, with the hash of phase 7. The first run pays for the extractor again. -- The budgets are 16,000, 32,000, and 64,000 characters. The reason for dropping 8,000 was - an estimate that the facts of phase 7 leave no room for a whole session. The runs then - showed blocks of facts of at most 5,626 characters, so the estimate was too high. +1. Keep the memory of a run. With `--keep`, the runner does not empty the namespaces, and + it keeps the database of each worker. The name of the database states what decides + the memory: the sample, the extractor variant, the episode setting, the fact sources, + the ranking, the worker count, and the last commit that changed `src/hotmemory`. The + runner records the name in the result file. +2. Reuse the memory. With `--reuse`, the runner opens the kept databases, skips + `capture`, and runs `recall`, the reader, and the judge. It refuses to run if a + database with the expected name does not exist. +3. Let `bench.diagnose` read a kept database, so that it does not replay `capture`. +4. A smoke run. `--smoke` runs 18 questions, 3 of each type from sample A, on + `MemoryStore`, in the process. A committed file lists the 18 ids. The smoke run checks + that a change runs and that the block looks right. It decides no gain: 18 questions can + be off by about 23 points, and `MemoryStore` ranks by vector only. +5. Keep finished questions. The runner appends each finished row to a file as it goes, + and `--resume` skips the rows that the file holds. A failed call to a model fails its + question and not the whole run. +6. Prove the reuse. Run `search` at 64,000 characters on sample A with `--keep`, then + `search` at 32,000 and `cited` at 64,000 with `--reuse`. Each answer must equal the + answer of the phase 8 run of the same configuration, from the caches. Record the time + of each run. +7. Update `bench/README.md` and `docs/benchmarks.md` with the new options and the times. Acceptance criteria: -- `MemoryStore` and `HotdataStore` return the same block for the same calls, in the - conformance suite. -- A capture with an episode writes the episode and the facts in one flush per table, and - a retried capture writes nothing new. -- `docs/benchmarks.md` reports accuracy and reader tokens for each value and budget. -- The best value scores more than 31 of 54 on sample A and more than 32 on sample B, the - results of phase 7. +- A run with `--reuse` gives the same block, answer, and mark for each question as a run + that captures, and it takes less than 5 minutes on sample A with cached replies. +- A smoke run with cached replies takes less than 3 minutes. +- A run that loses its model connection keeps its finished questions, and `--resume` + completes it. +- No file in `src/hotmemory` changes. Stop and ask if: -- The surface needs a new `Store` method or a new record field. -- The best value needs more reader tokens than the retrieval arm, 29,878 at the median. - Then memory costs as much as retrieval, and the owner decides whether the trade holds. -- The estimate of a run is over 50 dollars. +- A reused memory gives a different block than a fresh capture for any question. +- The kept databases of one sample need more than 10 GB on the local stack. -## Phase 9. The ranking +## Phase 10. The ranking, and episode chunks Goal: the default ranking of `HotdataStore` is the one that scores best, and the reason -for the gap between the two rankings is known. +for the gap between the two rankings is known. The reader gets the accuracy of phase 8 +for fewer tokens, because the block holds the chunks of a session that match, and not +the whole session. + +In phase 8, `search` at 64,000 characters scored 45 on both samples with about 12,800 +reader tokens at the median. At 32,000 characters it scored 43 on sample A with 5,715 +tokens. A session is one episode, so an episode that holds one needed turn still fills +the budget with every other turn. `load` already cuts a document into chunks of +`chunk_chars`, and each chunk is an episode. Tasks: @@ -211,28 +185,64 @@ Tasks: query with one arm left out at a time: BM25 and content, content and cue, and BM25 alone. Episodes have no cues, so in the retrieval run the cue arm adds nothing. Find which arm costs the most and on which question types. -2. Run the memory arm on sample A with each ranking, with the best configuration of - phases 7 and 8. -3. Propose the default to the owner: the vector ranking, or a rebuilt fusion. A rebuilt - fusion can rank BM25 inside the scope, or fuse two arms with no weights. Stop for the - agreement. -4. Build the agreed change, with `CHANGELOG.md`, `docs/contracts.md`, and - `docs/guarantees.md`. If the default changes, the oracle test and the benchmark runner - follow. -5. Run the agreed ranking on sample B, and rerun the retrieval score on all 419 questions. +2. Run the memory arm on sample A with each ranking, with `search` at 64,000 characters. + With phase 9, each ranking captures once and reuses the memory. +3. Propose the default ranking and the surface of episode chunks in `docs/contracts.md`, + and stop for the owner's agreement. The ranking: the vector ranking, or a rebuilt + fusion that ranks BM25 inside the scope or fuses two arms with no weights. The chunks, + as a starting point: `capture` with `episode` takes `chunk_chars`, cuts the text at + blank lines as `load` does, and stores chunk n under `-`. The extractor + still reads the whole text, so its prompt and its cache do not change. The proposal + names which chunks a fact cites, and how `"cited"` uses them. +4. Build the agreed change for both drivers, with `CHANGELOG.md`, `docs/contracts.md`, + `docs/guarantees.md`, and the skill commands. If the default ranking changes, the + oracle test and the benchmark runner follow. +5. Run `search` with chunks of 1,000, 2,000, and 4,000 characters, at budgets of 8,000, + 16,000, and 32,000 characters, on sample A, with reuse wherever the memory is the same. + Report accuracy against reader tokens, next to the phase 8 runs. +6. Run the best configuration on sample B, rerun the retrieval score on all 419 questions + with the agreed ranking, and diagnose the misses. Acceptance criteria: -- `docs/benchmarks.md` reports the ablation of task 1 and the retrieval score of the - agreed ranking. +- `docs/benchmarks.md` reports the ablation of task 1, the retrieval score of the agreed + ranking, and each chunk size and budget. - The search inside `capture` uses the same ranking as `recall`, and a test proves it. +- A configuration with chunks scores at least 45 on sample A and 45 on sample B with + fewer reader tokens at the median than 12,849, or the report states why none does. +- `MemoryStore` and `HotdataStore` return the same block for the same calls with chunks. Stop and ask if: - The ablation shows that the gap comes from the data and not from an arm, for example that BM25 favors a long session. - -## Phase 10. Supersede in capture +- Chunks need a new `Store` method or a new record field. +- No chunk size reaches 43 on sample A within 6,000 reader tokens at the median. + +## Phase 11. A large memory store + +Goal: numbers for hotmemory at 10,000, 100,000, and 1,000,000 memories: the time of a +write and of a search, and the accuracy of retrieval when the store is large. The owner +asked for it on 2026-10-16, and the tasks are planned when phase 10 closes. + +The parts discussed so far: + +- A synthetic scale test, with no reader model: load 10,000, 100,000, and 1,000,000 + records, from the facts and sessions that the benchmark already extracted. Measure the + rate of writes with table loads, and with native DML when the engine has it. Measure + search time at p50 and p95 for the fused and the vector ranking, for scopes that hold 1, + 10, and 100 percent of the rows, on the local stack and in the cloud. +- The index check. Today the vector rankings filter first and scan, because a filtered + search through the vector index misses rows loaded after the build. When the engine + fixes that, the vector rankings move to the index, and tests prove that every write, + revision, and delete is found through the vector index and the BM25 index. +- Retrieval at scale: LongMemEval-M, about 500 sessions for each question, scored by + `recall_any@k` with no reader model. +- A comparison with other memory layers in one harness, such as MemoryBench, with the + same questions, reader, and judge. It needs accounts for each service and the owner's + yes on its cost. + +## Phase 12. Supersede in capture Goal: when a session changes a fact, the old fact closes, so the reader sees one current value. Today a changed value gets a new key, and both values stay current. @@ -266,7 +276,7 @@ Acceptance criteria: - A changed value leaves one current record and one closed record with `valid_until` and `expired_at`, in both drivers. -- `recall` with `as_of` before the change returns the old value only after phase 11. +- `recall` with `as_of` before the change returns the old value only after phase 13. This phase states that limit in `docs/contracts.md`. Stop and ask if: @@ -274,7 +284,7 @@ Stop and ask if: - The extractor sets `replaces` on records that did not change in more than one case of ten in the offline test set. -## Phase 11. Search over history +## Phase 13. Search over history Goal: `recall(as_of=T)` returns what was true at T, including a fact that a later revision replaced. Today it reads current revisions only, and it applies `as_of` after @@ -309,7 +319,7 @@ Stop and ask if: - The as-of filter in SQL makes a local search slower than twice its time without the filter. -## Phase 12. The full benchmark and a second benchmark +## Phase 14. The full benchmark and a second benchmark Goal: the final configuration has numbers on all 500 questions, and on a second benchmark. @@ -317,7 +327,7 @@ benchmark. Tasks: 1. Run the retrieval arm and the memory arm on all 500 questions, with the configuration that - phases 7 to 11 agreed. Print the estimate first. The run needs the owner's yes, because + phases 7 to 13 agreed. Print the estimate first. The run needs the owner's yes, because it costs about ten times a sample run. 2. Add a LoCoMo runner to `bench/`, as a smoke test of 10 conversations and 1,540 questions. Its labels have known errors, which the research notes list. The report @@ -338,7 +348,7 @@ Stop and ask if: - The memory arm scores below the pass level on all 500 questions. The owner then decides whether the workstream closes or gets another phase. -## Phase 13. Release on PyPI +## Phase 15. Release on PyPI Goal: `pip install hotmemory` installs the first release. @@ -363,8 +373,9 @@ Acceptance criteria: These wait, and `roadmap.md` lists them under Deferred: -- Latency: native DML for writes, the filtered vector index, a conditional write, text - loaded only for ranked rows, and the per-request time of the cloud. +- Latency: a conditional write, text loaded only for ranked rows, and the per-request + time of the cloud. Native DML and the filtered vector index move to phase 11, where + they are measured at scale. - A model-backed extractor in an extra, and decision calls with a probability, such as a classifier that decides supersede or routes a recall. - The other ideas from the hindsight study of 2026-10-09: a chunk-hash delta on @@ -383,4 +394,4 @@ These wait, and `roadmap.md` lists them under Deferred: ## Questions for the owner -- Phase 13: the first version number, and whether to publish to TestPyPI first. +- Phase 15: the first version number, and whether to publish to TestPyPI first. diff --git a/docs/internal/roadmap.md b/docs/internal/roadmap.md index 5b2abfc..4fcd2af 100644 --- a/docs/internal/roadmap.md +++ b/docs/internal/roadmap.md @@ -25,18 +25,22 @@ that proves it. Agreed on 2026-10-13. `plan.md` holds the tasks, the acceptance criteria, and the stop conditions of each phase. The library stays deterministic: no code in `src/hotmemory` -calls a model. Latency is not a goal, and its work waits under Deferred. +calls a model. Latency is not a goal, and its work waits under Deferred, except the +measurements at scale of phase 11. On 2026-10-16 the owner added phases 9 to 11, and the +later phases moved to 12 to 15. | Phase | Delivers | Status | |---|---|---| | 6. The accuracy loop | A second sample of 54 questions, and the memory arm configured from the command line. The diagnosis split for each question type, and the baselines of both samples. | closed 2026-10-14, PR #18 | | 7. The extractor | A benchmark extractor that keeps what the assistant said, with sources, measured against the 19 misses where no stored fact held the answer. No library change. | closed 2026-10-14, PR #21 | -| 8. Episodes behind facts | `capture` stores its text as an episode that its facts cite. `recall` can add the cited or ranked episodes to its block, within one budget. | open, issue #22 | -| 9. The ranking | An ablation of the fused ranking, and a default ranking chosen by the benchmark. | proposed | -| 10. Supersede in capture | A fact can name the current record that it replaces, and `remember` closes that record. The extractor decides, and the library applies the rule. | proposed | -| 11. Search over history | `recall(as_of=T)` returns the revision valid at T, with the as-of filter in SQL, and a date window from the caller. | proposed | -| 12. The full benchmark | The final configuration on all 500 LongMemEval-S questions, and a LoCoMo smoke test. | proposed | -| 13. Release on PyPI | The release flow of the sibling repository `hotdata-langchain`, and the first release. | proposed | +| 8. Episodes behind facts | `capture` stores its text as an episode that its facts cite. `recall` can add the cited or ranked episodes to its block, within one budget. | closed 2026-10-15, PR #24 | +| 9. Faster evaluation | The runner keeps the memory of a run and reuses it for runs that change only `recall` or the reader. A smoke run of 18 questions on `MemoryStore`. A run keeps its finished questions when it stops. No library change. | open, issue #25 | +| 10. The ranking, and episode chunks | An ablation of the fused ranking, and a default ranking chosen by the benchmark. `capture` stores a session as chunks, so the block holds the chunks that match and not whole sessions, for fewer reader tokens. | proposed, issue #26 | +| 11. A large memory store | The time of a write and a search at 10,000, 100,000, and 1,000,000 memories, by scope size, locally and in the cloud. The vector rankings on the vector index once the engine fixes the filtered search, with tests that every record is found through both indexes. Retrieval on LongMemEval-M. Then a comparison with other memory layers in one harness. | proposed | +| 12. Supersede in capture | A fact can name the current record that it replaces, and `remember` closes that record. The extractor decides, and the library applies the rule. | proposed | +| 13. Search over history | `recall(as_of=T)` returns the revision valid at T, with the as-of filter in SQL, and a date window from the caller. | proposed | +| 14. The full benchmark | The final configuration on all 500 LongMemEval-S questions, and a LoCoMo smoke test. | proposed | +| 15. Release on PyPI | The release flow of the sibling repository `hotdata-langchain`, and the first release. | proposed | ## Deferred @@ -46,9 +50,10 @@ taken up now. - Adapters, planned as phase 6 of the first roadmap: a LangGraph `BaseStore` adapter in `hotdata-langchain`, then an MCP server. They wait until the memory contract scores well and is on PyPI. -- Latency, from phase 5: native DML for writes, the filtered vector index search, a - conditional write for more than one writer, text loaded only for ranked rows, and the - time of each cloud request. The second workstream measures accuracy first. +- Latency, from phase 5: a conditional write for more than one writer, text loaded only + for ranked rows, and the time of each cloud request. The second workstream measures + accuracy first. Native DML for writes and the filtered vector index search moved to + phase 11 on 2026-10-16, to be measured at scale. - Model calls, from the discussion of 2026-10-13: a model-backed extractor in an extra, and decision calls with a probability, for example to decide a supersede or to route a recall. The library stays deterministic in the second workstream. @@ -140,3 +145,16 @@ A closed phase keeps its row above with the date it closed and the pull request. B, and they are still the largest group. The marker of the diagnosis can miss facts, so that group can be smaller than the diagnosis shows. A reply that reaches the token limit gives no facts for its session, and a result file counts such sessions. +- Phase 8 closed on 2026-10-15 with PR #24, which closed issue #22. `capture` stores a + session as an episode that its facts cite, and `recall` adds episodes after the facts. + With `episodes="search"` at 64,000 characters, the memory arm scored 45 of 54 on sample + A and 45 on sample B, against 29 and 31 with facts only, at about 12,800 reader tokens + at the median. The retrieval arm scored 45 on sample A with 29,878 tokens. It changed + the plan for later phases in four ways. The misses with no stored fact went from 17 to + 5 on each sample, and in 8 of the 9 misses on each sample the block held an evidence + session, so the reader is now a larger loss than the extractor. The block shows a + source in its own namespace as `@`, so the run id stays out of every + prompt and a run replays `capture` from the cache. Each run captured the same memory + again for about 20 minutes, which led to phase 9. Whole sessions cost the tokens, which + led to the chunks of phase 10. An internet outage stopped four runs, and the local + engine refused one search with a stale BM25 index, which a second try did not show.