From 88ee4ecf4ed935d3c4be3030e3b0ee9240be53ec Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 15 Oct 2026 16:21:48 +0530 Subject: [PATCH 1/5] Propose episodes behind facts in the memory contract capture takes an optional episode key, and recall takes episodes: cited or search. The block shows a same-namespace source as key@revision. The plan records the owner's decisions of task 1 and the budgets of task 3. --- docs/contracts.md | 42 ++++++++++++++++++++++++++++++++++++------ docs/internal/plan.md | 23 ++++++++++++++++++++--- 2 files changed, 56 insertions(+), 9 deletions(-) diff --git a/docs/contracts.md b/docs/contracts.md index 072cffc..b1f3b1d 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -296,12 +296,12 @@ The memory contract is what an agent calls. It is a class named `Memory`, built | Operation | Arguments | Behavior | |---|---|---| | `remember` | facts, scope, actor | Writes facts that are already structured, and returns their ids in order. Each fact is a `Fact`, with `kind`, `content`, and the optional record fields. The key comes from the subject and a hash of the normalized content, so a retried call writes nothing new. The facts go through one writer. Thus one refused fact means that no fact is written. | -| `recall` | query, scopes, optional as_of, budget in characters, k | Searches the current revisions of kind `fact`, `profile`, and `procedure` in the allowed scopes for the top k records. It never returns an episode. With `as_of`, it keeps the ones that are valid at `as_of`, so a fact superseded after `as_of` is not returned. It returns the records inside the budget as a list and as one rendered block. The block labels each record with its sources and its validity span, and with nothing else. | +| `recall` | query, scopes, optional as_of, budget in characters, k, optional episodes | Searches the current revisions of kind `fact`, `profile`, and `procedure` in the allowed scopes for the top k records. With `as_of`, it keeps the ones that are valid at `as_of`, so a fact superseded after `as_of` is not returned. With `episodes`, it also returns episodes after the facts. `"cited"` gives the episodes that the facts of the block name in `sources`. `"search"` gives the top k episodes for the query. With no `episodes`, it returns no episode. It returns the records inside the budget as a list and as one rendered block. The facts take the budget first, and the episodes fill what is left. | | `candidates` | fact, scopes, k | Returns the top k hits of `search` for the content of the fact, sorted by distance, closest first. It makes no decision. A consolidator that the caller writes reads this before it calls `remember` or `supersede`. | | `supersede` | scope, key of the record to close, new fact, optional valid_from, actor | Closes the named record and writes the new fact as the next revision under its key, in one `put` with `close_previous`. The new `valid_from` is the argument, else that of the fact, else now. The `valid_until` of the old record becomes the `valid_from` of the new record, and the `expired_at` of the old record becomes now. If the key has no current revision, or the `valid_from` of the old record is later than that of the new record, the call refuses. The library decides nothing by itself. | | `forget` | scopes, and ids or a horizon | Deletes every revision of the key of each named id, or of each key whose current revision has a `forget_after` before the horizon. It finds the keys for a horizon with `list`, up to 10,000 records under each scope. A record already past its `forget_after` is hidden from `list`, so `sweep` deletes it. It checks every id before the first delete. An id that is not a valid record id, or is outside the scopes, makes the call refuse and delete nothing. It returns the deleted ids. | | `profile` | subject, scopes, budget in characters | Returns the current records of kind `fact`, `profile`, and `procedure` for the subject, grouped by `kind`. It returns them as a list and as one rendered block inside the budget. The block ends with the namespaces and record counts that `recall` can reach, so an agent knows what it can search for. This is the block that an agent always loads. | -| `capture` | text, scope, extractor, actor, observed_at | Calls the extractor of the caller with the text, `observed_at`, and the current records that `recall` returns for the text under the scope. Then it calls `remember` on the result, and returns the ids. A fact with no `observed_at` gets the one passed to `capture`. The extractor is a plain callable. The library ships no model and names no model. | +| `capture` | text, scope, extractor, actor, observed_at, optional episode | Calls the extractor of the caller with the text, `observed_at`, and the current records that `recall` returns for the text under the scope. Then it calls `remember` on the result, and returns the ids. A fact with no `observed_at` gets the one passed to `capture`. With `episode`, a key, it also stores the text as a record of kind `episode` under that key, and adds the id of the episode to the `sources` of each fact. The facts and the episode go through one writer, the facts first. If the episode is current with the same content, `capture` calls no extractor, writes nothing, and returns no id. The extractor is a plain callable. The library ships no model and names no model. | | `load` | document, text, scope, extractor, actor, observed_at, chunk_chars | Cuts a Markdown document into chunks. It stores chunk n as a record of kind `episode` under the key `-`, with n from 1 in four digits. For each chunk, it calls the extractor as `capture` does, on the chunk with its heading prefix. Each fact gets the id of the episode of its chunk added to its `sources`. The facts of a chunk are written before its episode. A chunk whose episode is current with the same content is skipped, with no call to the extractor. Thus a second load of the same text writes nothing new. It returns the ids of the episodes and the ids of the facts that the extractor returned. The document must be a valid key. | The key of a fact has two parts joined by `-`. The first part is the subject, with each @@ -317,10 +317,39 @@ The block of `recall` has one line for each record, in the order of the search: Each run of whitespace in the content becomes one space, so a record is always one line. Times are in ISO 8601 in UTC, to the second. A null `valid_from` shows as `unknown`, a -null `valid_until` as `now`, and a record with no sources as `none`. The block holds whole +null `valid_until` as `now`, and a record with no sources as `none`. A source that is the +id of a record in the same namespace shows as `@`, without the namespace. +For example, a fact under `team/alerts` with the source `team/alerts/notes-0001@1` shows +`[sources: notes-0001@1]`. Any other source shows in full. The block holds whole lines only, joined by newlines. It stops before the first line that does not fit in the budget. The list holds the records of the block. The budget counts the characters of -the block in this line format, and its default is 2,000. A consumer that renders the +the block in this line format, and its default is 2,000. + +With `episodes`, the fact lines come first, and they fit in the budget as above. Then, if +an episode fits in what is left, a line `episodes:` follows, and then each episode in this +form: + +```text +## @ [observed: ] + +``` + +The content of an episode keeps its line breaks. A null `observed_at` shows as `unknown`. +The block holds whole episodes only, and it stops before the first episode that does not +fit. Each part of the block is joined to the next by a newline, and the budget counts +every character. An episode never takes the place of a fact. + +The two values of `episodes` choose the episodes in two ways. + +- `"cited"` reads the sources of each fact of the block, in order. Each source that is + the id of a record of kind `episode` in the allowed scopes gives that revision, read + with `get`. An episode cited twice comes once, at its first place. A source that is not + such an id is skipped. +- `"search"` searches the current episodes in the allowed scopes with the query, for the + top k, in the order of the search. + +With `as_of`, only the episodes that are valid at `as_of` remain. The list holds the +facts of the block and then its episodes. A consumer that renders the records in its own format passes its own `budget` and `k`. Then the default does not cut the list first. @@ -365,8 +394,9 @@ extractor returns no fact still gets its episode. If the text of a chunk changes second load writes the next revision of its episode and calls the extractor again. If a document gets fewer chunks, the episodes of its old last chunks stay current. -An episode is the evidence that a fact names in `sources`, so `recall` and `profile` leave -it out. A consumer reads an episode with `get`, or joins the two record tables in SQL. A +An episode is the evidence that a fact names in `sources`. `profile` leaves it out, and +`recall` gives it only with `episodes`. A consumer also reads an episode with `get`, or +joins the two record tables in SQL. A namespace that holds only episodes shows a count of 0 in the block of `profile`. Two conditions make a record valid at time T. `valid_from` is null or at most T, and diff --git a/docs/internal/plan.md b/docs/internal/plan.md index 8f4596f..225297d 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -152,7 +152,7 @@ Tasks: 2. Build the agreed surface in `Memory`, for both drivers. The conformance suite, the ledger in `docs/guarantees.md`, the skill commands, and `CHANGELOG.md` follow. 3. The memory arm captures each session with an episode key, the hash of phase 7. Then it - recalls with each `episodes` value, at three budgets: 8,000, 32,000, and 64,000 + recalls with each `episodes` value, at three budgets: 16,000, 32,000, and 64,000 characters. The search inside `capture` does not change with these values, so each run after the first replays `capture` from the extractor cache (read from source, not observed). Only the reader and the judge pay for each value and budget. @@ -161,6 +161,25 @@ Tasks: 5. Report accuracy against reader tokens for each value on sample A, then the best one on sample B, with the diagnosis and the cost of each run. +Decisions of task 1, agreed by the owner on 2026-10-15. `docs/contracts.md` holds the +surface. + +- `capture` takes an optional `episode` key. The facts and the episode go through one + writer, the facts first. If the episode is current with the same content, `capture` + calls no extractor, writes nothing, and returns no id. +- `recall` takes `episodes`: none, `"cited"`, or `"search"`. Both values read with the + `Store` methods of today. `"search"` uses the same `k` as the facts. +- The block gives the facts first, then a line `episodes:`, then each episode under a + header `## @ [observed: ]`. It holds whole episodes only, + and it stops before the first episode that does not fit. +- A source that is the id of a record in the same namespace shows as `@`. + So the run id of the benchmark is in no prompt, and a run after the first replays + `capture` from the cache. The blocks of documents that `load` stored get shorter. +- The benchmark extractor stops setting `session:`. The episode key is + `session-`, with the hash of phase 7. The first run pays for the extractor again. +- The budgets are 16,000, 32,000, and 64,000 characters. At 8,000 characters, the facts + of phase 7 leave no room for a whole session (estimate, not observed). + Acceptance criteria: - `MemoryStore` and `HotdataStore` return the same block for the same calls, in the @@ -361,6 +380,4 @@ These wait, and `roadmap.md` lists them under Deferred: ## Questions for the owner -- Phase 8: agree the budgets to try, or change them from 8,000, 32,000, and 64,000 - characters. - Phase 13: the first version number, and whether to publish to TestPyPI first. From c501fc902b90274071f758d0739db7d6bfc28b00 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 15 Oct 2026 16:29:17 +0530 Subject: [PATCH 2/5] Store episodes in capture and return them from recall capture takes an episode key: remember writes the facts, each citing the episode, and then one put writes the episode. A capture of an unchanged episode calls no extractor. recall takes episodes, cited or search, and adds whole episodes after the facts inside the budget. The block shows a same-namespace source as key@revision. Conformance tests run on both drivers, and the ledger, skill commands, and changelog follow. --- CHANGELOG.md | 9 ++ README.md | 6 +- docs/contracts.md | 2 +- docs/guarantees.md | 6 +- docs/internal/architecture.md | 22 ++- docs/internal/plan.md | 8 +- skills/hotmemory/SKILL.md | 22 ++- skills/hotmemory/scripts/capture.py | 10 +- skills/hotmemory/scripts/recall.py | 7 +- src/hotmemory/contract.py | 154 ++++++++++++++++---- tests/test_contract.py | 211 ++++++++++++++++++++++++++++ tests/test_skill.py | 7 +- 12 files changed, 424 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8591baa..925bc0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,3 +56,12 @@ each one, and its failure message points to this file. - `Memory.recall` and `Memory.profile` read the records of kind `fact`, `profile`, and `procedure` only. They never return an episode, and the counts of `profile` leave episodes out. +- `Memory.capture` takes `episode`, a key. With it, `capture` also stores the text as an + episode under that key, after the facts, and each fact names the episode in `sources`. + A capture of the same text under the same key writes nothing and calls no extractor. +- `Memory.recall` takes `episodes`, `"cited"` or `"search"`. With it, the block adds the + episodes that its facts cite, or the top episodes for the query, after the facts and + inside the budget. Without it, `recall` returns no episode, as before. +- The block of `recall` and `profile` shows a source that is the id of a record in the + namespace of its line as `@`. Before, it showed the full id. The stored + sources do not change. diff --git a/README.md b/README.md index 0318ec9..55b3a83 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,10 @@ The library has two layers: fact, forget by id or by horizon, and load the profile of a subject. Extraction from raw text is optional, and it takes an extractor callable that the caller supplies. `load` cuts a Markdown document into chunks, stores each chunk as an episode, and extracts - facts that name their episode as a source. `recall` and `profile` return facts, - profiles, and procedures, never episodes. + facts that name their episode as a source. `capture` with an `episode` key stores its + text as an episode in the same way. `profile` returns facts, profiles, and procedures, + never episodes. `recall` returns them too, and adds episodes after the facts only when + the caller asks for them. The library runs in the process of the consumer and calls the Hotdata API with the API key of the consumer. There is no hotmemory server. The first consumer is an incident diff --git a/docs/contracts.md b/docs/contracts.md index b1f3b1d..4d07139 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -301,7 +301,7 @@ The memory contract is what an agent calls. It is a class named `Memory`, built | `supersede` | scope, key of the record to close, new fact, optional valid_from, actor | Closes the named record and writes the new fact as the next revision under its key, in one `put` with `close_previous`. The new `valid_from` is the argument, else that of the fact, else now. The `valid_until` of the old record becomes the `valid_from` of the new record, and the `expired_at` of the old record becomes now. If the key has no current revision, or the `valid_from` of the old record is later than that of the new record, the call refuses. The library decides nothing by itself. | | `forget` | scopes, and ids or a horizon | Deletes every revision of the key of each named id, or of each key whose current revision has a `forget_after` before the horizon. It finds the keys for a horizon with `list`, up to 10,000 records under each scope. A record already past its `forget_after` is hidden from `list`, so `sweep` deletes it. It checks every id before the first delete. An id that is not a valid record id, or is outside the scopes, makes the call refuse and delete nothing. It returns the deleted ids. | | `profile` | subject, scopes, budget in characters | Returns the current records of kind `fact`, `profile`, and `procedure` for the subject, grouped by `kind`. It returns them as a list and as one rendered block inside the budget. The block ends with the namespaces and record counts that `recall` can reach, so an agent knows what it can search for. This is the block that an agent always loads. | -| `capture` | text, scope, extractor, actor, observed_at, optional episode | Calls the extractor of the caller with the text, `observed_at`, and the current records that `recall` returns for the text under the scope. Then it calls `remember` on the result, and returns the ids. A fact with no `observed_at` gets the one passed to `capture`. With `episode`, a key, it also stores the text as a record of kind `episode` under that key, and adds the id of the episode to the `sources` of each fact. The facts and the episode go through one writer, the facts first. If the episode is current with the same content, `capture` calls no extractor, writes nothing, and returns no id. The extractor is a plain callable. The library ships no model and names no model. | +| `capture` | text, scope, extractor, actor, observed_at, optional episode | Calls the extractor of the caller with the text, `observed_at`, and the current records that `recall` returns for the text under the scope. Then it calls `remember` on the result, and returns the ids. A fact with no `observed_at` gets the one passed to `capture`. With `episode`, a key, it also stores the text as a record of kind `episode` under that key, and adds the id of the episode to the `sources` of each fact. `remember` writes the facts in one flush, and then one `put` writes the episode, as in `load`. If the episode is not written, a retry calls the extractor again. If the episode is current with the same content, `capture` calls no extractor, writes nothing, and returns no id. With `episode`, a blank text, a key that is not valid, or a key that holds another kind makes the call refuse before the extractor runs. The extractor is a plain callable. The library ships no model and names no model. | | `load` | document, text, scope, extractor, actor, observed_at, chunk_chars | Cuts a Markdown document into chunks. It stores chunk n as a record of kind `episode` under the key `-`, with n from 1 in four digits. For each chunk, it calls the extractor as `capture` does, on the chunk with its heading prefix. Each fact gets the id of the episode of its chunk added to its `sources`. The facts of a chunk are written before its episode. A chunk whose episode is current with the same content is skipped, with no call to the extractor. Thus a second load of the same text writes nothing new. It returns the ids of the episodes and the ids of the facts that the extractor returned. The document must be a valid key. | The key of a fact has two parts joined by `-`. The first part is the subject, with each diff --git a/docs/guarantees.md b/docs/guarantees.md index 70c3ba9..ef6e1b4 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -56,7 +56,11 @@ A row that no test proves yet names the phase that will prove it. | Does `HotdataStore` agree with `MemoryStore`? | Ranked by content vector alone, it returns the same records in the same order, with distances equal within 1e-6. With the fused ranking, it returns the same set when k covers every record. On the test fixture, the largest distance difference was 4.0e-8. | answered [measured, local] 2026-10-08 | `test_vector_ranking_matches_the_oracle`, `test_fused_ranking_returns_the_oracle_set` | | What does `load` store for a document? | One episode for each chunk, under the key `-`, with the first heading and the heading path at the top. Each fact that the extractor returns names the episode of its chunk in `sources`. A fact with no `valid_from` is valid from the `observed_at` of the load. A heading inside a fenced code block does not start a chunk, and no chunk body is longer than `chunk_chars`, unless one line is. | answered | `test_load_stores_each_chunk_as_an_episode_and_links_its_facts`, `test_a_loaded_fact_is_valid_from_the_observed_time`, `test_a_heading_inside_a_fence_is_not_a_heading`, `test_every_body_fits_the_size`, `test_a_cut_table_repeats_its_header` | | Does a second `load` of the same text write anything? | No. A chunk whose episode is current with the same content is skipped, and the extractor is not called for it, so this holds for an extractor that calls a model. A changed chunk writes the next revision of its episode. | answered | `test_a_second_load_of_the_same_text_writes_nothing_new`, `test_load_of_an_edited_chunk_writes_its_next_revision` | -| Do `recall` and `profile` return episodes? | No. They read the current records of kind `fact`, `profile`, and `procedure` only. A fact names its episodes in `sources`, so a consumer reads an episode with `get`. The counts of `profile` leave episodes out too. | answered | `test_recall_and_profile_leave_episodes_out`, `test_recall_block_matches_its_pinned_string`, `test_profile_block_matches_its_pinned_string` | +| Do `recall` and `profile` return episodes? | `profile` never does, and its counts leave episodes out. `recall` returns episodes only with `episodes`. Without it, `recall` reads the current records of kind `fact`, `profile`, and `procedure` only. | answered | `test_recall_and_profile_leave_episodes_out`, `test_recall_block_matches_its_pinned_string`, `test_profile_block_matches_its_pinned_string` | +| What does `capture` store with an episode key? | The text, as an episode under that key, with the `actor` and `observed_at` of the call. Each fact names the id of the episode in `sources`. `remember` writes the facts in one flush, and one `put` then writes the episode, so each table gets one flush. If the extractor raises, nothing is written. | answered | `test_capture_with_an_episode_stores_it_and_links_its_facts`, `test_capture_writes_one_flush_per_table`, `test_capture_with_a_failing_extractor_writes_no_episode`, `test_capture_refuses_a_bad_episode`, `test_capture_refuses_an_episode_key_that_holds_a_fact` | +| Does a retried `capture` with an episode key write anything? | No. If the episode is current with the same content, the extractor is not called and nothing is written, so this holds for an extractor that calls a model. | answered | `test_a_retried_capture_with_an_episode_writes_nothing_new` | +| Which episodes does `recall` add with `episodes`? | With `"cited"`, the episodes that the facts of the block name in `sources`, in order, once each, and only inside the allowed scopes. With `"search"`, the top k episodes for the query. `as_of` filters them as it filters facts. They follow the facts, whole, and the block stops before the first episode that does not fit. An episode never takes the place of a fact. | answered | `test_recall_cited_adds_the_episodes_of_the_facts`, `test_recall_cited_names_each_episode_once`, `test_recall_cited_skips_an_episode_outside_the_scopes`, `test_recall_search_adds_the_top_episodes`, `test_recall_episodes_never_take_the_place_of_a_fact`, `test_recall_stops_before_the_first_episode_that_does_not_fit`, `test_recall_as_of_filters_the_episodes`, `test_recall_refuses_an_unknown_episodes_value` | +| How does the block show a source? | In full, except the id of a record in the namespace of the line, which shows as `@`. The stored source keeps the full id. | answered | `test_recall_shows_a_source_in_its_namespace_as_key_and_revision` | | Can a `put` move a key between `episode` and another kind? | No. A key holds episodes only, or holds no episode at all. A `put` that crosses that line raises an error and writes nothing. A key can change between `fact`, `profile`, and `procedure`. | answered | `test_key_cannot_cross_the_episode_line` | | What does a `put` with `close_previous` write? | The new revision, and the previous row with `superseded_by`, `valid_until` set to the new `valid_from`, and `expired_at` set to the clock's time. A missing `valid_from`, or one earlier than the previous `valid_from`, raises an error, and the put or the whole flush writes nothing. Without `close_previous`, the previous span stays open. | answered | `test_close_previous_closes_the_current_revision`, `test_close_previous_refuses_a_span_it_cannot_close`, `test_close_previous_on_a_new_key_or_same_content`, `test_put_without_close_previous_leaves_the_span_open` | | Can a record have empty content? | No. `put` and a buffered `put` refuse content that is empty after normalization, and write nothing. | answered | `test_put_refuses_empty_content` | diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 8842598..848a01a 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -357,6 +357,14 @@ retried call write nothing new. The extractor sees the records that `recall` returns for the text, before the capture writes anything. A write is never visible inside the same turn. +With an `episode` key, `capture` first reads the current revision of that key with `get`. +If it holds the same content, the call ends there, with no extractor call. Else the +diagram above runs with the id of the next episode revision added to the `sources` of +each fact, and then one `put` writes the text to `episode_v1`. The two writes are two +flushes, one for each table, in that order. A flush of `HotdataStore` loads its tables at +the same time, so with one writer for both, the episode can land without the facts. A retry +would then skip the extractor. + ## Recall and search `recall` calls `search` with the filter `kind` in (`fact`, `profile`, `procedure`). That @@ -373,9 +381,21 @@ flowchart TD asof -- "no" --> render keep --> render["Render one line for each record"] render --> fit["Keep whole lines up to the budget"] - fit --> out["Return the records and the block"] + fit --> eps{"episodes given?"} + eps -- "no" --> out["Return the records and the block"] + eps -- "cited" --> get["get each episode that the facts
of the block name in sources"] + eps -- "search" --> find["search the episodes,
kind episode, top k"] + get --> whole["Add whole episodes while they fit
in what the facts left"] + find --> whole + whole --> out ``` +With `episodes="cited"`, `recall` sends one `get` for each distinct source that is an +episode id in the allowed scopes. With `episodes="search"`, it sends a second search with +the filter `kind` = `episode`, so that search reads `episode_v1`. The episodes that +`capture` and `load` write have no cues, so the fused ranking of that search is BM25 and +the content vector. + The `as_of` filter runs in the process, after the search. Thus `recall` with `as_of` can return fewer than k records. The search keeps k records before the filter, so a record that is valid at `as_of` but ranks below k is missing. diff --git a/docs/internal/plan.md b/docs/internal/plan.md index 225297d..d88924a 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -164,9 +164,11 @@ Tasks: Decisions of task 1, agreed by the owner on 2026-10-15. `docs/contracts.md` holds the surface. -- `capture` takes an optional `episode` key. The facts and the episode go through one - writer, the facts first. If the episode is current with the same content, `capture` - calls no extractor, writes nothing, and returns no id. +- `capture` takes an optional `episode` key. `remember` writes the facts, and then one + `put` writes the episode, as in `load`. The proposal said one writer, but a flush of + `HotdataStore` loads its tables at the same time, so the order of the two writes was + not fixed (read from source, not observed). If the episode is current with the same + content, `capture` calls no extractor, writes nothing, and returns no id. - `recall` takes `episodes`: none, `"cited"`, or `"search"`. Both values read with the `Store` methods of today. `"search"` uses the same `k` as the facts. - The block gives the facts first, then a line `episodes:`, then each episode under a diff --git a/skills/hotmemory/SKILL.md b/skills/hotmemory/SKILL.md index 98ee109..16c127f 100644 --- a/skills/hotmemory/SKILL.md +++ b/skills/hotmemory/SKILL.md @@ -42,7 +42,10 @@ python scripts/remember.py --memory-file memory.json --scope team/alerts --subje Print the records that match the query, one line each, inside the budget in characters. Each line holds the content, the sources, and the validity span. Add `--as-of` to keep -only the records that were valid at that time. +only the records that were valid at that time. Add `--episodes cited` to print, after the +facts, the episodes that they name in their sources. Add `--episodes search` to print +the episodes that match the query. Each episode is printed whole, under a line that +gives its key and its date, while it fits in the budget. ```sh python scripts/recall.py --memory-file memory.json --scope team --query "When does the disk fill?" \ @@ -89,13 +92,26 @@ python scripts/capture.py --memory-file memory.json --scope team/alerts --actor --extractor extractors:one_fact_per_sentence --text "The CPU spikes at noon. The disk fills at noon." ``` +Add `--episode` with a key to also store the text as an episode under that key. Each fact +then names the episode in its sources. If you capture the same text under the same key +again, the script writes nothing and calls no extractor. + +```sh +python scripts/capture.py --memory-file memory.json --scope team/alerts --actor sentence-splitter \ + --extractor extractors:one_fact_per_sentence --episode chat-0001 \ + --text "The pool ran out of nodes. The node group was at its ceiling." +python scripts/recall.py --memory-file memory.json --scope team --query "Why did the pool run out?" \ + --budget 1000 --episodes cited +``` + ## Load a document Store a Markdown document as episodes, one for each chunk, and run an extractor on each chunk. A chunk starts at a heading, never inside a code block, and starts with the first heading of the document and its heading path. Each fact names the episode of its chunk in -its sources. `recall` never returns an episode. The script prints the episode ids and then -the fact ids. If you load the same file again, it writes nothing and calls no extractor. +its sources. `recall` prints an episode only with `--episodes`. The script prints the +episode ids and then the fact ids. If you load the same file again, it writes nothing and +calls no extractor. ```sh python scripts/load.py --memory-file memory.json --scope team/alerts --document disk-incident \ diff --git a/skills/hotmemory/scripts/capture.py b/skills/hotmemory/scripts/capture.py index 05d45c5..8066990 100644 --- a/skills/hotmemory/scripts/capture.py +++ b/skills/hotmemory/scripts/capture.py @@ -1,4 +1,7 @@ -"""Run an extractor over a text, remember the facts it returns, and print their ids.""" +"""Run an extractor over a text, remember the facts it returns, and print their ids. + +With --episode, the text is also stored as an episode, and each fact names it in its sources. +""" import importlib @@ -12,9 +15,12 @@ ) args_parser.add_argument("--actor", default="") args_parser.add_argument("--observed-at", type=parse_time) +args_parser.add_argument("--episode", help="a key: also store the text as an episode under it") args = args_parser.parse_args() module, _, name = args.extractor.partition(":") extractor = getattr(importlib.import_module(module), name) with memory(args) as agent_memory: - ids = agent_memory.capture(args.text, args.scope, extractor, args.actor, args.observed_at) + ids = agent_memory.capture( + args.text, args.scope, extractor, args.actor, args.observed_at, args.episode + ) print(*ids, sep="\n") diff --git a/skills/hotmemory/scripts/recall.py b/skills/hotmemory/scripts/recall.py index 8831f23..d98d83f 100644 --- a/skills/hotmemory/scripts/recall.py +++ b/skills/hotmemory/scripts/recall.py @@ -8,7 +8,12 @@ args_parser.add_argument("--as-of", type=parse_time) args_parser.add_argument("--budget", type=int, default=2000) args_parser.add_argument("--k", type=int, default=10) +args_parser.add_argument( + "--episodes", choices=["cited", "search"], help="also print episodes after the facts" +) args = args_parser.parse_args() with memory(args) as agent_memory: - _, block = agent_memory.recall(args.query, args.scopes, args.as_of, args.budget, args.k) + _, block = agent_memory.recall( + args.query, args.scopes, args.as_of, args.budget, args.k, args.episodes + ) print(block) diff --git a/src/hotmemory/contract.py b/src/hotmemory/contract.py index 90fde11..7468284 100644 --- a/src/hotmemory/contract.py +++ b/src/hotmemory/contract.py @@ -7,7 +7,7 @@ from collections.abc import Callable, Sequence from dataclasses import dataclass, field, replace from datetime import UTC, datetime, timedelta -from typing import TypeAlias +from typing import Literal, TypeAlias from hotmemory._chunks import chunk_document from hotmemory._rules import check_prefix, is_duplicate, newest_first, next_revision, under_prefix @@ -35,6 +35,9 @@ COUNT_LIMIT = 1000 _NOT_IN_KEY = re.compile(r"[^A-Za-z0-9_-]") RECALLED_KINDS: tuple[Kind, ...] = ("fact", "profile", "procedure") +EPISODES_LINE = "episodes:" +Episodes: TypeAlias = Literal["cited", "search"] +"""Which episodes `recall` adds after the facts: those the facts cite, or a search over them.""" @dataclass(frozen=True, kw_only=True) @@ -77,15 +80,27 @@ def render_line(record: Record) -> str: The line is `- [sources: a, b] [valid: to ]`. Each run of whitespace in the content becomes one space. A null `valid_from` shows as `unknown`, - a null `valid_until` as `now`, and no sources as `none`. + a null `valid_until` as `now`, and no sources as `none`. A source that is the id of a + record in the namespace of `record` shows as `@`. """ content = " ".join(record.content.split()) - sources = ", ".join(record.sources) or "none" + sources = ", ".join(_shown_source(source, record.namespace) for source in record.sources) + sources = sources or "none" start = _time(record.valid_from, "unknown") end = _time(record.valid_until, "now") return f"- {content} [sources: {sources}] [valid: {start} to {end}]" +def render_episode(record: Record) -> str: + """Return the part of a rendered block for the episode `record`. + + The part is a header `## @ [observed: ]`, a newline, and + the content with its line breaks. A null `observed_at` shows as `unknown`. + """ + observed = _time(record.observed_at, "unknown") + return f"## {record.key}@{record.revision} [observed: {observed}]\n{record.content}" + + def fit_lines(lines: Sequence[str], budget: int) -> int: """Return how many of `lines`, from the first, fit in `budget` characters with newlines. @@ -164,22 +179,71 @@ def recall( as_of: datetime | None = None, budget: int = DEFAULT_BUDGET, k: int = 10, + episodes: Episodes | None = None, ) -> tuple[list[Record], str]: """Return the top current records for `query` under `scopes`, and their block. - The search returns up to `k` current records of the kinds in RECALLED_KINDS, so - it never returns an episode. With `as_of`, only the ones valid at `as_of` remain. - The block holds one line for each record, in order, and stops before the first - line that would pass `budget` characters. The list holds the records of the block. + The search returns up to `k` current records of the kinds in RECALLED_KINDS. With + `as_of`, only the ones valid at `as_of` remain. The block holds one line for each + record, in order, and stops before the first line that would pass `budget` + characters. With `episodes`, the block then holds the episodes that fit in what is + left, after a line `episodes:`: those that the facts of the block cite with + `"cited"`, or the top `k` episodes for `query` with `"search"`. The list holds the + records of the block, the facts first. """ check_count("budget", budget) + if episodes not in (None, "cited", "search"): + raise ValueError(f"episodes must be None, 'cited', or 'search', got {episodes!r}") recalled = Filter(kind=RECALLED_KINDS) records = [hit.record for hit in self._store.search(query, scopes, recalled, k=k)] if as_of is not None: records = [record for record in records if is_valid_at(record, as_of)] lines = [render_line(record) for record in records] count = fit_lines(lines, budget) - return records[:count], "\n".join(lines[:count]) + records, lines = records[:count], lines[:count] + if episodes is None: + return records, "\n".join(lines) + if episodes == "cited": + found = self._cited(records, _scopes(scopes)) + else: + kind = Filter(kind="episode") + found = [hit.record for hit in self._store.search(query, scopes, kind, k=k)] + if as_of is not None: + found = [episode for episode in found if is_valid_at(episode, as_of)] + parts: list[str] = [] + shown: list[Record] = [] + for episode in found: + more = [*parts, render_episode(episode)] + if len("\n".join([*lines, EPISODES_LINE, *more])) > budget: + break + parts, shown = more, [*shown, episode] + if not shown: + return records, "\n".join(lines) + return [*records, *shown], "\n".join([*lines, EPISODES_LINE, *parts]) + + def _cited(self, records: Sequence[Record], allowed: Sequence[tuple[str, ...]]) -> list[Record]: + """Return the episodes that the sources of `records` name, in order, without repeats. + + A source that is not a record id, is outside `allowed`, or names no episode is + skipped. + """ + cited: dict[str, Record | None] = {} + for record in records: + for source in record.sources: + if source in cited: + continue + cited[source] = None + try: + namespace, key = _slot(source) + except ValueError: + continue + if not any(under_prefix(namespace, scope) for scope in allowed): + continue + revision = int(source.rpartition("@")[2]) + found = self._store.get(namespace, key, revision) + if found is not None and found.kind == "episode": + cited[source] = found + return [episode for episode in cited.values() if episode is not None] def candidates(self, fact: Fact, scopes: Sequence[Sequence[str]], k: int = 5) -> list[Hit]: """Return up to `k` current records near the content of `fact`, closest first. @@ -326,14 +390,26 @@ def capture( extractor: Extractor, actor: str = "", observed_at: datetime | None = None, + episode: str | None = None, ) -> list[str]: """Call `extractor` on `text` and remember the facts that it returns. The extractor gets `text`, `observed_at`, and the records that `recall` returns for - `text` under `scope`. A fact with no `observed_at` gets `observed_at`. Returns the - ids that `remember` returns. + `text` under `scope`. A fact with no `observed_at` gets `observed_at`. With + `episode`, a key, `text` is also stored as an episode under that key, written after + the facts, and each fact gets the id of the episode added to its `sources`. If that + episode is current with the same content, the extractor is not called and nothing + is written. Raises ValueError if `episode` is not a valid key, holds another kind, + or `text` is blank. Returns the ids that `remember` returns. """ - return self._extract(text, scope, extractor, actor, observed_at, ()) + if episode is None: + return self._extract(text, scope, extractor, actor, observed_at, ()) + check_key(episode) + if not text.strip(): + raise ValueError("text must not be blank when capture stores an episode") + namespace = check_namespace(scope) + _, facts = self._episode(namespace, episode, text, extractor, actor, observed_at) + return facts def load( self, @@ -366,22 +442,40 @@ def load( for number, chunk in enumerate(chunk_document(text, chunk_chars), start=1): key = f"{document}-{number:04d}" check_key(key) - current = self._store.get(namespace, key) - if current is not None and current.kind != "episode": - raise ValueError(f"key {key!r} holds kind {current.kind!r}, not an episode") - if current is not None and is_duplicate(current, chunk, self._clock()): - episodes.append(current.id) - continue - episode = record_id(namespace, key, next_revision(current)) - facts.extend(self._extract(chunk, namespace, extractor, actor, observed_at, (episode,))) - written = self._store.put( - namespace, key, kind="episode", content=chunk, actor=actor, observed_at=observed_at - ) - if written != episode: - raise RuntimeError(f"episode {key!r} was written as {written!r}, not {episode!r}") - episodes.append(written) + episode, written = self._episode(namespace, key, chunk, extractor, actor, observed_at) + episodes.append(episode) + facts.extend(written) return episodes, facts + def _episode( + self, + namespace: tuple[str, ...], + key: str, + text: str, + extractor: Extractor, + actor: str, + observed_at: datetime | None, + ) -> tuple[str, list[str]]: + """Extract the facts of `text`, then store `text` as the episode under `key`. + + Each fact gets the id of the episode in its `sources`. If the episode is current + with the same content, nothing is called or written. Returns the id of the episode + and the ids of the facts. + """ + current = self._store.get(namespace, key) + if current is not None and current.kind != "episode": + raise ValueError(f"key {key!r} holds kind {current.kind!r}, not an episode") + if current is not None and is_duplicate(current, text, self._clock()): + return current.id, [] + episode = record_id(namespace, key, next_revision(current)) + facts = self._extract(text, namespace, extractor, actor, observed_at, (episode,)) + written = self._store.put( + namespace, key, kind="episode", content=text, actor=actor, observed_at=observed_at + ) + if written != episode: + raise RuntimeError(f"episode {key!r} was written as {written!r}, not {episode!r}") + return written, facts + def _extract( self, text: str, @@ -426,6 +520,16 @@ def _slot(record_id: str) -> tuple[tuple[str, ...], str]: return namespace, key +def _shown_source(source: str, namespace: tuple[str, ...]) -> str: + """Return `source` as the block shows it: `@` for a record in `namespace`.""" + prefix = "/".join(namespace) + "/" + rest = source.removeprefix(prefix) + key, separator, revision = rest.rpartition("@") + if rest == source or not separator or not key or "/" in key or not revision.isdigit(): + return source + return rest + + def _time(value: datetime | None, missing: str) -> str: """Return `value` in ISO 8601 in UTC to the second, or `missing` if it is None.""" if value is None: diff --git a/tests/test_contract.py b/tests/test_contract.py index a7dc042..483a01b 100644 --- a/tests/test_contract.py +++ b/tests/test_contract.py @@ -1,6 +1,7 @@ """Tests of the memory contract. Each test runs `Memory` over every driver.""" from datetime import UTC, datetime, timedelta +from typing import Any, cast import pytest @@ -587,3 +588,213 @@ def test_a_loaded_fact_is_valid_from_the_observed_time(memory: Memory) -> None: [], "", ) + + +CHAT = "user: The disk fills at night.\n\nassistant: Rotate the logs before noon." +LATER = "user: The pool ran out of nodes.\n\nassistant: Raise the node group ceiling." + + +def first_turn(text: str, observed_at: object, current: list[Record]) -> list[Fact]: + """Return one fact with the first turn of a chat, without its role.""" + turn = text.split("\n\n")[0].removeprefix("user: ") + return [Fact(kind="fact", subject="chat", content=turn, sources=("chat",))] + + +class CountingStore: + """A store that counts the calls of `put` and `writer`, and passes each call on.""" + + def __init__(self, store: Store) -> None: + self.inner = store + self.calls: list[str] = [] + + def __getattr__(self, name: str) -> object: + if name in ("put", "writer"): + self.calls.append(name) + return getattr(self.inner, name) + + +def test_capture_with_an_episode_stores_it_and_links_its_facts(memory: Memory) -> None: + observed = START - timedelta(days=2) + ids = memory.capture( + CHAT, SCOPE, first_turn, actor="extractor", observed_at=observed, episode="chat-0001" + ) + + episode = memory.store.get(SCOPE, "chat-0001") + assert episode is not None + assert (episode.kind, episode.content, episode.actor, episode.observed_at) == ( + "episode", + CHAT, + "extractor", + observed, + ) + [fact] = [memory.store.get(SCOPE, record_id.split("/")[-1].split("@")[0]) for record_id in ids] + assert fact is not None + assert (fact.content, fact.sources) == ("The disk fills at night.", ("chat", episode.id)) + + +def test_capture_writes_one_flush_per_table(store: Store, clock: FakeClock) -> None: + counting = CountingStore(store) + memory = Memory(cast(Store, counting), clock=clock) + memory.capture(CHAT, SCOPE, first_turn, episode="chat-0001") + assert counting.calls == ["writer", "put"] + + +def test_a_retried_capture_with_an_episode_writes_nothing_new( + memory: Memory, clock: FakeClock +) -> None: + calls: list[str] = [] + + def extractor(text: str, observed_at: object, current: list[Record]) -> list[Fact]: + calls.append(text) + return first_turn(text, observed_at, current) + + memory.capture(CHAT, SCOPE, extractor, episode="chat-0001") + before = memory.store.list(SCOPE, limit=1000) + clock.advance() + assert memory.capture(CHAT, SCOPE, extractor, episode="chat-0001") == [] + + assert calls == [CHAT] + assert memory.store.list(SCOPE, limit=1000) == before + assert all(len(memory.store.history(SCOPE, r.key)) == 1 for r in before) + + +def test_capture_with_a_failing_extractor_writes_no_episode(memory: Memory) -> None: + def extractor(text: str, observed_at: object, current: list[Record]) -> list[Fact]: + raise RuntimeError("the model is down") + + with pytest.raises(RuntimeError, match="down"): + memory.capture(CHAT, SCOPE, extractor, episode="chat-0001") + assert memory.store.list(SCOPE) == [] + memory.capture(CHAT, SCOPE, first_turn, episode="chat-0001") + assert memory.store.get(SCOPE, "chat-0001") is not None + + +@pytest.mark.parametrize( + ("episode", "text"), [("a/b", CHAT), ("a@b", CHAT), ("", CHAT), ("chat", " \n")] +) +def test_capture_refuses_a_bad_episode(memory: Memory, episode: str, text: str) -> None: + with pytest.raises(ValueError): + memory.capture(text, SCOPE, first_turn, episode=episode) + assert memory.store.list(SCOPE) == [] + + +def test_capture_refuses_an_episode_key_that_holds_a_fact(memory: Memory) -> None: + memory.store.put(SCOPE, "chat-0001", kind="fact", content="Not an episode.") + with pytest.raises(ValueError, match="episode"): + memory.capture(CHAT, SCOPE, first_turn, episode="chat-0001") + assert [record.key for record in memory.store.list(SCOPE)] == ["chat-0001"] + + +def test_recall_shows_a_source_in_its_namespace_as_key_and_revision(memory: Memory) -> None: + sources = ("team/alerts/chat-0001@2", "team/alerts/night/chat-0002@1", "team/x@1", "wiki") + memory.remember([Fact(kind="fact", content="The disk fills.", sources=sources)], SCOPE) + _, block = memory.recall("disk", [SCOPE]) + assert block == ( + "- The disk fills. [sources: chat-0001@2, team/alerts/night/chat-0002@1, team/x@1, wiki]" + " [valid: unknown to now]" + ) + + +def capture_two_chats(memory: Memory, clock: FakeClock) -> None: + memory.capture( + CHAT, SCOPE, first_turn, observed_at=datetime(2026, 9, 1, tzinfo=UTC), episode="chat-0001" + ) + clock.advance() + memory.capture( + LATER, SCOPE, first_turn, observed_at=datetime(2026, 9, 2, tzinfo=UTC), episode="chat-0002" + ) + + +PINNED_CITED = ( + "- The disk fills at night. [sources: chat, chat-0001@1] " + "[valid: 2026-09-01T00:00:00Z to now]\n" + "episodes:\n" + "## chat-0001@1 [observed: 2026-09-01T00:00:00Z]\n" + "user: The disk fills at night.\n\nassistant: Rotate the logs before noon." +) + + +def test_recall_cited_adds_the_episodes_of_the_facts(memory: Memory, clock: FakeClock) -> None: + capture_two_chats(memory, clock) + records, block = memory.recall("disk at night", [SCOPE], k=1, episodes="cited") + assert block == PINNED_CITED + assert [(record.kind, record.key) for record in records] == [ + ("fact", records[0].key), + ("episode", "chat-0001"), + ] + + +def test_recall_cited_names_each_episode_once(memory: Memory, clock: FakeClock) -> None: + capture_two_chats(memory, clock) + clock.advance() + memory.capture(LATER, SCOPE, first_turn, episode="chat-0003") + records, block = memory.recall("pool nodes", [SCOPE], k=1, budget=10_000, episodes="cited") + assert [record.key for record in records if record.kind == "episode"] == [ + "chat-0002", + "chat-0003", + ] + assert "[sources: chat, chat-0002@1, chat-0003@1]" in block + + +def test_recall_cited_skips_an_episode_outside_the_scopes(memory: Memory) -> None: + other = ("team", "billing") + memory.store.put(other, "chat-0009", kind="episode", content="The disk bill is due.") + sources = ("team/billing/chat-0009@1",) + memory.remember([Fact(kind="fact", content="The disk fills.", sources=sources)], SCOPE) + records, block = memory.recall("disk", [SCOPE], budget=10_000, episodes="cited") + assert [record.kind for record in records] == ["fact"] + assert "episodes:" not in block + + +def test_recall_search_adds_the_top_episodes(memory: Memory, clock: FakeClock) -> None: + capture_two_chats(memory, clock) + records, block = memory.recall("pool nodes ceiling", [SCOPE], k=1, episodes="search") + assert [record.kind for record in records] == ["fact", "episode"] + assert records[1].key == "chat-0002" + assert block.split("\n")[1:3] == [ + "episodes:", + "## chat-0002@1 [observed: 2026-09-02T00:00:00Z]", + ] + + +def test_recall_episodes_never_take_the_place_of_a_fact(memory: Memory, clock: FakeClock) -> None: + capture_two_chats(memory, clock) + facts, fact_block = memory.recall("disk at night", [SCOPE], k=1) + for budget in (len(fact_block), len(PINNED_CITED) - 1): + records, block = memory.recall( + "disk at night", [SCOPE], k=1, budget=budget, episodes="cited" + ) + assert (records, block) == (facts, fact_block) + records, block = memory.recall( + "disk at night", [SCOPE], k=1, budget=len(PINNED_CITED), episodes="cited" + ) + assert block == PINNED_CITED + + +def test_recall_stops_before_the_first_episode_that_does_not_fit( + memory: Memory, clock: FakeClock +) -> None: + capture_two_chats(memory, clock) + _, both = memory.recall("disk pool", [SCOPE], k=2, budget=10_000, episodes="search") + cut = len(both) - 1 + records, block = memory.recall("disk pool", [SCOPE], k=2, budget=cut, episodes="search") + assert [record.kind for record in records] == ["fact", "fact", "episode"] + assert both.startswith(block + "\n## ") + assert len(block) <= cut + + +def test_recall_as_of_filters_the_episodes(memory: Memory, clock: FakeClock) -> None: + capture_two_chats(memory, clock) + records, _ = memory.recall( + "pool nodes ceiling", + [SCOPE], + as_of=datetime(2026, 9, 1, 12, tzinfo=UTC), + budget=10_000, + episodes="search", + ) + assert [record.key for record in records if record.kind == "episode"] == ["chat-0001"] + + +def test_recall_refuses_an_unknown_episodes_value(memory: Memory) -> None: + with pytest.raises(ValueError, match="episodes"): + memory.recall("disk", [SCOPE], episodes=cast(Any, "all")) diff --git a/tests/test_skill.py b/tests/test_skill.py index 6c6f85e..847449b 100644 --- a/tests/test_skill.py +++ b/tests/test_skill.py @@ -35,7 +35,8 @@ def test_every_command_in_the_skill_file_exits_zero(tmp_path: Path) -> None: shutil.copytree(SKILL, folder, ignore=shutil.ignore_patterns("__pycache__")) path = os.pathsep.join([str(Path(sys.executable).parent), os.environ.get("PATH", "")]) found = commands() - assert len(found) == 8 + assert len(found) == 10 + printed = {} for command in found: result = subprocess.run( command, @@ -47,9 +48,13 @@ def test_every_command_in_the_skill_file_exits_zero(tmp_path: Path) -> None: check=False, ) assert result.returncode == 0, f"{command}\n{result.stderr}" + printed[command] = result.stdout records = (folder / "memory.json").read_text(encoding="utf-8") assert "The disk fills at noon." in records assert "The CPU spikes at noon." not in records assert '"key": "disk-incident-0001"' in records assert "The noon batch doubled the log volume." in records assert records.count("team/alerts/disk-incident-0001@1") > 1 + assert '"key": "chat-0001"' in records + [cited] = [out for command, out in printed.items() if "--episodes cited" in command] + assert "\nepisodes:\n## chat-0001@1 [observed: " in cited From 320e8918f69f08dbd85930e937648cb3c8731df8 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 15 Oct 2026 16:39:59 +0530 Subject: [PATCH 3/5] Capture sessions as episodes in the memory arm, and diagnose them The runner gains --session-episodes, which captures each session under the key session- with no session: source on the facts, and --episodes, the recall value. Each result row lists the episode keys of its block. The diagnosis replays capture the same way and records, for each miss, whether the block held an evidence session as an episode. --- bench/README.md | 15 +++++- bench/diagnose.py | 68 ++++++++++++++++++------- bench/e2e.py | 99 +++++++++++++++++++++++++++++------- bench/tests/test_diagnose.py | 55 +++++++++++++++++++- bench/tests/test_e2e.py | 59 ++++++++++++++++++++- 5 files changed, 254 insertions(+), 42 deletions(-) diff --git a/bench/README.md b/bench/README.md index 38c218c..da92ae4 100644 --- a/bench/README.md +++ b/bench/README.md @@ -87,6 +87,8 @@ The options are these: | `--recall-budget` | the `budget` of the memory recall, in characters | `8000` | | `--extractor` | a variant name in `VARIANTS` of `bench/prompts.py` | `baseline` | | `--no-fact-sources` | write the facts with no session source, as phases 5 and 6 did | off | +| `--session-episodes` | capture each session as an episode, with the facts citing it | off | +| `--episodes` | `none`, `cited`, or `search`, the `episodes` of the memory recall; a value other than `none` needs `--session-episodes` | `none` | | `--replay` | call no model, and stop at the first reply that the cache does not hold | off | | `--workers` | the number of processes that run the questions | `1` | | `--tag` | a suffix for the name of the result file | none | @@ -101,7 +103,14 @@ the session id. The block shows the sources to the reader, and the id of an evid session holds `answer`, so the source does not show the id. If an extractor reply reaches its limit of 4,000 output tokens, the session writes no facts. The cache keeps that reply, so a replay skips the same session. The result file counts these sessions in -`sessions_at_limit`. The result file is `bench/results/e2e-sample-[-].json`, +`sessions_at_limit`. + +With `--session-episodes`, `capture` stores each session as an episode under the key +`session-`, with the same hash. Each fact then cites that episode, and carries no +`session:` source. The block shows the source as `session-@1`. The source +holds no run id, so a second run with another `--episodes` or `--recall-budget` replays +`capture` from the cache. The result file lists the episode keys of each block in +`episodes_in_block`. The result file is `bench/results/e2e-sample-[-].json`, and it records each option. ## The checks of the results @@ -136,4 +145,6 @@ For the last cause, GPT-4o tells whether the marked facts in the block hold ever that the answer needs ("all present") or only some of them ("some present"). The command prints the groups for each question type, and writes them next to the result file, as `-diagnosis.json`. For each fact that holds the answer, the file gives its session -id, if the fact has a source. \ No newline at end of file +id, if the fact has a source. If the run recalled episodes, each miss also gets +`evidence_in_block`: whether the block held an evidence session as an episode. The table +then counts these misses in a last column. \ No newline at end of file diff --git a/bench/diagnose.py b/bench/diagnose.py index 0e688a7..9dd0f9c 100644 --- a/bench/diagnose.py +++ b/bench/diagnose.py @@ -6,7 +6,8 @@ and extractor variant that the result file records. A model marks the facts that hold the answer, and the search ranks every fact for the question. If a marked fact is in the block, a second model call tells whether the block holds every needed fact ("all present") or only -some of them ("some present"). +some of them ("some present"). If the run recalled episodes, each miss also records whether +the block held an evidence session as an episode, from the episode keys of the result file. uv run --group bench --env-file .env python -m bench.diagnose --result """ @@ -22,15 +23,14 @@ from pathlib import Path from typing import Any -from bench import prompts -from bench.data import DATA_DIR, DATA_FILE, QUESTION_TYPES, load +from bench.data import DATA_DIR, DATA_FILE, QUESTION_TYPES, Question, load from bench.e2e import ( - ACTOR, BASELINE, RECALL_BUDGET, RECALL_K, Config, - extractor, + capture_session, + episode_key, fact_source, local_store, ) @@ -113,7 +113,8 @@ def memory_config(result: dict[str, Any]) -> Config: """Return the configuration of the memory arm that a result file records. A result file of phase 5 records no `memory` section, and its run used `BASELINE`. A - result file of phase 6 records no `fact_sources`, and its facts carried no source. + result file of phase 6 records no `fact_sources`, and its facts carried no source. A + result file before phase 8 records no `session_episodes` and no `episodes`. """ found = result["config"].get("memory") if found is None: @@ -124,18 +125,40 @@ def memory_config(result: dict[str, Any]) -> Config: budget=found["budget"], extractor=found["extractor"], fact_sources=found.get("fact_sources", False), + session_episodes=found.get("session_episodes", False), + episodes=found.get("episodes", "none"), ) +def evidence_in_block(question: Question, shown: Sequence[str]) -> bool: + """Return True if an episode key in `shown` is that of an evidence session of `question`.""" + return any(episode_key(session_id) in shown for session_id in question.answer_session_ids) + + def table(rows: Sequence[dict[str, Any]]) -> str: - """Return the groups of misses for each question type, and in total, as Markdown.""" - columns = ("missed", *CAUSES[:3], "recalled, all present", "recalled, some present") + """Return the groups of misses for each question type, and in total, as Markdown. + + If the rows record `evidence_in_block`, the last column counts the misses whose block + held an evidence session as an episode. + """ + columns: tuple[str, ...] = ( + "missed", + *CAUSES[:3], + "recalled, all present", + "recalled, some present", + ) + episodes = any(row.get("evidence_in_block") is not None for row in rows) + if episodes: + columns = (*columns, "evidence episode in block") header = "| Type | " + " | ".join(columns) + " |" lines = [header, "|" + "---|" * (len(columns) + 1)] kinds = [kind for kind in QUESTION_TYPES if any(row["type"] == kind for row in rows)] for kind in [*kinds, "all"]: chosen = [row for row in rows if kind in ("all", row["type"])] groups = Counter(group(row) for row in chosen) + groups["evidence episode in block"] = sum( + bool(row.get("evidence_in_block")) for row in chosen + ) counts = [len(chosen), *(groups[name] for name in columns[1:])] lines.append(f"| {kind} | " + " | ".join(str(count) for count in counts) + " |") return "\n".join(lines) @@ -158,6 +181,9 @@ def main(argv: Sequence[str] | None = None) -> Path: result = json.loads(args.result.read_text()) config = memory_config(result) missed = [row["id"] for row in result["questions"] if not row["arms"]["memory"]["correct"]] + in_block = { + row["id"]: row["arms"]["memory"].get("episodes_in_block", []) for row in result["questions"] + } questions = {question.id: question for question in load(DATA_DIR / DATA_FILE)} replay = CachedModels(Offline(), CACHE_DIR / "replies.sqlite") marker = CachedModels(Remote(), CACHE_DIR / "replies.sqlite") @@ -169,15 +195,9 @@ def main(argv: Sequence[str] | None = None) -> Path: question = questions[question_id] namespace = ("bench", "lme_s", run_id, question.id) sessions = {fact_source(session.id): session.id for session in question.sessions} + sessions |= {episode_key(session.id): session.id for session in question.sessions} for session in question.sessions: - source = fact_source(session.id) if config.fact_sources else "" - memory.capture( - prompts.session_text(session), - namespace, - extractor(replay, config.variant, source, []), - actor=ACTOR, - observed_at=session.date, - ) + capture_session(memory, namespace, session, replay, config, []) store = stores[config.memory_ranking] facts = store.list(namespace, Filter(kind=RECALLED_KINDS), limit=10_000) marked = _mark(marker, question.question, question.answer, facts) @@ -205,6 +225,11 @@ def main(argv: Sequence[str] | None = None) -> Path: "best_rank": ranks, "cause": cause, "present": present, + "evidence_in_block": ( + None + if config.episodes == "none" + else evidence_in_block(question, in_block[question.id]) + ), } ) empty(namespace) @@ -221,6 +246,8 @@ def main(argv: Sequence[str] | None = None) -> Path: "budget": config.budget, "extractor": config.extractor, "fact_sources": config.fact_sources, + "session_episodes": config.session_episodes, + "episodes": config.episodes, }, "marker": JUDGE, "hotmemory_commit": commit, @@ -241,11 +268,14 @@ def main(argv: Sequence[str] | None = None) -> Path: def session_of_fact(fact: Record, sessions: dict[str, str]) -> str | None: """Return the session id of `fact` from its sources, or None if no source names one. - `sessions` maps the result of `fact_source` to the session id. + `sessions` maps the results of `fact_source` and of `episode_key` to the session id. A + source that is an episode id matches by its key. """ for source in fact.sources: - if source in sessions: - return sessions[source] + key = source.rpartition("/")[2].rpartition("@")[0] + for name in (source, key): + if name in sessions: + return sessions[name] return None diff --git a/bench/e2e.py b/bench/e2e.py index 838b78e..ff8b09f 100644 --- a/bench/e2e.py +++ b/bench/e2e.py @@ -26,7 +26,7 @@ from dataclasses import dataclass from datetime import UTC, datetime from pathlib import Path -from typing import Any +from typing import Any, cast from bench import embed, prompts from bench.data import ( @@ -36,6 +36,7 @@ DATA_SHA256, QUESTION_TYPES, Question, + Session, load, ) from bench.embed import CACHE_DIR, Tokens @@ -52,6 +53,7 @@ ) from bench.retrieval import RANKINGS, RESULTS_DIR, _commit from hotmemory import Embedder, Fact, Filter, Memory, Record, Store +from hotmemory.contract import Episodes HERE = Path(__file__).resolve().parent PER_TYPE = 9 @@ -59,6 +61,7 @@ RETRIEVED = 10 RECALL_K = 30 RECALL_BUDGET = 8000 +EPISODES = ("none", "cited", "search") COST_LIMIT = 50.0 EXTRACT_OVERHEAD = 700 ACTOR = "longmemeval-extractor" @@ -73,6 +76,9 @@ class Config: and recalls with `memory_ranking`, recalls `k` records into a block of at most `budget` characters, and extracts with the variant `extractor` of `prompts.VARIANTS`. With `fact_sources`, each fact carries the source of its session from `fact_source`. + With `session_episodes`, `capture` stores each session as an episode under the key + from `episode_key`, and the facts carry no session source. `episodes` is the value of + `recall`, and needs `session_episodes` unless it is `none`. """ retrieval_ranking: str = "fused" @@ -81,6 +87,8 @@ class Config: budget: int = RECALL_BUDGET extractor: str = "baseline" fact_sources: bool = True + session_episodes: bool = False + episodes: str = "none" def __post_init__(self) -> None: for ranking in (self.retrieval_ranking, self.memory_ranking): @@ -90,6 +98,10 @@ def __post_init__(self) -> None: raise ValueError(f"no extractor variant {self.extractor!r}") if self.k < 1 or self.budget < 1: raise ValueError("k and budget must be at least 1") + if self.episodes not in EPISODES: + raise ValueError(f"episodes must be one of {', '.join(EPISODES)}") + if self.episodes != "none" and not self.session_episodes: + raise ValueError("episodes needs session_episodes") @property def variant(self) -> prompts.Variant: @@ -156,7 +168,17 @@ def fact_source(session_id: str) -> str: The reader sees the sources in the block, and an evidence session id holds `answer`, so the source does not show the id. """ - return "session:" + hashlib.sha256(session_id.encode()).hexdigest()[:12] + return "session:" + _session_hash(session_id) + + +def episode_key(session_id: str) -> str: + """Return the key of the episode of a session: `session-` and the hash of `fact_source`.""" + return "session-" + _session_hash(session_id) + + +def _session_hash(session_id: str) -> str: + """Return the first 12 hex digits of the SHA-256 of `session_id`.""" + return hashlib.sha256(session_id.encode()).hexdigest()[:12] def extractor( @@ -205,6 +227,7 @@ class Arm: verdict: Reply extracted: int = 0 at_limit: int = 0 + episodes: tuple[str, ...] = () @property def correct(self) -> bool: @@ -218,11 +241,12 @@ def answer( context_chars: int, extracted: int = 0, at_limit: int = 0, + episodes: tuple[str, ...] = (), ) -> Arm: """Read `prompt` with READER, judge the answer with JUDGE, and return the arm.""" reply = models(READER, prompt, READER_MAX_TOKENS) verdict = models(JUDGE, prompts.judge(question, reply.text), JUDGE_MAX_TOKENS) - return Arm(context_chars, reply, verdict, extracted, at_limit) + return Arm(context_chars, reply, verdict, extracted, at_limit, episodes) def run_question( @@ -258,25 +282,51 @@ def run_question( written = 0 at_limit: list[str] = [] for session in question.sessions: - source = fact_source(session.id) if config.fact_sources else "" - written += len( - memory.capture( - prompts.session_text(session), - namespace, - extractor(models, config.variant, source, at_limit), - actor=ACTOR, - observed_at=session.date, - ) - ) - _, block = memory.recall( - question.question, [namespace], as_of=question.as_of, budget=config.budget, k=config.k + written += len(capture_session(memory, namespace, session, models, config, at_limit)) + records, block = memory.recall( + question.question, + [namespace], + as_of=question.as_of, + budget=config.budget, + k=config.k, + episodes=None if config.episodes == "none" else cast(Episodes, config.episodes), ) empty(namespace) + shown = tuple(record.key for record in records if record.kind == "episode") prompt = prompts.read_facts(question, block) - result["memory"] = answer(models, question, prompt, len(block), written, len(at_limit)) + result["memory"] = answer( + models, question, prompt, len(block), written, len(at_limit), shown + ) return result +def capture_session( + memory: Memory, + namespace: Sequence[str], + session: Session, + models: Models, + config: Config, + at_limit: list[str], +) -> list[str]: + """Run `capture` on one session as the memory arm does, and return the fact ids. + + With `config.session_episodes`, the session is stored as an episode under + `episode_key`. Else, with `config.fact_sources`, each fact carries `fact_source`. + """ + if config.session_episodes: + source, episode = "", episode_key(session.id) + else: + source, episode = (fact_source(session.id) if config.fact_sources else ""), None + return memory.capture( + prompts.session_text(session), + namespace, + extractor(models, config.variant, source, at_limit), + actor=ACTOR, + observed_at=session.date, + episode=episode, + ) + + def estimate( questions: Sequence[Question], tokens: Tokens, @@ -433,6 +483,7 @@ def row_of(question: Question, result: dict[str, Arm]) -> dict[str, Any]: "context_chars": found.context_chars, "facts_written": found.extracted, "sessions_at_limit": found.at_limit, + "episodes_in_block": list(found.episodes), "reader_input_tokens": found.answer.input_tokens, "answer": found.answer.text, "verdict": found.verdict.text, @@ -496,6 +547,14 @@ def main(argv: Sequence[str] | None = None) -> Path | None: action="store_true", help="write the facts with no session source, as phases 5 and 6 did", ) + parser.add_argument( + "--session-episodes", + action="store_true", + help="capture each session as an episode, and write the facts with no session source", + ) + parser.add_argument( + "--episodes", choices=EPISODES, default="none", help="the episodes value of recall" + ) parser.add_argument( "--replay", action="store_true", help="call no model: use cached replies, stop at a miss" ) @@ -521,6 +580,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: budget=args.recall_budget, extractor=args.extractor, fact_sources=not args.no_fact_sources, + session_episodes=args.session_episodes, + episodes=args.episodes, ) except ValueError as error: parser.error(str(error)) @@ -604,6 +665,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: "budget": config.budget, "extractor": config.extractor, "fact_sources": config.fact_sources, + "session_episodes": config.session_episodes, + "episodes": config.episodes, }, "arms": { "full": "every session of the question", @@ -614,8 +677,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: "memory": ( f"capture on each session in date order with the {config.extractor} " f"extractor, then recall with the {config.memory_ranking} ranking, k " - f"{config.k}, budget {config.budget} characters, as_of the end of the " - "question day" + f"{config.k}, budget {config.budget} characters, episodes " + f"{config.episodes}, as_of the end of the question day" ), }, "questions": len(questions), diff --git a/bench/tests/test_diagnose.py b/bench/tests/test_diagnose.py index 2cfe33f..d2fc473 100644 --- a/bench/tests/test_diagnose.py +++ b/bench/tests/test_diagnose.py @@ -5,8 +5,17 @@ import pytest -from bench.diagnose import block_ids, group, memory_config, session_of_fact, table, verdict -from bench.e2e import BASELINE, fact_source +from bench.data import Question +from bench.diagnose import ( + block_ids, + evidence_in_block, + group, + memory_config, + session_of_fact, + table, + verdict, +) +from bench.e2e import BASELINE, episode_key, fact_source from bench.models import READER, CachedModels, CacheMiss, Offline, Reply from hotmemory import Record @@ -60,6 +69,10 @@ def test_memory_config_reads_the_result_file() -> None: memory = {"ranking": "fused", "k": 30, "budget": 8000, "extractor": "detail"} sourced = memory_config({"config": {"memory": {**memory, "fact_sources": True}}}) assert (sourced.extractor, sourced.fact_sources) == ("detail", True) + assert (sourced.session_episodes, sourced.episodes) == (False, "none") + episodes = {**memory, "session_episodes": True, "episodes": "search"} + found = memory_config({"config": {"memory": episodes}}) + assert (found.session_episodes, found.episodes) == (True, "search") def test_session_of_fact_reads_the_source() -> None: @@ -67,6 +80,44 @@ def test_session_of_fact_reads_the_source() -> None: sourced = replace(fact("k"), sources=(fact_source("answer_a_1"),)) assert session_of_fact(sourced, sessions) == "answer_a_1" assert session_of_fact(fact("k"), sessions) is None + keys = {episode_key("answer_a_1"): "answer_a_1"} + cited = replace(fact("k"), sources=(f"bench/q/{episode_key('answer_a_1')}@1",)) + assert session_of_fact(cited, keys) == "answer_a_1" + + +def test_evidence_in_block_matches_the_episode_keys() -> None: + question = Question( + id="q", + type="multi-session", + question="?", + answer="!", + date=AS_OF, + answer_session_ids=("answer_a_1",), + sessions=(), + ) + assert evidence_in_block(question, [episode_key("answer_a_1")]) + assert not evidence_in_block(question, [episode_key("other")]) + assert not evidence_in_block(question, []) + + +def test_table_counts_the_evidence_episodes() -> None: + rows: list[dict[str, Any]] = [ + { + "type": "multi-session", + "cause": "not extracted", + "present": None, + "evidence_in_block": True, + }, + { + "type": "multi-session", + "cause": "not extracted", + "present": None, + "evidence_in_block": False, + }, + ] + lines = table(rows).splitlines() + assert lines[0].endswith("| evidence episode in block |") + assert lines[2] == "| multi-session | 2 | 2 | 0 | 0 | 0 | 0 | 1 |" def test_table_splits_the_recalled_group() -> None: diff --git a/bench/tests/test_e2e.py b/bench/tests/test_e2e.py index edd1a79..f4231e3 100644 --- a/bench/tests/test_e2e.py +++ b/bench/tests/test_e2e.py @@ -11,7 +11,9 @@ from bench.e2e import ( SAMPLES, Config, + capture_session, choose_sample, + episode_key, estimate, extractor, fact_source, @@ -21,7 +23,7 @@ summarize, ) from bench.models import EXTRACTOR, JUDGE, READER, CachedModels, Offline, Reply -from hotmemory import MemoryStore +from hotmemory import Memory, MemoryStore from tests.conftest import FakeClock, fake_embedder FIXTURE = Path(__file__).with_name("fixture.json") @@ -293,3 +295,58 @@ def complete(model: str, prompt: str, max_tokens: int, schema: dict[str, Any] | ) found = replay._db.execute("SELECT reply FROM replies WHERE key = 'old'").fetchone()[0] assert not Reply(**json.loads(found)).stopped + + +def test_config_refuses_episodes_without_session_episodes() -> None: + with pytest.raises(ValueError, match="session_episodes"): + Config(episodes="cited") + with pytest.raises(ValueError, match="episodes must be"): + Config(session_episodes=True, episodes="all") + + +def test_episode_key_hides_the_session_id() -> None: + key = episode_key("answer_a_1") + assert key == "session-" + fact_source("answer_a_1").removeprefix("session:") + assert "answer" not in key + + +@pytest.mark.parametrize("episodes", ["cited", "search"]) +def test_run_question_recalls_the_sessions_as_episodes( + questions: dict[str, Question], episodes: str +) -> None: + question = questions["q_one"] + store = MemoryStore(embedder=fake_embedder, clock=FakeClock()) + models = FakeModels() + + def empty(namespace: Sequence[str]) -> None: + for record in store.list(namespace, limit=1000): + store.delete(record.namespace, record.key) + + config = Config(budget=100_000, session_episodes=True, episodes=episodes) + result = run_question( + question, models, {"fused": store, "vector": store}, empty, "r1", ("memory",), config + ) + block = [p for m, p in models.prompts if m == READER and "several facts" in p[:60]][-1] + late = next(s for s in question.sessions if s.id == "answer_a_late") + assert f"[sources: {episode_key(late.id)}@1]" in block + assert "\nepisodes:\n## session-" in block + assert f"\n## {episode_key(late.id)}@1 [observed: " in block + assert "session:" not in block + assert "answer_" not in block + assert episode_key(late.id) in result["memory"].episodes + assert store.list(("bench",)) == [] + + +def test_capture_session_replays_the_same_extractor_prompts( + questions: dict[str, Question], +) -> None: + question = questions["q_one"] + config = Config(session_episodes=True) + seen: list[list[str]] = [] + for run in ("r1", "r2"): + models = FakeModels() + memory = Memory(MemoryStore(embedder=fake_embedder, clock=FakeClock())) + for session in question.sessions: + capture_session(memory, ("bench", run, question.id), session, models, config, []) + seen.append([p for m, p in models.prompts if m == EXTRACTOR]) + assert seen[0] == seen[1] From 1b2d3dc8bd41c9a4aa3bf19821ab71ea13b5cb15 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 15 Oct 2026 23:12:15 +0530 Subject: [PATCH 4/5] Report the episodes runs of phase 8 docs/benchmarks.md reports each episodes value and budget on sample A, the best value on sample B, the cost of each run, and the diagnosis. search at 64,000 characters scored 45 of 54 on both samples, against 29 and 31 with facts alone, at about 12,800 reader tokens. The README gets the row, and the contract keeps the budget note beside its default. --- README.md | 19 +++++---- docs/benchmarks.md | 92 +++++++++++++++++++++++++++++++++++++++++++ docs/contracts.md | 8 ++-- docs/internal/plan.md | 5 ++- 4 files changed, 111 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 55b3a83..b580a7d 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,8 @@ had these results with the vector ranking. For 98.3% of them, an evidence sessio the top 5. For 99.3% of them, every evidence session was in the top 10. The end-to-end score ran on two samples of 54 questions, sample A on 2026-10-10 and -sample B on 2026-10-14. The reader and the judge were `gpt-4o-2024-08-06`. +sample B on 2026-10-14. Later rows give their own date. The reader and the judge were +`gpt-4o-2024-08-06`. | What the reader got | Sample A | Sample B | |---|---|---| @@ -149,12 +150,16 @@ sample B on 2026-10-14. The reader and the judge were `gpt-4o-2024-08-06`. | The 10 sessions that the search ranked first | 45 of 54 (83.3%) | 48 of 54 (88.9%) | | The block of `recall`, after `capture` with a `gpt-4o-mini` extractor and the `baseline` prompt | 19 of 54 (35.2%) | 27 of 54 (50.0%) | | The same, on 2026-10-14, with the `detail` prompt and a session source on each fact | 31 of 54 (57.4%) | 32 of 54 (59.3%) | - -The memory contract scores below the full history. With the `baseline` prompt, the -extractor did not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 -misses on sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With 54 questions, each score can be about 10 points off, and the memory arm -differs by 14.8 points between the two samples. In the cloud, a search took 361 ms at p50 with the vector ranking, and -998 ms with the fused ranking. +| The same, on 2026-10-15, with each session stored as an episode, and `recall` with `episodes="search"` and a budget of 64,000 characters, about 12,800 tokens | 45 of 54 (83.3%) | 45 of 54 (83.3%) | + +Facts alone score below the full history. With the `baseline` prompt, the extractor did +not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 misses on +sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With the +episodes that a search adds after the facts, the memory arm scores above the full history +and equal to the 10 sessions on sample A, with less than half of their tokens. With 54 +questions, each score can be about 10 points off. With the `baseline` prompt, the memory +arm differed by 14.8 points between the two samples. In the cloud, a search took 361 ms at +p50 with the vector ranking, and 998 ms with the fused ranking. ## Documents diff --git a/docs/benchmarks.md b/docs/benchmarks.md index c1b0b4c..b59f10f 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -333,6 +333,93 @@ Phase 7 read the misses and found these facts: them. The marker reads all facts of a question in one call, 226 to 300 facts here. So the group with no stored fact can be smaller than the table shows, in each run. +### Episodes behind facts + +Phase 8 stored each session as an episode beside its facts, and let `recall` add episodes +to the block after the facts. The runs use the `detail` extractor of phase 7, and the +options `--session-episodes` and `--episodes` of the runner. `capture` stores each session +under the key `session-`, with the hash of phase 7, and each fact cites that episode. +The block shows the source as `session-@1`. The facts carry no `session:` +source. The two values of `episodes`: + +| Value | The episodes after the facts | +|---|---| +| `none` | No episode. The block holds the facts only. | +| `cited` | The episodes that the facts of the block cite, in the order of the facts. | +| `search` | The top 30 episodes of a fused search with the question. | + +Each block holds whole episodes only, and it stops before the first episode that does not +fit in the budget. `none` was run at 16,000 characters only. With k of 30, its largest +block held 5,344 characters on sample A and 5,626 on sample B, so a larger budget gives +the same block. + +Observed on 2026-10-15, at commit `320e891`, with 4 workers. + +| Run | Sample | Correct of 54 | Reader tokens at the median | Episodes in the block at the median | Gained and lost against `none` | Cost | File | +|---|---|---|---|---|---|---|---| +| `none`, 16,000 | A | 29 | 1,715 | 0 | | $0.52 | `e2e-sample-a-p8-none-16k.json` | +| `cited`, 16,000 | A | 31 | 1,728 | 0 | +3, -1 | $0.12 | `e2e-sample-a-p8-cited-16k.json` | +| `search`, 16,000 | A | 34 | 1,738 | 0 | +6, -1 | $0.08 | `e2e-sample-a-p8-search-16k.json` | +| `cited`, 32,000 | A | 38 | 6,300 | 2 | +10, -1 | $1.04 | `e2e-sample-a-p8-cited-32k.json` | +| `search`, 32,000 | A | 43 | 5,715 | 2 | +14, -0 | $0.75 | `e2e-sample-a-p8-search-32k.json` | +| `cited`, 64,000 | A | 44 | 12,446 | 4 | +15, -0 | at least $1.01 | `e2e-sample-a-p8-cited-64k.json` | +| `search`, 64,000 | A | 45 | 12,849 | 4 | +17, -1 | $1.93 | `e2e-sample-a-p8-search-64k.json` | +| `none`, 16,000 | B | 31 | 1,723 | 0 | | $0.57 | `e2e-sample-b-p8-none-16k.json` | +| `search`, 64,000 | B | 45 | 12,676 | 4 | +15, -1 | $1.91 | `e2e-sample-b-p8-search-64k.json` | + +The retrieval arm read 29,878 tokens at the median on sample A, and scored 45. `search` at +64,000 characters scored 45 on sample A and 45 on sample B, with 12,849 and 12,676 reader +tokens. Sample A and sample B moved the same way, from 29 to 45 and from 31 to 45. + +The `none` runs scored 29 on sample A and 31 on sample B, against 31 and 32 with the +`session:` source of phase 7. The new source changes the line length in the block, +so the list of known facts that `capture` gives the extractor can change. Then the +extractor gets a new prompt and a new reply. The `none` runs are the baseline of this +section. + +Each run after the first on a sample replayed `capture` from the extractor cache. A run +that changed only `episodes` or the budget paid for the reader and the judge only. The +first try of the runs with `cited` at 64,000 and every `search` run stopped at an internet +outage, and the next try of `cited` at 64,000 stopped at one engine error. Each later try +took the replies of the earlier tries from the cache. The log of the first try of `cited` at +64,000 was lost, so its cost is not known. The cost of a run is the sum of its tries. The +runs and the diagnoses cost at least $8.10. + +Correct answers by question type: + +| Question type | A `none` | A `cited` 64,000 | A `search` 32,000 | A `search` 64,000 | B `none` | B `search` 64,000 | +|---|---|---|---|---|---|---| +| single-session-user | 3 | 9 | 9 | 8 | 6 | 9 | +| single-session-assistant | 2 | 6 | 9 | 9 | 2 | 9 | +| single-session-preference | 4 | 7 | 5 | 6 | 1 | 5 | +| multi-session | 6 | 6 | 6 | 6 | 6 | 7 | +| knowledge-update | 8 | 8 | 8 | 8 | 7 | 7 | +| temporal-reasoning | 6 | 8 | 6 | 8 | 9 | 8 | +| All | 29 | 44 | 43 | 45 | 31 | 45 | + +The diagnosis of the `none` runs and of `search` at 64,000, with the commands above. The +last row counts the misses whose block held an evidence session as an episode. The +diagnoses cost $0.17 in total. + +| Cause | A `none` | A `search` 64,000 | B `none` | B `search` 64,000 | +|---|---|---|---|---| +| No stored fact states the answer or a value that it needs | 17 | 5 | 17 | 5 | +| A stored fact holds it, and the fused search ranked it below 30 | 1 | 0 | 0 | 0 | +| A marked fact was in the top 30, and `recall` left it out | 0 | 0 | 0 | 0 | +| The block holds every needed fact, and the reader missed | 1 | 2 | 0 | 1 | +| The block holds some of the needed facts | 6 | 2 | 6 | 3 | +| Misses | 25 | 9 | 23 | 9 | +| The block held an evidence session as an episode | not run | 8 | not run | 8 | + +The two target groups of the phase got smaller on both samples. The misses with no stored +fact went from 17 to 5 on each sample. The misses with some needed facts went from 6 to 2 +on sample A and from 6 to 3 on sample B. The `none` runs missed every question of the +type single-session-assistant with no stored fact, 7 on each sample. `search` at 64,000 +missed none of that type. In 8 of the 9 misses on each sample, the block held an evidence +session. A question of the type multi-session can need more sessions than the block held. +The type single-session-preference has the most misses with `search` at 64,000: 3 on +sample A and 4 on sample B. + ## Known limits - The end-to-end numbers come from samples of 54 questions, so one question moves an arm @@ -344,5 +431,10 @@ Phase 7 read the misses and found these facts: stronger extractor model was not measured. - The diagnosis marks the facts of a question in one model call, and it can miss facts in a long list. Its count of misses with no stored fact can be too high. +- The reader prompt of the memory arm calls the block "facts extracted from history chats", + also when the block holds episodes. A prompt that names the episodes was not tried. +- In the runs of phase 8, the local engine refused one fused search once, because it + found a BM25 index of `memory_v1` stale against its table. A second try of the same run + did not show the error. The cause was not found. - `recall` keeps the records that are valid at `as_of`, and a search over history does not exist yet. A question about a value before it changed can need a superseded record. diff --git a/docs/contracts.md b/docs/contracts.md index 4d07139..6924030 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -323,7 +323,9 @@ For example, a fact under `team/alerts` with the source `team/alerts/notes-0001@ `[sources: notes-0001@1]`. Any other source shows in full. The block holds whole lines only, joined by newlines. It stops before the first line that does not fit in the budget. The list holds the records of the block. The budget counts the characters of -the block in this line format, and its default is 2,000. +the block in this line format, and its default is 2,000. A consumer that renders the +records in its own format passes its own `budget` and `k`. Then the default does not +cut the list first. With `episodes`, the fact lines come first, and they fit in the budget as above. Then, if an episode fits in what is left, a line `episodes:` follows, and then each episode in this @@ -349,9 +351,7 @@ The two values of `episodes` choose the episodes in two ways. top k, in the order of the search. With `as_of`, only the episodes that are valid at `as_of` remain. The list holds the -facts of the block and then its episodes. A consumer that renders the -records in its own format passes its own `budget` and `k`. Then the default does not -cut the list first. +facts of the block and then its episodes. The block of `profile` has a group for each kind that has records, in the order fact, profile, procedure. A line `:` starts each group, and its records follow diff --git a/docs/internal/plan.md b/docs/internal/plan.md index d88924a..4fa0b84 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -179,8 +179,9 @@ surface. `capture` from the cache. The blocks of documents that `load` stored get shorter. - The benchmark extractor stops setting `session:`. The episode key is `session-`, with the hash of phase 7. The first run pays for the extractor again. -- The budgets are 16,000, 32,000, and 64,000 characters. At 8,000 characters, the facts - of phase 7 leave no room for a whole session (estimate, not observed). +- The budgets are 16,000, 32,000, and 64,000 characters. The reason for dropping 8,000 was + an estimate that the facts of phase 7 leave no room for a whole session. The runs then + showed blocks of facts of at most 5,626 characters, so the estimate was too high. Acceptance criteria: From b79648321c8e70d270f6f08a79d99091588d189e Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 15 Oct 2026 23:47:05 +0530 Subject: [PATCH 5/5] Leave a forgotten episode out of a cited block recall with episodes="cited" read each episode with get, which does not hide a record past its forget_after. The block could then show an episode that search hides. _cited now skips such an episode, and keeps a superseded revision, which the fact cites on purpose. --- CHANGELOG.md | 3 ++- docs/contracts.md | 3 ++- docs/guarantees.md | 2 +- src/hotmemory/contract.py | 17 ++++++++++++++--- tests/test_contract.py | 33 +++++++++++++++++++++++++++++++++ 5 files changed, 52 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 925bc0d..ab8d214 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,7 +61,8 @@ each one, and its failure message points to this file. A capture of the same text under the same key writes nothing and calls no extractor. - `Memory.recall` takes `episodes`, `"cited"` or `"search"`. With it, the block adds the episodes that its facts cite, or the top episodes for the query, after the facts and - inside the budget. Without it, `recall` returns no episode, as before. + inside the budget. Without it, `recall` returns no episode, as before. Neither value + returns an episode past its `forget_after`. - The block of `recall` and `profile` shows a source that is the id of a record in the namespace of its line as `@`. Before, it showed the full id. The stored sources do not change. diff --git a/docs/contracts.md b/docs/contracts.md index 6924030..c162a60 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -346,7 +346,8 @@ The two values of `episodes` choose the episodes in two ways. - `"cited"` reads the sources of each fact of the block, in order. Each source that is the id of a record of kind `episode` in the allowed scopes gives that revision, read with `get`. An episode cited twice comes once, at its first place. A source that is not - such an id is skipped. + such an id is skipped. An episode past its `forget_after` is skipped, as `search` + hides it. A superseded revision stays, because the fact cites that revision. - `"search"` searches the current episodes in the allowed scopes with the query, for the top k, in the order of the search. diff --git a/docs/guarantees.md b/docs/guarantees.md index ef6e1b4..ea71620 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -59,7 +59,7 @@ A row that no test proves yet names the phase that will prove it. | Do `recall` and `profile` return episodes? | `profile` never does, and its counts leave episodes out. `recall` returns episodes only with `episodes`. Without it, `recall` reads the current records of kind `fact`, `profile`, and `procedure` only. | answered | `test_recall_and_profile_leave_episodes_out`, `test_recall_block_matches_its_pinned_string`, `test_profile_block_matches_its_pinned_string` | | What does `capture` store with an episode key? | The text, as an episode under that key, with the `actor` and `observed_at` of the call. Each fact names the id of the episode in `sources`. `remember` writes the facts in one flush, and one `put` then writes the episode, so each table gets one flush. If the extractor raises, nothing is written. | answered | `test_capture_with_an_episode_stores_it_and_links_its_facts`, `test_capture_writes_one_flush_per_table`, `test_capture_with_a_failing_extractor_writes_no_episode`, `test_capture_refuses_a_bad_episode`, `test_capture_refuses_an_episode_key_that_holds_a_fact` | | Does a retried `capture` with an episode key write anything? | No. If the episode is current with the same content, the extractor is not called and nothing is written, so this holds for an extractor that calls a model. | answered | `test_a_retried_capture_with_an_episode_writes_nothing_new` | -| Which episodes does `recall` add with `episodes`? | With `"cited"`, the episodes that the facts of the block name in `sources`, in order, once each, and only inside the allowed scopes. With `"search"`, the top k episodes for the query. `as_of` filters them as it filters facts. They follow the facts, whole, and the block stops before the first episode that does not fit. An episode never takes the place of a fact. | answered | `test_recall_cited_adds_the_episodes_of_the_facts`, `test_recall_cited_names_each_episode_once`, `test_recall_cited_skips_an_episode_outside_the_scopes`, `test_recall_search_adds_the_top_episodes`, `test_recall_episodes_never_take_the_place_of_a_fact`, `test_recall_stops_before_the_first_episode_that_does_not_fit`, `test_recall_as_of_filters_the_episodes`, `test_recall_refuses_an_unknown_episodes_value` | +| Which episodes does `recall` add with `episodes`? | With `"cited"`, the episodes that the facts of the block name in `sources`, in order, once each, and only inside the allowed scopes. With `"search"`, the top k episodes for the query. Neither value returns an episode past its `forget_after`. `"cited"` keeps a superseded revision, because the fact cites that revision. `as_of` filters them as it filters facts. They follow the facts, whole, and the block stops before the first episode that does not fit. An episode never takes the place of a fact. | answered | `test_recall_cited_adds_the_episodes_of_the_facts`, `test_recall_cited_names_each_episode_once`, `test_recall_cited_skips_an_episode_outside_the_scopes`, `test_recall_search_adds_the_top_episodes`, `test_recall_episodes_never_take_the_place_of_a_fact`, `test_recall_stops_before_the_first_episode_that_does_not_fit`, `test_recall_as_of_filters_the_episodes`, `test_recall_refuses_an_unknown_episodes_value`, `test_recall_cited_skips_an_episode_past_its_forget_after`, `test_recall_cited_keeps_a_superseded_revision` | | How does the block show a source? | In full, except the id of a record in the namespace of the line, which shows as `@`. The stored source keeps the full id. | answered | `test_recall_shows_a_source_in_its_namespace_as_key_and_revision` | | Can a `put` move a key between `episode` and another kind? | No. A key holds episodes only, or holds no episode at all. A `put` that crosses that line raises an error and writes nothing. A key can change between `fact`, `profile`, and `procedure`. | answered | `test_key_cannot_cross_the_episode_line` | | What does a `put` with `close_previous` write? | The new revision, and the previous row with `superseded_by`, `valid_until` set to the new `valid_from`, and `expired_at` set to the clock's time. A missing `valid_from`, or one earlier than the previous `valid_from`, raises an error, and the put or the whole flush writes nothing. Without `close_previous`, the previous span stays open. | answered | `test_close_previous_closes_the_current_revision`, `test_close_previous_refuses_a_span_it_cannot_close`, `test_close_previous_on_a_new_key_or_same_content`, `test_put_without_close_previous_leaves_the_span_open` | diff --git a/src/hotmemory/contract.py b/src/hotmemory/contract.py index 7468284..9314bb3 100644 --- a/src/hotmemory/contract.py +++ b/src/hotmemory/contract.py @@ -10,7 +10,14 @@ from typing import Literal, TypeAlias from hotmemory._chunks import chunk_document -from hotmemory._rules import check_prefix, is_duplicate, newest_first, next_revision, under_prefix +from hotmemory._rules import ( + check_prefix, + is_duplicate, + is_forgotten, + newest_first, + next_revision, + under_prefix, +) from hotmemory._writer import check_count from hotmemory.filter import Filter from hotmemory.memory import utc_now @@ -225,7 +232,7 @@ def _cited(self, records: Sequence[Record], allowed: Sequence[tuple[str, ...]]) """Return the episodes that the sources of `records` name, in order, without repeats. A source that is not a record id, is outside `allowed`, or names no episode is - skipped. + skipped, and so is an episode past its `forget_after`. A superseded revision stays. """ cited: dict[str, Record | None] = {} for record in records: @@ -241,7 +248,11 @@ def _cited(self, records: Sequence[Record], allowed: Sequence[tuple[str, ...]]) continue revision = int(source.rpartition("@")[2]) found = self._store.get(namespace, key, revision) - if found is not None and found.kind == "episode": + if ( + found is not None + and found.kind == "episode" + and not is_forgotten(found, self._clock()) + ): cited[source] = found return [episode for episode in cited.values() if episode is not None] diff --git a/tests/test_contract.py b/tests/test_contract.py index 483a01b..7859866 100644 --- a/tests/test_contract.py +++ b/tests/test_contract.py @@ -746,6 +746,39 @@ def test_recall_cited_skips_an_episode_outside_the_scopes(memory: Memory) -> Non assert "episodes:" not in block +def test_recall_cited_skips_an_episode_past_its_forget_after( + memory: Memory, clock: FakeClock +) -> None: + memory.store.put( + SCOPE, + "chat-0001", + kind="episode", + content="The disk fills at night.", + forget_after=START + timedelta(hours=1), + ) + sources = ("team/alerts/chat-0001@1",) + memory.remember([Fact(kind="fact", content="The disk fills.", sources=sources)], SCOPE) + records, _ = memory.recall("disk", [SCOPE], budget=10_000, episodes="cited") + assert [record.kind for record in records] == ["fact", "episode"] + + clock.advance(timedelta(hours=2)) + for episodes in ("cited", "search"): + records, block = memory.recall("disk", [SCOPE], budget=10_000, episodes=episodes) + assert [record.kind for record in records] == ["fact"] + assert "episodes:" not in block + + +def test_recall_cited_keeps_a_superseded_revision(memory: Memory, clock: FakeClock) -> None: + memory.store.put(SCOPE, "chat-0001", kind="episode", content="The disk fills at night.") + sources = ("team/alerts/chat-0001@1",) + memory.remember([Fact(kind="fact", content="The disk fills.", sources=sources)], SCOPE) + clock.advance() + memory.store.put(SCOPE, "chat-0001", kind="episode", content="The disk fills at noon.") + records, block = memory.recall("disk", [SCOPE], budget=10_000, episodes="cited") + assert [(record.key, record.revision) for record in records[1:]] == [("chat-0001", 1)] + assert "The disk fills at night." in block + + def test_recall_search_adds_the_top_episodes(memory: Memory, clock: FakeClock) -> None: capture_two_chats(memory, clock) records, block = memory.recall("pool nodes ceiling", [SCOPE], k=1, episodes="search")