Phase 8: episodes behind facts

CHANGELOG.md

@@ -56,3 +56,13 @@ each one, and its failure message points to this file.
 - `Memory.recall` and `Memory.profile` read the records of kind `fact`, `profile`, and
   `procedure` only. They never return an episode, and the counts of `profile` leave
   episodes out.
+- `Memory.capture` takes `episode`, a key. With it, `capture` also stores the text as an
+  episode under that key, after the facts, and each fact names the episode in `sources`.
+  A capture of the same text under the same key writes nothing and calls no extractor.
+- `Memory.recall` takes `episodes`, `"cited"` or `"search"`. With it, the block adds the
+  episodes that its facts cite, or the top episodes for the query, after the facts and
+  inside the budget. Without it, `recall` returns no episode, as before. Neither value
+  returns an episode past its `forget_after`.
+- The block of `recall` and `profile` shows a source that is the id of a record in the
+  namespace of its line as `<key>@<revision>`. Before, it showed the full id. The stored
+  sources do not change.

README.md

@@ -21,8 +21,10 @@ The library has two layers:
   fact, forget by id or by horizon, and load the profile of a subject. Extraction from raw
   text is optional, and it takes an extractor callable that the caller supplies. `load`
   cuts a Markdown document into chunks, stores each chunk as an episode, and extracts
-  facts that name their episode as a source. `recall` and `profile` return facts,
-  profiles, and procedures, never episodes.
+  facts that name their episode as a source. `capture` with an `episode` key stores its
+  text as an episode in the same way. `profile` returns facts, profiles, and procedures,
+  never episodes. `recall` returns them too, and adds episodes after the facts only when
+  the caller asks for them.
 
 The library runs in the process of the consumer and calls the Hotdata API with the API key
 of the consumer. There is no hotmemory server. The first consumer is an incident
@@ -139,20 +141,25 @@ had these results with the vector ranking. For 98.3% of them, an evidence sessio
 the top 5. For 99.3% of them, every evidence session was in the top 10.
 
 The end-to-end score ran on two samples of 54 questions, sample A on 2026-10-10 and
-sample B on 2026-10-14. The reader and the judge were `gpt-4o-2024-08-06`.
+sample B on 2026-10-14. Later rows give their own date. The reader and the judge were
+`gpt-4o-2024-08-06`.
 
 | What the reader got | Sample A | Sample B |
 |---|---|---|
 | Every session of the question, about 114,000 tokens | 39 of 54 (72.2%) | 43 of 54 (79.6%) |
 | The 10 sessions that the search ranked first | 45 of 54 (83.3%) | 48 of 54 (88.9%) |
 | The block of `recall`, after `capture` with a `gpt-4o-mini` extractor and the `baseline` prompt | 19 of 54 (35.2%) | 27 of 54 (50.0%) |
 | The same, on 2026-10-14, with the `detail` prompt and a session source on each fact | 31 of 54 (57.4%) | 32 of 54 (59.3%) |
-
-The memory contract scores below the full history. With the `baseline` prompt, the
-extractor did not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27
-misses on sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With 54 questions, each score can be about 10 points off, and the memory arm
-differs by 14.8 points between the two samples. In the cloud, a search took 361 ms at p50 with the vector ranking, and
-998 ms with the fused ranking.
+| The same, on 2026-10-15, with each session stored as an episode, and `recall` with `episodes="search"` and a budget of 64,000 characters, about 12,800 tokens | 45 of 54 (83.3%) | 45 of 54 (83.3%) |
+
+Facts alone score below the full history. With the `baseline` prompt, the extractor did
+not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 misses on
+sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With the
+episodes that a search adds after the facts, the memory arm scores above the full history
+and equal to the 10 sessions on sample A, with less than half of their tokens. With 54
+questions, each score can be about 10 points off. With the `baseline` prompt, the memory
+arm differed by 14.8 points between the two samples. In the cloud, a search took 361 ms at
+p50 with the vector ranking, and 998 ms with the fused ranking.
 
 ## Documents
 

bench/README.md

@@ -87,6 +87,8 @@ The options are these:
 | `--recall-budget` | the `budget` of the memory recall, in characters | `8000` |
 | `--extractor` | a variant name in `VARIANTS` of `bench/prompts.py` | `baseline` |
 | `--no-fact-sources` | write the facts with no session source, as phases 5 and 6 did | off |
+| `--session-episodes` | capture each session as an episode, with the facts citing it | off |
+| `--episodes` | `none`, `cited`, or `search`, the `episodes` of the memory recall; a value other than `none` needs `--session-episodes` | `none` |
 | `--replay` | call no model, and stop at the first reply that the cache does not hold | off |
 | `--workers` | the number of processes that run the questions | `1` |
 | `--tag` | a suffix for the name of the result file | none |
@@ -101,7 +103,14 @@ the session id. The block shows the sources to the reader, and the id of an evid
 session holds `answer`, so the source does not show the id. If an extractor reply reaches
 its limit of 4,000 output tokens, the session writes no facts. The cache keeps that reply,
 so a replay skips the same session. The result file counts these sessions in
-`sessions_at_limit`. The result file is `bench/results/e2e-sample-<a or b>[-<tag>].json`,
+`sessions_at_limit`.
+
+With `--session-episodes`, `capture` stores each session as an episode under the key
+`session-<hash>`, with the same hash. Each fact then cites that episode, and carries no
+`session:<hash>` source. The block shows the source as `session-<hash>@1`. The source
+holds no run id, so a second run with another `--episodes` or `--recall-budget` replays
+`capture` from the cache. The result file lists the episode keys of each block in
+`episodes_in_block`. The result file is `bench/results/e2e-sample-<a or b>[-<tag>].json`,
 and it records each option.
 
 ## The checks of the results
@@ -136,4 +145,6 @@ For the last cause, GPT-4o tells whether the marked facts in the block hold ever
 that the answer needs ("all present") or only some of them ("some present"). The command
 prints the groups for each question type, and writes them next to the result file, as
 `<result>-diagnosis.json`. For each fact that holds the answer, the file gives its session
-id, if the fact has a source.
+id, if the fact has a source. If the run recalled episodes, each miss also gets
+`evidence_in_block`: whether the block held an evidence session as an episode. The table
+then counts these misses in a last column.

bench/diagnose.py

@@ -6,7 +6,8 @@
 and extractor variant that the result file records. A model marks the facts that hold the
 answer, and the search ranks every fact for the question. If a marked fact is in the block,
 a second model call tells whether the block holds every needed fact ("all present") or only
-some of them ("some present").
+some of them ("some present"). If the run recalled episodes, each miss also records whether
+the block held an evidence session as an episode, from the episode keys of the result file.
 
     uv run --group bench --env-file .env python -m bench.diagnose --result <file>
 """
@@ -22,15 +23,14 @@
 from pathlib import Path
 from typing import Any
 
-from bench import prompts
-from bench.data import DATA_DIR, DATA_FILE, QUESTION_TYPES, load
+from bench.data import DATA_DIR, DATA_FILE, QUESTION_TYPES, Question, load
 from bench.e2e import (
-    ACTOR,
     BASELINE,
     RECALL_BUDGET,
     RECALL_K,
     Config,
-    extractor,
+    capture_session,
+    episode_key,
     fact_source,
     local_store,
 )
@@ -113,7 +113,8 @@ def memory_config(result: dict[str, Any]) -> Config:
     """Return the configuration of the memory arm that a result file records.
 
     A result file of phase 5 records no `memory` section, and its run used `BASELINE`. A
-    result file of phase 6 records no `fact_sources`, and its facts carried no source.
+    result file of phase 6 records no `fact_sources`, and its facts carried no source. A
+    result file before phase 8 records no `session_episodes` and no `episodes`.
     """
     found = result["config"].get("memory")
     if found is None:
@@ -124,18 +125,40 @@ def memory_config(result: dict[str, Any]) -> Config:
         budget=found["budget"],
         extractor=found["extractor"],
         fact_sources=found.get("fact_sources", False),
+        session_episodes=found.get("session_episodes", False),
+        episodes=found.get("episodes", "none"),
     )
 
 
+def evidence_in_block(question: Question, shown: Sequence[str]) -> bool:
+    """Return True if an episode key in `shown` is that of an evidence session of `question`."""
+    return any(episode_key(session_id) in shown for session_id in question.answer_session_ids)
+
+
 def table(rows: Sequence[dict[str, Any]]) -> str:
-    """Return the groups of misses for each question type, and in total, as Markdown."""
-    columns = ("missed", *CAUSES[:3], "recalled, all present", "recalled, some present")
+    """Return the groups of misses for each question type, and in total, as Markdown.
+
+    If the rows record `evidence_in_block`, the last column counts the misses whose block
+    held an evidence session as an episode.
+    """
+    columns: tuple[str, ...] = (
+        "missed",
+        *CAUSES[:3],
+        "recalled, all present",
+        "recalled, some present",
+    )
+    episodes = any(row.get("evidence_in_block") is not None for row in rows)
+    if episodes:
+        columns = (*columns, "evidence episode in block")
     header = "| Type | " + " | ".join(columns) + " |"
     lines = [header, "|" + "---|" * (len(columns) + 1)]
     kinds = [kind for kind in QUESTION_TYPES if any(row["type"] == kind for row in rows)]
     for kind in [*kinds, "all"]:
         chosen = [row for row in rows if kind in ("all", row["type"])]
         groups = Counter(group(row) for row in chosen)
+        groups["evidence episode in block"] = sum(
+            bool(row.get("evidence_in_block")) for row in chosen
+        )
         counts = [len(chosen), *(groups[name] for name in columns[1:])]
         lines.append(f"| {kind} | " + " | ".join(str(count) for count in counts) + " |")
     return "\n".join(lines)
@@ -158,6 +181,9 @@ def main(argv: Sequence[str] | None = None) -> Path:
     result = json.loads(args.result.read_text())
     config = memory_config(result)
     missed = [row["id"] for row in result["questions"] if not row["arms"]["memory"]["correct"]]
+    in_block = {
+        row["id"]: row["arms"]["memory"].get("episodes_in_block", []) for row in result["questions"]
+    }
     questions = {question.id: question for question in load(DATA_DIR / DATA_FILE)}
     replay = CachedModels(Offline(), CACHE_DIR / "replies.sqlite")
     marker = CachedModels(Remote(), CACHE_DIR / "replies.sqlite")
@@ -169,15 +195,9 @@ def main(argv: Sequence[str] | None = None) -> Path:
             question = questions[question_id]
             namespace = ("bench", "lme_s", run_id, question.id)
             sessions = {fact_source(session.id): session.id for session in question.sessions}
+            sessions |= {episode_key(session.id): session.id for session in question.sessions}
             for session in question.sessions:
-                source = fact_source(session.id) if config.fact_sources else ""
-                memory.capture(
-                    prompts.session_text(session),
-                    namespace,
-                    extractor(replay, config.variant, source, []),
-                    actor=ACTOR,
-                    observed_at=session.date,
-                )
+                capture_session(memory, namespace, session, replay, config, [])
             store = stores[config.memory_ranking]
             facts = store.list(namespace, Filter(kind=RECALLED_KINDS), limit=10_000)
             marked = _mark(marker, question.question, question.answer, facts)
@@ -205,6 +225,11 @@ def main(argv: Sequence[str] | None = None) -> Path:
                     "best_rank": ranks,
                     "cause": cause,
                     "present": present,
+                    "evidence_in_block": (
+                        None
+                        if config.episodes == "none"
+                        else evidence_in_block(question, in_block[question.id])
+                    ),
                 }
             )
             empty(namespace)
@@ -221,6 +246,8 @@ def main(argv: Sequence[str] | None = None) -> Path:
                 "budget": config.budget,
                 "extractor": config.extractor,
                 "fact_sources": config.fact_sources,
+                "session_episodes": config.session_episodes,
+                "episodes": config.episodes,
             },
             "marker": JUDGE,
             "hotmemory_commit": commit,
@@ -241,11 +268,14 @@ def main(argv: Sequence[str] | None = None) -> Path:
 def session_of_fact(fact: Record, sessions: dict[str, str]) -> str | None:
     """Return the session id of `fact` from its sources, or None if no source names one.
 
-    `sessions` maps the result of `fact_source` to the session id.
+    `sessions` maps the results of `fact_source` and of `episode_key` to the session id. A
+    source that is an episode id matches by its key.
     """
     for source in fact.sources:
-        if source in sessions:
-            return sessions[source]
+        key = source.rpartition("/")[2].rpartition("@")[0]
+        for name in (source, key):
+            if name in sessions:
+                return sessions[name]
     return None