Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/board/ISSUES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED — a crashed consumer actor is respawned at once and the respawn is rescheduled forever (2026-10-13)

**Status:** OPEN. **Basis:** VERIFIED-IN-CODE (`crates/lance-graph-supervisor/src/supervisor.rs`, `RespawnG` arm); found by the codex review of #1536.
**Status:** RESOLVED 2026-10-13 by #1540 (merged `f2bf199`). The delayed cast is now the only respawn path and is scheduled from `ActorTerminated`; the `RespawnG` arm spawns once and does nothing if the slot is already live; the crash count is carried and restarts at 1 when the previous crash is more than five minutes old (spec §5.3); more than 10 crashes in that window escalate and the child stays down. TEST-PINNED in `tests/supervisor_respawn_backoff.rs` (one respawn per crash after its backoff; escalation; window reset), each disable-verified red. The bullets below describe the defect as found. **Was:** OPEN. **Basis:** VERIFIED-IN-CODE (`crates/lance-graph-supervisor/src/supervisor.rs`, `RespawnG` arm); found by the codex review of #1536.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep the issue open until the five-minute window is enforced

The implementation in supervisor.rs:324-330 only resets the count when the gap from the immediately preceding crash exceeds five minutes; it does not count crashes within a rolling five-minute window as required by spec §5.3. For example, with one crash per minute, the eleventh crash occurs ten minutes after the first and no five-minute interval contains more than ten crashes, yet the count reaches 11 and lines 336-345 leave the child down. The quiet-gap test does not cover this case, so marking the issue resolved and claiming escalation only for crashes “in that window” hides a remaining supervisor outage condition.

Useful? React with 👍 / 👎.


- On `ActorTerminated` the supervisor casts `RespawnG { crash_count: n }`. The arm computes the backoff delay, spawns a task that sleeps that delay and casts `RespawnG { crash_count: 0 }`, and then calls `spawn_consumer_actor` right away (the comment says the delayed cast is a retry "if we need to").
- The delayed `RespawnG` runs the same arm: another immediate spawn, another delayed cast. So the backoff never delays the first respawn, the crash count is reset to 0 on every round, and each round schedules the next one 100 ms later, whether or not the child is alive.
Expand Down
6 changes: 4 additions & 2 deletions .claude/board/LATEST_STATE.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## 2026-10-13 — main after #1505, #1506, #1507–#1509, #1514, #1516, #1518, #1520, #1523, #1524, #1525, #1526, #1529, #1530, #1531, #1534, #1536, #1537 (merged)
## 2026-10-13 — main after #1505, #1506, #1507–#1509, #1514, #1516, #1518, #1520, #1523, #1524, #1525, #1526, #1529, #1530, #1531, #1534, #1536, #1537, #1538, #1539, #1540 (merged)

- **On `main`, live:** `.cargo/config.toml` defaults to `-Ctarget-cpu=native` with `config-v3.toml` as the portable baseline and `debug = 0` everywhere (#1505); `compute_dag_kahn` checks all 256 field positions (#1506); R2IL `MIN`/`MAX` lowering onto mask-risc (#1518, the part outside glove-parity).
- **Merged then withdrawn by #1524 (D-CMR-RESET):** the class-method layer of `lance-graph-glove-parity` (`method`, `call`, `decl`, `gremlin`, `awareness`; #1507, #1508, #1509, #1514, #1516, #1518, #1520). Prior state: `backup/session-011BMJ-2026-10-12`. Inventory: entry `2026-10-12-session-reset-inventory.md`.
Expand All @@ -8,7 +8,9 @@
- **On `main` (#1530 `c864721`, #1531 `86e3174`):** the thinking-engine, cognitive-stack and symbiont images build on `rust:1.99.0-bookworm` against ndarray `master`; the thinking-engine images build v4 with `--target <host triple>`.
- **On `main` (#1534, merged `0a6708b`):** `lance-graph-r2il` is a workspace member; r2il, `lance-graph-nal` and `r2il-mask-abi-probe` take OGAR by git `main`, so the graph holds one `ogar-loco` and one `lance-graph-contract` (D-FOLD-CORE-a).
- **On `main` (#1537, merged `befbf1d`):** core `lance-graph` depends on mask-risc, Quack and R2IL by default and exposes them as `lance_graph::fold` (D-FOLD-CORE-b). NAL stays in `crates/lance-graph-nal` (D-FOLD-CORE-c dropped).
- **Docs only (#1536, merged `2b28a19`):** outer-membrane graph lab, epoch-indexed temporal graph agenda, and the ONA → NAL/R2IL fold-method program (W3–W9). Proposals; nothing in them is built by that PR. The supervisor's restart backoff is not applied (`ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED`).
- **Docs only (#1536, merged `2b28a19`):** outer-membrane graph lab, epoch-indexed temporal graph agenda, and the ONA → NAL/R2IL fold-method program (W3–W9). Proposals; nothing in them is built by that PR.
- **On `main` (#1538, merged `4ceaab3`):** D-NALP-3, `nal::Revision`: evidence revision as one grouped fold over stored `w`/`f`, refusing overlapping stamps and evidence without provenance. Where ONA saturates mid-fold and its answer depends on arrival order, the fold keeps the exact sum (pinned against the ONA oracle).
- **On `main` (#1540, merged `f2bf199`):** the supervisor applies its restart backoff; `ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED` resolved. #1539 (`3635109`) corrected #1536's docs and board.
- **CI (#1526, merged `abc0a1f`):** the native default plus `rust-cache` reused AVX-512 artifacts on runners without them (SIGILL in `dir-sim log in Lance`). `native` is now written out as each runner's exact feature list, and the cache is keyed to it.
- **Rules carried from this session (operator):** classid-dependent methods go through the existing `hotplug.rs` plug-and-play; NAL/NARS operations are global core methods (no classid check, no plug, no `BeliefClass`); internal pins are forbidden (the v3 pins in the Dockerfiles and wheel build were replaced in #1529).

Expand Down
14 changes: 14 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
## 2026-10-13 — lance-graph PR #1540 (merged `f2bf199`, branch `ccr-9f3b8075-74u47l-supervisor-backoff`, head `dd3ab5d`, 5 commits) — supervisor: apply the restart backoff

- **Added:** `ActorTerminated` computes the crash count (carried from the slot; restarts at 1 when the previous crash is more than `CRASH_WINDOW` = 5 min old), escalates past 10 crashes, otherwise sleeps `backoff_delay(n) = min(100 ms · 2^n, 30 s)` and casts the only `RespawnG`. The arm spawns once, skips a slot that is already live, and stores the count and crash time on the new slot. `tests/supervisor_respawn_backoff.rs` (paused tokio time): one respawn per crash after its backoff, crash 11 stays down, the count restarts after a quiet window; each guard disable-verified red.
- **Locked:** before, the arm spawned immediately and also scheduled a `RespawnG { crash_count: 0 }` that re-ran itself every 100 ms forever, so neither backoff nor escalation could take effect. The crash time uses `tokio::time::Instant` (a `std` instant ignores paused time, which hid the window bug from the test).
- **Deferred / open:** none; `ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED` resolved.
- **Confidence:** high; three tests, disable runs red, CI green.

## 2026-10-13 — lance-graph PR #1538 (merged `4ceaab3`, branch `ccr-9f3b8075-74u47l-nal-w3`, head `12f3291`, 5 commits) — D-NALP-3: NAL revision as one grouped fold

- **Added:** `nal::Revision` in `crates/lance-graph-nal`: groups evidence by the class its stamps define and folds stored `w`/`f` with `GroupCrossPowerSumsI32`; refuses overlapping stamps and evidence without provenance. `tests/w3.rs`, 11 tests against a scalar oracle and ONA pinned at `dc4efd0`, and the revision points OpenNARS printed for `revision_points.nal`.
- **Locked:** the fold computes the exact `w = Σwᵢ` and caps only the result. ONA caps pairwise, so once evidence saturates its answer depends on arrival order (measured on one fixture: 104 positive-first, 151 negative-first; the fold gives 128 either way). Documented as a divergence and pinned.
- **Deferred / open:** W4–W9 of the ONA program (per-row truth laws, inference over populations, the `NAL::narsese()` surface).
- **Confidence:** high; disable runs red, oracle parity up to ONA's mid-fold saturation.

## 2026-10-13 — lance-graph PR #1536 (merged `2b28a19`, branch `docs/outer-membrane-graph-lab-20261013`, head `93b0993`, 5 commits) — docs: outer membrane graph lab, epoch-indexed temporal graph agenda, ONA → NAL/R2IL fold-method program

- **Added:** three proposal documents under `docs/architecture/`: `OUTER_MEMBRANE_GRAPH_LAB_VERTICAL_SLICE.md` (ownership rule, callcenter/supervisor evidence table, a falsifiable two-node one-edge vertical slice), `EPOCH_TEMPORAL_GRAPH_RESEARCH_AGENDA.md` (E01–E20 exploration questions, three time axes, T0–T6 sequence), `ONA_NAL_R2IL_FOLD_METHOD_ASSIMILATION.md` (ONA source map, W3–W9 coding waves, falsifiers). No code.
Expand Down
2 changes: 1 addition & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Plan: `.claude/plans/nal-polyfill-v1.md`. Template: D-SQLP. No opcode, tenant or
| **D-NALP-0** | Plan, board rows | In PR | docs only |
| **D-NALP-1** | Statement layout + declared ClassView reading; additive `WideFieldMask` projection path on `ClassView`; binder's mask; term width decision | Partial: #1525 merged `08635ed` (NodeGuid reading `nal-statement-v1`, term width u16, aperture selection); ClassView wide projection and binder mask Queued | only the mask's lanes are read; undeclared reading and masked-out field refuse; Small = Wide answers |
| **D-NALP-2** | Query procedures: select, truth threshold, choice by confidence, support count | Shipped #1525 `08635ed` (`crates/lance-graph-nal`: NodeGuid key, byte apertures; 7 oracle tests, 7 disable runs red; 19–248 ns at 1,048,576 rows); support count and argmax not built | equal to a row-at-a-time oracle; one-pattern threshold disable red; no new opcode |
| **D-NALP-3** | Revision as `GroupCrossPowerSumsI32` over stored `w`/`f` | In PR (`nal::Revision`, key = evidence class from the stamps; refuses overlap and missing provenance; 10 tests, 7 disable runs red) | equal to exact integer revision; OpenNARS revision points; duplicate evidence counts once |
| **D-NALP-3** | Revision as `GroupCrossPowerSumsI32` over stored `w`/`f` | Shipped #1538 `4ceaab3` (`nal::Revision`, key = evidence class from the stamps; refuses overlap and missing provenance; 11 tests, 7 disable runs red; ONA's mid-fold saturation is a documented divergence, the fold keeps the exact sum) | equal to exact integer revision; OpenNARS revision points; duplicate evidence counts once |
| **D-NALP-4** | Per-row truth law into a lane: `MapPairU8` vs fixed-point multiply-high | Queued | insufficiency proof; per-row parity with `isa::truth`, error in codes; one shape also serves SQL nullable arithmetic |
| **D-NALP-5** | Deduction / induction / abduction / analogy over populations | Queued (after D-NALP-4) | equals pairwise scalar rule; abduction direction test red for the reversed statement |
| **D-NALP-6** | `lance-graph-nal`: typed `nal::{Operation}` surface (`NalOp`), parser (tokens only), binder over the reverse label index, `NAL::narsese()`, typed refusals | Queued | text and typed calls give byte-identical `Program`s; no string in the plan, rows or wire; ambiguous / unknown label refuses; lowers from the bound plan; no fallback; no `Program` outside procedures |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ The existing [`ExternalMembrane`](../../crates/lance-graph-callcenter/src/lance_
| Capability | Current evidence | Confidence / gap |
|---|---|---|
| Per-consumer actor slots, health | `supervisor.rs` | Implemented skeleton; confirm runtime integration before claiming production behavior |
| Restart backoff | `RespawnG` arm in `supervisor.rs` | **Not applied.** The arm schedules a delayed `RespawnG` with `crash_count: 0` and then calls `spawn_consumer_actor` immediately; the delayed message repeats both, so respawns start without waiting and keep being rescheduled every 100 ms (`ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED`) |
| Restart backoff | `RespawnG` arm in `supervisor.rs` | Applied (#1540). On `ActorTerminated` the supervisor waits `min(100 ms · 2^n, 30 s)` and then sends the only `RespawnG`; the count is carried to the new child, restarts at 1 after five quiet minutes, and more than 10 crashes in five minutes leave the child down (`ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED`, resolved) |
| Default mailbox target `1024` / per-module override | `DEFAULT_MAILBOX_CAPACITY`, `ModuleEntry.mailbox_capacity` | **Declared**, but `spawn_consumer_actor` currently calls `Actor::spawn_linked` without applying that capacity |
| Normal `DispatchToG` forwarding | `supervisor.rs::handle` | **Not implemented** for non-`Health` envelopes: returns `DispatchNotImplemented` |
| Inbound `DrainTask` queue | `callcenter/src/drain.rs` | **Unbounded** Tokio mpsc in realtime build, even though child mailbox backpressure is a design goal |
Expand Down
Loading