fix(supervisor): respawn a crashed consumer once, after its backoff

[AdaWorldAPI]

Fixes ISS-SUPERVISOR-RESPAWN-BACKOFF-NOT-APPLIED (filed in #1539).

The bug

In crates/lance-graph-supervisor/src/supervisor.rs, the RespawnG arm:

• spawned the dead child at once, so the backoff never delayed a respawn;
• also started a task that slept and cast RespawnG { crash_count: 0 }, which ran the same arm again. That started a chain that repeated every 100 ms for the life of the supervisor. Every crash added another chain.

A correction to the issue text: the crash count was not reset on every round. While the child is live, those chained respawns fail on the duplicate actor name in ractor's registry. What the chain did break is escalation: once crash 11 removes the slot, the next round of the chain finds the name free and respawns the child with count 0.

The fix

On ActorTerminated, the supervisor now:

1. checks the escalation threshold (more than 10 crashes means no respawn);
2. waits min(100ms * 2^n, 30s);
3. casts RespawnG with the crash count.

RespawnG spawns the child and gives the new slot that crash count. It does nothing if the slot is already live, and it never schedules itself again. The escalation log no longer names a ResetCrashCount message, which does not exist.

Escalation window (codex P1, added after review)

Spec §5.3 escalates on more than ten crashes within five minutes; the count used to be lifetime-wide. A crash more than five minutes after the previous one now restarts the count at 1. last_crash_ts records the crash time, carried through RespawnG's new crashed_at field and taken from tokio's clock. Pinned by the_count_restarts_after_a_quiet_window (written first, failed before the fix). Disable runs: forcing every crash to count from 1 fails all three tests; never resetting fails the window test.

Tests (written first, paused tokio time)

New file: tests/supervisor_respawn_backoff.rs.

Test	What it checks	On the old code
each_crash_respawns_once_after_its_backoff	Two crashes. At half the backoff the child is down; after the backoff it is live with crash count 1, then 2	FAILS: "crash 1: respawned before its 200ms backoff"
a_child_past_the_escalation_threshold_stays_down	Crashes 1–10 are respawned; crash 11 stays down for 120 s	FAILS: "crash 11 must not be respawned"

The two tests supervise different G slots, because ractor's registry is shared across the process and the tests run in parallel.

Each guard in the fix was removed in turn to confirm a test catches it:

Guard removed	Result
Escalation check	Escalation test fails
Carry the crash count (reset to 0 instead)	Both tests fail
The backoff sleep	Backoff test fails
"Slot already live" check	All tests pass. No test can catch this: ractor's registry refuses the duplicate name anyway. The check is marked defensive in its comment.

Gates

• cargo test -p lance-graph-supervisor --features supervisor: all suites pass. The new file was run three times in a row.
• cargo clippy -p lance-graph-supervisor --features supervisor --all-targets -- -D warnings: clean.
• cargo fmt --check: clean.
• The only Cargo.toml change is adding tokio's test-util feature to dev-dependencies (needed for paused time).

Board

The issue entry and the vertical-slice doc row both live on #1539, which has not merged yet. Once #1539 merges, I'll merge main into this branch and update both: the issue to RESOLVED, with the corrected "count reset" detail above, and the slice doc's backoff row to "applied".

Still open

After escalation, Health reports the child's crash count as 0 (is_live: false), because the slot is removed. That is unchanged by this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_011BMJnDBAfzcuQh5HeT39Rh

---

Generated by Claude Code

---

Generated by Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 67a3d806-7525-45f6-bb3f-da0b130ba308

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-13T18:05:12.665512Z	0c8e1d8	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c8e1d8ef1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".