Orchestrator: gracefully gate components the platform can't recover

docs/src/design/orchestrator/orchestrator-model.md

@@ -65,12 +65,17 @@ verification.
 
 ```
 Required   — attempt recovery; re-walk the chain after `Restored`; latch
-             `Locked` if the retry cap is reached
+             `Locked` once recovery is exhausted
 Isolable   — hold the component in reset; advance past it; continue the walk
 Cascading  — same as Isolable, and additionally hold in reset any component
              whose `depends_on` names this one
 ```
 
+Recovery is *exhausted* either when the retry cap is reached or when the
+platform reports `RecoveryUnavailable` — both run the same policy branch, so a
+platform that is out of recovery images gates an `Isolable` component rather
+than halting the platform.
+
 ### `RegionId`
 
 An opaque `u8` that groups components into a *recovery region*. Components
@@ -361,7 +366,10 @@ names **what** must happen to **which** component; the platform decides **how**.
 For example, `Effect::RecoverComponent(id)` says only "recover this component" —
 whether that resolves to a golden-image restore, an A/B slot swap, a streamed
 image, or a vendor-specific scheme is a platform/configuration decision, never
-encoded in the core.
+encoded in the core. The platform reports the outcome as an event: `Restored`
+once it has swapped in an untried image, `RecoveryUnavailable` when it has none
+left. Both are verdicts; `EffectError` stays reserved for a genuine actuation
+fault, which fails closed to `Locked` regardless of policy.
 
 The core never reads flash, never checks signatures, never observes reset lines.
 It only emits descriptions. The complete split:

services/orchestrator/sm/src/lib.rs

@@ -424,6 +424,42 @@ impl<const N: usize, const E: usize> Rot<N, E> {
         }
     }
 
+    /// Whether a `VerificationPassed(id)` may release `id`. Two conditions, and
+    /// both release sites check them through here so they cannot drift apart:
+    ///
+    /// - `id` is the component currently under verification (`chain[cursor]`,
+    ///   whose `VerifyFirmware` was just emitted). A verdict for any other id is
+    ///   stale or out-of-turn (cf. INV9 for `ComponentReady`).
+    /// - `id` is not gated. Gating does not move the cursor off the component,
+    ///   so a component isolated *while its verification was in flight* is still
+    ///   `chain[cursor]` when that verdict lands — and a stale pass must not
+    ///   undo the `AssertReset` that just held it (INV8).
+    fn may_release(&self, id: ComponentId) -> bool {
+        self.chain.get(self.cursor as usize).map(|(c, _)| c) == Some(&id) && !self.is_gated(id)
+    }
+
+    /// Recovery of `target` has run out of options. Gate it per policy:
+    /// `Isolable`/`Cascading` are skipped and the walk continues, while
+    /// `Required` (or an unknown id) reports the component and latches
+    /// [`State::Locked`]. Both exhaustion paths — the retry cap
+    /// ([`Event::Restored`] past `max_retry`) and the platform's
+    /// [`Event::RecoveryUnavailable`] — share this so they cannot diverge.
+    fn exhaust_recovery(&mut self, ctx: &mut Sink<E>, target: ComponentId) -> Outcome {
+        match self.gate_by_policy(ctx, target) {
+            Gating::Gated => {
+                self.clear_retry(target);
+                Outcome::Transition(State::PreSupervision)
+            }
+            // The report precedes the internal `Emit`, so it is actuated before
+            // the machine moves toward `Locked`.
+            Gating::NotGated => {
+                ctx.emit(Effect::ReportRecoveryFailed(target));
+                ctx.emit(Effect::Emit(Event::RecoveryFailed));
+                Outcome::Handled
+            }
+        }
+    }
+
     /// Shared `CorruptionDetected` handling, called from both `PreSupervision`
     /// (directly) and `SupervisingPlatform` (via its superstate handler).
     /// Delegates the policy interpretation to [`gate_by_policy`](Self::gate_by_policy)
@@ -498,13 +534,9 @@ impl<const N: usize, const E: usize> Rot<N, E> {
             // Cursor walk via Outcome::Handled — a self-transition would reset cursor.
             State::PreSupervision => match event {
                 Event::VerificationPassed(id) => {
-                    // Only the component currently under verification
-                    // (`chain[cursor]`, whose `VerifyFirmware` was just emitted)
-                    // may be released. A verdict for any other id is stale or
-                    // out-of-turn and is dropped, so a misordered or hostile
-                    // report cannot release an unverified component (cf. INV9
-                    // for `ComponentReady`).
-                    if self.chain.get(self.cursor as usize).map(|(c, _)| c) != Some(id) {
+                    // A misordered, stale, or hostile verdict cannot release an
+                    // unverified or already-isolated component.
+                    if !self.may_release(*id) {
                         return Outcome::Handled;
                     }
                     // The component passed its check — it has recovered, so its
@@ -596,11 +628,8 @@ impl<const N: usize, const E: usize> Rot<N, E> {
                     }
                 }
                 Event::VerificationPassed(id) => {
-                    // Only the component currently under verification
-                    // (`chain[cursor]`) may be released; a verdict for any other
-                    // id is stale or out-of-turn and is dropped (cf. INV9 for
-                    // `ComponentReady`).
-                    if self.chain.get(self.cursor as usize).map(|(c, _)| c) != Some(id) {
+                    // Same guard as the `PreSupervision` release site.
+                    if !self.may_release(*id) {
                         return Outcome::Handled;
                     }
                     // The component passed its check — it has recovered, so its
@@ -707,26 +736,19 @@ impl<const N: usize, const E: usize> Rot<N, E> {
                     if attempts < self.max_retry {
                         Outcome::Transition(State::PreSupervision)
                     } else {
-                        // Retries exhausted: gate via the same `gate_by_policy`
-                        // the runtime-corruption path uses, so the two can never
-                        // disagree. Gated → continue the walk; NotGated
-                        // (Required/unknown) → lock down.
-                        match self.gate_by_policy(ctx, failed) {
-                            Gating::Gated => {
-                                self.clear_retry(failed);
-                                Outcome::Transition(State::PreSupervision)
-                            }
-                            // `Required`, or an unknown/missing id: report the
-                            // component that forced the halt, then lock down.
-                            // The report precedes the internal `Emit`, so it is
-                            // actuated before the machine moves toward `Locked`.
-                            Gating::NotGated => {
-                                ctx.emit(Effect::ReportRecoveryFailed(failed));
-                                ctx.emit(Effect::Emit(Event::RecoveryFailed));
-                                Outcome::Handled
-                            }
-                        }
+                        self.exhaust_recovery(ctx, failed)
+                    }
+                }
+                Event::RecoveryUnavailable(id) => {
+                    // Same target guard as `Restored`: a verdict for a
+                    // component other than the one under recovery is dropped.
+                    if *id != failed {
+                        return Outcome::Handled;
                     }
+                    // No `bump_retry`: the platform is authoritative about being
+                    // out of sources, so this short-circuits the cap rather than
+                    // waiting for it to run out on non-progressing restores.
+                    self.exhaust_recovery(ctx, failed)
                 }
                 Event::RecoveryFailed => Outcome::Transition(State::Locked),
                 _ => Outcome::Super,
@@ -918,6 +940,13 @@ pub struct EffectError;
 ///   `VerifyFirmware` covers code that cannot run or rewrite its own flash
 ///   between the check and the release. A reset that merely pulses would let a
 ///   component resume before verification and void that guarantee.
+/// - **`EffectError` means actuation failed, not "policy says no".** It is
+///   fail-closed: it latches [`State::Locked`] from any state. In particular,
+///   an [`Effect::RecoverComponent`] with no untried recovery source left is
+///   *not* an error — the driver returns `Ok` and reports
+///   [`Event::RecoveryUnavailable`], which routes through [`FailurePolicy`].
+///   Returning `Err` there locks the whole platform down for a component the
+///   board may have classified `Isolable`.
 /// - **A failed [`Effect::LatchLockdown`] is a hard fault.** Lockdown is the top
 ///   of the escalation ladder — the core has nothing stronger to emit and
 ///   will *believe* it is `Locked`. The driver must treat that failure as

services/orchestrator/sm/src/model.rs

@@ -43,9 +43,11 @@ pub enum ComponentKind {
     Passive,
 }
 
-/// Recovery-failure classification: what the machine does once a required
-/// component's restore attempts are **exhausted** (its per-component retry
-/// count reaches `max_retry`). Every verification or corruption failure enters
+/// Recovery-failure classification: what the machine does once a component's
+/// recovery is **exhausted** — either its per-component retry count reaches
+/// `max_retry`, or the platform reports [`Event::RecoveryUnavailable`] because
+/// it has no remaining recovery source. Every verification or corruption
+/// failure enters
 /// [`State::Recovering`] and is retried first, regardless of this
 /// classification — CSA's "recover first" principle. This value is consulted
 /// only after retries are exhausted.
@@ -213,6 +215,11 @@ pub enum Event {
     CorruptionDetected(ComponentId),
     /// This component has been restored from its configured recovery source.
     Restored(ComponentId),
+    /// The platform driver is out of recovery sources for this component.
+    /// Routed through [`FailurePolicy`] (gate, not the fail-closed
+    /// [`Event::EffectFailed`] lockdown). Authoritative: exhausts recovery at
+    /// once, bypassing the retry cap.
+    RecoveryUnavailable(ComponentId),
     /// A required component's recovery was exhausted.
     RecoveryFailed,
     /// The platform driver's boot-progress watchdog fired: `id` did not report its
@@ -261,6 +268,7 @@ impl Event {
             | Event::BootConfirmed(id)
             | Event::CorruptionDetected(id)
             | Event::Restored(id)
+            | Event::RecoveryUnavailable(id)
             | Event::Timeout(id) => Some(*id),
             Event::PowerGood(_)
             | Event::AttestationChallenge
@@ -307,6 +315,14 @@ pub enum Effect {
     /// per configuration policy. The core only names the component to
     /// recover; it does not encode how recovery is performed.
     ///
+    /// Two-phase: this starts recovery, and the driver later reports the
+    /// verdict as an event — [`Event::Restored`] once it has swapped in an
+    /// untried image, or [`Event::RecoveryUnavailable`] when it has none left.
+    /// "Out of images" is a verdict, not an actuation failure: returning
+    /// `EffectError` for it would latch [`State::Locked`] regardless of the
+    /// component's [`FailurePolicy`]. Reserve `EffectError` for a genuine swap
+    /// fault (bus error, hardware fault).
+    ///
     /// `attempt` is this component's consecutive-recovery count (0 on the first
     /// attempt), taken straight from the core's own retry counter — the same
     /// value the retry cap is measured against. It rides on the effect so the