Skip to content

Orchestrator: gracefully gate components the platform can't recover - #445

Open
rusty1968 wants to merge 4 commits into
OpenPRoT:mainfrom
rusty1968:recovery-unavailable
Open

Orchestrator: gracefully gate components the platform can't recover#445
rusty1968 wants to merge 4 commits into
OpenPRoT:mainfrom
rusty1968:recovery-unavailable

Conversation

@rusty1968

Copy link
Copy Markdown
Collaborator

The intent of this PR is to allow the platform to boot degraded when a non-required (Isolable/Cascading) component cannot be recovered, instead of hard locking.

@rusty1968
rusty1968 requested review from chrysh and leongross August 26, 2026 21:02
@rusty1968
rusty1968 marked this pull request as ready for review August 27, 2026 00:10
@chrysh

chrysh commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

may_release fixes the INV8 hole, but it also removes the only thing that kept the walk alive when chain[cursor] is gated mid-verification. handle_corruption's Gated arm returns Handled with the cursor untouched, so once the stale pass is dropped, no event can advance the walk anymore: PreSupervision has no supervisor, so Timeout is discarded, and in AwaitingReady the later ComponentReady sees cursor < chain.len() and settles in AwaitingReady(None). The platform never reaches Ready. That's a boot halted on a component the board classified Isolable — exactly what this PR is supposed to prevent.

The new stale_verdict_* tests don't catch it because they bind _state. These two fail on the current branch, the other 87 pass:

/// A component gated while its verification is in flight must not park the
/// walk: the rest of the chain still verifies and the platform boots degraded.
#[test]
fn walk_completes_past_component_gated_under_verification() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::passive_required()),
            (C1, ComponentAttrs::passive_isolable()),
            (C2, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // → VerifyFirmware(C1) in flight
            Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
            Event::VerificationPassed(C1), // the in-flight verdict, now stale
            Event::VerificationPassed(C2),
        ],
    );
    assert!(effects.contains(&Effect::VerifyFirmware(C2))); // fails: never emitted
    assert_eq!(state, State::Ready); // fails: parked in PreSupervision
}

/// Same hole in `AwaitingReady`: `ComponentReady` must finish the walk, not
/// settle in `AwaitingReady(None)` forever.
#[test]
fn awaiting_ready_completes_past_component_gated_under_verification() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::active_required()),
            (C1, ComponentAttrs::passive_isolable()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // → AwaitingReady(C0); VerifyFirmware(C1) in flight
            Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
            Event::VerificationPassed(C1), // the in-flight verdict, now stale
            Event::ComponentReady(C0),
        ],
    );
    assert!(!effects.contains(&Effect::ReleaseReset(C1)));
    assert_eq!(state, State::Ready); // fails: stuck in AwaitingReady(None)
}

The verdict for a gated component might never arrive at all, so the re-arm belongs at the gating site, not the drop site: advance the walk when gating lands on chain[cursor]. And AwaitingReady's done-check needs to be "no ungated component at or after cursor" instead of cursor < chain.len().

@chrysh

chrysh commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

orchestrator-machine.md seems to need updating (e.g. RecoveryUnavailable)

@chrysh chrysh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may_release makes the stale pass inert, but the stale fail for the same gated component is still honored: VerificationFailed(C1) right after CorruptionDetected(C1) enters Recovering(C1) and emits RecoverComponent for a component policy just took out of service. Entering recovery also runs quiesce_all, so every healthy live component is reset for a device the re-walk will skip anyway. The end state self-heals — the re-walk skips the gated component — but we rewrite the flash of an isolated device and re-boot the whole platform for nothing. And with a driver that answers RecoverComponent with Err, this stale fail escalates all the way to Locked.

Shouldn't a verdict for a gated component be inert in both polarities, not just the pass? Same hole in the AwaitingReady arm.

This fails on the current branch:

/// A verdict for a gated component is inert regardless of polarity: the stale
/// `VerificationFailed` must not start a recovery episode for a component
/// policy just took out of service.
#[test]
fn stale_verdict_never_recovers_an_isolated_component() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::passive_required()),
            (C1, ComponentAttrs::passive_isolable()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // → VerifyFirmware(C1) in flight
            Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
            Event::VerificationFailed(C1), // the in-flight verdict, now stale
        ],
    );
    assert_ne!(state, State::Recovering(C1)); // fails: episode started
    assert!(!effects
        .iter()
        .any(|e| matches!(e, Effect::RecoverComponent { id, .. } if *id == C1)));
    // Collateral: entering `Recovering` quiesces the already-released C0.
    assert!(!effects.contains(&Effect::AssertReset(C0)));
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants