Orchestrator: gracefully gate components the platform can't recover - #445
Orchestrator: gracefully gate components the platform can't recover#445rusty1968 wants to merge 4 commits into
Conversation
|
The new /// A component gated while its verification is in flight must not park the
/// walk: the rest of the chain still verifies and the platform boots degraded.
#[test]
fn walk_completes_past_component_gated_under_verification() {
let (effects, state) = drive(
chain(&[
(C0, ComponentAttrs::passive_required()),
(C1, ComponentAttrs::passive_isolable()),
(C2, ComponentAttrs::passive_required()),
]),
&[
BOOT,
Event::VerificationPassed(C0), // → VerifyFirmware(C1) in flight
Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
Event::VerificationPassed(C1), // the in-flight verdict, now stale
Event::VerificationPassed(C2),
],
);
assert!(effects.contains(&Effect::VerifyFirmware(C2))); // fails: never emitted
assert_eq!(state, State::Ready); // fails: parked in PreSupervision
}
/// Same hole in `AwaitingReady`: `ComponentReady` must finish the walk, not
/// settle in `AwaitingReady(None)` forever.
#[test]
fn awaiting_ready_completes_past_component_gated_under_verification() {
let (effects, state) = drive(
chain(&[
(C0, ComponentAttrs::active_required()),
(C1, ComponentAttrs::passive_isolable()),
]),
&[
BOOT,
Event::VerificationPassed(C0), // → AwaitingReady(C0); VerifyFirmware(C1) in flight
Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
Event::VerificationPassed(C1), // the in-flight verdict, now stale
Event::ComponentReady(C0),
],
);
assert!(!effects.contains(&Effect::ReleaseReset(C1)));
assert_eq!(state, State::Ready); // fails: stuck in AwaitingReady(None)
}The verdict for a gated component might never arrive at all, so the re-arm belongs at the gating site, not the drop site: advance the walk when gating lands on |
|
orchestrator-machine.md seems to need updating (e.g. RecoveryUnavailable) |
chrysh
left a comment
There was a problem hiding this comment.
may_release makes the stale pass inert, but the stale fail for the same gated component is still honored: VerificationFailed(C1) right after CorruptionDetected(C1) enters Recovering(C1) and emits RecoverComponent for a component policy just took out of service. Entering recovery also runs quiesce_all, so every healthy live component is reset for a device the re-walk will skip anyway. The end state self-heals — the re-walk skips the gated component — but we rewrite the flash of an isolated device and re-boot the whole platform for nothing. And with a driver that answers RecoverComponent with Err, this stale fail escalates all the way to Locked.
Shouldn't a verdict for a gated component be inert in both polarities, not just the pass? Same hole in the AwaitingReady arm.
This fails on the current branch:
/// A verdict for a gated component is inert regardless of polarity: the stale
/// `VerificationFailed` must not start a recovery episode for a component
/// policy just took out of service.
#[test]
fn stale_verdict_never_recovers_an_isolated_component() {
let (effects, state) = drive(
chain(&[
(C0, ComponentAttrs::passive_required()),
(C1, ComponentAttrs::passive_isolable()),
]),
&[
BOOT,
Event::VerificationPassed(C0), // → VerifyFirmware(C1) in flight
Event::CorruptionDetected(C1), // → AssertReset(C1), isolated
Event::VerificationFailed(C1), // the in-flight verdict, now stale
],
);
assert_ne!(state, State::Recovering(C1)); // fails: episode started
assert!(!effects
.iter()
.any(|e| matches!(e, Effect::RecoverComponent { id, .. } if *id == C1)));
// Collateral: entering `Recovering` quiesces the already-released C0.
assert!(!effects.contains(&Effect::AssertReset(C0)));
}
The intent of this PR is to allow the platform to boot degraded when a non-required (Isolable/Cascading) component cannot be recovered, instead of hard locking.