sn-manager: EVM preflight + auto-rollback (v2.6.1-testnet fix)#311
Merged
Merged
Conversation
Adds two symmetric branches to the sn-manager auto-updater's
checkAndUpdateCombined loop:
* Forward-block preflight — refuses to install a supernode target
tag >= v2.6.0 when the chain has the evm upgrade-module active and
~/.supernode/config.yml has no supernode.evm_key_name set. The current
binary is kept in place, the current_version in sn-manager config is
NOT touched, and ~/.sn-manager/update-blocked.log is written with
reason + supernode config mtime so a second check with unchanged mtime
is a no-op (sticky until the operator remediates).
* Auto-rollback — when the CURRENTLY-INSTALLED supernode is already
>= v2.6.0 on an evm-active chain with no evm_key_name (i.e. the node
is in a crash loop from an earlier auto-update), fetches v2.5.0-testnet
(hardcoded rollback target), activates it via versionMgr, persists
updates.current_version, writes ~/.sn-manager/rolled-back.log and a
.needs_restart marker.
Both branches only fire when the chain query succeeds. A gRPC-dial or
ModuleVersions error fails OPEN (allow) so a transient chain outage
cannot itself trigger a block or a rollback.
decidePreflight is a pure predicate over (chainHasEVM, evmKeyName,
current, target) and is covered by a table-driven test matching the
invariant table in the PR body. Operator-explicit code paths in
cmd/{start,init,use,get}.go intentionally bypass the preflight (the
operator asked for a specific version); this is documented in the
preflight comment block.
sn-manager status surfaces both markers by dumping the log contents
under 'Update Blocked:' / 'Rolled Back:' headers.
Deps: promotes google.golang.org/grpc + cosmossdk.io/x/upgrade to
direct requires to reuse upgradetypes.NewQueryClient (same call pattern
as supernode/cmd/evmigration.go).
Signed-off-by: Matee ullah Malik <mateeullahmalik@hotmail.com>
j-rafique
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an EVM-migration-aware preflight + auto-rollback branch to the sn-manager auto-updater. Ships as v2.6.1-testnet to remediate the 14-node crash-loop caused by v2.6.0-testnet on unmigrated (legacy secp256k1) fleet nodes.
Root cause / timeline
supernode.evm_key_nameset in~/.supernode/config.ymlto start on a chain with theevmupgrade module active. Testnet chain has EVM active.evm_key_name, refuses to start. sn-manager restart-loops the crashed process.Immediate mitigation (already done): v2.6.0-testnet release drafted on GitHub to block new pulls. This PR is the permanent fix.
Invariant table
Design
Two symmetric branches at the single decision point (
updater.checkAndUpdateCombined):A. Forward-block preflight — before installing the downloaded target:
supernode.config.lumera.grpc_addrforupgradetypes.ModuleVersions("evm")evm_key_nameempty → ABORTupdates.current_versionmutation~/.sn-manager/update-blocked.logwith reason + timestamp + target +supernode_config_mtimesn-manager statussurfacesUpdate Blocked: truewith reason + target + sinceB. Auto-rollback — same code path, opposite direction:
evm_key_nameempty → stuck nodev2.5.0-testnet(hardcoded rollback target) from GitHub if not already installed locallyversionMgr.InstallVersion+SetCurrentVersionupdates.current_version = "v2.5.0-testnet"~/.sn-manager/rolled-back.log+.needs_restartsn-manager statussurfacesRolled Back: truewith from/to/reason/atC. Sticky-until-config-changes — after a block, subsequent check cycles compare current supernode config mtime against the mtime recorded in the block log; if unchanged, forward-check is skipped. Once the operator edits
~/.supernode/config.yml(addingevm_key_name), the mtime advances and the marker is auto-cleared on the next allow decision.Fail-open on chain unreachability: a gRPC dial or
ModuleVersionserror returnspreflightAllowwith an INFO log. A transient chain outage must NEVER cause a block or a rollback.Predicate-refactor pattern
decidePreflight(preflightInputs) → (preflightDecision, reason)is a pure function with no I/O and no logging. The runtime helperpreflightCheckperforms the I/O then delegates todecidePreflight. This is the invariant-first-coding §Anti-pattern 6 pattern — the invariant is table-driven testable without booting a network client.Test matrix
All 9 rows of the invariant table PASS as table-driven cases in
TestDecidePreflight_Table(internal/updater/preflight_test.go).Additional coverage:
TestParseGRPCAddr— host/TLS inference for host:port, https://, http://, bareTestQueryEVMModuleActive_FailOpenOnEmpty— empty grpc_addr → error → caller must fail-openTestPerformRollback_HappyPath— pre-installed binary, symlink swap, config persist,.needs_restart, rollback logTestWriteReadBlockLogMTime— round-trip mtime, clearTestReadSupernodeEVMKeyName/TestReadSupernodeGRPCAddr— config-parse extensionsTestIsV260OrAbove— core-version comparison ignoring prerelease (v2.6.0-testnettreated as >=v2.6.0)Local verification:
Deployment plan
v2.6.1-testnetfrom master (a subsequent step, gated on this PR merge — NOT part of this PR).v2.6.0-testnetrelease stays drafted on GitHub (kept for history; no operator will ever pull it because the newer semver tag exists).v2.6.1-testneton its next 10-minute check cycle — the new sn-manager binary contains the rollback branch which will fire on the next iteration and revert supernode tov2.5.0-testnet, restoring the node to a runnable state until the operator completes their EVM migration.Post-merge validation
Operator can run on any node:
Current Version: v2.6.0-testnetandStatus: Not running (process dead)in a loop.v2.6.1-testnetpicks up: output showsCurrent Version: v2.5.0-testnet,Rolled Back: true, plus from/to/reason.v2.6.1-testnet+as latest:Update Blocked: truewith target + reason + migration doc link.Explicit notes
MIN_SNMANAGER_VERmechanism is deferred to a subsequent PR per user directive. This PR relies on sn-manager already having auto-updated to v2.6.0-testnet on the crashed nodes (which it did), so v2.6.1-testnet ships as a normal semver-forward tag and is picked up by the existing check loop.cmd/{start,init,use,get}.goare intentional — those are operator-explicit code paths (sn-manager use vX,sn-manager get vX) where the operator has consciously requested a specific version. Comment inupdater.godocuments this.AutoUpdater.checkAndUpdateCombined. The rollback branch is ALSO only reachable through this same function; no separate ticker or startup routine.Cross-references
Rollback plan (for THIS PR, not the incident rollback)
If the preflight itself misbehaves in production, DO NOT revert-and-republish v2.6.0-testnet — that is what caused the outage. Instead cut a
v2.6.2-testnetthat reverts this commit; sn-manager on all nodes will pick up the semver-forward tag and revert to the pre-preflight behavior. Rollback log format is a stable text file (line-prefixed key: value); no on-disk migration is required.