Fix #774: monitor can assign a primary an unreachable goal state during failover

[dimitri]

Problem

Two rules in group_state_machine.c could combine to assign the primary a goal state its own KeeperFSM[] (src/bin/pg_autoctl/fsm.c) has no transition path to reach from its current state, fataling the keeper forever with "does not know how to reach state ... from ...".

• Rule 1 (ProceedGroupStateForPrimaryNode, "no healthy standby in the quorum"): reassigns the primary to wait_primary whenever secondaryQuorumNodesCount == 0. That count drops to zero the instant a failover candidate leaves SECONDARY (e.g. converges to prepare_promotion) — even though the candidate is the failover in progress, not a lost standby.
• Rule 2 ("prepare_promotion -> stop_replication"): assigns the primary demote_timeout as soon as the candidate reports prepare_promotion, unconditionally. demote_timeout has a real KeeperFSM[] edge from primary/join_primary/apply_settings/draining — but not from wait_primary. If Rule 1 already reassigned the primary to wait_primary, Rule 2 hands out an unreachable demote_timeout.

Fix

• Rule 1 now skips its reassignment when IsFailoverInProgress() is true (the candidate already being in prepare_promotion counts).
• Rule 2 now only assigns demote_timeout when the primary's reported state is actually one KeeperFSM[] has an edge to (primary, join_primary, apply_settings, draining), deferring otherwise — the next node_active() round re-evaluates.

Testing

Added src/monitor/sql/failover_candidate_leaves_secondary.sql, a pg_regress test with two scenarios, each isolating one of the two guards via direct state manufacture (the same technique stale_primary_report.sql already uses for otherwise hard-to-reach states). Neither needs the isolation tester — both are pure sequencing issues inside ProceedGroupStateFromContext.

Verified via:

• Authoritative make installcheck (pg_regress in Docker) — all 13 monitor tests + isolation tests pass.
• Empirically reproduced both bugs pre-fix (temporarily reverted each guard, rebuilt, confirmed the unreachable goal state is assigned and the test fails), then confirmed both fixes independently load-bearing.

Issues

Fixes #774.

Also looked at #883 and #369 as candidates for the same root cause:

• Concurrently triggering failovers with multiple standbys causes FSM to get stuck #369: unrelated, already fixed by Fix when to allow for a failover in multiple standbys. #370 in a different code path (node_metadata.c/node_active_protocol.c, guarding failover-source node selection) — not touching this rule at all.
• Impossible / unresolveable state after crash - How to recover? #883: likely the same underlying bug (the FATAL signature and triggering event match Rule 2 exactly, and the maintainer's own closing PR Improve documentation thanks to user feedback. #909 links directly to How to recover from a current/assigned state combination without a possible state transition path? #774), but it was closed via a documentation-only recovery procedure rather than a code fix, so I can't fully confirm it's the identical code path on 1.6.3-era code. Leaving a comment on Impossible / unresolveable state after crash - How to recover? #883 pointing here rather than closing it outright.