Skip to content

Add 'pg_autoctl inspect fsm mermaid' phase diagrams for the keeper FSM#1164

Merged
dimitri merged 4 commits into
mainfrom
feature/fsm-mermaid-phases
Jul 26, 2026
Merged

Add 'pg_autoctl inspect fsm mermaid' phase diagrams for the keeper FSM#1164
dimitri merged 4 commits into
mainfrom
feature/fsm-mermaid-phases

Conversation

@dimitri

@dimitri dimitri commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

What

Replaces the hand-maintained Graphviz FSM diagram (docs/fsm.png, regenerated by hand via pg_autoctl inspect fsm gv | dot, last touched in 2021) with five smaller diagrams generated directly from KeeperFSM[] (src/bin/pg_autoctl/fsm.c), so they can never drift out of sync with the actual state machine.

The full FSM is 20 states and 77 transitions — legible as a reference table, but too dense to read at a glance as a single diagram. New commands split it by phase of a node's life:

pg_autoctl inspect fsm mermaid init
pg_autoctl inspect fsm mermaid steady-state
pg_autoctl inspect fsm mermaid failover
pg_autoctl inspect fsm mermaid maintenance
pg_autoctl inspect fsm mermaid removal

Each prints a Mermaid stateDiagram-v2 program, embedded directly in docs/failover-state-machine.rst via sphinxcontrib.mermaid (already a docs dependency).

Design notes

  • Phase classification is a small, hand-curated (current, assigned) -> phase table in fsm_mermaid.c, verified at runtime: every transition lands in exactly one diagram (a log_warn fires if a future KeeperFSM[] change ever adds an edge the table doesn't cover — nothing silently disappears).
  • Citus doesn't get a 6th diagram. Verified all 15 Citus-specific (current, assigned) pairs already exist under NODE_KIND_ANY — a separate Citus diagram would be graph-isomorphic to "failover," so it's called out as a note there instead.
  • join_primary is excluded. It's long deprecated and no longer assigned to any node (see the existing "Join_primary" doc section); KeeperFSM[] still carries its 9 transitions for backward compatibility with on-disk state from old versions, but showing them just adds noise. 77 edges -> 68 shown.
  • States shared across diagrams are explicit, not hidden. Every diagram annotates each of its states with note right of X naming every other phase that state also appears in.
  • Colour scheme via Mermaid classDef/class, consistent across all five: grey (lifecycle/admin), blue (primary-like), green (secondary-like), red (primary going away), purple (maintenance), amber (multi-standby election machinery).
  • pg_autoctl inspect fsm gv (Graphviz) is left in place for anyone who wants the single full graph (join_primary included) for their own tooling — it's just no longer the documented way to visualize the FSM.

Verification

  • Native build (make -C src/bin/pg_autoctl) clean, no warnings.
  • All 5 inspect fsm mermaid <phase> commands run and produce valid Mermaid syntax — caught and fixed one real bug along the way: a colon inside note text collides with Mermaid's note right of X : text delimiter grammar (verified against mermaid@11.12.1 directly).
  • Docs built with Sphinx and the rendered HTML checked in a browser: all 5 diagrams render without errors, colours and cross-diagram notes display correctly, join_primary confirmed absent everywhere.
  • Edge-count cross-check: 68 edges total across the 5 diagrams, matching 77 real KeeperFSM[] transitions minus the 9 involving join_primary, with zero dropped and zero duplicated.

dimitri added 2 commits July 26, 2026 22:33
…nual Graphviz doc image

Splits the 77-transition keeper FSM into 5 phase diagrams (init,
steady-state, failover, maintenance, removal) generated directly from
KeeperFSM[], since the full graph is too dense to read as one image.
Citus-specific transitions are not shown separately: every one of the
15 Citus-specific (current, assigned) pairs already exists under
NODE_KIND_ANY, so a standalone Citus diagram would be graph-isomorphic
to the others.

Colour scheme groups states by role (meta/primary/secondary/demoting/
maintenance/election) via Mermaid classDef/class. States appearing in
more than one phase are annotated in each diagram via 'note right of'
naming the other phase(s), verified against a hand-curated but fully
covered (current, assigned) -> phase classification table.

Docs: docs/failover-state-machine.rst now embeds the 5 generated
diagrams (sphinxcontrib.mermaid, already a dependency) in place of the
old fsm.png/dot pipeline; pg_autoctl inspect fsm gv is left in the code
for anyone who wants the single full graph, but is no longer the
documented way to visualize the FSM.
join_primary is no longer assigned to any node (see the existing
'Join_primary' doc section) -- KeeperFSM[] still carries its 9
transitions for backward compatibility with on-disk state from old
versions, but showing them just adds noise to diagrams describing
current behaviour. Filtered in the generator itself (skips any edge
touching JOIN_PRIMARY_STATE) and removed the now-dead rows from the
phase classification table. 77 edges -> 68 edges across the five
diagrams; verified zero dropped/warned edges and zero remaining
join_primary mentions after rebuilding.
@dimitri dimitri self-assigned this Jul 26, 2026
@dimitri dimitri added the docs The documentation needs more work label Jul 26, 2026
dimitri added 2 commits July 26, 2026 23:16
…adow table

The previous commit introduced a separate FsmPhaseMap[] array duplicating
(current, assigned) pairs already present in KeeperFSM[] -- real double
maintenance: every future KeeperFSM[] edit would need a matching, easy to
forget update to a second table, with only a runtime warning (not the
compiler) catching drift, and no check at all for stale entries left
behind in FsmPhaseMap after a KeeperFSM[] row changes.

Add FsmPhase as a field directly on KeeperFSMTransition (fsm.h), and set
it on each NODE_KIND_ANY row in KeeperFSM[] (fsm.c) -- the 77 real
transitions, curated by hand once against their own transition comments,
same classification as before, just living in the one table that already
exists. Citus-specific rows and the NO_STATE sentinel are left at the
implicit zero value (FSM_PHASE_NONE): they're skipped by the mermaid
generator's existing NODE_KIND_ANY filter before phase is ever consulted.

fsm_mermaid.c now walks KeeperFSM[] directly and groups by
transition.phase, exactly the way print_fsm_for_graphviz() already walks
it to dump every row -- no separate lookup table, no separate loop. The
missing-classification safety net moves with it: a NODE_KIND_ANY row
left at FSM_PHASE_NONE still logs a warning instead of silently vanishing
from every diagram.

Verified behaviour-preserving: all 5 generated diagrams are byte-for-byte
identical to the pre-refactor output, zero warnings on any of them, and
'pg_autoctl inspect fsm gv' (which never looks at the new field) is
unaffected. citus_indent (via Docker) made no formatting changes.
@dimitri
dimitri merged commit 1730573 into main Jul 26, 2026
9 of 10 checks passed
@dimitri
dimitri deleted the feature/fsm-mermaid-phases branch July 26, 2026 23:01
dimitri added a commit that referenced this pull request Jul 27, 2026
Pre-existing missing blank line from #1164, unrelated to the #774 fix
in this branch but blocking this PR's style-check CI job since
citus_indent lints the whole tree. Verified with `make docker-check`
(clean) and `sh ci/banned.h.sh` (clean).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs The documentation needs more work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant