Skip to content

fix: persist KRaft bootstrap node id in cluster status - #1061

Open
razvan wants to merge 2 commits into
mainfrom
fix/bootstrap-quorum
Open

razvan wants to merge 2 commits into
mainfrom
fix/bootstrap-quorum

Conversation

@razvan

@razvan razvan commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #1060

The bootstrap controller is now chosen once and recorded in the KafkaCluster status:

  • New status field status.kraftBootstrapControllerNodeId. At the first reconcile the operator sets it to the lowest controller node.id, and never changes it afterwards. If the cached KafkaCluster has no value yet, the operator reads the live object before choosing, so an out-of-date cache can't lead to a different choice.
  • Written before the StatefulSets. A newly chosen value is saved to the status before any controller StatefulSet is applied. The end-of-reconcile status update always includes the field, so server-side apply doesn't remove it.
  • Controller start command. Only the recorded controller formats with --standalone. All other controllers, including those in role groups added later, format with --no-initial-controllers and join through the quorum-manager sidecar. Adding a controller role group no longer restarts the existing controllers.
  • Refactoring. The node ID calculation, including the cluster-wide hash-collision check, moved from pod_descriptors() into a reusable function, pod_node_ids(). Validation uses it to pick the bootstrap controller.

Behavior changes

  • In KRaft mode, a node ID hash collision between role groups is now reported during validation rather than later in the build step. The error is the same.
  • The CRD has a new optional status field.

Known limitations

  • If the status is lost, the operator chooses the bootstrap controller again from the current spec.
  • If the bootstrap controller itself loses its PVC, it can still bootstrap a new quorum. This limitation is already documented.

Tests

  • Unit tests:
    • the stored bootstrap controller wins over a lower node.id added later;
    • with nothing stored, the lowest node.id is chosen;
    • ZooKeeper mode gets no bootstrap controller;
    • the controller start command uses the stored id, and never uses --standalone when none is stored.
  • kuttl (operations-kraft): a new step adds a controller role group (extra) that hashes lower than default. It checks that the quorum has 4 voters when queried through both role groups, and that status.kraftBootstrapControllerNodeId still points to default-0. The next step removes extra again, which also covers removing a voter.

Docs

  • kraft-controller.adoc describes how the bootstrap controller is chosen and recorded.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@razvan razvan self-assigned this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KRaft: adding a controller role group can bootstrap a second quorum

1 participant