Skip to content

fix: KRaft remove headless service from bootstrap controller list - #1063

Open
razvan wants to merge 2 commits into
mainfrom
fix/drop-headless-service-from-bootstrap
Open

razvan wants to merge 2 commits into
mainfrom
fix/drop-headless-service-from-bootstrap

Conversation

@razvan

@razvan razvan commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #1062

In KRaft mode, a controller that starts without metadata, such as a new pod after scaling up, finds the quorum leader through controller.quorum.bootstrap.servers. Without Kerberos, that list had one entry per controller role group: the role group's headless Service. That Service also publishes pods that aren't ready, so a starting controller could resolve the name to its own address. The connection to itself succeeds, so Kafka never tried another address. The pod stayed unattached, never joined the quorum, and the StatefulSet rollout stalled.

controller.quorum.bootstrap.servers now always lists the individual controller pods, as it already did for Kerberos clusters. Kafka picks a random entry for every fetch attempt, so a pod that picks itself only loses that one attempt and then reaches a voter that knows the leader.

Changes

  • kraft_controllers() returns the FQDN of every controller pod for all clusters. The Kerberos-only branch and its parameter are gone.
  • Broker and controller configs both use the new list.
  • Docs: the KRaft internal-details section describes the list and its scaling impact; the Kerberos section refers to it.
  • Changelog: the unreleased feat: add support for dynamic KRaft quorum scaling #1010 entry mentions the per-pod list.

Unchanged:

  • The quorum-manager sidecar and the preStop hook. Both read the list from controller.properties and already handled a list of pods under Kerberos.
  • The headless Service. Pod DNS names depend on it, and existing controllers must be able to reach a pod that isn't ready yet in order to add it as a voter.

Behavior change

The list now changes whenever the number of controller replicas changes, so scaling a controller role group restarts all controller and broker pods. This only affects clusters without Kerberos or OPA. Kerberos clusters already restarted because of this list, and OPA clusters because super.users lists every pod.

Tests

  • Unit tests: the list has one entry per controller pod across role groups, excludes brokers, and grows with the replica count.
  • kuttl: operations-kraft scales controllers from 3 to 5 and back. New controllers must join without manual intervention. smoke-kraft and kraft-kerberos check for regressions.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KRaft: a new controller can bootstrap against itself and never join the quorum

1 participant