feat: add support for dynamic KRaft quorum scaling

.gitignore

@@ -23,3 +23,5 @@ tilt_options.json
 .envrc
 
 .DS_Store
+
+.worktrees/

CHANGELOG.md

@@ -6,6 +6,17 @@ All notable changes to this project will be documented in this file.
 
 ### Changed
 
+- The dynamic KRaft quorum created by the operator is now scaled automatically. Previously,
+  manual intervention was needed after every scale operation.
+  This change introduces a new side-car container (`quorum-manager`) to all controller pods
+  that adds the new controller to the voter list.
+  On termination, a new `preStop` hook on the controller container (`kafka`) removes the pod from
+  the voter list before shutdown.
+  The property `controller.quorum.bootstrap.servers` now contains the headless service names
+  of all controller role groups instead of individual peer host names. This prevents the
+  restart controller from restarting all pods in the quorum when a new one is added/deleted.
+  The controller `StatefulSet` is now scaled using `OrderedReady` instead of the `Parallel` strategy
+  to ensure only one voter is added/removed at a time and thus keep the quorum healthy ([#1010]).
 - Internal operator refactoring: introduce a build() step in the reconciler that
   assembles all relevant Kubernetes resources before anything is applied ([#985]).
 - Bump stackable-operator to 0.116.0 ([#994], [#1011]).
@@ -34,12 +45,24 @@ All notable changes to this project will be documented in this file.
   See [our internal issue](https://github.com/stackabletech/hdfs-operator/issues/626) and [the fix](https://github.com/kube-rs/kube/pull/2042) for details ([#998]).
 - The operator now watches all resources that it creates and early-exits the reconcile action when the
   cluster is marked for deletion ([#1014]).
+- A KRaft cluster without a `controllers` role is now rejected during validation.
+  Previously, the operator would create a cluster consisting only of `brokers` which would never
+  become healthy ([#1010]).
+
+### Removed
+
+- BREAKING: The broker pod's separate `kcat-prober` sidecar container has been removed; its
+  `kcat`-based readiness probe now runs directly on the `kafka` container instead (`kcat` has
+  shipped in the same product image as `kafka` since #527, so the dedicated container/image was
+  no longer needed). The `kcat-prober` value is no longer accepted in a broker's
+  `logging.containers` CRD field ([#1010]).
 
 [#985]: https://github.com/stackabletech/kafka-operator/pull/985
 [#990]: https://github.com/stackabletech/kafka-operator/pull/990
 [#994]: https://github.com/stackabletech/kafka-operator/pull/994
 [#998]: https://github.com/stackabletech/kafka-operator/pull/998
 [#1000]: https://github.com/stackabletech/kafka-operator/pull/1000
+[#1010]: https://github.com/stackabletech/kafka-operator/pull/1010
 [#1011]: https://github.com/stackabletech/kafka-operator/pull/1011
 [#1014]: https://github.com/stackabletech/kafka-operator/pull/1014
 [#1017]: https://github.com/stackabletech/kafka-operator/pull/1017

docs/modules/kafka/pages/usage-guide/kraft-controller.adoc

@@ -85,33 +85,29 @@ KRaft mode requires major configuration changes compared to ZooKeeper:
 * `cluster-id`: This is set to the `metadata.name` of the KafkaCluster resource during initial formatting
 * `node.id`: This is a calculated integer, hashed from the `role` and `rolegroup` and added `replica` id.
 * `process.roles`: Will always only be `broker` or `controller`. Mixed `broker,controller` servers are not supported.
-* The operator configures a static voter list containing the controller pods. Controllers are not dynamically managed.
+* Each controller pod runs an additional `quorum-manager` sidecar container that, on startup, admits the pod into
+  the KRaft voter set.
+  On pod restart or termination, the controller is removed from the voter set by the `preStop` hook on the `kafka` container.
+* Exactly one controller (the one with the numerically lowest `node.id` among all controller pod descriptors) formats
+  with `kafka-storage.sh format --standalone`, bootstrapping a single-node quorum by itself.
+  Every other controller formats with `--no-initial-controllers` and joins purely through the sidecar's `add-controller` call.
+  Brokers always format with `--no-initial-controllers` too; they are never voters.
+* `controller.quorum.bootstrap.servers` points at each controller role group's own headless Service DNS name, not
+  individual pod addresses.
 
 == Known Issues
 
 * Automatic migration from Apache ZooKeeper to KRaft is not supported.
-* Scaling controller replicas might lead to unstable clusters.
 * Kerberos is currently not supported for KRaft in all versions.
+* The quorum is created once by the controller with the lowest `node.id` using `--standalone`. If this controller
+  loses it's PVC, a new conflicting quorum is created on restart.
+* A Controller that loses its persistent volume is not re-admitted to the voter set automatically, because it
+  returns with a new directory ID while the quorum still lists the old one.
 
-== Troubleshooting
+==== Prevention
 
-=== Cluster does not start
-
-Check that at least a quorum (majority) of controllers are reachable.
-
-=== Frequent leader elections
-
-Likely caused by controller resource starvation or unstable Kubernetes scheduling.
-
-=== Migration issues (ZooKeeper to KRaft)
-
-Ensure Kafka version 3.9.x and higher and follow the official migration documentation.
-The Stackable Kafka operator currently does not support the migration.
-
-=== Scaling issues
-
-The https://developers.redhat.com/articles/2024/11/27/dynamic-kafka-controller-quorum[Dynamic scaling] is only supported from Kafka version 3.9.0.
-If you are using older versions, automatic scaling may not work properly (e.g. adding or removing controller replicas).
+* Use a StorageClass backed by network-attached storage, so that a node failure does not imply volume loss.
+* Do not delete Controller PVCs as part of routine maintenance.
 
 == Kraft migration guide
 

docs/modules/kafka/partials/supported-versions.adoc

@@ -7,10 +7,7 @@
 * 3.9.2 (LTS)
 * 3.9.1 (deprecated)
 
-Support for clusters running in Kraft mode (which includes Apache Kafka 4.x.x) is experimental because it has not been thoroughly tested in production environments yet.
+Support for clusters running in Kraft mode (which includes Apache Kafka >= 4.x) is experimental due to the following known issues:
 
-Also there are some known issues such as:
-
-* Controller scaling is not reliable.
 * Kerberos authentication is not tested yet.
 * Service exposition is not definitive.

extra/crds.yaml

@@ -181,86 +181,6 @@ spec:
                                     description: Configuration per logger
                                     type: object
                                 type: object
-                              kcat-prober:
-                                anyOf:
-                                - required:
-                                  - custom
-                                - {}
-                                - {}
-                                description: Log configuration of the container
-                                properties:
-                                  console:
-                                    description: Configuration for the console appender
-                                    nullable: true
-                                    properties:
-                                      level:
-                                        description: |-
-                                          The log level threshold.
-                                          Log events with a lower log level are discarded.
-                                        enum:
-                                        - TRACE
-                                        - DEBUG
-                                        - INFO
-                                        - WARN
-                                        - ERROR
-                                        - FATAL
-                                        - NONE
-                                        - null
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  custom:
-                                    description: Log configuration provided in a ConfigMap
-                                    properties:
-                                      configMap:
-                                        description: ConfigMap containing the log configuration files
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  file:
-                                    description: Configuration for the file appender
-                                    nullable: true
-                                    properties:
-                                      level:
-                                        description: |-
-                                          The log level threshold.
-                                          Log events with a lower log level are discarded.
-                                        enum:
-                                        - TRACE
-                                        - DEBUG
-                                        - INFO
-                                        - WARN
-                                        - ERROR
-                                        - FATAL
-                                        - NONE
-                                        - null
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  loggers:
-                                    additionalProperties:
-                                      description: Configuration of a logger
-                                      properties:
-                                        level:
-                                          description: |-
-                                            The log level threshold.
-                                            Log events with a lower log level are discarded.
-                                          enum:
-                                          - TRACE
-                                          - DEBUG
-                                          - INFO
-                                          - WARN
-                                          - ERROR
-                                          - FATAL
-                                          - NONE
-                                          - null
-                                          nullable: true
-                                          type: string
-                                      type: object
-                                    default: {}
-                                    description: Configuration per logger
-                                    type: object
-                                type: object
                               vector:
                                 anyOf:
                                 - required:
@@ -717,86 +637,6 @@ spec:
                                           description: Configuration per logger
                                           type: object
                                       type: object
-                                    kcat-prober:
-                                      anyOf:
-                                      - required:
-                                        - custom
-                                      - {}
-                                      - {}
-                                      description: Log configuration of the container
-                                      properties:
-                                        console:
-                                          description: Configuration for the console appender
-                                          nullable: true
-                                          properties:
-                                            level:
-                                              description: |-
-                                                The log level threshold.
-                                                Log events with a lower log level are discarded.
-                                              enum:
-                                              - TRACE
-                                              - DEBUG
-                                              - INFO
-                                              - WARN
-                                              - ERROR
-                                              - FATAL
-                                              - NONE
-                                              - null
-                                              nullable: true
-                                              type: string
-                                          type: object
-                                        custom:
-                                          description: Log configuration provided in a ConfigMap
-                                          properties:
-                                            configMap:
-                                              description: ConfigMap containing the log configuration files
-                                              nullable: true
-                                              type: string
-                                          type: object
-                                        file:
-                                          description: Configuration for the file appender
-                                          nullable: true
-                                          properties:
-                                            level:
-                                              description: |-
-                                                The log level threshold.
-                                                Log events with a lower log level are discarded.
-                                              enum:
-                                              - TRACE
-                                              - DEBUG
-                                              - INFO
-                                              - WARN
-                                              - ERROR
-                                              - FATAL
-                                              - NONE
-                                              - null
-                                              nullable: true
-                                              type: string
-                                          type: object
-                                        loggers:
-                                          additionalProperties:
-                                            description: Configuration of a logger
-                                            properties:
-                                              level:
-                                                description: |-
-                                                  The log level threshold.
-                                                  Log events with a lower log level are discarded.
-                                                enum:
-                                                - TRACE
-                                                - DEBUG
-                                                - INFO
-                                                - WARN
-                                                - ERROR
-                                                - FATAL
-                                                - NONE
-                                                - null
-                                                nullable: true
-                                                type: string
-                                            type: object
-                                          default: {}
-                                          description: Configuration per logger
-                                          type: object
-                                      type: object
                                     vector:
                                       anyOf:
                                       - required:

rust/operator-binary/src/controller.rs

@@ -611,6 +611,14 @@ pub(crate) mod test_support {
 
     /// Runs the real validate step against a minimal (auth/OPA-free) fixture.
     pub fn validated_cluster(kafka: &v1alpha1::KafkaCluster) -> ValidatedCluster {
+        validate_err(kafka).expect("validate should succeed for the test fixture")
+    }
+
+    /// Runs the real validate step against a minimal (auth/OPA-free) fixture, without unwrapping
+    /// the result. Used for tests asserting on a specific validation failure.
+    pub fn validate_err(
+        kafka: &v1alpha1::KafkaCluster,
+    ) -> Result<ValidatedCluster, super::validate::Error> {
         validate(
             kafka,
             DereferencedObjects {
@@ -621,7 +629,6 @@ pub(crate) mod test_support {
             },
             &operator_environment(),
         )
-        .expect("validate should succeed for the test fixture")
     }
 }