Skip to content

Commit e99d0ee

Browse files
razvanadwk67
razvan
and
adwk67
authored
chore: support kraft migration (#923)
* allow kafka controllers running with zookeeper metadata manager * add zookeeper migration properties to controllers * validate controller role only if controllers are defined * move cluster id to env var * disable automatic broker.id generation * disable default tls * add test manifests * Revert "disable default tls" This reverts commit 88f319d. * add internal listener to controller properties. * update test manifests * add internal broker listener TLS settings to controller.properties * update migration manifests * fix metadata manager casing * update kraft kuttl tests * update changelog and start documenting the migration process * make meta manager optional and implement sanity checks * add deprecation notice * kraft migration guide * spelling and casing * typo * zk -> kraft mm2 example * add field to support manual broker id assignments * update docs to match decision * update start command to new broker map decision * backwards compatibility: disable broker id generation only if necessary * update changelog * update changelog * minor cleanups * update tests and remove static quorum property * clarify mm2 example purpose * clarify kraft actions * proper english is nice * implement review feedback * Apply suggestions from code review Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --------- Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com>
1 parent 7d70202 commit e99d0ee

32 files changed

Lines changed: 1054 additions & 118 deletions

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ All notable changes to this project will be documented in this file.
1515
### Changed
1616

1717
- Refactor: move server configuration properties from the command line to configuration files. ([#911]).
18+
- Add support for ZooKeeper to KRaft migration ([#923]).
1819
- Bump testing-tools to `0.3.0-stackable0.0.0-dev` ([#925]).
1920

2021
### Removed
@@ -26,6 +27,7 @@ All notable changes to this project will be documented in this file.
2627
[#911]: https://github.com/stackabletech/kafka-operator/pull/911
2728
[#914]: https://github.com/stackabletech/kafka-operator/pull/914
2829
[#915]: https://github.com/stackabletech/kafka-operator/pull/915
30+
[#923]: https://github.com/stackabletech/kafka-operator/pull/923
2931
[#925]: https://github.com/stackabletech/kafka-operator/pull/925
3032
[#927]: https://github.com/stackabletech/kafka-operator/pull/927
3133
[#928]: https://github.com/stackabletech/kafka-operator/pull/928

‎deploy/helm/kafka-operator/crds/crds.yaml‎

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -787,6 +787,61 @@ spec:
787787
- configMapName
788788
type: object
789789
type: object
790+
brokerIdPodConfigMapName:
791+
description: |-
792+
Enable users to manually assign Kafka broker ids.
793+
794+
Name of a ConfigMap containing a mapping of broker IDs to pod names.
795+
The ConfigMap must contain a key for every broker pod in the cluster with the following format:
796+
`<pod-name>: <broker-id>`
797+
798+
Example:
799+
```
800+
---
801+
apiVersion: v1
802+
kind: ConfigMap
803+
metadata:
804+
name: brokeridmapping
805+
data:
806+
simple-kafka-broker-default-0: "2000"
807+
simple-kafka-broker-default-1: "2001"
808+
simple-kafka-broker-default-2: "2002"
809+
```
810+
This is necessary when migrating from ZooKeeper to Kraft mode to retain existing broker IDs
811+
because previously broker ids were generated by Kafka and not the operator.
812+
nullable: true
813+
type: string
814+
metadataManager:
815+
description: |-
816+
Metadata manager to use for the Kafka cluster.
817+
818+
IMPORTANT: This property will be removed as soon as Kafka 3.x support is dropped.
819+
820+
Possible values are `zookeeper` and `kraft`.
821+
822+
If not set, defaults to:
823+
824+
- `zookeeper` for Kafka versions below `4.0.0`.
825+
- `kraft` for Kafka versions `4.0.0` and higher.
826+
827+
Using `zookeeper` for Kafka versions `4.0.0` and higher is not supported.
828+
829+
When set to `kraft`, the operator will perform the following actions:
830+
831+
* Generate the Kafka cluster id.
832+
* Assign broker roles and configure controller quorum voters in the `broker.properties` files.
833+
* Format storage before (re)starting Kafka brokers.
834+
* Remove ZooKeeper related configuration options from the `broker.properties` files.
835+
836+
Some of them cannot be performed with the ZooKeeper mode.
837+
838+
This property is also useful when migrating from ZooKeeper to Kraft mode because it permits the operator
839+
to reconcile controllers while still using ZooKeeper for brokers.
840+
enum:
841+
- zookeeper
842+
- kraft
843+
nullable: true
844+
type: string
790845
tls:
791846
default:
792847
internalSecretClass: tls
@@ -830,7 +885,7 @@ spec:
830885
Provide the name of the ZooKeeper [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery)
831886
here. When using the [Stackable operator for Apache ZooKeeper](https://docs.stackable.tech/home/nightly/zookeeper/)
832887
to deploy a ZooKeeper cluster, this will simply be the name of your ZookeeperCluster resource.
833-
This can only be used up to Kafka version 3.9.x. Since Kafka 4.0.0, ZooKeeper suppport was dropped.
888+
This can only be used up to Kafka version 3.9.x. Since Kafka 4.0.0, ZooKeeper support was dropped.
834889
Please use the 'controller' role instead.
835890
nullable: true
836891
type: string

‎docs/modules/kafka/examples/kraft_migration/01-setup.yaml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
apiVersion: v1
3+
kind: Namespace
4+
metadata:
5+
labels:
6+
stackable.tech/vendor: Stackable
7+
name: kraft-migration
8+
---
9+
apiVersion: zookeeper.stackable.tech/v1alpha1
10+
kind: ZookeeperCluster
11+
metadata:
12+
name: simple-zk
13+
namespace: kraft-migration
14+
spec:
15+
image:
16+
productVersion: 3.9.4
17+
pullPolicy: IfNotPresent
18+
servers:
19+
roleGroups:
20+
default:
21+
replicas: 1
22+
---
23+
apiVersion: zookeeper.stackable.tech/v1alpha1
24+
kind: ZookeeperZnode
25+
metadata:
26+
name: simple-kafka-znode
27+
namespace: kraft-migration
28+
spec:
29+
clusterRef:
30+
name: simple-zk
31+
---
32+
apiVersion: secrets.stackable.tech/v1alpha1
33+
kind: SecretClass
34+
metadata:
35+
name: kafka-internal-tls
36+
spec:
37+
backend:
38+
autoTls:
39+
ca:
40+
secret:
41+
name: secret-provisioner-kafka-internal-tls-ca
42+
namespace: kraft-migration
43+
autoGenerate: true
44+
---
45+
apiVersion: authentication.stackable.tech/v1alpha1
46+
kind: AuthenticationClass
47+
metadata:
48+
name: kafka-client-auth-tls
49+
spec:
50+
provider:
51+
tls:
52+
clientCertSecretClass: kafka-client-auth-secret
53+
---
54+
apiVersion: secrets.stackable.tech/v1alpha1
55+
kind: SecretClass
56+
metadata:
57+
name: kafka-client-auth-secret
58+
spec:
59+
backend:
60+
autoTls:
61+
ca:
62+
secret:
63+
name: secret-provisioner-tls-kafka-client-ca
64+
namespace: kraft-migration
65+
autoGenerate: true
66+
---
67+
apiVersion: v1
68+
kind: ConfigMap
69+
metadata:
70+
name: broker-ids
71+
namespace: kraft-migration
72+
data:
73+
simple-kafka-broker-default-0: "2000"
74+
simple-kafka-broker-default-1: "2001"
75+
simple-kafka-broker-default-2: "2002"
76+
---
77+
apiVersion: kafka.stackable.tech/v1alpha1
78+
kind: KafkaCluster
79+
metadata:
80+
name: simple-kafka
81+
namespace: kraft-migration
82+
spec:
83+
image:
84+
productVersion: 3.9.1
85+
pullPolicy: IfNotPresent
86+
clusterConfig:
87+
metadataManager: zookeeper
88+
brokerIdPodConfigMapName: broker-ids
89+
authentication:
90+
- authenticationClass: kafka-client-auth-tls
91+
tls:
92+
internalSecretClass: kafka-internal-tls
93+
serverSecretClass: tls
94+
zookeeperConfigMapName: simple-kafka-znode
95+
brokers:
96+
roleGroups:
97+
default:
98+
replicas: 3

‎docs/modules/kafka/examples/kraft_migration/02-start-controllers.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
apiVersion: kafka.stackable.tech/v1alpha1
3+
kind: KafkaCluster
4+
metadata:
5+
name: simple-kafka
6+
namespace: kraft-migration
7+
spec:
8+
image:
9+
productVersion: 3.9.1
10+
pullPolicy: IfNotPresent
11+
clusterConfig:
12+
metadataManager: zookeeper
13+
authentication:
14+
- authenticationClass: kafka-client-auth-tls
15+
tls:
16+
internalSecretClass: kafka-internal-tls
17+
serverSecretClass: tls
18+
zookeeperConfigMapName: simple-kafka-znode
19+
brokerIdPodConfigMapName: broker-ids
20+
brokers:
21+
envOverrides:
22+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
23+
roleGroups:
24+
default:
25+
replicas: 3
26+
controllers:
27+
roleGroups:
28+
default:
29+
replicas: 3
30+
envOverrides:
31+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
32+
configOverrides:
33+
controller.properties:
34+
zookeeper.metadata.migration.enable: "true" # Enable migration mode so the controller can read metadata from ZooKeeper.

‎docs/modules/kafka/examples/kraft_migration/03-migrate-metadata.yaml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
apiVersion: kafka.stackable.tech/v1alpha1
3+
kind: KafkaCluster
4+
metadata:
5+
name: simple-kafka
6+
namespace: kraft-migration
7+
spec:
8+
image:
9+
productVersion: 3.9.1
10+
pullPolicy: IfNotPresent
11+
clusterConfig:
12+
metadataManager: zookeeper
13+
authentication:
14+
- authenticationClass: kafka-client-auth-tls
15+
tls:
16+
internalSecretClass: kafka-internal-tls
17+
serverSecretClass: tls
18+
zookeeperConfigMapName: simple-kafka-znode
19+
brokerIdPodConfigMapName: broker-ids
20+
brokers:
21+
envOverrides:
22+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
23+
roleGroups:
24+
default:
25+
replicas: 3
26+
configOverrides:
27+
broker.properties:
28+
inter.broker.protocol.version: "3.9" # - Latest value known to Kafka 3.9.1
29+
zookeeper.metadata.migration.enable: "true" # - Enable migration mode so the broker can participate in metadata migration.
30+
controller.listener.names: "CONTROLLER"
31+
controller.quorum.bootstrap.servers: "simple-kafka-controller-default-0.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093,simple-kafka-controller-default-1.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093,simple-kafka-controller-default-2.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093"
32+
controllers:
33+
roleGroups:
34+
default:
35+
replicas: 3
36+
envOverrides:
37+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
38+
configOverrides:
39+
controller.properties:
40+
zookeeper.metadata.migration.enable: "true" # Enable migration mode so the controller can read metadata from ZooKeeper.

‎docs/modules/kafka/examples/kraft_migration/04-migrate-brokers.yaml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
apiVersion: kafka.stackable.tech/v1alpha1
3+
kind: KafkaCluster
4+
metadata:
5+
name: simple-kafka
6+
namespace: kraft-migration
7+
spec:
8+
image:
9+
productVersion: 3.9.1
10+
pullPolicy: IfNotPresent
11+
clusterConfig:
12+
metadataManager: zookeeper
13+
authentication:
14+
- authenticationClass: kafka-client-auth-tls
15+
tls:
16+
internalSecretClass: kafka-internal-tls
17+
serverSecretClass: tls
18+
zookeeperConfigMapName: simple-kafka-znode
19+
brokerIdPodConfigMapName: broker-ids
20+
brokers:
21+
envOverrides:
22+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
23+
roleGroups:
24+
default:
25+
replicas: 3
26+
configOverrides:
27+
broker.properties:
28+
controller.listener.names: "CONTROLLER"
29+
controller.quorum.bootstrap.servers: "simple-kafka-controller-default-0.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093,simple-kafka-controller-default-1.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093,simple-kafka-controller-default-2.simple-kafka-controller-default-headless.kraft-migration.svc.cluster.local:9093"
30+
process.roles: "broker"
31+
node.id: "${env:REPLICA_ID}"
32+
controllers:
33+
roleGroups:
34+
default:
35+
replicas: 3
36+
envOverrides:
37+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
38+
configOverrides:
39+
controller.properties:
40+
zookeeper.metadata.migration.enable: "true" # Enable migration mode so the controller can read metadata from ZooKeeper.

‎docs/modules/kafka/examples/kraft_migration/05-kraft-mode.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
apiVersion: kafka.stackable.tech/v1alpha1
3+
kind: KafkaCluster
4+
metadata:
5+
name: simple-kafka
6+
namespace: kraft-migration
7+
spec:
8+
image:
9+
productVersion: 3.9.1
10+
pullPolicy: IfNotPresent
11+
clusterConfig:
12+
metadataManager: kraft
13+
authentication:
14+
- authenticationClass: kafka-client-auth-tls
15+
tls:
16+
internalSecretClass: kafka-internal-tls
17+
serverSecretClass: tls
18+
brokerIdPodConfigMapName: broker-ids
19+
brokers:
20+
envOverrides:
21+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"
22+
roleGroups:
23+
default:
24+
replicas: 3
25+
configOverrides:
26+
broker.properties:
27+
controller.listener.names: "CONTROLLER"
28+
controllers:
29+
roleGroups:
30+
default:
31+
replicas: 3
32+
envOverrides:
33+
KAFKA_CLUSTER_ID: "MyCya7hbTD-Hay8PgCsCYA"

0 commit comments

Comments
 (0)