Skip to content

fix: Re-emit existing discovery CM if listener is not ready - #1105

Merged
adwk67 merged 14 commits into
mainfrom
fix/discovery-cm-reemit
Nov 11, 2026
Merged

adwk67 merged 14 commits into
mainfrom
fix/discovery-cm-reemit

Conversation

@adwk67

@adwk67 adwk67 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Description

Part of https://github.com/stackabletech/internal-issues/issues/167

--- PASS: kuttl (210.31s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/znode_zookeeper-latest-3.9.6_openshift-false (26.52s)
        --- PASS: kuttl/harness/cluster-operation_zookeeper-latest-3.9.6_openshift-false (54.25s)
        --- PASS: kuttl/harness/logging_zookeeper-3.9.6_openshift-false (97.66s)
        --- PASS: kuttl/harness/smoke_zookeeper-3.9.6_use-server-tls-true_use-client-auth-tls-true_openshift-false (83.52s)
        --- PASS: kuttl/harness/delete-rolegroup_zookeeper-3.9.6_openshift-false (129.54s)
PASS

kuttl: the cluster-operation test has a new step 60-recreate-role-listener that deletes the role Listener and asserts that the discovery ConfigMap keeps its uid, data and resourceVersion once the Listener is re-created.

Manual checks on kind, with product images at stackableVersion: 26.3.0 vs. the CI image built from this PR 0.0.0-pr1105 :-

Scenario Result
Fresh deploy The discovery ConfigMap is written once, already with addresses; no version with empty ZOOKEEPER/ZOOKEEPER_HOSTS. Druid, which copies the ZooKeeper address from this ConfigMap into its own config, rolls out once (as on 26.3) instead of twice (as on main).
Role Listener deleted on a running cluster The ConfigMap is unchanged (same uid and resourceVersion) while the Listener is re-created; no consumer rollouts.
Cluster paused before the Listener has an address No reconcile errors while paused, no ConfigMap. Once unpaused, the ConfigMap is created with addresses straight away.
Upgrade 26.3.0 → PR-1105, role Listener present Only the label change that every discovery ConfigMap gets on this upgrade; data unchanged, no empty version.
Upgrade 26.3.0 → PR-1105, role Listener missing (26.3.0 does not re-create a deleted one) The stored ConfigMap is re-emitted before the new Listener has an address; data unchanged, no empty version (main writes empty values here).
Discovery ConfigMap deleted Re-created at once, with addresses.
Discovery ConfigMap deleted while paused Stays absent while paused; re-created with addresses on unpause.
Repeated reconciles No ConfigMap change.
listenerClass change The ConfigMap switches to the new address once the Listener publishes it, and back when reverted; no empty version.

What users notice

26.3.0 and 26.7.0 handle the role Listener the same way: the operator does not watch Listeners,
and the cluster reconcile fails while the Listener has no address. The differences below apply
to an upgrade from either release.

  • No reconcile error while the role Listener has no address yet Before, the reconcile failed with "has no ingress addresses" (or a port-not-found error) and skipped the status update and orphan cleanup until the address appeared. Now it succeeds.
  • The discovery ConfigMap content stays the same in that window. An existing ConfigMap keeps its values; a new one is created only once the address is known. Consumers never see empty ZOOKEEPER or ZOOKEEPER_HOSTS values. On a fresh deploy, Druid rolls out once, the same as on 26.3.0.
  • The ConfigMap follows the address faster. Before, a change of the Listener's address only reached the ConfigMap on the next reconcile triggered by something else; now the ConfigMap is updated as soon as the Listener publishes the new address. Until then the old address is still advertised.
  • A deleted role Listener is re-created. On 26.3.0 it stayed missing as no reconcile was triggered. Now it is re-created, and the discovery ConfigMap is left untouched meanwhile, so consumers do not roll out.
  • While the cluster is paused, a discovery ConfigMap that does not exist yet is not created; it appears on unpause.
  • On upgrade, the discovery ConfigMap gets one new resourceVersion, for a label change only; its data is unchanged.

Compared with the current main (unreleased), this PR removes the extra ConfigMap version with
empty ZOOKEEPER/ZOOKEEPER_HOSTS on every fresh deploy and every time the role Listener is
re-created, and with it the extra rollout of consumers such as Druid that copy the address into
their own config.

ZookeeperZnode discovery ConfigMaps are unchanged by this PR: a ZookeeperZnode still fails its
reconcile until the role Listener has an address.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • (Integration-)Test cases added
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@adwk67 adwk67 self-assigned this Nov 5, 2026
@adwk67
adwk67 marked this pull request as ready for review November 6, 2026 09:23
@adwk67 adwk67 moved this to Development: Waiting for Review in Stackable Engineering Nov 6, 2026
@siegfriedweber siegfriedweber moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 11, 2026
siegfriedweber
siegfriedweber previously approved these changes Nov 11, 2026

@siegfriedweber siegfriedweber left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I have some comments that I would classify as "could have".

Comment thread rust/operator-binary/src/discovery.rs Resolved
Comment thread rust/operator-binary/src/discovery.rs Resolved
Comment thread rust/operator-binary/src/discovery.rs Resolved
@adwk67
adwk67 added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 37f03d7 Nov 11, 2026
13 checks passed
@adwk67
adwk67 deleted the fix/discovery-cm-reemit branch November 11, 2026 16:31
@adwk67 adwk67 moved this from Development: In Review to Development: Done in Stackable Engineering Nov 11, 2026
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

3 participants