Add configurable MaxConcurrentReconciles for CHK controller#2033
Open
miguel-signoz wants to merge 1 commit into
Open
Add configurable MaxConcurrentReconciles for CHK controller#2033miguel-signoz wants to merge 1 commit into
miguel-signoz wants to merge 1 commit into
Conversation
sunsingerus
requested changes
Jul 7, 2026
sunsingerus
left a comment
sunsingerus
left a comment
Collaborator
There was a problem hiding this comment.
Full cycle with CRD modification is needed as well
Add reconcileCHKsThreadsNumber config option to control the number of concurrent ClickHouseKeeperInstallation reconciliations. Previously hardcoded to 1 via controller-runtime defaults, this caused a significant bottleneck at scale when many CHK resources needed reconciliation simultaneously. The new setting mirrors the existing reconcileCHIsThreadsNumber for CHI resources. Defaults to 1 to preserve existing behavior. Closes Altinity#2032 Signed-off-by: miguel-signoz <miguel@signoz.io>
miguel-signoz
force-pushed
the
0.27.2-chk-concurrent-reconciles
branch
from
858ed9d to
09d6b17
Compare
Author
|
Updated added reconcileCHKsThreadsNumber to CRD schemas, config files, helm values, and JSON schema across all deploy targets. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
reconcileCHKsThreadsNumberconfig option to control the number of concurrent ClickHouseKeeperInstallation reconciliations.The CHK controller currently uses controller-runtime's default
MaxConcurrentReconciles=1, meaning all CHK resources are reconciled serially. This becomes a significant bottleneck at scale — with 1,000+ CHK resources and ~30-40 seconds per reconciliation (due to force restarts and hardcoded sleeps), a full reconciliation cycle takes 9-12 hours. During this time, newly created CHK resources are added to the back of the FIFO queue and can wait hours before the operator begins reconciling them.The new setting mirrors the existing
reconcileCHIsThreadsNumberfor CHI resources. Defaults to 1 to preserve existing behavior.Changes
cmd/operator/app/thread_keeper.go: Add.WithOptions(controller.Options{MaxConcurrentReconciles: ...})to the controller-runtime builder chainpkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go: AddReconcileCHKsThreadsNumberfield toOperatorConfigReconcileRuntimestruct, default constant, and normalization logicConfig YAML
Closes #2032
Signed-off-by: miguel-signoz miguel@signoz.io