Generic active-active reconciliation: an ownership/coordination control plane to complement shardSelector

[prashantbh]

Is your feature request related to a problem? Please describe.

JOSDK's leader election provides high availability but a single active replica: standbys run informers and maintain caches, while only the leader processes events. This is efficient in steady state, but during fleet-wide churn (for example, when informers relist after an API-server outage and every resource must be reconciled again) recovery is serialized through one replica's reconciliation executor while standby capacity stays idle. Scaling the leader (a larger pod, higher reconciliation parallelism) is ultimately bounded by a single JVM.

JOSDK 5.4 and the Fabric8 client expose the server-side shardSelector from KEP-5866, which lets each replica List/Watch only its hash range. That is the read-path mechanism, but as the 5.4 notes state, operators must coordinate replica ranges themselves. A static shard selector alone is effectively hash(resource) % replicaCount: scaling remaps ranges, a failed replica's range is unserved until its pod restarts, and range changes have no safe handoff (dual-owner or gap windows). It does not provide assignment, rebalancing on scale, failover/takeover, drained handoff, or reconciliation fencing.

Describe the solution you'd like

A generic coordination/ownership control plane in JOSDK that turns static sharding into dynamic, fault-tolerant sharding, and that can drive the shardSelector where the cluster supports it. The following pieces are framework-generic (not specific to any resource type):

• Replica membership: which replicas are alive and have synchronized caches.
• Deterministic shard assignment and rebalancing, using stable logical shards that are independent of the replica count, so scaling does not remap the whole keyspace.
• Reconciliation admission based on ownership: only the owning replica processes an event.
• Assignment generations/epochs for ordering and fencing.
• Drain-before-activate ownership transfer: the previous owner drains in-flight reconciliations before the new owner begins.
• Failure takeover: membership expiry plus safety checks before another replica assumes ownership.
• Re-enqueue of owned resources after ownership changes.
• Hooks to fence mutations: revalidate ownership immediately before a side effect.

The control plane can drive the shardSelector: the assignment sets each replica's hash range and updates it on rebalance/failover with safe draining, giving end-to-end dynamic read-plus-reconcile sharding. On clusters without the KEP-5866 feature (alpha in Kubernetes 1.36 and off by default), the same assignment is enforced with full informers and a local ownership filter, so the model also works on older clusters. The read path and the ownership control plane stay decoupled: the selector is derived from the assignment rather than being the assignment.

Describe alternatives you've considered

• Static shardSelector alone: no dynamic rebalancing, failover, or safe handoff (equivalent to hash % replicaCount).
• Namespace partitioning via separate operator installations: manual, cannot split a hot namespace, and each install is still leader/standby so standbys stay idle.
• A Lease per resource: fine-grained ownership but thousands of coordination objects and continuous write traffic.
• An external work queue (Kafka-like): natural distribution but an external dependency and a departure from the informer-based model.

Additional context

This proposal comes from a FLIP (FLink Improvement Proposal) for horizontally scaling the Flink Kubernetes Operator (active-active reconciliation). Most of the coordination layer is framework-generic and could benefit other operators: in that operator, the only operator-specific part is fencing non-Kubernetes side effects (Flink REST calls), while membership, assignment, admission, generations, draining, transfer, and Kubernetes-mutation fencing are all generic.

One integration consideration for shardSelector adoption is hash-key choice and cross-resource caches. KEP-5866 hashes on metadata.uid or metadata.namespace. A common operator pattern uses a CR type as both a primary and a secondary (dependency) resource. UID-sharding the primary informers then still requires complete secondary informers of those types on every replica for cross-resource lookups, which removes the read-path savings; namespace-keying colocates related resources but cannot split a hot namespace. A generic design should let the assignment/selector mapping account for secondary and dependency caches.

Design document: https://docs.google.com/document/d/161ST30OPcyQk8UHyz4Gm1j8tz9avfXW3KCZa4jqrdtc/edit?usp=sharing