You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature] Store-side readiness for rolling replacement: registration marks Up before partition restore, and a stopped Store stays Up in every shard group for keepAlive-timeout #3229
Two gaps make it impossible to write an honest barrier for rolling Store replacement (the OnDelete procedure a production image roll needs):
StoreNodeService.register() persists StoreState.Up before the Store runs restoreLocalPartitionEngine(), and a failure there is only logged, so Up never means restored.
A stopped Store stays Up and listed in all its shard groups until its keep-alive entry expires (store.keepAlive-timeout, 300 s) plus the 60 s patrol. Measured 2026-09-22 on master 83ef9f3 (3+3+3, kind): a Store pod deleted and back after 150 s never left Up; a barrier polling /v1/shardGroups on the PD leader every 2 s reported full membership and one leader per group on every sample of the whole outage.
The consequence: an operator (or any automation) deleting Stores one at a time and gating on PD's view can pass the gate while the previous Store is still down, ending at one live replica per group. Today the only discriminating signal is the Store's own GET :8520/v1/partition/{id} per group, which is ad hoc.
Ask: a Store-side readiness signal that means "registered, partitions restored, raft groups caught up" (the Store counterpart of #3212 for the Server), exposed both as an endpoint suitable for a Kubernetes readinessProbe and to PD; and/or PD exposing heartbeat age (or a suspect state) so a stale Up is distinguishable from a live one. Complements #3136, which asks for the recovery progress metrics; this is the boolean gate on top.
The Helm chart in #3218 documents the manual workaround in two places. The chart README's Upgrading section keeps the Store roll: values-cluster.yaml sets store.updateStrategy.type=OnDelete, and the gate between deletions is Pod Ready plus the per-group /v1/shardGroups check. The rest moved to the operations page in apache/hugegraph-doc#494 (section 6, "Rolling Store images safely", linking this issue): the closer :8520/v1/partition/<groupId> check on the Store's own REST port, and the statement that no endpoint reports restoration-complete. When this fix merges: if #3218 has merged, add a chart TODO to gate on the new signal and shrink the procedure in both places; else comment there to update and re-test.
(Edited 2026-10-02: locations updated after the README was slimmed in 3996bb127.)
Hi @imbajin, PR #3362 is open for this issue, and I'm following up on the remaining review points. Could you please assign this issue to me before I continue with the revisions?
Activity
bitflicker64 commentedon Sep 24, 2026
The Helm chart in #3218 documents the manual workaround in two places. The chart README's Upgrading section keeps the Store roll:
values-cluster.yamlsetsstore.updateStrategy.type=OnDelete, and the gate between deletions is PodReadyplus the per-group/v1/shardGroupscheck. The rest moved to the operations page in apache/hugegraph-doc#494 (section 6, "Rolling Store images safely", linking this issue): the closer:8520/v1/partition/<groupId>check on the Store's own REST port, and the statement that no endpoint reports restoration-complete. When this fix merges: if #3218 has merged, add a chart TODO to gate on the new signal and shrink the procedure in both places; else comment there to update and re-test.(Edited 2026-10-02: locations updated after the README was slimmed in
3996bb127.)pallavithegod commentedon Oct 16, 2026
Hi @imbajin, PR #3362 is open for this issue, and I'm following up on the remaining review points. Could you please assign this issue to me before I continue with the revisions?