Skip to content

[Feature] Store-side readiness for rolling replacement: registration marks Up before partition restore, and a stopped Store stays Up in every shard group for keepAlive-timeout #3229

Open

Description

@bitflicker64

Feature Description (功能描述)

Two gaps make it impossible to write an honest barrier for rolling Store replacement (the OnDelete procedure a production image roll needs):

  1. StoreNodeService.register() persists StoreState.Up before the Store runs restoreLocalPartitionEngine(), and a failure there is only logged, so Up never means restored.
  2. A stopped Store stays Up and listed in all its shard groups until its keep-alive entry expires (store.keepAlive-timeout, 300 s) plus the 60 s patrol. Measured 2026-09-22 on master 83ef9f3 (3+3+3, kind): a Store pod deleted and back after 150 s never left Up; a barrier polling /v1/shardGroups on the PD leader every 2 s reported full membership and one leader per group on every sample of the whole outage.

The consequence: an operator (or any automation) deleting Stores one at a time and gating on PD's view can pass the gate while the previous Store is still down, ending at one live replica per group. Today the only discriminating signal is the Store's own GET :8520/v1/partition/{id} per group, which is ad hoc.

Ask: a Store-side readiness signal that means "registered, partitions restored, raft groups caught up" (the Store counterpart of #3212 for the Server), exposed both as an endpoint suitable for a Kubernetes readinessProbe and to PD; and/or PD exposing heartbeat age (or a suspect state) so a stale Up is distinguishable from a live one. Complements #3136, which asks for the recovery progress metrics; this is the boolean gate on top.

Activity

  1. bitflicker64 commented on Sep 24, 2026

    @bitflicker64
    ContributorAuthor

    The Helm chart in #3218 documents the manual workaround in two places. The chart README's Upgrading section keeps the Store roll: values-cluster.yaml sets store.updateStrategy.type=OnDelete, and the gate between deletions is Pod Ready plus the per-group /v1/shardGroups check. The rest moved to the operations page in apache/hugegraph-doc#494 (section 6, "Rolling Store images safely", linking this issue): the closer :8520/v1/partition/<groupId> check on the Store's own REST port, and the statement that no endpoint reports restoration-complete. When this fix merges: if #3218 has merged, add a chart TODO to gate on the new signal and shrink the procedure in both places; else comment there to update and re-test.

    (Edited 2026-10-02: locations updated after the README was slimmed in 3996bb127.)

  2. pallavithegod commented on Oct 16, 2026

    @pallavithegod

    Hi @imbajin, PR #3362 is open for this issue, and I'm following up on the remaining review points. Could you please assign this issue to me before I continue with the revisions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions