Backfill and resilver backup store - #1073
Open
enigbe wants to merge 14 commits into
Open
Conversation
This commit adds `TierStore`, a tiered `KVStore` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary and backup. Reads do not consult the backup store during normal operation. Unpaginated listings expose the logical contents of the primary and ephemeral tiers without consulting the backup store; paginated listings currently expose only the primary tier. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals Assisted-by: Amp (AI coding agent)
TierStore needs a single persistent ordering domain before it can provide correct pagination across primary and ephemeral stores because the wrapped stores' native ordering and pagination tokens are not comparable. In this commit, we create an internal SQLite index database automatically whenever ephemeral storage is configured and give it a persistent identity as well as an exclusive SQLite lock for TierStore's lifetime. This is a prefactor and only establishes the internal storage plumbing; listing behavior remains unchanged. Assisted-by: Amp (AI coding agent)
Primary and ephemeral stores maintain independent creation orders, so their paginated listings cannot be merged while preserving the PaginatedKVStore ordering contract. In this commit, the local TierStore index records logical key membership in one SQLite ordering domain. We initialize each namespace from the primary store on first use, preserving its existing paginated order, then maintain the index after writes and removals. We also route both list methods through the index so keys from either tier share a consistent creation order. Assisted-by: Amp (AI coding agent)
Value-store and index updates cannot be committed atomically, so failures can leave keys partially created or removed across storage tiers. In this commit we: - Persist creation and removal intent in the local index database before applying external changes. We recover pending operations before listing or modifying their namespace, including rechecking recovery under the per-key lock to prevent queued operations from racing with newly recorded journal entries. - Roll "creates" forward to all required stores, remove keys from the listing index before deleting their value copies, and retain failed operations for retry. Keep ordinary updates unjournaled and reject unindexed existing values as corruption. - Add deterministic failure and synchronization instrumentation to test recovery across interrupted writes, removals, primary/backup divergence, and queued operations. Assisted-by: Amp (AI coding agent)
The local index store supplies an opaque pagination token, but returning that token directly would allow callers to reuse it with another logical namespace or a different index database. In this commit we wrap the index token in a versioned TierStore token containing the logical namespace identity and persistent index database ID and we validate this context before passing the opaque token back to the index store, rejecting any malformed, unsupported, or mismatched tokens. Assisted-by: Amp (AI coding agent)
Nodes adopting tiered storage may already hold cache values in primary storage, while a missing index leaves existing ephemeral values without a recoverable position in the cross-store ordering. For this commit, we: - Prepare namespaces on reads as well as writes and listings so first access cannot bypass migration or journal recovery. - Rebuild missing indexes from primary ordering, discard ephemeral values whose ordering cannot be recovered, and move indexed cache values to ephemeral storage without changing their index positions. - Copy values before removing primary and backup copies, and persist a completion marker so interrupted reconciliation can safely resume without repeating it on every access. - Preserve namespace-specific cache routing for future namespace changes. Assisted-by: Amp (AI coding agent)
Preparing an entire namespace before every read allows one unrecoverable journal entry to block reads of unrelated keys. Cache migration has similar coupling because readiness is recorded for the whole namespace and all cache keys are reconciled together. In this commit, we: - Initialize the ordering index before reads, then hold the requested key's operation lock continuously while recovering its journal entry, reconciling its cache placement, and reading its value. - Track cache readiness per logical key and retain the original key identity as metadata to detect hash collisions. - Keep writes, removals, and listings on the existing namespace-wide preparation path for now (addressed in follow up). - Add coverage for unrelated pending operations, pending removals, independent cache migration, and lock re-entry deadlocks. Assisted-by: Amp (AI coding agent)
Preparing an entire namespace before every write or removal allows one unrecoverable journal entry to block mutations of unrelated keys sharing that namespace. In this commit, we: - Initialize the ordering index before mutations, then recover journal state and reconcile cache placement for only the requested key while holding its existing operation lock. - Reserve complete namespace recovery and cache reconciliation for listing operations. - Remove the obsolete journal-list snapshot gate and add coverage showing that a stuck key does not block unrelated writes or removals, while listings retain their deliberate fail-fast behavior. - Verify that touching one cache key no longer migrates another cache key in the same namespace. Assisted-by: Amp (AI coding agent)
TierStore remains internal until NodeBuilder provisions and installs its storage backends. Add builder options for local ephemeral and backup SQLite stores, wrap the configured primary store in TierStore, and pass the resulting store into node construction. When ephemeral storage is enabled, automatically create the persistent ordering index in the node's storage directory and require TierStore to own it exclusively. Update filesystem-backed tests and add integration coverage confirming that configured backup storage receives durable primary-backed data. Assisted-by: Amp (AI coding agent)
Add the opt-in storage-tier feature, gate tier-specific APIs and implementation, and preserve direct store usage when disabled. Document the feature, expose it on docs.rs, and run targeted tier storage tests in CI. AI-assisted: Developed with Amp.
Add a shared helper for creating temporary storage paths, loggers, and cleanup guards across TierStore tests.
A newly configured backup and a previously configured backup that missed primary-only operations are both potentially incomplete. Without durable synchronization metadata, TierStore cannot distinguish either case from a backup containing the current primary state. In this commit, we: - Persist an opaque synchronization generation in the authoritative primary store and compare it with the backup's last completed generation. - Classify missing or mismatched backup completion records as requiring synchronization, while preserving matching generations across restarts. - Rotate an existing primary generation when restarting without the backup so later primary-only operations invalidate its previous completion record. - Avoid creating synchronization metadata for stores that have never configured a backup, preventing an unnecessary write on every startup. - Initialize backup synchronization metadata after NodeBuilder finishes configuring TierStore and test new, stale, synchronized, and removed-backup cases. This commit only establishes synchronization detection. Copying primary data into a new or stale backup follows separately. Assisted-by: Amp (AI coding agent)
A write can be interrupted while a backup is configured, leaving unfinished work in the journal. If the node then restarts without that backup, recovery must not wait forever for a store that was deliberately removed. Updates to existing keys were also not journaled. If the primary write succeeded but the backup write failed, TierStore did not retain the value needed to finish that update later. Separately, a matching primary generation alone does not prove a backup is current. If the local TierStore index has been replaced, the old index may have contained unfinished operations that were lost with it, and generation comparison alone would not detect this. In this commit, we: - Journal updates that must be written to both primary and backup storage. - Allow unfinished creates, updates, and removals to finish using only the primary store after restarting without the backup. - Rotate the primary synchronization generation first, ensuring that the missing backup is recognized as out of date when it is configured again. - Reopen an existing TierStore index even when the backup is no longer configured, so unfinished journal entries can still be recovered. - Store the index database identity alongside the primary generation in the backup completion record, and treat a completion from another index as requiring synchronization. - Add tests for failed updates, recovery without a backup, attempts to recover before backup synchronization has been initialized, completion encoding, and index replacement. A later commit will copy the current primary data into a new or outdated backup. Assisted-by: Amp (AI coding agent)
A newly configured backup starts empty, while a backup restored after primary-only operation may contain missing or stale values. Treating either backup as current could leave it unusable for recovery. In this commit, we: - Add exhaustive key enumeration to the dynamic store interface when tiered storage is enabled, while preserving the existing requirements for builds without tiered storage. - Recover pending journal operations before taking the primary-store snapshot. - Copy all durable primary values into the backup and remove values that no longer exist in primary. - Exclude ephemeral cache values from the backup and preserve TierStore's synchronization metadata during stale-value cleanup. - Write the backup completion record only after every synchronization step succeeds, ensuring interrupted attempts are retried safely. - Run required backup synchronization during node construction and fail the build if synchronization cannot complete. - Implement `MigratableKVStore` for existing test stores so the tiered-storage feature continues compiling and exercising their original behavior. - Add coverage for successful synchronization, backup-only configuration, journal recovery, stale-value removal, metadata preservation, and retries after failures at each stage. Assisted-by: Amp (AI coding agent)
|
I've assigned @tnull as a reviewer! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR closes #954 by ensuring a configured
TierStorebackup is synchronized before the node starts. It handles two cases:How it works
TierStorerecords the primary generation and local index identity with which a backup last synchronized.Back-filling a new backup
When a backup is configured for the first time, it has no completion record and is classified as requiring synchronization.
TierStorefirst recovers pending journal operations so primary reflects the authoritative durable state.Re-silvering a stale backup
When a node restarts without a previously configured backup,
TierStorerotates the primary synchronization generation before allowing pending operations to finish against primary alone. Updates are journaled alongside creates and removals so interrupted writes remain recoverable.When the backup returns:
TierStorecompares its completion record with the current primary generation and index identity. A mismatch, or replacement of the local index, marks the backup as stale.Both processes run during node construction. The build fails if a configured backup cannot be synchronized safely before the node starts. When tiered storage is enabled, exhaustive primary-key enumeration is provided through
MigratableKVStore. Tests cover successful synchronization and retryable failures.PR stack
This PR is the second part of tiered data storage and is "stacked" on top of #692 — Support tiered data storage, which introduces the underlying
TierStoreimplementation and native builder integration.