Skip to content

feat(write): expose base snapshot IDs per partition and bucket - #517

Merged
SteNicholas merged 1 commit into
apache:mainfrom
lxy-9602:rt-write-base-snap
Nov 5, 2026
Merged

SteNicholas merged 1 commit into
apache:mainfrom
lxy-9602:rt-write-base-snap

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158

Expose the snapshot used by each partition-bucket writer, since lazy restoration can select different snapshots for different buckets.

Initialize the value from the actual restore result and advance it after successful real-time refresh. Partial refresh failures retain the previous value for failed buckets. The getter performs no storage I/O.

Tests

Added or extended coverage for:

  • Lazy restoration across snapshots and partitions, empty tables, ignored previous files, and missing writers.
  • Real-time refresh failures, retries, and shared contexts.
  • Partial bucket failures, concurrent refreshes, and snapshots with unchanged offsets.
  • Compaction refresh failures and retries.
  • Format-table rejection and deduplicate refresh preparation failures.

API and Format

Add the pure virtual API:

virtual Result<std::optional<int64_t>> GetBaseSnapshotId(
    const std::map<std::string, std::string>& partition, int32_t bucket) const = 0;

Returns nullopt when no snapshot has been used, and an error for missing bucket writers or format tables.

No storage format or protocol changes.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit 41ed4ea into apache:main Nov 5, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants