refactor(partitions): make purge recovery testable with SimStorage

[diegomrsantos]

Which issue does this PR address?

Closes #4239.
Relates to #4128 and #4130.

Rationale

The existing storage purge test uses its own cleanup sequence, while cluster simulation retains consumer offsets in memory across restarts. Testing recovery after power loss needs the production cleanup decisions and state reloaded from the simulated filesystem.

What changed?

Production purge now shares its offset cleanup and completion phase with a focused SimStorage harness through DurableStorage, preserving the current behavior after errors, including #4128. The harness starts after the message reset and reloads purge.gen, individual consumer offsets, and group offsets through shared production recovery code.

Controls cover both offset durability policies, cleanup of files absent from the live maps, and Next returning the complete fresh history after power loss. Fresh messages are durable before the crash, and their persistence leaves the separate offset directories untouched; the failure regression and checks of retries with acknowledged fresh writes remain followup work.

Local Execution

• Passed cargo fmt --all and cargo sort --no-format --workspace.
• Passed cargo clippy --offline --all-features --all-targets -- -D warnings across the workspace.
• Passed cargo test --offline -p journal -p partitions -p server -p simulator --lib: 1,012 passed, 7 ignored on macOS. The server suite ran outside the sandbox to allow its local HTTP listener.
• Passed taplo fmt --check over all tracked TOML files.
• Documentation generated with --cap-lints warn. The strict documentation link check encounters existing unresolved references in unmodified journal and server files.
• Pre-commit hooks are not installed; the checks above ran manually.

AI Usage

Codex generated the implementation and tests and drafted this description. Local verification is recorded above.

[codecov]

Codecov Report

❌ Patch coverage is 92.26006% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.73%. Comparing base (d1cb5ae) to head (b7336f4).

Files with missing lines	Patch %	Lines
core/server/src/partition_helpers.rs	71.42%	14 Missing ⚠️
core/journal/src/durable_storage.rs	89.18%	8 Missing and 4 partials ⚠️
core/partitions/src/iggy_partition.rs	84.48%	4 Missing and 5 partials ⚠️
core/server/src/offset_recovery.rs	85.45%	4 Missing and 4 partials ⚠️
core/partitions/src/offset_storage.rs	94.00%	6 Missing ⚠️
core/simulator/src/storage/purge.rs	99.62%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #4240       +/-   ##
=============================================
- Coverage     87.53%   70.73%   -16.80%     
- Complexity     1575     1576        +1     
=============================================
  Files          1283     1282        -1     
  Lines        223461   198202    -25259     
  Branches     186824   161567    -25257     
=============================================
- Hits         195605   140207    -55398     
- Misses        23141    53372    +30231     
+ Partials       4715     4623       -92     

Components	Coverage Δ
Rust Core	67.85% <92.26%> (-20.77%)	⬇️
Java SDK	68.70% <ø> (+0.01%)	⬆️
C# SDK	77.50% <ø> (+0.09%)	⬆️
Python SDK	90.97% <ø> (ø)
PHP SDK	85.67% <ø> (ø)
Node SDK	96.43% <ø> (+0.06%)	⬆️
Go SDK	70.11% <ø> (+0.05%)	⬆️

Files with missing lines	Coverage Δ
core/simulator/src/storage.rs	84.01% <100.00%> (+0.03%)	⬆️
core/simulator/src/storage/purge.rs	99.62% <99.62%> (ø)
core/partitions/src/offset_storage.rs	94.05% <94.00%> (+0.20%)	⬆️
core/server/src/offset_recovery.rs	83.22% <85.45%> (+3.00%)	⬆️
core/partitions/src/iggy_partition.rs	93.01% <84.48%> (+0.18%)	⬆️
core/journal/src/durable_storage.rs	83.60% <89.18%> (+2.54%)	⬆️
core/server/src/partition_helpers.rs	83.37% <71.42%> (+0.40%)	⬆️

... and 357 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[diegomrsantos]

Suggested review path: start with the two controls, then follow the shared production code they exercise. This PR connects purge completion and bookmark recovery to the existing DurableStorage trait, which DiskStorage and SimStorage already implemented. It prepares the testing foundation for #4239 while preserving the directory sync failure behavior in #4128.

A consumer offset is a bookmark: a saved offset of 2 means Next starts at message 3. Purge must clear those bookmarks so consumers can read replacement messages from the beginning.

The two test scenarios make this observable:

Scenario after power loss	Bookmark loaded into the new partition	Messages returned by Next
No purge happened	2	3, 4
Purge completed before fresh messages arrived	None	0, 1, 2, 3, 4

Both scenarios cover individual consumers and groups under Replicated and Persisted offset policies.

In the first test, the fixture writes and explicitly syncs the bookmarks and messages before the crash. No data loss is expected. “Recovery” means creating a new partition and loading its state from the surviving files. The simulated crash discards unsynced changes, preventing recovery from relying on cached writes. This control also rejects a recovery implementation that always returns empty bookmark maps. Because the fixture explicitly syncs under both policies, it does not claim that every ordinary Replicated write survives power loss.

In the second test, the partition starts at the purge completion phase, with message history already reset and old bookmarks still present. It also creates bookmark files absent from the live maps, requiring the directory sweep to find them. Assertions check that completion clears the live bookmarks, durability tracking, and files. After saving fresh messages and simulating power loss, the test reloads the completion marker and consumer state into a new partition, then checks the actual messages returned by Next. Reusing offsets 0 through 4 is intentional: stale bookmark 2 would still be within range but would silently skip the first three messages.

For the production changes, the useful review points are:

1. Purge completion: production passes DiskStorage; the harness passes SimStorage to the same cleanup and completion method. Check that the extraction preserves ordering and error handling, especially continuing after offset deletion or directory sync failures. The caller must already have reset message history and excluded concurrent writes.
2. Offset persistence: the existing disk entry points delegate to helpers accepting storage. Check record formats, replacement ordering, and who owns directory sync. File sync preserves contents; directory sync preserves changes to names. Syncing the purge marker's parent does not sync the separate consumer and group directories.
3. Consumer recovery: the harness uses the server's loader and map initialization rather than retaining old maps. Check that validation, clamping, and persistence tracking still follow the existing boot behavior. The directory scan adapter preserves the disk scan's filtering and bounded worker behavior.
4. Fresh message persistence: it syncs the message journal and its parent, leaving the offset directories untouched. A global simulated writeback here could accidentally make a failed bookmark deletion durable and conceal the intended future failure case.

These are controls for successful cleanup and recovery. They exercise production bookmark logic against simulated storage, but do not inject sync failures, exercise the message reset phase or purge retries, or run full server startup. The known bug can therefore remain while both tests pass. The remaining regression work must introduce the relevant directory sync failure and check recovery and retries, including preservation of acknowledged fresh messages.

[hubcio]

@diegomrsantos please rebase - i introduced a small-ish server refactor in #4089

[hubcio]

@diegomrsantos could you please analyze if this problem with purge is related to failing bdd tests in #3608 ?

[diegomrsantos]

@hubcio could we track the investigation into #3608 in that PR? This PR focuses on making purge recovery testable and preserves the current behavior.

Apart from the requested rebase, is there anything else you’d like changed here before merging?

[hubcio]

@diegomrsantos i will check this PR in ~1h

[hubcio]

looked at the purge and offset recovery extraction. the shared cleanup, completion and recovery paths keep the order and the error handling from master, and both new controls run under both durability policies against the simulated filesystem. one warning stands, and it is the #4128 shape: the offset directory sweep is silent on a failed scan and still records the generation, so the fix belongs there and not in this refactor. the rest are small cleanups.

[diegomrsantos]

Thanks for the reviews, @hubcio and @mmodzelewski!

[diegomrsantos]

@hubcio @mmodzelewski could one of you rerun the C++ BDD job? It failed with a 504 while downloading a Bazel dependency, before the tests started. I don't have permission to rerun it.

[hubcio]

i will restart once all jobs are finished