perf(storage): stop producing and reading the UUID membership index

[DecisionNerd]

Closes #1902

Slice of #1881, decision 5. New generations no longer produce or read the UUID membership index (topology/uuid-membership/manifest.json, identities-v5-*.uuidx, node-surrogates-v5-*.uuidx, topology-receipt.json). It duplicated the UUID-ordered node_uuid/edge_uuid columns of the published Parquet at 17-25 B/edge. Under the pre-v1 policy the format changes in place; a project that still carries the files opens normally and the files are ignored. The ordinal node-identity facet that shares the directory is unchanged.

What changed

• TopologyIdentityProbe (topology_identity.rs) answers every identity question from the published topology Parquet. A sorted, deduplicated candidate set prunes row groups by min/max, then pages by the Parquet column index, and decodes only the selected pages of node_uuid (+ node_id) or edge_uuid. UuidProbeMetrics gains pages_considered and pages_read; TopologyWriteWork carries both.
• Node and edge UUIDs share one namespace. Validation and the writer commit refuse a UUID that is a live node, a live edge or a deleted entity, and refuse a missing edge endpoint.
• Deleted UUIDs stay spent through topology/deleted_identities.parquet (sorted, unique, carried forward by each generation that deletes; a graph that never deletes never has one). It replaces the index's tombstones.
• The bulk builder no longer spills sorted edge UUIDs to scratch for the index, and the staged encoder keeps no retained parent index. Both encode only the ordinal facet.
• A footer-cache bug the move exposed: hydration hard-links one object under many workspace paths, and a cache hit returned a fragment pointing at the first workspace's (possibly deleted) path. A hit is now re-pointed at the path asked for (regression test a_cached_footer_never_sends_a_probe_to_the_path_it_was_first_read_from, red before the fix).
• Probe hardening from review: a fragment replaced after its footer was read is refused when decoded (pruning metadata would no longer describe it), a UUID held by two rows is a typed error, and absent or truncated statistics mean "may hold" (all tested, each test killed by a mutation). The probe takes no lock: it is advisory for validation and authoritative at commit, where the rewrite lock refuses a commit whose probed generation moved.
• G500 certification evidence drops the index's fsync/write counters (scripts/ci/schemas/g500-certification.schema.json, its validator test, scale_g500_ladder.rs), and the ladder treats uuid_and_surrogates as node-bearing.

Reader inventory (every consumer of the index on main, and where it reads now)

Reader	Now
Writer endpoint lookup (register_existing_endpoints) and commit validation (uuid_membership/topology_delta.rs)	TopologyIdentityProbe + deleted-identities record
Staged shaping/encoding parent snapshot and retained artifacts (graph_construction.rs, shape.rs, graph_construction_encoding.rs, inventory.rs)	removed; nothing retained
Bulk builder identity stream and edge-UUID scratch (bulk.rs, bulk/identities.rs, bulk/scratch_edges.rs)	removed
Index reader, rebuild, construction encoder, orphan GC (uuid_membership/{probing,rebuild,construction,maintenance}.rs)	removed; ordinal-facet paths kept
label_membership_counts.rs (labels by UUID)	probe over the pinned TopologyFiles
ordinal_identity_v4.rs discovery freshness check, project_generation.rs rebuild-if-missing, runtime_entity_labels.rs, mutator.rs	removed / probe
API node selectors by UUID or handle (node_selector.rs, added by #1926)	cached_identity_probe; the 1M-capped scan stays for label and property selectors only
API bulk validation (bulk_construction.rs), branches/import_graph.rs, graph_publication.rs, composite_publish.rs, maintenance.rs, resumable_construction.rs, lib.rs cache field	probe, or rebuild-if-stale removed
Search node identity (graphforge-search/src/node_identity.rs)	ordinal authority; without one, rows are checked for repeats only
Hydration/materialization, export, verify, import (graph_admission.rs, staging.rs, durable_rewrite.rs, graph files inventory)	no reader; legacy files are ordinary inventory entries; new generations carry none

Remaining references to uuid-membership/ are the ordinal facet (ordinal-v4-*, forward-v4-*, tombstones-v4-*).

Acceptance evidence

• No membership artifacts in new generations. neither_bulk_route_publishes_a_membership_index (memory and scratch routes), the staged-encoder inventory assertion in encoding_publication/tests.rs, and a_new_project_carries_no_membership_index_and_round_trips.
• Refusals. commit_refuses_a_uuid_the_published_topology_already_holds (node/node, node/edge, edge/edge, edge reusing a node UUID, at the writer commit), the bulk_construction publication/normalization tests (duplicate node and edge, edge equal to node, missing endpoint, deleted UUID), and the legacy and new-project API tests.
• Page pruning, shown with counters and with bytes read.
  • Fixture of 4 row groups x 10 pages written with the production Parquet properties: a candidate set inside one page reads 1 of 40 pages; two row groups, 2 pages; a value in the gap between pages reads 0; every page, 40; a file without a page index falls back to row-group pruning.
  • Through the production writer: 45,000 nodes, 3 pages; endpoints in one page pages_read=1/3 (identity_bytes_read=106,588), endpoints in the first and last page 2/3 (115,990; the dictionary page is read once per probed chunk).
  • Real reads: the probe is attributed to the caller's lifecycle capture (rayon workers now carry it). One page read 1,510 bytes, every page 21,862, of a 26,312-byte file (page_pruning_reduces_the_bytes_the_file_system_serves).
• Reopen and recovery. legacy_membership_index (real pre-change project: opens, appends, refuses, deletes, exports, verifies, imports, deleted UUID still spent after the round trip), the construction crash tests (ordinal_encoding_crashes_recover_every_durable_boundary, bulk_builder killed-in-any-pass tests, lane crash tests), and identity_first_touch (a flipped node Parquet is refused by the commit that probes it).
• Portable round trip. The new-project test exports a bundle, asserts the tar names no index file (the same check finds them in the legacy bundle), verifies, imports, and refuses a reused UUID on the import.

Mutation proofs (each applied to the committed tree, run, reverted)

Mutation	Result
taken() ignores live nodes and edges	6 tests fail (writer refusal, 2 bulk publication, deleted-identity probe, legacy, new project)
API live_nodes reports every candidate live	5 fail (missing-endpoint normalization and publication, deleted-identity probe, legacy, new project)
writer defaults an absent surrogate to 0	1 fails (wave13_validation_display_and_disappeared_endpoint_are_structured)
deleted-identity record ignored	3 fail
edge fragments never probed	5 fail
page-index pruning disabled	6 of 10 probe tests fail
row selection dropped (counters unchanged)	the bytes-served test fails
probe resolves nothing	5 probe tests fail
crashed-attempt ordinal residue not cleared	ordinal_encoding_crashes_recover_every_durable_boundary fails

Verification

• make check: exit 0.
• graphforge-storage nextest: 1796 passed, 0 failed, 9 skipped. graphforge-api nextest: 1668 passed, 0 failed, 11 skipped (includes fix(api): resolve UUID selectors by identity and keep single-pass algorithms off the iteration budget #1926's analyst_verb_limits: BFS from a UUID on 1,000,100 nodes). API BDD: 118 scenarios passed.
• Run with GF_TEST_TMPDIR=/tmp/...: the launcher's default root sits inside the enclosing checkout when the worktree lives under .claude/worktrees, which makes 17 repository::* tests see Git-tracked data and one socket test exceed SUN_LEN. Those pass with a short ext4 root; they are environmental.
• Python/Node bindings were not rebuilt or run: no binding source changed. The public change is to Rust-internal graphforge-storage exports (UuidMembershipIndex and the index maintenance functions are gone; TopologyIdentityProbe is new), and the workspace compiles under clippy.

Append-validation time (acceptance 3), contended host

S20 graph (1,048,576 nodes, 16,777,216 edges) receiving an S18-sized append (262,144 nodes, 4,194,304 edges, existing endpoints), gf import-session validate and commit, base 171120cbd against this branch, release builds in separate target dirs (binary sha256 differ), 5 alternating pairs, host load average 14-27 on 16 cores. Full method, commands, digests and per-step load are on the issue: #1902 (comment)

pair	validate+commit base -> candidate	per edge (us)
1	564.5 s -> 66.7 s (-88%)	134.6 -> 15.9
2	156.6 s -> 64.8 s (-59%)	37.3 -> 15.5
3	158.0 s -> 74.7 s (-53%)	37.7 -> 17.8
4	172.0 s -> 76.8 s (-55%)	41.0 -> 18.3
5	169.7 s -> 58.4 s (-66%)	40.5 -> 13.9

The candidate was faster in all five pairs, in both orders, on validate, commit and the sum, and in four pairs its mean load was the higher one. Per-edge ingest cost did not regress. This is contended-host evidence for the direction, not a size: a quiet-host run is still needed for a number to quote. While measuring I found that main's bulk builder fails S20 builds on a nearly full disk (captured encoded source identity or length changed: the installer records an encoded shard's allocated bytes before writeback, which can add an extent block); it is not caused by this PR and is described on the issue.

Accepted behaviour changes for projects written before this change

Both are accepted under the pre-v1 in-place format policy and documented in docs/book/architecture/uuid-membership-index.md:

• A UUID deleted before the upgrade becomes reusable: the deleted-identities record holds only deletions made since. Entities deleted afterwards stay spent.
• Search verifies nodes by repeat-check when a project has no ordinal facet.

The stale index files in an upgraded project are inert: no code reads them, orphan collection considers only canonical ordinal artifact names, and they stay in the graph-files inventory (and travel through hydration, export, verify and import) until a future cleanup drops them.

Follow-ups

• Probed chunks read their dictionary page; noted on epic(storage): build initial graphs in parallel passes derived from the published generation #1881.

🤖 Generated with Claude Code

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 2375bf24-660d-4f74-80aa-8fe1660cb6af

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}