Repository navigation
Store keys hash_prefix/schema_prefix are accepted but not honored by writers or GC #1486
Description
Found during review of #1479 (GC). The per-store config keys hash_prefix, schema_prefix, and filepath_prefix are accepted and validated by settings (Config.get_store_spec → _validate_prefix_separation; also listed in storage_adapter.py COMMON_SPEC_KEYS) — but only <filepath@> consumes them, and only as reserved-namespace guards (builtin_codecs/filepath.py: filepath may not write under hash_prefix/schema_prefix; must stay under filepath_prefix).
Neither the hash writer (hash_registry.build_hash_path — fixed _hash/{schema}/... layout, now via the HASH_STORAGE_PREFIX constant) nor the schema-addressed writer (storage.build_object_path — {schema}/{table}/{pk}/...) nor GC honors them as relocation knobs.
Why this needs a decision
- A user who sets
stores.x.hash_prefix = "content"expecting relocation gets a validated, silent no-op — misleading config surface. - Worse, the trap is loaded: if a future change wires
hash_prefixinto the writer without teaching GC, hash objects land outside_hash/, get enumerated by the schema-addressed walk, match no schema reference, and are deleted as orphans — data loss. (fix(#1469): codec-driven GC reference discovery + file-level orphan matching #1479 added a comment onHASH_STORAGE_PREFIXflagging exactly this.)
Options
- Declare the layout fixed (recommended): document
hash_prefix/schema_prefixas what they actually are today — reserved-namespace declarations for<filepath@>cohabitation validation, not relocation settings — and rename or clearly re-document them (e.g. in the object-store-configuration spec). - Wire them through: honor the prefixes in
build_hash_path/build_object_pathAND ingc.list_stored_hashes/list_schema_pathsin the same change, with migration guidance for existing stores. Larger and of unclear demand.
Either way, the current state — validate-and-ignore — should not survive. Related: the <filepath@>-cohabitation GC hazard noted on #1479 (unattributable files) and the fail-safe planned with #1445/#1478.
Activity
fix(gc): honor filepath_prefix as a protected namespace; re-document …
dimitri-yatsenko commentedon Jul 6, 2026
Being resolved within PR #1479 per review (option 1 + GC-side teeth): the keys are re-documented as namespace declarations — not relocation settings — and
gc.list_schema_pathsnow skips the declaredfilepath_prefixsubtree so user-managed files can never be collected (regression-tested, incl. pinning the undeclared-cohabitation hazard). The relocation semantics is explicitly not supported;HASH_STORAGE_PREFIXremains the fixed layout. Will auto-close on merge of #1479.dimitri-yatsenko commentedon Jul 6, 2026
Correction to my comment above, from continued review:
get_store_specdefaultshash_prefixto_hashandschema_prefixto_schema(settings.py:633-634), so this wasn't pure validate-and-ignore — filepath validation did reserve_hash/by default. The sharper defects, both now handled on #1479:hash_prefixun-reserved the real namespace: the key is declarative (writers never consume it), so setting e.g.hash_prefix="content"replaced the default and left the actual_hash/layout unprotected — precisely when a user misread it as a relocation knob. filepath validation now always reserves the fixedHASH_STORAGE_PREFIXin addition to declared values (commit 6e139a3, with a unit test pinning the override case).schema_prefix's default_schemais vestigial — schema-addressed objects live in root-level{schema}/directories, so the reservation protects a location nothing writes to. Documented in the settings docstring; whether to retire the key remains open for whoever revisits store-spec surface.dimitri-yatsenko commentedon Jul 6, 2026
Final direction, per maintainer review (commit c74a830 on #1479): option 2 — the settings are authoritative. The documentation was the tiebreaker:
how-to/configure-storagedescribes the sections as configured viahash_prefix(default_hash/) andschema_prefix(default_schema/), andmigrate-to-v20gives the path format parameterized as{location}/{hash_prefix}/{schema}/{hash}. The code was out of spec, not the settings.Wired through:
put_hash/build_hash_pathtakehash_prefixfrom the store spec.build_object_path(codec + staged-insert) takesschema_prefixfrom the spec — new writes land under the documented_schema/section by default (previously root-level{schema}/…, which was the out-of-spec behavior).hash_prefixnow moves the writer and the reservation together, so the earlier hardcoded-reservation fix is superseded and reverted.HASH_STORAGE_PREFIXremoved (defaults remain as builder fallbacks). My earlier "fixed layout convention" comments are all corrected.End-to-end test pins a store with
hash_prefix="content"+schema_prefix="objects": writers place objects there, GC scans and collects there. Closes via #1479 as before.