feat(proof): SN100 live gaps — publish path, topic disable, P1b allocator (P1b, draft)

[echobt]

DO NOT MERGE. Draft only, by Owner instruction (Mathis LOCK 2026-09-15 SN100).
Keep this draft — and #297 / #298 — open and unmerged until the Owner's 100% LIVE
gate passes and the Owner gives an explicit GO. Auto-merge is forbidden and none
is configured.

What this is

P1b: the five gaps between the P1a RLM topic-install work and a 100% LIVE dynamic
topic. Stacked on droid/9f68584e-sn100-p1a-rlm-topic-install (#298) — review
that first; this branch is its delta (9 commits) on top of b735f335.

HEAD: 37fa0920

The five gaps

#	Gap	Where it landed
1	Publish path without a localhost rewrite proxy	87e32024 — gateway forwards the one publish route; the challenge serves its own prefix
2	topic disable fail-closed	b6d84340 — proof_topic_gate (migration 0026), CLI, submit-path 403
3	P1b allocator: vms_per_submission=1, concurrent 2-submit ⇒ 2 VMs	f703108e, 37fa0920 — pin enforced at submit; per-submission leases; shared-VM path serialized
4	Strip remaining tbench hardcode in challenge/gateway/orch	ab1b5320 — results contract, harness default, router guard test
5	Baseline → scorable wiring (install without --skip-baseline)	8805b849, afa8c815 — topic baseline / topic seal

1. Publish through the gateway, no rewrite proxy

proof-admin topic install publishes to POST /challenge/proof/v1/admin/proof/topics.
Both hops refused it: the gateway answered 403 for every /v1/admin/* path, and
the challenge's topic mux swallowed /challenge/proof/… as a topic id with no
registered routes (404). The operator's workaround was an ephemeral rewrite proxy.

• Gateway forwards exactly one admin route: POST, the publish path, Proof
  challenge id, and a well-formed Authorization: Bearer <token>. The scheme is
  required — a bare value, another scheme, or a lowercase scheme is a 401. (The
  challenge's own admin_ok still accepts a bare token on a master-local call;
  that is a different surface, and the gateway is the public edge.) Every other
  admin route, the topic-id form, and a GET keep their 403.
• Challenge serves its own routes under its own prefix (/challenge/proof/…), so
  one publish URL works against the gateway and the service directly.
• Topic routes may not claim v1/admin… (refused at install, never resolved by
  the mux), so no topic route can read like an operator route.

2. topic disable is a real switch

The CLI commands were stubs that exited 3, so the only way to stop a topic taking
submissions was to re-sign its document — a ceremony that does not wait for an incident.

• proof_topic_gate (migration 0026) is an append-only journal: newest row per topic
  is the state, the rows before it are who/when/why. enabled clears a disable.
• The CLI appends the row and refuses an id that is not published (a typo cannot
  silently "disable" nothing); topic show prints the gate.
• The challenge reads it on POST /v1/submissions before the nonce is spent: a
  disabled topic is a 403 naming the reason, an unreadable gate is a 503, and a
  re-submit after topic enable lands with the same signature and nonce.
• GET /v1/proof/topics and /v1/proof/topics/{id} carry disabled
  (+ disabled_reason).

3. One VM per submission, and two submissions at once

vms_per_submission = 1 was recorded in the install journal and read by nothing; the
scorer held a per-topic lease for a whole run, so two submissions to one topic
were a queue.

• The submit path reads the pin from the topic's newest install row and refuses any
  other value with 503, no row, nonce unspent. A topic with no install row is not
  refused for that.
• RlmScorer leases per submission ((topic_id, digest)); the topic's shared
  state (lifecycle, best pointer) is written under a separate topic lock taken only
  for a write, so a paid run never blocks another run.
• Where the second VM comes from: a topic that selects an in-guest runner gets one
  experiment VM per paid job, so two submissions are two VMs in parallel. A topic that
  selects none has a single VM shared with the cheap jobs, and the KVM host refuses a
  second concurrent job on one VM — so that path takes a per-topic lock for the whole
  run (inspect included) and the second submission waits rather than racing into a 409.
  For the Owner's LIVE 2-VM check this means the topic must select an in-guest runner
  (the tb4 bundle shape does).
• The lifecycle is a phase marker, not a per-run counter: a second submission while
  one is evaluating keeps evaluating, and the scorer's in-flight count decides
  whether a persisted evaluating is stale or live.
• The promotion compare-and-swap still refuses a crown whose best pointer moved.

4. tbench hardcode in scoring/routes

The scoring path had one true topic-shaped default: the Python harness defaulted an
absent results_contract pin to tbench-harbor-v1. It now defaults to the generic
harbor-trials-v1; the tbench-named id stays accepted because it is a wire value
a signed document may pin (and a signed document cannot be edited). No production Rust
branches on a topic id — gateway-core has a test asserting the router mentions none.

5. Baseline → scorable

topic install --drive-rlm (without --skip-baseline) left the topic at baselining
and nothing could turn that into a scorable topic.

• topic baseline <id> reads the stored measurement and prints the primary and the
  metrics_commitment an open document must carry (computed from the stored run
  and the pin's image digest).
• topic seal <id> --document <open.json> [--publish] hands that measurement to
  TopicSetup::mark_sealed — the same call the runtime's tests drive — then publishes
  through the admin route. The seal is wired to an unwired orchestrator and a
  refusing key probe, so sealing can never provision anything.
• A failed publish is retryable: re-running the same command detects "already open
  under this document" and publishes without re-running the lifecycle move. A draft, a
  different signature, or any other lifecycle still goes through mark_sealed and
  still gets its refusal.
• The install prints Scorable / NOT scorable yet — <what is missing> and --json
  reports scorable + remaining.

Greptile

Three review rounds; four P1s raised, all fixed with a test that fails without the fix:

Finding	Fix
Gateway forwarded a non-Bearer Authorization value	scheme + non-empty token required (9515ad3f)
TTL reaping leaked the in-flight count, pinning the topic	reaping releases the count with the lease (37fa0920)
A failed publish could not be retried through topic seal	already-sealed detection skips only the lifecycle move (37fa0920)
Two paid submissions without an experiment binding shared the topic VM	shared-VM path serialized per topic (37fa0920)

Latest review: 5/5, no outstanding issue.

Verification

• cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings — clean
• cargo test --workspace — 76 suites pass; two failures are pre-existing on
  b735f335 (chmod 000-based tests that root bypasses, and cargo deny advisories)
• xtask loc-cap, consensus-lint, spec-check, design-check, external-docs-check — all pass
• New tests include: gateway publish forwarding (incl. the bare-header 401), challenge-prefix
  routing, admin-namespace reservation, disable→submit→enable with an unspent nonce, the
  listing flag, the allocator pin, two concurrent submits ⇒ two runs, two experiment VMs
  per topic, shared-VM serialization (fails with vm vm-0 is running a job when the lock
  is removed), the in-flight reap balance, and the seal commitment round trip.

LOC cap

Three crates went over the 1500-line cap, so ab1b5320 extracts per the repository's
own convention (gateway-core::admin_route / ::proxy_paths, proof-http::operator,
proof-submit::SubmitBody, and the new proof-topic-ops). No behaviour change.

Out of scope (Owner LIVE, after this tip)

Owner A→Z: baseline measured on metal, miner → pf_, the 2-VM check (needs a topic that
selects an in-guest runner), and disable fail-closed against the live topic. This branch
spends nothing and tips no metal.

[echobt]

@greptileai review

Draft (do not merge). Stacked on #298. Gap 1 of 5 in this push: the operator publish now reaches the challenge through the gateway (and through the challenge's own prefix) without a rewrite proxy; see the PR body for the exact gates and tests.

[greptile-apps]

Greptile Summary

The reviewed fixes address the previously reported authorization, publish-retry, abandoned-run accounting, and shared-VM concurrency behavior. The author has marked this pull request as a draft; it should not be merged until it is marked ready.

Confidence Score: 5/5

The reviewed findings are addressed; once the pull request is marked ready, no outstanding reviewed issue blocks merging.

The previous gateway finding is fixed: public operator-route forwarding now requires the Bearer scheme and a non-empty token. The previous publish-retry finding is fixed: retrying publication for the same signed document already open in the local lifecycle does not repeat the lifecycle transition.

_{Reviews (3): Last reviewed commit: "fix(proof): three concurrency/retry defe..." | Re-trigger Greptile}

[greptile-apps]

Require Bearer Scheme

The gateway treats any nonempty Authorization header as an operator bearer because it falls back to Some(raw) when the value does not start with Bearer . As a result, Authorization: operator-token forwards the newly exposed publish request to Proof even though the gateway contract requires a nonempty Bearer authorization value. Require the Bearer prefix and a nonempty token suffix before forwarding.

How this was verified: An in-process gateway request with a raw Authorization value reached the mocked Proof upstream and returned 201.

Artifacts

Gateway administrative publish contract script

• Authored executable script that creates isolated base and PR-head worktrees, injects the in-process Axum/Wiremock probe, and captures both executed runs; it is the exact validation source used.

Gateway publish behavior before change

• Captured base-commit output for requests through an in-process gateway and mocked Proof upstream; all publish and administrative variants return 403 before the change.

Gateway publish behavior after change

• Captured PR-head output showing intended routing behavior and forwarding of a non-Bearer Authorization value.

_{Ran code and verified through T-Rex}

[echobt]

@greptileai review

All five gaps are now in this branch (7 commits). The P1 you raised is fixed in 9515ad3f: has_operator_bearer now requires the Bearer scheme with a non-empty token — a bare value, another scheme, and a lowercase scheme are all a 401 — and both the unit test and the proxy integration test drive those cases.

Also in since your last review: the LOC-cap refactor (ab1b5320) that extracted gateway-core::admin_route (where that predicate now lives), gateway-core::proxy_paths, proof-http::operator, proof-submit::SubmitBody, and the new proof-topic-ops.

Still a draft: do not merge.

[greptile-apps]

Failed publishes cannot retry

The seal is committed locally before the remote publish is attempted. If publishing fails, the topic is already open locally but remains unpublished remotely. Retrying topic seal --publish calls mark_sealed again, which rejects the now-open topic before the publish request can run, leaving the operator without a recovery path.

Artifacts

Evidence from the check

• The authored script runs the focused lifecycle integration test that performs the first seal and the rejected retry, ending with the exact assertions exercised.

Command output from the check

• The initial command capture ran the focused real lifecycle test successfully, showing the executable test and its zero exit status.

Command output from the check

• The final command capture ran the same focused test successfully and prints the assertions that first open/version-2 transition succeeds and a second seal is illegal, establishing the unrecoverable retry state.

_{Ran code and verified through T-Rex}

[echobt]

@greptileai review

All three P1s are fixed in 37fa0920, each with a test that fails without the fix:

1. Reaping leaked the in-flight count — reap_abandoned now releases the count with the lease. Test: after a reap the count is 1 (was 2), then 0, then a fresh run is admitted.
2. Failed publish could not be retried — topic seal detects "already open under this document" and skips mark_sealed for that case only; a draft, a different signature, or any other lifecycle still goes through it and still gets its refusal. SealOutcome::already_sealed is on the wire for --json.
3. Two paid submissions shared one VM — RlmScorer serializes the shared-VM path on a per-topic lock held for the whole run (inspect included). An experiment topic takes no lock, so its submissions stay parallel.

For (3) the test fake now models the host's one-job-per-VM rule and can slow a job; with the lock removed the test fails with vm vm-0 is running a job, and passes with it.

Still a draft: do not merge.