Skip to content

proof: raise MAX_PROOF_DEADLINE_S_CEILING to 14400 (B1 staging) - #300

Draft
echobt wants to merge 1 commit into
droid/9822d526-sn100-100-live-gaps-p1b-disafrom
droid/933f76bf-b1-raise-max-proof-deadline
Draft

echobt wants to merge 1 commit into
droid/9822d526-sn100-100-live-gaps-p1b-disafrom
droid/933f76bf-b1-raise-max-proof-deadline

Conversation

@echobt

@echobt echobt commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Why

Staging Gate1 (Arch B1) needs a topic whose signed
eval_executor.max_proof_deadline_s is ≥ 14400 s for the measured Harbor
first-5 pack. The live tip binary refuses it and crash-loops
proof-challenge:

max_proof_deadline_s_ceiling 14400 must be 1..=7200

The lock is crates/proof-task/src/executor.rs
pub const MAX_PROOF_DEADLINE_S_CEILING: u64 = 7_200;, also read by pin
validation (ProofPin::validate_executor) and by the topic
eval_executor deadline check (TopicEvalExecutor::validate).

What changed

  1. MAX_PROOF_DEADLINE_S_CEILING 7200 → 14400 (four hours), with its
    doc comment updated.
  2. Committed config/proof-pin.toml default ceiling → 14400.
  3. Every test / fixture / assertion that hardcoded 7200 as the crate
    lock maximum
    moved with it. Boundary probes that meant "just over the
    ceiling" now use 14401 (and 14400 where they meant "exactly at the
    ceiling"):
    • crates/proof-task/src/executor.rs (constant + lock test)
    • crates/proof-task/src/pin.rs (TOML round-trip fixture; the
      tighten-never-loosen test now asserts the crate lock itself is legal,
      then still asserts lock + 1 and 0 are refused)
    • crates/proof-task/tests/committed_pin.rs
    • crates/proof-executor/src/lib.rs, crates/proof-executor/src/plan.rs
    • crates/proof-http/src/lib.rs (public /v1/status + /v1/proof/executor)
    • crates/proof-harvest/src/lib.rs (override-refusal message)
    • xtask/src/proof_executor_offer.rs (ceremony helper refusal)
  4. Docs / env example follow the ceiling: docs/PROOF.md,
    docs/COMPLETENESS.md, docs/external-miner/proof.md,
    docs/runbooks/proof-submit-e2e.md,
    deploy/env/proof-challenge.env.example.

Not changed: the topic / offer deadlines themselves (7200 stays the
example offer deadline and the live offer's own value), the
tbench miner doc's live topic row, ctx's 7200 s submit wait,
PROOF_PROXY_DEADLINE_SECS, and the Lium retry-hint clamp — those are
per-request budgets, not the pin lock. Gate5/6 behavior is untouched: no
Harbor seal logic, no metal, no topic documents, holdouts, or RLM install
path.

Semantics preserved

Tighten-only is unchanged. A pin may still only lower the ceiling, a
topic may only lower the offer's deadline, and the live offer may be
shorter than both. Only the outer bound moved.

Verification

  • cargo test -p proof-task -p proof-executor — pass.
  • cargo test --workspace --no-fail-fast — only 4 failures, all
    pre-existing and environment-caused: this sandbox runs as uid 0, so
    chmod 000 on a fixture directory does not deny reads
    (CAP_DAC_OVERRIDE). Each passes when run as an unprivileged user
    (setpriv --reuid=65534): seed_pf_allocator_refuses_boot_when_a_topic_dir_cannot_be_read,
    artefact::tests::max_zip_numeric_id_fails_closed_when_a_topic_dir_cannot_be_read,
    agent_tests::deadline_cut_still_persists_work_tree,
    agent_tests::paid_run_fails_closed_when_work_tree_cannot_be_synced.
    None are touched by this change.
  • cargo fmt --all -- --check — clean.
  • cargo clippy --workspace --all-targets -- -D warnings — clean.
  • xtask loc-cap / consensus-lint / spec-check / design-check /
    external-docs-check — all OK.

Merge status

DRAFT. Do not merge, do not squash, do not enable auto-merge — Mathis
Merge HOLD until Owner staging A→Z 6/6.

Gates

  • Greptile review (@greptileai review if silent)

The pin hard-lock refused `max_proof_deadline_s_ceiling > 7200`
(`max_proof_deadline_s_ceiling 14400 must be 1..=7200`), which crash-looped
proof-challenge on a host whose pin asked for 14400 and blocked staging
Gate1 from setting the topic `eval_executor.max_proof_deadline_s >= 14400`
the measured Harbor first-5 pack needs.

Raise the crate lock to four hours (14400) and move every test / fixture /
committed-pin assertion that hardcoded 7200 as the *ceiling* with it. The
tighten-only rules are unchanged: a pin may still only lower the ceiling,
a topic may only lower the offer deadline, and the live offer may be
shorter than both. Boundary probes now use 14401 as "over the ceiling".

Docs, the committed `config/proof-pin.toml` default ceiling, and the
deploy env example follow. No Harbor seal logic, topic document, holdout,
or RLM install path is touched.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

B1 staging unlock: 7200 → 14400 in crates/proof-task/src/executor.rs, plus the committed config/proof-pin.toml ceiling and every test/fixture that hardcoded 7200 as the crate lock maximum (boundary probes now use 14401).

Tighten-only semantics are unchanged: a pin may still only lower the ceiling, a topic may only lower the offer deadline, and the live offer may be shorter than both. Only the outer bound moved.

Draft / Merge HOLD — do not merge, do not squash, no auto-merge.

@echobt

echobt commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Correction to the line above (the constant name was eaten by shell interpolation):

B1 staging unlock: MAX_PROOF_DEADLINE_S_CEILING 7200 → 14400 in crates/proof-task/src/executor.rs, plus the committed config/proof-pin.toml ceiling and every test/fixture that hardcoded 7200 as the crate lock maximum (boundary probes now use 14401).

@greptile-apps

greptile-apps Bot commented Sep 15, 2026

Copy link
Copy Markdown

Greptile Summary

This change permits longer Proof runs, but synchronous submission still stops waiting after the former two-hour window. A valid long-running submission can report failure to the caller even when scoring completes and the result is stored later. Do not merge until the gateway and default client submission waits are aligned with the supported run duration. The requested Draft / Merge HOLD remains in effect.

Confidence Score: 4/5

Not safe to merge: a supported long-running Proof submission can return a timeout despite completing successfully afterward.

One reproduced reliability failure remains in the synchronous submission path: the accepted execution duration exceeds the unchanged gateway and default-client wait budgets.

Files Needing Attention: crates/gateway-core/src/proxy_detach.rs and crates/ctx-client/src/lib.rs need their synchronous Proof submission timeouts aligned with the duration admitted by crates/proof-task/src/executor.rs.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex posted a finding-comment-proof for a P1 finding and linked it to the corresponding review comment.
  • T-Rex captured and organized artifacts for the P1 finding, including the Rust source and shell script used in the proof, along with related logs.
  • T-Rex completed a general-contract-validation-proof showing a deadline discrepancy between the executor's four-hour deadline and the configured constants, resulting in a 504 upstream timeout after a 7,201-second simulated score.

View all artifacts

T-Rex Ran code and verified through T-Rex

Comments Outside Diff (1)

  1. General comment

    P1 Four-hour Proof offers can time out synchronously at the 7,200-second gateway boundary

    • Bug
      • crates/proof-task/src/executor.rs:25 raises MAX_PROOF_DEADLINE_S_CEILING from 7,200 to 14,400 seconds, and the current executor-plan contract accepts a 14,400-second deadline. The Proof submission route remains synchronously awaited by the public gateway. Its detached hop times out at 7,200 seconds with HTTP 504 Gateway Timeout and body upstream deadline; the default ctx client uses the same 7,200-second submit timeout. Executed contract evidence shows that a spawned score finishing at 7,201 seconds persists after the gateway returns 504, so the client does not receive the expected post-score 201 Created.
    • Cause
      • The allowed proof execution deadline was raised without raising or removing the synchronous gateway timeout (crates/gateway-core/src/proxy_detach.rs:16,67-70) and the default ctx submit timeout (crates/ctx-client/src/lib.rs:28,99,220-223). The submit handler deliberately awaits scoring for 201-after-score while detached scoring survives cancellation.
    • Fix
      • Align all synchronous submission wait budgets with the accepted executor deadline (at least 14,400 seconds, plus any required transport margin), including PROOF_PROXY_DEADLINE_SECS and DEFAULT_SUBMIT_TIMEOUT_SECS; or change the API contract to return an accepted/queued response before scoring and provide durable polling. Add an integration test covering an accepted deadline above 7,200 seconds through gateway and ctx-client behavior.

    T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "feat(proof): raise MAX_PROOF_DEADLINE_S_..." | Re-trigger Greptile

pub const MAX_PROOF_DEADLINE_S_CEILING: u64 = 7_200;
/// topic may declare (four hours). A pin may only tighten it below this
/// lock, and the live offer may be shorter still.
pub const MAX_PROOF_DEADLINE_S_CEILING: u64 = 14_400;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Align submission timeouts

This change permits Proof deadlines up to 14,400 seconds, but synchronous submission still has a 7,200-second gateway deadline and the default client uses the same limit. A valid proof that finishes after two hours returns a gateway or client timeout instead of the expected 201 Created, even though its detached scoring work can finish and persist afterward. Raise those submission budgets to cover the supported deadline range, including any transport margin, or change the submission contract to acknowledge work before scoring completes.

Knowledge Base Used: Proof tasks, challenge service, and scoring

Artifacts

Command output from the check

Command output from the check

  • Executed the authored paused-time gateway contract test against the PR state; it reports HTTP 504 Gateway Timeout with `upstream deadline` while a 7,201-second detached score persists, confirming the mismatch.

Evidence from the check

  • The authored Rust test invokes the real gateway detached Proof timeout function and verifies its 504 response and post-timeout score persistence.

Evidence from the check

  • The authored shell command temporarily installs the narrow test with Tokio test time enabled, runs it, and restores repository files afterward.

Command output from the check

  • Executed the current executor plan boundary test, which includes acceptance of a 14,400-second override, confirming the changed admission side.

View artifacts

T-Rex Ran code and verified through T-Rex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant