diff --git a/docs/external-miner/README.md b/docs/external-miner/README.md index 350bc173f..9cb304ae7 100644 --- a/docs/external-miner/README.md +++ b/docs/external-miner/README.md @@ -46,7 +46,7 @@ always the live list. Topics that additionally have a written miner guide here: | Topic | Guide | What it scores | |-------|-------|----------------| -| `tbench` | [proof-tbench.md](./proof-tbench.md) | `custom` topic on `success_rate`, `discovery` payout, Firecracker sister, miner-supplied model key | +| `tbench` | [proof-tbench.md](./proof-tbench.md) | `custom` topic on `success_rate`, `discovery` payout, **scoring on** (experiment VM, not deferred). Miner-supplied OpenRouter key. Read live `ctx proof status` | Emission: `bounty` 2000 bps, `proof` 8000 bps (sum 10000). Off challenges have no row and earn 0. diff --git a/docs/external-miner/proof-tbench.md b/docs/external-miner/proof-tbench.md index 290166c6c..a4a27002d 100644 --- a/docs/external-miner/proof-tbench.md +++ b/docs/external-miner/proof-tbench.md @@ -17,29 +17,44 @@ Every number, digest, and rule below is **topic data**: it comes from a signed document an operator publishes and can re-publish at any time. Nothing about `tbench` is compiled into the network binaries. Read the live document before you spend anything; if this page and the live document disagree, the live -document wins. - -## Status right now: submits are accepted but not scored - -`tbench` sets `constraints.params.defer_scoring = "true"`. That puts it in -`deferred_topics` on `GET /challenge/proof/v1/status`: the host **accepts** your -submission and stores it as **`queued`** (**201**), and then evaluates nothing. -No pod is rented, no VM boots, no judge call is made, and no mass is recorded -until the operator lifts the flag and drains the queue, oldest first and one at -a time. - -So today: - -- A well-formed submit is **201 `queued`**, not a score. -- `can_score` on `/challenge/proof/v1/status` is **`false`** and - `scorable_topics` does **not** list `tbench`. That is expected for a deferring - topic and is not an outage — a topic in `deferred_topics` is the one case where - `can_score: false` does not mean **503**. -- A queued row is the only non-terminal state. `ctx proof show --wait` keeps - polling through it for as long as the operator's install takes. - -Queue for a topic that is not yet scoring only if you accept that a `queued` -row is a place in line, not a payment. A ready light is permission to try. +document wins. Do not copy a digest from this page — read +`ctx proof status` and `ctx proof topics`. + +## Status right now: scoring is on + +Live prod (checked 2026-09-10) scores `tbench`. A well-formed submit is +**evaluated**, not parked: + +- `can_score` is **`true`** +- `open_topics` contains `tbench` +- `scorable_topics` contains `tbench` +- `deferred_topics` is **empty** (`[]`) +- `registered_custom` and `custom_ready` both contain `tbench` + +So today a well-formed submit is **201** with a scored row (`awaiting_admin`, +`rejected`, or `champion`) — not **201 `queued`**. The host boots an +experiment VM, ticks the checklist, and (on a green checklist) runs your +artefact against the operator pack. That can take up to +`eval_executor.max_proof_deadline_s` (7200 s today). `ctx proof show --wait` +polls until the row is terminal. + +The operator's sealed bar is currently a **stub** `custom_value` of `0.5` +(kept in the host's `baselines.json` until a measured n=15 is sealed). You +never see the recipe — only the public `script_sha256` / `metrics_commitment`. +Beat whatever the live sealed bar is by `epsilon_rel`; do not assume a +measured n=15, and do not invent a digest or a bar from this page. + +**Historical / may return.** `tbench` previously set +`constraints.params.defer_scoring = "true"`, which put it in `deferred_topics`: +submits answered **201 `queued`** and nothing was evaluated until the operator +lifted the flag. That flag is **off** on the live document today. If the +operator re-publishes with `defer_scoring = "true"`, `tbench` leaves +`scorable_topics`, `can_score` may drop if nothing else is scorable, and +submits go back to **201 `queued`**. Read `ctx proof status` before you spend +— this page is not a substitute for the live fields. + +A ready light is permission to try, not a payment guarantee. Emission still +reaches anyone only through a **sealed** weights bundle (`ctx weights`). ## 0. Install `ctx` @@ -66,13 +81,13 @@ Read these fields for `tbench`: | Field | What you want to see | |-------|----------------------| | `open_topics` | contains `tbench` — otherwise the topic is not published and a submit is **400** `unknown topic` / `topic is not open` | -| `deferred_topics` | contains `tbench` while scoring is deferred: submits answer **201 `queued`** | -| `scorable_topics` | contains `tbench` once the topic is actually being scored. While it is deferred, it is not here | -| `queued_submissions` | how many rows are already waiting across topics — your place in line | -| `registered_custom` / `custom_ready` | must both contain `tbench`. A registered id missing from `custom_ready` means its topic VM is not usable and the topic answers **503** once it stops deferring | +| `scorable_topics` | contains `tbench` while scoring is on (today). A well-formed submit is evaluated | +| `deferred_topics` | **empty** while scoring is on. If `tbench` appears here again, the operator has re-set `defer_scoring` and submits answer **201 `queued`** | +| `queued_submissions` | rows still waiting from a previous deferral (drained oldest first) — `0` while nothing is deferred | +| `registered_custom` / `custom_ready` | must both contain `tbench`. A registered id missing from `custom_ready` means its topic VM is not usable and the topic answers **503** | | `custom_family_wired` | `true`. `tbench` is a `custom`-family topic, so `live_harvest_wired` (the Lium `nll` / `throughput` harvest) says nothing about it | | `baseline_sealed` | `true`. An open topic without both seal hashes is **503** | -| `eval_image_digest` | a `sha256:…` pin. Empty is **503** | +| `eval_image_digest` | a `sha256:…` pin. Empty is **503**. Read it live; do not copy one from a guide | Read the digests, the judge `inference_offer`, and the `eval_executor` from this endpoint rather than from any document. They are re-pinned by the @@ -96,25 +111,34 @@ What the signed document carries today, and what each field means for you: | `metric.epsilon_rel` | `0.05` | A pass must beat the bar (sealed baseline, or the reigning champion) by this **relative** margin | | `constraints.task_slice` | `tb4-first-15` | The scored slice. It is an opaque runner input: the task content is not in this repository and must not be in your code | | `constraints.model_pin` | `moonshotai/kimi-k3` | Every paid model call in your run must name exactly this model | -| `constraints.firecracker_required` | `true` | A run without the host's sister-guest attestation is not evidence: **503**, no row, no host fallback | -| `constraints.params.baseline_runner` | an in-guest runner id | Selects the **experiment VM** path: one dedicated Firecracker VM per paid job, created for the job and destroyed after it | -| `constraints.params.experiment_pack_digest` | a `sha256:` pin | The operator's experiment pack, re-hashed by the host before any jail. Not yours to supply | +| `constraints.firecracker_required` | `true` | A run without the host's Firecracker attestation is not evidence: **503**, no row, no host fallback. On `tbench` that attestation is the **experiment VM** (`network: egress-allowlist`), not a sister guest | +| `constraints.params.baseline_runner` | `rlm_fc_in_guest_harbor` | Selects the **experiment VM** path: one dedicated Firecracker VM per paid job, created for the job and destroyed after it. Read the live value; this id is topic data | +| `constraints.params.experiment_pack_digest` | a `sha256:` pin | The operator's experiment pack, re-hashed by the host before any jail. Not yours to supply or patch. Read it live | | `constraints.params.miner_byok` | `OPENROUTER_API_KEY` | You bring the model key — see § 4 | -| `constraints.params.defer_scoring` | `"true"` | Submits are queued, not scored (§ Status) | +| `constraints.params.defer_scoring` | **absent** (scoring on) | If this returns as `"true"`, submits queue again (§ Status) | | `flops_budget` | `2e18` | Topic document field. **Not** a reject gate on `tbench`: the host ignores `declared_flops` vs measured FLOPs | | `eval_executor.max_proof_deadline_s` | `7200` | Your run is cut at this wall clock; a cut run is **503** with the run's `stdout_tail` | | `payout_mode` | `discovery` | Pass floor plus novelty pool — see § 7 | -| `baseline` | sealed | `script_sha256` + `metrics_commitment`, seed `42`. You never see the recipe, only the commitments | +| `baseline` | sealed | `script_sha256` + `metrics_commitment`, seed `42`. Standing bar is a stub `custom_value` of `0.5` until a measured n=15 is sealed. You never see the recipe | | `status` | `open` | Only `open` accepts submits | `GET /v1/proof/topics` never returns holdout records, and `holdout_commitment` is a commitment, not data. There is nothing to read there. -## 3. Build and serve the artefact +## 3. Miner artefact / attach surface + +`tbench` is a `custom` topic, so **`artifact_uri` is required** — the guest +agent fetches the bytes from your locator inside the experiment VM. A submit +without one is a **400** with no row. -`tbench` is a `custom` topic, so **`artifact_uri` is required** — the runner -fetches the bytes from your locator inside the topic VM. A submit without one -is a **400** with no row. +The attach surface is the guest-runner **contract** in +[`deploy/guest/runners/README.md`](../../deploy/guest/runners/README.md) +plus the versioned adaptor +[`rlm_fc_in_guest_harbor`](../../deploy/guest/runners/rlm_fc_in_guest_harbor/README.md) +(the live `baseline_runner`). That adaptor is what evaluate execs; this page +is what you must ship so it has a Harbor agent to attach. + +### Build the file you will serve Artefact identity is **the served file, verbatim**: @@ -155,24 +179,99 @@ Env the run sees: `PROOF_SEED`, `PROOF_MODEL_PIN`, `PROOF_TASK_SLICE`, - Serve **that exact file** at `artifact_uri` and keep it. Re-running `tar` later produces different bytes (mtimes, member order) and therefore a different digest, and the run is refused rather than run on a substitute. -- The host re-hashes exactly the bytes the runner fetched before it boots your - guest, and refuses gzip, non-tar bytes, an archive with no file content, or - bytes that do not match your `artifact_digest`. +- The host re-hashes exactly the bytes the guest fetched before it continues, + and refuses gzip, non-tar bytes, an archive with no file content, or bytes + that do not match your `artifact_digest`. - A digest of nothing — the sha256 of zero bytes or of an empty tar — is a **400** with no row, whatever case you spell it in. - On the experiment-VM path the guest agent streams your artefact under a hard **64 MiB** cap. Ship a recipe, not a weight dump. -Your code runs **inside a Firecracker guest the host boots**, against the -operator's pinned experiment pack. You cannot produce the sandbox attestation +### What the guest sees after unpack + +The guest agent fetches, verifies, and unpacks your tar into +`PROOF_ARTIFACT_DIR`. `$PROOF_WORK_DIR/artifact.tar` holds the verbatim bytes. +`tar -cf recipe.tar recipe/` unpacks as `PROOF_ARTIFACT_DIR/recipe/…`. + +The operator pack is a **different** tree: `PROOF_PACK_DIR` (and +`PROOF_PACK_DIGEST`) is staged by the host from the topic's +`experiment_pack_digest`. Miners do **not** patch the pack. Tasks, the +evaluator, and the default baseline agent live there. Your submit is the +artefact only. + +Preferred miner layout — put your Harbor agent at one of: + +| After unpack | How you tared it | +|-------------|------------------| +| `$PROOF_ARTIFACT_DIR/agent` | agent at the archive root (`tar -cf recipe.tar agent` or equivalent) | +| `$PROOF_ARTIFACT_DIR/recipe/agent` | the usual `tar -cf recipe.tar recipe/` with `recipe/agent/…` inside | + +**Evaluate must use the miner agent**, not the pack's default. Harbor +`-a` / `--agent` is a built-in name or a Python import path +(`module.path:ClassName`), **not** a filesystem path: the adaptor imports +your class from `$PROOF_ARTIFACT_DIR/agent` or +`$PROOF_ARTIFACT_DIR/recipe/agent` and passes that import path. A +`recipe/run.sh` with no Harbor agent dir is **not** a substitute — evaluate +fails closed rather than falling back to `terminus-2`. Baseline is +operator-owned (pack + the topic's agent param) only when no miner artefact +is staged. Do not copy the pack into your tar. + +### Off-limits (checklist red, no spend) + +These fail **before any paid inference**. One red item is a persisted +`rejected` row: + +| Do not | Why | +|--------|-----| +| Reseed | `same_seed`: every paid call and every scored episode uses the topic baseline seed (`42` today). No per-miner reseeding | +| Hardcode TB4 task ids, answers, fixtures, or success paths | `no_tb4_hardcoding`. Only the signed topic carries `task_slice` | +| Patch the evaluator, the metric path, the pack, or short-circuit scoring | `no_eval_short_circuit`. You do not own `PROOF_PACK_DIR` | +| Embed an OpenRouter key (`sk-or-…`) in the artefact, the claim, or the manifest | The key is BYOK `env` only. A key in the tree is not how the guest authenticates, and it will leak into the artefacts zip | + +Write logs under `$PROOF_WORK_DIR` / `$PROOF_OUTPUT_DIR` while the job +runs. stdout / stderr are only a 64 KiB rolling tail. Extra files in those +directories **die with the VM** — they are not copied into the artefacts +zip. + +### Artefacts zip + +The host collects a zip per scored row at +`{topic_id}/{submission_id}.zip` (checklist rule `artefacts_zip`): + +- your unpacked recipe (`artifact/`) +- `report.json` (the adaptor's measurement; absent on a pre-spend reject) +- `checklist.json` (the rule items with evidence) +- `baseline_ref.json` +- `logs/runner.log` — the redacted stdout/stderr tail the guest returned. + Files you wrote under `PROOF_WORK_DIR` / `PROOF_OUTPUT_DIR` besides + `report.json` / `checklist.json` are **not** in this zip. + +A red-checklist reject ships no `report.json`. You do not upload this zip; +the host builds it. Putting secrets in the recipe still lands them in +`artifact/`. Do not print keys: the tail is redacted, but the recipe is not. + +Your code runs **inside a Firecracker experiment VM the host boots**, against +the operator's pinned pack. You cannot produce the sandbox attestation yourself and you do not need to: the host — not the harness, not the RLM — -stamps it. A guest-measured `flops_used` may appear on the verdict as -telemetry. **`tbench` does not reject or cheat-code on `declared_flops` vs -measured FLOPs** — anti-cheat is the signed topic checklist, sister -attestation, BYOK env, and hotkey signature, not a hardcoded TFLOP budget. +stamps it (`mode: experiment_vm`, `network: egress-allowlist`). A +guest-measured `flops_used` may appear on the verdict as telemetry. +**`tbench` does not reject or cheat-code on `declared_flops` vs measured +FLOPs** — anti-cheat is the signed topic checklist, Firecracker attestation, +BYOK env, and hotkey signature, not a hardcoded TFLOP budget. ## 4. The model key is yours (BYOK) +`tbench` runs on the **experiment-VM** path +(`baseline_runner = rlm_fc_in_guest_harbor`). That VM has **HTTPS egress** on +the operator's allowlist (model provider + whatever the pack pulls). Your +OpenRouter calls leave the guest over that allowlist. + +That is **not** the other custom-family path. Topics that do not select an +in-guest runner still score in a **sister** Firecracker guest with +**network none**: no outbound HTTPS, no BYOK call from inside the sister. +`tbench` is not that path. If you plan as if the guest is offline, your +key never reaches the provider and the run dies on inference. + The topic pins `moonshotai/kimi-k3` and sets `constraints.params.miner_byok = "OPENROUTER_API_KEY"`. Its checklist rule `miner_byok_openrouter` reads: @@ -212,16 +311,17 @@ What the host does with it: `GET /v1/submissions`, or in `/v1/status`, and it is blanked to `[REDACTED]` in any run log or evidence your own run prints it into. - **It is kept in a private file, not a process value**, from the moment it is - accepted until your row is terminal — so a topic that queues your row today - still has your key when the operator drains it. If the host loses it before - the run, your row stays `queued` and the drain answers **503**; it never - scores your work on the operator's account. + accepted until your row is terminal. Scoring-on uses it on this submit. If + `defer_scoring` returns, the same vault still has the key when the operator + drains. If the host loses it before the run, a drain answers **503** with + the row untouched; it never scores your work on the operator's account. One thing to be clear about, because guessing here costs money: **`X-Lium-Api-Key` is not this key, and is not identity.** It is the Lium header from [proof.md](./proof.md), used by the Lium harvest families for **compute**. It never carries a model-provider key and it never authenticates -you. `env` pays for inference; `X-Lium-Api-Key` pays for machines. +you. `env` pays for inference; `X-Lium-Api-Key` pays for machines. `tbench` +does not rent Lium for your run. Never commit an OpenRouter key, and never put one in your claim, your manifest, or a repository you publish as `artifact_uri`. @@ -241,6 +341,16 @@ ctx proof sign \ --json ``` +`--train-dataset` is a **contamination label**, not a training run. An +honest id is enough when you did not train (a harness name such as +`my-harness-v0` is fine). An **empty** manifest — no +`train_dataset_ids` and no `train_content_hashes` — is not a clean +contamination check: on `tbench` it is a persisted **`rejected`** row with +**no rent**. `ctx` refuses to build one client-side and names it +`contamination_evidence_missing`. The stored row's `verdict.failed` is +`{"evidence_missing":{"field":"contamination_evidence"}}` — there is no +stable rejection-code field on the submit response. + That prints `miner_hotkey`, `hotkey_signature`, `submit_nonce`, the exact `manifest`, and `domain` (`base-proof-submit-v1`) without posting anything. Then submit: @@ -278,7 +388,7 @@ Fields the host reads on `POST /challenge/proof/v1/submissions`: | `artifact_uri` | **yes** | Required because `tbench` is a `custom` topic | | `claim` | yes | One English sentence of what improved. Signed | | `declared_flops` | no | Optional, default `0`. Still bound into the signature if you send it. **Ignored as a scoring gate** on `tbench` | -| `manifest.train_content_hashes` / `manifest.train_dataset_ids` | yes | Declare at least one. Signed | +| `manifest.train_content_hashes` / `manifest.train_dataset_ids` | yes | Declare at least one. An honest harness id is enough when you did not train. Signed. Empty → `rejected` (`verdict.failed.evidence_missing.field` = `contamination_evidence`; `ctx` names it `contamination_evidence_missing`) | | `env` | **yes** | `{"OPENROUTER_API_KEY": "sk-or-…"}` — the topic's `miner_byok` variable (§ 4). **Not** signed, never echoed back. Missing → **400** without spending your nonce | The signature covers the hotkey, `topic_id`, `artifact_digest`, @@ -291,13 +401,10 @@ it from this page. The `(miner_hotkey, submit_nonce)` pair is accepted **once**, reserved before any row exists. Re-posting the identical body is **401 `submit_nonce reused`**. -To re-send the same artefact, sign again with a fresh nonce — on a deferring -topic that returns the **existing** row (**200**), not a second one: one run per -artefact per topic. - -An empty manifest is not a clean contamination check. On `tbench` it is queued -now and becomes a persisted **`rejected`** row with `contamination_evidence_missing` -and no rent when the queue drains. `ctx` refuses to build one client-side. +To re-send the same artefact, sign again with a fresh nonce. While scoring is +on, that is a **new** run (a new row). The **200** existing-row answer is the +defer path (`deferred_topics`): one queued-or-scored row per artefact per +topic, not a second paid run. ## 6. Watch the row @@ -310,7 +417,7 @@ curl -sS https://gateway.cortex.foundation/challenge/proof/v1/submissions/ | `state` | Meaning on `tbench` | |---------|---------------------| -| `queued` | Accepted and stored, **not evaluated**. No rent, no VM, no judge call, no mass. The only non-terminal state; the operator drains the queue later and the row becomes one of the three below | +| `queued` | **Not** the live answer while scoring is on. Only if `tbench` is back in `deferred_topics`: accepted and stored, **not evaluated**. No VM, no judge call, no mass. The only non-terminal state; the operator drains later | | `awaiting_admin` | Clean pass, mass recorded. The operator audit is informational | | `rejected` | A gate failed: contamination, unreproduced claim, a red checklist item. Pre-eval rejects spend nothing. FLOP accounting is **not** a reject gate on `tbench` | | `champion` | Promoted. On this topic promotion is automatic: a passing run with a green checklist that beats the sealed bar or the reigning best by `epsilon_rel`. Proof pays on pass, not on the crown | @@ -349,10 +456,10 @@ texts in the live topic are the contract; the ids published today are: |---------|----------------------| | `same_seed` | Every paid call and every scored episode uses the topic baseline seed. No per-miner reseeding | | `no_tb4_hardcoding` | Your harness must not hardcode task ids, answers, fixtures, or success paths. Only the signed topic carries `task_slice` | -| `no_eval_short_circuit` | The evaluator, the metric path, and the sandbox are untouched — no short-circuiting the checklist or the scoring | +| `no_eval_short_circuit` | The evaluator, the metric path, and the sandbox are untouched — no short-circuiting the checklist or the scoring; do not patch `PROOF_PACK_DIR` | | `miner_byok_openrouter` | You supply the OpenRouter key for the pinned model; operator keys are never injected into your guest | -| `firecracker_sister` | Your code runs only in the Firecracker guest the host booted, and the report must carry that attestation | -| `artefacts_zip` | Scored artefacts persist as a zip under the topic's artefact root, per submission | +| `firecracker_sister` | Your code runs only in the Firecracker guest the host booted, and the report must carry that attestation. On `tbench` that guest is the **experiment VM** (egress allowlist), not a network-none sister | +| `artefacts_zip` | Scored artefacts persist as `{topic_id}/{submission_id}.zip`: recipe, `report.json`, `checklist.json`, runner logs (§ 3) | | `auto_promote_best` | Only a passing, green-checklist run that beats the bar by `epsilon_rel` is promoted | | `rlm_topic_setup_autonomous` | The topic's RLM drives setup, baseline, evaluation, and promotion from the signed document alone | @@ -370,18 +477,19 @@ you will actually meet on `tbench`: | Answer | When | Row? | |--------|------|------| -| **201** `queued` | The normal answer today: `tbench` is in `deferred_topics` | yes, queued | -| **200** existing row | Same artefact + hotkey re-sent with a fresh nonce, before or after the queue drained | existing row | +| **201** scored (`awaiting_admin` / `rejected` / `champion`) | The normal answer today: `tbench` is in `scorable_topics` | yes | +| **201** `queued` | Only if `tbench` is in `deferred_topics` again (`defer_scoring` returned) | yes, queued | +| **200** existing row | Same artefact + hotkey re-sent with a fresh nonce **on the defer path**, before or after the queue drained | existing row | | **400** `unknown topic` / `topic is not open` | `tbench` is not published, or is outside its epoch window | no | | **400** `artifact_uri is required for custom topics` | You left the locator out. `tbench` is `custom` | no | | **400** `artifact_digest is the sha256 of empty input …` | You hashed nothing, or an empty tar | no | | **400** invalid `miner_hotkey` / `artifact_digest` | Not exactly 64 lowercase hex. The host never normalises a hex field | no | | **401** `hotkey_signature required` / `invalid` | Missing signature, or a `claim`, `declared_flops`, `manifest`, or nonce that differs from what you signed | no | | **401** `submit_nonce required` / `invalid` / `reused` | Missing, not 64 lowercase hex, or a replay. Sign again with a fresh nonce | no | -| **503** `custom metric … has no registered runner` / `not wired` | `tbench` is not in `registered_custom` / `custom_ready`. Only reachable once the topic stops deferring | no | +| **503** `custom metric … has no registered runner` / `not wired` | `tbench` is not in `registered_custom` / `custom_ready` | no | | **503** empty `eval_image_digest` / unsealed baseline / missing judge offer | The host cannot score. Fail-closed, never a sim fallback | no | | **503** `proof deadline … exceeded` | Your run did not finish inside `max_proof_deadline_s`; the body carries `stdout_tail` | no | -| **201** `rejected` + `contamination_evidence_missing` | Empty manifest, at drain time | yes, rejected | +| **201** `rejected` + empty manifest | Empty `train_dataset_ids` and `train_content_hashes`. `ctx` names it `contamination_evidence_missing`; the row's `verdict.failed` is `evidence_missing` / `contamination_evidence` | yes, rejected | | **201** `rejected` + red checklist | A topic rule failed on your artefact — before any paid inference | yes, rejected | Never commit your OpenRouter key or `LIUM_API_KEY`, and never hand anyone a diff --git a/docs/external-miner/proof.md b/docs/external-miner/proof.md index 157065ae3..02fa6301c 100644 --- a/docs/external-miner/proof.md +++ b/docs/external-miner/proof.md @@ -126,7 +126,7 @@ Topics that have a written miner guide in this repo: | Topic | Guide | Family / payout | |-------|-------|-----------------| -| `tbench` | [proof-tbench.md](./proof-tbench.md) | `custom` (`success_rate`) / `discovery` | +| `tbench` | [proof-tbench.md](./proof-tbench.md) | `custom` (`success_rate`) / `discovery`. **Scoring is on** (experiment VM). Do not assume it is deferred — read `ctx proof status` | A guide is a convenience, not the contract: the signed document returned by `ctx proof topics` wins wherever the two disagree, and a topic without a guide diff --git a/docs/external-miner/troubleshoot.md b/docs/external-miner/troubleshoot.md index 5f8903a43..5decc3b52 100644 --- a/docs/external-miner/troubleshoot.md +++ b/docs/external-miner/troubleshoot.md @@ -36,9 +36,15 @@ Install `ctx` from [README](./README.md). Proof miners pay Lium | `400` `declared_flops exceeds the topic budget` | Harvest `nll` / `throughput` only: `declared_flops > topic.flops_budget`. Custom / agent (`tbench`) ignore this gate | Cap declared FLOPs on harvest topics; omit or send `0` on `tbench` | | `400` invalid hotkey / `artifact_digest` | Not exactly 64 lowercase hex (`0x`, uppercase, whitespace) | Post the same lowercase bytes you signed; the host never normalises a hex field | | `401` `hotkey_signature` / `submit_nonce` required, invalid, or reused | Unsigned, wrong key, a `claim` / `declared_flops` / `manifest` that differs from what was signed, or a replayed `(hotkey, submit_nonce)` | `ctx proof sign` over `base-proof-submit-v1` with a fresh nonce; post the manifest you signed. `X-Lium-Api-Key` is not identity | -| `rejected` with `contamination_evidence_missing` | Empty `manifest` | Declare `train_content_hashes` or `train_dataset_ids` | +| `rejected` with `contamination_evidence_missing` | Empty `manifest` | Declare `train_content_hashes` or `train_dataset_ids`. On `tbench` an honest harness id is enough when you did not train; empty still rejects with no rent | | `rejected` with contamination / cheat code | Holdout overlap, unreproduced claim, strawman AdamW, … | Read the verdict `cheat_codes`. Contamination rejects without rent | | HTTP 503 on submit | Empty `eval_image_digest`, zero open topics, unsealed baseline, harvest down, or missing/closed RLM judge | `ctx proof status` → `can_score`. Live digest is `sha256:78b614a1…`. Empty digest still 503. Do not invent a digest. Nothing was rented | +| `201` `queued` on `tbench` | `tbench` is in `deferred_topics` (`defer_scoring=true`) | Read `ctx proof status`. **Live scoring is on**: `can_score=true`, `scorable_topics` contains `tbench`, `deferred_topics=[]`. Queued is not the current path; see [proof-tbench.md](./proof-tbench.md) | +| `tbench` submit but no OpenRouter traffic | Planned as a network-none sister guest | `tbench` is the **experiment VM** path (`baseline_runner=rlm_fc_in_guest_harbor`) with HTTPS egress. Sister-with-network-none is the other custom path. BYOK `env.OPENROUTER_API_KEY` is required | + +`tbench` is **not** deferred-only. Scoring status is live data: `open_topics`, +`scorable_topics`, `deferred_topics`, `custom_ready`. The topic page is +[proof-tbench.md](./proof-tbench.md). ## Off challenges diff --git a/xtask/src/external_docs_check.rs b/xtask/src/external_docs_check.rs index e0322ecb0..5d5a64892 100644 --- a/xtask/src/external_docs_check.rs +++ b/xtask/src/external_docs_check.rs @@ -126,6 +126,8 @@ const PAGE_PINS: &[(&str, &[&str])] = &[ "tbench", "deferred_topics", "queued", + "scorable_topics", + "scorable_topics` contains `tbench", "artifact_uri", "submit_nonce", "base-proof-submit-v1", @@ -136,6 +138,7 @@ const PAGE_PINS: &[(&str, &[&str])] = &[ "X-Lium-Api-Key", "PROOF_ARTIFACT_DIR", "recipe/agent", + "rlm_fc_in_guest_harbor", ], ), ]; @@ -603,11 +606,16 @@ mod tests { "tbench", "deferred_topics", "queued", + "scorable_topics", + "scorable_topics` contains `tbench", "artifact_uri", "submit_nonce", "success_rate", "discovery", "OPENROUTER_API_KEY", + "PROOF_ARTIFACT_DIR", + "recipe/agent", + "rlm_fc_in_guest_harbor", ] { assert!(pins.contains(&needle), "topic page must pin {needle:?}"); }