An evaluation harness for LLM resolvers on settled Polymarket / UMA markets. Given a market question, its formal resolution criteria, and a frozen bundle of dated documents, a resolver must return a structured verdict: outcome, confidence, reasoning, cited sources, detected conflicts. Ground truth is how the market actually settled — so the benchmark scores the resolver against money, not against a model's judgment.
Status: the harness is complete, one model tier has been run, and the contamination control has landed. Two things are not done and are marked where they belong rather than left for a reader to find: the expensive-model arm (its free-tier quota served 20 of 300 calls, so the cost/quality axis is missing) and the manual review of the 50 questions (so conflict detection is unscored and the bundles are of unverified quality — which, given the result below, is now the most load-bearing gap in the project). See PLAN.md for the build plan.
gemini-3.5-flash-lite, 50 questions × 3 arms × 3 repeats, plus both closed-book controls.
Baselines are in the same table because an accuracy number cannot be read without them.
config n q cov acc brier conf-wrong ece sch-ok abst $/res p50 p95
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
always_no 850 0 1.000 0.500 0.500 — — — — — — —
always_yes 850 0 1.000 0.500 0.500 — — — — — — —
random_p50 850 0 1.000 0.500 0.250 — — — — — — —
market_price_at_T-24h 50 0 1.000 0.860 0.095 — — — — — — —
a0 flash 12 5 0.080 — — — — — — — — —
a0 flash-lite 150 50 1.000 0.533 0.253 0.361 0.192 0.840 0.247 0.002 1499.000 2555.000
a1 flash-lite 150 50 1.000 0.527 0.247 0.387 0.248 0.873 0.247 0.002 1694.000 2817.000
a2 flash 8 4 0.053 — — — — — — — — —
a2 flash-lite 150 50 1.000 0.487 0.219 0.434 0.114 0.920 0.367 0.002 1497.000 2611.000
closed_book flash-lite 50 50 1.000 0.020 0.260 0.980 0.100 1.000 0.960 0.000 990.000 1094.000
closed_book_forced flash-lite 50 50 1.000 0.520 0.335 0.500 0.278 1.000 0.000 0.000 1028.000 1223.000
cov is coverage — the share of attempted resolutions the provider actually served — and q is the
number of distinct questions behind the row. q, not n, is the sample size: the bootstrap
resamples questions, so three repeats of five questions is a sample of five.
The two flash rows print dashes because they are not measurements. That tier's free-tier daily
quota served 20 of 300 calls, and what survived was five questions for A0 and four for A2 — not a
random five, but the first ones the run happened to reach. Subsampling the full A0 run shows the
mean width of the 95% accuracy interval at 0.78 for four questions, 0.67 for eight, 0.50 for
fifteen and 0.41 for twenty; below fifteen the interval covers more than half the scale. The floor
is set at twenty questions, and below it the quality columns are withheld everywhere — table,
evidence_lift and the McNemar comparisons alike, so the number is not merely relocated. The rows
stay visible because an arm that was attempted and starved is a fact about the run.
The result is negative, and it is published as it stands. No arm is distinguishable from a coin
flip: A0's 95% interval on accuracy is [0.393, 0.667], which contains 0.500, and the naive
always_no and always_yes rows both score exactly 0.500 because the drawn set is balanced by
construction. Neither prompt change helped — McNemar against A0 gives p=1.0 for A1 and p=0.63 for
A2, on 3–4 discordant questions out of 50. The differences in the table are 1–5 points and the
smallest difference this dataset can detect is 10–11 points, so they are not results.
The headline metric is confident_wrong_rate — P(wrong | confidence ≥ 0.9) — and it is the
number that makes the negative result concrete rather than merely disappointing: when this model
says it is at least 90% sure, it is wrong about 36% of the time. That is a calibration failure,
not a knowledge failure, and it is the failure mode the project set out to measure.
Two things underneath the flat accuracy column are worth more than the column:
- Abstention is doing real work. A2 — the arm that explicitly offers
UNRESOLVABLE— abstains on 36.7% of questions against A0's 24.7%, and when it does commit it is right 77.7% of the time against A0's 71.4%. Its Brier falls to 0.219 and its ECE from 0.192 to 0.114. Overall accuracy still drops, because an abstention is scored as wrong; both numbers are true and the table shows the one the plan specified. - The model abstains in the wrong places. On the contested set — markets where UMA token holders actually disputed the outcome — A0 abstains on 10.0% of questions against 28.3% on the main set. A resolver that understood its own uncertainty would do the opposite.
Also measured: self-consistency across the three repeats is 0.96 / 0.90 / 0.88 for A0 / A1 / A2, so A2 buys its calibration with some instability. Citation grounding is 1.000 — across 550 scored resolutions the model never once cited a source id that was not in its bundle. Cache hit rate is 0.000, confirming that this model does not use the implicit cache at any prompt length we send.
The full numbers, intervals and per-arm breakdowns are in
results/results.json.
Predicted probability against observed frequency, five equal-count bins, dot area proportional to bin weight. The diagonal is perfect calibration; everything below it is overconfidence.
evidence_lift = accuracy(given-evidence) − accuracy(closed-book) is the number that says whether
this benchmark measures reading or memory. It is now measured, and it is close to zero:
| configuration | accuracy | lift over the control |
|---|---|---|
a0 — question, criteria, documents |
0.533 | +0.013 |
a1 — forced enumeration |
0.527 | +0.007 |
a2 — abstention offered |
0.487 | −0.033 |
closed_book_forced — no documents, must answer |
0.520 | — |
always_no — a constant |
0.500 | — |
Four thousand eight hundred tokens of dated, on-topic documents move accuracy from 0.520 to 0.533. The control's own interval is [0.38, 0.66] and A0's is [0.393, 0.667]; they overlap almost exactly. A2 scores below the control. Whatever the given-evidence arms are doing, it is not visible in accuracy.
Two explanations survive this, and the benchmark cannot yet tell them apart. Either the model is not reading the documents, or the documents do not contain the answer — because the article for each question was chosen by an automatic ranker and never confirmed by a person. At least one is known to be wrong: market 1693734, "Will Ukraine re-enter Rodynske by April 30?", was given List of national days of mourning (2000–2019). That is the manual review of §3.2 step 5, and this result promotes it from a nice-to-have to the thing that decides what the whole table means.
The plan's closed-book arm tells the model: "if you do not know, answer UNRESOLVABLE rather than guessing." It obeyed, on 48 of 50 questions. Its accuracy of 0.020 is therefore an obedience rate, and subtracting it produced an evidence lift of 0.513 — the largest and most flattering number in the report, measuring a difference in abstention policy rather than contamination.
closed_book_forced withdraws the option: same questions, no documents, must answer YES or NO with
a confidence. That arm answers 50 of 50, abstains never, and scores 0.520 — the base rate, reached
by answering NO almost every time. The lift is computed against that arm and no other, and
results.json carries evidence_lift_control naming which arm it was taken against, because a lift
whose control is unstated cannot be checked. Both arms stay in the table: the gap between 0.020 and
0.520 is the finding about what an instruction to abstain does to a control.
Conflict detection is unscored for the same kind of reason. conflict_recall needs the 15
hand-marked bundles of §4.2, and that marking is part of the manual review that has not
happened yet, so the metric reads None rather than zero. What the run does show is that the model
emits conflict notes on bundles with no planted conflict — measured precision against the labelled
set is 0.0 — but with nothing labelled that number says more about the missing labels than about the
model.
50 questions: a 40-question main set of markets that settled without dispute, and a separate
10-question contested set. Every rejected candidate is recorded with its reason in
data/rejected.jsonl — the rejection list is the evidence that selection
happened rather than "took the first 50". The draw is seeded and its constraints are recorded in
data/selection-manifest.json.
The funnel, from closed Polymarket markets in a 12-month window above $200k volume:
| Stage | Remaining |
|---|---|
| Harvested candidates | 4,947 |
− multi-outcome markets presented as binary (negRisk) |
1,719 |
| − not strictly Yes/No, no clean 0/1 settlement, empty criteria | 1,707 |
| − excluded domains, mention markets | 1,639 |
| − too little price history to label difficulty | 1,528 |
| Collapsed to one question per event, then per question | 710 |
| Stratified draw | 50 |
The base rate is measured, not assumed. The unscreened pool settles 81% NO; the screened undisputed pool, 72% NO (1,052 of 1,460). A resolver that always answers NO therefore scores 0.72 on a natural sample of this domain. Forcing 20 YES / 20 NO puts that naive baseline at 0.50, and it is published in the same table as every model result.
Sports and asset-price markets are excluded rather than capped. UMA's own production proposer reports 99.3% accuracy on exactly those two categories: they resolve by looking up a number, which is a different skill from reading contradictory prose, and including them would let a resolver buy accuracy without doing the task this benchmark is about.
One question per event, and per question. The screened markets belong to only ~900 Gamma events, and the largest event alone holds 61 of them — "Will Israel strike Iran by March 31 / by April 30 / by May 31…". Those are one question with the date moved, so drawing several would inflate the apparent sample size while adding no independent evidence.
Collapsing by event is not enough. Gamma files "Will Trump talk to Zelenskyy in November?" and "… in December?" as separate events, and the first draw duly picked up both, along with two versions of "Japan prime minister Ishiba out". So questions are also compared with their deadlines stripped, and near-identical ones collapse: that removed a further 174 restatements from the main pool. Collapsing twice is what makes n=40 mean forty things.
Difficulty comes from the price series, but not by the rule originally planned. The plan called
for max|price − 0.5| over the final 48 hours, with < 0.15 meaning hard. Measured on 120 random
markets, that statistic is ~0.4995 for every market and labels 100% of them easy — a market
closes precisely when its price has converged to 0 or 1, so any window ending at settlement
contains the converged point and max picks it. Two changes fix it: the window ends 48 hours
before settlement, and the statistic is the median over that window. So hard now means
something defensible — for a whole week, traders with money at stake put the question within 0.15
of even. Thresholds are unchanged.
Domains are regional. The plan's five domains do not survive contact with the pool: only three are populated, and "geopolitics" alone covers 71% of it, so a 12-per-domain cap over three domains admits at most 36 questions and cannot reach 40. Splitting geopolitics by region restores feasibility, but the real gain is that the cap now bites where the concentration is — otherwise twelve "geopolitics" questions would be twelve Middle East questions from twelve different events.
The draw that comes out, before manual review:
| main set | contested set | |
|---|---|---|
| questions | 40 | 10 |
| YES / NO | 20 / 20 | 5 / 5 |
| hard / medium / easy | 12 / 12 / 16 | 0 / 4 / 6 |
| largest domain (cap 12 / 4) | 6 | 2 |
| distinct events | 40 | 10 |
The domain cap never binds — the draw spreads across all seven domains rather than filling them in order, so the largest is 6 of 40 where 12 was allowed.
Each question ships with 2–5 dated documents — 47 of 50 have five — in
bundles.jsonl, built once and committed. Wikipedia is
the source because a revision id pins the text exactly: the same revid returns
the same bytes in a year, when the article has been rewritten. Every source
carries revid, as_of and a permalink; attribution and the share-alike terms
that come with it are in LICENSE-DATA.md.
Revisions are taken at settlement, not at the question's deadline. Settlement is the moment the outcome entered the public record, so a revision at that instant can state it; one at the nominal deadline usually cannot. This is a resolution benchmark, not a forecasting one — the evidence is meant to contain the answer, and the job is to read it correctly.
Tying a question to an article is the part that does not automate. Plain search
gets it wrong most of the time, and not narrowly: TikTok banned in 2025?
returns "Censorship of TikTok", a worldwide survey, while the article that
resolves it ranks third; Will House and Senate pass funding bill by November 30? returns a different bill from a different month. Ranking candidates by how
hard each article was being edited around the settlement date — the signal
search does not have — puts the right one first or second in most cases, and in
the candidate list in all of them. A person picks from that list; the ones not
picked become the bundle's distractors, which is what §4.1 asks a distractor to
be.
Two departures from the plan, both recorded rather than papered over:
- No oracle source. The plan takes it from the market's resolution-source
URL; Gamma returns
resolutionSourceempty for all 50. Reading UMA's on-chain ancillary data would recover it and is a different integration. So bundles run 2–5 documents rather than the planned 4–8, and the "what the oracle actually looked at" arm is missing. - Conflicts are marked by hand, and are not marked yet. 15 of 50 bundles are
meant to carry the same article as it stood 30 days earlier. Whether that older
revision disagrees rather than merely says less is a judgement, and
conflict_recallmeasures nothing if it is guessed — so it currently reports nothing at all.
The article for each bundle is currently the one the ranking chose, not one a person confirmed. At least one is wrong: "Will Ukraine re-enter Rodynske by April 30?" was given "List of national days of mourning (2000–2019)", a table with no prose. Until the review below happens, the results table rests on bundles of unverified quality, and some part of the flat accuracy column may belong to the bundles rather than to the model.
Step 5 pending: manual review of all 50 against
data/review-sheet.md. Machine screening cannot tell whether a question
is answerable from documents at all, and the draw still contains candidates that look like it is
not — "Nothing Ever Happens: US Strike Edition" needs context the bundle will not carry. Rejections
there are appended to the ledger with a reason and the sets are redrawn; because the draw is
seeded, surviving questions keep their places.
A resolver returns one JSON object: outcome, confidence, reasoning, the sources it read
and the conflicts_detected between them (schema.py). Whether a cited
source_id actually exists in the bundle is checked outside the schema
(grounding.py) — a fabricated id is well-formed, so pydantic cannot
see it, and hallucinated citations get their own metric rather than being counted as valid.
A broken answer is a measured refusal, not a crashed run.
| Event | Action | Recorded as |
|---|---|---|
| Invalid JSON or schema | one repair retry carrying the validation error | repair_rate |
| Still invalid | hard_fail, run continues, scored as wrong |
hard_fail_rate |
| Timeout / 429 / 5xx | exponential backoff, 3 attempts, then hard_fail |
not a quality signal |
| 400 / 401 / 404 | immediate hard_fail — retrying cannot help |
harness fault, reported apart |
| Declined by a safety classifier | hard_fail, never repaired |
refusal, reported apart |
The last two rows are not in the plan and were added because collapsing them into the others would misattribute blame. A refusal arrives as HTTP 200 with empty content, so a parser sees "no JSON" and files it as a schema failure — the model gets charged with a structure problem it never had. Fifty bad-key errors likewise mean the harness is broken, not that a model failed fifty times.
Structured outputs are deliberately not used. The API can constrain the response to the schema,
which would make schema_valid_rate, repair_rate and hard_fail_rate read 1.0 / 0 / 0 by
construction — the three reliability numbers deleted by the feature that appears to guarantee them.
The contract is stated in the prompt and the model has to meet it. Code fences and surrounding
prose are tolerated (it is the same object, and rejecting them would measure instruction-following
instead); a missing field, a wrong type or an extra key is not.
Prompt caching is not an optimisation here but a structural constraint. One bundle is reused across
three repeats and three arms, and caching is a prefix match — so the arm's wording sits after the
evidence rather than in the system prompt. Put it in the system prompt and every arm opens its own
cache entry: the run still works, the numbers are identical, and the only symptom is the bill.
Cache reads are priced at 0.1× and writes at 1.25×, so cost_usd_per_resolution is computed from
the reported usage per call rather than from a token estimate.
temperature is not sent anywhere. Sonnet 5 rejects a non-default value outright, so determinism
was never on offer to begin with — which is what the three repeats and the published
self_consistency_rate are for.
| Tier | Default (Google, free) | Alternative (Anthropic, paid) |
|---|---|---|
| cheap | gemini-3.5-flash-lite — $0.30 / $2.50 per 1M |
claude-haiku-4-5 — $1.00 / $5.00 |
| expensive | gemini-3.5-flash — $1.50 / $9.00 per 1M |
claude-sonnet-5 — $2.00 / $10.00 |
Two providers, which is the ceiling PLAN §15 sets, and a second implementation is what
keeps the one-file Resolver claim honest rather than asserted.
The default pair runs on Google's free tier, so the §12 budget stops binding. Cost is still
reported, at list price. A benchmark that prints $0.000 per resolution has not demonstrated cost
awareness; it has deleted the metric. What this costs to run at scale is the answerable question, so
every call is priced from its reported usage at published rates and the run states that it was
executed free.
Both defaults are from the same family so that A3 varies the model tier and nothing else — in
particular they take the same reasoning control. That control is thinking_level="minimal", not
thinking_budget=0: the latter is a 400 on gemini-3.5-flash-lite, so a pair built on it would not
be a pair. minimal is nominally a floor rather than an off switch, and it measures
thoughts_token_count == 0 on every callable 3.x model.
Four things here were established by calling the API, and three of them contradict the docs.
- The 2.5 family was the first choice and is unavailable.
models.list()still returnsgemini-2.5-flashand-flash-lite, andgenerateContentanswers 404 — no longer available to new users;gemini-2.5-proanswers 429 with a zero free-tier quota. Listing is not availability. - The cached prefix is 2,303–5,240 tokens, median 4,332 (counted, not estimated at 1.33 tokens per word — that estimate was 8% low). 15 of 50 sit below the 4,096 floor of the 3.5 family.
- The cheap model appears not to cache at all. Four identical calls at 4,822 tokens — well clear
of the floor — returned zero cached tokens every time, while
gemini-3.5-flashhit on its second identical call. Its published cost therefore carries no cache discount. - A cache hit is partial, not all-or-nothing: 2,032 of 4,822 tokens. Which is why cost is read from
the reported
usageper call rather than derived from a floor and an assumption.
The two tiers also differ by an order of magnitude in latency — ~1.6 s for flash-lite against 15–21 s for flash — and flash returned a 503 high demand mid-probe, which the retry policy above handles as transient. Both are properties of the run, so both are reported rather than smoothed away.
Free-tier caveats, stated because they are conditions on the result and not footnotes: per-day and per-minute request quotas apply, and 850 calls will be paced by them rather than by the harness; Google may use free-tier inputs to improve its products, including human review, which is acceptable here only because every input is already public (Wikipedia revisions and published market questions) — and which is why no key, no credential and no private text ever enters a prompt.
CI replays recorded responses instead of calling the API, so the gate runs on every pull request with no key and no network. Failures are recorded too — a captured 429 is re-raised on replay, which is how the backoff path above is exercised in CI for free.
UMA's Optimistic Oracle settles a market by proposal-and-bond, with a dispute window; if a dispute is opened, the outcome is decided by a vote of UMA token holders. That is human judgment — deferred and weighted by money, but human. So "externally settled" is accurate and "objective" is not.
This is handled by splitting rather than by hedging: the main set uses only undisputed markets, where the ground truth really is external. The contested set is reported separately, and the interesting number there is not accuracy but the abstention rate — a good resolver should decline where people with money on the line disagreed.
Disputes are rare — Polymarket documents roughly 1.5% of proposals as disputed — so the contested set is small by nature, and OOBench does not try to be the authority on it. Prior work has already measured LLM agreement with post-dispute UMA votes at n=259; this set of 10 exists to ask the different question above.
Small differences. Fifty questions is a small sample, and the paired test that gets the most out
of it still only sees the questions where two arms disagree. At n=50 the smallest detectable
difference in accuracy runs from 12.5 points (when the arms disagree on one question in ten) to
28 points (when they disagree on half), at 80% power. The plan this repository follows guessed
~10 points; that figure is optimistic under every discordance rate, and the real one is computed per
comparison and published in results.json rather than assumed. A difference below it is not a
result, however pleasing the direction.
On this run the arms turned out to disagree even less often than one question in ten — A1 differs
from A0 on 3 of 50 and A2 on 4 — so the published figures are 9.7 and 11.2 points, below the
12.5 above rather than in contradiction with it. Low discordance is not good news: two arms that
almost never differ are two arms with almost nothing to compare. Where discordance is zero the MDE
is reported as nothing at all, because the formula scales with its square root and a floor of 1e-6
returns 0.0014 — a claim to detect a tenth of a point, from two arms that never once disagreed.
Search. The evidence bundle is fixed. A resolver that would have found the decisive document
scores the same as one that would not, and nothing here says whether a model can locate evidence —
only what it does with evidence it is handed. That is the trade the design makes deliberately:
fixed bundles are what make runs reproducible, contamination measurable, and conflicts_detected
checkable, because the conflicts are planted and their locations known.
Model drift. The CI gate replays recorded responses, so it is deterministic by construction. It catches regressions in the harness — extraction, validation, grounding, the metric formulas — and it catches an edited prompt as a cassette miss. It cannot catch the model answering differently next month. That is the nightly live run's job, and when the two diverge it opens an issue rather than failing a build, because provider drift is a finding and not a broken commit.
Anything outside English Wikipedia. Every document is an English Wikipedia revision. The resolver is never tested on a press release, a court filing, a chart, or a source in another language, and the questions are the ones a Wikipedia article happens to cover well.
Calibration under instruction. Confidence is self-reported and never elicited twice, so
confident_wrong_rate measures what the model volunteers, not what it would say if pressed.
uv sync
export GEMINI_API_KEY=... # free tier; https://aistudio.google.com/apikey
make dataset && make eval && make reportmake dataset stops in the middle by design: step 5 of the selection procedure is a person reading
data/review-sheet.md. Everything already harvested is committed, so a
clone can skip straight to make eval.
make eval runs the three given-evidence arms three times each and names the run for the moment it
started; make report renders the most recent one. The grid is ragged on purpose, so the other arms
are separate invocations into the same run:
make eval ARMS=closed_book REPEATS=1 RUN=main # the §3.3 control as the plan spells it
make eval ARMS=closed_book_forced REPEATS=1 RUN=main # the same, abstention withdrawn
make eval MODEL=flash ARMS=a0,a2 RUN=main # the expensive tier (§8, A3)A run is resumable and a re-run is free: records are keyed by (model, arm, repeat, market), and a resolution the provider never served — a 429, a timeout — is retried rather than treated as done.
That resumability is why records.jsonl holds more lines than the report scores: a retried key is
appended again and the later row wins. On the published run the log has 951 lines and 850 scored
resolutions. results.json states all three — n_rows_written, n_rows_superseded, n_records —
rather than leaving a reader to count lines, get a different number, and wonder which one was
massaged.
make report renders the most recent run under results/runs/, which is whichever one was written
last. If you record cassettes after an eval, pass --run explicitly rather than letting the gate
slice become the published table.
make eval reads GEMINI_API_KEY, or ANTHROPIC_API_KEY with PROVIDER=anthropic. Neither is
needed for the gate, which replays cassettes:
make gate # ruff + mypy --strict + pytest, no network and no keysWithout make (Windows), every target is a one-line wrapper — see the Makefile.
Most prediction-market benchmarks score forecasting — predicting an event whose outcome is not yet known. OOBench scores resolution: the outcome is already known and settled, and the question is whether a model can read a fixed evidence bundle and say what happened, with defensible confidence.
| Work | Task | Relation to OOBench |
|---|---|---|
| ForecastBench (Karger et al.) | Forecasting. 1,000 rolling questions with no known answer at submission time; contamination-free by construction | Different task. Their design solves contamination by never asking about the past; OOBench asks about the past and therefore has to measure contamination instead (see Evidence lift) |
| Halawi et al., 2024 | Forecasting with a retrieval-augmented system, evaluated against crowd aggregates | Different task; the retrieve-then-reason structure is the ancestor of the given-evidence setup here |
| KalshiBench | Closed-book forecasting, scored on calibration (ECE, Brier) | Closest on metrics, different on task. It asks whether a model knows what it doesn't know about the future; OOBench asks the same about the past, with evidence in front of it |
| Multi-agent AI oracle systems | Same task. 1,189 settled Kalshi markets, shared date-filtered evidence layer, single-LLM vs. ensemble | The nearest neighbour, at 24× the scale. See below |
| LLMs for UMA dispute arbitration | Reproducing UMA's post-dispute token-holder vote on 259 disputed Polymarket markets | Overlaps the contested set, and is much larger. OOBench does not try to beat it on agreement; the contested set here asks a different question — whether a resolver abstains where humans with money disputed |
| PolyBench | Live forecasting and simulated trading on Polymarket order books | Different task and different objective (returns, not truth) |
| UMA OOTruthBot | Production AI proposer for the Optimistic Oracle | Reports ~95% on Yes/No markets and 99.3% on sports and asset pricing — the reason OOBench excludes price markets and caps sports (see Dataset) |
How OOBench differs from its nearest neighbour. The multi-agent oracle study answers a
question OOBench does not ask (does an ensemble beat a single model?) and its headline explanation
is persuasive error propagation — "confidently wrong models flip correct ones during debate" —
yet it reports no calibration metric of any kind. Confident wrongness is named as the mechanism
and never measured. That gap is OOBench's entire premise: confident_wrong_rate is the headline
number here, not accuracy. Four further differences follow from treating this as an engineering
artifact rather than a model comparison:
- Frozen evidence. Their bundles come from a live retrieval API at evaluation time, so the
benchmark is not reproducible once the index moves. OOBench pins each document to a Wikipedia
revidand commits it, so a run in 2027 sees the 2026 bundle. - A published base rate. They report 83.43% accuracy on a set that is 59% NO, with no naive
baseline in the table. OOBench forces 20 YES / 20 NO and prints
always_noon the same line as every model. - Engineering metrics. Cost per resolution, latency, schema validity, repair rate and citation grounding are absent from all of the work above. They are the point here.
- Planted conflicts. 15 of 50 bundles contain a deliberately introduced contradiction with a
hidden label, which is what makes
conflicts_detecteda scoreable field rather than free text.
Code: MIT (see LICENSE). Evidence text is drawn from Wikipedia revisions under
CC BY-SA; attribution will live in data/LICENSE-DATA.md.