test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) - #1438
test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414)#1438devin-ai-integration[bot] wants to merge 6 commits into
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
|
Runtime verification — parser fuzz lane ✅Tested end-to-end from the shell on this branch (Node 22, no UI). Failure detection actually works (the key claim)Temporarily injected faults into The printed Green runs, determinism, scale
Corpus, replay lane, workflow
Replay-script quoted-token fix (before/after)Repo tree clean ( |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
d70a045 to
cf3c5d0
Compare
Review: changes requestedThis is directionally solid—the target registry, deterministic mutator, worker cursor, and replay-script regression are good foundations—but it is not ready for the #1412/#1414 contract yet.
Readiness evidence is also incomplete: GitHub has no |
…tests (#1414) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
@thymikee addressed all three in [P1] Scheduled-lane envelope. #1430 is not merged, so there is nothing to consume yet — I implemented the equivalent here rather than blocking. { "schemaVersion": 1, "lane": "parser-fuzz", "result": "pass",
"startedAt": "…", "finishedAt": "…", "durationMs": 2229,
"provenance": { "commitSha": …, "ref": …, "workflowRunId": …, "workflowRunAttempt": …,
"nodeVersion": "v22.14.0", "tool": "scripts/fuzz/run.ts", "corpusEntries": 8 },
"config": { "mode": "generate", "seed": 1, "iterations": 300, "caseTimeoutMs": 2000, "targets": [...] },
"targetRuns": [ { "target": "cli-args", "cases": 300, "failures": 0, "durationMs": 490 }, ... ],
"failures": [ /* incl. artifact path */ ], "reproCommands": [ ... ] }The upload step is now [P1] Artifact → corpus promotion. Correct — the replay path returned before promotion could run. Tested end-to-end in [P2] Tests of the invariant/watchdog itself. Added Writing that test found a real watchdog bug: worker startup (type stripping + parser imports) counted against the per-case budget, so under load a fast untyped throw could be misreported as a On the readiness evidence: a Locally green: |
|
Re-review at head 5a382f6: artifact-to-corpus promotion and the tests-of-tests are fixed, but three P1 blockers remain. First, issue 1430 makes the shared standard artifact envelope and scheduled health/freshness integration a birth requirement for this new lane; the PR still uses a lane-specific envelope with no shared monitoring integration. Second, if the self-check fails, the fuzz step never runs, so no envelope is written; the always-run upload warns and the summary then fails while trying to read the missing file. Every terminal path needs a valid envelope and summary. Third, the watchdog interval starts before the worker reports ready, so slow worker startup can still be misclassified as a hung parser case. Start case timing only after ready and add a delayed-startup regression test. All PR checks are green, but no Replay Nightly run exists for this head, so the scheduled route itself is unvalidated. |
…tchdog after ready (#1414) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
@thymikee all three addressed in 1. Shared envelope (#1430). #1430 has no branch or PR yet, so instead of a second lane-specific shape I extracted the standard writer to 2. Every terminal path has an envelope. Self-check and fuzz now write to separate artifact subdirs ( 3. Watchdog arms only after Local: On the unvalidated scheduled route: |
|
Re-review at head 0f4d99a: the artifact promotion, tests-of-tests, self-check failure handling, and worker-ready watchdog issues are fixed. Two blockers remain. First, umbrella issue 1412 explicitly requires issue 1430 health/freshness monitoring to land before or with this scheduled lane; this PR adds an envelope producer but no derived scheduled-lane health consumer, dark-lane alerting, or freshness exposure, and no Replay Nightly run exists for this head. Second, malformed workflow-dispatch options throw in readFuzzOptions before run() enters its envelope-writing catch, so the claim that every config/crash terminal path produces an envelope is still false. Move option parsing under the guarded outcome path and cover malformed input. Ordinary PR checks are green, but the scheduled route is not yet ship-ready. |
|
Both addressed in 1. Envelope on the option-parse path. Option parsing now happens inside the guarded path: 2. Health/freshness consumer (#1430). #1430 still has no branch, so rather than ship the lane without its watcher I built the consumer here:
Run against the live API just now, which is itself the dark-lane evidence you asked for — it immediately caught a real outage rather than a synthetic one:
Blocked on you for the scheduled-route run. I tried to dispatch it and my token can't: Local gates: |
|
Re-review at head dd700b6: malformed fuzz options are now guarded and tested, but the new scheduled-health path has three P1 blockers. First, scheduled-lane-health.yml omits actions: read while calling the Actions workflow-runs API; with an explicit permissions map the omitted scope is none, so collection will 403. Second, health.ts has no top-level terminal error path, so an API failure exits before writing its snapshot or own required error envelope. Third, a readable workflow with zero scheduled runs is marked dark immediately, contradicting the two-cadence grace and falsely alerting for newly born lanes before their first due run. Add the permission, make terminal failures emit an error envelope, and model first-due/two-cadence grace with dispatched-route evidence. Ordinary PR checks are green, but neither scheduled route has run on this head. |
|
Re-review at 1. 2. Terminal error path — 3. First-due grace — the zero-run case now goes through Live dry-run against the real API on this head (classification unchanged, and the new registration lookup doesn't perturb it): Local gates: format/typecheck/lint green, fallow Still blocked on the dispatched-route evidence, and I need you for it. Both attempts get
Otherwise, from an account with dispatch rights: |
|
Latest delta fixes One P1 remains: first-run grace is anchored to workflow Also replace the new raw No live scheduled/dispatch evidence exists yet; iOS Smoke is currently red on an unrelated pre-existing timing assertion, so this head is not fully green. |
|
Fixed in P1 — grace anchored to the wrong date. You're right, and That's the commit that changed the number of Process-execution rule. Replaced with Local: format/typecheck/lint green, fallow On the red check: all 22 checks on this PR are green ( Still no dispatched-route evidence: |
Summary
Closes #1414 (Track A of #1412). Parsers are the front door for agent-authored input; nothing asserted they fail well. This adds a fuzz lane over the five parser entry points and enforces exactly one invariant:
Targets (
scripts/fuzz/targets.ts):parseArgs(strict flags),parseSelectorChain,parseReplayScriptDetailed,readCliBatchStepsJson,parseMaestroProgram. Adding a parser to the lane is one entry in that array.It already caught a real bug.
readQuotedReplayTokenscans for the closing quote and then hands the literal toJSON.parse, so a.adline likefill @e1 --text "hello wor\ld"(invalid escape, raw tab, or\u0000) escaped as a bareSyntaxError. Now a typedINVALID_ARGSwith a hint naming the escaping rule; three of those inputs are pinned in the corpus and insrc/replay/__tests__/script.test.ts.Hang detection is real, not elapsed-time bookkeeping. A synchronous parser cannot be interrupted from inside its own tick, so cases run in a worker thread that publishes its case index into a
SharedArrayBufferbefore each case; the runner watchdogs that cursor and, when it stops advancing past--case-timeout-ms, reports the exact input and terminates the thread. The worker postsreadybefore the first case so startup (type stripping, parser imports) is not charged to a case budget.The harness is tested against itself.
scripts/fuzz/self-check-targets.tsholds three broken-on-purpose targets (bareError,AppErrorwith a blank hint,for(;;){}) that onlyregistry.tsresolves, so a normal run never sees them.pnpm fuzz:parsers --self-check(nightly step) andscripts/fuzz/harness.test.ts(unit lane) require each failure kind to be reported — otherwise a regressed classifier or watchdog would leave every test green.No new dependency. #1413 will bring fast-check; this lane only needs reproducible cases it can print as a repro command, so it uses a seeded mulberry32 PRNG over a seed corpus + mutators (hostile chunks, truncation, slice duplication, long runs). Same seed = same cases everywhere.
Envelope, promotion, lanes.
<artifact-dir>/run-envelope.json(schemaVersion, commit/ref/run provenance, seed + config, per-target cases/failures/durations, result, repro commands); the nightly uploads it withif: always()and echoes it into the step summary, so a green scheduled run is auditable. obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 is standardizing this shape; when it lands this module becomes the mapping onto its shared writer.repro:andpromote: … --append-corpus, and--input-file --append-corpusappends the downloaded artifact toscripts/fuzz/corpus/regressions.json(sorted, deduped).scripts/fuzz/corpus-replay.test.tsreplays the whole corpus plus every target's seeds in the unit lane (~40ms).Parser Fuzz Laneridesreplays-nightly.yml(device-free, ubuntu, 50k cases/target, seeded bygithub.run_number).Verification, locally:
--self-checkand the fuzz unit tests stable across repeat runs.throw new Error(...)/for(;;){}injected into aparseSelectorChainbranch are reported asuntyped-throw/hang: case N did not finish within …mswith working repro commands (both reverted).pnpm format:check,pnpm lint,pnpm typecheck,pnpm check:fallow --base origin/main,pnpm check:layering,pnpm check:affected:test, andpnpm test:unit(518 files / 4578 tests) green.Link to Devin session: https://app.devin.ai/sessions/45070ada3381444c89ac3bb95b79a611
Requested by: @thymikee