Skip to content

[rig-sampler] Improve missing-required-field error messages with richer type descriptions#42

Merged
pelikhan merged 1 commit into
mainfrom
rig-sampler/22-config-normalizer-00b5e5f3ce892494
Jul 23, 2026
Merged

[rig-sampler] Improve missing-required-field error messages with richer type descriptions#42
pelikhan merged 1 commit into
mainfrom
rig-sampler/22-config-normalizer-00b5e5f3ce892494

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Samples run

Five consecutive samples were executed via the stub Copilot SDK runner (npm run sample):

# Sample Result Notes
22 22-config-normalizer.ts ✅ Pass Output { normalized: s.unknown, warnings: s.array(s.string) }s.unknown is correctly loose here; no repair needed
23 23-schema-inference.ts ✅ Pass Nested object array in output; stub synthesised valid output on first turn
24 24-error-message-improver.ts ✅ Pass Uses s.optional(s.string) for context; optional field correctly excluded from required
25 25-migration-guide.ts ✅ Pass Deeply nested s.array(s.object({ before, after })) — first-turn success
26 26-design-review.ts ✅ Pass Mirrors sample 25's migration-guide spec; no repair turns observed

All 5 samples completed in a single turn with no repair loops.

Improvement: richer missing-field error messages

What was observed

Samples 23, 25, and 26 all declare output schemas that include required array, object, and nullable fields. When a model omits one of these fields, the existing validator surfaced:

$.items: missing required field (expected value)

The word "value" is unhelpful — it gives the model no signal about what kind of value to supply during a repair loop.

What changed

skills/rig/rig.ts — extracted a describeSchemaType helper that is called instead of the inline ternary:

  • s.array(...) → reports "array"
  • s.object(...) / s.record(...) → reports "object"
  • s.nullable(s.number) → reports "number | null"
  • s.enum("open", "closed") → reports "open" | "closed" (inline enum values)
  • Primitive schemas (s.string, s.boolean, etc.) → unchanged, still reports the type name

src/rig.test.ts — added three new cases to the "missing required field error message" describe block covering the array, nullable, and enum improvements.

Why this matters

The repair prompt (defaultRepairPrompt) includes the exact validation error string. A precise message like "expected number | null" or "expected array" is directly actionable for the model in a repair turn, reducing the chance of a second incorrect attempt.

Generated by Daily Rig Sampler · sonnet46 79.7 AIC · ⌖ 8.18 AIC · ⊞ 5.4K ·

…ptions

When a required object field is absent, the validation error now reports
a meaningful type hint for all schema kinds — array, object, nullable,
and enum — instead of falling back to the generic 'value'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review July 23, 2026 13:25
@pelikhan
pelikhan merged commit 56c2bc2 into main Jul 23, 2026
3 checks passed
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd — one minor gap found; leaving as COMMENT (no blocking issues).

📋 Key Themes & Highlights

Finding

  • Missing test for object branch: describeSchemaType has five branches; only four are tested. The untested properties/additionalProperties branch would silently regress to "value", exactly undoing the intent of this PR.

Positive Highlights

  • ✅ Clean extraction of describeSchemaType — single-responsibility, well-named, easy to unit-test independently if needed
  • ✅ Recursive nullable handling (describeSchemaType(inner) + " | null") is elegant and correct
  • ✅ Enum values are quoted via JSON.stringify, so string values render with quotes (e.g. "open" | "closed") — this is exactly what a model needs to produce valid JSON
  • ✅ Three new tests are well-structured and assertive; the pattern is consistent with the existing describe block
  • ✅ The fallback return "value" is preserved — safe for unknown/future schema shapes

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 27.4 AIC · ⌖ 4.21 AIC · ⊞ 6.3K
Comment /matt to run again

Comment thread skills/rig/rig.ts
return (schema as EnumSchema).enum.map((v: Json) => JSON.stringify(v)).join(" | ");
}
if ("items" in schema) {
return "array";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The "object" branch in describeSchemaType (the properties / additionalProperties check) has no corresponding test. If that branch regresses, the repair prompt silently falls back to "value" — the exact issue this PR was designed to fix.

💡 Suggested test
it("reports a missing required object field with type 'object'", () => {
  const schema = s.object({ meta: s.object({ key: s.string }), name: s.string });
  const result = analyzeResponse(JSON.stringify({ name: "x" }), schema, "test", 1);
  expect(result.ok).toBe(false);
  if (!result.ok) {
    expect(result.error.message).toContain("missing required field");
    expect(result.error.message).toContain("$.meta");
    expect(result.error.message).toContain("object");
  }
});

All four describeSchemaType non-trivial branches (array, nullable, enum, object) should have coverage to match the precision the other three tests establish.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant