[rig-sampler] Improve missing-required-field error messages with richer type descriptions#42
Conversation
…ptions When a required object field is absent, the validation error now reports a meaningful type hint for all schema kinds — array, object, nullable, and enum — instead of falling back to the generic 'value'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd — one minor gap found; leaving as COMMENT (no blocking issues).
📋 Key Themes & Highlights
Finding
- Missing test for
objectbranch:describeSchemaTypehas five branches; only four are tested. The untestedproperties/additionalPropertiesbranch would silently regress to"value", exactly undoing the intent of this PR.
Positive Highlights
- ✅ Clean extraction of
describeSchemaType— single-responsibility, well-named, easy to unit-test independently if needed - ✅ Recursive nullable handling (
describeSchemaType(inner) + " | null") is elegant and correct - ✅ Enum values are quoted via
JSON.stringify, so string values render with quotes (e.g."open" | "closed") — this is exactly what a model needs to produce valid JSON - ✅ Three new tests are well-structured and assertive; the pattern is consistent with the existing describe block
- ✅ The fallback
return "value"is preserved — safe for unknown/future schema shapes
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 27.4 AIC · ⌖ 4.21 AIC · ⊞ 6.3K
Comment /matt to run again
| return (schema as EnumSchema).enum.map((v: Json) => JSON.stringify(v)).join(" | "); | ||
| } | ||
| if ("items" in schema) { | ||
| return "array"; |
There was a problem hiding this comment.
[/tdd] The "object" branch in describeSchemaType (the properties / additionalProperties check) has no corresponding test. If that branch regresses, the repair prompt silently falls back to "value" — the exact issue this PR was designed to fix.
💡 Suggested test
it("reports a missing required object field with type 'object'", () => {
const schema = s.object({ meta: s.object({ key: s.string }), name: s.string });
const result = analyzeResponse(JSON.stringify({ name: "x" }), schema, "test", 1);
expect(result.ok).toBe(false);
if (!result.ok) {
expect(result.error.message).toContain("missing required field");
expect(result.error.message).toContain("$.meta");
expect(result.error.message).toContain("object");
}
});All four describeSchemaType non-trivial branches (array, nullable, enum, object) should have coverage to match the precision the other three tests establish.
Samples run
Five consecutive samples were executed via the stub Copilot SDK runner (
npm run sample):22-config-normalizer.ts{ normalized: s.unknown, warnings: s.array(s.string) }—s.unknownis correctly loose here; no repair needed23-schema-inference.ts24-error-message-improver.tss.optional(s.string)forcontext; optional field correctly excluded from required25-migration-guide.tss.array(s.object({ before, after }))— first-turn success26-design-review.tsAll 5 samples completed in a single turn with no repair loops.
Improvement: richer missing-field error messages
What was observed
Samples 23, 25, and 26 all declare output schemas that include required
array,object, andnullablefields. When a model omits one of these fields, the existing validator surfaced:The word
"value"is unhelpful — it gives the model no signal about what kind of value to supply during a repair loop.What changed
skills/rig/rig.ts— extracted adescribeSchemaTypehelper that is called instead of the inline ternary:s.array(...)→ reports"array"s.object(...)/s.record(...)→ reports"object"s.nullable(s.number)→ reports"number | null"s.enum("open", "closed")→ reports"open" | "closed"(inline enum values)s.string,s.boolean, etc.) → unchanged, still reports the type namesrc/rig.test.ts— added three new cases to the"missing required field error message"describe block covering the array, nullable, and enum improvements.Why this matters
The repair prompt (
defaultRepairPrompt) includes the exact validation error string. A precise message like"expected number | null"or"expected array"is directly actionable for the model in a repair turn, reducing the chance of a second incorrect attempt.