Skip to content

feat(ai): record expected MCP failure messages on the dispatcher span - #1961

Open
Makisuo wants to merge 18 commits into
perf/run-sql-trace-seek-row-capfrom
feat/mcp-failure-message-span
Open

Makisuo wants to merge 18 commits into
perf/run-sql-trace-seek-row-capfrom
feat/mcp-failure-message-span

Conversation

@Makisuo

@Makisuo Makisuo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Invalid-input and budget failures recorded their message only in the Warn log, so finding out why a tool fails meant joining logs to traces by trace id. That join was how the run_sql failure analysis behind #1959 and #1960 was done. The message, capped at 500 characters, is now maple.mcp.error.message on McpToolDispatcher.call, next to error.type and maple.mcp.error.category.

Checked with dispatcher.test.ts, which now asserts the attribute, plus the scoped apps/ai typecheck and bun run lint:core.

Stacked on #1960.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Invalid-input and budget failures kept their message only in the Warn log,
so measuring a tool's failure reasons meant joining logs to spans by trace.
The message, capped at 500 characters, is now maple.mcp.error.message.
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 171abc32-b27e-46a6-a078-05fcb62815d8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
A capped attribute on an existing span, in the repo's maple.* namespace, pinned by a test that fails without it.
no findings · tests covered · risk low

Records an expected MCP 4xx's message as maple.mcp.error.message (500-char cap) on the McpToolDispatcher.call span, so tool failure reasons no longer need a logs join. Contained and safe to merge.

  • recordExpectedMcpFailure annotates the dispatcher span with the capped error.message
  • dispatcher.test.ts asserts the attribute on the decode-error path

Production impact

Telemetry this change adds and removes (1)
  • ➕ attribute maple.mcp.error.message · apps/ai/src/mcp/expected-failures.ts:60
What was checked
  • formatDecodeFailure emits only parameter names, types and descriptions, never argument values (decode-issues.ts:245)
  • Traced the McpInvalidInputError/McpAuthInvalidError messages through params.ts, resolve-tenant.ts, source-errors.ts: ids, field names and count limits, no credentials or contact data
  • Attribute follows CLAUDE.md's maple.* span-attribute rule; ingest has no attribute length or key allow-list to truncate it

8f6b6c1 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo Makisuo added the preview:web Deploy a Workers-only PR preview (no ingest/Electric); see docs/pr-previews.md label Oct 30, 2026
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 00:47 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🍁 Maple PR preview

The preview is ready:

Service Preview URL
Web app-pr-1961.maple.dev
API api-pr-1961.maple.dev

Web-only preview (preview:web): seeded demo data, no ingest gateway or Electric sync. Add preview for the full stack.

Smoke test: passed

Sign in with the PR preview form on the sign-in page and the preview access key (PREVIEW_SIGNIN_KEY in Infisical), or run bun run preview:signin 1961 for a signed-in link.

Deployed from 7d6b523 · View workflow run

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 00:52 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
One namespaced span attribute on an existing catch path, pinned by dispatcher.test.ts:273; the message it carries is already logged at Warn beside it.
no findings · tests covered · risk low

recordExpectedMcpFailure now also writes the capped error message to maple.mcp.error.message on the dispatcher span, so a tool's 4xx reasons no longer need a trace-to-log join. Contained, tested and safe to merge.

Production impact

Telemetry this change adds and removes (1)
  • ➕ attribute maple.mcp.error.message · apps/ai/src/mcp/expected-failures.ts:60
What was checked
  • Only the seven tags in MCP_EXPECTED_FAILURE_STATUS reach the attribute; McpQueryError and McpAuthUnavailableError return early (expected-failures.ts:54)
  • The same message is already logged unchanged at Warn (expected-failures.ts:64), so the diff exposes no new class of data to telemetry
  • describeMalformedBearer quotes only an unexpanded placeholder, never a real key (resolve-tenant.ts:98-104)

43a154e · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:12 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:13 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:22 — with GitHub Actions Active
@Makisuo

Makisuo commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Preview test (app-pr-1961 / api-pr-1961, commit 43a154e)

A separate verification agent called the preview's MCP run_sql directly: 106 tool calls, run with a short-lived API key for the Preview org.

Claim Result
Column fixes (service.name, Events.Name, trace_id, ServiceVersion, backticked table, t.trace_id in a join) Pass. Each returns rewrites and a "Ran with" note.
No rewrite in literals, comments, AS service, CTE names, Severity, a shared column in a join Pass
Stops after 3 rewrites; a later failure names the rewrites already run Pass
Errors free of internal database names and org id Fail at 43a154e: the preview's pr_1961. database and the org id leaked. The scrubber only knew maple_us/maple_eu. Fixed in 92a6440 (any database name, alias kept, leftover org ids blanked); recheck below once redeployed.
Unbounded trace-id lookups Pass. Identical rows to the same query with a 30-day window, for single ids, IN lists of 3 and 100, if()/countIf, NOT, OR, aliases, a subquery and a nonexistent id.
Row cap Pass. No LIMIT / LIMIT 5000 returns 1,000 rows plus the notice; LIMIT 1000 (with or without OFFSET, or ;) gets no notice.
Regressions (aggregations, $__orgFilter(alias) joins, CTE, UNION ALL, missing org filter rejected) Pass

Not changed here: the maple CLI drops rewrites, notices, and all but the first error line, so CLI users don't see these yet. That's a follow-up in apps/cli.

Traces (deployment.environment = pr-1961)

Span Calls p50 p95 max
McpTool.runSql 106 84 ms 454 ms 609 ms
RawSql.execute 131 47 ms 206 ms 428 ms
WarehouseQueryService.executeSql 126 49 ms 204 ms 428 ms
  • Statements per call: 85 calls ran one statement (p50 73 ms), 19 ran two because of a rewrite or a seek fallback (p50 140 ms), and 2 ran four because of three rewrites (max 518 ms). A rerun costs about one extra warehouse round trip.
  • Span attribute: maple.mcp.error.message is on all 16 invalid-input failures; query failures keep their message in the span status.
  • No N+1 or slow spans beyond the per-statement audit insert, which already existed.

@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Four-line span-attribute addition pinned by the dispatcher test; the message sources I traced carry no secrets or user data.
no findings · tests covered · risk low

Adds maple.mcp.error.message, capped at 500 characters, to the dispatcher span for expected MCP 4xx failures, so a tool's reason no longer needs a join to logs. Safe to merge.

  • recordExpectedMcpFailure annotates the current span with the failure message, capped at 500 characters
  • dispatcher.test.ts asserts the attribute for a decode failure
What was checked
  • Auth messages echo only ${VAR}/<...> placeholders; real maple_ak_ keys return early (resolve-tenant.ts:99-104)
  • Budget, not-ready and unavailable messages are static guidance, not caller input (map-warehouse-error.ts:47)
  • raw-sql-error-text.ts reached this branch from the base, so the PR diff is only expected-failures.ts and the test

3fca428 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:24 — with GitHub Actions Active
@Makisuo

Makisuo commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Recheck on the redeployed preview (3fca428): the four error-text repros from the preview test now contain no pr_1961. and no org id, and keep the caller's alias (FROM logs AS l). Error-text claim: pass. A follow-up commit keeps parentheses balanced for an unaliased wrapper inside parentheses (from the review note on #1959).

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:33 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Four added lines annotate an existing dispatcher span with an already-logged message; no call site, message content or test was left unchecked.
no findings · tests covered · risk low

Records the expected MCP 4xx failure message as maple.mcp.error.message on the McpToolDispatcher.call span, capped at 500 characters, so a tool's failure reason needs no join to logs. Contained and safe to merge.

  • recordExpectedMcpFailure annotates the current span with maple.mcp.error.message, truncated to MAX_SPAN_MESSAGE_CHARS
  • dispatcher.test.ts pins the attribute on the decode-failure path
What was checked
  • Every recordExpectedMcpFailure call site (dispatcher and resolve-tenant) passes a constant-text or identifier-only message; no token, email or record reaches the attribute
  • git diff --stat base..head lists only these two files; the other files named as changed since the last review came in from the base branch

4710390 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:35 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:39 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
One attribute added next to existing MCP span keys, its content already emitted in the Warn log on the same call, pinned by dispatcher.test.ts:273.
no findings · tests covered · risk low

Records the capped failure message as maple.mcp.error.message on the MCP dispatcher span so a tool failure's reason is readable without joining logs to traces. Contained and tested; safe to merge.

What was checked
  • error.message already left the process via the existing Warn log annotation on the same call (expected-failures.ts:64), so the span carries no new field
  • Messages for the six expected tags name tools, parameters and timestamps; describeMalformedBearer echoes only placeholder-shaped tokens (resolve-tenant.ts:98)
  • New key follows the maple.* convention and sits beside the existing error.type / maple.mcp.error.category spellings; no second spelling of a message key exists in live spans

fb08629 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:51 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
One bounded attribute on an existing span; both the attribute and its 500-char cap are pinned by tests, and no message it copies carries a credential.
no findings · tests covered · risk low

Records the expected MCP 4xx failure message, capped at 500 characters, as maple.mcp.error.message on the McpToolDispatcher.call span, so a tool's failure reason no longer needs a logs-to-traces join. Contained and safe to merge.

What was checked
  • McpAuthInvalidError echoes the bearer only when it is a ${VAR}/<placeholder> name, never a real key (resolve-tenant.ts:98)
  • No error.message span key exists in the org's live attributes, so maple.mcp.error.message is not a second spelling
  • Whole SQL already reaches spans as db.query.text, so the copied message adds no new exposure

9f9c473 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:54 — with GitHub Actions Active
@maple-review-bot

Copy link
Copy Markdown

Note

⏳ Maple is reviewing this pull request at 0eafacb. This comment updates with the review when it finishes.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 01:57 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Two-line annotation on an existing Ok-status span, pinned by a new cap test; the sql-identifiers files named as changed are no longer in this diff.
no findings · tests covered · risk low

Adds maple.mcp.error.message, capped at 500 characters, to the McpToolDispatcher.call span for expected MCP 4xx failures so the reason no longer needs a join to logs. The attribute, its name and its tests are sound; safe to merge.

What was checked
  • Cap test fails if the slice is dropped (dispatcher.test.ts:260, 5,000-char message)
  • Decode text carries tool and parameter names, not argument values (decode-issues.ts:245)
  • The same message was already logged uncapped at Warn (expected-failures.ts:64), so no new data class reaches telemetry

bb10186 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:00 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:01 — with GitHub Actions Active
@Makisuo

Makisuo commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Live recheck on the preview at bb10186 (the review-round identifier fixes), calling MCP run_sql directly:

SQL shape Expected Result
SELECT service.name ... FROM traces rewrite to ServiceName Rewritten, with a rewrites entry and the "Ran with" note
FROM (SELECT ...) service (alias without AS) no rewrite Agent's own unknown-identifier error, not rewritten
FROM traces service ... $__orgFilter(service) no rewrite Not rewritten; FROM and org filter intact
t.body with traces t joined to logs l no rewrite Not rewritten
l.body with logs l rewrite to l.Body Rewritten

The quoted, dotted table alias fix (8fd3173) is in the deploy running now; unit tests cover it.

@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Telemetry-only attribute in a shared helper, with both the 500-char cap and the dispatcher span's attribute pinned by tests.
no findings · tests covered · risk low

Records the expected-failure message, capped at 500 characters, as maple.mcp.error.message on the MCP dispatcher span so a tool's failure reason no longer needs a logs-to-traces join. Telemetry-only and covered by tests; safe to merge.

  • recordExpectedMcpFailure adds a capped maple.mcp.error.message attribute to the current span
  • dispatcher.test.ts pins the cap and the attribute on the dispatch span
What was checked
  • Message text is warehouse/validation text: live maple.mcp.error.message values show no personal data or credentials
  • maple.mcp.error.message follows the maple.* rule; no existing error.message span key to duplicate (explore_attributes)
  • Cap applies on every path through recordExpectedMcpFailure (dispatcher.ts:95-146, resolve-tenant.ts:213,335)

4139263 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:04 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Four added lines put a capped, PII-free message on an existing span, pinned by a test that fails if the cap or the attribute goes away.
no findings · tests covered · risk low

recordExpectedMcpFailure now also puts a 500-character prefix of the failure message on the current span as maple.mcp.error.message, so a tool's failure reason is queryable without joining to logs. Both the attribute and its cap are asserted; safe to merge.

  • recordExpectedMcpFailure records maple.mcp.error.message, capped by MAX_SPAN_MESSAGE_CHARS
  • The Warn log keeps the uncapped message, so log detail is unchanged
  • dispatcher.test.ts pins the cap and the decode-error message on the call span

Before merge

What was checked
  • No credential can reach the attribute: describeMalformedBearer returns undefined for maple_ak_ keys and JWTs, and only echoes unexpanded placeholders (lib/resolve-tenant.ts:99-104)
  • run_sql error text is scrubbed of the org id before recording (lib/raw-sql-error-text.ts:30-32)
  • The cap test can fail: String(undefined) yields "undefined", which breaks the startsWith prefix assertion (dispatcher.test.ts:269-273)

ba6dae8 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:07 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
Contained telemetry-only change on an existing span attribute, pinned by a dispatcher test; the raw-sql files named in the kickoff are no longer in this head's diff.
no findings · tests covered · risk low

Records the expected-failure message as maple.mcp.error.message on the McpToolDispatcher.call span, capped at 500 characters, alongside the existing error.type and status attributes. Safe to merge.

  • recordExpectedMcpFailure annotates the span with the first 500 characters of the failure message
What was checked
  • Attribute key follows the repo's maple.* custom-attribute rule (CLAUDE.md) and matches the existing maple.mcp.error.category spelling
  • The cap test fails if the attribute is absent (String(undefined) breaks startsWith) and if the truncation is removed
  • Auth failure messages reaching this helper are fixed strings (resolve-tenant.ts:171-262), so no bearer token lands in the attribute

9a7c367 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:10 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:24 — with GitHub Actions Active
@Makisuo

Makisuo commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Live recheck of the raw-SQL gate fix (7db5041) on the preview at 9a7c367:

SQL Before Now
FROM traces JOIN logs l ... $__orgFilter(l), correctly spelled Rejected: $__orgFilter(l) must name a table Returns rows
Same query with l.body Rejected by the gate Rewritten to l.Body, returns rows
FROM traces JOIN (SELECT 'org_abc' AS OrgId) f ... $__orgFilter(f) Rejected Still rejected

@maple-review-bot

maple-review-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge
A four-line span annotation with both presence and the 500-character cap pinned by tests in dispatcher.test.ts; no behavior outside telemetry changes.
no findings · tests covered · risk low

Adds maple.mcp.error.message — a 500-character prefix of the failure text — to the dispatcher span so an expected MCP 4xx can be diagnosed from traces without joining logs. Contained and tested; safe to merge.

  • recordExpectedMcpFailure annotates maple.mcp.error.message, capped at MAX_SPAN_MESSAGE_CHARS
  • dispatcher.test.ts pins the cap and that a short message arrives whole
What was checked
  • Every mapped tag reaches the annotator through recordExpectedMcpFailure (apps/ai/src/mcp/dispatcher.ts:94-148), so budget and invalid-input failures both get the attribute
  • The cap test fails if the attribute is absent, since String(undefined) is not a prefix of the message
  • Message builders traced for secrets: the malformed-bearer text echoes only placeholder names matched by PLACEHOLDER (apps/ai/src/mcp/lib/resolve-tenant.ts:92-104)

7d6b523 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview October 30, 2026 02:27 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
pr-preview — 7d6b523f Deployed Oct 30, 2026 by Makisuo via End-to-end tests against the preview #7801
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

preview:web Deploy a Workers-only PR preview (no ingest/Electric); see docs/pr-previews.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant