Skip to content

fix(core): distinguish WebFetch URL errors from network errors#33393

Open
lexlian wants to merge 1 commit into
anomalyco:devfrom
lexlian:fix/webfetch-invalid-url-error
Open

fix(core): distinguish WebFetch URL errors from network errors#33393
lexlian wants to merge 1 commit into
anomalyco:devfrom
lexlian:fix/webfetch-invalid-url-error

Conversation

@lexlian

@lexlian lexlian commented Jun 22, 2026

Copy link
Copy Markdown

Issue for this PR

Closes #33073

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

WebFetch's URL validation collapses every failure (malformed URL, wrong scheme, transport failure, timeout, response too large, MIME rejection, HTML conversion crash) into the same "Unable to fetch <url>" message. The model cannot tell whether it sent a bad URL or hit an unreachable host. The reporter's "Expected behavior" asks for a clear distinction between format errors and network errors.

This PR introduces two-layer URL validation, matching Claude Code's WebFetch shape:

  1. Schema-level parseability check via Schema.makeFilter on the url field rejects inputs that new URL() cannot parse. These surface through the standard tool-input channel as Invalid tool input: Invalid URL\n at ["url"] (the same path every other tool's schema rejection takes via Tool.make's built-in Schema.decodeUnknown step in packages/core/src/tool/tool.ts:82).

  2. Tool-layer scheme check (kept from the original assertHttpUrl) rejects non-http/https schemes with a typed InvalidUrlError. These surface as Invalid URL <url>: URL must use http:// or https://, carrying the reason into the message so the model knows why the request was rejected.

Network, timeout, body-size, MIME, and HTML-conversion failures keep the generic Unable to fetch <url> message so existing tests and downstream consumers do not break.

Implementation notes

  • Replaces Effect.try({ catch: (error) => error }) at the validation site with Effect.gen + Effect.fail(InvalidUrlError) so the tagged error reaches the boundary mapper without erasure through unknown → E. (Effect.try's catch returns its value as the failure type, but the surrounding Effect.mapError(() => …) was discarding it.)
  • Removes the now-unused assertHttpUrl helper (line 81-83 of the old file) — the same check is inlined in the new Effect.gen.
  • The second Effect.try({ catch: (error) => error }) site added by PR fix(core): bound web tool failures #33259 for HTML-to-Markdown conversion (lines 162-165) has the same generic-message anti-pattern but does not cause user-visible failures today. Left alone in this PR; flagged in the solution doc as a follow-up.

Why two layers (and not just Schema-level)

Claude Code uses a JSON-schema format: "url" constraint to reject unparseable URLs before the tool runs, and a separate tool-layer check for the http/https scheme. Two layers produce two distinct error categories — exactly the "format vs. network" distinction the bug report asks for. The tool-layer scheme check is also strictly stricter than Claude Code's (Claude Code leaks file:// to the network and returns a confusing socket error — cross-checked 2026-06-22).

How did you verify your code works?

$ cd packages/core && bun test ./test/tool-webfetch.test.ts
14 pass, 0 fail, 35 expect() calls  [814ms]

$ bun typecheck
Tasks: 23 successful, 23 total  (FULL TURBO)

$ bun test
# 1028 pass across 132 files. The 3 unrelated failures
# (ProjectCopy/Worktree, Watcher/.git, LocationServiceMap timeouts)
# are pre-existing — none touch webfetch.

Manual verification: invoked webfetch with not-a-url, ftp://example.com, file:///etc/passwd, and an unreachable host via the live tool:

Input Output
not-a-url Invalid tool input: Invalid URL\n at ["url"] (no permission, no network)
ftp://example.com Invalid URL ftp://example.com: URL must use http:// or https:// (no permission, no network)
file:///etc/passwd Invalid URL file:///etc/passwd: URL must use http:// or https:// (no permission, no network)
unreachable host Unable to fetch <url> (permission + network, as before)

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Notes for the reviewer

  • Security posture unchanged: this fix only changes the message for the file:// rejection (from Unable to fetch file:///etc/passwd to Invalid URL file:///etc/passwd: URL must use http:// or https://). The rejection itself, the absence of permission/network calls, and the read-tool-only path for local files are all preserved unchanged (verified against packages/opencode/src/tool/read.ts:241-260 and packages/core/src/location-mutation.ts:37).
  • Cross-check: Claude Code's WebFetch was probed with the same inputs on 2026-06-22 to validate the design. Findings recorded in DEVS/fixes/SOLUTION-33073.md ("Cross-check: Claude Code WebFetch" and "Security context: local file access" sections).

WebFetch's URL validation collapsed every failure (malformed URL,
wrong scheme, transport failure, timeout, response too large, MIME
rejection, HTML conversion crash) into the same "Unable to fetch"
message. The model could not tell whether it sent a bad URL or hit
an unreachable host.

This commit introduces two-layer URL validation, matching Claude
Code's WebFetch shape:

1. Schema-level parseability check via Schema.makeFilter rejects
   inputs that `new URL()` cannot parse. These surface through the
   standard tool-input channel as
   "Invalid tool input: Invalid URL\n  at [\"url\"]".
2. Tool-layer scheme check (kept from the original assertHttpUrl)
   rejects non-http/https schemes with a typed InvalidUrlError.
   These surface as
   "Invalid URL <url>: URL must use http:// or https://".

Network, timeout, body-size, MIME, and HTML-conversion failures
keep the generic "Unable to fetch <url>" message so existing tests
and downstream consumers do not break.

Replaces Effect.try({ catch: (error) => error }) at the validation
site with Effect.gen + Effect.fail(InvalidUrlError) so the tagged
error reaches the boundary mapper without erasure. Removes the
now-unused assertHttpUrl helper.

Adds two new tests (parse error, ftp scheme) and updates the
existing file:///etc/passwd test to expect the new typed message.

Closes anomalyco#33073
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] WebFetch tool URL validation is a no-op: invalid URLs pass through silently

1 participant