fix(auth): ignore non-metadata JSON when probing for protected resource metadata

[easyinplay]

Why

AuthorizationManager probes its base URL first when looking for RFC 9728 protected resource metadata, and probe_resource_metadata_url treats any 200 there as "this URL is the metadata document". ResourceServerMetadata has only optional fields, so an unrelated JSON object deserializes into an all-None value, and validate_resource_metadata_resource then fails hard with Protected resource metadata missing required resource field. The error propagates out of resolve_metadata(), so the .well-known fallbacks on the following lines never run.

Servers that answer GET / with a JSON health payload hit this. Against https://mcp.tavily.com today:

GET https://mcp.tavily.com/
200 application/json  {"status":"healthy","message":"MCP server is running"}

GET https://mcp.tavily.com/.well-known/oauth-protected-resource
404

GET https://mcp.tavily.com/.well-known/oauth-protected-resource/mcp
200 {"resource":"https://mcp.tavily.com/mcp",
     "authorization_servers":["https://mcp.tavily.com/"],
     "scopes_supported":["openid","offline_access"],
     "bearer_methods_supported":["header"]}

GET https://mcp.tavily.com/.well-known/oauth-authorization-server
200 {"issuer":"https://mcp.tavily.com/","authorization_endpoint":"https://mcp.tavily.com/authorize",
     "token_endpoint":"https://mcp.tavily.com/token","registration_endpoint":"https://mcp.tavily.com/register"}

The server publishes valid metadata at both well-known locations, using the RFC 9728 path-insertion form. resolve_metadata() still returns Metadata error: Protected resource metadata missing required resource field, because it stops at the health payload and never reaches either. The workaround is to configure the endpoint path rather than the host, which is not always what the caller has.

This is the JSON-object half of #810. That PR made a non-JSON body at the base URL a soft failure so discovery could continue. A JSON body that is not a metadata document still deserializes, so it takes the hard path instead.

Sampling the origin root of 36 reachable public remote MCP servers, 18 answer 200. All 18 are currently read as "this URL is the protected resource metadata document"; 17 of them escape only because their body is HTML or markdown rather than a JSON object.

Standards

RFC 9728 Section 3 requires the metadata document to live at a URL formed by inserting a well-known URI string into the resource identifier, and Section 5.1 lets a 401 point at it through WWW-Authenticate. The 2026-07-28 authorization server discovery requirements list the same two mechanisms and no others. The resource URL itself is not a metadata location under either, so a 200 from it carries no metadata claim.

RFC 9728 Section 3.2 allows additional members in the document, so tightening deserialization is not an option here: real documents carry fields this struct does not model, including the bearer_methods_supported in the Tavily document above.

What this changes

fetch_resource_metadata_from_url returns Ok(None) when the parsed document has none of resource, authorization_server, or authorization_servers, with a debug! line, matching how the same function already handles a non-200 status and a body that is not JSON. Discovery then continues to the well-known paths and to authorization server metadata.

A document carrying any of those fields still goes through validate_resource_metadata_resource unchanged, so a server that advertises a metadata URL through WWW-Authenticate and serves a malformed document there still gets a hard error. protected_resource_discovery_rejects_missing_resource and protected_resource_discovery_rejects_mismatched_resource both reach the document through an explicit challenge pointer, and both stay green.

Alternative

The narrower reading is that the base URL should never be treated as a metadata location at all, only as a source of a WWW-Authenticate challenge. That is what the TypeScript SDK does: discoverOAuthProtectedResourceMetadata only ever fetches /.well-known/oauth-protected-resource{path} or a URL taken from the challenge. Scoping the StatusCode::OK arm of probe_resource_metadata_url to the well-known probe would have the same effect here, and no existing test covers the base-URL-200 path, so that shape stays green too. I went with the document-shape check because it keeps working for servers that do serve metadata at the endpoint itself. Happy to send the other shape instead if you prefer it.

Test plan

resolve_metadata_ignores_non_metadata_json_at_the_base_url mirrors resolve_metadata_reports_authorization_server_metadata, with the base URL answering a health payload twice instead of 404: once for the probe, once for the fetch, which is what a real server does. Before the change it fails with

called `Result::unwrap()` on an `Err` value: MetadataError("Protected resource metadata missing required resource field")

and after it passes.

cargo +nightly fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p rmcp --lib --features auth

cargo test -p rmcp --lib --features auth reports 373 passed; 1 failed. The one failure is default_http_client_preserves_connection_failure_cause, which asserts on an OS connection-refused string and fails the same way on an unmodified checkout of this branch point on a non-English Windows host.

[DaleSeo]

discover_resource_metadata_url has already returned the base URL before rejecting the response body, so returning Ok(None) sends control back to resolve_metadata and skips all the protected-resource well-known candidates.

[easyinplay]

You're right, and the first commit message named the mechanism without the fix following through: it says the base URL 200 makes the well-known fallbacks never run, and then only softened the failure that came after. Pushed a second commit that fixes it where you point.

The deciding line is in probe_resource_metadata_url:

match response.status() {
    StatusCode::OK => Ok(Some(url.clone())),

"200 means this url is the document" holds for the .well-known candidates it is called with in the loop, and not for the first call, which is passed the resource itself. RFC 9728 publishes the document at the well-known URI and advertises it through the resource_metadata parameter of a challenge, so a 200 from the resource is the resource answering. Only the 401 branch says anything at that first call, which is now probe_resource_endpoint_for_challenge; the .well-known probe is unchanged.

What that was costing, from the recorded requests with the old probe and a health payload at the base URL:

GET https://mcp.example.com/
GET https://mcp.example.com/
GET https://auth.example.com/.well-known/oauth-authorization-server
GET https://auth.example.com/.well-known/openid-configuration

The base URL twice, then straight to authorization server discovery. https://mcp.example.com/.well-known/oauth-protected-resource is never requested, so a document published there is unreachable no matter what the fetch does with the body.

The check from the first commit stays. A .well-known url can answer 200 with something unrelated too, and every field of ResourceServerMetadata being optional turns that into an all-None value that fails validation fatally rather than falling through.

Two tests, and both fail with the corresponding half reverted:

• resolve_metadata_reaches_the_well_known_document_past_a_non_metadata_base_url asserts the well-known url is actually requested and that resolution comes back as ProtectedResourceMetadata. Restore the old probe call and it fails on the assertion above, printing the four requests.
• resolve_metadata_ignores_a_well_known_url_that_is_not_a_metadata_document covers the remaining guard. Drop the guard and it fails with MetadataError("Protected resource metadata missing required resource field"), the error this PR started from.

The test I had before asserted the fall-through you flagged, so it is gone.

cargo clippy --all-targets --all-features -- -D warnings is clean and cargo test -p rmcp --all-features is 499 passed, with default_http_client_preserves_connection_failure_cause failing identically on an untouched main here (it asserts a connection error string my platform words differently).

[DaleSeo]

When WWW-Authenticate points to the metadata URL, the server is explicitly saying that this document contains the metadata. If the body is {}, it used to raise a hard missing required resource field error. Now it returns Ok(None), so resolve_metadata_from_challenge continues with authorization server discovery and then falls back to the legacy endpoint. As a result, it drops the RFC 8707 resource binding without reporting it.

[DaleSeo]

The reasoning behind the second commit seems to apply to this loop too. Once the probe returns a soft failure, the loop cannot resume because the candidates have already been skipped.

[DaleSeo]

When WWW-Authenticate points to the metadata URL, the server is explicitly saying that this document contains the metadata. If the body is {}, it used to raise a hard missing required resource field error. Now it returns Ok(None), so resolve_metadata_from_challenge continues with authorization server discovery and then falls back to the legacy endpoint. As a result, it drops the RFC 8707 resource binding without reporting it.

[easyinplay]

Both are right, and they are the same seam from two sides: the guard sat in the shared fetch, while the two callers need opposite answers to "this document is not metadata".

Your two comments also sort into two axes, which is what I used to look for the rest of them:

• what the document is — every field of ResourceServerMetadata is optional, so the type cannot tell "not this document" from "this document is broken"
• where that decision is made — deciding once, outside the loop, is what let a single answer end discovery

Four commits. The first two are your comments; the last two are the same two axes applied to the rest of the path, so this does not come back a fifth time.

The loop (83743b0). discover_resource_metadata_url returned the first candidate that answered 200 and left; the body was read afterwards, outside the loop, so a candidate answering 200 with something else was only recognised once the remaining candidates were gone. The probe already held that body and discarded it, which also cost the winning candidate a second GET. The loop body now reads the body it already has, so a candidate that is not the document costs one request and the loop moves on.

Where the url came from (b165775). A .well-known candidate is a guess, so its answer only rules out that candidate. An advertised url has no better alternative to move on to, so it reports what the server got wrong instead of degrading quietly.

Origin has to survive the next hop (f2cbb68). The field-presence guard was only half the decision. validate_resource_metadata_resource ran afterwards, outside the loop, with four fatal returns that did not look at origin — so a .well-known candidate answering {"error":"not_found","resource":"/.well-known/oauth-protected-resource"} (a catch-all handler, which is what makes this common) failed on resource field is not a valid URL and took the whole run down: remaining candidates untried, authorization server discovery untried, legacy fallback unreachable. That is the bug this PR started from, reached through a different field. JSON parse, field presence and validation now happen in one place and all three answer by origin.

Nothing is requested twice (aaf529e + e7cebf1). Two leftovers on that axis:

• An advertised url that answers 404, or HTML, was soft — but the block returned instead of falling through, so a challenge pointing at a .well-known url that a reverse proxy does not route skipped the candidate loop entirely, including the path where the document actually was. It now falls through. The split is whether the server answered a document there: 200 with a body that parses is answered-and-wrong (hard), 404 or non-JSON is not-here (soft, keep looking).
• With the loop no longer stopping at the first 401, a pointer from one candidate can be another candidate, so that url could be fetched twice. A HashSet of what has been requested covers both that and the advertised pointer.

e7cebf1 is the same idea one level down: a server publishing both authorization_server and authorization_servers with the same value — the usual way to straddle the draft and RFC 9728 — had that server walked twice, all four well-known shapes each time.

Six new tests, each verified to fail with its own half reverted. Two examples:

• reverting the validation split fails with MetadataError("Protected resource metadata resource field is not a valid URL") — the catch-all case above
• reverting the fall-through leaves the challenge test resolving past the document to an authorization server it was never meant to reach

Two existing tests fed the document twice to satisfy the duplicated GET; both lost the duplicate.

Three things I noticed on the same thread and left out of this PR, since they are wider than what it started as. Happy to open issues, or fold any of them in if you would rather:

1. fetch_authorization_metadata reads a url from generate_discovery_urls and a url named in a protected resource document's authorization_servers the same way — both soft. If none of the named servers can be reached, resolution continues on to the legacy endpoints, so the document's choice of authorization server does not reach the caller. source does carry LegacyEndpointFallback, so the information is available to callers that read it. Every named server being rejected by the SSRF guard ends up on the same path. Whether those two are worth telling apart from an ordinary fallback looks more like a design call than a fix, so I left it alone.
2. When an advertised url is soft-failed on the challenge path, resolve_metadata re-runs the whole discovery, so a url that is both the challenge pointer and a well-known candidate is fetched twice across the two passes.
3. An issuer mismatch aborts the authorization server loop rather than trying the next candidate. That reads deliberate to me (fail closed on a poisoned server), and it matches how transport errors are handled — but no test covers a document naming more than one server, so nothing pins the intent.

One behaviour note on this PR as a whole: 09c5e51 means a server publishing the document directly at the resource url is no longer discovered that way. That is the point of your second comment, but it is a break for anyone relying on it and no test records it as intentional.

cargo clippy --all-targets --all-features -- -D warnings is clean and cargo test -p rmcp --all-features is 761 passed, with the same platform-worded connection-error assertion failing as before.