Add a host allowlist for creative asset rewriting

[dhruv8sh]

Summary

• Adds [rewrite] include_domains, a host allowlist for creative asset rewriting.
  • Why: today every absolute asset URL in a winning creative is proxied through /first-party/proxy, and each proxied fetch appends the EC ID, so every third-party asset host receives it.
  • Non-empty list: only matching asset hosts are proxied, and every other asset keeps its original URL. exclude_domains still wins over the list, and click-through links are never affected.
• /first-party/sign applies the same policy for absolute and // input.
  • Off-list host: gets 502, the same as an excluded host, so the creative runtime loads the raw URL directly.
  • 403: still means only "blocked by proxy.allowed_domains".
• Validation is strict. Entries that can never match a URL host are rejected at ts config push. That includes schemes, ports, paths, stray dots, unbracketed IPv6, and non-ASCII names, which must be written in punycode.
• Fixes an existing bug: on the inline SSAT/page-bids path, <link imagesrcset> was proxied twice, wrapping the proxy URL inside another proxy URL.

Implements the approved design in #1248: docs/superpowers/specs/2026-10-07-1231-creative-asset-host-allowlist-design.md.

Stacked on #1252. This branch builds on the #1234 implementation, which introduces the shared normalizer and host-policy split.

• Commits under review: only the last four, 48d59b8d, ef8ad67f, 9d9fb273 and 2dfe7e7b.
• The six commits below them (d8f04390…ee5715f1) are cherry-picks of Make creative click rewriting a separate switch #1252 with identical patch IDs, and are reviewed there.
• Base: spec/1231-creative-asset-host-allowlist (**Spec**: creative asset host allowlist #1248).
• Merge order: **Spec**: creative asset host allowlist #1248 and **Spec**: Separate creative click rewrite switch #1249, then Make creative click rewriting a separate switch #1252. After that, this branch is rebased onto main and only the four Add a host allowlist for creative asset rewriting #1231 commits remain.

Changes

These are the #1231 commits only.

File	Change
crates/trusted-server-core/src/settings.rs	include_domains field (left out of the blob when empty), load-time normalization, strict validate_include_domains, and the include clause in should_proxy_asset via is_host_permitted; tests
crates/trusted-server-core/src/creative.rs	Allowlist matrix across every asset handler on all three HTML paths, plus CSS bodies. link[href] no longer rewrites imagesrcset; the [imagesrcset] handler rewrites it once. Tests and module docs
crates/trusted-server-core/src/proxy.rs	is_host_permitted made pub(crate); sign decline log; an 8-case GET/POST sign matrix covering 200, 502 and 403
crates/trusted-server-core/src/config_payload.rs	Default-omission and round-trip tests
docs/guide/configuration.md, creative-processing.md, first-party-proxy.md, api-reference.md	Allowlist semantics, validation rules, the rollback warning, and the sign 502. Fixes the *.cdn.example.com apex example
trusted-server.example.toml	Commented-out include_domains example
CHANGELOG.md	Added include_domains; Fixed the inline imagesrcset double proxy

Closes

Closes #1231

Test plan

• cargo test-fastly && cargo test-axum. Also cargo test-fastly-reuse, cargo test-cloudflare, cargo test-spin, ./scripts/test-cli.sh and the parity tests.
• cargo clippy-fastly && cargo clippy-axum. All 8 clippy aliases pass, and every intermediate commit passes clippy-fastly.
• cargo fmt --all -- --check
• JS tests: cd crates/trusted-server-js/lib && npx vitest run (no JS changes)
• JS format: cd crates/trusted-server-js/lib && npm run format (no JS changes)
• Docs format: cd docs && npm run format, plus the markdown prettier check outside docs/
• WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
• Manual testing via fastly compute serve
• Other: the imagesrcset regression test was seen failing on the unfixed code. Make creative click rewriting a separate switch #1252's switch-matrix tests pass together with this branch's allowlist tests.

Rollout: deploy the binary first, then add include_domains and run ts config push. Rollback: remove include_domains and push first, then roll back the binary; older binaries reject the key because of deny_unknown_fields.

Hardening note:

• Invalid entries: an invalid include_domains entry fails settings validation with a config error rather than a panic. Covered by rewrite_include_domains_reject_malformed_entries.
• Bad exclude_domains entries: empty and bare "*" entries are dropped with a warning instead of rejected, so existing configs keep loading after an upgrade.

Checklist

• Changes follow AGENTS.md conventions
• No unwrap() in production code — use expect("should ...")
• Uses tracing macros (not println!)
• New code has tests
• No secrets or credentials committed

[dhruv8sh]

@prk-Jr Your review comments on the spec PRs (#1248 and #1249) are addressed in the spec branches. Those changes are merged here, along with the matching code changes: the include_domains host round-trip check, the warning for exclude_domains entries that can never match, the <link> let-chain, keeping <base target>, the doc warning on the public rewrite functions, and the operator doc and CHANGELOG corrections. One thing differs from what we discussed: in clicks-only mode <base> still loses its href, because /auction click URLs are root-relative and would otherwise resolve against the bidder's base. The spec explains the tradeoff.

[aram356]

Summary

This PR adds [rewrite] include_domains and applies it to every asset handler on all three HTML paths, to CSS bodies, and to /first-party/sign. It also fixes the inline <link imagesrcset> double proxy. The core change holds up:

• A mutation pass over the new code (12 mutants, run on the PR merged with main) caught 10.
• The merged tree passes clippy-fastly and test-fastly.
• Every proxy URL builder goes through asset_target, so nothing bypasses the allowlist.

Three things block merging:

• the conflict with main;
• the #1252 commits this branch carries;
• an exclude_domains warning that tells operators to delete entries that work.

4 of the inline comments below carry a one-click GitHub suggestion. Use Commit suggestion, or Add suggestion to batch to apply several at once, to add them as commits on the PR branch. The other comments describe the fix in prose, because it spans several hunks or files.

Blocking

🔧 wrench

• The exclude_domains warning flags entries that do match, including the trailing-dot workaround. See inline at crates/trusted-server-core/src/settings.rs:708.

Non-blocking

🤔 thinking

• Sign declines log at error and count as 5xx on every adapter. See inline at crates/trusted-server-core/src/proxy.rs:1692.
• The include grammar rejects real hosts (_), and the 0x clause is redundant. See inline at crates/trusted-server-core/src/settings.rs:3494.

♻️ refactor

• The validation error doesn't name the bad entry. See inline at crates/trusted-server-core/src/settings.rs:3450.

⛏ nitpick

• Unset rewrite_clicks follows rewrite_creatives on the auction paths. See inline at docs/guide/creative-processing.md:762.
• The spec status is stale. See inline at docs/superpowers/specs/2026-10-07-1231-creative-asset-host-allowlist-design.md:7.

Cross-cutting / body-level findings

• 🔧 Conflicts with main. Merging origin/main (d2f0d8c) conflicts in two files:

  • CHANGELOG.md: both sides add Fixed entries. Keep both.
  • settings.rs normalize_deserialized: #1221 made image_optimizer.normalize() return Result. The resolution is:

          self.rewrite.normalize();
          self.image_optimizer.normalize()?;

  With that resolution, the merged tree passes cargo fmt --check, cargo clippy-fastly and cargo test-fastly (2964 core tests).

• 🔧 This branch carries #1252, including #1252's open blocking threads. The base is now main, so this diff includes the rewrite_clicks work in the same state as #1252's 7c5eca0da. Review 5507201594 on #1252 still has six open 🔧 threads:

  • auction-orchestration.md;
  • two CHANGELOG items;
  • the example TOML;
  • the creative-processing.md guard sentence;
  • the #1234 spec status.

  Merging this PR first would land all of them. As the description plans, land #1252 first, then rebase so only the #1231 commits remain. Two related problems:

  • The description is out of date. It names spec/1231-… as the base and lists four commits under review. The base is main, and the #1231 work now also includes f7885c199, e1332c9fe, 2af735d1d and the spec feedback in cce4e2a5e.
  • The shared Fixed entry in CHANGELOG.md no longer matches #1252's copy. This one adds "and other entries that can never match a host are logged with a warning". The plan says the shared text is identical in both PRs, so whichever lands second will conflict on that line.

• 📌 The trailing-dot bypass needs its follow-up issue. The spec's Follow-ups section says to track it in a follow-up issue, and I found none. https://blocked.example.com./x has host_str() blocked.example.com., so it escapes exclude_domains and still gets ts-ec. The spec suggests stripping one trailing . before matching. Until then, an exclude entry written with the trailing dot is the stopgap, which is why the wrench above matters.

CI Status

• CLAUDE.md symlink guard: PASS
• browser integration tests: PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• cargo fmt: PASS (required)
• cargo test: PASS (required)
• cargo test (axum native): PASS
• cargo test (cross-adapter parity): PASS
• cargo test (ts CLI, native) (macos-latest): PASS
• cargo test (ts CLI, native) (ubuntu-latest): PASS
• format-docs: PASS (required)
• format-typescript: PASS (required)
• integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• prepare integration artifacts: PASS
• vitest: PASS

[aram356]

🔧 wrench: This warning says an entry "never matches a URL host". But check_include_domain implements the stricter include_domains grammar, so it also flags entries that do match. With the locked url 2.5.8:

exclude_domains entry	URL it matches
blocked.example.com.	https://blocked.example.com./x (host_str() keeps the trailing dot)
ads_cdn.example.com	https://ads_cdn.example.com/x
*.192.0.2.1	https://192.0.2.1/x (apex rule in is_host_allowed)

The first row matters most. The spec's Follow-ups section says trailing-dot hosts bypass exclude_domains, and an entry written with the trailing dot is the only way to close that bypass today. This warning, plus configuration.md ("fix them, or their hosts stay proxied"), tells the operator to delete that entry, and deleting it turns ts-ec forwarding back on for the host.

I reproduced this with the PR-head ts binary. ts config validate --no-env on exclude_domains = ["ads_cdn.example.com", "https://cdn.example.com"] exits 0 and prints:

rewrite.exclude_domains: entry `ads_cdn.example.com` never matches a URL host (character outside a-z, 0-9, `-` and `.`)
rewrite.exclude_domains: entry `https://cdn.example.com` never matches a URL host (character outside a-z, 0-9, `-` and `.`)

The suggestion gives exclude_domains its own check. The URL parser decides (check_host_round_trip on the entry minus *.). Two shapes that no fetchable host has are still flagged: a * other than a leading *., and a leading . or an empty label. All three entries in rewrite_exclude_domains_keep_entries_that_never_match still warn.

The warning also has no test. Deleting the whole loop leaves every include/rewrite/imagesrcset/proxy_sign test passing (I ran a mutation pass on the PR merged with main). Once the check is a function, it can be tested. This test sits outside the hunk, so add it by hand next to rewrite_exclude_domains_keep_entries_that_never_match. It passes with the suggestion applied:

    #[test]
    fn rewrite_exclude_domains_flag_only_entries_that_never_match() {
        for entry in [
            "https://cdn.example.com",
            "cdn.example.com:443",
            "[2001:0db8::1]",
            "cdn.*.example.com",
            ".example.com",
            "bücher.example",
        ] {
            assert!(
                Rewrite::check_exclude_domain(entry).is_err(),
                "should flag `{entry}`, which never matches a URL host"
            );
        }
        for entry in [
            "blocked.example.com.",
            "ads_cdn.example.com",
            "*.192.0.2.1",
            "*.example.com",
            "[::1]",
        ] {
            assert!(
                Rewrite::check_exclude_domain(entry).is_ok(),
                "should not flag `{entry}`, which matches a URL host"
            );
        }
    }

Suggested change

	// An `exclude_domains` entry that can never equal a URL host keeps its
	// host proxied and receiving `ts-ec`. Rejecting it would stop existing
	// blobs from loading, so warn once per entry and keep it.
	for pattern in &self.exclude_domains {
	if let Err(reason) = check_include_domain(pattern) {
	log::warn!(
	"rewrite.exclude_domains: entry `{pattern}` never matches a URL host ({reason})"
	);
	}
	}
	false
	}
	// An `exclude_domains` entry that can never equal a URL host keeps its
	// host proxied and receiving `ts-ec`. Rejecting it would stop existing
	// blobs from loading, so warn once per entry and keep it.
	for pattern in &self.exclude_domains {
	if let Err(reason) = Self::check_exclude_domain(pattern) {
	log::warn!(
	"rewrite.exclude_domains: entry `{pattern}` never matches a URL host ({reason})"
	);
	}
	}
	}
	/// Checks one normalized `exclude_domains` entry, returning why no fetched
	/// URL can match it.
	///
	/// Looser than the `include_domains` grammar on purpose: a trailing `.`, a
	/// `_` and a wildcard IP address all still match a URL host, and a
	/// trailing-dot entry is the only way to exclude `blocked.example.com.`.
	fn check_exclude_domain(pattern: &str) -> Result<(), &'static str> {
	let host = pattern.strip_prefix("*.").unwrap_or(pattern);
	let labels = host.strip_suffix('.').unwrap_or(host);
	if host.contains('*') {
	Err("`*` other than a leading `*.`")
	} else if labels.split('.').any(str::is_empty) {
	Err("leading `.` or empty label")
	} else {
	check_host_round_trip(host)
	}
	}

[aram356]

♻️ refactor: When a list contains a bad entry, the error doesn't say which entry it is. With the PR-head ts binary, include_domains = ["assets.example.com", "img.example.com", "cdn.example.com:443", "*.static.example.net"] fails ts config validate --no-env with only:

[ts] typed app-config failed validation: rewrite.include_domains: rewrite.include_domains entries must be a host name, an optional leading `*.`, a dotted-quad IPv4 address or a bracketed IPv6 address

The CLI prints message and drops the value and reason params. The edge load path shows even less: validation_error_summary prints only the code (rewrite.include_domains: invalid_rewrite_include_domain). Putting the entry and the reason into the message fixes what operators see at ts config validate and push. The params stay for structured consumers, and the codes the tests match on don't change.

With the suggestion applied, the same config prints:

[ts] typed app-config failed validation: rewrite.include_domains: rewrite.include_domains entry `cdn.example.com:443` must be a host name with an optional leading `*.`, a dotted-quad IPv4 address or a bracketed IPv6 address (character outside a-z, 0-9, `-` and `.`)

Suggested change

	if !pattern.is_ascii() {
	let mut err = ValidationError::new("non_ascii_rewrite_include_domain");
	err.add_param("value".into(), pattern);
	err.message = Some(
	"rewrite.include_domains entries must be ASCII; write internationalized domain names in punycode (xn--) form"
	.into(),
	);
	return Err(err);
	}
	if let Err(reason) = check_include_domain(pattern) {
	let mut err = ValidationError::new("invalid_rewrite_include_domain");
	err.add_param("value".into(), pattern);
	err.add_param("reason".into(), &reason);
	err.message = Some(
	"rewrite.include_domains entries must be a host name, an optional leading `*.`, a dotted-quad IPv4 address or a bracketed IPv6 address"
	.into(),
	);
	return Err(err);
	}
	if !pattern.is_ascii() {
	let mut err = ValidationError::new("non_ascii_rewrite_include_domain");
	err.add_param("value".into(), pattern);
	err.message = Some(
	format!(
	"rewrite.include_domains entry `{pattern}` is not ASCII; write internationalized domain names in punycode (xn--) form"
	)
	.into(),
	);
	return Err(err);
	}
	if let Err(reason) = check_include_domain(pattern) {
	let mut err = ValidationError::new("invalid_rewrite_include_domain");
	err.add_param("value".into(), pattern);
	err.add_param("reason".into(), &reason);
	err.message = Some(
	format!(
	"rewrite.include_domains entry `{pattern}` must be a host name with an optional leading `*.`, a dotted-quad IPv4 address or a bracketed IPv6 address ({reason})"
	)
	.into(),
	);
	return Err(err);
	}

[aram356]

🤔 thinking: The doc comment says this rejects entries that "can never equal a URL host", but the character class also rejects hosts that URLs do carry:

• _: Url::parse("https://cdn_assets.example.com/") gives host_str() == "cdn_assets.example.com", and url::Host::parse round-trips it. The PR-head ts config validate rejects include_domains = ["cdn_assets.example.com"] with the generic message. An operator whose asset host has an underscore can only allowlist it through a wider wildcard such as *.example.com.
• 0x (line 3504): check_host_round_trip already rejects every 0x label that url reads as a number. foo.0xabc fails to parse, and 0xc0000201 renders as 192.0.2.1. The clause only adds false rejections such as foo.0xcdn, which host_str() returns unchanged. Deleting it leaves the suite passing (mutation run).

If strict letter-digit-hyphen names are the intent, say so in the doc comment and configuration.md, and leave the code. Otherwise:

    if !name.bytes().all(|byte| {
        byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'-' | b'.' | b'_')
    }) {
        return Err("character outside a-z, 0-9, `-`, `_` and `.`");
    }

    let numeric_last_label = last_label.bytes().all(|byte| byte.is_ascii_digit());

Then move cdn_assets.example.com from rewrite_include_domains_reject_malformed_entries to rewrite_include_domains_accept_hosts_and_ip_literals, and update the grammar bullet in configuration.md and the spec. This has to be applied by hand: the test lists are in another hunk, and the code change alone fails the reject test.

[aram356]

🤔 thinking: The spec (decision 3) logs declines at debug because "declines are expected in normal operation once an allowlist is set". The Err returned here undoes that. Every adapter's http_error logs Error occurred: … at error level before it builds the 502: fastly app.rs:999, axum app.rs:120, cloudflare app.rs:329 and spin app.rs:472.

Once include_domains is set, most ad-tech hosts are off-list. Each image or iframe that a creative inserts at runtime then costs a sign round-trip that comes back 502. That adds one error log line and one 5xx in edge metrics, and the client logs tsjs-creative: sign HTTP error 502 (proxy_sign.ts:62). That noise can bury real proxy failures and trip 5xx alerts. exclude_domains already behaved this way, but exclusions are rare; with an allowlist, a decline becomes the common case.

Options, smallest first:

1. Return the decline as Ok with a 502 response built here. That skips http_error's log line and keeps the status and the JS contract. proxy_sign_rejects_excluded_urls and proxy_sign_rejects_excluded_urls_case_insensitively call expect_err, so they would read the status from the response instead.
2. Use a 4xx status, such as 422, for policy declines. proxy_sign.ts already falls back to a direct load on any status other than 403, so 5xx stays reserved for real upstream failures. This changes the documented 502 in api-reference.md and first-party-proxy.md.

Separately, the # Errors section on this function (lines 1621-1624, outside the diff) still lists only parse, scheme, host and proxy.allowed_domains failures. It doesn't mention the exclude_domains / include_domains decline.

[aram356]

⛏ nitpick: "depends only on rewrite_clicks" reads as if an unset rewrite_clicks were its own state. On /auction and SSAT/page-bids, an unset rewrite_clicks follows rewrite_creatives (rewrites_auction_clicks). So with rewrite_creatives = false and rewrite_clicks unset, links are not wrapped.

Suggested change

	- Click-through links (`<a href>`, `<area href>`) are not affected. Whether
	they are wrapped in `/first-party/click` depends only on `rewrite_clicks` and
	`exclude_domains`.
	- Click-through links (`<a href>`, `<area href>`) are not affected. Whether
	they are wrapped in `/first-party/click` depends only on `rewrite_clicks`
	(which follows `rewrite_creatives` on the auction paths when unset) and
	`exclude_domains`.

[aram356]

⛏ nitpick: #1248 was closed in favor of this PR, so the spec ships here and "Draft" will be stale once this merges. This is the same item as the #1234 spec thread on #1252. #1248 closed with prk-Jr's CHANGES_REQUESTED still standing, so the approval of this PR is also the spec's approval.

One related line: the Rollout section (line 377) says that with no include_domains, "asset behavior is unchanged except for the exclude_domains case fix". Deploying the binary alone also changes three things:

• inline <link imagesrcset> output (the double-proxy fix);
• unparseable absolute URLs are now left untouched;
• the new exclude_domains warnings appear.

Suggested change

	**Status:** Draft (design validated by an exploratory implementation; see the plan, `docs/superpowers/plans/2026-10-07-1231-creative-asset-host-allowlist.md`)
	**Status:** Implemented in #1253 (see the plan, `docs/superpowers/plans/2026-10-07-1231-creative-asset-host-allowlist.md`)

[ChristianPavilonis]

Please address the host-classification issue: valid DNS names such as 0xproxy are rejected as IPv4 candidates, preventing otherwise valid allowlist configurations.

Also clarify the changelog and operator guidance: include_domains controls newly rewritten or signed asset targets, not fetch-time enforcement. Previously issued proxy URLs can still forward EC IDs to hosts subsequently removed from the list. This is a documentation correction, not a request to expand runtime policy.

[ChristianPavilonis]

[P3] Do not classify every 0x-prefixed hostname as IPv4

Valid DNS hosts such as 0xproxy and a.0xproxy are rejected because their final label starts with 0x, even though the remainder is not hexadecimal and the URL parser treats them as domain names. Consequently, include_domains = ["0xproxy"] or ["*.0xproxy"] fails configuration validation instead of allowing those hosts. Use the URL parser's actual host classification; the existing round-trip check can still reject malformed or noncanonical hosts.

Suggested change

	let last_label = name.rsplit('.').next().unwrap_or(name);
	let numeric_last_label =
	last_label.bytes().all(|byte| byte.is_ascii_digit()) || last_label.starts_with("0x");
	let numeric_last_label = matches!(url::Host::parse(name), Ok(url::Host::Ipv4(_)));

[ChristianPavilonis]

[P3] Qualify the claim that only included hosts receive EC IDs

Non-blocking documentation/contract correction; correctness and architecture lanes. The claim that “only those hosts receive the EC ID on proxied fetches” is broader than the implementation. For example, obtain a creative-generated proxy URL for cdn.example.net, then configure an include list that excludes it without changing the publisher secret. Static creative URLs have no tsexp (creative.rs:600–643), signature validation does not consult rewrite policy (proxy.rs:2071–2152), and /first-party/proxy still forwards the EC ID subject only to proxy.allowed_domains (proxy.rs:1506–1528). Previously delivered creatives can therefore continue forwarding IDs to now-off-list hosts. Qualify the changelog and operator guidance: the list controls newly rewritten/signed asset targets, not previously issued URLs or fetch-time enforcement. This preserves the spec’s explicit non-goals; it does not propose expanding runtime policy.