Add a host allowlist for creative asset rewriting

CHANGELOG.md

@@ -33,14 +33,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `rewrite.exclude_domains` matching is now case-insensitive and ignores surrounding whitespace. Entries written with uppercase letters previously never matched and now take effect, so those hosts stop being proxied and click-wrapped, and `/first-party/sign` now rejects them with `502`; it used to sign them, or return `403` when the host was also outside `proxy.allowed_domains`. Empty and bare `"*"` `exclude_domains` entries, which never matched a host, are dropped at load with a warning, and other entries that can never match a host are logged with a warning. Absolute `http(s)` creative URLs that cannot be parsed (for example, a host containing a space) are now left untouched; previously the trimmed value was written back re-quoted (with `https:` prepended to `//` input, so `src=" //exa mple.example/x "` became `https://exa mple.example/x`), an unparseable quoted URL in `<style>` or `style=` was re-quoted through `cssparser::serialize_string`, and a link also gained `data-tsclick`. `/first-party/sign` now answers malformed input and non-http(s) schemes with `unsupported url` instead of `invalid url` / `unsupported scheme`; both still map to `502`. Audit `exclude_domains` for mixed-case entries before upgrading.
 - `[auction].allowed_context_keys` now serializes in sorted, deduplicated order, so ESI template-cache fingerprints and `ts config diff`/`push` envelope hashes are stable across loads. Template fingerprints also sort object keys independently of `serde_json/preserve_order`. Existing envelopes may show a one-time allowlist reorder after upgrading; push once to settle it. The updated fingerprint format causes one template-cache miss per cached page after deployment.
 - TSJS-generated envelopes now send `trustedServer.params.storedRequest: false`, preventing accidental PBS stored lookups without suppressing eligible non-PBS demand. PBS filters unusable impressions after overrides; explicit `true` and omission in valid envelopes retain inline-first stored fallback. A malformed envelope disables stored fallback for the entire slot, including independent direct demand left unusable after overrides. Publisher intent survives repeated and refresh auctions. Deploy compatible server admission everywhere before serving the new JS, and retain it during rollback while cached clients remain. See the Prebid deployment guide.
 - Protocol-relative creative URLs now honor `rewrite.exclude_domains`, so excluded creative assets stay direct and excluded absolute or protocol-relative URLs submitted to `/first-party/sign` are rejected.
 - Server-side ad template bids now always carry `hb_adid` in `window.tsjs.bids`. Bidders that return neither a Prebid Cache UUID nor an `adid` previously produced no `hb_adid` at all, so no `hb_adid` GPT targeting key was set and the Universal Creative render bridge had nothing to match — the winning creative never rendered. The OpenRTB bid `id`, which is mandatory per spec, is now the last-resort source; `cache_id` and `adid` still take priority where present. Blank `cacheId`/`adid` values no longer win that precedence and emit an unusable empty `hb_adid`, and `hb_cache_host`/`hb_cache_path` are now emitted only alongside a real Prebid Cache UUID — without one they pointed the Universal Creative at a guaranteed cache miss instead of letting it fall through to the inline creative.
+- Inline SSAT/page-bids creatives no longer proxy `<link rel="preload" imagesrcset>` candidates twice. The `imagesrcset` attribute is now rewritten once, by the same handler on every creative path; previously the inline path's absolute proxy URLs were wrapped in a second `/first-party/proxy` URL.
 
 ### Added
 
-- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `<base>` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
+- `[rewrite] include_domains` limits creative asset rewriting to listed hosts. When non-empty, only asset URLs on matching hosts are proxied through `/first-party/proxy`, so only those hosts receive the EC ID on proxied fetches. Other assets keep their original URL, and `/first-party/sign` returns `502` for them so the creative runtime loads them directly. Click-through links are not affected, and `exclude_domains` still wins. Entries must be host names, optionally prefixed with `*.`, dotted-quad IPv4 addresses, or bracketed IPv6 addresses; entries with a scheme, port, path, spaces or non-ASCII characters are rejected at load. Upgrading: deploy the binary first, then push a config that sets `include_domains`. Rolling back: remove `include_domains`, push the resulting config, then roll back the binary; older binaries reject the key because `[rewrite]` denies unknown fields.
+- Added `[auction].rewrite_clicks` to control creative click-through wrapping (`<a href>`/`<area href>` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `<base>` handling and creative TSJS injection run when either setting is on. A bidder `<base target>` is now kept with its `href` dropped, instead of removing the whole element, so raw links keep opening where the bidder intended. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary. Older binaries tie click wrapping to `rewrite_creatives` in both directions, so rolling back turns clicks back on for `rewrite_creatives = true` with `rewrite_clicks = false`, and off for `rewrite_creatives = false` with `rewrite_clicks = true`.
+- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (asset URL conversion to `/first-party/proxy`, bidder `<base>` removal; creative TSJS injection on `POST /auction` only). Click-through wrapping is controlled by `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset. Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.
 - Added typed APS renderer transport for direct auctions and GAM/Prebid Universal Creative, using a minimized one-bid envelope, a fragment-bound nonce, and an opaque sandboxed renderer endpoint.

crates/trusted-server-cli/tests/config_env_overlay.rs

@@ -29,6 +29,7 @@ ids = ["trusted_server_secrets"]
 "#;
 const REWRITE_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CREATIVES";
 const SANITIZE_ENV: &str = "TRUSTED_SERVER__AUCTION__SANITIZE_CREATIVES";
+const CLICKS_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CLICKS";
 const GAM_ATTRIBUTION_ENV: &str = "TRUSTED_SERVER__INTEGRATIONS__GPT__GAM_ATTRIBUTION_ENABLED";
 const AD_TEMPLATES_ENABLED_ENV: &str = "TRUSTED_SERVER__CREATIVE_OPPORTUNITIES__ENABLED";
 const PROVIDER_ENDPOINT_ENV: &str = "TRUSTED_SERVER__AUCTION__PROVIDERS__PBS-MAIN__ENDPOINT";
@@ -82,6 +83,76 @@ fn validate_with_overlay(project: &MigratedProject, raw_value: &str) -> Output {
         .expect("should run ts config validate")
 }
 
+fn pushed_auction_with_env(
+    project: &MigratedProject,
+    key: &str,
+    raw_value: &str,
+) -> serde_json::Value {
+    let output = Command::new(env!("CARGO_BIN_EXE_ts"))
+        .args(["config", "push", "--adapter", "axum", "--manifest"])
+        .arg(&project.manifest_path)
+        .arg("--app-config")
+        .arg(&project.config_path)
+        .args(["--yes", "--no-diff"])
+        .current_dir(project.directory.path())
+        .env(key, raw_value)
+        .output()
+        .expect("should run ts config push");
+    assert!(
+        output.status.success(),
+        "config push should succeed: {}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+
+    let local_store_path = project
+        .directory
+        .path()
+        .join(".edgezero/local-config-trusted_server_config.json");
+    let local_store: serde_json::Value = serde_json::from_str(
+        &fs::read_to_string(local_store_path).expect("should read pushed local config"),
+    )
+    .expect("should parse local config store");
+    let envelope_json = local_store
+        .as_object()
+        .and_then(|entries| entries.values().next())
+        .and_then(serde_json::Value::as_str)
+        .expect("should contain a blob envelope");
+    let envelope: serde_json::Value =
+        serde_json::from_str(envelope_json).expect("should parse blob envelope");
+    envelope["data"]["auction"].clone()
+}
+
+#[test]
+fn rewrite_clicks_environment_override_applies_when_leaf_present() {
+    let project = migrated_project();
+    let mut document = fs::read_to_string(&project.config_path)
+        .expect("should read migrated config")
+        .parse::<DocumentMut>()
+        .expect("should parse migrated config");
+    document["auction"]["rewrite_clicks"] = value(true);
+    fs::write(&project.config_path, document.to_string()).expect("should write config");
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert_eq!(
+        auction["rewrite_clicks"],
+        serde_json::Value::Bool(false),
+        "pushed config should contain the rewrite_clicks environment override"
+    );
+}
+
+#[test]
+fn rewrite_clicks_environment_override_is_ignored_without_leaf() {
+    let project = migrated_project();
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert!(
+        auction.get("rewrite_clicks").is_none(),
+        "an override for a missing leaf should be ignored and the unset default omitted"
+    );
+}
+
 #[test]
 fn config_validate_explains_legacy_provider_list_migration() {
     let project = migrated_project();

crates/trusted-server-core/src/auction/README.md

@@ -108,8 +108,10 @@ as `pbs-main`.
 response.
 
 - `sanitize_creatives = true` strips executable markup. It is opt-in.
-- `rewrite_creatives = true` rewrites eligible URLs through first-party routes
-  and removes bidder `<base>` elements. It is enabled by default.
+- `rewrite_creatives = true` rewrites eligible asset URLs through first-party
+  routes and removes bidder `<base>` elements. It is enabled by default.
+- `rewrite_clicks` wraps creative links in `/first-party/click`. Unset, it
+  follows `rewrite_creatives`.
 - The publisher inline delivery path uses absolute first-party URLs without
   injecting the direct endpoint's creative runtime.
 - Creatives over the configured hard cap are rejected.

crates/trusted-server-core/src/auction/endpoints.rs

@@ -84,10 +84,14 @@ const MAX_AUCTION_BODY_SIZE: usize = 256 * 1024;
 /// ## Response
 ///
 /// Returns an `OpenRTB 2.x` response. Creative HTML is inlined in each bid's
-/// `adm` field after mandatory server-side sanitization. First-party resource
-/// and click URL rewriting plus creative TSJS injection are enabled by default;
-/// setting [`auction.rewrite_creatives`][`crate::auction_config_types::AuctionConfig::rewrite_creatives`]
-/// to `false` skips only that rewrite pass.
+/// `adm` field after optional sanitization
+/// ([`auction.sanitize_creatives`][`crate::auction_config_types::AuctionConfig::sanitize_creatives`]).
+/// First-party asset rewriting
+/// ([`auction.rewrite_creatives`][`crate::auction_config_types::AuctionConfig::rewrite_creatives`])
+/// and click wrapping
+/// ([`auction.rewrite_clicks`][`crate::auction_config_types::AuctionConfig::rewrite_clicks`])
+/// are enabled by default. Bidder `<base>` removal and creative TSJS injection
+/// run when either is on.
 ///
 /// ## Scroll, refresh, and SPA navigation
 ///

crates/trusted-server-core/src/auction/formats.rs

@@ -316,14 +316,16 @@ pub(crate) struct OpenRtbResponseConversion {
 ///
 /// Creative HTML in the `adm` field is optionally sanitized and optionally
 /// rewritten according to the auction configuration
-/// ([`AuctionConfig::sanitize_creatives`], opt-in, and
-/// [`AuctionConfig::rewrite_creatives`], default-on); with both disabled the
-/// creative ships exactly as the bidder returned it, subject to the 1 MiB
-/// per-creative cap. Typed renderers are serialized in the response extension
+/// ([`AuctionConfig::sanitize_creatives`], opt-in;
+/// [`AuctionConfig::rewrite_creatives`] for assets, default-on; and
+/// [`AuctionConfig::rewrite_clicks`] for links, following `rewrite_creatives`
+/// when unset); with all disabled the creative ships exactly as the bidder
+/// returned it, subject to the 1 MiB per-creative cap. Typed renderers are serialized in the response extension
 /// instead of entering that pipeline at all.
 ///
 /// [`AuctionConfig::sanitize_creatives`]: crate::auction_config_types::AuctionConfig::sanitize_creatives
 /// [`AuctionConfig::rewrite_creatives`]: crate::auction_config_types::AuctionConfig::rewrite_creatives
+/// [`AuctionConfig::rewrite_clicks`]: crate::auction_config_types::AuctionConfig::rewrite_clicks
 ///
 /// # Errors
 ///
@@ -373,9 +375,10 @@ pub(crate) fn convert_to_openrtb_response_with_report(
         let height = to_openrtb_i32(bid.height, "height", &bid_context);
 
         // Ordinary markup goes through the configured creative processing:
-        // sanitization is opt-in, rewriting is on by default, and with both
-        // disabled the creative ships exactly as the bidder returned it. A typed
-        // renderer is serialized separately and never enters that pipeline.
+        // sanitization is opt-in, asset rewriting and click wrapping are on by
+        // default, and with all three disabled the creative ships exactly as
+        // the bidder returned it. A typed renderer is serialized separately and
+        // never enters that pipeline.
         let serialize_renderer = |renderer: &BidRenderer| {
             (BidExt {
                 trusted_server: BidTrustedServerExt { renderer },
@@ -398,12 +401,13 @@ pub(crate) fn convert_to_openrtb_response_with_report(
             let processed = creative::process_auction_creative(settings, raw_creative);
 
             log::debug!(
-                "Processed creative for auction {} slot {} bidder {} (sanitize {}, rewrite {}, raw {} bytes, output {} bytes)",
+                "Processed creative for auction {} slot {} bidder {} (sanitize {}, rewrite {}, clicks {}, raw {} bytes, output {} bytes)",
                 auction_request.id,
                 slot_id,
                 bid.bidder,
                 settings.auction.sanitize_creatives,
                 rewrite_creatives,
+                settings.auction.rewrites_auction_clicks(),
                 raw_creative.len(),
                 processed.len()
             );
@@ -1413,6 +1417,37 @@ mod tests {
         );
     }
 
+    #[test]
+    fn convert_to_openrtb_response_wraps_clicks_without_rewriting_assets() {
+        let mut settings = make_settings();
+        settings.auction.sanitize_creatives = false;
+        settings.auction.rewrite_creatives = false;
+        settings.auction.rewrite_clicks = Some(true);
+        let auction_request = make_auction_request();
+        let result = make_result(make_complete_creative_bid());
+
+        let response = convert_to_openrtb_response(&result, &settings, &auction_request, false)
+            .expect("should convert creative with click rewriting only");
+        let adm = response_adm(response);
+
+        assert!(
+            adm.contains("/first-party/click?tsurl=") && adm.contains("data-tsclick"),
+            "should wrap the landing link: {adm}"
+        );
+        assert!(
+            !adm.contains("/first-party/proxy?tsurl="),
+            "should not proxy any asset: {adm}"
+        );
+        assert!(
+            adm.contains(r#"src="https://cdn.example.com/ad.png""#),
+            "should keep the image URL direct: {adm}"
+        );
+        assert!(
+            adm.contains("tsjs-unified.min.js"),
+            "should inject the creative runtime for the click guard: {adm}"
+        );
+    }
+
     #[test]
     fn sanitize_creatives_defaults_to_disabled() {
         let config = crate::auction_config_types::AuctionConfig::default();

crates/trusted-server-core/src/auction/orchestrator.rs

@@ -4696,6 +4696,7 @@ mod tests {
                 enabled: true,
                 sanitize_creatives: true,
                 rewrite_creatives: true,
+                rewrite_clicks: None,
                 providers: AuctionConfig::legacy_provider_map(&[]),
                 bidders: Default::default(),
                 mediator: None,