From cf164189665d2af6cbd2b2e35e012806026e729e Mon Sep 17 00:00:00 2001 From: Jason E Date: Mon, 24 Aug 2026 19:02:18 -0500 Subject: [PATCH 001/230] Add request phase timing design spec and implementation plan (#1069) * Add request phase timing design spec (Server-Timing subtimings + access telemetry) * Address review round 1: freeze point, template-cache naming, snapshot semantics, KV scope, geo carry, route template, sink confirmation, sampling and query model, config rollback * Address review round 2: auction-wait placement modes, conservative private-only header emission, non-null sorting key with service identity, coarse publisher route template, telemetry snapshot and outage behavior, tinybird flag decoupling, adapter phase semantics * Add request phase timing implementation plan * Address engineer review: KV timing decorator, try_lock sampling, route metadata extension, adapter-derived env, typed template-cache state, adapter-owned emission context, per-mode delivery semantics, Axum outer wrapper --- .../plans/2026-08-24-request-phase-timing.md | 1038 +++++++++++++++++ .../2026-08-24-request-phase-timing-design.md | 553 +++++++++ 2 files changed, 1591 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-24-request-phase-timing.md create mode 100644 docs/superpowers/specs/2026-08-24-request-phase-timing-design.md diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md new file mode 100644 index 000000000..9ff3905f4 --- /dev/null +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -0,0 +1,1038 @@ +# Request Phase Timing Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every application response attributes its own server time by phase via a +Server-Timing header and a sampled Tinybird access-telemetry row. + +**Architecture:** A core `RequestTimings` handle (Arc-shared, infallible recording) +collects phase spans always-on; the Fastly adapter freezes and emits at +`send_edgezero_response` immediately before `into_parts()`; a post-send emitter ships +one NDJSON row to the Tinybird Events API with a bounded, 2xx-validated await. + +**Tech Stack:** Rust 2024, `edgezero` HTTP types, Fastly Compute (wasm32-wasip1, +Viceroy tests), Axum (native tests), Tinybird Events API. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`: the +plan argues from the spec; executors read both. Spec section numbers are cited per +task. + +## Global Constraints + +- Errors use `error-stack` (`Report`); errors defined with + `derive_more::Display`; never thiserror, never anyhow (except the Spin entry point). +- No `unwrap()` in production code; `expect("should ...")` only. Assertion messages + `"should ..."`. Tests use Arrange-Act-Assert. +- No inline comments; comments on their own line above the code. +- Functions never exceed 7 arguments; use a struct instead (this bit + `ec_finalize_response` in review; the timings handle travels inside existing state). +- No local imports inside functions; `use super::*` only in `#[cfg(test)]`. +- Only example/fictional data in tests and docs (`example.com` domains). +- Recording is infallible: saturating math, lock failure drops the sample, no panics + (spec 5, 13). +- Vendor identity never appears in emitted surfaces: the filter span is `ts-filter` + (spec 3). +- Test commands: `cargo test-axum` (native, fast inner loop), `cargo test-fastly` + (Viceroy) for adapter tasks. Before PR handoff: the full CI gate list in + `CLAUDE.md`. +- Commit style: sentence case, imperative, no prefixes, no trailers. + +## File Structure + +| File | Responsibility | +| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `crates/trusted-server-core/src/request_timing.rs` (new) | `Phase`, `AuctionWaitPlacement`, `RequestTimings`, `PhaseSpan`, `TimingSnapshot`, header rendering | +| `crates/trusted-server-core/src/access_telemetry.rs` (new) | `RouteClass`, `publisher_route_template`, `AccessTelemetrySnapshot`, `AccessEventRow` NDJSON | +| `crates/trusted-server-core/src/geo.rs` (modify) | `GeoLookupState` response-extension type | +| `crates/trusted-server-core/src/settings.rs` (modify) | `ObservabilitySettings`, tinybird flag decoupling, access validation | +| `crates/trusted-server-core/src/publisher.rs` (modify) | `ts-origin`, `ts-template-cache`, auction-wait spans | +| `crates/trusted-server-core/src/ec/kv.rs` (modify) | `ts-kv` at the graph abstraction | +| `crates/trusted-server-adapter-fastly/src/main.rs` (modify) | T0, appbuild span, freeze point, `DeliveryOutcome`, post-send emission ordering | +| `crates/trusted-server-adapter-fastly/src/app.rs` (modify) | filter span, geo span + `GeoLookupState` attach, route class assignment | +| `crates/trusted-server-adapter-fastly/src/middleware.rs` (modify) | finalize consumes `GeoLookupState` | +| `crates/trusted-server-adapter-fastly/src/tinybird.rs` (modify) | access sink with confirmed delivery | +| `crates/trusted-server-adapter-axum/src/` (modify) | terminal freeze layer, header emission | +| `tinybird/datasources/access_logs_raw.datasource` (modify) | phase-column schema, non-null sorting key | +| `trusted-server.example.toml` (modify) | `[observability]`, tinybird keys | + +Out of scope for this plan: the Grafana dashboard JSON (separate telemetry repo, +spec 11) and Cloudflare/Spin emission wiring (spec non-goal). + +--- + +### Task 1: Core `RequestTimings` + +**Files:** + +- Create: `crates/trusted-server-core/src/request_timing.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` (add `pub mod request_timing;`) +- Test: same file, `#[cfg(test)]` + +**Interfaces:** + +- Consumes: nothing (leaf module; `std::time`, `std::sync`). +- Produces (later tasks rely on these exact names): + - `pub enum Phase { AppBuild, Filter, Geo, EcKv, Origin, TemplateCacheLookup, AuctionWait, Stream }` + - `pub enum AuctionWaitPlacement { PreHeader, InStream }` + - `#[derive(Clone)] pub struct RequestTimings` with: + - `pub fn new() -> Self` + - `pub fn record(&self, phase: Phase, dur: Duration)` (saturating accumulate) + - `pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration)` + - `pub fn span(&self, phase: Phase) -> PhaseSpan` (records on drop) + - `pub fn mark_headers_ready(&self)` (first call wins) + - `pub fn mark_request_elapsed(&self)` (first call wins) + - `pub fn set_resp_bytes(&self, bytes: u64)` + - `pub fn server_timing_value(&self) -> Option` + - `pub fn snapshot(&self) -> TimingSnapshot` + - `pub struct TimingSnapshot { pub time_elapsed_ms: Option, pub request_elapsed_ms: Option, pub appbuild_ms: Option, pub filter_ms: Option, pub geo_ms: Option, pub kv_ms: Option, pub origin_ms: Option, pub template_cache_ms: Option, pub auction_wait_ms: Option, pub stream_ms: Option, pub auction_wait_placement: Option, pub resp_bytes: Option }` + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!(value.contains("ts-filter;dur=9.1"), "should render one decimal: {value}"); + assert!(!value.contains("ts-geo"), "should omit unrecorded phases: {value}"); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!(timings.server_timing_value().is_none(), "should require the snapshot"); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!(timings.snapshot().time_elapsed_ms, first, "should not restamp"); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [Phase::AppBuild, Phase::Filter, Phase::Geo, Phase::EcKv, Phase::Origin, Phase::TemplateCacheLookup] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!(!value.to_ascii_lowercase().contains("datadome"), "should mask vendors"); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: compile FAIL, module does not exist. + +- [ ] **Step 3: Implement** + +```rust +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +const PHASE_COUNT: usize = 8; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + AppBuild, + Filter, + Geo, + EcKv, + Origin, + TemplateCacheLookup, + AuctionWait, + Stream, +} + +impl Phase { + fn index(self) -> usize { /* match self -> 0..=7 */ } + + /// Header entry name; row-only phases return None. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + PreHeader, + InStream, +} + +struct Inner { + t0: Instant, + phases: [Option; PHASE_COUNT], + headers_ready_total: Option, + request_elapsed: Option, + auction_wait_placement: Option, + resp_bytes: Option, +} + +#[derive(Clone)] +pub struct RequestTimings(Arc>); +``` + +Implementation notes (all bodies in this task, none deferred): + +- Every method takes `if let Ok(mut inner) = self.0.try_lock()` and silently + returns otherwise: contention and poison both drop the sample instead of waiting, + per the infallibility constraint. +- `record` accumulates with `saturating_add` semantics + (`Some(existing.saturating_add(dur))`). +- `mark_headers_ready` and `mark_request_elapsed` write `t0.elapsed()` only when the + slot is `None`. +- `server_timing_value` returns `None` unless `headers_ready_total` is set; renders + `ts-total` first from the stored snapshot, then the six header phases in enum order + with `{:.1}` millisecond formatting (`dur.as_secs_f64() * 1000.0`). +- `PhaseSpan { timings: RequestTimings, phase: Phase, started: Instant }`; `Drop` + calls `record(self.phase, self.started.elapsed())`. +- `TimingSnapshot` converts each `Duration` with + `u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)`. +- `impl Default for RequestTimings` delegates to `new()`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: all 7 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/request_timing.rs crates/trusted-server-core/src/lib.rs +git commit -m "Add RequestTimings phase collection and Server-Timing rendering" +``` + +--- + +### Task 2: Settings: `[observability]`, tinybird decoupling, access validation + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs` +- Modify: `trusted-server.example.toml` + +**Interfaces:** + +- Produces: + - `pub struct ObservabilitySettings { pub server_timing_enabled: bool }` as + `settings.observability`, `#[serde(default)]` on the field and + `#[serde(skip_serializing_if = "ObservabilitySettings::is_default")]`. + - `TinybirdSettings.auction_enabled: bool` (`#[serde(default = "default_true")]`). + - `prepare_runtime` validation: `access_enabled` requires `enabled`, non-empty + `api_host`, `secret_store`, `access_dataset`, `access_token_secret`, + `max_body_bytes > 0`, and `access_sample_rate > 0.0`. + +- [ ] **Step 1: Write the failing tests** (in `settings.rs` tests module) + +```rust +#[test] +fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!(!settings.observability.server_timing_enabled, "should default off"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" + ); +} + +#[test] +fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!(format!("{err:?}").contains("access_sample_rate"), "should name the field"); +} + +#[test] +fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!(!settings.tinybird.auction_enabled, "should disable auction emission"); + assert!(settings.tinybird.access_enabled, "should enable access emission"); +} + +#[test] +fn auction_enabled_defaults_true_for_existing_configs() { + let settings = settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!(settings.tinybird.auction_enabled, "should preserve current behavior"); +} +``` + +Also REPLACE the existing rejection test +(`tinybird_access_enabled_is_rejected_until_emitter_is_wired`, `settings.rs:4123`) +with a wiring test asserting a fully-specified access config is accepted. + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core observability access_enabled auction_enabled` +Expected: compile FAIL (`observability` field missing). + +- [ ] **Step 3: Implement** + +- Add `ObservabilitySettings` (derive `Debug, Clone, Default, PartialEq, Deserialize, +Serialize`, `#[serde(deny_unknown_fields)]`), with + `fn is_default(&self) -> bool { *self == Self::default() }`. +- Add the `observability` field to `Settings` with the serde attributes above. +- Add `auction_enabled` to `TinybirdSettings` with `default_true()`; update + `Default for TinybirdSettings`. +- Extend `TinybirdSettings::prepare_runtime` with the access validation matrix; error + messages name the failing field (`"tinybird.access_sample_rate must be > 0 when +access_enabled"` and so on). +- `trusted-server.example.toml`: add a commented `[observability]` block with + `server_timing_enabled = false` present-but-false and the env-override note (the + overlay cannot create a missing leaf), plus `auction_enabled`/access keys in the + tinybird section comments. +- Gate the auction sink: in `crates/trusted-server-adapter-fastly/src/app.rs`, + `auction_sink_from_settings` condition becomes + `settings.tinybird.enabled && settings.tinybird.auction_enabled`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core` then `cargo test-fastly` (the sink gate +touches the Fastly adapter). +Expected: PASS, including the replaced wiring test. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/settings.rs trusted-server.example.toml crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Add observability settings and decouple tinybird access and auction emission" +``` + +--- + +### Task 3: Fastly freeze point, header emission, `DeliveryOutcome` + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (entry T0, appbuild span, + `send_edgezero_response`) +- Test: `crates/trusted-server-adapter-fastly/src/app.rs` tests module (route-level + tests run under Viceroy) + +**Interfaces:** + +- Consumes: `RequestTimings`, `Phase` (Task 1); + `trusted_server_core::cache_policy::cache_control_headers_are_private_or_no_store`. +- Produces: + - `RequestTimings` inserted into request extensions at dispatch + (`core_req.extensions_mut().insert(timings.clone())`), alongside the existing + `config_store`/`device_signals`/`client_info` inserts. + - `send_edgezero_response(response, effects, timings) -> DeliveryOutcome` where + `pub(crate) struct DeliveryOutcome { pub bytes: u64, pub result: DeliveryResult }` + and `pub(crate) enum DeliveryResult { Complete, Error }` (streaming partial + detection lands in Task 6). + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn server_timing_emitted_on_private_response_when_enabled() { + // Arrange: settings with observability.server_timing_enabled = true; publisher + // route fixture whose response is Cache-Control: private, no-store. + // Act: dispatch through the full adapter path. + // Assert: + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!(header.contains("ts-total;dur="), "should carry the stored total"); + assert_eq!( + header.matches("ts-total").count(), 1, + "should emit exactly one TS-owned metric set" + ); +} + +#[test] +fn server_timing_absent_when_flag_off() { /* same fixture, flag false: no ts-total */ } + +#[test] +fn server_timing_absent_on_cacheable_responses() { + // tsjs route (public, max-age=31536000, immutable) and a bare max-age=60 response: + // both must carry no ts-total even with the flag on. +} + +#[test] +fn preexisting_server_timing_values_survive() { + // Fixture response already carrying Server-Timing: upstream;dur=1 stays present + // alongside the appended TS set. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly server_timing` +Expected: FAIL, no header emitted. + +- [ ] **Step 3: Implement** + +In `edgezero_main` (`main.rs`): + +```rust +let timings = RequestTimings::new(); +{ + let _appbuild = timings.span(Phase::AppBuild); + // existing: open_trusted_server_config_store() + build_app_with_state() +} +``` + +Move the config-store open inside the span scope. Insert `timings.clone()` into +request extensions before dispatch. Thread the handle into both send sites and the +error paths by value (it is a cheap clone). + +In `send_edgezero_response`, immediately before `response.into_parts()`: + +```rust +timings.mark_headers_ready(); +let conclusively_private = + cache_control_headers_are_private_or_no_store(response.headers()); +if settings_enabled_server_timing && conclusively_private { + if let Some(value) = timings.server_timing_value() { + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response.headers_mut().append(header::SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } + } +} +``` + +`settings_enabled_server_timing` arrives inside a small +`SendContext { timings: RequestTimings, server_timing_enabled: bool }` so the +function stays at or under seven parameters. Return `DeliveryOutcome` with per-mode +semantics: buffered bodies capture the byte count from the body length before +`send_to_client()` (which returns no delivery result) and report complete-on-return; +the streaming branch gains a counting writer in Task 6. Existing callers ignore the +outcome in this task (Task 8 consumes it). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/main.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Emit Server-Timing at the send freeze point on conclusively private responses" +``` + +--- + +### Task 4: Filter span and geo span with `GeoLookupState` dedupe + +**Files:** + +- Modify: `crates/trusted-server-core/src/geo.rs` (add `GeoLookupState`) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` + (`run_pre_route_filters` wrapper, `build_ec_request_state` geo span + state attach) +- Modify: `crates/trusted-server-adapter-fastly/src/middleware.rs` and `main.rs` + (`resolve_geo_for_response` consumes carried state) + +**Interfaces:** + +- Consumes: `RequestTimings` from request extensions (Task 3). +- Produces: + - `pub enum GeoLookupState { NotAttempted, Attempted, Resolved(GeoInfo) }` in + `trusted_server_core::geo`, attached as a response extension on every exit path + that attempted a lookup (including the asset fallback). + - `resolve_geo_for_response` gains the carried state as input: live lookup only on + `NotAttempted`; `Attempted` is never retried; fallback lookups are wrapped in + `timings.span(Phase::Geo)`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Counting geo stub: dispatch a publisher route; assert lookup count == 1 and + // x-geo-country still set on the response. +} + +#[test] +fn failed_lookup_is_not_retried() { + // Stub returns None once; assert GeoLookupState::Attempted carried and the + // finalize path performs zero further lookups. +} + +#[test] +fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // Asset route: response extension holds GeoLookupState, EcFinalizeState absent. +} + +#[test] +fn filter_span_recorded_when_request_filter_runs() { + // Registry fixture with a test request filter; assert snapshot().filter_ms is Some. +} + +#[test] +fn geo_lookup_skipped_for_unauthorized_responses() { + // Existing 401 rule preserved: no lookup, state NotAttempted. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly geo_ filter_span` +Expected: FAIL (lookup count 2; no `GeoLookupState`). + +- [ ] **Step 3: Implement** + +- `GeoLookupState` derives `Debug, Clone`; store in response extensions from the + dispatch layer right after `build_ec_request_state` resolves (or fails) its lookup. +- Wrap the `build_ec_request_state` lookup and any finalize fallback lookup in + `timings.span(Phase::Geo)` (accumulating slot handles the repeat case). +- Wrap `run_pre_route_filters` (`app.rs:751`) in `timings.span(Phase::Filter)`, + recording only when at least one filter is registered (skip the span when the + registry has no request filters, so the header omits `ts-filter` on unconfigured + deployments). +- `resolve_geo_for_response(response, carried: &GeoLookupState, client_ip, lookup)` + keeps the 401 short-circuit first, then matches the carried state. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS including untouched existing geo header tests. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/geo.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/middleware.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record filter and geo spans and dedupe the per-request geo lookup" +``` + +--- + +### Task 5: Core spans: origin, template cache, KV abstraction + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (origin send ~4496, template + cache lookup ~4391 on current main) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (graph-level `ts-kv`) +- Test: `publisher.rs` and `ec/kv.rs` test modules + +**Interfaces:** + +- Consumes: `RequestTimings` read from request extensions inside + `handle_publisher_request`; `KvIdentityGraph` gains + `pub fn with_timings(self, timings: RequestTimings) -> Self` (builder-style, + optional field), set where the graph is constructed in `main.rs`. +- Produces: `origin_ms`, `template_cache_ms`, `kv_ms` populated in snapshots. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn origin_span_covers_the_publisher_fetch() { + // Stubbed origin with a small injected delay; assert snapshot().origin_ms is Some. +} + +#[test] +fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode fixture: template_cache_ms None. Shared-mode eligible fixture: + // template_cache_ms Some. +} + +#[test] +fn kv_span_accumulates_across_graph_operations() { + // Stub KV recording two operations through a TimedKvStore-wrapped graph; assert + // kv_ms Some and covers both (accumulated, not last-write). +} + +#[test] +fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent read through the decorated RuntimeServices store: kv_ms Some. + // Pull-sync graph built from the untimed store: records nothing. +} + +#[test] +fn ec_finalize_kv_lands_before_freeze() { + // Adapter-level (test-fastly): EC-enabled fixture with eids cookies; assert the + // emitted header contains ts-kv, proving the freeze point sits after finalize. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core origin_span template_cache_span kv_span` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- `handle_publisher_request` reads the handle: + `let timings = req.extensions().get::().cloned().unwrap_or_default();` + (a defaulted handle records into nothing that ever renders, keeping non-adapter + tests unchanged). +- Origin: `let origin_span = timings.span(Phase::Origin);` immediately before + `services.http_client().send(platform_request).await`; `drop(origin_span)` when the + response headers are available (directly after the `match` arm binds the response). +- Template cache: same guard pattern around + `services.template_cache().lookup_or_reserve(key).await`. +- KV: add `TimedKvStore` (new type in `crates/trusted-server-core/src/platform/`), + a decorator implementing `PlatformKvStore` that wraps `Arc` + plus a `RequestTimings` handle and records `Phase::EcKv` around every trait + method. Every request-path `KvIdentityGraph` construction site (request setup, + identify, admin lookup, batch sync, finalization) receives the timed store; + consent-store access through `RuntimeServices` uses the same decorator; pull-sync + constructs its graph from the untimed store explicitly (add a test asserting the + pull-sync store records nothing). `ec_finalize_response` keeps seven arguments: + the handle rides inside the store the graph already receives. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` then `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/ec/kv.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record origin, template cache, and KV phase spans in core" +``` + +--- + +### Task 6: Body-phase capture: stream, auction wait placement, bytes + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (seam wait + buffered wait) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (stream drive timing, + `DeliveryOutcome.bytes`) +- Test: `publisher.rs` tests + adapter tests + +**Interfaces:** + +- Consumes: `record_auction_wait` (Task 1), `DeliveryOutcome` (Task 3). +- Produces: `stream_ms`, `auction_wait_ms` + placement, `resp_bytes`, + `mark_request_elapsed()` called by the adapter immediately after the stream drive + returns. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn streaming_seam_wait_records_in_stream_placement() { + // Streaming fixture with a delayed auction: placement InStream, and + // stream_ms >= auction_wait_ms. +} + +#[test] +fn buffered_template_miss_records_pre_header_placement() { + // Shared-template authorized miss (buffered finalizer): placement PreHeader; the + // wait is recorded even though headers had not committed. +} + +#[test] +fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + // Adapter: after send, snapshot has resp_bytes Some(body_len) and + // request_elapsed_ms Some; request_elapsed excludes post-send emitter time by + // construction (asserted by ordering test in Task 8). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly seam_wait buffered_template delivery_outcome` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Streaming path: around the `collect_stream_auction(...)` await inside the body + stream, measure with `Instant::now()` and call + `timings.record_auction_wait(AuctionWaitPlacement::InStream, waited)`. The handle + reaches the stream closure through `OwnedProcessResponseParams`/assembly params (it + is `Clone`; add a field). +- Buffered path (`buffer_publisher_response_async` and the shared-template miss + finalizer): same measurement with `AuctionWaitPlacement::PreHeader`. +- Adapter stream drive: wrap the `block_on(stream_asset_body(...))` region with a + counting writer that tallies bytes and observes truncation/error, record + `Phase::Stream` with the elapsed drive time, populate `DeliveryOutcome` with + bytes and Complete/Partial/Error, call `timings.set_resp_bytes(bytes)` and + `timings.mark_request_elapsed()` immediately after the drive returns, before + anything else post-send. Buffered responses keep the Task 3 complete-on-return + semantics; `body_mode` distinguishes the regimes in the row. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Capture stream duration, auction wait placement, and response bytes" +``` + +--- + +### Task 7: `AccessTelemetrySnapshot`, route class, route template + +**Files:** + +- Create: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (RouteMetadata attach at + handler wrappers), `main.rs` (snapshot build at freeze point), + `crates/trusted-server-core/src/publisher.rs` (typed template-cache state + extension) + +**Interfaces:** + +- Consumes: `TimingSnapshot` (Task 1), `GeoLookupState` (Task 4). +- Produces: + - `pub enum RouteClass { PublisherHtml, Tsjs, IntegrationProxy, Ec, AuctionApi, Other }` + with `pub fn as_str(&self) -> &'static str` (snake_case values from the spec). + - `pub fn publisher_route_template(path: &str) -> String`: `/` plus first segment + filtered to `[a-z0-9_-]`, truncated to 32 chars, plus `/*` when deeper; empty or + disallowed first segments render `/other/*`. + - `pub struct AccessTelemetrySnapshot { pub method: String, pub status: u16, pub route_class: RouteClass, pub route_template: String, pub publisher_domain: String, pub env: String, pub service_id: String, pub pop: String, pub ts_version: String, pub country: String, pub template_cache_state: String, pub body_mode: &'static str, pub sample_rate: f64 }` + - `pub fn access_event_row(snapshot: &AccessTelemetrySnapshot, timings: &TimingSnapshot, event_ts_epoch_ms: u64) -> String` (one NDJSON line). + +- [ ] **Step 1: Write the failing tests** (adversarial, per spec 9) + +```rust +#[test] +fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route template comes from the route table: "/_ts/admin/ec/{id}". + // Assert a row built for that route never contains a 64-hex EC id fixture. +} + +#[test] +fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); +} + +#[test] +fn row_serializes_nulls_for_missing_phases() { + // Sparse TimingSnapshot: absent phases serialize as JSON null, dimension fields + // never null (unknown sentinel). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core access_telemetry route_template` +Expected: compile FAIL. + +- [ ] **Step 3: Implement** + +Row serialization via `serde_json::json!` mapping spec section 9 column names exactly +(`time_elapsed_ms`, `appbuild_ms`, ..., `auction_wait_placement` as +`pre_header|in_stream|none`). `pop`/`service_id` read from Fastly env +(`FASTLY_SERVICE_ID`, `FASTLY_POP`), defaulting `"unknown"`; `env` derived by the +adapter from `FASTLY_IS_STAGING` (the `x-ts-env` input), never from `Settings`. +Route identity travels as a typed `RouteMetadata` response extension +(`pub struct RouteMetadata { pub route_class: RouteClass, pub route_template: String }` +in `access_telemetry.rs`): each named-route handler wrapper attaches its matched +route-table pattern verbatim, and the fallback and tsjs handlers attach their class +plus the coarse template; the freeze point consumes the extension (no `RouteClass` +column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: +make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set +at every point that writes `x-ts-template-cache` so header and extension cannot +drift; the row reads the extension. The snapshot is built unconditionally in +`send_edgezero_response` right after `mark_headers_ready()` and returned inside +`DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` and `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/access_telemetry.rs crates/trusted-server-core/src/lib.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Add access telemetry snapshot, route classes, and coarse route templates" +``` + +--- + +### Task 8: Access sink with confirmed delivery + post-send ordering + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/tinybird.rs` (access sink) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (post-send ordering) + +**Interfaces:** + +- Consumes: `AccessTelemetrySnapshot` + `access_event_row` (Task 7), settings flags + (Task 2), `DeliveryOutcome` (Tasks 3/6). +- Produces: `pub(crate) async fn emit_access_event(client: &FastlyPlatformHttpClient, target: &TinybirdEventsTarget, row: String) -> Result<(), Report>`, + sending via the adapter's stateless platform client (the blocking variant, + post-delivery), checking `response.status().is_success()`, warning with status + otherwise. The transport context is adapter-owned and route-independent (target + derived from settings once at entry), so asset, admin, and error responses emit + without `RuntimeServices` or `EcFinalizeState`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn access_emitter_posts_ndjson_and_validates_2xx() { + // RecordingHttpClient returning 202: assert URI is /v0/events?name=access_logs_raw, + // body is the row, Authorization bearer from the secret stub. +} + +#[test] +fn access_emitter_warns_and_drops_on_non_2xx() { + // RecordingHttpClient returning 422: emit returns Err naming the status; no retry + // request recorded (exactly one request seen). +} + +#[test] +fn sampled_out_requests_emit_nothing() { + // access_sample_rate stub decision false: RecordingHttpClient sees zero requests. +} + +#[test] +fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // Instrumented stubs record call order; assert request_elapsed snapshot precedes + // pull-sync dispatch which precedes the telemetry send. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly access_emitter post_send_order` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Reuse `TinybirdEventsTarget` with a second constructor + `from_access_config(config: TinybirdSettings)` using `access_dataset` and + `access_token_secret`. +- Sampling decision: `fn sampled_in(rate: f64, entropy: u64) -> bool` where entropy is + derived from the event timestamp nanos XOR a per-request counter (no `rand` + dependency; document that uniformity is approximate and sufficient). +- `main.rs` post-send, in order: `timings.mark_request_elapsed()` (already placed in + Task 6), existing pull-sync dispatch unchanged, then when + `settings.tinybird.enabled && settings.tinybird.access_enabled` and sampled in: + build the row from `outcome.snapshot` + `timings.snapshot()`, call + `emit_access_event`, log one warning on `Err`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" +``` + +--- + +### Task 9: Tinybird datasource schema + +**Files:** + +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +**Interfaces:** + +- Consumes: column names exactly as serialized by `access_event_row` (Task 7). +- Produces: the deployed schema contract for the dashboard (separate repo). + +- [ ] **Step 1: Rewrite the schema** per spec section 9: keep + `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, + `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + dimension columns non-nullable `LowCardinality(String)` and phase columns + `Nullable(UInt32)`; drop `path` and `cache_state`; set + `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + +- [ ] **Step 2: Validate** with the tinybird toolchain if available locally + (`tb check` / project tests under `tinybird/tests`); otherwise assert the file + parses by review and rely on rollout step 4's remote verification. Add a fixture row + in `tinybird/fixtures` matching `access_event_row` output. + +- [ ] **Step 3: Commit** + +```bash +git add tinybird/datasources/access_logs_raw.datasource tinybird/fixtures +git commit -m "Extend access_logs_raw with phase columns and a non-null sorting key" +``` + +--- + +### Task 10: Axum adapter emission + +**Files:** + +- Modify: `crates/trusted-server-adapter-axum/src/` (terminal layer at the response + serialization boundary; locate the equivalent of the Fastly send path) +- Test: axum adapter tests (`cargo test-axum`) + +**Interfaces:** + +- Consumes: `RequestTimings`, header emission helper. Extract the emission block from + Task 3 into a shared core helper so both adapters call one function: + `pub fn append_server_timing_if_private(response: &mut Response, timings: &RequestTimings, enabled: bool)` + in `request_timing.rs` (move the Fastly inline logic here and re-point Task 3's call + site). +- Produces: Axum responses carry the header under the same conservative predicate; + `ts-appbuild` absent by construction (state built at startup); router-generated + 404/405 covered by the terminal layer; `/health` excluded by route match. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn axum_emits_header_on_private_response() { /* flag on, private response: ts-total present, ts-appbuild absent */ } + +#[test] +fn axum_404_carries_header_when_private() { /* router-generated 404 passes through the terminal layer */ } + +#[test] +fn axum_health_is_excluded() { /* /health: no ts-total */ } +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum axum_emits axum_404 axum_health` +Expected: FAIL. + +- [ ] **Step 3: Implement** an outer service wrapper around the `RouterService` + inside `AxumDevServer` (not router middleware, which router-generated 404/405 + responses bypass and which returns before body serialization): create + `RequestTimings::new()` per request in the wrapper, insert into request + extensions, and on the wrapper's response side call `mark_headers_ready()` + + `append_server_timing_if_private(...)`, skipping the `/health` path by match. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-axum/src crates/trusted-server-core/src/request_timing.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit Server-Timing from the Axum terminal layer with adapter-specific semantics" +``` + +--- + +### Task 11: Full gate, docs, and PR + +- [ ] **Step 1: Docs.** Add a short operator section to `docs/guide/configuration.md`: + the `[observability]` flag, the tinybird access keys, the deploy/rollback ordering + from spec section 12 (binary first, config second; config first on rollback), and + the conservative emission rule. Run `cd docs && npm run format`. + +- [ ] **Step 2: Full CI gate list** from `CLAUDE.md`: + `cargo fmt --all -- --check`; all six clippy aliases; `test-fastly`, `test-axum`, + `test-cloudflare`, `test-spin`; the integration-tests parity suite; JS build/test + and formats. Cloudflare/Spin compile the new core modules (collection only), which + is exactly what the non-goal requires. + +- [ ] **Step 3: Commit docs, push the branch, open the implementation PR** referencing + the spec PR #1069 and issue #1068, with the rollout section of the spec quoted as + the deployment checklist (staging pass-through + MISS/HIT replay before production + flag-on). + +--- + +## Self-Review + +- Spec coverage: sections 5 (Task 1), 12 (Task 2), 7 (Tasks 3, 10), 8/8a (Tasks 4, + 10), 6 (Tasks 4-6), 9 (Tasks 7, 9), 10 (Task 8), 13 (Tasks 1, 3, 8), 14 (test + steps throughout), 15 steps 1-4 (Task 11 + deployment checklist). Section 11 + (dashboard) is explicitly out of scope for this repo's plan. +- Type consistency: `RequestTimings`/`TimingSnapshot`/`RouteClass`/ + `AccessTelemetrySnapshot`/`DeliveryOutcome` names and signatures match across + Tasks 1, 3, 6, 7, 8, 10. +- Known intentional deferral: `DeliveryResult::Partial` detection is named in Task 3 + and wired when the stream drive reports bytes in Task 6; no other deferrals. diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md new file mode 100644 index 000000000..06f27c1c2 --- /dev/null +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -0,0 +1,553 @@ +# Request phase timing: Server-Timing subtimings and access telemetry + +**Date:** 2026-08-24 +**Status:** Approved design, revised for review rounds 1 and 2, pending implementation +plan. +**Scope:** `trusted-server-core`, Fastly and Axum adapters, `tinybird/` schema, +performance dashboard (separate repo). + +--- + +## 1. Problem + +On 2026-08-21 a production deployment (publisher redacted, `prospect-a.example`) showed +an episodic stall: for a window of roughly 40 minutes, every request that reached the +application path carried a uniform extra ~600 ms of Fastly `time-elapsed`, and then +recovered to 20-50 ms with no deploy or config change we could observe. `/health` +(2-4 ms, short-circuits before app construction) and `/_ts/debug/ja4` (6-9 ms, settings +load only) stayed fast throughout, so the stall lived between app construction and +response send. + +Attributing that window required a live probing session: route-by-route bisection, +cookie-deletion experiments, and an eight-agent code trace. The trace found no +unconditional await on the path that could cost 570 ms, and exactly two +config-conditional candidates (the pre-route request filter's synchronous verification +POST, and EC identity KV writes before send), plus one dependency shared by every +application route (two geo hostcalls per request). We could not tell which one stalled, +because nothing in the response says where server time went. + +The Compute CPU budget is ~50 ms per request, so a large `time-elapsed` strongly +suggests wall-clock time outside active guest CPU: dependency awaits are the leading +explanation, with platform scheduling and hostcall queueing as the residual ones. The +comparison figure here is the fronting delivery layer's `time-elapsed` Server-Timing +entry, observed at its deliver phase. Either way, these are exactly the numbers a +response can carry about itself. + +## 2. Goals + +1. Every normal application response attributes its own server time by phase in a + standard header. Browsers expose the values to same-origin JavaScript via + `PerformanceResourceTiming.serverTiming`, so RUM tooling that reads that API can + surface the breakdown. Whether a given vendor or the publisher's own monitoring + extension actually collects it is verified separately in rollout; the publisher + extension needs a small change to render it. +2. The same numbers flow to Tinybird so we hold p50/p95/p99 per phase, per route class, + per PoP, per deployed version, and a future stall window self-diagnoses in one query. +3. No additional awaited I/O before first byte. The pre-send cost is a handful of + monotonic clock reads, one small allocation at entry, and rendering one header; + telemetry emission happens strictly after the last body byte. + +Scope note: phases cover the application lifecycle after T0. The `/health` and +`/_ts/debug/ja4` short-circuits, config-store open failures, and request-conversion +failures bypass the lifecycle and emit nothing. Requests served entirely by the +fronting cache never reach the guest and produce neither header entries nor rows. + +## 3. Non-goals + +- No trailer-based Server-Timing for body-phase spans (browsers do not expose trailer + values to JavaScript). +- No per-filter naming in any emitted surface. The request-filter span is `ts-filter` + regardless of which filter runs; vendor identity stays out of headers and telemetry. +- No Cloudflare or Spin emission wiring in v1. Core collection is adapter-neutral; those + adapters can wire emission later without core changes. +- No Tinybird endpoint pipe and no rollup materialized views in v1. Grafana queries the + datasource through the ClickHouse connector, matching the auction dashboards; rollups + only if panel latency demands them. +- No sampling of the header. The header is all-traffic when enabled; only Tinybird rows + sample. +- No cross-request circuit breaker for telemetry emission. Compute runs one isolate per + request; there is no shared mutable state to hold breaker state. The controls are the + bounded per-request cost and the `access_sample_rate` lever (section 10). + +## 4. Design overview + +``` +adapter entry (T0) + | RequestTimings::new() -> shared handle + v +app construction ................ ts-appbuild (adapter) +pre-route request filters ....... ts-filter (adapter wrapper) +geo lookup (single, deduped) .... ts-geo (adapter; result carried forward) +template cache lookup ........... ts-template-cache (core: publisher.rs) +origin fetch to resp headers .... ts-origin (core: publisher.rs) +EC identity KV, pre-send ........ ts-kv (core: KV abstraction) +auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in this mode) + | +send_edgezero_response, immediately before into_parts(): + mark_headers_ready() snapshot (unconditional) + build AccessTelemetrySnapshot (unconditional) + append Server-Timing header (flag-gated, only on conclusively private responses) + | +headers committed; body streams + auction hold at seam .......... auction_wait_ms (row only; in-stream in this mode) + stream duration, bytes ........ stream_ms, resp_bytes (row only) + | +post-send (adapter main): + request_elapsed snapshot, then existing pull-sync, then: + sample gate -> one NDJSON row -> Tinybird Events API + bounded response await, 2xx validated +``` + +Collection is always-on and flag-free, including the `mark_headers_ready()` snapshot. +Two independent flags gate emission: the header (`observability.server_timing_enabled`) +and the telemetry row (`tinybird.access_enabled`). + +## 5. `RequestTimings` (core) + +New module `crates/trusted-server-core/src/request_timing.rs`. + +- `Phase`: a closed enum: `AppBuild`, `Filter`, `Geo`, `EcKv`, `Origin`, + `TemplateCacheLookup`, `AuctionWait`, `Stream`. Header rendering covers the first six + plus the stored total; the last two are row-only. +- Inner state: one fixed-size array of `Option` slots indexed by phase, + `t0: Instant`, `headers_ready_total: Option`, + `auction_wait_placement: Option` (`PreHeader` or `InStream`), + and `resp_bytes: Option`. Phases that repeat within a request (geo, KV) + accumulate by saturating addition into the same slot. +- `mark_headers_ready()`: stores `t0.elapsed()` once at the response-commit boundary, + unconditionally, before either emission flag is consulted. The header renders this + stored value as `ts-total`; the telemetry row reads the same stored value as + `time_elapsed_ms`. The two surfaces cannot disagree, and the row stays correct when + the header flag is off. Full request duration is captured separately as + `request_elapsed_ms`, snapshotted immediately after the body-stream drive returns and + before any other post-send work, so pull-sync and telemetry emission are never + included in it. +- Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses + three boundaries: adapter entry to core handlers, the streaming body closure (records + body-phase spans after the response object has been handed off), and the adapter's + post-send emission read. Access is exclusively `try_lock()`: a contended or + poisoned lock drops the sample immediately rather than waiting, so recording can + never delay a request. +- Recording API: `timings.record(Phase::Geo, dur)` and a scope guard + `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration + math; a non-monotonic reading records zero rather than panicking. The auction-wait + recorder takes the placement explicitly so the two modes cannot be conflated. +- Rendering: `server_timing_value(&self) -> Option` produces + `ts-total;dur=41.2, ts-appbuild;dur=18.4, ts-filter;dur=9.1` with durations in + milliseconds at one decimal. Phases never recorded are omitted. Returns `None` when + `mark_headers_ready()` has not run. + +`Instant` is already used freely in the guest (`publisher.rs`, `auction/telemetry.rs`), +so no new clock abstraction is needed. + +## 6. Span taxonomy and recording sites + +| Entry | Measures | Site | +| ------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| `ts-total` | T0 to `mark_headers_ready()` at the response-commit boundary | stored snapshot | +| `ts-appbuild` | config-store open, Settings parse, orchestrator + registry + router build | Fastly `main.rs` around `open_trusted_server_config_store()` + `build_app_with_state()` | +| `ts-filter` | pre-route request filters, end to end (backend ensure, secret read, POST) | around `run_pre_route_filters` (`app.rs:751`) | +| `ts-geo` | geo hostcall (single after dedupe; accumulates if any path still repeats) | `build_ec_request_state` (`app.rs:410`) and timed finalizer fallback lookups | +| `ts-kv` | EC identity KV operations before response send (see enumeration below) | the shared KV abstraction | +| `ts-origin` | publisher backend send to response headers available (read-through cache hit or miss) | `publisher.rs` around the origin `send` | +| `ts-template-cache` | template cache `lookup_or_reserve`, including the hit-path full-body read | `publisher.rs` around the lookup | + +Naming follows the completed template-cache terminology migration (`x-ts-template-cache` +is the emitted header on `main`; `c2` naming is retired). + +`ts-kv` is instrumented by a timing decorator implementing `PlatformKvStore` that +wraps the store handed to request-scoped consumers, because no single existing +abstraction covers the taxonomy: EC graph operations go through `KvIdentityGraph` +while consent persistence uses `PlatformKvStore` directly, and graphs are constructed +independently in request setup, identify, admin lookup, batch sync, and finalization. +Every request-path graph construction receives the timed store; pull-sync explicitly +constructs its graph from an untimed store. Consent-store reads pass through the same +decorator and are timed like any other store call. Included pre-send operations: EC +generation `create_or_revive`, identify-path graph reads and evaluation, finalize-path +`ingest_eid_cookies`/`upsert_partner_ids` and withdrawal tombstones, consent-store +reads on consent routes, and batch-sync graph access when it runs before send. +Explicitly excluded: pull-sync work, which runs strictly after `send_to_client` and is +invisible to both surfaces. The decorator measures store-call latency only: no value +passing through it is read, parsed, or recorded, and the emitted surfaces carry no +consent or identity payloads. This feature therefore needs no consent gate; it is the +site measuring its own infrastructure, not processing user data. The timings handle +reaches `ec_finalize_response` inside the graph it already receives; that function +keeps the repository maximum of seven arguments and does not gain an eighth. + +Row-only fields: + +| Field | Measures | Site | +| -------------------- | --------------------------------------------------------- | ------------------------------------------------ | +| `auction_wait_ms` | wait on the dispatched auction (placement varies by mode) | seam hold (streaming) or buffered finalizer wait | +| `body_mode` | `streamed` or `buffered` response assembly | set where the response body is built | +| `stream_ms` | headers committed to last body byte | adapter around the body-stream drive | +| `resp_bytes` | bytes written to the client body | same | +| `request_elapsed_ms` | T0 to immediately after the body-stream drive returns | post-send snapshot, before pull-sync | + +Auction-wait placement is not universal. On the ordinary streaming path the wait +happens at the `` seam inside the body stream and nests inside `stream_ms`. On +buffered paths (the Fastly shared-template authorized miss, which buffers the full +transform and auction before returning a response, and every Axum response) the wait +completes before headers commit. The row therefore carries `body_mode` plus +`auction_wait_placement` (`pre_header` or `in_stream`), and derivations are +conditional: + +- `in_stream`: `stream_other_ms = greatest(coalesce(stream_ms, 0) - coalesce(auction_wait_ms, 0), 0)`. +- `pre_header`: `auction_wait_ms` joins the pre-header phase set, and `stream_other_ms = coalesce(stream_ms, 0)`. + +`unattributed_ms = greatest(coalesce(time_elapsed_ms, 0) - (coalesce(appbuild_ms, 0) + +coalesce(filter_ms, 0) + coalesce(geo_ms, 0) + coalesce(kv_ms, 0) + +coalesce(origin_ms, 0) + coalesce(template_cache_ms, 0) + pre-header auction wait), 0)`. +Every phase column is nullable, so every query-time formula wraps each term in +`coalesce(column, 0)` and every subtraction in `greatest(..., 0)`; query tests cover +sparse phase combinations. + +## 7. Freeze point and header emission + +The freeze-and-emit point is `send_edgezero_response` (Fastly `main.rs`), immediately +before `response.into_parts()`. This is the single choke point every send path shares, +and it runs after everything that can still mutate the response: the router middleware, +entry-point finalize (`apply_finalize_headers`, asset-policy reapplication), EC +finalization and its KV work, and terminal filter/privacy effects. +`apply_finalize_headers` itself does not emit; the `HEADER_X_TS_FINALIZED` sentinel +marks middleware finalization, not header commitment, and must not be treated as the +timing boundary. + +At the freeze point, in order: `mark_headers_ready()` (unconditional), the +`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`observability.server_timing_enabled`, append one `Server-Timing` header from +`server_timing_value()`. Append semantics, never insert: an origin-supplied +Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, +`hit-state`) are additive per the header's list semantics. + +Header emission is conservative: it happens only when the response is conclusively +non-storable by any shared cache, meaning `Cache-Control` contains `private` or +`no-store` (the existing `cache_control_headers_are_private_or_no_store` predicate). +Anything else, including bare `max-age`, `s-maxage` without `private`, +heuristically-cacheable responses with no cache header at all, and anything a fronting +cache override might store, emits no header, because a stored object would replay one +request's timings for its full lifetime. The long-lived immutable `tsjs` asset route +is the concrete excluded case. The snapshot and the telemetry row are unaffected by +this skip, so excluded routes still report through Tinybird. + +The Axum adapter applies the same emission rule at its terminal point before response +serialization, with adapter-specific phase semantics (section 8a). + +## 8. Geo lookup dedupe (rider) + +Today every dispatched request pays two geo hostcalls for one answer: request-phase in +`build_ec_request_state` (`app.rs:410`) and response-phase in +`FinalizeResponseMiddleware` (`middleware.rs:83`, alternate site `main.rs:285`). + +Plain request extensions cannot carry the result out: the middleware moves the request +context into `next.run(ctx)` and holds only the response afterward. The resolved geo +travels on a dedicated `GeoLookupState` response extension, attached on every exit +path that attempted a lookup, including the asset fallback, which runs +`build_ec_request_state` and then returns without `EcFinalizeState` (which is why +`EcFinalizeState` is not an acceptable carrier). States: `NotAttempted`, +`Attempted(None)` (lookup ran and failed, do not retry), and `Resolved(GeoInfo)`. The +finalize path consumes the carried value and performs a live lookup only in the +`NotAttempted` state; those legitimate fallback lookups (admin, batch, error paths) +are themselves timed into `ts-geo` so degraded geo cannot hide inside +`unattributed_ms`. The 401 rule (`resolve_geo_for_response` skips lookup for +unauthorized responses) is preserved. + +## 8a. Adapter phase semantics + +The Fastly adapter is the reference implementation of the taxonomy. Axum differs +structurally and its emissions are defined accordingly rather than pretending parity: + +- `ts-appbuild` is absent: Axum builds application state once at startup. +- `body_mode` is always `buffered`: the Axum HTTP client buffers upstream bodies, so + `stream_ms` measures buffered-body write-out and `auction_wait_placement` is always + `pre_header`. +- The freeze point is an outer service wrapper around the `RouterService` inside + `AxumDevServer`, not router middleware: router-generated 404/405 responses bypass + router middleware, and middleware returns before Axum serializes the body. The + wrapper sees every response including router-generated ones; `/health` is excluded + by path match inside the wrapper. +- Axum emits the header only; no Tinybird rows in v1 (unchanged). + +Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). + +## 9. Access telemetry row + +Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. + +Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the +`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. + +Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put +EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality +and user-generated content (search terms, usernames, emails in slugs). Replaced by +`route_template`: + +- Named routes: the matched route-table pattern verbatim, parameters left as + placeholders. +- Publisher fallback: a coarse fixed template, `/` plus the first path segment + restricted to a bounded allowlisted charset, plus `/*` when deeper (for example + `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it + redacts long tokens but preserves short identifiers and arbitrary slugs. +- Tests are adversarial, not just the happy path: a literal EC identifier on the admin + route, an email address in a path segment, search-term-shaped segments, and + overlong segments must all normalize to bounded, content-free templates. + +Added columns (all dimension columns non-nullable with an `unknown` sentinel, because +ClickHouse sorting keys cannot contain nullable columns): + +``` +`service_id` LowCardinality(String), -- FASTLY_SERVICE_ID; immutable deployment identity +`publisher_domain` LowCardinality(String), -- matches auction schema +`env` LowCardinality(String), -- adapter-derived: production | staging | unknown +`route_class` LowCardinality(String), -- publisher_html | tsjs | integration_proxy | ec | auction_api | other +`route_template` String, -- bounded, normalized; replaces path +`body_mode` LowCardinality(String), -- streamed | buffered +`auction_wait_placement` LowCardinality(String), -- pre_header | in_stream | none +`appbuild_ms` Nullable(UInt32), +`filter_ms` Nullable(UInt32), +`geo_ms` Nullable(UInt32), +`kv_ms` Nullable(UInt32), +`origin_ms` Nullable(UInt32), +`template_cache_ms` Nullable(UInt32), +`auction_wait_ms` Nullable(UInt32), +`stream_ms` Nullable(UInt32), +`request_elapsed_ms` Nullable(UInt32), +`resp_bytes` Nullable(UInt64), +`template_cache_state` LowCardinality(String), -- from the typed response extension, not the public header +`country` LowCardinality(String), +`ts_version` LowCardinality(String), +`pop` LowCardinality(String) -- FASTLY_POP, 'unknown' when absent +``` + +The matched route pattern does not survive dispatch today, so a typed +`RouteMetadata` response extension carries `route_class` and `route_template`: each +named-route handler wrapper attaches its route-table pattern verbatim (handlers +serving multiple patterns attach the one that matched), and the fallback and tsjs +handlers attach their class plus the coarse template. The freeze point consumes the +extension; nothing reconstructs routes from a handler enum or path regex. + +Typed sources only: `env` is adapter-owned, derived from the same Fastly +`FASTLY_IS_STAGING` input that drives `x-ts-env` (`Settings` has no environment +field and does not gain one). `template_cache_state` comes from a typed response +extension, not the `x-ts-template-cache` header (operator-configured response +headers can override managed headers): the currently private +`TemplateCacheResponseState` in `publisher.rs` becomes a typed response extension, +and every state transition sets the managed header and the extension together so +the two can never drift. `service_id` and `pop` come from the Fastly environment. `cache_state` +from the reserved schema is dropped: the guest cannot observe the fronting cache, and +guest-visible cache behavior is already carried by `template_cache_state` and +`origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are +invisible by construction and the dashboard documentation says so. + +Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, +status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +also carries an `event_date` predicate so the primary index prunes; rollout validates +the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the +reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the +reserved datasource was ever deployed to the remote workspace; if it was, this schema +ships as a versioned replacement datasource with a cutover, not an in-place edit. + +## 10. Emission mechanics + +- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before + `into_parts()` consumes the response. It captures method, status, route metadata + (from the `RouteMetadata` extension), and typed dimension states (`env`, + `template_cache_state`, geo country). It exists because nothing else survives to + post-send on every path: the request is consumed by dispatch, the response by + `into_parts()`, and `EcFinalizeState` is absent on asset, admin, and error paths. +- The emitter's transport context is adapter-owned and route-independent: the + Events API target (backend spec, secret store name, dataset, token secret, sample + rate) derives from settings once at entry in `main.rs`, and the HTTP client is the + adapter's stateless platform client. Asset, admin, and error responses therefore + emit without `RuntimeServices` or `EcFinalizeState`. +- `send_edgezero_response` returns a delivery outcome instead of `()`, with + per-mode semantics because the two body paths observe different things. Streamed + bodies: a counting writer reports bytes written and distinguishes complete, + partial (truncated), and error outcomes. Buffered bodies: `send_to_client()` + returns no delivery result, so the byte count is captured from the body length + before the send and the outcome is complete-on-return with no partial detection; + `body_mode` in the row keeps the two regimes distinguishable in analysis. +- Ordering after the body-stream drive returns: snapshot `request_elapsed_ms` first, + run the existing pull-sync dispatch unchanged, then telemetry emission last, so + pull-sync is never delayed behind the ingest await and never included in + `request_elapsed_ms`. +- Sampling: uniform per-request decision against `tinybird.access_sample_rate`. No + client stickiness. Sampled-out requests are silent; every other drop (row build + failure, send failure, non-2xx) logs one warning naming the reason. There is no + cross-request warning suppression (per-request isolates hold no shared state); the + overload controls are the 2 s bounded await, the single-warning-per-request cap, and + `access_sample_rate` pushed down by config as the operational abort lever. Ingest + health is monitored from the Tinybird side via ingestion freshness on the + datasource, which catches quarantine and schema rejection that per-request warnings + cannot. +- Transport: one NDJSON row to the Tinybird Events API: same `api_host`, reserved + `access_dataset` and `access_token_secret`, 2 s first-byte and between-bytes + timeouts, `max_body_bytes` guard, no retry. +- Delivery confirmation: unlike the auction sink, which starts `send_async` and drops + the pending response (it runs before delivery completes and cannot afford to wait), + the access emitter runs after the client has the full response and therefore awaits + the bounded ingest response and validates 2xx. A non-2xx or timeout logs a warning + with the status. +- Budget: at `access_sample_rate = 1.0` this adds one backend request per request to + the service, after delivery; during a Tinybird outage each such request holds its + sandbox for up to the bounded timeout. The sample rate is the budget control; 1.0 is + a diagnosis setting, not a steady state, and rollout treats sustained emission + warnings as the signal to dial it down. +- Axum adapter: emits the header only; no Tinybird rows in v1. + +## 11. Dashboard and query model + +No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the +ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, +matching the auction dashboards. + +Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the +telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with +the revenue and auction dashboards. Panels: + +- Phase percentiles (p50/p95/p99) by `route_class`, per phase column. +- Stacked phase breakdown over time using the non-overlapping set: `appbuild_ms`, + `filter_ms`, `geo_ms`, `kv_ms`, `origin_ms`, `template_cache_ms`, pre-header + auction wait (where `auction_wait_placement = 'pre_header'`), and derived + `unattributed_ms`. In-stream auction wait and derived `stream_other_ms` chart in a + separate body-phase panel and never stack with pre-header phases. +- PoP split, `ts_version` overlay, template-cache state rates. +- Stall panel: rows with `request_elapsed_ms > 500` (post-body total, so body-only + stalls are caught) grouped by dominant phase, where `unattributed_ms` competes as a + phase so the panel cannot confidently blame a small measured span while most time is + uninstrumented. + +All derivations use the `coalesce`/`greatest` forms from section 6; query tests cover +sparse phase combinations and both `auction_wait_placement` modes. + +Sampling semantics for every aggregate: `sample_rate` must be operationally stable +within any queried window. Quantile panels filter strictly to a single `sample_rate` +value. Volume panels weight each row by `1.0 / sample_rate` (the inverse-probability +estimator is `sum(1.0 / sample_rate)` over emitted rows; `count() / rate` is valid +only when the query is already filtered to one rate). Pooled unweighted quantiles +across a rate change are documented as invalid. + +## 12. Config surface + +```toml +[observability] +# Append TS phase timings to the Server-Timing response header. +server_timing_enabled = false # example default +``` + +New `ObservabilitySettings` struct with the single boolean, default off, standard +environment override (`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED`). +Collection has no flag: the flags gate the two emission surfaces independently. + +Tinybird flag structure: `tinybird.enabled` today arms the auction sink by itself, so +"enable Tinybird for access telemetry" would silently enable auction emission too. The +master flag is demoted to transport-only (host, store, credentials), and each emitter +gets its own switch: a new `tinybird.auction_enabled` defaulting to `true` (preserving +current behavior for existing configs) and the reserved `tinybird.access_enabled` +defaulting to `false`. A settings test locks the decoupling in both directions. + +Validation when `access_enabled = true`: `tinybird.enabled`, non-empty `api_host`, +non-empty `secret_store`, `access_dataset`, and `access_token_secret`, a positive +`max_body_bytes`, and `access_sample_rate > 0`. An armed-but-silent configuration +(`access_enabled = true`, `access_sample_rate = 0`) is a configuration error, not a +valid state; disabling is done with the flag, not the rate. + +Rollback and compatibility, because `Settings` is `deny_unknown_fields`: + +- Deployment order is binary first, config second. Rollback order is config first + (remove the `[observability]` table and any new tinybird keys), binary second. A + config containing the new fields must never be pushed while a pre-observability + binary can still run. +- Config serialization omits the table when it equals the default, so round-tripping a + config through tooling does not inject a field an older binary rejects. A + compatibility test asserts the serialized default config parses under the previous + schema. +- The environment-variable overlay cannot create a missing leaf, so the key ships + present-but-false in the base operator TOML (the same pattern the GPT integration + documents in `trusted-server.example.toml`) and is flipped by config push. + +## 13. Error handling + +- Recording is infallible: saturating math, lock-failure drops the sample, no panics. +- Header rendering failure (defensive `HeaderValue::from_str` error) logs and skips + the header. +- Row emission failure logs one warning naming the reason and drops the row. The + response has already been delivered; there is nothing to degrade. + +## 14. Testing + +- Core unit tests: phase accumulation, saturating math, `mark_headers_ready()` + idempotence and both-surface consistency, render format (one decimal, omission of + unrecorded phases), row serialization shape, auction-wait placement recording. +- Adapter tests (Fastly via Viceroy, Axum native): header present and well-formed on a + conclusively-private publisher route with the flag on; absent with the flag off; + absent on the shared-cacheable tsjs route and on a bare `max-age` response with the + flag on; exactly one TS-owned metric set (a single `ts-total`) with every + pre-existing Server-Timing value preserved, across all send paths; `ts-kv` captures + EC finalize work (proving the freeze point sits after it). +- Body-mode tests: ordinary streaming (in-stream wait nested in `stream_ms`), + Fastly shared-template authorized miss (buffered, pre-header wait), and Axum + (always buffered), each asserting placement and non-negative derivations. +- Geo dedupe: finalize consumes `Resolved`; no retry on `Attempted(None)`; live + lookup only on `NotAttempted`; fallback lookups timed into `ts-geo`; asset-fallback + path carries `GeoLookupState` without `EcFinalizeState`; 401 skip preserved. +- Route template: adversarial normalization tests (literal EC identifier on the admin + route, email address in a segment, search-term segments, overlong segments) all + producing bounded content-free templates. +- Settings: the access validation matrix including the armed-but-silent rejection; + auction/access flag decoupling in both directions; the former rejection test becomes + the wiring test; the serialized-default-config compatibility test against the + previous schema. +- Sink tests: `RecordingHttpClient` pattern; assert URI, NDJSON body shape, token + header, 2xx validation and warning on non-2xx, skip when sampled out, ordering after + pull-sync. +- Query tests: derivation formulas against sparse rows and both placements. + +## 15. Rollout and verification + +1. Land collection + freeze point + header emission behind the flag, off everywhere. + Full CI gate. +2. Staging deploy with the flag on. Delivery-layer verification is two-sided: a + pass-through request confirming the appended Server-Timing survives the fronting + VCL, and a MISS-then-HIT replay against a cacheable route confirming no stale + timing header is ever served from cache. Fallback if the VCL clobbers the header: a + one-line VCL change on the delivery service, or mirroring the value to + `x-ts-timing` while that lands. +3. Production flag on. Confirm + `performance.getEntriesByType('navigation')[0].serverTiming` shows `ts-*` entries + in a real browser session, and separately confirm what the publisher's RUM tooling + actually collects; the publisher monitoring extension renders it only after a small + change on their side. +4. Verify whether `access_logs_raw` exists in the remote Tinybird workspace. If yes, + ship the schema as a versioned replacement with cutover; if no, edit in place. + Validate the dashboard panel queries with `EXPLAIN` against the sorting key. Then + land the row schema, sink, and settings changes; sample at 1.0 during stall + diagnosis with ingestion-freshness monitoring on the datasource; then the + dashboard. +5. Success criterion: the next stall window is attributable from one response header + or one dashboard query, with no live probing session. + +## 16. Overhead + +Roughly ten monotonic clock reads, two stored snapshots, and one ~130-byte header per +request; one sampled HTTP POST with a bounded await after the response has fully +streamed. No allocation in the hot path beyond the one `Arc` at entry, the +`AccessTelemetrySnapshot` at the freeze point, and the rendered header string. + +## 17. Decisions and open questions + +- **Public exposure is a decision, not an open question.** The header is all-traffic + when enabled. Rationale: values are durations only; the delivery layer already + exposes `hit-state` and `time-elapsed` publicly on every response; filter vendor + identity is masked; emission is restricted to conclusively-private responses so no + cache can replay stale timings. Revisit (quantization or gating) only if a concrete + abuse surfaces. +- The fronting delivery layer's Server-Timing pass-through is unverified until the + first staging deploy (step 2). This is the only known external dependency. +- Body-phase capture threads the timings handle into the streaming closure in + `publisher.rs`; the exact seam is an implementation-plan detail, with the constraint + that a dropped handle (error paths, early client disconnect) must still yield a + valid row with null body-phase fields and a recorded delivery outcome. +- The stall window itself remains unattributed until this ships. If it recurs first, + the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset + path versus HTML path) is the fallback. From 052eadad04af7964933178d4482881302de0c1b5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 19:31:18 -0500 Subject: [PATCH 002/230] Add RequestTimings phase collection and Server-Timing rendering --- crates/trusted-server-core/src/lib.rs | 1 + .../trusted-server-core/src/request_timing.rs | 437 ++++++++++++++++++ 2 files changed, 438 insertions(+) create mode 100644 crates/trusted-server-core/src/request_timing.rs diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 48e92faed..b8a1a5718 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -61,6 +61,7 @@ pub mod proxy; pub mod publisher; pub mod redacted; pub mod request_signing; +pub mod request_timing; pub mod response_privacy; pub mod rsc_flight; pub(crate) mod s3_sigv4; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs new file mode 100644 index 000000000..b0cb7b0dd --- /dev/null +++ b/crates/trusted-server-core/src/request_timing.rs @@ -0,0 +1,437 @@ +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +/// Number of [`Phase`] variants; sizes the fixed-slot duration array in +/// [`Inner`]. +const PHASE_COUNT: usize = 8; + +/// A distinct stage of request handling that duration can be attributed to. +/// +/// Variants map to fixed slots in [`RequestTimings`], in declaration order. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + /// Time spent constructing the app/handler before request processing + /// begins. + AppBuild, + /// Time spent in request/response filtering (e.g. HTML rewriting). + Filter, + /// Time spent resolving geographic signals for the request. + Geo, + /// Time spent reading or writing the Edge Cookie key-value store. + EcKv, + /// Time spent waiting on the origin fetch. + Origin, + /// Time spent looking up a cached template. + TemplateCacheLookup, + /// Time spent waiting on the auction. Row-only: never rendered as a + /// `Server-Timing` header entry. + AuctionWait, + /// Time spent streaming the response body. Row-only: never rendered as + /// a `Server-Timing` header entry. + Stream, +} + +impl Phase { + /// Maps this variant to its fixed slot in the [`Inner::phases`] array. + fn index(self) -> usize { + match self { + Self::AppBuild => 0, + Self::Filter => 1, + Self::Geo => 2, + Self::EcKv => 3, + Self::Origin => 4, + Self::TemplateCacheLookup => 5, + Self::AuctionWait => 6, + Self::Stream => 7, + } + } + + /// `Server-Timing` header entry name; row-only phases return `None`. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +/// Where in the response the auction wait occurred. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + /// The auction was awaited before response headers were sent. + PreHeader, + /// The auction was awaited while streaming the response body. + InStream, +} + +/// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. +struct Inner { + /// Instant the request started; the reference point for elapsed marks. + t0: Instant, + /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. + phases: [Option; PHASE_COUNT], + /// Elapsed time at the first [`RequestTimings::mark_headers_ready`] call. + headers_ready_total: Option, + /// Elapsed time at the first [`RequestTimings::mark_request_elapsed`] + /// call. + request_elapsed: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + resp_bytes: Option, +} + +/// Per-request phase timing collector. +/// +/// Cheap to clone (an [`Arc`] handle) and safe to share across threads and +/// async tasks handling the same request. Every method is infallible: lock +/// contention or poisoning silently drops the sample rather than blocking or +/// panicking. +#[derive(Clone)] +pub struct RequestTimings(Arc>); + +impl RequestTimings { + /// Starts a new collector with its clock reference (`t0`) set to now. + #[must_use] + pub fn new() -> Self { + Self(Arc::new(Mutex::new(Inner { + t0: Instant::now(), + phases: [None; PHASE_COUNT], + headers_ready_total: None, + request_elapsed: None, + auction_wait_placement: None, + resp_bytes: None, + }))) + } + + /// Accumulates `dur` into `phase`'s running total. + /// + /// Repeated calls for the same phase saturate-add rather than overwrite. + /// Drops the sample silently on lock contention or poisoning. + pub fn record(&self, phase: Phase, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = phase.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + } + + /// Records an auction wait duration under [`Phase::AuctionWait`] and + /// stores its placement. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = Phase::AuctionWait.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + inner.auction_wait_placement = Some(placement); + } + + /// Starts a guard that records elapsed time into `phase` when dropped. + #[must_use] + pub fn span(&self, phase: Phase) -> PhaseSpan { + PhaseSpan { + timings: self.clone(), + phase, + started: Instant::now(), + } + } + + /// Stamps the elapsed time since `t0` as `headers_ready_total`, the + /// first time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_headers_ready(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.headers_ready_total.is_none() { + inner.headers_ready_total = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as `request_elapsed`, the first + /// time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_request_elapsed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.request_elapsed.is_none() { + inner.request_elapsed = Some(inner.t0.elapsed()); + } + } + + /// Records the response body size in bytes. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn set_resp_bytes(&self, bytes: u64) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + inner.resp_bytes = Some(bytes); + } + + /// Renders a `Server-Timing` header value, or `None` before + /// [`RequestTimings::mark_headers_ready`] has been called. + /// + /// `ts-total` is rendered first from `headers_ready_total`, followed by + /// the recorded header-bearing phases (`ts-appbuild`, `ts-filter`, + /// `ts-geo`, `ts-kv`, `ts-origin`, `ts-template-cache`) in enum + /// declaration order. Unrecorded phases are omitted. Durations are + /// rendered as milliseconds with one decimal place. Drops the sample + /// silently (returning `None`) on lock contention or poisoning. + #[must_use] + pub fn server_timing_value(&self) -> Option { + let inner = self.0.try_lock().ok()?; + let total = inner.headers_ready_total?; + let mut entries = vec![format_entry("ts-total", total)]; + for phase in HEADER_PHASES { + let Some(name) = phase.header_name() else { + continue; + }; + if let Some(dur) = inner.phases[phase.index()] { + entries.push(format_entry(name, dur)); + } + } + Some(entries.join(", ")) + } + + /// Captures the current state as a [`TimingSnapshot`]. + /// + /// Returns an all-`None` snapshot on lock contention or poisoning, + /// consistent with the infallibility of every other method. + #[must_use] + pub fn snapshot(&self) -> TimingSnapshot { + let Ok(inner) = self.0.try_lock() else { + return TimingSnapshot::default(); + }; + TimingSnapshot { + time_elapsed_ms: duration_ms(inner.headers_ready_total), + request_elapsed_ms: duration_ms(inner.request_elapsed), + appbuild_ms: duration_ms(inner.phases[Phase::AppBuild.index()]), + filter_ms: duration_ms(inner.phases[Phase::Filter.index()]), + geo_ms: duration_ms(inner.phases[Phase::Geo.index()]), + kv_ms: duration_ms(inner.phases[Phase::EcKv.index()]), + origin_ms: duration_ms(inner.phases[Phase::Origin.index()]), + template_cache_ms: duration_ms(inner.phases[Phase::TemplateCacheLookup.index()]), + auction_wait_ms: duration_ms(inner.phases[Phase::AuctionWait.index()]), + stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), + auction_wait_placement: inner.auction_wait_placement, + resp_bytes: inner.resp_bytes, + } + } +} + +impl Default for RequestTimings { + fn default() -> Self { + Self::new() + } +} + +/// The header-bearing phases (see [`Phase::header_name`]), in the enum +/// declaration order [`RequestTimings::server_timing_value`] renders them in. +const HEADER_PHASES: [Phase; 6] = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, +]; + +/// Formats one `Server-Timing` entry as `name;dur=`. +fn format_entry(name: &str, dur: Duration) -> String { + format!("{name};dur={:.1}", dur.as_secs_f64() * 1000.0) +} + +/// Converts a recorded [`Duration`] to whole milliseconds, saturating to +/// [`u32::MAX`] instead of overflowing. +fn duration_ms(dur: Option) -> Option { + dur.map(|dur| u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)) +} + +/// RAII guard returned by [`RequestTimings::span`] that records its own +/// elapsed lifetime into the originating phase when dropped. +pub struct PhaseSpan { + /// The collector this span reports into on drop. + timings: RequestTimings, + /// The phase this span's elapsed time is recorded under. + phase: Phase, + /// The instant the span was created. + started: Instant, +} + +impl Drop for PhaseSpan { + fn drop(&mut self) { + self.timings.record(self.phase, self.started.elapsed()); + } +} + +/// A point-in-time, plain-data view of a [`RequestTimings`] collector. +/// +/// All durations are whole milliseconds; unrecorded phases are `None`. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TimingSnapshot { + /// Elapsed time from request start to + /// [`RequestTimings::mark_headers_ready`], in milliseconds. + pub time_elapsed_ms: Option, + /// Elapsed time from request start to + /// [`RequestTimings::mark_request_elapsed`], in milliseconds. + pub request_elapsed_ms: Option, + /// Accumulated [`Phase::AppBuild`] duration, in milliseconds. + pub appbuild_ms: Option, + /// Accumulated [`Phase::Filter`] duration, in milliseconds. + pub filter_ms: Option, + /// Accumulated [`Phase::Geo`] duration, in milliseconds. + pub geo_ms: Option, + /// Accumulated [`Phase::EcKv`] duration, in milliseconds. + pub kv_ms: Option, + /// Accumulated [`Phase::Origin`] duration, in milliseconds. + pub origin_ms: Option, + /// Accumulated [`Phase::TemplateCacheLookup`] duration, in milliseconds. + pub template_cache_ms: Option, + /// Accumulated [`Phase::AuctionWait`] duration, in milliseconds. + pub auction_wait_ms: Option, + /// Accumulated [`Phase::Stream`] duration, in milliseconds. + pub stream_ms: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + pub auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + pub resp_bytes: Option, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!( + value.contains("ts-filter;dur=9.1"), + "should render one decimal: {value}" + ); + assert!( + !value.contains("ts-geo"), + "should omit unrecorded phases: {value}" + ); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!( + timings.server_timing_value().is_none(), + "should require the snapshot" + ); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!( + timings.snapshot().time_elapsed_ms, + first, + "should not restamp" + ); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + ] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!( + !value.to_ascii_lowercase().contains("datadome"), + "should mask vendors" + ); + } +} From 7505fb888e4bacf86504cf3a73a34c3502cf5d17 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 20:27:41 -0500 Subject: [PATCH 003/230] Add observability settings and decouple tinybird access and auction emission --- .../src/tinybird.rs | 41 ++++- crates/trusted-server-core/src/settings.rs | 154 ++++++++++++++++-- trusted-server.example.toml | 29 ++++ 3 files changed, 205 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index f2df61744..08b5811fc 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -22,9 +22,14 @@ const TINYBIRD_BETWEEN_BYTES_TIMEOUT: Duration = Duration::from_secs(2); const TINYBIRD_MAX_ROWS_PER_AUCTION_BATCH: usize = 512; /// Build the configured auction telemetry sink. +/// +/// Auction emission requires both the Tinybird master toggle +/// (`tinybird.enabled`) and the auction-specific toggle +/// (`tinybird.auction_enabled`), so access-log telemetry can be enabled +/// independently without also emitting auction events. #[must_use] pub(crate) fn auction_sink_from_settings(settings: &Settings) -> Arc { - if settings.tinybird.enabled { + if settings.tinybird.enabled && settings.tinybird.auction_enabled { Arc::new(FastlyTinybirdAuctionTelemetrySink::new( settings.tinybird.clone(), )) @@ -443,6 +448,7 @@ mod tests { fn enabled_config() -> TinybirdSettings { TinybirdSettings { enabled: true, + auction_enabled: true, api_host: "api.us-east.aws.tinybird.co".to_owned(), secret_store: "ts_secrets".to_owned(), auction_dataset: "auction_events_raw".to_owned(), @@ -455,6 +461,39 @@ mod tests { } } + #[test] + fn sink_from_settings_disables_when_auction_enabled_is_false() { + let settings = Settings { + tinybird: TinybirdSettings { + auction_enabled: false, + ..enabled_config() + }, + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + !sink.is_enabled(), + "auction telemetry should stay off when auction_enabled is false, even if tinybird.enabled is true" + ); + } + + #[test] + fn sink_from_settings_enables_when_both_toggles_are_true() { + let settings = Settings { + tinybird: enabled_config(), + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + sink.is_enabled(), + "auction telemetry should be on when both tinybird.enabled and tinybird.auction_enabled are true" + ); + } + #[test] fn events_uri_targets_dataset_on_region_host() { assert_eq!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index ddc8ac612..bcb467fb1 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1704,9 +1704,16 @@ impl Proxy { /// Direct Tinybird Events API telemetry configuration. #[derive(Debug, Clone, Deserialize, Serialize)] pub struct TinybirdSettings { - /// Master enablement for auction telemetry ingestion. + /// Master enablement for Tinybird telemetry. Required by both auction and + /// access-log emission; each is independently toggled below. #[serde(default)] pub enabled: bool, + /// Emit auction telemetry when `enabled`. Defaults to `true` so existing + /// configs preserve their current auction-emission behavior after + /// upgrading; set `false` to silence auction events while keeping + /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). + #[serde(default = "default_true")] + pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] pub api_host: String, @@ -1719,19 +1726,26 @@ pub struct TinybirdSettings { /// Secret key containing the auction datasource APPEND token. #[serde(default = "default_tinybird_auction_token_secret")] pub auction_token_secret: String, - /// Reserved for future access-log telemetry. + /// Emit access-log telemetry when `enabled`, independent of + /// `auction_enabled`. /// - /// `true` is rejected until an access-log emitter is wired, so operators - /// cannot enable a setting that silently emits nothing. + /// `true` requires `enabled`, non-empty `api_host`/`secret_store`/ + /// `access_dataset`/`access_token_secret`, `max_body_bytes > 0`, and + /// `access_sample_rate > 0.0`. This prevents an armed-but-silent sampler + /// that enables the flag but emits nothing. #[serde(default)] pub access_enabled: bool, - /// Future access-log Events API datasource name. + /// Access-log Events API datasource name. Required non-empty when + /// `access_enabled`. #[serde(default = "default_tinybird_access_dataset")] pub access_dataset: String, - /// Future Secret Store key containing the access-log datasource APPEND token. + /// Secret Store key containing the access-log datasource APPEND token. + /// Required non-empty when `access_enabled`. #[serde(default = "default_tinybird_access_token_secret")] pub access_token_secret: String, - /// Future fraction of requests to emit for optional access telemetry. + /// Fraction of requests to emit for access telemetry. Must be greater + /// than `0.0` when `access_enabled`, so an operator cannot enable access + /// telemetry while sampling it away entirely. #[serde(default)] pub access_sample_rate: f64, /// Defensive maximum NDJSON body size for one Events API request. @@ -1767,6 +1781,7 @@ impl Default for TinybirdSettings { fn default() -> Self { Self { enabled: false, + auction_enabled: default_true(), api_host: String::new(), secret_store: default_tinybird_secret_store(), auction_dataset: default_tinybird_auction_dataset(), @@ -1790,6 +1805,12 @@ impl TinybirdSettings { self.access_token_secret = self.access_token_secret.trim().to_owned(); } + /// Validate this settings block, including the access-telemetry matrix: + /// `access_enabled` requires `enabled`, a non-empty `api_host`, + /// `secret_store`, `access_dataset`, and `access_token_secret`, a + /// `max_body_bytes` above the defensive floor enforced below, and an + /// `access_sample_rate` greater than `0.0`. Auction emission is + /// independently gated by `auction_enabled` and validated the same way. fn prepare_runtime(&mut self) -> Result<(), Report> { self.normalize(); if !(0.0..=1.0).contains(&self.access_sample_rate) { @@ -1802,9 +1823,9 @@ impl TinybirdSettings { message: "tinybird.max_body_bytes must be at least 1024".to_owned(), })); } - if self.access_enabled { + if self.access_enabled && !self.enabled { return Err(Report::new(TrustedServerError::Configuration { - message: "tinybird.access_enabled is reserved for future access-log telemetry; no emitter is currently wired".to_owned(), + message: "tinybird.access_enabled requires tinybird.enabled".to_owned(), })); } if !self.enabled { @@ -1818,10 +1839,19 @@ impl TinybirdSettings { .to_owned(), })); } - if self.enabled { + if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; } + if self.access_enabled { + validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; + validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + if self.access_sample_rate <= 0.0 { + return Err(Report::new(TrustedServerError::Configuration { + message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), + })); + } + } Ok(()) } } @@ -2588,6 +2618,29 @@ pub enum AuctionDebugCommentFormat { Pretty, } +/// Request-observability toggles exposed to operators. +/// +/// The default table must stay omitted from serialized config blobs: this +/// struct denies unknown fields, so an older binary loading a config blob +/// carrying an `[observability]` table it does not know would reject it, +/// breaking rollback. See [`Settings::observability`]. +#[derive(Debug, Clone, Default, PartialEq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ObservabilitySettings { + /// Emit the `Server-Timing` response header with per-phase request + /// timing. Defaults to `false` (off). + #[serde(default)] + pub server_timing_enabled: bool, +} + +impl ObservabilitySettings { + /// True when every field is at its default, i.e. observability is fully + /// disabled and the table can be omitted from serialized output. + fn is_default(&self) -> bool { + *self == Self::default() + } +} + /// Tester-cookie endpoint configuration. #[derive(Debug, Default, Clone, Deserialize, Serialize)] pub struct TesterCookieConfig { @@ -2633,6 +2686,12 @@ pub struct Settings { pub tinybird: TinybirdSettings, #[serde(default)] pub debug: DebugConfig, + /// Request-observability toggles. The default table is omitted from + /// serialized config blobs so a config round-tripped without change + /// still parses under a prior binary's schema; see + /// [`ObservabilitySettings`]. + #[serde(default, skip_serializing_if = "ObservabilitySettings::is_default")] + pub observability: ObservabilitySettings, } impl Settings { @@ -3354,6 +3413,14 @@ mod tests { use crate::redacted::Redacted; use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + /// Parses `extra` appended to the shared test fixture TOML, mirroring the + /// `format!("{}\n...", crate_test_settings_str())` pattern used throughout + /// this module's other tests. + fn settings_from_toml_with(extra: &str) -> Result> { + let toml = format!("{}\n{extra}", crate_test_settings_str()); + Settings::from_toml(&toml) + } + #[test] fn auction_debug_comment_options_default_matches_serde_defaults() { let opts = AuctionDebugCommentOptions::default(); @@ -3565,17 +3632,68 @@ mod tests { } #[test] - fn tinybird_access_enabled_is_rejected_until_emitter_is_wired() { - let toml = format!( - "{}\n[tinybird]\naccess_enabled = true\n", - crate_test_settings_str() + fn tinybird_access_enabled_with_full_config_is_accepted() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept a fully-specified access telemetry config"); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" ); + } - let err = Settings::from_toml(&toml) - .expect_err("should reject access telemetry before emitter exists"); + #[test] + fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!( + format!("{err:?}").contains("access_sample_rate"), + "should name the field" + ); + } + + #[test] + fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!( + !settings.tinybird.auction_enabled, + "should disable auction emission" + ); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" + ); + } + + #[test] + fn auction_enabled_defaults_true_for_existing_configs() { + let settings = + settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!( + settings.tinybird.auction_enabled, + "should preserve current behavior" + ); + } + + #[test] + fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!( + !settings.observability.server_timing_enabled, + "should default off" + ); + let toml = toml::to_string(&settings).expect("should serialize settings"); assert!( - format!("{err:?}").contains("tinybird.access_enabled"), - "should report unsupported tinybird.access_enabled setting: {err:?}" + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" ); } diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 71f0f8f78..c52a299b3 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -222,6 +222,35 @@ verbosity = "redacted" # JSON request/response bodies remain strings exactly as captured. format = "compact" +[tinybird] +enabled = false +# Regional Tinybird API host, no scheme/port/path, e.g. "api.us-east.aws.tinybird.co". +api_host = "" +secret_store = "ts_secrets" +auction_dataset = "auction_events_raw" +auction_token_secret = "tinybird_auction_append_token" +# Defaults to true: existing configs keep emitting auction telemetry once +# `enabled` is turned on. Set false to silence auction events while keeping +# `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). +auction_enabled = true +# Access-log telemetry, decoupled from auction emission. `true` requires +# `enabled`, non-empty api_host/secret_store/access_dataset/access_token_secret, +# max_body_bytes > 0, and access_sample_rate > 0.0; an armed-but-silent sampler +# (access_enabled = true with access_sample_rate = 0) fails config load. +access_enabled = false +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +# Fraction (0.0-1.0) of requests to emit for access telemetry when access_enabled. +access_sample_rate = 0.0 +# Defensive maximum NDJSON body size, in bytes, for one Events API request. +max_body_bytes = 1048576 + +[observability] +# Keep this leaf present so the environment override can apply; the overlay +# cannot create a missing configuration leaf. The Server-Timing header stays +# off until enabled. +server_timing_enabled = false + [creative_opportunities] # Set to false to disable server-side ad templates while retaining slot definitions # and direct POST /auction callers. Structurally inactive templates use the From 4ce413ff87621429f731fc3a62a5267e43579c18 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:01:00 -0500 Subject: [PATCH 004/230] Add test coverage for the access_enabled without tinybird.enabled rejection --- crates/trusted-server-core/src/settings.rs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index bcb467fb1..1aecf80a9 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3643,6 +3643,21 @@ mod tests { ); } + #[test] + fn access_enabled_requires_tinybird_enabled() { + // access_enabled = true with tinybird.enabled omitted (defaults + // false) must be rejected: access telemetry cannot run without the + // master toggle on. + let err = settings_from_toml_with( + "[tinybird]\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect_err("should reject access telemetry without tinybird.enabled"); + assert!( + format!("{err:?}").contains("tinybird.access_enabled"), + "should name the field: {err:?}" + ); + } + #[test] fn access_enabled_requires_positive_sample_rate() { // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. From f83092ab2c7acf81ae3bdf0b2c81d97b3c5941e4 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:20:34 -0500 Subject: [PATCH 005/230] Emit Server-Timing at the send freeze point on conclusively private responses --- .../trusted-server-adapter-fastly/src/app.rs | 109 +++++++++++- .../trusted-server-adapter-fastly/src/main.rs | 157 ++++++++++++++++-- 2 files changed, 248 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 41e5e65ee..8f21d89ac 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -1311,7 +1311,9 @@ mod tests { use bytes::Bytes; use edgezero_core::body::Body; use edgezero_core::context::RequestContext; - use edgezero_core::http::{Method, Response, StatusCode, header, request_builder}; + use edgezero_core::http::{ + Method, Response, StatusCode, header, request_builder, response_builder, + }; use edgezero_core::key_value_store::NoopKvStore; use edgezero_core::params::PathParams; use edgezero_core::router::RouterService; @@ -1333,6 +1335,7 @@ mod tests { PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, PlatformSelectResult, RuntimeServices, }; + use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; fn settings_with_missing_consent_store() -> Settings { @@ -2750,4 +2753,108 @@ mod tests { "the filter's response-header effect must be threaded out" ); } + + /// Joins every instance of a response header into one comma-separated + /// string (mirroring how a client sees repeated header fields), or + /// `None` if the header is absent. + fn response_header(response: &Response, name: &str) -> Option { + let values: Vec<&str> = response + .headers() + .get_all(name) + .iter() + .filter_map(|value| value.to_str().ok()) + .collect(); + if values.is_empty() { + None + } else { + Some(values.join(", ")) + } + } + + #[test] + fn server_timing_emitted_on_private_response_when_enabled() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!( + header.contains("ts-total;dur="), + "should carry the stored total: {header}" + ); + assert_eq!( + header.matches("ts-total").count(), + 1, + "should emit exactly one TS-owned metric set" + ); + } + + #[test] + fn server_timing_absent_when_flag_off() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, false); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit server-timing when the flag is off" + ); + } + + #[test] + fn server_timing_absent_on_cacheable_responses() { + // tsjs route policy: public, long max-age, immutable. + let mut tsjs_response = response_builder() + .header("cache-control", "public, max-age=31536000, immutable") + .body(Body::empty()) + .expect("should build a tsjs-style response fixture"); + // A bare shared-cacheable response with no private/no-store directive. + let mut public_response = response_builder() + .header("cache-control", "max-age=60") + .body(Body::empty()) + .expect("should build a bare max-age response fixture"); + + crate::apply_server_timing_header(&mut tsjs_response, &RequestTimings::new(), true); + crate::apply_server_timing_header(&mut public_response, &RequestTimings::new(), true); + + assert!( + response_header(&tsjs_response, "server-timing").is_none(), + "should not emit on the public immutable tsjs cache policy" + ); + assert!( + response_header(&public_response, "server-timing").is_none(), + "should not emit on a bare shared-cacheable max-age response" + ); + } + + #[test] + fn preexisting_server_timing_values_survive() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .header("server-timing", "upstream;dur=1") + .body(Body::empty()) + .expect("should build a private response fixture carrying an upstream Server-Timing"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = + response_header(&response, "server-timing").expect("should still carry a header"); + assert!( + header.contains("upstream;dur=1"), + "should preserve the pre-existing entry: {header}" + ); + assert!( + header.contains("ts-total"), + "should append the TS-owned set: {header}" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index a19d0485d..b04b84fe0 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -5,13 +5,17 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; +use edgezero_core::http::{ + HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, +}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::cache_policy::EdgeCacheHeader; +use trusted_server_core::cache_policy::{ + EdgeCacheHeader, cache_control_headers_are_private_or_no_store, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -24,6 +28,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -48,6 +53,12 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -111,19 +122,27 @@ fn edgezero_main(mut req: FastlyRequest) { return; } - let config_store = match open_trusted_server_config_store() { - Ok(cs) => cs, - Err(e) => { - log::error!("failed to open config store: {e}"); - FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) - .with_body_text_plain("Internal Server Error") - .send_to_client(); - return; - } - }; + let timings = RequestTimings::new(); - let (app, app_state) = TrustedServerApp::build_app_with_state(); + let (config_store, app, app_state) = { + let _appbuild = timings.span(Phase::AppBuild); + let config_store = match open_trusted_server_config_store() { + Ok(cs) => cs, + Err(e) => { + log::error!("failed to open config store: {e}"); + FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) + .with_body_text_plain("Internal Server Error") + .send_to_client(); + return; + } + }; + let (app, app_state) = TrustedServerApp::build_app_with_state(); + (config_store, app, app_state) + }; let settings_snapshot = app_state.as_ref().map(|state| Arc::clone(&state.settings)); + let server_timing_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.observability.server_timing_enabled); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -171,6 +190,7 @@ fn edgezero_main(mut req: FastlyRequest) { core_req.extensions_mut().insert(config_store); core_req.extensions_mut().insert(device_signals); core_req.extensions_mut().insert(client_info); + core_req.extensions_mut().insert(timings.clone()); match futures::executor::block_on(app.router().oneshot(core_req)) { Ok(response) => response, Err(error) => edge_error_response(error), @@ -213,7 +233,14 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); return; } @@ -228,7 +255,14 @@ fn edgezero_main(mut req: FastlyRequest) { Ok(settings) => { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send( &settings, &partner_registry, @@ -250,7 +284,14 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings, + server_timing_enabled, + }, + ); } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -323,6 +364,68 @@ fn run_edgezero_pull_sync_after_send( } } +/// Per-response context threaded into [`send_edgezero_response`] so the +/// function stays at or under seven parameters. +struct SendContext { + /// The request's phase-timing collector. + timings: RequestTimings, + /// Whether `observability.server_timing_enabled` is set. + server_timing_enabled: bool, +} + +/// Outcome of handing a finalized response to the client. +#[allow(dead_code)] +pub(crate) struct DeliveryOutcome { + /// Response body size in bytes. + pub bytes: u64, + /// Whether delivery completed or failed partway. + pub result: DeliveryResult, +} + +/// Whether [`send_edgezero_response`] completed delivery or failed partway. +pub(crate) enum DeliveryResult { + /// The response was handed to the client in full. + Complete, + /// Delivery failed partway through. + Error, +} + +/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is +/// enabled and the response is conclusively private, appends the rendered +/// `Server-Timing` header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +pub(crate) fn apply_server_timing_header( + response: &mut HttpResponse, + timings: &RequestTimings, + server_timing_enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !server_timing_enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -331,8 +434,14 @@ fn run_edgezero_pull_sync_after_send( fn send_edgezero_response( mut response: HttpResponse, request_filter_effects: Option<&RequestFilterEffects>, -) { + context: &SendContext, +) -> DeliveryOutcome { apply_terminal_response_effects(&mut response, request_filter_effects); + apply_server_timing_header( + &mut response, + &context.timings, + context.server_timing_enabled, + ); let (parts, body) = response.into_parts(); @@ -348,15 +457,29 @@ fn send_edgezero_response( if let Err(e) = streaming_body.finish() { log::error!("failed to finish EdgeZero streaming body: {e}"); } + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Complete, + } } Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Error, + } } } } once => { + let bytes = + u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); + DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + } } } } From 4b19c44a3cbb9abda692fd52054293fb0fcb8d56 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:56:02 -0500 Subject: [PATCH 006/230] Record filter and geo spans and dedupe the per-request geo lookup --- .../trusted-server-adapter-fastly/src/app.rs | 342 ++++++++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 33 +- .../src/middleware.rs | 80 +++- crates/trusted-server-core/src/geo.rs | 18 + .../src/integrations/registry.rs | 11 + 5 files changed, 447 insertions(+), 37 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 8f21d89ac..0cf80fb74 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -114,6 +114,7 @@ use trusted_server_core::ec::identify::{cors_preflight_identify, handle_identify use trusted_server_core::ec::kv::KvIdentityGraph; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, @@ -133,6 +134,7 @@ use trusted_server_core::request_signing::{ handle_deactivate_key, handle_rotate_key, handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::{ProxyAssetRoute, Settings}; use trusted_server_core::settings_data::{ default_config_key, default_config_store_name, get_settings_from_config_store, @@ -355,6 +357,17 @@ impl EcRequestState { services: self.services, } } + + /// Derives the carried [`GeoLookupState`] from this request's geo lookup + /// outcome, so response-phase finalize can reuse it instead of repeating + /// the lookup. `build_ec_request_state` always attempts the lookup, so + /// `None` here means the lookup ran and failed, not that it was skipped. + fn geo_lookup_state(&self) -> GeoLookupState { + match &self.geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + } + } } /// Derives device signals from the request's `User-Agent` header. @@ -410,13 +423,21 @@ fn build_ec_request_state( let eids_cookie = crate::extract_cookie_value(req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(req, COOKIE_SHAREDID); - let geo_info = services - .geo() - .lookup(services.client_info().client_ip) - .unwrap_or_else(|e| { - log::warn!("geo lookup failed during EC setup: {e}"); - None - }); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let geo_info = { + let _span = timings.span(Phase::Geo); + services + .geo() + .lookup(services.client_info().client_ip) + .unwrap_or_else(|e| { + log::warn!("geo lookup failed during EC setup: {e}"); + None + }) + }; let (ec_context, setup_error) = match EcContext::read_from_request_with_geo(settings, req, services, geo_info.as_ref()) { @@ -482,6 +503,18 @@ async fn run_pre_route_filters( req: &mut Request, geo_info: Option<&GeoInfo>, ) -> PreRoute { + // Only recorded when a filter is actually registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let _span = state + .registry + .has_request_filters() + .then(|| timings.span(Phase::Filter)); + match state .registry .filter_request(RequestFilterRegistryInput { @@ -515,6 +548,7 @@ fn attach_dispatch_extensions( ec: EcRequestState, effects: RequestFilterEffects, ) -> Response { + response.extensions_mut().insert(ec.geo_lookup_state()); response.extensions_mut().insert(ec.into_finalize_state()); if !effects.response_headers.is_empty() { response.extensions_mut().insert(effects); @@ -810,7 +844,15 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback(state, services, req, asset_route, &effects).await; + return dispatch_asset_fallback( + state, + services, + req, + asset_route, + &effects, + ec.geo_lookup_state(), + ) + .await; } // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -900,7 +942,10 @@ fn asset_response_carries_body(method: &Method, status: StatusCode) -> bool { /// [`AssetProxyCachePolicy`] out via response extensions so `edgezero_main` /// can reapply protected cache directives after finalization. EC finalization /// is intentionally skipped: no [`EcFinalizeState`] is attached, matching the -/// legacy `should_finalize_ec = false` behavior for asset responses. +/// legacy `should_finalize_ec = false` behavior for asset responses. The +/// caller's [`GeoLookupState`] is still attached, since `build_ec_request_state` +/// already attempted the lookup before the asset route was matched — this is +/// the one exit path that carries geo state without an `EcFinalizeState`. /// /// Like legacy `route_request`, asset bodies are streamed straight to the client /// with no cap: the origin stream is attached to the response and `edgezero_main` @@ -915,6 +960,7 @@ async fn dispatch_asset_fallback( req: Request, asset_route: &ProxyAssetRoute, effects: &RequestFilterEffects, + geo_state: GeoLookupState, ) -> Response { log::info!("No explicit route matched; proxying via configured asset route"); @@ -936,6 +982,7 @@ async fn dispatch_asset_fallback( } response.extensions_mut().insert(cache_policy); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -944,6 +991,7 @@ async fn dispatch_asset_fallback( response .extensions_mut() .insert(AssetProxyCachePolicy::NoStorePrivate); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -1319,21 +1367,23 @@ mod tests { use edgezero_core::router::RouterService; use std::net::{IpAddr, Ipv4Addr}; use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; use error_stack::Report; use futures::executor::block_on; use serde_json::json; - use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; + use trusted_server_core::constants::{HEADER_X_GEO_COUNTRY, HEADER_X_GEO_INFO_AVAILABLE}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::error::TrustedServerError; + use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::{ HeaderMutation, IntegrationRegistry, IntegrationRequestFilter, RequestFilterDecision, RequestFilterEffects, RequestFilterInput, }; use trusted_server_core::platform::{ - ClientInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformHttpClient, - PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, - PlatformSelectResult, RuntimeServices, + ClientInfo, GeoInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformGeo, + PlatformHttpClient, PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, + PlatformResponse, PlatformSelectResult, RuntimeServices, }; use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; @@ -1481,19 +1531,19 @@ mod tests { ); } - /// Builds a router whose `AppState` uses a registry containing the given - /// request filters (and no routes), so dispatch-level request-filter - /// behavior can be exercised without a real integration. - fn router_with_request_filters( + /// Builds an `AppState` whose registry contains the given request + /// filters (and no routes), so dispatch-level request-filter behavior can + /// be exercised without a real integration. + fn state_with_request_filters( filters: Vec>, - ) -> RouterService { + ) -> Arc { let settings = test_settings(); let orchestrator = trusted_server_core::auction::build_orchestrator(&settings) .expect("should build orchestrator"); let registry = IntegrationRegistry::from_request_filters(filters); let default_kv_store = Arc::new(crate::platform::UnavailableKvStore) as Arc; - let state = Arc::new(super::AppState { + Arc::new(super::AppState { auction_telemetry_sink: Arc::new( trusted_server_core::auction::NoopAuctionTelemetrySink, ), @@ -1501,8 +1551,15 @@ mod tests { orchestrator: Arc::new(orchestrator), registry: Arc::new(registry), default_kv_store, - }); - TrustedServerApp::routes_for_state(&state) + }) + } + + /// Builds a router on top of [`state_with_request_filters`] so + /// dispatch-level request-filter behavior can be exercised end-to-end. + fn router_with_request_filters( + filters: Vec>, + ) -> RouterService { + TrustedServerApp::routes_for_state(&state_with_request_filters(filters)) } /// Continues routing while mutating the request and emitting a response @@ -2532,6 +2589,61 @@ mod tests { ); } + #[test] + fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // The asset-route fallback is the one exit path that skips + // EcFinalizeState but must still carry GeoLookupState, since + // build_ec_request_state (and its geo lookup) already ran before the + // asset route was matched. Without this, the finalize step would + // silently repeat the lookup for every asset request. + let settings = Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + + [proxy] + + [[proxy.asset_routes]] + prefix = "/.image/" + origin_url = "https://assets.example.com" + "#, + ) + .expect("should parse asset-route settings"); + let state = build_state_from_settings(settings).expect("should build state"); + let router = TrustedServerApp::routes_for_state(&state); + + let response = route(&router, empty_request(Method::GET, "/.image/banner.png")); + + assert!( + response.extensions().get::().is_some(), + "asset-route responses should still carry GeoLookupState even though \ + EC finalization is skipped" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "asset-route responses must skip EC finalization (no EcFinalizeState)" + ); + } + struct FixedBackend; impl PlatformBackend for FixedBackend { @@ -2652,6 +2764,7 @@ mod tests { req, asset_route, &effects, + trusted_server_core::geo::GeoLookupState::NotAttempted, )); assert_eq!( @@ -2695,6 +2808,193 @@ mod tests { ); } + /// A [`PlatformGeo`] stub that counts every `lookup` call and always + /// returns the same canned result, used to prove the request-phase geo + /// lookup is never repeated during finalize. + struct CountingGeo { + calls: Arc, + result: Option, + } + + impl PlatformGeo for CountingGeo { + fn lookup(&self, _: Option) -> Result, Report> { + self.calls.fetch_add(1, Ordering::SeqCst); + Ok(self.result.clone()) + } + } + + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + + fn runtime_services_with_geo(geo: Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(geo) + .client_info(ClientInfo::default()) + .build() + } + + #[test] + fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Dispatching a publisher route runs build_ec_request_state, which + // attempts the geo lookup once and carries the result via + // GeoLookupState. The finalize step (resolve_geo_for_response) must + // reuse that carried value instead of calling the geo backend again. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: Some(sample_geo_info()), + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState"); + assert!( + matches!(carried, GeoLookupState::Resolved(_)), + "a successful lookup should carry Resolved" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not repeat a resolved geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + + let mut response = response; + geo_info + .expect("geo info should have resolved") + .set_response_headers(&mut response); + assert!( + response.headers().get(HEADER_X_GEO_COUNTRY).is_some(), + "x-geo-country should still be set on the response after reusing the carried geo" + ); + } + + #[test] + fn failed_lookup_is_not_retried() { + // When the request-phase lookup fails (returns None), dispatch must + // carry GeoLookupState::Attempted rather than NotAttempted, and + // finalize must not retry it. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: None, + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState even for a failed lookup"); + assert!( + matches!(carried, GeoLookupState::Attempted), + "a failed lookup should carry Attempted, not Resolved or NotAttempted" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not retry a failed geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + assert!( + geo_info.is_none(), + "no geo info should be available after a failed lookup" + ); + } + + #[test] + fn filter_span_recorded_when_request_filter_runs() { + // The Filter phase span should only be recorded when the registry + // actually has a request filter registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let state = state_with_request_filters(vec![Arc::new(RecordingRequestFilter)]); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_some(), + "should record the Filter phase span when a request filter is registered and runs" + ); + } + + #[test] + fn filter_span_not_recorded_when_no_request_filters_registered() { + // Mirror test: an empty registry must never record the Filter span, + // even though run_pre_route_filters still runs (as a no-op loop). + let state = state_with_request_filters(Vec::new()); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_none(), + "should omit the Filter phase span when no request filters are registered" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b04b84fe0..ae9c600fb 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -24,6 +24,7 @@ use trusted_server_core::ec::pull_sync::{ }; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::TrustedServerError; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; @@ -209,14 +210,30 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + let geo_lookup_state = response + .extensions_mut() + .remove::() + .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { if let Some(settings) = settings_snapshot.as_deref() { - apply_entry_point_finalize_headers(settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } else { match load_settings_from_config_store() { Ok(settings) => { - apply_entry_point_finalize_headers(&settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + &settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } Err(e) => { log::warn!("entry-point finalize skipped: failed to reload settings: {e:?}"); @@ -319,8 +336,11 @@ fn apply_entry_point_finalize_headers( settings: &Settings, response: &mut HttpResponse, client_ip: Option, + geo_state: &GeoLookupState, + timings: &RequestTimings, ) { - let geo_info = resolve_geo_for_response(response, client_ip, |client_ip| { + let geo_info = resolve_geo_for_response(response, geo_state, client_ip, |client_ip| { + let _span = timings.span(Phase::Geo); FastlyPlatformGeo.lookup(client_ip).unwrap_or_else(|e| { log::warn!("entry-point geo lookup failed: {e}"); None @@ -891,9 +911,10 @@ mod tests { .body(EdgeBody::empty()) .expect("should build response"); - let geo_info = resolve_geo_for_response(&response, None, |_| { - panic!("should skip entry-point geo lookup for 401 responses"); - }); + let geo_info = + resolve_geo_for_response(&response, &GeoLookupState::NotAttempted, None, |_| { + panic!("should skip entry-point geo lookup for 401 responses"); + }); apply_finalize_headers(&settings, geo_info.as_ref(), &mut response); assert_eq!( diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 18f309c68..17ecc13a4 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -24,8 +24,9 @@ use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_SERVICE_VERSION, HEADER_X_GEO_INFO_AVAILABLE, HEADER_X_TS_ENV, HEADER_X_TS_VERSION, }; -use trusted_server_core::geo::GeoInfo; +use trusted_server_core::geo::{GeoInfo, GeoLookupState}; use trusted_server_core::platform::PlatformGeo; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::Settings; pub(crate) const HEADER_X_TS_FINALIZED: &str = "x-ts-finalized"; @@ -68,6 +69,12 @@ impl FinalizeResponseMiddleware { impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let client_ip = FastlyRequestContext::get(ctx.request()).and_then(|c| c.client_ip); + let timings = ctx + .request() + .extensions() + .get::() + .cloned() + .unwrap_or_default(); let mut response = match next.run(ctx).await { Ok(r) => r, @@ -77,7 +84,13 @@ impl Middleware for FinalizeResponseMiddleware { } }; - let geo_info = resolve_geo_for_response(&response, client_ip, |ip| { + let carried = response + .extensions() + .get::() + .cloned() + .unwrap_or(GeoLookupState::NotAttempted); + let geo_info = resolve_geo_for_response(&response, &carried, client_ip, |ip| { + let _span = timings.span(Phase::Geo); self.geo.lookup(ip).unwrap_or_else(|e| { log::warn!("geo lookup failed: {e}"); None @@ -142,14 +155,20 @@ impl Middleware for AuthMiddleware { // Shared geo resolution helper // --------------------------------------------------------------------------- -/// Resolves geo for a response, skipping the lookup for 401 responses. +/// Resolves geo for a response, skipping the lookup for 401 responses and +/// reusing a request-phase lookup when one was already carried. /// -/// Returns `None` for authentication rejections (401) without calling `lookup_geo` -/// to avoid unnecessary work and exposing geo data to unauthenticated callers. -/// All other responses call `lookup_geo` and return its result. +/// Returns `None` for authentication rejections (401) without consulting +/// `carried` or calling `lookup_geo`, to avoid unnecessary work and exposing +/// geo data to unauthenticated callers. Otherwise dispatches on `carried`: +/// a [`GeoLookupState::Resolved`] value is reused as-is, a +/// [`GeoLookupState::Attempted`] value is treated as no geo info without +/// retrying the lookup, and [`GeoLookupState::NotAttempted`] falls back to +/// calling `lookup_geo`. /// /// Used by both [`FinalizeResponseMiddleware`] and the entry-point finalization -/// in `main.rs` so the 401-skip rule is defined in one place. +/// in `main.rs` so the 401-skip rule and the dedupe rule are each defined in +/// one place. /// /// # Parity note /// @@ -161,6 +180,7 @@ impl Middleware for AuthMiddleware { /// server or the upstream origin. pub(crate) fn resolve_geo_for_response( response: &Response, + carried: &GeoLookupState, client_ip: Option, lookup_geo: F, ) -> Option @@ -168,9 +188,12 @@ where F: FnOnce(Option) -> Option, { if response.status() == StatusCode::UNAUTHORIZED { - None - } else { - lookup_geo(client_ip) + return None; + } + match carried { + GeoLookupState::Resolved(geo) => Some(geo.clone()), + GeoLookupState::Attempted => None, + GeoLookupState::NotAttempted => lookup_geo(client_ip), } } @@ -277,6 +300,19 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + struct FixedGeo(Option); impl PlatformGeo for FixedGeo { @@ -639,6 +675,30 @@ mod tests { ); } + #[test] + #[allow(clippy::panic)] + fn geo_lookup_skipped_for_unauthorized_responses() { + // The 401 short-circuit in resolve_geo_for_response must win + // regardless of what state the request phase carried in, and must + // never invoke the fallback lookup closure. + let mut response = empty_response(); + *response.status_mut() = StatusCode::UNAUTHORIZED; + + for carried in [ + GeoLookupState::NotAttempted, + GeoLookupState::Attempted, + GeoLookupState::Resolved(sample_geo_info()), + ] { + let geo_info = resolve_geo_for_response(&response, &carried, None, |_| { + panic!("401 responses must never trigger a geo lookup"); + }); + assert!( + geo_info.is_none(), + "401 responses should never resolve geo info, regardless of carried state" + ); + } + } + // --------------------------------------------------------------------------- // AuthMiddleware::handle tests // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-core/src/geo.rs b/crates/trusted-server-core/src/geo.rs index 63f7907f5..fe5785d26 100644 --- a/crates/trusted-server-core/src/geo.rs +++ b/crates/trusted-server-core/src/geo.rs @@ -48,6 +48,24 @@ impl GeoInfo { } } +/// Carries the outcome of a request-phase geo lookup across to +/// response-phase finalization, so a finalize consumer can reuse it instead +/// of performing a second lookup for the same request. +/// +/// Attached as a response extension on every exit path that attempted a +/// lookup, including the asset-route fallback (which does not carry an EC +/// finalize state). +#[derive(Debug, Clone)] +pub enum GeoLookupState { + /// No lookup has been attempted for this request. + NotAttempted, + /// A lookup ran and failed (or returned no result). This must not be + /// retried: finalize treats it the same as no geo info being available. + Attempted, + /// A lookup ran and resolved geo info. + Resolved(GeoInfo), +} + fn insert_geo_header(headers: &mut http::HeaderMap, name: http::header::HeaderName, value: &str) { match HeaderValue::from_str(value) { Ok(header_value) => { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 280eae847..376f5e492 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -893,6 +893,17 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// Return true when at least one integration request filter is + /// registered. + /// + /// Adapters use this to decide whether to record a request-filter phase + /// timing span, so unconfigured deployments (no request filters) omit + /// that entry from observability output entirely. + #[must_use] + pub fn has_request_filters(&self) -> bool { + !self.inner.request_filters.is_empty() + } + /// Run pre-routing request filters. /// /// Request header mutations are applied immediately so later filters and From ce295f1405a0f5be2ee7363bd4f6ba7f3fc94587 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 22:37:53 -0600 Subject: [PATCH 007/230] Record origin, template cache, and KV phase spans in core --- .../trusted-server-adapter-fastly/src/app.rs | 132 +++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 184 +++++++++++++++++- crates/trusted-server-core/src/ec/kv.rs | 22 +++ .../trusted-server-core/src/platform/mod.rs | 2 + .../src/platform/timed_kv.rs | 180 +++++++++++++++++ crates/trusted-server-core/src/publisher.rs | 99 +++++++++- 6 files changed, 595 insertions(+), 24 deletions(-) create mode 100644 crates/trusted-server-core/src/platform/timed_kv.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0cf80fb74..5cf7f9709 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -120,7 +120,9 @@ use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, RequestFilterRegistryOutcome, }; -use trusted_server_core::platform::{ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices, TimedKvStore, +}; use trusted_server_core::proxy::{ AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign, @@ -222,13 +224,18 @@ fn warn_if_certificate_check_disabled(settings: &Settings) { pub(crate) fn runtime_services_for_consent_route( settings: &Settings, runtime_services: &RuntimeServices, + timings: &RequestTimings, ) -> Result> { let Some(store_name) = settings.consent.consent_store.as_deref() else { return Ok(runtime_services.clone()); }; open_kv_store(store_name) - .map(|store| runtime_services.clone().with_kv_store(store)) + .map(|store| { + let timed_store = + Arc::new(TimedKvStore::new(store, timings.clone())) as Arc; + runtime_services.clone().with_kv_store(timed_store) + }) .map_err(|e| { Report::new(TrustedServerError::KvStore { store_name: store_name.to_string(), @@ -451,7 +458,7 @@ fn build_ec_request_state( // Bot gate: suppress KV-backed EC writes for unrecognized clients, except // consent withdrawals. Revocations keep the write path so tombstones stay // authoritative even for privacy-extension-heavy clients. - let kv_graph = crate::maybe_identity_graph(settings); + let kv_graph = crate::identity_graph_with_timing(settings, &timings); let finalize_kv_graph = if setup_error.is_none() && (is_real_browser || ec_consent_withdrawn(ec_context.consent())) { @@ -585,7 +592,12 @@ async fn execute_named( // Deliberately do not use an EC request-state graph: that // copy is bot-gated, while operators use curl for this // authenticated diagnostic. - let kv = crate::maybe_identity_graph(&state.settings); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::identity_graph_with_timing(&state.settings, &timings); handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) } NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), @@ -656,7 +668,12 @@ async fn run_named_route( if req.method() == Method::OPTIONS { cors_preflight_identify(&state.settings, &req) } else { - let kv = crate::require_identity_graph(&state.settings)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::require_identity_graph_with_timing(&state.settings, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; handle_identify( &state.settings, @@ -673,7 +690,13 @@ async fn run_named_route( // The auction reads consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but // cannot be opened, matching legacy behavior. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -701,7 +724,13 @@ async fn run_named_route( // Like the auction, page-bids reads consent data, so the consent KV // store must be available — fail closed with 503 when configured but // unopenable, matching legacy. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -746,12 +775,18 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> let is_real_browser = device_signals.looks_like_browser(); let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); - let result = crate::require_identity_graph(&state.settings).and_then(|kv| { - let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; - let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - handle_batch_sync(&kv, &partner_registry, &limiter, req) - }); + let result = + crate::require_identity_graph_with_timing(&state.settings, &timings).and_then(|kv| { + let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; + let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); + handle_batch_sync(&kv, &partner_registry, &limiter, req) + }); let mut response = result.unwrap_or_else(|e| http_error(&e)); // Legacy parity: batch-sync responses still pass through @@ -870,7 +905,12 @@ async fn dispatch_fallback( // Publisher pages read consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but cannot // be opened, matching legacy behavior. - match runtime_services_for_consent_route(&state.settings, services) { + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + match runtime_services_for_consent_route(&state.settings, services, &timings) { Ok(publisher_services) => { // Run the server-side auction with the configured creative- // opportunity slots and collect dispatched bids from the lazy @@ -2995,6 +3035,72 @@ mod tests { ); } + fn settings_with_consent_and_ec_store() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [consent] + consent_store = "consent_store" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse settings with consent and EC KV stores configured") + } + + #[test] + fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent-store access threaded through RuntimeServices uses the same + // TimedKvStore decorator as request-path KvIdentityGraph + // construction, so a read through it records Phase::EcKv. + let settings = settings_with_consent_and_ec_store(); + let services = streaming_runtime_services(); + let timings = RequestTimings::new(); + + let consent_services = + super::runtime_services_for_consent_route(&settings, &services, &timings) + .expect("should open the configured consent store"); + let _ = block_on(consent_services.kv_store().get_bytes("consent-read-key")); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "a consent-store read through the decorated RuntimeServices store should record Phase::EcKv" + ); + + // Pull-sync's identity graph is built by `require_identity_graph`, + // which takes no `timings` parameter at all — the untimed store it + // constructs cannot record into any handle, including a fresh one. + let graph = crate::require_identity_graph(&settings) + .expect("should construct the pull-sync identity graph"); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let _ = graph.get(ec_id); + + let pull_sync_timings = RequestTimings::new(); + pull_sync_timings.mark_headers_ready(); + assert!( + pull_sync_timings.snapshot().kv_ms.is_none(), + "pull-sync's untimed graph construction has no timings handle to record into" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ae9c600fb..ca97f6475 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -27,7 +27,7 @@ use trusted_server_core::error::TrustedServerError; use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; -use trusted_server_core::platform::RuntimeServices; +use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; @@ -248,7 +248,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(ec_state) = ec_state { if let Some(settings) = settings_snapshot.as_deref() { - match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { send_edgezero_response( response, @@ -270,7 +270,8 @@ fn edgezero_main(mut req: FastlyRequest) { } else { match load_settings_from_config_store() { Ok(settings) => { - match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + { Ok(partner_registry) => { send_edgezero_response( response, @@ -353,10 +354,11 @@ fn apply_edgezero_ec_finalize( settings: &Settings, ec_state: &EcFinalizeState, response: &mut HttpResponse, + timings: &RequestTimings, ) -> Result> { let partner_registry = PartnerRegistry::from_config(&settings.ec.partners)?; let finalize_kv_graph = if ec_state.use_finalize_kv { - maybe_identity_graph(settings) + identity_graph_with_timing(settings, timings) } else { None }; @@ -575,12 +577,21 @@ fn build_ja4_debug_response(req: &FastlyRequest) -> FastlyResponse { .with_body(body) } -pub(crate) fn maybe_identity_graph(settings: &Settings) -> Option { - settings - .ec - .ec_store - .as_ref() - .map(|store_name| KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, for request-path callers with a `RequestTimings` handle. +/// +/// Returns `None` when `ec.ec_store` is not configured, matching +/// [`require_identity_graph_with_timing`]'s contract on every other axis. +pub(crate) fn identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Option { + settings.ec.ec_store.as_ref().map(|store_name| { + KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + )) + }) } fn run_pull_sync_after_send( @@ -603,6 +614,12 @@ fn run_pull_sync_after_send( /// Constructs a `KvIdentityGraph` from settings, or returns an error if the /// `ec_store` config is not set. +/// +/// Deliberately untimed: pull-sync (this function's only caller) runs after +/// `send_edgezero_response`'s Server-Timing freeze point, so a decorated +/// store here would record into a handle nothing ever renders. +/// Request-path callers with a `RequestTimings` handle use +/// [`require_identity_graph_with_timing`] instead. pub(crate) fn require_identity_graph( settings: &Settings, ) -> Result> { @@ -615,6 +632,27 @@ pub(crate) fn require_identity_graph( Ok(KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) } +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, or returns an error if the `ec_store` config is not set. +/// +/// Request-path sibling of [`require_identity_graph`], which pull-sync uses +/// unwrapped because pull-sync runs after the Server-Timing freeze point. +pub(crate) fn require_identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Result> { + let store_name = settings.ec.ec_store.as_deref().ok_or_else(|| { + Report::new(TrustedServerError::KvStore { + store_name: "ec.ec_store".to_owned(), + message: "ec.ec_store is not configured".to_owned(), + }) + })?; + Ok(KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + ))) +} + /// Extracts a named cookie value from the request's `Cookie` header. pub(crate) fn extract_cookie_value(req: &HttpRequest, name: &str) -> Option { let cookie_header = req.headers().get("cookie").and_then(|v| v.to_str().ok())?; @@ -644,6 +682,7 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { use super::*; + use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; @@ -980,4 +1019,129 @@ mod tests { "should include sec-ch-ua-platform fallback" ); } + + fn ec_finalize_settings() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [[ec.partners]] + name = "Example Partner" + source_domain = "example.com" + api_token = "test-vendor-token-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse EC finalize test settings") + } + + /// Minimal `RuntimeServices` for `EcFinalizeState.services`. Real + /// `FastlyPlatform*` handles are used as inert placeholders: EC + /// finalization never calls through them, it only satisfies the field. + fn inert_runtime_services() -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore) + as Arc) + .backend(Arc::new(crate::platform::FastlyPlatformBackend)) + .http_client(Arc::new(crate::platform::FastlyPlatformHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(trusted_server_core::platform::ClientInfo::default()) + .build() + } + + #[test] + fn ec_finalize_kv_lands_before_freeze() { + // A pre-seeded EC entry (see fastly.toml's ec_identity_store fixture) + // for a returning user carrying an eids cookie that matches the + // configured partner. This drives ec_finalize_response into + // ingest_eid_cookies, which reads and writes the KV identity graph. + let settings = ec_finalize_settings(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let eids = serde_json::json!([{ + "source": "example.com", + "uids": [{ "id": "example-uid", "atype": 1 }] + }]); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode(eids.to_string()); + let request = edgezero_core::http::request_builder() + .method(fastly::http::Method::GET) + .uri("https://test-publisher.com/article") + .header("cookie", format!("ts-ec={ec_id}; ts-eids={eids_cookie}")) + .body(EdgeBody::empty()) + .expect("should build EC finalize test request"); + + let services = inert_runtime_services(); + let geo_info = trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }; + let ec_context = trusted_server_core::ec::EcContext::read_from_request_with_geo( + &settings, + &request, + &services, + Some(&geo_info), + ) + .expect("should read EC context from a non-regulated request"); + assert!( + ec_context.ec_was_present(), + "the pre-seeded ts-ec cookie should be recognized" + ); + + let ec_state = EcFinalizeState { + ec_context, + use_finalize_kv: true, + eids_cookie: Some(eids_cookie), + sharedid_cookie: None, + is_real_browser: true, + services, + }; + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(EdgeBody::empty()) + .expect("should build EC finalize response fixture"); + let timings = RequestTimings::new(); + + // Mirrors edgezero_main's ordering: EC finalize runs, then the freeze + // point (apply_server_timing_header, called just before + // response.into_parts() inside send_edgezero_response) renders the + // header. Calling both directly exercises exactly this order without + // requiring a live Fastly client connection. + apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + .expect("should finalize EC response"); + apply_server_timing_header(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|v| v.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.contains("ts-kv"), + "the freeze point must run after EC finalization recorded KV time: {header}" + ); + } } diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 3572581ce..dc9885fde 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -810,6 +810,28 @@ mod tests { assert!(ts > 0, "should return a nonzero timestamp"); } + #[test] + fn kv_span_accumulates_across_graph_operations() { + let timings = crate::request_timing::RequestTimings::new(); + let graph = KvIdentityGraph::new(crate::platform::TimedKvStore::new( + crate::ec::kv_backend::test_support::InMemoryEcKv::new("test-store"), + timings.clone(), + )); + + graph + .create("ec-1", &live_entry()) + .expect("should create entry through the timed store"); + graph + .get("ec-1") + .expect("should read the entry back through the timed store"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should accumulate Phase::EcKv across both graph operations, not just the last write" + ); + } + #[test] fn serialize_entry_produces_valid_json() { let entry = KvEntry::tombstone(1000); diff --git a/crates/trusted-server-core/src/platform/mod.rs b/crates/trusted-server-core/src/platform/mod.rs index 1c5bf4c2a..e6ed3ef90 100644 --- a/crates/trusted-server-core/src/platform/mod.rs +++ b/crates/trusted-server-core/src/platform/mod.rs @@ -42,6 +42,7 @@ mod template_assembly; mod template_cache; #[cfg(test)] pub(crate) mod test_support; +mod timed_kv; mod traits; mod types; @@ -67,6 +68,7 @@ pub use template_cache::{ TemplateEntry, TemplateMetadata, TemplateMetadataEncodeError, UnavailableTemplateCache, VaryHeaderValues, VarySpec, }; +pub use timed_kv::TimedKvStore; pub use traits::{PlatformBackend, PlatformConfigStore, PlatformGeo, PlatformSecretStore}; pub use types::{ ClientInfo, GeoInfo, PlatformBackendSpec, RuntimeServices, RuntimeServicesBuilder, StoreId, diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs new file mode 100644 index 000000000..0594b7cef --- /dev/null +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -0,0 +1,180 @@ +//! Latency-only timing decorator for KV store handles. +//! +//! [`TimedKvStore`] wraps an inner store plus a [`RequestTimings`] handle and +//! records [`Phase::EcKv`] around every call. It implements both +//! [`PlatformKvStore`] (for consent-store access obtained through +//! [`RuntimeServices`](super::RuntimeServices)) and [`EcKvStore`] (for +//! [`KvIdentityGraph`](crate::ec::kv::KvIdentityGraph) construction sites), +//! because no single existing abstraction covers the whole `ts-kv` taxonomy: +//! EC graph operations go through [`EcKvStore`] while consent persistence +//! uses [`PlatformKvStore`] directly. +//! +//! The decorator measures store-call latency only: it never reads, parses, +//! or logs any value passing through it. + +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; +use edgezero_core::key_value_store::{KvError, KvPage, KvStore as PlatformKvStore}; +use error_stack::Report; + +use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteOutcome}; +use crate::error::TrustedServerError; +use crate::request_timing::{Phase, RequestTimings}; + +/// Wraps `inner` plus a [`RequestTimings`] handle, recording [`Phase::EcKv`] +/// around every store call made through it. +pub struct TimedKvStore { + /// The wrapped store handle. + inner: S, + /// The request's phase-timing collector. + timings: RequestTimings, +} + +impl TimedKvStore { + /// Creates a decorator around `inner` that records into `timings`. + #[must_use] + pub fn new(inner: S, timings: RequestTimings) -> Self { + Self { inner, timings } + } +} + +#[async_trait(?Send)] +impl PlatformKvStore for TimedKvStore> { + async fn get_bytes(&self, key: &str) -> Result, KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.get_bytes(key).await + } + + async fn put_bytes(&self, key: &str, value: Bytes) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes(key, value).await + } + + async fn put_bytes_with_ttl( + &self, + key: &str, + value: Bytes, + ttl: Duration, + ) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes_with_ttl(key, value, ttl).await + } + + async fn delete(&self, key: &str) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key).await + } + + async fn list_keys_page( + &self, + prefix: &str, + cursor: Option<&str>, + limit: usize, + ) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.list_keys_page(prefix, cursor, limit).await + } +} + +impl EcKvStore for TimedKvStore { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.lookup(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.insert(key, write) + } + + fn count_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.count_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key) + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration as StdDuration; + + use super::*; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + + #[test] + fn ec_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + store + .insert( + "key-a", + EcKvWrite { + body: "{}", + metadata: "{}", + ttl: StdDuration::from_secs(60), + mode: crate::ec::kv_backend::EcKvWriteMode::Add, + }, + ) + .expect("should insert into the in-memory store"); + store.lookup("key-a").expect("should read back the entry"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv across both store calls" + ); + } + + #[test] + fn store_name_is_not_timed() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + assert_eq!(store.store_name(), "test-store"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_none(), + "store_name is a metadata accessor, not a store operation" + ); + } + + #[test] + fn platform_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(crate::platform::UnavailableKvStore); + let store = TimedKvStore::new(inner, timings.clone()); + + // UnavailableKvStore errors on every call; the decorator still times + // the attempt regardless of outcome. + futures::executor::block_on(async { + let _ = store.get_bytes("key").await; + let _ = store.put_bytes("key", Bytes::from_static(b"value")).await; + }); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv even when the inner store errors" + ); + } +} diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index b0d63b82a..07d6704a7 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,6 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; +use crate::request_timing::{Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -4044,6 +4045,14 @@ pub async fn handle_publisher_request( ) -> Result> { log::debug!("Proxying request to publisher_origin"); + // A defaulted handle records into nothing that ever renders, so tests + // that don't populate the request extension are unaffected. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + // Adapter fallbacks prepare this before EC/cookie handling. Keep this // idempotent call as a direct-handler safety net and for focused tests. let gpt_diagnostics = @@ -4476,7 +4485,10 @@ pub async fn handle_publisher_request( // not be served a shared template even if that template is perfectly cacheable. let mut template_cache_reservation = None; if let Some(key) = template_cache_key.as_ref() { - match services.template_cache().lookup_or_reserve(key).await { + let template_cache_span = timings.span(Phase::TemplateCacheLookup); + let template_cache_lookup = services.template_cache().lookup_or_reserve(key).await; + drop(template_cache_span); + match template_cache_lookup { Ok(crate::platform::TemplateCacheLookup::Hit(entry)) => { log::debug!("template_cache hit: {} bytes", entry.body.len()); @@ -4577,6 +4589,7 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + let origin_span = timings.span(Phase::Origin); let mut response = match services.http_client().send(platform_request).await { Ok(platform_response) => platform_response.response, Err(err) => { @@ -4594,6 +4607,7 @@ pub async fn handle_publisher_request( })); } }; + drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7737,6 +7751,35 @@ mod tests { .expect("should proxy publisher request") } + #[tokio::test] + async fn origin_span_covers_the_publisher_fetch() { + let settings = create_test_settings(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run_publisher_proxy(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin phase span around the publisher fetch" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! @@ -8973,6 +9016,60 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode: no shared-cache key is ever computed, so the lookup + // never runs and the span is never recorded. + let inline_settings = create_test_settings(); + let inline_stub = Arc::new(StubHttpClient::new()); + inline_stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let inline_services = build_services_with_http_client( + inline_stub as Arc, + ); + let mut inline_request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let inline_timings = RequestTimings::new(); + inline_request + .extensions_mut() + .insert(inline_timings.clone()); + + let _inline_response = + run_publisher_proxy(&inline_settings, &inline_services, inline_request).await; + + inline_timings.mark_headers_ready(); + assert!( + inline_timings.snapshot().template_cache_ms.is_none(), + "inline mode should never run the template-cache lookup" + ); + + // Shared-mode eligible: a matched slot plus a template-cache-eligible + // assembly mode compute a cache key, so the lookup runs and is timed. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + let mut request = navigation_request(); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().template_cache_ms.is_some(), + "a shared-cache-eligible request should record the TemplateCacheLookup span" + ); + } + #[tokio::test] async fn only_the_cold_miss_uses_the_platform_assembler() { let stub = Arc::new(StubHttpClient::new()); From c222ae280931999fdd822473581ac22a632cab27 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 00:57:06 -0700 Subject: [PATCH 008/230] Capture stream duration, auction wait placement, and response bytes --- .../trusted-server-adapter-fastly/src/main.rs | 269 ++++++++++++++++-- crates/trusted-server-core/src/publisher.rs | 210 +++++++++++++- 2 files changed, 462 insertions(+), 17 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ca97f6475..5747dc8fd 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,5 @@ use std::sync::Arc; +use std::time::Instant; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -405,10 +406,15 @@ pub(crate) struct DeliveryOutcome { } /// Whether [`send_edgezero_response`] completed delivery or failed partway. +#[derive(Debug, PartialEq, Eq)] pub(crate) enum DeliveryResult { /// The response was handed to the client in full. Complete, - /// Delivery failed partway through. + /// Delivery started but did not finish cleanly: some bytes reached the + /// client's transport before a stream error, or the transport could not + /// be closed cleanly after every byte was written. + Partial, + /// Delivery failed before any bytes reached the client. Error, } @@ -448,6 +454,93 @@ pub(crate) fn apply_server_timing_header( } } +/// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written +/// to the inner writer. +/// +/// Wraps the client transport during a streaming drive so a truncated or +/// failed drive still reports how many bytes actually reached it, instead of +/// the placeholder `0` a failed/aborted drive would otherwise report. +struct CountingWriter { + inner: W, + bytes: u64, +} + +impl CountingWriter { + fn new(inner: W) -> Self { + Self { inner, bytes: 0 } + } + + /// Bytes successfully written to the inner writer so far. + fn bytes(&self) -> u64 { + self.bytes + } + + fn into_inner(self) -> W { + self.inner + } +} + +impl std::io::Write for CountingWriter { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + let written = self.inner.write(buf)?; + self.bytes = self.bytes.saturating_add(written as u64); + Ok(written) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +/// Drives a streaming `EdgeZero` body through `output`, tallying bytes written +/// and timing the drive into `timings`. +/// +/// Stamps `resp_bytes` and `request_elapsed` immediately once the drive +/// returns — before the caller does anything transport-specific (finishing +/// the streaming body, logging) — so `request_elapsed` never includes that +/// work. Returns the counting writer (so the caller can recover both the +/// tallied byte count and the wrapped transport) alongside the drive's +/// result. +fn drive_streaming_body( + body: EdgeBody, + output: W, + timings: &RequestTimings, +) -> (CountingWriter, Result<(), Report>) { + let mut counting = CountingWriter::new(output); + let drive_started = Instant::now(); + let result = futures::executor::block_on(stream_asset_body(body, &mut counting)); + timings.record(Phase::Stream, drive_started.elapsed()); + timings.set_resp_bytes(counting.bytes()); + timings.mark_request_elapsed(); + (counting, result) +} + +/// Classifies a completed streaming drive into a [`DeliveryResult`]. +/// +/// A drive that failed after writing at least one byte delivered a truncated +/// response rather than nothing at all, so it is [`DeliveryResult::Partial`], +/// not [`DeliveryResult::Error`]. +fn classify_stream_delivery( + drive_result: &Result<(), Report>, + bytes: u64, +) -> DeliveryResult { + match drive_result { + Ok(()) => DeliveryResult::Complete, + Err(_) if bytes > 0 => DeliveryResult::Partial, + Err(_) => DeliveryResult::Error, + } +} + +/// Stamps `resp_bytes`/`request_elapsed` for an already-materialized body, +/// immediately before it is handed to the Fastly client transport, and +/// returns its byte length. +fn record_buffered_delivery(body: &EdgeBody, timings: &RequestTimings) -> u64 { + let bytes = u64::try_from(body.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + timings.set_resp_bytes(bytes); + timings.mark_request_elapsed(); + bytes +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -473,30 +566,40 @@ fn send_edgezero_response( parts, EdgeBody::empty(), )); - let mut streaming_body = skeleton.stream_to_client(); - match futures::executor::block_on(stream_asset_body(body, &mut streaming_body)) { - Ok(()) => { - if let Err(e) = streaming_body.finish() { - log::error!("failed to finish EdgeZero streaming body: {e}"); - } - DeliveryOutcome { - bytes: 0, + let (counting, drive_result) = + drive_streaming_body(body, skeleton.stream_to_client(), &context.timings); + let bytes = counting.bytes(); + let streaming_body = counting.into_inner(); + // Computed before `drive_result` is matched by value below, since + // the `Err` arm there moves its `Report` out. + let result = classify_stream_delivery(&drive_result, bytes); + match drive_result { + Ok(()) => match streaming_body.finish() { + Ok(()) => DeliveryOutcome { + bytes, result: DeliveryResult::Complete, + }, + Err(e) => { + // Every byte was handed to the transport (the drive + // above returned Ok), but the transport itself could + // not close cleanly — the client may still see a + // truncated response. + log::error!("failed to finish EdgeZero streaming body: {e}"); + DeliveryOutcome { + bytes, + result: DeliveryResult::Partial, + } } - } + }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { - bytes: 0, - result: DeliveryResult::Error, - } + DeliveryOutcome { bytes, result } } } } once => { - let bytes = - u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + let bytes = record_buffered_delivery(&once, &context.timings); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); DeliveryOutcome { bytes, @@ -687,7 +790,9 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; + use trusted_server_core::request_timing::AuctionWaitPlacement; fn test_settings() -> Settings { Settings::from_toml( @@ -1144,4 +1249,136 @@ mod tests { "the freeze point must run after EC finalization recorded KV time: {header}" ); } + + #[test] + fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + let timings = RequestTimings::new(); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello "), + bytes::Bytes::from_static(b"world"), + ])); + + let (counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + let bytes = counting.bytes(); + let outcome = DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + }; + + assert_eq!( + counting.into_inner(), + b"hello world", + "should write every byte to the underlying transport" + ); + assert_eq!( + outcome.bytes, + "hello world".len() as u64, + "DeliveryOutcome.bytes should equal the streamed body length" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("hello world".len() as u64), + "should stamp resp_bytes to the tallied byte count" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed once the drive returns" + ); + } + + #[test] + fn buffered_delivery_stamps_bytes_and_request_elapsed() { + let timings = RequestTimings::new(); + let body = EdgeBody::from(b"a buffered body".to_vec()); + + let bytes = record_buffered_delivery(&body, &timings); + + assert_eq!( + bytes, + "a buffered body".len() as u64, + "should report the buffered body length" + ); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("a buffered body".len() as u64), + "should stamp resp_bytes for the buffered path too" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed for the buffered path too" + ); + } + + #[test] + fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { + // A streaming seam wait (Task 6, publisher.rs) records into the same + // `RequestTimings` handle the adapter drives with. `Phase::Stream` + // wraps the entire drive, so it must cover — and therefore be at + // least as large as — any `AuctionWait` recorded while the body was + // being polled. + let timings = RequestTimings::new(); + let wait_timings = timings.clone(); + let stream = futures::stream::once(async move { + let waited = Duration::from_millis(5); + std::thread::sleep(waited); + wait_timings.record_auction_wait(AuctionWaitPlacement::InStream, waited); + bytes::Bytes::from_static(b"") + }); + let body = EdgeBody::stream(stream); + + let (_counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "should preserve the placement recorded from inside the polled body" + ); + let auction_wait_ms = snapshot + .auction_wait_ms + .expect("should record the auction wait"); + let stream_ms = snapshot.stream_ms.expect("should record the stream drive"); + assert!( + stream_ms >= auction_wait_ms, + "the drive's Phase::Stream span must cover the in-stream auction wait: \ + stream_ms={stream_ms} auction_wait_ms={auction_wait_ms}" + ); + } + + #[test] + fn classify_stream_delivery_treats_bytes_written_before_an_error_as_partial() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 42), + DeliveryResult::Partial, + "bytes already on the wire before a stream error is a truncated delivery" + ); + } + + #[test] + fn classify_stream_delivery_treats_an_error_with_no_bytes_as_error() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 0), + DeliveryResult::Error, + "a failure before any byte reached the client is a clean failure, not a truncation" + ); + } + + #[test] + fn classify_stream_delivery_treats_ok_as_complete() { + assert_eq!( + classify_stream_delivery(&Ok(()), 123), + DeliveryResult::Complete + ); + } } diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 07d6704a7..1943d9b3c 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,7 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; -use crate::request_timing::{Phase, RequestTimings}; +use crate::request_timing::{AuctionWaitPlacement, Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -1617,6 +1617,12 @@ pub struct OwnedProcessResponseParams { /// rescanned from the output, which cannot tell a `nonce` attribute from the same /// word inside a script. pub(crate) csp_nonce_observed: Option>, + /// Per-request phase-timing handle, carried into the streaming/buffered + /// finalizers so the `` seam wait can be recorded with the right + /// [`AuctionWaitPlacement`]. Cheap to clone (an `Arc` handle); a request that + /// never attached one to its extensions gets a fresh, unattached collector + /// that nothing ever renders. + pub(crate) timings: RequestTimings, } /// Response-authorized template cache insert inputs. The key is built before origin lookup; the @@ -1860,6 +1866,8 @@ pub async fn buffer_publisher_response_async( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, ) .await; @@ -2019,6 +2027,7 @@ fn build_template_assembly_params( request_scheme: &str, price_granularity: PriceGranularity, ad_bids_state: AdBidsState, + timings: RequestTimings, ) -> OwnedProcessResponseParams { OwnedProcessResponseParams { csp_nonce_observed: None, @@ -2041,6 +2050,7 @@ fn build_template_assembly_params( price_granularity, gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings, } } @@ -2383,6 +2393,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }, ) .await; @@ -2501,6 +2513,7 @@ pub async fn publisher_response_into_streaming_response( &orchestrator, &services, &settings, + AuctionWaitPlacement::InStream, ) .await; // Collection reached a terminal result; disarm only now @@ -2522,6 +2535,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }; while let Some(step) = hold_step_next_chunk( @@ -2857,6 +2872,7 @@ pub async fn stream_publisher_body_async( orchestrator, services, settings, + AuctionWaitPlacement::PreHeader, ) .await; if body.is_stream() { @@ -2923,6 +2939,8 @@ pub async fn stream_publisher_body_async( services, settings, request_origin: request_origin(¶ms.request_scheme, ¶ms.request_host), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, }, ) @@ -3490,6 +3508,12 @@ struct AuctionCollectDeps<'a> { settings: &'a Settings, /// Trusted request origin (`scheme://host`) for absolute inline creative URLs. request_origin: String, + /// Phase-timing handle the collect step records the auction wait into. + timings: RequestTimings, + /// Where this collect call sits relative to response headers: streaming + /// callers await inside the body already handed to the client, buffered + /// callers await before anything has been sent. + placement: AuctionWaitPlacement, } /// Run the close-body hold loop for HTML bodies, collecting the auction before @@ -3870,6 +3894,11 @@ async fn emit_abandoned_auction( /// Collect a dispatched auction before a non-HTML body streams: there is no /// `` to inject into, so bids are written to state up front and the /// auction telemetry completes immediately. +/// +/// `placement` records where this wait sits relative to response headers — the +/// caller decides, since this collector runs from both the buffered finalizer +/// (headers not yet committed) and the true streaming path (headers already +/// sent, this body only just started being polled). async fn collect_non_html_auction( dispatched: DispatchedAuction, telemetry: AuctionTelemetryCarry, @@ -3877,12 +3906,14 @@ async fn collect_non_html_auction( orchestrator: &AuctionOrchestrator, services: &RuntimeServices, settings: &Settings, + placement: AuctionWaitPlacement, ) { let auction_id = telemetry .auction_request .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3890,6 +3921,9 @@ async fn collect_non_html_auction( &make_collect_context(settings, services, &placeholder), ) .await; + params + .timings + .record_auction_wait(placement, wait_started.elapsed()); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3931,6 +3965,8 @@ async fn collect_stream_auction( services, settings, request_origin, + timings, + placement, } = deps; let auction_id = telemetry .auction_request @@ -3939,9 +3975,11 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; + timings.record_auction_wait(*placement, wait_started.elapsed()); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4538,6 +4576,7 @@ pub async fn handle_publisher_request( request_scheme, price_granularity, ad_bids_state.clone(), + timings.clone(), ); params.seam_ad_slots = seam_ad_slots.clone(); params.dispatched_auction = dispatched_auction.take(); @@ -4910,6 +4949,7 @@ pub async fn handle_publisher_request( dispatched_auction, price_granularity, gpt_diagnostics: Some(gpt_diagnostics), + timings: timings.clone(), }), }) } @@ -7546,6 +7586,7 @@ mod tests { price_granularity: Default::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -14592,6 +14633,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }, }; let mut output = Vec::new(); @@ -14641,6 +14684,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }; // Passthrough processor: the ordering contract is about collection, not // HTML rewriting, so keep the emitted bytes verbatim. @@ -15517,6 +15562,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15570,6 +15616,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15612,6 +15659,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::from_stream(futures::stream::iter(vec![Ok::<_, io::Error>( bytes::Bytes::from_static(b"live"), @@ -15732,6 +15780,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"body{background:url('https://origin.example.com/"), @@ -15790,6 +15839,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = gzip_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15851,6 +15901,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = deflate_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15912,6 +15963,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15973,6 +16025,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -16022,6 +16075,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16221,6 +16275,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"hello"), @@ -16290,6 +16345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; // The `` that triggers bid injection lives in the SECOND gzip // member. `flate2::read::GzDecoder` decodes only the first member, so @@ -16358,6 +16414,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![bytes::Bytes::from_static( b"body{background:url('https://origin.example.com/asset.png')}", @@ -16419,6 +16476,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -16568,6 +16626,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16958,6 +17017,7 @@ mod tests { price_granularity: PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } }; let make_stream_response = || PublisherResponse::Stream { @@ -17020,6 +17080,149 @@ mod tests { assert_bodiless_abandoned(&buffered_sink); } + #[test] + fn streaming_seam_wait_records_in_stream_placement() { + // The true Fastly streaming path: `publisher_response_into_streaming_response` + // hands back a lazy body after headers have already been committed by + // `stream_to_client()`. The `` seam wait polled from inside that body + // must therefore be attributed `InStream`, never `PreHeader`. + let settings = Arc::new(create_test_settings()); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(publisher_response_into_streaming_response( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + noop_services(), + )) + .expect("streaming finalize should succeed"); + + // The wait is only recorded once the lazy body is actually polled — the + // finalizer call above only constructs it. + let drained = futures::executor::block_on( + response + .into_body() + .into_bytes_bounded(settings.publisher.max_buffered_body_bytes), + ) + .expect("body should drain"); + assert!( + String::from_utf8_lossy(&drained).contains("hello"), + "should still stream the document" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "the streaming seam wait must be attributed InStream" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + + #[test] + fn buffered_template_miss_records_pre_header_placement() { + // The buffered finalizer materializes the entire response — headers and + // body — before any of it reaches the client. Even though the wait runs + // through the same `` seam code path as the streaming finalizer + // above, headers have not committed here, so it must be attributed + // `PreHeader`. (This exercises the same collect step a shared-template + // authorized miss rides through: `template_cache_key`'s presence only + // changes what happens *after* collection — whether the transformed bytes + // are stored — not where the wait itself is measured.) + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let services = noop_services(); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(buffer_publisher_response_async( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + )) + .expect("buffered finalize should succeed"); + + let html = String::from_utf8( + response + .into_body() + .into_bytes() + .unwrap_or_default() + .to_vec(), + ) + .expect("should be valid UTF-8"); + assert!(html.contains("hello"), "should still assemble the document"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "the buffered finalizer's wait must be attributed PreHeader even though \ + it shares the seam code path with the streaming finalizer" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + #[test] fn publisher_response_streaming_finalize_processes_gzip_stream() { let compressed = @@ -17142,6 +17345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -17214,6 +17418,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -17269,6 +17474,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let bogus_body = EdgeBody::from(b"not gzip".to_vec()); @@ -17382,6 +17588,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); stream_publisher_body(body, &mut output, ¶ms, &settings, ®istry) @@ -17444,6 +17651,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); From 8b028bb90aca365eddd4f106e3bc2f5ec818735b Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 09:27:44 -0700 Subject: [PATCH 009/230] Add access telemetry snapshot, route classes, and coarse route templates --- .../trusted-server-adapter-fastly/src/app.rs | 196 +++++++- .../trusted-server-adapter-fastly/src/main.rs | 283 +++++++++++- .../src/access_telemetry.rs | 418 ++++++++++++++++++ crates/trusted-server-core/src/constants.rs | 2 + crates/trusted-server-core/src/lib.rs | 1 + crates/trusted-server-core/src/publisher.rs | 78 +++- 6 files changed, 967 insertions(+), 11 deletions(-) create mode 100644 crates/trusted-server-core/src/access_telemetry.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 5cf7f9709..0a0cc8c5a 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -98,6 +98,7 @@ use edgezero_core::http::{ }; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::access_telemetry::{RouteClass, RouteMetadata, publisher_route_template}; use trusted_server_core::auction::AuctionTelemetrySink; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; @@ -303,6 +304,12 @@ fn uses_dynamic_tsjs_fallback(method: &Method, path: &str) -> bool { *method == Method::GET && path.starts_with("/static/tsjs=") } +/// Coarse route template for every `tsjs` bundle request, used as the +/// `route_template` in the [`RouteMetadata`] attached by the tsjs branch of +/// [`dispatch_fallback`]. Actual filenames vary by module/hash; the prefix +/// alone is the route identity that matters for access telemetry. +const TSJS_ROUTE_TEMPLATE: &str = "/static/tsjs=*"; + // --------------------------------------------------------------------------- // EC request state // --------------------------------------------------------------------------- @@ -846,12 +853,28 @@ async fn dispatch_fallback( PreRoute::Continue { effects } => effects, }; + // Assigned exactly once, per branch below, alongside the routing + // decision itself, so the access-telemetry route identity always + // reflects which branch actually dispatched the request — including + // when that branch's handler errors. The asset-route sub-branch is an + // early return handled separately by `dispatch_asset_fallback`, so it + // never reaches (or needs to assign) this binding. + let route_metadata: Option; + let result = if uses_dynamic_tsjs_fallback(&method, &path) { + route_metadata = Some(RouteMetadata { + route_class: RouteClass::Tsjs, + route_template: TSJS_ROUTE_TEMPLATE.to_owned(), + }); handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SurrogateControl) } else if state.registry.has_route(&method, &path) { // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + route_metadata = Some(RouteMetadata { + route_class: RouteClass::IntegrationProxy, + route_template: publisher_route_template(&path), + }); state .registry .handle_proxy(ProxyDispatchInput { @@ -890,6 +913,11 @@ async fn dispatch_fallback( .await; } + route_metadata = Some(RouteMetadata { + route_class: RouteClass::PublisherHtml, + route_template: publisher_route_template(&path), + }); + // Generate an EC ID if needed — mirrors the legacy catch-all arm. // Only for document navigations by recognised browsers; subresource // requests may lack consent signals such as Sec-GPC. @@ -958,7 +986,10 @@ async fn dispatch_fallback( } }; - let response = result.unwrap_or_else(|e| http_error(&e)); + let mut response = result.unwrap_or_else(|e| http_error(&e)); + if let Some(metadata) = route_metadata { + response.extensions_mut().insert(metadata); + } attach_dispatch_extensions(response, ec, effects) } @@ -1154,6 +1185,10 @@ struct NamedRoute { path: &'static str, primary_methods: &'static [Method], handler: NamedRouteHandler, + /// Access-telemetry traffic category for this row. Attached verbatim + /// alongside `path` (the route-table pattern) to every response this + /// route produces — see [`named_route_handler`]. + route_class: RouteClass, } const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ @@ -1171,21 +1206,25 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/.well-known/trusted-server.json", primary_methods: &[Method::GET], handler: NamedRouteHandler::TrustedServerDiscovery, + route_class: RouteClass::Other, }, NamedRoute { path: "/verify-signature", primary_methods: &[Method::POST], handler: NamedRouteHandler::VerifySignature, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/rotate", primary_methods: &[Method::POST], handler: NamedRouteHandler::RotateKey, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/deactivate", primary_methods: &[Method::POST], handler: NamedRouteHandler::DeactivateKey, + route_class: RouteClass::Ec, }, // Admin EC lookup: the bare route reads the EC ID from the caller's // `ts-ec` cookie; the parameterized route takes an explicit EC ID. @@ -1193,11 +1232,13 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/ec", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/ec/{id}", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with // an ingestion preview. Pure request inspection — no KV access. @@ -1205,6 +1246,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/eids", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEidsLookup, + route_class: RouteClass::Ec, }, // The legacy non-`/_ts` aliases (`/admin/keys/*`) are denied locally with a // 404 instead of executing key operations: the production basic-auth handler @@ -1216,36 +1258,43 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/admin/keys/rotate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/admin/keys/deactivate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/api/v1/batch-sync", primary_methods: &[Method::POST], handler: NamedRouteHandler::BatchSync, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/api/v1/identify", primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::Identify, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/set-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::SetTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/clear-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::ClearTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/auction", primary_methods: &[Method::POST], handler: NamedRouteHandler::Auction, + route_class: RouteClass::AuctionApi, }, // GET runs the SPA re-auction; OPTIONS is denied in-handler as a CORS // preflight guard for this side-effecting endpoint. @@ -1253,6 +1302,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, // Deprecated double-underscore alias. tsjs bundles served before the // `/_ts/page-bids` rename keep requesting this path from already-loaded @@ -1263,21 +1313,29 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_LEGACY_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, + // Classified `Other` rather than `IntegrationProxy`: that class is + // reserved for `state.registry.handle_proxy` (the js-integration proxy + // dispatch in `dispatch_fallback`), which these first-party proxy routes + // do not go through. NamedRoute { path: "/first-party/proxy", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyProxy, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/click", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyClick, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/sign", primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartySign, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/proxy-rebuild", @@ -1286,16 +1344,35 @@ const NAMED_ROUTES: &[NamedRoute] = &[ // POST is blocked by CORS and the guard navigates here for a 302 instead. primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartyProxyRebuild, + route_class: RouteClass::Other, }, ]; +/// Wraps [`execute_named`], attaching a [`RouteMetadata`] extension carrying +/// `route_class` and the route-table pattern (`route_template`, verbatim, +/// with parameters left as placeholders) to every response the handler +/// produces — including its early-return diagnostic and setup-error arms, +/// since the attachment happens once around the whole future rather than in +/// each branch. fn named_route_handler( state: Arc, handler: NamedRouteHandler, + route_class: RouteClass, + route_template: &'static str, ) -> impl Fn(RequestContext) -> HandlerFuture + Clone + Send + Sync + 'static { move |ctx: RequestContext| { let state = Arc::clone(&state); - Box::pin(execute_named(state, ctx, handler)) + Box::pin(async move { + execute_named(state, ctx, handler) + .await + .map(|mut response| { + response.extensions_mut().insert(RouteMetadata { + route_class, + route_template: route_template.to_owned(), + }); + response + }) + }) } } @@ -1354,7 +1431,12 @@ impl TrustedServerApp { router = router.route( route.path, method.clone(), - named_route_handler(Arc::clone(state), route.handler), + named_route_handler( + Arc::clone(state), + route.handler, + route.route_class, + route.path, + ), ); } @@ -1392,7 +1474,8 @@ mod tests { use super::{ AppState, NAMED_ROUTES, NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, - TrustedServerApp, build_per_request_services, build_state_from_settings, + RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, + build_per_request_services, build_state_from_settings, publisher_route_template, startup_error_router, }; use base64::Engine as _; @@ -2223,6 +2306,111 @@ mod tests { ); } + /// `Authorization: Basic` header value for `test_settings()`'s + /// `^/_ts/admin` handler (`admin` / `admin-pass`). + fn admin_basic_auth_header() -> edgezero_core::http::HeaderValue { + let credentials = base64::engine::general_purpose::STANDARD.encode("admin:admin-pass"); + format!("Basic {credentials}") + .parse() + .expect("should parse basic-auth header value") + } + + #[test] + fn named_route_attaches_the_table_pattern_verbatim_even_with_a_real_id_in_the_path() { + // A named-route response must carry the route-TABLE pattern + // (`{id}` left as a placeholder), never the caller's actual matched + // path segment — this is what keeps a real EC identifier out of + // access telemetry, independent of anything the row-serialization + // layer does. + let router = test_router(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let mut req = empty_request(Method::GET, &format!("/_ts/admin/ec/{ec_id}")); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("named-route responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/ec/{id}"); + assert!( + !metadata.route_template.contains(ec_id), + "the attached template must never contain the matched id" + ); + } + + #[test] + fn named_route_attaches_metadata_even_on_a_read_only_diagnostic_early_return() { + // AdminEidsLookup is handled by an early-return arm inside + // execute_named, before the normal EC lifecycle runs (see the + // "read-only diagnostics" comment there). named_route_handler wraps + // the whole future, so the attachment must still happen here too. + let router = test_router(); + let mut req = empty_request(Method::GET, "/_ts/admin/eids"); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("even a read-only diagnostic early-return response should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/eids"); + } + + #[test] + fn tsjs_fallback_attaches_tsjs_route_metadata() { + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/static/tsjs=tsjs-unified.min.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("tsjs fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Tsjs); + assert_eq!(metadata.route_template, TSJS_ROUTE_TEMPLATE); + } + + #[test] + fn integration_proxy_fallback_attaches_integration_proxy_route_metadata() { + // test_settings() enables the prebid integration, which registers a + // proxy route at /integrations/prebid/bundle.js. + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/integrations/prebid/bundle.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("integration-proxy fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); + assert_eq!( + metadata.route_template, + publisher_route_template("/integrations/prebid/bundle.js") + ); + } + + #[test] + fn publisher_fallback_attaches_publisher_html_route_metadata() { + let router = test_router(); + let response = route(&router, empty_request(Method::GET, "/news/some-article")); + + let metadata = response + .extensions() + .get::() + .expect("publisher fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::PublisherHtml); + assert_eq!(metadata.route_template, "/news/*"); + } + #[test] fn browser_device_signals_from_extension_reach_ec_finalize_state() { // Regression guard for the EdgeZero JA4/H2 signal loss: `edgezero_main` diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 5747dc8fd..c09663d13 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -14,9 +14,13 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; +use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; +use trusted_server_core::constants::{ + ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -30,6 +34,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::publisher::TemplateCacheResponseState; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -145,6 +150,18 @@ fn edgezero_main(mut req: FastlyRequest) { let server_timing_enabled = settings_snapshot .as_deref() .is_some_and(|settings| settings.observability.server_timing_enabled); + // Both read once here rather than at each `send_edgezero_response` call + // site: if `app_state` failed to build, there is no settings snapshot to + // read them from at all, so every call site would need the same + // degraded-mode fallback. `access_sample_rate` defaults to `0.0` (never + // sampled in) and `publisher_domain` to `"unknown"` in that case. + let access_sample_rate = settings_snapshot + .as_deref() + .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let publisher_domain = settings_snapshot.as_deref().map_or_else( + || "unknown".to_owned(), + |settings| settings.publisher.domain.clone(), + ); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -159,8 +176,11 @@ fn edgezero_main(mut req: FastlyRequest) { req.set_header("fastly-ssl", "1"); } - // Capture client IP before the request is consumed by dispatch. + // Capture client IP and method before the request is consumed by + // dispatch: nothing else survives to the freeze point in + // `send_edgezero_response`, which only receives the response. let client_ip = req.get_client_ip_addr(); + let request_method = req.get_method_str().to_owned(); // Strip any client-supplied x-ts-tls-* headers before injecting the trusted // values from the Fastly SDK. Must run after sanitize_fastly_forwarded_headers. @@ -211,9 +231,13 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + // Read rather than pop: the access-telemetry snapshot built later in + // `send_edgezero_response` reads this same extension, so it must still + // be attached to `response` at that point. let geo_lookup_state = response - .extensions_mut() - .remove::() + .extensions() + .get::() + .cloned() .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { @@ -257,6 +281,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -280,6 +307,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send( @@ -309,6 +339,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings, server_timing_enabled, + method: request_method, + publisher_domain, + access_sample_rate, }, ); } @@ -349,6 +382,18 @@ fn apply_entry_point_finalize_headers( }) }); apply_finalize_headers(settings, geo_info.as_ref(), response); + + // This path runs only when the middleware chain was bypassed (e.g. a + // router-level 404/405 for an unregistered method), so `geo_state` may + // still be `NotAttempted` even after a fresh lookup just ran above. + // Write the resolved outcome back so the access-telemetry snapshot built + // later in `send_edgezero_response` sees what was actually looked up, + // not the stale carried-in state. + let resolved_state = match &geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); } fn apply_edgezero_ec_finalize( @@ -394,6 +439,13 @@ struct SendContext { timings: RequestTimings, /// Whether `observability.server_timing_enabled` is set. server_timing_enabled: bool, + /// The request's HTTP method, captured before the request was consumed + /// by dispatch. + method: String, + /// The configured publisher domain. + publisher_domain: String, + /// The configured access-telemetry sample rate. + access_sample_rate: f64, } /// Outcome of handing a finalized response to the client. @@ -403,6 +455,9 @@ pub(crate) struct DeliveryOutcome { pub bytes: u64, /// Whether delivery completed or failed partway. pub result: DeliveryResult, + /// Access-telemetry dimensions captured for this response at the + /// freeze point. + pub snapshot: AccessTelemetrySnapshot, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -558,6 +613,12 @@ fn send_edgezero_response( context.server_timing_enabled, ); + // Built unconditionally, right after the freeze point and before + // `into_parts()` consumes `response`: nothing else survives to + // post-send on every path (the request was consumed by dispatch, and + // `EcFinalizeState` is absent on asset, admin, and error paths). + let snapshot = build_access_telemetry_snapshot(&response, context); + let (parts, body) = response.into_parts(); match body { @@ -578,6 +639,7 @@ fn send_edgezero_response( Ok(()) => DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, }, Err(e) => { // Every byte was handed to the transport (the drive @@ -588,13 +650,18 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Partial, + snapshot, } } }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { bytes, result } + DeliveryOutcome { + bytes, + result, + snapshot, + } } } } @@ -604,11 +671,89 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, } } } } +/// Builds the [`AccessTelemetrySnapshot`] for `response` at the +/// `Server-Timing` freeze point. +/// +/// Reads route identity, geo country, and template-cache state from typed +/// response extensions rather than the headers those extensions back — +/// operator-configured response headers can override a managed header, so +/// reading a header here could silently drift from what actually happened. +/// Falls back to `"unknown"`/[`RouteClass::Other`] sentinels when an +/// extension was never attached (router-generated, asset, and other +/// responses that never passed through a `RouteMetadata`-attaching +/// wrapper). +fn build_access_telemetry_snapshot( + response: &HttpResponse, + context: &SendContext, +) -> AccessTelemetrySnapshot { + let (route_class, route_template) = match response.extensions().get::() { + Some(metadata) => (metadata.route_class, metadata.route_template.clone()), + None => (RouteClass::Other, "unknown".to_owned()), + }; + + let country = match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => info.country.clone(), + Some(GeoLookupState::Attempted | GeoLookupState::NotAttempted) | None => { + "unknown".to_owned() + } + }; + + let template_cache_state = response + .extensions() + .get::() + .map_or_else(|| "unknown".to_owned(), |state| state.as_str().to_owned()); + + let body_mode = if matches!(response.body(), EdgeBody::Stream(_)) { + "streamed" + } else { + "buffered" + }; + + AccessTelemetrySnapshot { + method: context.method.clone(), + status: response.status().as_u16(), + route_class, + route_template, + publisher_domain: context.publisher_domain.clone(), + env: resolve_env_dimension(), + service_id: env_var_or_unknown(ENV_FASTLY_SERVICE_ID), + pop: env_var_or_unknown(ENV_FASTLY_POP), + ts_version: env_var_or_unknown(ENV_FASTLY_SERVICE_VERSION), + country, + template_cache_state, + body_mode, + sample_rate: context.access_sample_rate, + } +} + +/// Derives the `env` access-telemetry dimension from the same +/// `FASTLY_IS_STAGING` input that drives the `x-ts-env` response header +/// (see [`apply_finalize_headers`]), never from [`Settings`] — `Settings` +/// has no environment field and does not gain one for this. +/// +/// `"unknown"` covers contexts where the variable is entirely absent (for +/// example native unit tests run outside Fastly Compute); on the Fastly +/// platform the variable is always present, as either `"1"` or not. +fn resolve_env_dimension() -> String { + match std::env::var(ENV_FASTLY_IS_STAGING) { + Ok(value) if value == "1" => "staging".to_owned(), + Ok(_) => "production".to_owned(), + Err(_) => "unknown".to_owned(), + } +} + +/// Reads a Fastly-provided environment variable, defaulting to `"unknown"` +/// when unset. +fn env_var_or_unknown(name: &str) -> String { + std::env::var(name).unwrap_or_else(|_| "unknown".to_owned()) +} + /// Apply every late response mutation, then restore privacy invariants before headers commit. fn apply_terminal_response_effects( response: &mut HttpResponse, @@ -820,6 +965,26 @@ mod tests { .expect("should parse test settings") } + /// A minimal [`AccessTelemetrySnapshot`] fixture for tests that only + /// need a `DeliveryOutcome` to exist, not its telemetry content. + fn sample_access_snapshot() -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class: RouteClass::Other, + route_template: "/other/*".to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + #[test] fn health_response_short_circuits_get_health() { let req = FastlyRequest::get("https://example.com/health"); @@ -1264,6 +1429,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot: sample_access_snapshot(), }; assert_eq!( @@ -1313,6 +1479,115 @@ mod tests { ); } + fn send_context_fixture() -> SendContext { + SendContext { + timings: RequestTimings::new(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 0.25, + } + } + + #[test] + fn access_snapshot_defaults_when_no_extensions_are_attached() { + // Router-generated 404/405 responses and other paths that never pass + // through a RouteMetadata-attaching wrapper must still produce a + // usable snapshot: RouteClass::Other and "unknown" sentinels, never + // a missing/panicking build. + let response = response_builder() + .status(404) + .body(EdgeBody::empty()) + .expect("should build response"); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!(snapshot.status, 404); + assert_eq!(snapshot.method, "GET"); + assert!(matches!(snapshot.route_class, RouteClass::Other)); + assert_eq!(snapshot.route_template, "unknown"); + assert_eq!(snapshot.country, "unknown"); + assert_eq!(snapshot.template_cache_state, "unknown"); + assert_eq!(snapshot.body_mode, "buffered"); + assert_eq!(snapshot.publisher_domain, "test-publisher.com"); + assert_eq!(snapshot.sample_rate, 0.25); + } + + #[test] + fn access_snapshot_reads_route_geo_and_template_cache_extensions() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(RouteMetadata { + route_class: RouteClass::AuctionApi, + route_template: "/auction".to_owned(), + }); + response.extensions_mut().insert(GeoLookupState::Resolved( + trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }, + )); + response + .extensions_mut() + .insert(TemplateCacheResponseState::Hit); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert!(matches!(snapshot.route_class, RouteClass::AuctionApi)); + assert_eq!(snapshot.route_template, "/auction"); + assert_eq!(snapshot.country, "US"); + assert_eq!(snapshot.template_cache_state, "hit"); + } + + #[test] + fn access_snapshot_treats_attempted_geo_lookup_as_unknown_country() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(GeoLookupState::Attempted); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!( + snapshot.country, "unknown", + "an attempted-but-unresolved lookup must not surface a stale country" + ); + } + + #[test] + fn access_snapshot_body_mode_reflects_the_response_body_variant() { + let streamed = response_builder() + .status(200) + .body(EdgeBody::stream(futures::stream::empty())) + .expect("should build streaming response"); + let buffered = response_builder() + .status(200) + .body(EdgeBody::from(b"hi".to_vec())) + .expect("should build buffered response"); + let context = send_context_fixture(); + + assert_eq!( + build_access_telemetry_snapshot(&streamed, &context).body_mode, + "streamed" + ); + assert_eq!( + build_access_telemetry_snapshot(&buffered, &context).body_mode, + "buffered" + ); + } + #[test] fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { // A streaming seam wait (Task 6, publisher.rs) records into the same diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs new file mode 100644 index 000000000..9daf77abc --- /dev/null +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -0,0 +1,418 @@ +//! Access telemetry: route classification and the per-request access log row. +//! +//! Extends the reserved `access_logs_raw` Tinybird datasource with bounded, +//! content-free route identity (see [`RouteClass`] and +//! [`publisher_route_template`]) instead of the raw request path, which would +//! otherwise carry identifiers, search terms, and other user-generated +//! content into a 30-day dataset. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` +//! section 9. + +use serde_json::json; + +use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; + +/// Maximum number of characters kept from a publisher path's first segment +/// by [`publisher_route_template`]. +const MAX_SEGMENT_LEN: usize = 32; + +/// Coarse traffic category for one response, used as a `LowCardinality` +/// dimension in the access telemetry row. +/// +/// Assigned per route by the adapter at dispatch time (see +/// [`RouteMetadata`]) rather than reconstructed from a handler enum or a +/// path regex at emission time, so the mapping lives in exactly one place. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RouteClass { + /// A publisher-origin page served through the assembly/auction pipeline. + PublisherHtml, + /// The unified `tsjs` static bundle. + Tsjs, + /// A request served by a registered [`crate::integrations::IntegrationProxy`]. + IntegrationProxy, + /// An Edge Cookie identity endpoint (verify, rotate, identify, admin + /// lookups, batch sync). + Ec, + /// The server-side auction or SPA re-auction (`page-bids`) endpoint. + AuctionApi, + /// Everything else: discovery, tester-cookie toggles, denied legacy + /// aliases, and any response with no attached [`RouteMetadata`]. + Other, +} + +impl RouteClass { + /// Renders this variant as the `snake_case` string stored in the + /// `route_class` column. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::PublisherHtml => "publisher_html", + Self::Tsjs => "tsjs", + Self::IntegrationProxy => "integration_proxy", + Self::Ec => "ec", + Self::AuctionApi => "auction_api", + Self::Other => "other", + } + } +} + +/// Route identity for one response, carried from dispatch to the freeze +/// point as a response extension. +/// +/// The matched route pattern does not otherwise survive dispatch: the +/// request is consumed by the router, and nothing else records which +/// route-table row (or coarse fallback bucket) produced the response. Each +/// named-route handler wrapper attaches its matched route-table pattern +/// verbatim; the publisher fallback and `tsjs` handlers attach their class +/// plus a coarse template ([`publisher_route_template`] for the former). +#[derive(Debug, Clone)] +pub struct RouteMetadata { + /// Coarse traffic category for this response. + pub route_class: RouteClass, + /// Bounded, content-free route identifier. For named routes this is the + /// route-table pattern verbatim (e.g. `/_ts/admin/ec/{id}`); for + /// publisher-fallback traffic it is the output of + /// [`publisher_route_template`]. + pub route_template: String, +} + +/// Normalizes a publisher-fallback request path into a bounded, +/// content-free route template. +/// +/// Returns `/` plus the first path segment, lowercased and restricted to +/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a +/// trailing `/*` appended when the path has additional segments beyond the +/// first. The root path `/` maps to itself. An empty first segment, or one +/// containing any character outside the allowlist (after lowercasing), +/// maps to `/other/*` — the segment is rejected outright rather than +/// filtered, so no fragment of a disallowed segment (an email address, a +/// search phrase) ever reaches the row. +/// +/// This is deliberately coarser than the auction-telemetry path +/// normalizer, which redacts long tokens but preserves short identifiers +/// and arbitrary slugs; that normalizer is not sufficient for a dataset +/// this broad. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::publisher_route_template; +/// +/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); +/// assert_eq!(publisher_route_template("/"), "/"); +/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// ``` +#[must_use] +pub fn publisher_route_template(path: &str) -> String { + if path == "/" { + return "/".to_owned(); + } + + let trimmed = path.strip_prefix('/').unwrap_or(path); + let (first_segment, rest) = match trimmed.split_once('/') { + Some((first, rest)) => (first, rest), + None => (trimmed, ""), + }; + let has_more_depth = !rest.is_empty(); + + let lowered = first_segment.to_ascii_lowercase(); + let is_allowlisted = !lowered.is_empty() + && lowered + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + + if !is_allowlisted { + return "/other/*".to_owned(); + } + + let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); + if has_more_depth { + format!("/{truncated}/*") + } else { + format!("/{truncated}") + } +} + +/// A point-in-time view of the access-log dimensions for one response, +/// captured unconditionally at the `Server-Timing` freeze point. +/// +/// Built from typed response extensions ([`RouteMetadata`], the geo +/// lookup state, and the template-cache response state) plus adapter-owned +/// environment values, never from the public response headers those +/// extensions back — operator-configured response headers can override a +/// managed header, so reading the header instead of the extension would let +/// the row silently drift from the extension of record. +#[derive(Debug, Clone)] +pub struct AccessTelemetrySnapshot { + /// The request's HTTP method (e.g. `GET`). + pub method: String, + /// The response's HTTP status code. + pub status: u16, + /// Coarse traffic category (see [`RouteClass`]). + pub route_class: RouteClass, + /// Bounded, content-free route identifier (see + /// [`publisher_route_template`]). + pub route_template: String, + /// The configured publisher domain. + pub publisher_domain: String, + /// Adapter-derived deployment environment: `production`, `staging`, or + /// `unknown`. + pub env: String, + /// Fastly service ID, or `unknown` when unavailable. + pub service_id: String, + /// Fastly POP code, or `unknown` when unavailable. + pub pop: String, + /// Trusted Server build/version identifier, or `unknown` when + /// unavailable. + pub ts_version: String, + /// Two-letter geo country code, or `unknown` when no geo lookup + /// resolved one. + pub country: String, + /// Template-cache outcome for this response, or `unknown` when the + /// response never passed through the assembly pipeline. + pub template_cache_state: String, + /// Whether the response body was streamed or buffered to the client. + pub body_mode: &'static str, + /// The configured access-telemetry sample rate at the time this + /// response was handled. + pub sample_rate: f64, +} + +/// Renders one NDJSON access-log row for the Tinybird Events API. +/// +/// Column names match spec section 9 exactly. Phase columns come from +/// `timings` and serialize as JSON `null` for phases that were never +/// recorded; every dimension column comes from `snapshot` and is a +/// non-nullable string (callers are expected to substitute an `unknown` +/// sentinel rather than leave a dimension empty). `event_date` is omitted: +/// the datasource derives it from `event_ts` by default. +#[must_use] +pub fn access_event_row( + snapshot: &AccessTelemetrySnapshot, + timings: &TimingSnapshot, + event_ts_epoch_ms: u64, +) -> String { + let auction_wait_placement = match timings.auction_wait_placement { + Some(AuctionWaitPlacement::PreHeader) => "pre_header", + Some(AuctionWaitPlacement::InStream) => "in_stream", + None => "none", + }; + + let row = json!({ + "event_ts": format_event_timestamp(event_ts_epoch_ms), + "method": snapshot.method, + "status": snapshot.status, + "time_elapsed_ms": timings.time_elapsed_ms, + "sample_rate": snapshot.sample_rate, + "service_id": snapshot.service_id, + "publisher_domain": snapshot.publisher_domain, + "env": snapshot.env, + "route_class": snapshot.route_class.as_str(), + "route_template": snapshot.route_template, + "body_mode": snapshot.body_mode, + "auction_wait_placement": auction_wait_placement, + "appbuild_ms": timings.appbuild_ms, + "filter_ms": timings.filter_ms, + "geo_ms": timings.geo_ms, + "kv_ms": timings.kv_ms, + "origin_ms": timings.origin_ms, + "template_cache_ms": timings.template_cache_ms, + "auction_wait_ms": timings.auction_wait_ms, + "stream_ms": timings.stream_ms, + "request_elapsed_ms": timings.request_elapsed_ms, + "resp_bytes": timings.resp_bytes, + "template_cache_state": snapshot.template_cache_state, + "country": snapshot.country, + "ts_version": snapshot.ts_version, + "pop": snapshot.pop, + }); + row.to_string() +} + +/// Formats `epoch_ms` as a `ClickHouse` `DateTime64(3)`-compatible string +/// (`%Y-%m-%d %H:%M:%S%.3f`), matching the format the auction telemetry +/// sink already uses for `event_ts`. +/// +/// Falls back to the Unix epoch when `epoch_ms` cannot be represented as a +/// valid timestamp, which never happens for a real wall-clock reading. +fn format_event_timestamp(epoch_ms: u64) -> String { + let epoch_ms = i64::try_from(epoch_ms).unwrap_or(i64::MAX); + let timestamp = chrono::DateTime::::from_timestamp_millis(epoch_ms) + .unwrap_or(chrono::DateTime::::UNIX_EPOCH); + timestamp.format("%Y-%m-%d %H:%M:%S%.3f").to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A 64-hex-plus-suffix EC identifier, matching the format the admin + /// EC lookup route (`/_ts/admin/ec/{id}`) accepts as a path parameter. + const SYNTHETIC_EC_ID: &str = + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + + fn unknown_snapshot(route_class: RouteClass, route_template: &str) -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class, + route_template: route_template.to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + + #[test] + fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route templates come from the route table verbatim, never + // from the matched request path, so a row built for this route can + // never carry the caller's actual EC id — even when a caller + // supplies one shaped exactly like a real one. + let snapshot = unknown_snapshot(RouteClass::Ec, "/_ts/admin/ec/{id}"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + + assert!( + !row.contains(SYNTHETIC_EC_ID), + "row must never contain a literal EC identifier: {row}" + ); + assert!( + row.contains("/_ts/admin/ec/{id}"), + "row should still carry the route-table pattern: {row}" + ); + } + + #[test] + fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!( + publisher_route_template("/news/some-article-slug"), + "/news/*" + ); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); + } + + #[test] + fn publisher_route_template_rejects_empty_first_segment() { + assert_eq!( + publisher_route_template("//double-slash"), + "/other/*", + "an empty first segment should not be treated as allowlisted" + ); + } + + #[test] + fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + assert_eq!( + publisher_route_template("/News/Article"), + "/news/*", + "should lowercase before validating and truncating" + ); + } + + #[test] + fn row_serializes_nulls_for_missing_phases() { + let snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + for field in [ + "time_elapsed_ms", + "appbuild_ms", + "filter_ms", + "geo_ms", + "kv_ms", + "origin_ms", + "template_cache_ms", + "auction_wait_ms", + "stream_ms", + "request_elapsed_ms", + "resp_bytes", + ] { + assert!( + parsed[field].is_null(), + "unrecorded phase `{field}` should serialize as null: {row}" + ); + } + + for field in [ + "service_id", + "publisher_domain", + "env", + "route_class", + "route_template", + "body_mode", + "template_cache_state", + "country", + "ts_version", + "pop", + ] { + assert!( + parsed[field].is_string(), + "dimension `{field}` must never be null: {row}" + ); + } + assert_eq!(parsed["auction_wait_placement"], "none"); + } + + #[test] + fn row_serializes_recorded_phases_as_numbers() { + let snapshot = unknown_snapshot(RouteClass::AuctionApi, "/auction"); + let timings = TimingSnapshot { + time_elapsed_ms: Some(12), + request_elapsed_ms: Some(15), + appbuild_ms: Some(1), + filter_ms: Some(2), + geo_ms: Some(3), + kv_ms: Some(4), + origin_ms: Some(5), + template_cache_ms: Some(6), + auction_wait_ms: Some(7), + stream_ms: Some(8), + auction_wait_placement: Some(AuctionWaitPlacement::InStream), + resp_bytes: Some(1024), + }; + let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["appbuild_ms"], 1); + assert_eq!(parsed["stream_ms"], 8); + assert_eq!(parsed["resp_bytes"], 1024); + assert_eq!(parsed["auction_wait_placement"], "in_stream"); + } + + #[test] + fn route_class_renders_snake_case() { + assert_eq!(RouteClass::PublisherHtml.as_str(), "publisher_html"); + assert_eq!(RouteClass::Tsjs.as_str(), "tsjs"); + assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); + assert_eq!(RouteClass::Ec.as_str(), "ec"); + assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Other.as_str(), "other"); + } + + #[test] + fn format_event_timestamp_matches_clickhouse_datetime64_shape() { + // 2023-11-14T22:13:20.000Z + let rendered = format_event_timestamp(1_700_000_000_000); + assert_eq!(rendered, "2023-11-14 22:13:20.000"); + } +} diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index e1152b1e7..1f85facb8 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -34,6 +34,8 @@ pub const HEADER_X_TS_ENV: HeaderName = HeaderName::from_static("x-ts-env"); // Fastly environment variables pub const ENV_FASTLY_SERVICE_VERSION: &str = "FASTLY_SERVICE_VERSION"; pub const ENV_FASTLY_IS_STAGING: &str = "FASTLY_IS_STAGING"; +pub const ENV_FASTLY_SERVICE_ID: &str = "FASTLY_SERVICE_ID"; +pub const ENV_FASTLY_POP: &str = "FASTLY_POP"; // Common standard header names used across modules pub const HEADER_USER_AGENT: HeaderName = HeaderName::from_static("user-agent"); diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index b8a1a5718..de1bbad21 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -31,6 +31,7 @@ ) )] +pub mod access_telemetry; pub(crate) mod asset_image_optimizer; pub mod auction; pub mod auction_config_types; diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 1943d9b3c..6a30611c8 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -91,21 +91,44 @@ const DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); const HEADER_X_TS_TEMPLATE_CACHE: &str = "x-ts-template-cache"; const HEADER_X_TS_ASSEMBLY: &str = "x-ts-assembly"; -#[derive(Clone, Copy, PartialEq, Eq)] -enum TemplateCacheResponseState { +/// Outcome of a template-cache lookup/store attempt for one response. +/// +/// Set on every response that passes through the assembly pipeline via +/// [`set_template_cache_response_state`], which writes both the +/// `x-ts-template-cache` response header and this same value as a typed +/// response extension, so the two can never drift. Access telemetry reads +/// the extension rather than the header, since operator-configured response +/// headers can override a managed header but cannot touch extensions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TemplateCacheResponseState { + /// The cached template was found and reused. Hit, + /// No cached template existed; the cache store is reserved for this + /// content type. MissReserved, + /// No cached template existed; one was stored after assembly. MissStored, + /// No cached template existed; storing the freshly assembled template + /// failed. MissStoreError, + /// The request bypassed the cache lookup. BypassRequest, + /// The response bypassed the cache store. BypassResponse, + /// The response's content type is not supported by the template cache. Unsupported, + /// The cached template entry was invalid and could not be reused. Invalid, + /// A backend error prevented the cache lookup or store. BackendError, } impl TemplateCacheResponseState { - const fn as_str(self) -> &'static str { + /// Renders this variant as the string written to the + /// `x-ts-template-cache` header and the `template_cache_state` access + /// telemetry column. + #[must_use] + pub const fn as_str(self) -> &'static str { match self { Self::Hit => "hit", Self::MissReserved => "miss-reserved", @@ -128,6 +151,7 @@ fn set_template_cache_response_state( HEADER_X_TS_TEMPLATE_CACHE, HeaderValue::from_static(state.as_str()), ); + response.extensions_mut().insert(state); } #[derive(Clone, Copy, PartialEq, Eq)] @@ -9057,6 +9081,54 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_response_extension_matches_the_header_on_every_transition() { + // The typed extension and the `x-ts-template-cache` header are written + // together by a single setter, so they must always agree — access + // telemetry reads the extension precisely because it cannot drift from + // what an operator-configured header override might otherwise show. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + + queue_shareable_html(&stub); + + let cold = run(&settings, &services, navigation_request()).await; + let cold_header = cold + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let cold_extension = cold + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + cold_extension, + cold_header.as_deref(), + "the cold-fill extension must match the header" + ); + assert_eq!(cold_extension, Some("miss-stored")); + + let warm = run(&settings, &services, navigation_request()).await; + let warm_header = warm + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let warm_extension = warm + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + warm_extension, + warm_header.as_deref(), + "the warm-hit extension must match the header" + ); + assert_eq!(warm_extension, Some("hit")); + } + #[tokio::test] async fn template_cache_span_recorded_only_when_lookup_runs() { // Inline mode: no shared-cache key is ever computed, so the lookup From 48acd816d7044c5a6d77d6780e0afdf1473bd214 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:00:30 -0700 Subject: [PATCH 010/230] Emit confirmed access telemetry rows after pull-sync post-send --- .../trusted-server-adapter-fastly/src/main.rs | 207 ++++++++++- .../src/tinybird.rs | 336 +++++++++++++++++- 2 files changed, 524 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index c09663d13..65e89cd1c 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,5 +1,5 @@ use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -14,7 +14,9 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; +use trusted_server_core::access_telemetry::{ + AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, +}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; @@ -275,7 +277,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -287,6 +289,7 @@ fn edgezero_main(mut req: FastlyRequest) { }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); + emit_access_telemetry_after_send(settings, &outcome, &timings); return; } Err(e) => { @@ -301,7 +304,7 @@ fn edgezero_main(mut req: FastlyRequest) { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -317,6 +320,7 @@ fn edgezero_main(mut req: FastlyRequest) { &partner_registry, &ec_state, ); + emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -333,17 +337,31 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { - timings, + timings: timings.clone(), server_timing_enabled, method: request_method, publisher_domain, access_sample_rate, }, ); + // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec + // finalize branch above failed), so there is no pull-sync dispatch here + // at all — telemetry is the only post-send step. Reload settings when + // `app_state` never built, matching the fallback used earlier in this + // function for entry-point finalize headers. + match settings_snapshot.as_deref() { + Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), + None => match load_settings_from_config_store() { + Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), + Err(e) => { + log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); + } + }, + } } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -432,6 +450,59 @@ fn run_edgezero_pull_sync_after_send( } } +/// Builds and emits the access-telemetry row for one delivered response, +/// when access telemetry is enabled and this request is sampled in. +/// +/// Called last at every `send_edgezero_response` call site in +/// [`edgezero_main`] — after `run_edgezero_pull_sync_after_send` on the two +/// EC-finalized paths, and directly after send on the asset/admin/error +/// fallback path, which never builds an [`EcFinalizeState`] or route-scoped +/// `RuntimeServices` at all. The Tinybird transport context is therefore +/// constructed fresh from `settings` here rather than threaded through +/// either of those per-route types, so every response class can emit. +/// +/// Sampled-out requests return silently — that is the expected, high-volume +/// case and not worth a log line. Every other drop (row build, token load, +/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) +/// logs exactly one warning naming the reason. +fn emit_access_telemetry_after_send( + settings: &Settings, + outcome: &DeliveryOutcome, + timings: &RequestTimings, +) { + if !settings.tinybird.enabled || !settings.tinybird.access_enabled { + return; + } + + let since_epoch = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default(); + let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); + // Entropy for the sampling decision: the timestamp's nanosecond + // resolution XORed with a per-request value already on hand + // (`outcome.bytes`), so two requests handled in the same instance never + // collide on sampling decisions purely because they read the same + // millisecond. There is no `rand` crate dependency here — see + // `tinybird::sampled_in`. + let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); + let entropy = entropy_nanos ^ outcome.bytes; + + if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + return; + } + + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); + let result = futures::executor::block_on(tinybird::emit_access_event( + &platform::FastlyPlatformHttpClient, + &target, + row, + )); + if let Err(error) = result { + log::warn!("access telemetry emission dropped: {error:?}"); + } +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -935,6 +1006,7 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1656,4 +1728,127 @@ mod tests { DeliveryResult::Complete ); } + + /// Records `"telemetry"` into a shared order log instead of sending a + /// real request, standing in for the adapter's platform HTTP client in + /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. + struct OrderingHttpClient { + log: Arc>>, + } + + #[async_trait::async_trait(?Send)] + impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { + async fn send( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformResponse, + Report, + > { + self.log + .lock() + .expect("should lock order log") + .push("telemetry"); + let response = response_builder() + .status(edgezero_core::http::StatusCode::ACCEPTED) + .body(EdgeBody::empty()) + .expect("should build ordering test response"); + Ok(trusted_server_core::platform::PlatformResponse::new( + response, + )) + } + + async fn send_async( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformPendingRequest, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + + async fn select( + &self, + _pending_requests: Vec, + ) -> Result< + trusted_server_core::platform::PlatformSelectResult, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + } + + #[test] + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // `edgezero_main` cannot be driven directly in a unit test (it + // consumes a live `fastly::Request::from_client()`), and + // `run_edgezero_pull_sync_after_send` has no injectable seam of its + // own — it dispatches through the real identity-graph pull-sync + // path, which needs a configured EC KV store, partner registry, and + // rate limiter wired together. This test instead exercises the two + // REAL functions `edgezero_main` calls that DO have a testable seam + // — `send_edgezero_response` (which stamps `request_elapsed` before + // returning, per Task 6/7) and `tinybird::emit_access_event` (the + // telemetry send added by this task) — around an instrumented + // stand-in for the pull-sync dispatch call, in the exact order + // `edgezero_main` places them. + // + // This proves the elapsed-before-telemetry leg from real production + // code (the assertion below reads the real `timings` snapshot + // between the two calls). The pull-sync-before-telemetry leg is a + // source-order invariant in `edgezero_main`'s three call sites + // (verified by code review, not by this test) because + // `run_edgezero_pull_sync_after_send` itself has no seam to + // instrument — see task-8-report.md for this residual. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + let timings = RequestTimings::new(); + let response = response_builder() + .body(EdgeBody::from("ok")) + .expect("should build response"); + + let outcome = send_edgezero_response( + response, + None, + &SendContext { + timings: timings.clone(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 1.0, + }, + ); + assert!( + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should already be stamped before pull-sync/telemetry run" + ); + + // Stand-in for `run_edgezero_pull_sync_after_send`, which has no + // injectable seam (see the test doc comment above). + log.lock().expect("should lock order log").push("pull_sync"); + + let http_client = OrderingHttpClient { + log: Arc::clone(&log), + }; + let target = tinybird::TinybirdEventsTarget::from_access_config( + trusted_server_core::settings::TinybirdSettings { + api_host: "api.us-east.aws.tinybird.co".to_owned(), + ..trusted_server_core::settings::TinybirdSettings::default() + }, + ); + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + + futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) + .expect("should send access telemetry"); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 08b5811fc..5025f9a73 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -11,10 +11,13 @@ use trusted_server_core::auction::telemetry::{ }; use trusted_server_core::error::TrustedServerError; use trusted_server_core::platform::{ - PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, StoreName, + PlatformBackend as _, PlatformBackendSpec, PlatformHttpClient, PlatformHttpRequest, + PlatformSecretStore as _, RuntimeServices, StoreName, }; use trusted_server_core::settings::{Settings, TinybirdSettings}; +use crate::platform::{FastlyPlatformBackend, FastlyPlatformSecretStore}; + const TINYBIRD_EVENTS_PATH: &str = "/v0/events"; const TINYBIRD_NDJSON_CONTENT_TYPE: &str = "application/x-ndjson"; const TINYBIRD_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(2); @@ -45,7 +48,7 @@ struct FastlyTinybirdAuctionTelemetrySink { } #[derive(Debug, Clone)] -struct TinybirdEventsTarget { +pub(crate) struct TinybirdEventsTarget { api_host: String, dataset: String, secret_store: StoreName, @@ -69,6 +72,27 @@ impl TinybirdEventsTarget { max_body_bytes: config.max_body_bytes, } } + + /// Builds the Events API target for the access-log datasource. + /// + /// Shares [`from_config`](Self::from_config)'s host/secret-store/ + /// body-size-limit derivation, but points at `access_dataset` and + /// `access_token_secret` instead of the auction pair, so access-log + /// emission never shares a datasource or token with auction telemetry + /// even though both configs come from the same [`TinybirdSettings`]. + pub(crate) fn from_access_config(config: TinybirdSettings) -> Self { + let uri = tinybird_events_uri(&config.api_host, &config.access_dataset); + let backend_spec = tinybird_backend_spec(&config.api_host); + Self { + api_host: config.api_host, + dataset: config.access_dataset, + secret_store: StoreName::from(config.secret_store), + token_secret: config.access_token_secret, + uri, + backend_spec, + max_body_bytes: config.max_body_bytes, + } + } } impl FastlyTinybirdAuctionTelemetrySink { @@ -208,6 +232,149 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { } } +// --------------------------------------------------------------------------- +// Access telemetry: confirmed-delivery emitter +// --------------------------------------------------------------------------- + +/// Bucket count [`sampled_in`] maps `entropy` into. +/// +/// Large enough that `rate` values with several significant digits (e.g. +/// `0.015`) still land in a distinct bucket instead of rounding away, while +/// staying well inside `u64` range once multiplied by `rate`. +const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; + +/// Decides whether one request's access-telemetry row should be emitted. +/// +/// `entropy` should vary from request to request — callers derive it from +/// the wall-clock event timestamp `XORed` with a cheap per-request value (see +/// the call site in `main.rs`). There is no `rand` crate dependency here: +/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping +/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against +/// `rate` is not cryptographically uniform (the low bits of a timestamp are +/// not perfectly evenly distributed), but access-telemetry sampling only +/// needs an approximately even sample, not a provably unbiased one. +/// +/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns +/// `true`, independent of `entropy`, so both boundary configurations behave +/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` +/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), +/// but this function stays total rather than leaning on that invariant. +#[must_use] +pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { + if rate >= 1.0 { + return true; + } + if rate <= 0.0 { + return false; + } + let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; + entropy % ACCESS_SAMPLE_BUCKETS < threshold +} + +/// Loads and validates the access-log APPEND token from the Fastly secret store. +/// +/// Constructs [`FastlyPlatformSecretStore`] directly instead of routing +/// through [`RuntimeServices`]: access-telemetry emission runs post-delivery +/// for every response class — including asset, admin, and error responses +/// that never build a route-scoped `RuntimeServices` — so the transport +/// context here must be adapter-owned and route-independent rather than +/// threaded from wherever the route happened to construct one. +fn load_access_token(target: &TinybirdEventsTarget) -> Result> { + let token = FastlyPlatformSecretStore + .get_string(&target.secret_store, &target.token_secret) + .change_context(TrustedServerError::Proxy { + message: "Tinybird access append token unavailable".to_owned(), + })?; + let token = token.trim().to_owned(); + if token.is_empty() { + return Err(Report::new(TrustedServerError::Proxy { + message: "Tinybird access append token is empty".to_owned(), + })); + } + Ok(token) +} + +/// Builds the Events API POST request for one access-log row. +fn build_access_events_request( + target: &TinybirdEventsTarget, + body: String, + auth_header: HeaderValue, +) -> Result> { + request_builder() + .method(Method::POST) + .uri(target.uri.as_str()) + .header(header::AUTHORIZATION, auth_header) + .header(header::CONTENT_TYPE, TINYBIRD_NDJSON_CONTENT_TYPE) + .body(Body::from(body)) + .change_context(TrustedServerError::Proxy { + message: "failed to build Tinybird Events API request".to_owned(), + }) +} + +/// Sends one confirmed access-log row to the Tinybird Events API and waits +/// for the response. +/// +/// Unlike [`FastlyTinybirdAuctionTelemetrySink::emit_auction_events`] (fire- +/// and-forget, dispatched mid-request so it never adds latency to the +/// response), this runs post-delivery: the response has already reached the +/// client, so there is no latency budget left to protect, and the send can +/// afford to wait for — and validate — the reply. `client` is the adapter's +/// stateless platform HTTP client in production +/// ([`crate::platform::FastlyPlatformHttpClient`]); accepting it as `&dyn +/// PlatformHttpClient` here (rather than that concrete type) is what lets +/// tests substitute a recording double instead of performing a real network +/// send, matching how [`RuntimeServices::http_client`] is consumed +/// elsewhere. `target` is derived from settings once at the post-send call +/// site rather than threaded through any per-route state. +/// +/// A non-2xx status is reported as `Err` naming the status; there is no +/// retry — the caller logs exactly one warning and moves on. +/// +/// # Errors +/// +/// Returns `Err` when the access-log APPEND token cannot be loaded, the +/// backend cannot be registered, the request cannot be built or sent, or the +/// Tinybird Events API responds with a non-2xx status. +pub(crate) async fn emit_access_event( + client: &dyn PlatformHttpClient, + target: &TinybirdEventsTarget, + row: String, +) -> Result<(), Report> { + let token = load_access_token(target)?; + let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; + let backend_name = FastlyPlatformBackend + .ensure(&target.backend_spec) + .change_context(TrustedServerError::Proxy { + message: "Tinybird backend registration failed".to_owned(), + })?; + let request = build_access_events_request(target, row, auth_header)?; + + log::info!( + "sending access telemetry to Tinybird dataset={} host={} backend={}", + target.dataset, + target.api_host, + backend_name + ); + + let response = client + .send(PlatformHttpRequest::new(request, backend_name)) + .await + .change_context(TrustedServerError::Proxy { + message: "failed to send Tinybird access telemetry request".to_owned(), + })?; + + if response.response.status().is_success() { + Ok(()) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request failed with status {}", + response.response.status() + ), + })) + } +} + fn tinybird_backend_spec(api_host: &str) -> PlatformBackendSpec { PlatformBackendSpec { scheme: "https".to_owned(), @@ -327,25 +494,28 @@ mod tests { body: Vec, } + /// Records outbound requests and, for [`PlatformHttpClient::send`] (the + /// blocking variant `emit_access_event` uses), returns a synthetic + /// response carrying `respond_status` instead of performing a real + /// network send. #[derive(Default)] struct RecordingHttpClient { requests: Mutex>, select_calls: Mutex, + respond_status: Mutex, } - #[async_trait::async_trait(?Send)] - impl PlatformHttpClient for RecordingHttpClient { - async fn send( - &self, - _request: PlatformHttpRequest, - ) -> Result> { - Err(Report::new(PlatformError::Unsupported)) + impl RecordingHttpClient { + /// Status [`PlatformHttpClient::send`] should reply with. Irrelevant + /// to auction-sink tests, which only exercise `send_async`. + fn respond_with(status: u16) -> Self { + Self { + respond_status: Mutex::new(status), + ..Self::default() + } } - async fn send_async( - &self, - request: PlatformHttpRequest, - ) -> Result> { + fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; let (parts, body) = request.request.into_parts(); let headers = parts @@ -369,6 +539,35 @@ mod tests { .lock() .expect("should lock recorded requests") .push(recorded); + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for RecordingHttpClient { + async fn send( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); + let status = *self + .respond_status + .lock() + .expect("should lock configured response status"); + let response = edgezero_core::http::response_builder() + .status( + edgezero_core::http::StatusCode::from_u16(status) + .expect("should build a valid test status code"), + ) + .body(edgezero_core::body::Body::empty()) + .expect("should build test response"); + Ok(PlatformResponse::new(response)) + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); Ok(PlatformPendingRequest::new(()).with_backend_name("tinybird-backend")) } @@ -698,6 +897,117 @@ mod tests { ); } + #[test] + fn access_emitter_posts_ndjson_and_validates_2xx() { + // `ts_secrets`/`tinybird_access_append_token` is seeded in + // fastly.toml's `[local_server.secret_stores]` fixture (value + // "test-tinybird-access-append-token"), so `emit_access_event` can + // load a real token through Viceroy without a secret-store test + // double — the same fixture backs the auction-token secret used + // above. + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(202); + let row = r#"{"status":200}"#.to_owned(); + + futures::executor::block_on(emit_access_event(&http_client, &target, row.clone())) + .expect("should accept a 202 response"); + + let requests = http_client + .requests + .lock() + .expect("should lock recorded requests"); + assert_eq!(requests.len(), 1, "should send exactly one request"); + assert_eq!( + requests[0].uri, + "https://api.us-east.aws.tinybird.co/v0/events?name=access_logs_raw" + ); + assert_eq!(requests[0].method, Method::POST.to_string()); + assert_eq!( + header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), + Some("Bearer test-tinybird-access-append-token") + ); + assert_eq!( + std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), + row, + "should send the row verbatim as the request body" + ); + } + + #[test] + fn access_emitter_warns_and_drops_on_non_2xx() { + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(422); + + let result = futures::executor::block_on(emit_access_event( + &http_client, + &target, + r#"{"status":422}"#.to_owned(), + )); + + let error = result.expect_err("a 422 response should be reported as an error"); + assert!( + error.to_string().contains("422"), + "error should name the failing status: {error}" + ); + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 1, + "should not retry after a non-2xx response" + ); + } + + #[test] + fn sampled_out_requests_emit_nothing() { + // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, + // entropy) { emit_access_event(...) }`): `emit_access_event` is only + // reached when `sampled_in` returns `true`. With a `0.0` rate it + // never does, for any entropy, so the http client should never see + // a request. + let http_client = RecordingHttpClient::respond_with(202); + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let rate = 0.0; + let entropy = 123_456_789_u64; + + if sampled_in(rate, entropy) { + futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) + .expect("should send when sampled in"); + } + + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 0, + "sampled-out requests must never reach emit_access_event" + ); + } + + #[test] + fn sampled_in_boundary_rates_are_unconditional() { + assert!( + sampled_in(1.0, 0), + "a 1.0 sample rate should always sample in" + ); + assert!( + sampled_in(1.0, u64::MAX), + "a 1.0 sample rate should always sample in regardless of entropy" + ); + assert!( + !sampled_in(0.0, 0), + "a 0.0 sample rate should never sample in" + ); + assert!( + !sampled_in(0.0, u64::MAX), + "a 0.0 sample rate should never sample in regardless of entropy" + ); + } + fn header_value<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> { headers .iter() From 72d57557827453d18409494e4301cc7fd6fe3734 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:10:07 -0700 Subject: [PATCH 011/230] Extend access_logs_raw with phase columns and a non-null sorting key --- .../datasources/access_logs_raw.datasource | 28 +++++++++++++++---- tinybird/fixtures/access_logs_raw.ndjson | 1 + 2 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 tinybird/fixtures/access_logs_raw.ndjson diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 42f214e07..a484964db 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -1,19 +1,37 @@ DESCRIPTION > - Optional sampled Trusted Server access telemetry rows. Disabled by default in Fastly config. + Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), - `path` String, `status` UInt16, `time_elapsed_ms` UInt32, - `cache_state` LowCardinality(Nullable(String)), - `country` LowCardinality(String), `sample_rate` Float64, + `service_id` LowCardinality(String), + `publisher_domain` LowCardinality(String), + `env` LowCardinality(String), + `route_class` LowCardinality(String), + `route_template` String, + `body_mode` LowCardinality(String), + `auction_wait_placement` LowCardinality(String), + `appbuild_ms` Nullable(UInt32), + `filter_ms` Nullable(UInt32), + `geo_ms` Nullable(UInt32), + `kv_ms` Nullable(UInt32), + `origin_ms` Nullable(UInt32), + `template_cache_ms` Nullable(UInt32), + `auction_wait_ms` Nullable(UInt32), + `stream_ms` Nullable(UInt32), + `request_elapsed_ms` Nullable(UInt32), + `resp_bytes` Nullable(UInt64), + `template_cache_state` LowCardinality(String), + `country` LowCardinality(String), + `ts_version` LowCardinality(String), + `pop` LowCardinality(String), `event_date` Date DEFAULT toDate(event_ts) ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, path, status, method" +ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" TTL "event_date + INTERVAL 30 DAY" TOKEN ts_access_ingest APPEND diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson new file mode 100644 index 000000000..3c82c5ca6 --- /dev/null +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -0,0 +1 @@ +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From c50be0323d258cae2b449dd4cb7ea98e232e115d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:17:02 -0700 Subject: [PATCH 012/230] Widen time_elapsed_ms to nullable so dropped snapshots cannot quarantine rows --- .../specs/2026-08-24-request-phase-timing-design.md | 3 ++- tinybird/datasources/access_logs_raw.datasource | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 06f27c1c2..d9bbd4bdd 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -275,7 +275,8 @@ Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the -`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. +`mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the +snapshot), `sample_rate`, `event_date`, 30-day TTL. Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index a484964db..918f3d5fd 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -5,7 +5,7 @@ SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), `status` UInt16, - `time_elapsed_ms` UInt32, + `time_elapsed_ms` Nullable(UInt32), `sample_rate` Float64, `service_id` LowCardinality(String), `publisher_domain` LowCardinality(String), From 0aead79727d81cde132f03be94a022db9ef18cb8 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 12:24:22 -0700 Subject: [PATCH 013/230] Emit Server-Timing from the Axum terminal layer with adapter-specific semantics --- crates/trusted-server-adapter-axum/Cargo.toml | 6 +- crates/trusted-server-adapter-axum/src/app.rs | 32 ++- crates/trusted-server-adapter-axum/src/lib.rs | 3 + .../trusted-server-adapter-axum/src/main.rs | 71 ++++- .../trusted-server-adapter-axum/src/timing.rs | 242 ++++++++++++++++++ .../trusted-server-adapter-fastly/src/main.rs | 51 +--- .../trusted-server-core/src/request_timing.rs | 91 +++++++ 7 files changed, 438 insertions(+), 58 deletions(-) create mode 100644 crates/trusted-server-adapter-axum/src/timing.rs diff --git a/crates/trusted-server-adapter-axum/Cargo.toml b/crates/trusted-server-adapter-axum/Cargo.toml index 15b6ee59d..09e8c77d2 100644 --- a/crates/trusted-server-adapter-axum/Cargo.toml +++ b/crates/trusted-server-adapter-axum/Cargo.toml @@ -20,6 +20,7 @@ path = "src/main.rs" [dependencies] async-trait = { workspace = true } +axum = { workspace = true } edgezero-adapter-axum = { workspace = true, features = ["axum"] } edgezero-core = { workspace = true } error-stack = { workspace = true } @@ -27,12 +28,11 @@ futures = { workspace = true } log = { workspace = true } reqwest = { workspace = true } simple_logger = { workspace = true } -tokio = { workspace = true, features = ["rt-multi-thread", "macros", "sync", "time"] } +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "net", "signal", "sync", "time"] } +tower = { workspace = true, features = ["util"] } trusted-server-core = { workspace = true } [dev-dependencies] -axum = { workspace = true } base64 = { workspace = true } temp-env = { workspace = true } tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } -tower = { workspace = true, features = ["util"] } diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4b71d07ce..4bdf27d01 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -565,15 +565,7 @@ impl Hooks for TrustedServerApp { } fn routes() -> RouterService { - let state = match build_state() { - Ok(s) => s, - Err(ref e) => { - log::error!("failed to build application state: {:?}", e); - return startup_error_router(e); - } - }; - - build_router(&state) + Self::routes_with_server_timing_flag().0 } } @@ -594,6 +586,28 @@ impl TrustedServerApp { let state = build_state_with_settings(settings)?; Ok(build_router(&state)) } + + /// Build the router alongside whether `Server-Timing` emission is + /// enabled, read from the same settings snapshot used to build the + /// router. + /// + /// The Axum dev server's terminal timing layer ([`crate::timing`]) needs + /// this flag once at startup: unlike the Fastly adapter, which rebuilds + /// `Settings` per request, the Axum dev server builds its application + /// state once and reuses the same [`RouterService`] for every request. + #[must_use] + pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + let state = match build_state() { + Ok(s) => s, + Err(ref e) => { + log::error!("failed to build application state: {:?}", e); + return (startup_error_router(e), false); + } + }; + + let server_timing_enabled = state.settings.observability.server_timing_enabled; + (build_router(&state), server_timing_enabled) + } } fn build_router(state: &Arc) -> RouterService { diff --git a/crates/trusted-server-adapter-axum/src/lib.rs b/crates/trusted-server-adapter-axum/src/lib.rs index 2f15e566d..b1d4c3dd8 100644 --- a/crates/trusted-server-adapter-axum/src/lib.rs +++ b/crates/trusted-server-adapter-axum/src/lib.rs @@ -10,3 +10,6 @@ pub mod app; pub mod middleware; /// Platform-trait implementations backed by env vars and `reqwest`. pub mod platform; +/// Terminal timing layer wrapping the Axum dev server's tower `Service` +/// boundary with the request-phase `Server-Timing` freeze point. +pub mod timing; diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index 960982176..b8bc28ae2 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -1,6 +1,16 @@ -use edgezero_adapter_axum::dev_server::{AxumDevServer, AxumDevServerConfig}; -use edgezero_core::app::Hooks as _; +use std::net::SocketAddr; + +use axum::Router; +use edgezero_adapter_axum::dev_server::AxumDevServerConfig; +use edgezero_adapter_axum::service::EdgeZeroAxumService; +use edgezero_core::router::RouterService; +use tokio::net::TcpListener; +use tokio::runtime::Builder as RuntimeBuilder; +use tokio::signal; +use tower::Service as _; +use tower::service_fn; use trusted_server_adapter_axum::app::TrustedServerApp; +use trusted_server_adapter_axum::timing::TimingService; #[allow(clippy::print_stderr)] fn main() { @@ -20,13 +30,66 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let router = TrustedServerApp::routes(); - if let Err(err) = AxumDevServer::with_config(router, config).run() { + let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); + if let Err(err) = run(router, server_timing_enabled, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } } +/// Runs the Axum dev server with the request-phase timing terminal layer +/// ([`trusted_server_adapter_axum::timing::TimingService`]) wrapped around +/// `EdgeZeroAxumService`, ahead of `axum::serve`. +/// +/// This does not use `edgezero_adapter_axum::dev_server::AxumDevServer::run`: +/// that helper only accepts a bare [`RouterService`] and builds its own +/// `EdgeZeroAxumService` and `axum::Router` internally, with no seam for an +/// outer service wrapper. Router-generated 404/405 responses bypass +/// `RouterBuilder::middleware` (see `trusted_server_adapter_axum::timing`), +/// so the freeze point has to wrap the tower `Service` boundary itself. +/// Driving `axum::serve` directly here mirrors that helper's own internal +/// bind/wrap/serve/shutdown sequence closely enough to keep behavior +/// identical for callers (`PORT` env var, ctrl-c graceful shutdown). +/// +/// # Errors +/// +/// Returns an error if the Tokio runtime fails to start, the listener fails +/// to bind, or the underlying serve loop errors. +fn run( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; + runtime.block_on(serve(router, server_timing_enabled, config)) +} + +async fn serve( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let listener = TcpListener::bind(config.addr).await?; + + let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); + let axum_router = Router::new().fallback_service(service_fn(move |req| { + let mut svc = service.clone(); + async move { svc.call(req).await } + })); + let make_service = axum_router.into_make_service_with_connect_info::(); + + let server = axum::serve(listener, make_service); + if config.enable_ctrl_c { + server + .with_graceful_shutdown(async { + let _ctrl_c = signal::ctrl_c().await; + }) + .await + } else { + server.await + } +} + /// Read a port number from the `PORT` environment variable. /// /// Returns `None` when the variable is unset. Exits non-zero if the value diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs new file mode 100644 index 000000000..832823c15 --- /dev/null +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -0,0 +1,242 @@ +//! Terminal timing layer for the Axum dev server. +//! +//! [`TimingService`](crate::timing::TimingService) wraps the tower `Service` +//! boundary the Axum dev server's router sits behind: it creates a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector per request, threads it through request extensions so +//! downstream core handlers can record into it, and on the way back stamps +//! `mark_headers_ready` and appends the `Server-Timing` header via +//! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). +//! +//! This wraps *outside* `RouterService` rather than registering as +//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits +//! `RouterInner::dispatch` before its middleware chain ever runs, so +//! middleware never sees those responses. By the time a response reaches +//! this layer -- after `RouterService::oneshot` inside +//! `EdgeZeroAxumService::call` has already converted any dispatch error into +//! a plain response -- every response is covered uniformly, router-generated +//! or not. +//! +//! `/health` is excluded by path match before a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector is even created: health checks never carry timing data on any +//! adapter. +//! +//! Unlike the Fastly adapter (state built per request, adding +//! `Phase::AppBuild` to the rendered header), the Axum dev server builds its +//! application state once at startup. There is no per-request app-build +//! interval to measure, so `ts-appbuild` never appears in the header here. + +use std::convert::Infallible; +use std::future::Future; +use std::pin::Pin; +use std::task::{Context, Poll}; + +use axum::body::Body as AxumBody; +use axum::http::{Request, Response}; +use tower::Service; +use trusted_server_core::request_timing::{RequestTimings, append_server_timing_if_private}; + +/// Path excluded from timing collection and `Server-Timing` emission: health +/// checks never carry timing data on any adapter. +const HEALTH_PATH: &str = "/health"; + +/// Wraps an inner Axum tower service with the request-phase timing freeze +/// point described in the module docs. +#[derive(Clone)] +pub struct TimingService { + inner: S, + server_timing_enabled: bool, +} + +impl TimingService { + /// Wraps `inner`, appending `Server-Timing` when `server_timing_enabled` + /// is set and the response is conclusively private. + #[must_use] + pub fn new(inner: S, server_timing_enabled: bool) -> Self { + Self { + inner, + server_timing_enabled, + } + } +} + +impl Service> for TimingService +where + S: Service, Response = Response, Error = Infallible> + + Clone + + Send + + 'static, + S::Future: Send + 'static, +{ + type Error = Infallible; + type Future = Pin> + Send>>; + type Response = Response; + + fn call(&mut self, mut req: Request) -> Self::Future { + let mut inner = self.inner.clone(); + + // Excluded before a collector is even created: `/health` never + // carries timing data, on any adapter. + if req.uri().path() == HEALTH_PATH { + return Box::pin(async move { inner.call(req).await }); + } + + let server_timing_enabled = self.server_timing_enabled; + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + Box::pin(async move { + let mut response = inner.call(req).await?; + append_server_timing_if_private(&mut response, &timings, server_timing_enabled); + Ok(response) + }) + } + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use axum::http::header::CACHE_CONTROL; + use axum::http::{HeaderValue, StatusCode}; + use edgezero_adapter_axum::service::EdgeZeroAxumService; + use edgezero_core::body::Body as EdgeBody; + use edgezero_core::context::RequestContext; + use edgezero_core::error::EdgeError; + use edgezero_core::http::response_builder; + use edgezero_core::router::RouterService; + use tower::{ServiceExt as _, service_fn}; + + /// Builds a private (`cache-control: private, no-store`) response for a + /// handler under test. + fn private_ok_response() -> Result { + Ok(response_builder() + .status(StatusCode::OK) + .header("cache-control", "private, no-store") + .body(EdgeBody::from("ok")) + .expect("should build a private response fixture")) + } + + /// Reads a response header as a UTF-8 string, or `None` if absent. + fn header(response: &Response, name: &str) -> Option { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_emits_header_on_private_response() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + assert!( + !server_timing.contains("ts-appbuild"), + "the Axum dev server builds state once at startup, so there is no \ + per-request app-build interval to render: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_404_carries_header_when_private() { + // An empty router has no routes at all, so any path dispatches + // through `RouterInner::dispatch`'s `NotFound` branch -- exactly the + // path that bypasses `RouterBuilder::middleware`. The router's own + // `EdgeError::into_response` does not attach `Cache-Control`, so a + // small wrapping service forces the response private here, standing + // in for whatever upstream layer would normally mark a genuinely + // private 404. This proves the freeze point still runs for a + // router-generated response without weakening + // `append_server_timing_if_private`'s real gating logic. + let empty_router = RouterService::builder().build(); + let inner = EdgeZeroAxumService::new(empty_router); + let force_private = service_fn(move |req: Request| { + let mut svc = inner.clone(); + async move { + let mut response = svc.call(req).await?; + response + .headers_mut() + .insert(CACHE_CONTROL, HeaderValue::from_static("private, no-store")); + Ok::<_, Infallible>(response) + } + }); + let mut service = TimingService::new(force_private, true); + + let request = Request::builder() + .uri("/does-not-exist") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should still be the router's own not-found response" + ); + let server_timing = header(&response, "server-timing") + .expect("a router-generated 404 must still carry the header when private"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_health_is_excluded() { + let router = RouterService::builder() + .get("/health", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/health") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "/health must never carry a server-timing header" + ); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 65e89cd1c..07c15b8b8 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -6,9 +6,7 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{ - HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, -}; +use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; @@ -17,9 +15,7 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::access_telemetry::{ AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, }; -use trusted_server_core::cache_policy::{ - EdgeCacheHeader, cache_control_headers_are_private_or_no_store, -}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, }; @@ -37,7 +33,7 @@ use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::publisher::TemplateCacheResponseState; -use trusted_server_core::request_timing::{Phase, RequestTimings}; +use trusted_server_core::request_timing::{Phase, RequestTimings, append_server_timing_if_private}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -62,12 +58,6 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; -/// `Server-Timing` header name. Not present in the `http` crate's `header` -/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the -/// same `HeaderName::from_static` pattern used in -/// `trusted_server_core::constants`. -const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); - /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -544,40 +534,17 @@ pub(crate) enum DeliveryResult { Error, } -/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is -/// enabled and the response is conclusively private, appends the rendered -/// `Server-Timing` header. -/// -/// Always stamps `mark_headers_ready` regardless of whether the header is -/// rendered, so the collector's `ts-total` reflects the moment headers -/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` -/// value set upstream survives alongside the TS-owned set. A response is -/// never promoted to shared-cacheable just because the header would -/// otherwise be omitted: this only gates emission, it does not touch -/// `Cache-Control`. +/// Thin Fastly-adapter wrapper around +/// [`append_server_timing_if_private`], the freeze point shared with the +/// Axum adapter's terminal timing layer. See that function's doc for the +/// emission rules (always stamps `mark_headers_ready`; appends rather than +/// overwrites; never promotes a response to shared-cacheable). pub(crate) fn apply_server_timing_header( response: &mut HttpResponse, timings: &RequestTimings, server_timing_enabled: bool, ) { - timings.mark_headers_ready(); - - let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); - if !server_timing_enabled || !conclusively_private { - return; - } - - let Some(value) = timings.server_timing_value() else { - return; - }; - match HeaderValue::from_str(&value) { - Ok(header_value) => { - response - .headers_mut() - .append(HEADER_SERVER_TIMING, header_value); - } - Err(error) => log::warn!("skipping server-timing header: {error}"), - } + append_server_timing_if_private(response, timings, server_timing_enabled); } /// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index b0cb7b0dd..ccd393555 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -7,6 +7,16 @@ use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; +use http::{HeaderName, HeaderValue, Response}; + +use crate::cache_policy::cache_control_headers_are_private_or_no_store; + +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Number of [`Phase`] variants; sizes the fixed-slot duration array in /// [`Inner`]. const PHASE_COUNT: usize = 8; @@ -253,6 +263,46 @@ impl Default for RequestTimings { } } +/// Stamps [`RequestTimings::mark_headers_ready`] and, when `enabled` and the +/// response is conclusively private, appends the rendered `Server-Timing` +/// header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +/// +/// Generic over the response body type so every adapter's terminal layer can +/// call the same emission logic regardless of which body type its HTTP stack +/// uses. +pub fn append_server_timing_if_private( + response: &mut Response, + timings: &RequestTimings, + enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// The header-bearing phases (see [`Phase::header_name`]), in the enum /// declaration order [`RequestTimings::server_timing_value`] renders them in. const HEADER_PHASES: [Phase; 6] = [ @@ -434,4 +484,45 @@ mod tests { "should mask vendors" ); } + + #[test] + fn append_server_timing_emits_on_private_response_when_enabled() { + let mut response = Response::builder() + .header("cache-control", "private, no-store") + .body(()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|value| value.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.starts_with("ts-total;dur="), + "should lead with the stored total: {header}" + ); + } + + #[test] + fn append_server_timing_marks_headers_ready_even_when_not_emitted() { + let mut response = Response::builder() + .header("cache-control", "max-age=60") + .body(()) + .expect("should build a shared-cacheable response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + assert!( + response.headers().get("server-timing").is_none(), + "should not emit on a shared-cacheable response" + ); + assert!( + timings.server_timing_value().is_some(), + "should still stamp mark_headers_ready so ts-total reflects the freeze point" + ); + } } From e9cf5b97a3677e734184b2604abc7df325e37ec5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 15:50:12 -0700 Subject: [PATCH 014/230] Document the observability and access telemetry configuration surface --- docs/guide/configuration.md | 126 ++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a1f172429..675aaf40a 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -77,6 +77,8 @@ fail and the service will return its startup-error response. | `[request_signing]` | Ed25519 request signing | | `[auction]` | Auction orchestration | | `[integrations.*]` | Partner integrations (Prebid, Next.js, etc.) | +| `[observability]` | Server-Timing header emission | +| `[tinybird]` | Auction and access telemetry transport | ## Example: Production Setup @@ -1805,6 +1807,130 @@ Rollback to the legacy entry point is no longer controlled by runtime config keys. Use the normal deployment rollback path to restore a pre-cleanup service version if that is required. +## Observability and Access Telemetry Configuration + +Settings for the `Server-Timing` response header and the sampled +access-telemetry sink. Both are off by default and are independent switches: +enabling one does not enable the other. + +### `[observability]` + +| Field | Type | Required | Default | Description | +| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | + +**Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded +phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and +`ts-template-cache`) as a standard `Server-Timing` header, in milliseconds +with one decimal place. An unrecorded phase is omitted from the header +rather than rendered as zero. + +**Emission is conservative**: the header is appended only on responses that +are conclusively private, meaning `Cache-Control` contains `private` or +`no-store`. A response that is heuristically cacheable, carries a bare +`max-age`, or has no cache header at all never receives the header, because a +shared-cache object would otherwise replay one request's timings for its +entire stored lifetime. The long-lived, shared-cacheable `tsjs` asset route is +the concrete case this excludes. The header is appended, never inserted, so +an origin-supplied `Server-Timing` value and any entries the fronting +delivery layer adds are preserved alongside the TS entries. + +The Axum adapter applies the same private-response rule at its own terminal +point before serializing the response, and emits the header only; it does not +send access-telemetry rows. + +**Example**: + +```toml +[observability] +server_timing_enabled = true +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED=true +``` + +::: tip Present-but-false by default +`server_timing_enabled` ships as `false` in the base operator config rather +than being left out, even though `false` is also its default. The +environment-variable overlay can only override a leaf that already exists in +the parsed TOML; it cannot create a missing one. Keeping the leaf present lets +`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED` take effect without an +extra edit to add the table first. +::: + +### `[tinybird]` access telemetry keys + +`[tinybird]` configures a shared Events API transport (`enabled`, `api_host`, +`secret_store`, and per-sink dataset and token fields) used by two +independent emitters: auction telemetry (`auction_dataset`, +`auction_token_secret`) and access telemetry. The keys below cover the +access-telemetry sink and the shared enable flags. + +| Field | Type | Required | Default | Description | +| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | + +**Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird +sinks separately so that turning on one does not silently turn on (or leave +off) the other; a settings test locks this decoupling in both directions. +Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at +config load as an armed-but-silent configuration; use `access_enabled` itself +to turn the sink off, not the sample rate. Enabling `access_enabled` also +requires the shared transport fields (`enabled`, non-empty `api_host`, +`secret_store`, `access_dataset`, `access_token_secret`, and a positive +`max_body_bytes`) to already be set. + +**Example**: + +```toml +[tinybird] +enabled = true +api_host = "api.tinybird.example.com" +secret_store = "ts_secrets" +auction_enabled = true + +# Access-log telemetry, decoupled from auction emission. +access_enabled = true +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +access_sample_rate = 0.05 +max_body_bytes = 1048576 +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__TINYBIRD__ACCESS_ENABLED=true +TRUSTED_SERVER__TINYBIRD__ACCESS_SAMPLE_RATE=0.05 +TRUSTED_SERVER__TINYBIRD__AUCTION_ENABLED=true +``` + +A sampled request emits one access-telemetry row to `access_dataset` after +the response has already been delivered to the client, so ingest never delays +the response the reader sees. + +### Deploy and rollback ordering + +::: warning `Settings` rejects unknown fields; order matters +Both `[observability].server_timing_enabled` and the new `[tinybird]` access +keys are new fields on a config schema that uses `deny_unknown_fields`, so an +older binary fails to load a config that carries them. + +**Deploying**: upgrade the binary first, then push a config containing the +new fields second. Never push a config with these fields while a +pre-observability binary can still receive it. + +**Rolling back**: reverse the order. Remove the `[observability]` table and +any new `[tinybird]` access keys from the config and push that first, then +roll back the binary second. +::: + ## Validation ### Automatic Validation From 7942c74aa6c2a298fed80dfade8ee3059c34cc7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 18:45:28 -0700 Subject: [PATCH 015/230] Normalize telemetry method, guard zero sample rate, and mirror geo write-back in middleware Three final-review fixes for access telemetry correctness: - Normalize the HTTP method to an allowlist (GET/HEAD/POST/PUT/DELETE/ PATCH/OPTIONS, else "other") inside access_event_row, so a client- controlled extension-method token can never inflate the LowCardinality method column, regardless of which adapter builds the row. - Guard emit_access_telemetry_after_send against snapshots carrying a degraded sample_rate of 0.0 (captured on the app-state-build-failure fallback path), which could otherwise be sampled in by freshly reloaded settings and corrupt the sum(1.0/sample_rate) volume estimator. - Mirror the geo lookup write-back from apply_entry_point_finalize_headers into FinalizeResponseMiddleware::handle, so a middleware-finalized response that resolved geo via fallback carries the resolved GeoLookupState for the access-telemetry snapshot instead of showing country "unknown". --- .../trusted-server-adapter-fastly/src/main.rs | 88 ++++++++++++++++++- .../src/middleware.rs | 73 +++++++++++++++ .../src/access_telemetry.rs | 79 ++++++++++++++++- 3 files changed, 235 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 07c15b8b8..ec2370272 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -452,9 +452,12 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. Every other drop (row build, token load, -/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) -/// logs exactly one warning naming the reason. +/// case and not worth a log line. A snapshot carrying a degraded +/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, +/// since it only occurs on an already-degraded path. Every other drop (row +/// build, token load, send, or non-2xx status — all folded into +/// `emit_access_event`'s `Result`) logs exactly one warning naming the +/// reason. fn emit_access_telemetry_after_send( settings: &Settings, outcome: &DeliveryOutcome, @@ -477,7 +480,11 @@ fn emit_access_telemetry_after_send( let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); let entropy = entropy_nanos ^ outcome.bytes; - if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + if !should_sample_access_row( + outcome.snapshot.sample_rate, + settings.tinybird.access_sample_rate, + entropy, + ) { return; } @@ -493,6 +500,39 @@ fn emit_access_telemetry_after_send( } } +/// Whether one response's access-telemetry row should be emitted, combining +/// the degraded-snapshot guard with the sampling roll. +/// +/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] +/// itself (the value serialized into the row's `sample_rate` column, which +/// the documented volume estimator divides by as `1.0 / sample_rate`). +/// `settings_sample_rate` is the rate used for the sampling decision at +/// call time. The two can diverge: when `app_state` fails to build, +/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to +/// `0.0` before any settings ever load, but the two settings-reload +/// emission sites still gate and sample using the *reloaded* settings' +/// (nonzero) rate. Without this guard, such a row could be sampled in and +/// emitted while carrying `sample_rate: 0.0`, corrupting the volume +/// estimator. Dropping these rows is acceptable: they only occur on an +/// already-degraded path, consistent with this pipeline's fail-quiet +/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so +/// the guard is unit-testable without a network seam. +/// +/// Callers must already have applied the coarse +/// `tinybird.enabled`/`access_enabled` gate. +#[must_use] +fn should_sample_access_row( + snapshot_sample_rate: f64, + settings_sample_rate: f64, + entropy: u64, +) -> bool { + if snapshot_sample_rate <= 0.0 { + return false; + } + + tinybird::sampled_in(settings_sample_rate, entropy) +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -1818,4 +1858,44 @@ mod tests { "pull-sync must dispatch before telemetry emits" ); } + + #[test] + fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { + // A snapshot captured on the app-state-build-failure fallback path + // carries `sample_rate: 0.0`. Even when the reloaded settings' rate + // would sample every request in (1.0), the row must not emit — + // otherwise it would claim `sample_rate: 0.0` and corrupt the + // `sum(1.0 / sample_rate)` volume estimator. + assert!( + !should_sample_access_row(0.0, 1.0, 0), + "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" + ); + assert!( + !should_sample_access_row(0.0, 1.0, u64::MAX), + "the degraded-rate guard must not depend on the entropy value" + ); + } + + #[test] + fn should_sample_access_row_rejects_a_negative_sample_rate() { + assert!( + !should_sample_access_row(-1.0, 1.0, 0), + "a negative snapshot sample_rate is equally degraded and must not emit" + ); + } + + #[test] + fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { + // With a healthy (nonzero) snapshot sample_rate, the outcome should + // match `tinybird::sampled_in` exactly, since that is the only + // remaining decision. + assert!( + should_sample_access_row(0.25, 1.0, 0), + "a settings rate of 1.0 always samples in, independent of entropy" + ); + assert!( + !should_sample_access_row(0.25, 0.0, 0), + "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 17ecc13a4..ae1efe7c3 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -97,6 +97,18 @@ impl Middleware for FinalizeResponseMiddleware { }) }); + // Write the resolved outcome back so a downstream access-telemetry + // snapshot (built from response extensions after finalize) sees + // what was actually looked up here rather than the stale carried-in + // state — mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`), which writes back for the + // same reason. + let resolved_state = match &geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); + apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response .headers_mut() @@ -594,6 +606,67 @@ mod tests { ); } + #[test] + fn finalize_handle_writes_back_resolved_geo_state_after_fallback_lookup() { + // The request phase never attempted a geo lookup (no GeoLookupState + // extension on the handler's response), so the middleware resolves + // one via the fallback closure. That resolved outcome must be + // written back into response extensions -- mirroring + // apply_entry_point_finalize_headers in main.rs -- so a downstream + // access-telemetry snapshot sees the freshly resolved country + // instead of a stale/missing GeoLookupState. + let settings = settings_with_response_headers(vec![]); + let middleware = FinalizeResponseMiddleware::new( + Arc::new(settings), + Arc::new(FixedGeo(Some(sample_geo_info()))), + ); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => { + assert_eq!( + info.country, "US", + "should carry the fallback-resolved geo info" + ); + } + other => { + panic!("expected GeoLookupState::Resolved after a fallback lookup, got {other:?}") + } + } + } + + #[test] + fn finalize_handle_writes_back_attempted_geo_state_when_fallback_finds_nothing() { + // The fallback lookup ran but resolved no geo info. The middleware + // must still record that the lookup was attempted, so a later + // consumer of the extension does not mistake this for + // GeoLookupState::NotAttempted and retry the lookup. + let settings = settings_with_response_headers(vec![]); + let middleware = + FinalizeResponseMiddleware::new(Arc::new(settings), Arc::new(FixedGeo(None))); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should write back Attempted when the fallback lookup finds no geo info" + ); + } + #[test] fn finalize_handle_marks_response_as_finalized() { let settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 9daf77abc..1642a1334 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -16,6 +16,42 @@ use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; /// by [`publisher_route_template`]. const MAX_SEGMENT_LEN: usize = 32; +/// Normalizes an HTTP method token into the bounded set of values stored in +/// the `method` `LowCardinality` column. +/// +/// HTTP permits arbitrary extension-method tokens (`PROPFIND`, `MKCOL`, or +/// any client-supplied garbage), and the token on an inbound request is +/// entirely client controlled. Capturing one verbatim into a 30-day +/// `LowCardinality(String)` column would let a single caller inflate that +/// column's cardinality without bound and would violate this dataset's +/// bounded-dimension privacy rule (see the module doc). Every standard +/// method maps to its uppercase form; anything else maps to `"other"`. Runs +/// inside [`access_event_row`] rather than at each capture site, so every +/// row-building path is covered regardless of how `method` was populated. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::normalize_method; +/// +/// assert_eq!(normalize_method("get"), "GET"); +/// assert_eq!(normalize_method("PROPFIND"), "other"); +/// assert_eq!(normalize_method(""), + "other", + "an unbounded client-controlled token must not reach the row verbatim" + ); + } + + #[test] + fn row_normalizes_method_even_when_snapshot_carries_a_raw_token() { + // The normalizer runs inside `access_event_row` so every row-building + // path is covered, regardless of what the snapshot's `method` field + // holds — a caller-controlled extension method must never leak into + // the row unnormalized. + let mut snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + snapshot.method = "PROPFIND".to_owned(); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["method"], "other"); + } } From d8fcb5e933e0cc8f0df15e98613dfce43636bd0f Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 19:51:06 -0700 Subject: [PATCH 016/230] Add a local dev config envelope generator example --- .../examples/local_dev_config.rs | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 crates/trusted-server-core/examples/local_dev_config.rs diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs new file mode 100644 index 000000000..19826231d --- /dev/null +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -0,0 +1,126 @@ +//! Generate a ready-to-use local dev config envelope for the Axum adapter. +//! +//! Reads `trusted-server.example.toml`, replaces the placeholder secrets with +//! random values, flips the flags a local smoke test needs, validates the +//! result through [`trusted_server_core::settings::Settings::from_toml`], and +//! prints the blob envelope JSON that +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! +//! The random values are time-and-pid seeded, not cryptographic. This tool +//! exists for throwaway local test instances only; never use its output for a +//! deployed service. +//! +//! Usage: +//! +//! ```text +//! cargo run -p trusted-server-core --example local_dev_config \ +//! --target -- [origin-url] [--realistic] +//! ``` +//! +//! `origin-url` defaults to `https://www.example.com`. By default every +//! response is forced `Cache-Control: private, no-store` so the Server-Timing +//! header is visible on all routes; pass `--realistic` to keep the origin's +//! own cache policy instead. + +use std::time::{SystemTime, UNIX_EPOCH}; + +/// Deliberately non-cryptographic generator for local placeholder secrets. +struct WeakRandom(u64); + +impl WeakRandom { + fn from_environment() -> Self { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .subsec_nanos() as u64; + let secs = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs(); + let pid = std::process::id() as u64; + Self(nanos ^ (secs << 20) ^ (pid << 40) ^ 0x9e37_79b9_7f4a_7c15) + } + + fn next(&mut self) -> u64 { + let mut x = self.0; + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + self.0 = x; + x + } + + fn hex(&mut self, chars: usize) -> String { + let mut out = String::with_capacity(chars); + while out.len() < chars { + out.push_str(&format!("{:016x}", self.next())); + } + out.truncate(chars); + out + } +} + +#[allow(clippy::print_stdout, clippy::print_stderr)] +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + let realistic = args.iter().any(|a| a == "--realistic"); + let origin = args + .iter() + .find(|a| !a.starts_with("--")) + .cloned() + .unwrap_or_else(|| "https://www.example.com".to_string()); + + let template = std::fs::read_to_string("trusted-server.example.toml") + .expect("should read trusted-server.example.toml from the repo root"); + + let mut random = WeakRandom::from_environment(); + let mut config = template + .replace( + "password = \"replace-with-admin-password-32-bytes\"", + &format!("password = \"{}\"", random.hex(48)), + ) + .replace( + "proxy_secret = \"change-me-proxy-secret\"", + &format!("proxy_secret = \"{}\"", random.hex(48)), + ) + .replace( + "passphrase = \"trusted-server-placeholder-secret\"", + &format!("passphrase = \"{}\"", random.hex(48)), + ) + .replace( + "server_timing_enabled = false", + "server_timing_enabled = true", + ); + + let origin_line = config + .lines() + .find(|line| line.starts_with("origin_url = ")) + .expect("should find the origin_url line in the template") + .to_string(); + config = config.replace(&origin_line, &format!("origin_url = \"{origin}\"")); + + if !realistic { + config = config.replace( + "# [response_headers]", + "[response_headers]\n\"Cache-Control\" = \"private, no-store\"", + ); + } + + let settings = trusted_server_core::settings::Settings::from_toml(&config) + .expect("should validate the generated local config"); + let data = serde_json::to_value(&settings).expect("should serialize settings"); + let generated_at = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs() + .to_string(); + let envelope = edgezero_core::blob_envelope::BlobEnvelope::new(data, generated_at); + println!( + "{}", + serde_json::to_string(&envelope).expect("should serialize the envelope") + ); + eprintln!( + "local dev envelope generated: origin={origin} force_private={}", + !realistic + ); +} From 7cf7d86712c996e80ce1cd4ae436df7974a69732 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 20:41:15 -0700 Subject: [PATCH 017/230] Add JSONPaths and expression sorting key to the access datasource --- .../2026-08-24-request-phase-timing-design.md | 7 ++- .../datasources/access_logs_raw.datasource | 60 ++++++++++--------- 2 files changed, 36 insertions(+), 31 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d9bbd4bdd..d8791a799 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -340,8 +340,11 @@ guest-visible cache behavior is already carried by `template_cache_state` and `origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are invisible by construction and the dashboard documentation says so. -Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, -status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +Sorting key: `(toDate(event_ts), service_id, publisher_domain, env, route_class, +pop, status)`. Every column carries a `json:$.` path (the Events API rejects +NDJSON into a datasource without JSONPaths, discovered live); `event_date` was +dropped in favor of the sorting-key expression because a DEFAULT column cannot +carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query also carries an `event_date` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 918f3d5fd..062b884ba 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -2,36 +2,38 @@ DESCRIPTION > Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > - `event_ts` DateTime64(3), - `method` LowCardinality(String), - `status` UInt16, - `time_elapsed_ms` Nullable(UInt32), - `sample_rate` Float64, - `service_id` LowCardinality(String), - `publisher_domain` LowCardinality(String), - `env` LowCardinality(String), - `route_class` LowCardinality(String), - `route_template` String, - `body_mode` LowCardinality(String), - `auction_wait_placement` LowCardinality(String), - `appbuild_ms` Nullable(UInt32), - `filter_ms` Nullable(UInt32), - `geo_ms` Nullable(UInt32), - `kv_ms` Nullable(UInt32), - `origin_ms` Nullable(UInt32), - `template_cache_ms` Nullable(UInt32), - `auction_wait_ms` Nullable(UInt32), - `stream_ms` Nullable(UInt32), - `request_elapsed_ms` Nullable(UInt32), - `resp_bytes` Nullable(UInt64), - `template_cache_state` LowCardinality(String), - `country` LowCardinality(String), - `ts_version` LowCardinality(String), - `pop` LowCardinality(String), - `event_date` Date DEFAULT toDate(event_ts) + `event_ts` DateTime64(3) `json:$.event_ts`, + `method` LowCardinality(String) `json:$.method`, + `status` UInt16 `json:$.status`, + `time_elapsed_ms` Nullable(UInt32) `json:$.time_elapsed_ms`, + `sample_rate` Float64 `json:$.sample_rate`, + `service_id` LowCardinality(String) `json:$.service_id`, + `publisher_domain` LowCardinality(String) `json:$.publisher_domain`, + `env` LowCardinality(String) `json:$.env`, + `route_class` LowCardinality(String) `json:$.route_class`, + `route_template` String `json:$.route_template`, + `body_mode` LowCardinality(String) `json:$.body_mode`, + `auction_wait_placement` LowCardinality(String) `json:$.auction_wait_placement`, + `appbuild_ms` Nullable(UInt32) `json:$.appbuild_ms`, + `filter_ms` Nullable(UInt32) `json:$.filter_ms`, + `geo_ms` Nullable(UInt32) `json:$.geo_ms`, + `kv_ms` Nullable(UInt32) `json:$.kv_ms`, + `origin_ms` Nullable(UInt32) `json:$.origin_ms`, + `template_cache_ms` Nullable(UInt32) `json:$.template_cache_ms`, + `auction_wait_ms` Nullable(UInt32) `json:$.auction_wait_ms`, + `stream_ms` Nullable(UInt32) `json:$.stream_ms`, + `request_elapsed_ms` Nullable(UInt32) `json:$.request_elapsed_ms`, + `resp_bytes` Nullable(UInt64) `json:$.resp_bytes`, + `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, + `country` LowCardinality(String) `json:$.country`, + `ts_version` LowCardinality(String) `json:$.ts_version`, + `pop` LowCardinality(String) `json:$.pop` ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" -TTL "event_date + INTERVAL 30 DAY" +ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" +TTL "toDate(event_ts) + INTERVAL 30 DAY" + +FORWARD_QUERY > + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop TOKEN ts_access_ingest APPEND From 600746f957f2a92c49fca93cea4d7d1a9c10de7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:16 -0700 Subject: [PATCH 018/230] Use web_time Instant on request timing paths std::time::Instant::now() panics on wasm32-unknown-unknown, so every publisher request on the Cloudflare adapter trapped when the timing collector was constructed, and the two auction-wait sites would trap once an auction dispatched. web_time re-exports std's Instant on every other target, so Fastly, Axum, and Spin behavior is unchanged. The publisher.rs sites are qualified locally because that module's std Instant import still serves the pre-existing template-cache sites, which are out of scope here. --- crates/trusted-server-core/src/publisher.rs | 10 ++++++++-- crates/trusted-server-core/src/request_timing.rs | 6 +++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 6a30611c8..53d09bfc6 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3937,7 +3937,10 @@ async fn collect_non_html_auction( .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3999,7 +4002,10 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index ccd393555..6ac9a9692 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -5,9 +5,13 @@ //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; +use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; +// `std::time::Instant::now()` panics on `wasm32-unknown-unknown` (the +// Cloudflare adapter's target); `web_time` re-exports std's `Instant` on +// every other target. +use web_time::Instant; use crate::cache_policy::cache_control_headers_are_private_or_no_store; From 3d7e697cad8d1fb6b3206249d66a54e117d40527 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:33 -0700 Subject: [PATCH 019/230] Reject opaque identifier segments in publisher route templates The character allowlist alone does not bound identity: [a-z0-9_-] is exactly the alphabet UUIDs, hex ids, reset tokens, and article slugs are built from, and truncating to 32 characters still leaves a globally unique prefix. A first segment now rejects whole to /other/* when it exceeds 32 characters or carries more than 7 ASCII digits, alongside the existing charset rejection. Year archives and hyphenated section names still pass. Extends the adversarial tests to the publisher-fallback path with UUID, hex-id, token, and slug shapes, and fixes the stale event_date reference in the row-builder doc. --- .../src/access_telemetry.rs | 114 +++++++++++++++--- .../2026-08-24-request-phase-timing-design.md | 11 +- 2 files changed, 105 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 1642a1334..28e391f7e 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,10 +12,20 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum number of characters kept from a publisher path's first segment -/// by [`publisher_route_template`]. +/// Maximum length of a publisher path's first segment before +/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments +/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a +/// truncated prefix of either would still be identifying, so the segment +/// is rejected whole rather than truncated. const MAX_SEGMENT_LEN: usize = 32; +/// Maximum number of ASCII digits in a publisher path's first segment +/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, +/// base36 ids, and reset tokens are digit-heavy; real section names carry +/// at most a year (`2026`) or a small version number, so a segment with +/// more digits than this is treated as an identifier, not a name. +const MAX_SEGMENT_DIGITS: usize = 7; + /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -116,18 +126,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a -/// trailing `/*` appended when the path has additional segments beyond the -/// first. The root path `/` maps to itself. An empty first segment, or one -/// containing any character outside the allowlist (after lowercasing), -/// maps to `/other/*` — the segment is rejected outright rather than -/// filtered, so no fragment of a disallowed segment (an email address, a -/// search phrase) ever reaches the row. +/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has +/// additional segments beyond the first. The root path `/` maps to itself. +/// A first segment is rejected to `/other/*` — outright, never filtered or +/// truncated, so no fragment of it ever reaches the row — when it: +/// +/// - is empty, or contains any character outside the allowlist after +/// lowercasing (an email address, a search phrase); +/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex +/// tokens, and full article slugs all exceed it — a truncated prefix of +/// any of these would still be identifying); or +/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque +/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; +/// publisher section names are words, at most a year or a version +/// number. /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// this broad. Short all-alpha slugs on single-segment paths are +/// indistinguishable from section names and still pass; the bound here is +/// shape-based, not semantic. /// /// # Examples /// @@ -137,6 +156,10 @@ pub struct RouteMetadata { /// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); /// assert_eq!(publisher_route_template("/"), "/"); /// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// assert_eq!( +/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// "/other/*" +/// ); /// ``` #[must_use] pub fn publisher_route_template(path: &str) -> String { @@ -156,16 +179,17 @@ pub fn publisher_route_template(path: &str) -> String { && lowered .chars() .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; + let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - if !is_allowlisted { + if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { return "/other/*".to_owned(); } - let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); if has_more_depth { - format!("/{truncated}/*") + format!("/{lowered}/*") } else { - format!("/{truncated}") + format!("/{lowered}") } } @@ -220,8 +244,9 @@ pub struct AccessTelemetrySnapshot { /// `timings` and serialize as JSON `null` for phases that were never /// recorded; every dimension column comes from `snapshot` and is a /// non-nullable string (callers are expected to substitute an `unknown` -/// sentinel rather than leave a dimension empty). `event_date` is omitted: -/// the datasource derives it from `event_ts` by default. +/// sentinel rather than leave a dimension empty). There is no `event_date` +/// column: the datasource's sorting key derives the date via +/// `toDate(event_ts)`. #[must_use] pub fn access_event_row( snapshot: &AccessTelemetrySnapshot, @@ -338,12 +363,65 @@ mod tests { ); assert_eq!( publisher_route_template(&format!("/{}", "a".repeat(500))), - format!("/{}", "a".repeat(32)), - "should bound segment length" + "/other/*", + "should reject overlong segments whole rather than truncate" ); assert_eq!(publisher_route_template("/search terms here"), "/other/*"); } + #[test] + fn publisher_route_template_rejects_opaque_identifier_segments() { + // Every row here passes the character allowlist (`[a-z0-9_-]` is + // exactly what UUIDs, hex ids, and tokens are built from) and must + // be caught by the length and digit-count bounds instead. A + // truncated prefix of any of these would still be identifying, so + // rejection must be whole-segment. + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), + "/other/*", + "should reject a UUID (36 chars) by length" + ); + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), + "/other/*", + "should reject a UUID first segment on deeper paths too" + ); + assert_eq!( + publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), + "/other/*", + "should reject a 32-char hex id by digit count" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a1".repeat(32))), + "/other/*", + "should reject a 64-char token by length" + ); + assert_eq!( + publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), + "/other/*", + "should reject a reset token by length" + ); + assert_eq!( + publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + "/other/*", + "should reject a full article slug by length" + ); + } + + #[test] + fn publisher_route_template_keeps_digit_light_section_names() { + assert_eq!( + publisher_route_template("/2026/08/some-article"), + "/2026/*", + "a year archive segment should pass the digit bound" + ); + assert_eq!( + publisher_route_template("/wp-content/themes/site/app.css"), + "/wp-content/*", + "a hyphenated section name should pass" + ); + } + #[test] fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d8791a799..214d146a1 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -289,9 +289,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. +- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` + when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 + ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated + prefix of any of those is still identifying, so the length and digit bounds reject + the segment outright. - Tests are adversarial, not just the happy path: a literal EC identifier on the admin - route, an email address in a path segment, search-term-shaped segments, and - overlong segments must all normalize to bounded, content-free templates. + route, an email address in a path segment, search-term-shaped segments, overlong + segments, UUIDs, hex ids, reset tokens, and full article slugs must all normalize + to bounded, content-free templates. Added columns (all dimension columns non-nullable with an `unknown` sentinel, because ClickHouse sorting keys cannot contain nullable columns): From 38043d7464362d44519153a09fe850bacc256b58 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:41 -0700 Subject: [PATCH 020/230] Address access telemetry review feedback - Gate building the access snapshot on tinybird.enabled and access_enabled, threaded through SendContext: a disabled deployment (the default) no longer pays env reads and String allocations on the pre-send path. DeliveryOutcome.snapshot becomes Option and the emitter treats None as nothing to send. - Classify asset-fallback responses as route_class asset with the operator-configured route prefix as the template, instead of landing in the other/unknown bucket alongside 404s. - Pin Phase::index() to PHASE_COUNT with a uniqueness-and-bounds test so a future variant fails the suite instead of panicking at runtime. - Drop the tautological sampled-out emission test; the 0.0-rate behavior is covered by sampled_in_boundary_rates_are_unconditional. - Clarify that the local dev config env var name genuinely triples trusted_server_config (prefix, store, key) rather than reading as a find/replace mistake. --- .../trusted-server-adapter-fastly/src/app.rs | 11 +++- .../trusted-server-adapter-fastly/src/main.rs | 51 +++++++++++++++---- .../src/tinybird.rs | 28 ---------- .../examples/local_dev_config.rs | 7 ++- .../src/access_telemetry.rs | 4 ++ .../trusted-server-core/src/request_timing.rs | 34 +++++++++++++ 6 files changed, 93 insertions(+), 42 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0a0cc8c5a..3a4ceda8f 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -902,7 +902,14 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback( + // The template is the operator-configured route prefix, so it + // is bounded and content-free by construction (unlike request + // paths, which need `publisher_route_template`). + let asset_metadata = RouteMetadata { + route_class: RouteClass::Asset, + route_template: format!("{}/*", asset_route.prefix.trim_end_matches('/')), + }; + let mut response = dispatch_asset_fallback( state, services, req, @@ -911,6 +918,8 @@ async fn dispatch_fallback( ec.geo_lookup_state(), ) .await; + response.extensions_mut().insert(asset_metadata); + return response; } route_metadata = Some(RouteMetadata { diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ec2370272..934f98402 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -150,6 +150,9 @@ fn edgezero_main(mut req: FastlyRequest) { let access_sample_rate = settings_snapshot .as_deref() .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let access_telemetry_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.tinybird.enabled && settings.tinybird.access_enabled); let publisher_domain = settings_snapshot.as_deref().map_or_else( || "unknown".to_owned(), |settings| settings.publisher.domain.clone(), @@ -276,6 +279,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -303,6 +307,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send( @@ -336,6 +341,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method, publisher_domain, access_sample_rate, + access_telemetry_enabled, }, ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec @@ -467,6 +473,12 @@ fn emit_access_telemetry_after_send( return; } + // No snapshot means access telemetry was disabled when the response + // was sent (the flag is read once, before dispatch); nothing to emit. + let Some(snapshot) = &outcome.snapshot else { + return; + }; + let since_epoch = SystemTime::now() .duration_since(UNIX_EPOCH) .unwrap_or_default(); @@ -481,14 +493,14 @@ fn emit_access_telemetry_after_send( let entropy = entropy_nanos ^ outcome.bytes; if !should_sample_access_row( - outcome.snapshot.sample_rate, + snapshot.sample_rate, settings.tinybird.access_sample_rate, entropy, ) { return; } - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let row = access_event_row(snapshot, &timings.snapshot(), epoch_ms); let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); let result = futures::executor::block_on(tinybird::emit_access_event( &platform::FastlyPlatformHttpClient, @@ -547,6 +559,12 @@ struct SendContext { publisher_domain: String, /// The configured access-telemetry sample rate. access_sample_rate: f64, + /// Whether `tinybird.enabled` and `tinybird.access_enabled` were both + /// set when settings were first read. Gates building the + /// [`AccessTelemetrySnapshot`] at all: the snapshot costs env reads and + /// `String` allocations on the pre-send path, which a disabled + /// deployment (the default) should not pay. + access_telemetry_enabled: bool, } /// Outcome of handing a finalized response to the client. @@ -557,8 +575,9 @@ pub(crate) struct DeliveryOutcome { /// Whether delivery completed or failed partway. pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the - /// freeze point. - pub snapshot: AccessTelemetrySnapshot, + /// freeze point. `None` when access telemetry was disabled at snapshot + /// time; the emitter treats that as nothing to send. + pub snapshot: Option, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -691,11 +710,15 @@ fn send_edgezero_response( context.server_timing_enabled, ); - // Built unconditionally, right after the freeze point and before - // `into_parts()` consumes `response`: nothing else survives to - // post-send on every path (the request was consumed by dispatch, and - // `EcFinalizeState` is absent on asset, admin, and error paths). - let snapshot = build_access_telemetry_snapshot(&response, context); + // Built right after the freeze point and before `into_parts()` + // consumes `response`: nothing else survives to post-send on every + // path (the request was consumed by dispatch, and `EcFinalizeState` + // is absent on asset, admin, and error paths). Skipped entirely when + // access telemetry is disabled, so the default configuration pays no + // env reads or allocations here. + let snapshot = context + .access_telemetry_enabled + .then(|| build_access_telemetry_snapshot(&response, context)); let (parts, body) = response.into_parts(); @@ -1508,7 +1531,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, - snapshot: sample_access_snapshot(), + snapshot: Some(sample_access_snapshot()), }; assert_eq!( @@ -1565,6 +1588,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 0.25, + access_telemetry_enabled: true, } } @@ -1827,6 +1851,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 1.0, + access_telemetry_enabled: true, }, ); assert!( @@ -1847,7 +1872,11 @@ mod tests { ..trusted_server_core::settings::TinybirdSettings::default() }, ); - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + let snapshot = outcome + .snapshot + .as_ref() + .expect("should build a snapshot when access telemetry is enabled"); + let row = access_event_row(snapshot, &timings.snapshot(), 0); futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) .expect("should send access telemetry"); diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 5025f9a73..1158ada68 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -960,34 +960,6 @@ mod tests { ); } - #[test] - fn sampled_out_requests_emit_nothing() { - // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, - // entropy) { emit_access_event(...) }`): `emit_access_event` is only - // reached when `sampled_in` returns `true`. With a `0.0` rate it - // never does, for any entropy, so the http client should never see - // a request. - let http_client = RecordingHttpClient::respond_with(202); - let target = TinybirdEventsTarget::from_access_config(enabled_config()); - let rate = 0.0; - let entropy = 123_456_789_u64; - - if sampled_in(rate, entropy) { - futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) - .expect("should send when sampled in"); - } - - assert_eq!( - http_client - .requests - .lock() - .expect("should lock recorded requests") - .len(), - 0, - "sampled-out requests must never reach emit_access_event" - ); - } - #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs index 19826231d..acbb61b6c 100644 --- a/crates/trusted-server-core/examples/local_dev_config.rs +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -3,8 +3,11 @@ //! Reads `trusted-server.example.toml`, replaces the placeholder secrets with //! random values, flips the flags a local smoke test needs, validates the //! result through [`trusted_server_core::settings::Settings::from_toml`], and -//! prints the blob envelope JSON that -//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! prints the blob envelope JSON that the Axum adapter's +//! `TRUSTED_SERVER_CONFIG_{STORE}_{KEY}` environment variable expects. With +//! the default store and key both named `trusted_server_config`, the +//! concrete variable resolves (not a typo) to +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG`. //! //! The random values are time-and-pid seeded, not cryptographic. This tool //! exists for throwaway local test instances only; never use its output for a diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 28e391f7e..2aac6d7d3 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -81,6 +81,9 @@ pub enum RouteClass { Ec, /// The server-side auction or SPA re-auction (`page-bids`) endpoint. AuctionApi, + /// A response proxied through a configured asset route (the + /// non-document fallback for scripts, styles, images, and fonts). + Asset, /// Everything else: discovery, tester-cookie toggles, denied legacy /// aliases, and any response with no attached [`RouteMetadata`]. Other, @@ -97,6 +100,7 @@ impl RouteClass { Self::IntegrationProxy => "integration_proxy", Self::Ec => "ec", Self::AuctionApi => "auction_api", + Self::Asset => "asset", Self::Other => "other", } } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 6ac9a9692..8d1b92cc1 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -385,6 +385,40 @@ pub struct TimingSnapshot { mod tests { use super::*; + #[test] + fn every_phase_index_is_unique_and_in_bounds() { + // `PHASE_COUNT` and `Phase::index()` are hand-synced; nothing at + // compile time ties them together. A new variant whose `index()` + // returns `PHASE_COUNT` would panic at runtime on first `record`, + // contradicting the module's no-panics claim, so this test fails + // first instead. (A variant missing from this list is a compile + // error via the exhaustive `match` in `index()` once added there.) + let phases = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + Phase::AuctionWait, + Phase::Stream, + ]; + let mut seen = [false; PHASE_COUNT]; + for phase in phases { + let index = phase.index(); + assert!( + index < PHASE_COUNT, + "should be in bounds: {phase:?} -> {index}" + ); + assert!(!seen[index], "should be unique: {phase:?} -> {index}"); + seen[index] = true; + } + assert!( + seen.iter().all(|slot| *slot), + "should cover every phases-array slot" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); From 274241b9ebb0f9bcabff420bc56a3a82ab1920a0 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:00:29 -0700 Subject: [PATCH 021/230] Add auction timeline offsets spec amendment Adds section 18 to the request phase timing spec: three first-call-wins T0 offsets (auction dispatched, resolved, committed) on RequestTimings, emitted as additive nullable columns on access_logs_raw with auction_id as the join key to the per-bidder auction dataset. Answers the overlap-proof questions the two existing clocks cannot: when the auction started relative to request entry, when the final bid landed, and when targeting was committed toward GAM. --- .../2026-08-24-request-phase-timing-design.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1..56c2685b3 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -562,3 +562,108 @@ streamed. No allocation in the hot path beyond the one `Arc` at entry, the - The stall window itself remains unattributed until this ships. If it recurs first, the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset path versus HTML path) is the fallback. + +## 18. Auction timeline offsets (follow-up increment) + +Status: spec amendment for a follow-up PR; not part of the initial implementation +(#1074). Builds only on machinery that spec sections 5, 9, and 10 already define. + +### Problem + +The pipeline has two clocks that never meet. The auction dataset +(`auction_events_raw`, PR #813) measures the auction internally: `total_time_ms` +from auction start to terminal, `provider_response_time_ms` per bidder call. Its +clock starts when the auction observation is created, so nothing places those +numbers on the request timeline. The access row is T0-anchored but records only +`auction_wait_ms`: time the handler was blocked at collect, deliberately not the +auction's own timeline. + +That leaves three questions unanswerable today: + +1. At what request-relative time did the auction start (dispatch leave the edge)? +2. At what request-relative time did the auction resolve (final bid or timeout)? +3. At what request-relative time were the results committed toward GAM? + +These are the overlap-proof questions. A client-side wrapper cannot dispatch until +the browser boots (t≈3000ms on measured prospect pages); the server-side auction +dispatches while the origin fetch is in flight. Proving that requires all +milestones on one clock. + +### Design + +Three first-call-wins marks on `RequestTimings`, in the style of +`mark_headers_ready()`, each storing `Option` since T0: + +| Mark | Recorded at | Meaning | +| --------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| `mark_auction_dispatched()` | immediately before `orchestrator.dispatch_auction` returns control to the caller (`publisher.rs` dispatch site) | bid requests have left the edge | +| `mark_auction_resolved()` | immediately after `collect_dispatched_auction` returns, both collect sites | final bid returned or auction timed out; terminal either way | +| `mark_auction_committed()` | immediately after `write_bids_to_state` returns, both call sites | winning bids are in page state, available to the response pipeline | + +Notes on the definitions: + +- "Committed toward GAM" is defined as `write_bids_to_state` returning: the common + point in buffered and streaming modes where targeting becomes part of the + response. TS never calls GAM server-side; the browser's GPT call carries the + targeting, and that half of the timeline belongs to client-side measurement. + The edge proves when targeting was available; the client proves when GAM saw it. +- First-call-wins on all three marks. A request produces at most one publisher-path + auction today; if a second auction ever occurs in one request, the row describes + the first and the auction dataset still carries both in full. +- Same locking and failure model as every other `RequestTimings` write: `try_lock`, + drop on contention, saturating conversion at serialization. + +### Row changes + +Four additive columns on `access_logs_raw`, all populated from the +`TimingSnapshot` at the existing freeze/emission points (no new emission path): + +``` +`auction_dispatched_ms` Nullable(UInt32), `json:$.auction_dispatched_ms` +`auction_resolved_ms` Nullable(UInt32), `json:$.auction_resolved_ms` +`auction_committed_ms` Nullable(UInt32), `json:$.auction_committed_ms` +`auction_id` String, `json:$.auction_id` +``` + +- The three offsets are null when no auction ran (the common case: assets, EC + endpoints, auction-disabled deployments). Null means "no auction", never "zero". +- `auction_id` is the telemetry auction UUID already present on every + `auction_events_raw` row, carried onto the access row as the join key between + the T0 timeline and per-bidder detail. Sentinel `none` when no auction ran, + matching the non-nullable-dimension convention of section 9. It is a random + UUID, not identity-bearing; unbounded cardinality is accepted for the same + reason it is accepted in the auction dataset. +- Schema evolution is additive with JSONPaths on every new column and + `FORWARD_QUERY` carrying the existing columns, per the deployed datasource's + established evolution path. Verified with `tb --cloud deploy --check` before + deploy. + +### Interpretation model + +Combined with existing columns, one access row now reads as a timeline: + +``` +t=0 ......... request entry +t=D ......... auction_dispatched_ms (bids out; origin fetch typically in flight) +t=R ......... auction_resolved_ms (R - D ~ auction duration; join auction_id + for the per-bidder long pole) +t=C ......... auction_committed_ms (targeting in page state) +t=H ......... time_elapsed_ms (headers committed) +``` + +Derivations the dashboard can add without schema help: auction duration on the +request clock (`R - D`), commit latency (`C - R`), and overlap ratio (share of +`R - D` that ran concurrently with `ts-origin`). `auction_wait_ms` keeps its +existing meaning (blocked time only) and is now interpretable next to the +timeline: `R - D` minus `auction_wait_ms` approximates how much of the auction +was absorbed by work the request needed anyway. + +### Scope + +- Fastly emits; Axum, Cloudflare, and Spin collect the marks but do not emit, + matching section 8a adapter semantics. +- No header emission for any of these values: they are post-hoc analysis fields, + and two of the three are typically unknown at the header freeze point in + streaming mode. +- No config surface: the marks are always-on collection like every other phase, + gated at emission by the existing `tinybird.access_enabled`. From 29d45e8c8087173ef8daa221898892b58a399989 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 022/230] Add auction timeline offsets implementation plan --- .../2026-08-26-auction-timeline-offsets.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md new file mode 100644 index 000000000..08e14dabc --- /dev/null +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -0,0 +1,59 @@ +# Auction Timeline Offsets Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Record three T0-anchored auction milestones (dispatched, resolved, committed) plus the auction id on `RequestTimings`, and emit them as four additive columns on the `access_logs_raw` row. + +**Architecture:** Follows spec section 18 exactly. All state lives in the existing `RequestTimings` inner (same `try_lock`/first-call-wins/saturating model as `mark_headers_ready`); the row builder reads the values from `TimingSnapshot`, so no new emission path and no adapter changes. + +**Tech Stack:** Rust (core crate only), Tinybird datasource file. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` section 18. + +## Global Constraints + +- Marks are first-call-wins; `try_lock` only; a contended lock drops the sample. +- Null offsets mean "no auction ran", never zero. `auction_id` sentinel is `none`. +- Column names: `auction_dispatched_ms`, `auction_resolved_ms`, `auction_committed_ms`, `auction_id`; JSONPaths `json:$.`; FORWARD_QUERY extended in the same order. +- Dispatch mark records only on `DispatchAuctionOutcome::Dispatched`; a failed dispatch leaves all three offsets null (the auction dataset still records the failure). +- No header emission, no config surface, no changes outside `trusted-server-core` and `tinybird/`. + +--- + +### Task 1: RequestTimings marks and snapshot fields + +**Files:** +- Modify: `crates/trusted-server-core/src/request_timing.rs` + +**Interfaces:** +- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` + +- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. +- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins. +- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone. +- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`. +- [ ] `cargo test-fastly request_timing`, commit. + +### Task 2: Publisher call sites + +**Files:** +- Modify: `crates/trusted-server-core/src/publisher.rs` + +**Interfaces:** +- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. + +- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` +- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`. +- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`. +- [ ] `cargo test-fastly`, commit. + +### Task 3: Row columns and datasource + +**Files:** +- Modify: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +- [ ] `access_event_row`: add the three offset keys (nullable) and `auction_id` with `none` sentinel, after the existing phase keys. +- [ ] Extend `row_serializes_nulls_for_missing_phases` and `row_serializes_recorded_phases_as_numbers` for the new keys. +- [ ] Datasource: four schema columns with JSONPaths (`Nullable(UInt32)` ×3, `String`), appended at the end of SCHEMA and FORWARD_QUERY so existing column order stays stable. +- [ ] Full gates: fmt, clippy (all six), test-fastly/axum/cloudflare/spin, parity. Commit. From 46911a647347adc13791171995839e43a92c56fb Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 023/230] Record T0-anchored auction timeline offsets in the access row Implements spec section 18: three first-call-wins marks on RequestTimings (dispatched at the DispatchAuctionOutcome::Dispatched arm, resolved after collect at both sites, committed after write_bids_to_state at both sites), carried through TimingSnapshot into four additive access_logs_raw columns: auction_dispatched_ms, auction_resolved_ms, auction_committed_ms, and auction_id as the join key to the per-bidder auction dataset. Null offsets mean no auction ran; a failed dispatch records nothing. FORWARD_QUERY fills the new columns with typed defaults for pre-existing rows. No header emission, no config surface, no adapter changes: the values ride the existing snapshot and the tinybird.access_enabled gate. --- .../wrangler.integration.generated.toml | 16 +++ .../src/access_telemetry.rs | 19 +++ crates/trusted-server-core/src/publisher.rs | 14 ++ .../trusted-server-core/src/request_timing.rs | 134 ++++++++++++++++++ .../datasources/access_logs_raw.datasource | 8 +- 5 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 000000000..263403193 --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d3..083adbe8e 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -286,6 +286,10 @@ pub fn access_event_row( "stream_ms": timings.stream_ms, "request_elapsed_ms": timings.request_elapsed_ms, "resp_bytes": timings.resp_bytes, + "auction_dispatched_ms": timings.auction_dispatched_ms, + "auction_resolved_ms": timings.auction_resolved_ms, + "auction_committed_ms": timings.auction_committed_ms, + "auction_id": timings.auction_id.as_deref().unwrap_or("none"), "template_cache_state": snapshot.template_cache_state, "country": snapshot.country, "ts_version": snapshot.ts_version, @@ -463,6 +467,9 @@ mod tests { "stream_ms", "request_elapsed_ms", "resp_bytes", + "auction_dispatched_ms", + "auction_resolved_ms", + "auction_committed_ms", ] { assert!( parsed[field].is_null(), @@ -488,6 +495,10 @@ mod tests { ); } assert_eq!(parsed["auction_wait_placement"], "none"); + assert_eq!( + parsed["auction_id"], "none", + "auction_id should carry the none sentinel when no auction ran" + ); } #[test] @@ -506,6 +517,10 @@ mod tests { stream_ms: Some(8), auction_wait_placement: Some(AuctionWaitPlacement::InStream), resp_bytes: Some(1024), + auction_dispatched_ms: Some(9), + auction_resolved_ms: Some(10), + auction_committed_ms: Some(11), + auction_id: Some("33333333-3333-3333-3333-333333333333".to_owned()), }; let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); let parsed: serde_json::Value = @@ -515,6 +530,10 @@ mod tests { assert_eq!(parsed["stream_ms"], 8); assert_eq!(parsed["resp_bytes"], 1024); assert_eq!(parsed["auction_wait_placement"], "in_stream"); + assert_eq!(parsed["auction_dispatched_ms"], 9); + assert_eq!(parsed["auction_resolved_ms"], 10); + assert_eq!(parsed["auction_committed_ms"], 11); + assert_eq!(parsed["auction_id"], "33333333-3333-3333-3333-333333333333"); } #[test] diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc6..3981c386b 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3951,6 +3951,8 @@ async fn collect_non_html_auction( params .timings .record_auction_wait(placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + params.timings.mark_auction_resolved(); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3960,6 +3962,9 @@ async fn collect_non_html_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + params.timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4010,6 +4015,8 @@ async fn collect_stream_auction( .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; timings.record_auction_wait(*placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + timings.mark_auction_resolved(); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4023,6 +4030,9 @@ async fn collect_stream_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4340,6 +4350,10 @@ pub async fn handle_publisher_request( .await { DispatchAuctionOutcome::Dispatched(dispatched) => { + // T0-anchored timeline mark (spec section 18): bid + // requests have left the edge. A failed dispatch never + // marks, so all three auction offsets stay null for it. + timings.mark_auction_dispatched(observation.auction_id.to_string()); auction_request_for_telemetry = Some(auction_request); auction_observation = Some(observation); Some(dispatched) diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1..edcd9bfc2 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -106,6 +106,19 @@ struct Inner { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. resp_bytes: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_dispatched`] call. + auction_dispatched: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_resolved`] call. + auction_resolved: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_committed`] call. + auction_committed: Option, + /// Telemetry auction UUID recorded by the first + /// [`RequestTimings::mark_auction_dispatched`] call; joins the access + /// row to the per-bidder auction dataset. + auction_id: Option, } /// Per-request phase timing collector. @@ -128,6 +141,10 @@ impl RequestTimings { request_elapsed: None, auction_wait_placement: None, resp_bytes: None, + auction_dispatched: None, + auction_resolved: None, + auction_committed: None, + auction_id: None, }))) } @@ -200,6 +217,53 @@ impl RequestTimings { } } + /// Stamps the elapsed time since `t0` as the auction dispatch offset and + /// records the telemetry auction id, the first time this is called. + /// + /// Called when `dispatch_auction` reports the bid requests dispatched; + /// a failed dispatch never records, so all three auction offsets stay + /// `None` for it. Subsequent calls are no-ops (first call wins). Drops + /// the sample silently on lock contention or poisoning. + pub fn mark_auction_dispatched(&self, auction_id: String) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_dispatched.is_none() { + inner.auction_dispatched = Some(inner.t0.elapsed()); + inner.auction_id = Some(auction_id); + } + } + + /// Stamps the elapsed time since `t0` as the auction resolve offset (the + /// final bid returned or the auction timed out), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_resolved(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_resolved.is_none() { + inner.auction_resolved = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as the auction commit offset + /// (winning bids written into page state), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_committed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_committed.is_none() { + inner.auction_committed = Some(inner.t0.elapsed()); + } + } + /// Records the response body size in bytes. /// /// Drops the sample silently on lock contention or poisoning. @@ -257,6 +321,10 @@ impl RequestTimings { stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), auction_wait_placement: inner.auction_wait_placement, resp_bytes: inner.resp_bytes, + auction_dispatched_ms: duration_ms(inner.auction_dispatched), + auction_resolved_ms: duration_ms(inner.auction_resolved), + auction_committed_ms: duration_ms(inner.auction_committed), + auction_id: inner.auction_id.clone(), } } } @@ -379,6 +447,18 @@ pub struct TimingSnapshot { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. pub resp_bytes: Option, + /// T0 offset at which the auction dispatched (bid requests left the + /// edge), or `None` when no auction ran. + pub auction_dispatched_ms: Option, + /// T0 offset at which the auction resolved (final bid or timeout), or + /// `None` when no auction ran. + pub auction_resolved_ms: Option, + /// T0 offset at which winning bids were committed into page state, or + /// `None` when no auction ran. + pub auction_committed_ms: Option, + /// Telemetry auction UUID joining this row to the auction dataset, or + /// `None` when no auction ran. + pub auction_id: Option, } #[cfg(test)] @@ -419,6 +499,60 @@ mod tests { ); } + #[test] + fn auction_marks_are_first_call_wins_and_snapshot_maps_them() { + let timings = RequestTimings::new(); + timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + // Second calls must not overwrite the first-recorded values. + timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + + let snapshot = timings.snapshot(); + assert!( + snapshot.auction_dispatched_ms.is_some(), + "should record the dispatch offset" + ); + assert!( + snapshot.auction_resolved_ms.is_some(), + "should record the resolve offset" + ); + assert!( + snapshot.auction_committed_ms.is_some(), + "should record the commit offset" + ); + assert_eq!( + snapshot.auction_id.as_deref(), + Some("11111111-1111-1111-1111-111111111111"), + "should keep the first-recorded auction id" + ); + } + + #[test] + fn snapshot_without_auction_marks_yields_none_for_all_offsets() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_dispatched_ms, None, + "should stay None when no auction dispatched" + ); + assert_eq!( + snapshot.auction_resolved_ms, None, + "should stay None when no auction resolved" + ); + assert_eq!( + snapshot.auction_committed_ms, None, + "should stay None when no auction committed" + ); + assert_eq!( + snapshot.auction_id, None, + "should carry no auction id when no auction ran" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 062b884ba..4441de384 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -27,13 +27,17 @@ SCHEMA > `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, `country` LowCardinality(String) `json:$.country`, `ts_version` LowCardinality(String) `json:$.ts_version`, - `pop` LowCardinality(String) `json:$.pop` + `pop` LowCardinality(String) `json:$.pop`, + `auction_dispatched_ms` Nullable(UInt32) `json:$.auction_dispatched_ms`, + `auction_resolved_ms` Nullable(UInt32) `json:$.auction_resolved_ms`, + `auction_committed_ms` Nullable(UInt32) `json:$.auction_committed_ms`, + `auction_id` String `json:$.auction_id` ENGINE "MergeTree" ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" TTL "toDate(event_ts) + INTERVAL 30 DAY" FORWARD_QUERY > - SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop, CAST(NULL AS Nullable(UInt32)) AS auction_dispatched_ms, CAST(NULL AS Nullable(UInt32)) AS auction_resolved_ms, CAST(NULL AS Nullable(UInt32)) AS auction_committed_ms, 'none' AS auction_id TOKEN ts_access_ingest APPEND From 1fa9f8cc09df889aec42039b13b7a108d1df9d47 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:29 -0700 Subject: [PATCH 024/230] Remove generated integration wrangler config from tracking The Cloudflare integration harness writes wrangler.integration.generated.toml at test time; it was swept into the previous commit by accident. Ignore it so local CI=1 runs cannot commit it again. --- .gitignore | 1 + .../wrangler.integration.generated.toml | 16 ---------------- 2 files changed, 1 insertion(+), 16 deletions(-) delete mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/.gitignore b/.gitignore index 24b9e06aa..96ffa2a5c 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,4 @@ src/*.html # leftover local build artifacts (node_modules, target, dist) that remain on disk. /crates/js/ /crates/integration-tests/ +wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml deleted file mode 100644 index 263403193..000000000 --- a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml +++ /dev/null @@ -1,16 +0,0 @@ -name = "trusted-server" -main = "build/index.js" -compatibility_date = "2024-09-23" -# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the -# outbound `CacheMode::NoStore` cache bypass under this compatibility date. -compatibility_flags = ["nodejs_compat", "cache_option_enabled"] -# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. - -[[kv_namespaces]] -binding = "TRUSTED_SERVER_KV" -id = "ci-local-kv" - -[vars] -# Placeholder replaced by the integration test harness with a JSON object that -# contains the runtime Trusted Server app-config blob envelope. -TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' From 2ef7635437d50b1c4ec3eac57321aeb168970c0c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 025/230] Reject single-segment publisher paths in route templates The first path segment is only a section name when the path has depth: under a /%postname%/ permalink structure every article is a single-segment path, so keeping those segments verbatim put full article slugs into the 30-day dataset, against spec section 9. Depth is now required for a named template; single-segment paths, root landing pages included, bucket to /other/*. Route slicing keeps route_class and multi-segment templates like /news/*. --- .../src/access_telemetry.rs | 69 ++++++++++++++----- 1 file changed, 50 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d3..aaad46633 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -130,27 +130,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has -/// additional segments beyond the first. The root path `/` maps to itself. -/// A first segment is rejected to `/other/*` — outright, never filtered or -/// truncated, so no fragment of it ever reaches the row — when it: +/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: +/// depth is what makes the first segment a section name (`/news/*`) rather +/// than the document itself. The root path `/` maps to itself. Everything +/// else is rejected to `/other/*` — outright, never filtered or truncated, +/// so no fragment of a rejected path ever reaches the row: /// -/// - is empty, or contains any character outside the allowlist after -/// lowercasing (an email address, a search phrase); -/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex -/// tokens, and full article slugs all exceed it — a truncated prefix of -/// any of these would still be identifying); or -/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque -/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; -/// publisher section names are words, at most a year or a version -/// number. +/// - single-segment paths (`/my-post-title` under a `/%postname%/` +/// permalink structure is a per-article slug that no shape heuristic +/// can separate from a section name); +/// - an empty first segment, or one containing any character outside the +/// allowlist after lowercasing (an email address, a search phrase); +/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a +/// truncated prefix of a UUID or token would still be identifying); or +/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits +/// (hex ids, base36 ids, and reset tokens are digit-heavy; section +/// names carry at most a year or a version number). /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. Short all-alpha slugs on single-segment paths are -/// indistinguishable from section names and still pass; the bound here is -/// shape-based, not semantic. +/// this broad. /// /// # Examples /// @@ -193,7 +193,11 @@ pub fn publisher_route_template(path: &str) -> String { if has_more_depth { format!("/{lowered}/*") } else { - format!("/{lowered}") + // A single-segment path's first segment is the document, not a + // section: `/my-post-title` under WordPress `/%postname%/` is a + // per-article slug, and no shape heuristic can separate it from a + // section name. Only depth >= 2 makes the first segment a section. + "/other/*".to_owned() } } @@ -436,11 +440,37 @@ mod tests { } #[test] - fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + fn publisher_route_template_rejects_single_segment_paths() { + // A single-segment path's first segment is the document itself + // (WordPress `/%postname%/` puts every article at depth 1), so no + // shape heuristic can separate a slug from a section name; depth + // is the only safe signal. Root-level landing pages pay for this + // deliberately. + assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); + assert_eq!( + publisher_route_template("/my-post-title/"), + "/other/*", + "a trailing slash should not count as depth" + ); + assert_eq!( + publisher_route_template("/1234567"), + "/other/*", + "a numeric post id at the digit boundary should still reject" + ); + assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); + assert_eq!( + publisher_route_template("/about"), + "/other/*", + "root-level landing pages reject too; only depth makes a section" + ); + } + + #[test] + fn publisher_route_template_lowercases_before_allowlisting() { assert_eq!( publisher_route_template("/News/Article"), "/news/*", - "should lowercase before validating and truncating" + "should lowercase before validating" ); } @@ -524,6 +554,7 @@ mod tests { assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); assert_eq!(RouteClass::Ec.as_str(), "ec"); assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Asset.as_str(), "asset"); assert_eq!(RouteClass::Other.as_str(), "other"); } From 082461d10eeabf1e5658dd27ebd89f4b6068af3d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 026/230] Sample access rows with real randomness at the snapshot rate The bucket-quantized sampler truncated rates below one in a million to a zero threshold (silently emitting nothing) and quantized other low rates downward while rows still carried the configured rate, biasing the sum(1.0 / sample_rate) volume estimator. Its no-rand premise was also wrong: rand::thread_rng() is WASI-backed on this target and the EC generation path already relies on it. The sampler is now a direct uniform-roll comparison, and the roll gates on the rate stored in the snapshot itself, so emission probability and the row's sample_rate column cannot diverge; the divergence guard and its tests are removed. Also per review: the settings-reload fallback in the post-send path could never emit (no snapshot exists when settings were absent) and is removed; the dead_code allow on DeliveryOutcome narrows to the one collected-but-unemitted field; and the post-send ordering test is narrowed to the leg it actually proves, that request_elapsed is stamped when send returns. --- Cargo.lock | 1 + .../trusted-server-adapter-fastly/Cargo.toml | 1 + .../trusted-server-adapter-fastly/src/main.rs | 243 +++--------------- .../src/tinybird.rs | 83 +++--- 4 files changed, 85 insertions(+), 243 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e29380b77..201060a52 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5372,6 +5372,7 @@ dependencies = [ "futures", "log", "log-fastly", + "rand 0.8.6", "serde", "serde_json", "sha2 0.10.9", diff --git a/crates/trusted-server-adapter-fastly/Cargo.toml b/crates/trusted-server-adapter-fastly/Cargo.toml index 47cc609b2..3835d6469 100644 --- a/crates/trusted-server-adapter-fastly/Cargo.toml +++ b/crates/trusted-server-adapter-fastly/Cargo.toml @@ -29,6 +29,7 @@ serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } trusted-server-core = { workspace = true } +rand = { workspace = true } url = { workspace = true } urlencoding = { workspace = true } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 934f98402..b5b08ab2e 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,6 @@ use std::sync::Arc; + +use rand::Rng as _; use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; @@ -346,17 +348,12 @@ fn edgezero_main(mut req: FastlyRequest) { ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec // finalize branch above failed), so there is no pull-sync dispatch here - // at all — telemetry is the only post-send step. Reload settings when - // `app_state` never built, matching the fallback used earlier in this - // function for entry-point finalize headers. - match settings_snapshot.as_deref() { - Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), - None => match load_settings_from_config_store() { - Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), - Err(e) => { - log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); - } - }, + // at all — telemetry is the only post-send step. When `app_state` never + // built there is nothing to emit either: `access_telemetry_enabled` was + // necessarily false without a settings snapshot, so the outcome carries + // no access snapshot, and reloading settings here could not change that. + if let Some(settings) = settings_snapshot.as_deref() { + emit_access_telemetry_after_send(settings, &outcome, &timings); } } @@ -458,9 +455,9 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. A snapshot carrying a degraded -/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, -/// since it only occurs on an already-degraded path. Every other drop (row +/// case and not worth a log line. The sampling roll uses the rate stored on +/// the snapshot itself, so the emission probability always matches the +/// row's `sample_rate` column by construction. Every other drop (row /// build, token load, send, or non-2xx status — all folded into /// `emit_access_event`'s `Result`) logs exactly one warning naming the /// reason. @@ -483,20 +480,12 @@ fn emit_access_telemetry_after_send( .duration_since(UNIX_EPOCH) .unwrap_or_default(); let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); - // Entropy for the sampling decision: the timestamp's nanosecond - // resolution XORed with a per-request value already on hand - // (`outcome.bytes`), so two requests handled in the same instance never - // collide on sampling decisions purely because they read the same - // millisecond. There is no `rand` crate dependency here — see - // `tinybird::sampled_in`. - let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); - let entropy = entropy_nanos ^ outcome.bytes; - - if !should_sample_access_row( - snapshot.sample_rate, - settings.tinybird.access_sample_rate, - entropy, - ) { + // Sample with the rate stored on the snapshot itself — the same value + // serialized into the row's `sample_rate` column — so the emission + // probability and the row's claimed rate cannot diverge, which the + // documented `sum(1.0 / sample_rate)` volume estimator depends on. + let roll = rand::thread_rng().r#gen::(); + if !tinybird::sampled_in(snapshot.sample_rate, roll) { return; } @@ -512,39 +501,6 @@ fn emit_access_telemetry_after_send( } } -/// Whether one response's access-telemetry row should be emitted, combining -/// the degraded-snapshot guard with the sampling roll. -/// -/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] -/// itself (the value serialized into the row's `sample_rate` column, which -/// the documented volume estimator divides by as `1.0 / sample_rate`). -/// `settings_sample_rate` is the rate used for the sampling decision at -/// call time. The two can diverge: when `app_state` fails to build, -/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to -/// `0.0` before any settings ever load, but the two settings-reload -/// emission sites still gate and sample using the *reloaded* settings' -/// (nonzero) rate. Without this guard, such a row could be sampled in and -/// emitted while carrying `sample_rate: 0.0`, corrupting the volume -/// estimator. Dropping these rows is acceptable: they only occur on an -/// already-degraded path, consistent with this pipeline's fail-quiet -/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so -/// the guard is unit-testable without a network seam. -/// -/// Callers must already have applied the coarse -/// `tinybird.enabled`/`access_enabled` gate. -#[must_use] -fn should_sample_access_row( - snapshot_sample_rate: f64, - settings_sample_rate: f64, - entropy: u64, -) -> bool { - if snapshot_sample_rate <= 0.0 { - return false; - } - - tinybird::sampled_in(settings_sample_rate, entropy) -} - /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -568,11 +524,12 @@ struct SendContext { } /// Outcome of handing a finalized response to the client. -#[allow(dead_code)] pub(crate) struct DeliveryOutcome { /// Response body size in bytes. pub bytes: u64, - /// Whether delivery completed or failed partway. + /// Whether delivery completed or failed partway. Collected as + /// groundwork; not yet emitted on any surface. + #[allow(dead_code)] pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the /// freeze point. `None` when access telemetry was disabled at snapshot @@ -672,6 +629,11 @@ fn drive_streaming_body( /// A drive that failed after writing at least one byte delivered a truncated /// response rather than nothing at all, so it is [`DeliveryResult::Partial`], /// not [`DeliveryResult::Error`]. +/// +/// The `Ok(())` arm exists for the classifier's totality, not for the +/// production caller: `send_edgezero_response` consumes this value only in +/// its `Err` branch and re-derives the success outcome from +/// `streaming_body.finish()`. fn classify_stream_delivery( drive_result: &Result<(), Report>, bytes: u64, @@ -1036,7 +998,6 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; - use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1760,83 +1721,14 @@ mod tests { ); } - /// Records `"telemetry"` into a shared order log instead of sending a - /// real request, standing in for the adapter's platform HTTP client in - /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. - struct OrderingHttpClient { - log: Arc>>, - } - - #[async_trait::async_trait(?Send)] - impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { - async fn send( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformResponse, - Report, - > { - self.log - .lock() - .expect("should lock order log") - .push("telemetry"); - let response = response_builder() - .status(edgezero_core::http::StatusCode::ACCEPTED) - .body(EdgeBody::empty()) - .expect("should build ordering test response"); - Ok(trusted_server_core::platform::PlatformResponse::new( - response, - )) - } - - async fn send_async( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformPendingRequest, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - - async fn select( - &self, - _pending_requests: Vec, - ) -> Result< - trusted_server_core::platform::PlatformSelectResult, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - } - #[test] - fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { - // `edgezero_main` cannot be driven directly in a unit test (it - // consumes a live `fastly::Request::from_client()`), and - // `run_edgezero_pull_sync_after_send` has no injectable seam of its - // own — it dispatches through the real identity-graph pull-sync - // path, which needs a configured EC KV store, partner registry, and - // rate limiter wired together. This test instead exercises the two - // REAL functions `edgezero_main` calls that DO have a testable seam - // — `send_edgezero_response` (which stamps `request_elapsed` before - // returning, per Task 6/7) and `tinybird::emit_access_event` (the - // telemetry send added by this task) — around an instrumented - // stand-in for the pull-sync dispatch call, in the exact order - // `edgezero_main` places them. - // - // This proves the elapsed-before-telemetry leg from real production - // code (the assertion below reads the real `timings` snapshot - // between the two calls). The pull-sync-before-telemetry leg is a - // source-order invariant in `edgezero_main`'s three call sites - // (verified by code review, not by this test) because - // `run_edgezero_pull_sync_after_send` itself has no seam to - // instrument — see task-8-report.md for this residual. - let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + fn request_elapsed_is_stamped_when_send_returns() { + // `edgezero_main`'s post-send ordering (pull-sync before telemetry) + // is a source-order invariant with no injectable seam, so this test + // deliberately proves only the leg that has one: by the time + // `send_edgezero_response` returns, `request_elapsed` is already + // stamped, so everything `edgezero_main` runs afterwards (pull-sync, + // telemetry emission) is excluded from `request_elapsed_ms`. let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1854,77 +1746,14 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( - timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should already be stamped before pull-sync/telemetry run" - ); - - // Stand-in for `run_edgezero_pull_sync_after_send`, which has no - // injectable seam (see the test doc comment above). - log.lock().expect("should lock order log").push("pull_sync"); - let http_client = OrderingHttpClient { - log: Arc::clone(&log), - }; - let target = tinybird::TinybirdEventsTarget::from_access_config( - trusted_server_core::settings::TinybirdSettings { - api_host: "api.us-east.aws.tinybird.co".to_owned(), - ..trusted_server_core::settings::TinybirdSettings::default() - }, - ); - let snapshot = outcome - .snapshot - .as_ref() - .expect("should build a snapshot when access telemetry is enabled"); - let row = access_event_row(snapshot, &timings.snapshot(), 0); - - futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) - .expect("should send access telemetry"); - - assert_eq!( - *log.lock().expect("should lock order log"), - vec!["pull_sync", "telemetry"], - "pull-sync must dispatch before telemetry emits" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { - // A snapshot captured on the app-state-build-failure fallback path - // carries `sample_rate: 0.0`. Even when the reloaded settings' rate - // would sample every request in (1.0), the row must not emit — - // otherwise it would claim `sample_rate: 0.0` and corrupt the - // `sum(1.0 / sample_rate)` volume estimator. - assert!( - !should_sample_access_row(0.0, 1.0, 0), - "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" - ); assert!( - !should_sample_access_row(0.0, 1.0, u64::MAX), - "the degraded-rate guard must not depend on the entropy value" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_negative_sample_rate() { - assert!( - !should_sample_access_row(-1.0, 1.0, 0), - "a negative snapshot sample_rate is equally degraded and must not emit" - ); - } - - #[test] - fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { - // With a healthy (nonzero) snapshot sample_rate, the outcome should - // match `tinybird::sampled_in` exactly, since that is the only - // remaining decision. - assert!( - should_sample_access_row(0.25, 1.0, 0), - "a settings rate of 1.0 always samples in, independent of entropy" + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should be stamped by the time send returns" ); assert!( - !should_sample_access_row(0.25, 0.0, 0), - "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + outcome.snapshot.is_some(), + "the access snapshot should exist for the enabled context" ); } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 1158ada68..97a1ffd3e 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -236,39 +236,26 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { // Access telemetry: confirmed-delivery emitter // --------------------------------------------------------------------------- -/// Bucket count [`sampled_in`] maps `entropy` into. -/// -/// Large enough that `rate` values with several significant digits (e.g. -/// `0.015`) still land in a distinct bucket instead of rounding away, while -/// staying well inside `u64` range once multiplied by `rate`. -const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; - /// Decides whether one request's access-telemetry row should be emitted. /// -/// `entropy` should vary from request to request — callers derive it from -/// the wall-clock event timestamp `XORed` with a cheap per-request value (see -/// the call site in `main.rs`). There is no `rand` crate dependency here: -/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping -/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against -/// `rate` is not cryptographically uniform (the low bits of a timestamp are -/// not perfectly evenly distributed), but access-telemetry sampling only -/// needs an approximately even sample, not a provably unbiased one. +/// `roll` is a uniform draw from `[0, 1)`; callers pass +/// `rand::thread_rng().r#gen::()`, which the wasm32-wasip1 guest backs with +/// real WASI randomness (the EC generation path already relies on this and +/// the CI wasm release build verifies it). Comparing the draw directly +/// against `rate` keeps the sampling probability exactly `rate` for every +/// positive value: there is no bucket quantization, so rates below one in a +/// million sample proportionally instead of never, and emitted rows' +/// `sample_rate` matches the probability they were sampled at, which the +/// `sum(1.0 / sample_rate)` volume estimator depends on. /// -/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns -/// `true`, independent of `entropy`, so both boundary configurations behave -/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` -/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), -/// but this function stays total rather than leaning on that invariant. +/// `rate <= 0.0` never samples and `rate >= 1.0` always samples, for any +/// `roll` in `[0, 1)`. `0.0` cannot actually occur while `access_enabled` +/// is `true` (`Settings` validation requires `access_sample_rate > 0.0` in +/// that case), but this function stays total rather than leaning on that +/// invariant. #[must_use] -pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { - if rate >= 1.0 { - return true; - } - if rate <= 0.0 { - return false; - } - let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; - entropy % ACCESS_SAMPLE_BUCKETS < threshold +pub(crate) fn sampled_in(rate: f64, roll: f64) -> bool { + roll < rate } /// Loads and validates the access-log APPEND token from the Fastly secret store. @@ -963,20 +950,44 @@ mod tests { #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( - sampled_in(1.0, 0), + sampled_in(1.0, 0.0), "a 1.0 sample rate should always sample in" ); assert!( - sampled_in(1.0, u64::MAX), - "a 1.0 sample rate should always sample in regardless of entropy" + sampled_in(1.0, 0.999_999), + "a 1.0 sample rate should sample in for the largest roll" + ); + assert!( + !sampled_in(0.0, 0.0), + "a 0.0 sample rate should never sample in, even on a zero roll" + ); + assert!( + !sampled_in(-1.0, 0.0), + "a negative rate should never sample in" + ); + } + + #[test] + fn sampled_in_keeps_exact_probability_for_tiny_rates() { + // The previous bucket-quantized sampler truncated rates below one + // in a million to a zero threshold, silently emitting nothing. + // Direct comparison keeps every positive rate proportional. + let rate = 0.000_000_1; + assert!( + sampled_in(rate, rate / 2.0), + "a roll below a tiny positive rate should sample in" + ); + assert!( + !sampled_in(rate, rate * 2.0), + "a roll above a tiny positive rate should sample out" ); assert!( - !sampled_in(0.0, 0), - "a 0.0 sample rate should never sample in" + !sampled_in(0.000_001_9, 0.000_001_95), + "no downward quantization: the boundary sits exactly at the rate" ); assert!( - !sampled_in(0.0, u64::MAX), - "a 0.0 sample rate should never sample in regardless of entropy" + sampled_in(0.000_001_9, 0.000_001_85), + "rolls just under the rate should sample in" ); } From f3ee47381858e2030ee3e040f773d565f39be0ca Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 027/230] Drop the origin span before error-path auction telemetry On origin failure with a dispatched auction, the origin span guard stayed alive through the emit_abandoned_auction await, so ts-origin and origin_ms absorbed Tinybird emission time. The span now closes when the send resolves, before either branch, with an error-path regression test. --- crates/trusted-server-core/src/publisher.rs | 55 ++++++++++++++++++++- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc6..4c7933795 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -4658,8 +4658,13 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + // The span must end when the origin responds (or fails), before any + // abandonment-telemetry await below — otherwise `ts-origin` would + // absorb Tinybird emission time on the error path. let origin_span = timings.span(Phase::Origin); - let mut response = match services.http_client().send(platform_request).await { + let origin_send_result = services.http_client().send(platform_request).await; + drop(origin_span); + let mut response = match origin_send_result { Ok(platform_response) => platform_response.response, Err(err) => { if let Some(dispatched) = dispatched_auction.take() { @@ -4676,7 +4681,6 @@ pub async fn handle_publisher_request( })); } }; - drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7851,6 +7855,53 @@ mod tests { ); } + #[tokio::test] + async fn origin_span_is_recorded_when_the_origin_send_fails() { + // Regression guard for the review finding that the origin span + // guard stayed alive through the error branch (and its + // abandonment-telemetry await): the span must close when the send + // resolves, so a failed fetch still records `origin_ms` and the + // error branch's own work is excluded from it. + let settings = create_test_settings(); + // No queued response: the stub client fails the origin send. + let stub = Arc::new(StubHttpClient::new()); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let mut ec_context = EcContext::read_from_request(&settings, &request, &services) + .expect("should read EC context"); + let result = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + request, + EdgeCacheHeader::SurrogateControl, + ) + .await; + + assert!(result.is_err(), "should surface the origin failure"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin span even when the origin send fails" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! From 46b3c7f99ab6bf9335d20e17922f0ddeb5866690 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 028/230] Address remaining review feedback on timing surfaces and docs - Pin HEADER_PHASES against Phase::header_name() in the phase-index test, closing the second hand-synced list. - Add RouteClass::Asset to the snake_case rendering test; rename the lowercasing test to say what it does. - Give the Axum adapter a named, fully configured construction path (TrustedServerApp::dev_server_service) so server_timing_enabled is never silently discarded; the tuple API is private now. - Document that Server-Timing is client-visible when enabled, in the configuration guide's observability section. - Replace stale event_date references in the spec, plan, and dashboard guidance with the toDate(event_ts) sorting-key expression, and state the single-segment rejection rule in spec section 9. --- crates/trusted-server-adapter-axum/src/app.rs | 19 ++++++++++++++++++- .../trusted-server-adapter-axum/src/main.rs | 14 +++++--------- .../trusted-server-core/src/request_timing.rs | 13 +++++++++++++ docs/guide/configuration.md | 13 +++++++++++++ .../plans/2026-08-24-request-phase-timing.md | 5 +++-- .../2026-08-24-request-phase-timing-design.md | 16 ++++++++++------ 6 files changed, 62 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4bdf27d01..aeda26a84 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1,6 +1,7 @@ use core::future::Future; use std::sync::Arc; +use edgezero_adapter_axum::service::EdgeZeroAxumService; use edgezero_core::app::Hooks; use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; @@ -587,6 +588,22 @@ impl TrustedServerApp { Ok(build_router(&state)) } + /// The dev server's fully configured tower service: the application + /// router wrapped in the terminal timing layer + /// ([`crate::timing::TimingService`]), with `server_timing_enabled` + /// read from the same settings snapshot that built the router. + /// + /// This is the standard construction path for serving this adapter. + /// [`Hooks::routes`] satisfies the `Hooks` trait contract and returns + /// the bare router without the timing layer; callers who serve traffic + /// should use this instead so `server_timing_enabled` is never + /// silently discarded. + #[must_use] + pub fn dev_server_service() -> crate::timing::TimingService { + let (router, server_timing_enabled) = Self::routes_with_server_timing_flag(); + crate::timing::TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled) + } + /// Build the router alongside whether `Server-Timing` emission is /// enabled, read from the same settings snapshot used to build the /// router. @@ -596,7 +613,7 @@ impl TrustedServerApp { /// `Settings` per request, the Axum dev server builds its application /// state once and reuses the same [`RouterService`] for every request. #[must_use] - pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + fn routes_with_server_timing_flag() -> (RouterService, bool) { let state = match build_state() { Ok(s) => s, Err(ref e) => { diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index b8bc28ae2..4e360ea41 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -3,7 +3,6 @@ use std::net::SocketAddr; use axum::Router; use edgezero_adapter_axum::dev_server::AxumDevServerConfig; use edgezero_adapter_axum::service::EdgeZeroAxumService; -use edgezero_core::router::RouterService; use tokio::net::TcpListener; use tokio::runtime::Builder as RuntimeBuilder; use tokio::signal; @@ -30,8 +29,8 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); - if let Err(err) = run(router, server_timing_enabled, config) { + let service = TrustedServerApp::dev_server_service(); + if let Err(err) = run(service, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } @@ -56,22 +55,19 @@ fn main() { /// Returns an error if the Tokio runtime fails to start, the listener fails /// to bind, or the underlying serve loop errors. fn run( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; - runtime.block_on(serve(router, server_timing_enabled, config)) + runtime.block_on(serve(service, config)) } async fn serve( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let listener = TcpListener::bind(config.addr).await?; - let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); let axum_router = Router::new().fallback_service(service_fn(move |req| { let mut svc = service.clone(); async move { svc.call(req).await } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1..cbb269b1e 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -417,6 +417,19 @@ mod tests { seen.iter().all(|slot| *slot), "should cover every phases-array slot" ); + + // `HEADER_PHASES` is a second hand-synced list: a phase that gains + // a `header_name()` but is never added there silently renders + // nothing in the `Server-Timing` value. + let header_bearing: Vec = phases + .into_iter() + .filter(|phase| phase.header_name().is_some()) + .collect(); + assert_eq!( + header_bearing, + HEADER_PHASES.to_vec(), + "should render every header-bearing phase, in declaration order" + ); } #[test] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 675aaf40a..729801ecb 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1846,6 +1846,19 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: warning Client-visible latency disclosure +The `Server-Timing` header is sent to every client on eligible responses, +not only to operators: browsers expose the values to same-origin JavaScript +via `PerformanceResourceTiming.serverTiming`, and any caller can read the +raw header. Enabling it publishes measured per-phase server latency, +including KV read timing on the public identity endpoints (`ts-kv`) and +origin/cache behaviour on publisher pages (`ts-origin`, +`ts-template-cache`). This is standard `Server-Timing` practice and the +values are durations only, but treat the flag as a diagnostic aid to enable +deliberately, not a general always-on toggle, unless disclosing those +timings to all clients is acceptable for the deployment. +::: + **Environment Override**: ```bash diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 9ff3905f4..02cda64d8 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -928,10 +928,11 @@ git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" - [ ] **Step 1: Rewrite the schema** per spec section 9: keep `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, - `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + `sample_rate Float64` + 30-day TTL; add the columns from spec 9 with dimension columns non-nullable `LowCardinality(String)` and phase columns `Nullable(UInt32)`; drop `path` and `cache_state`; set - `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + `ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status"` + (`event_date` was dropped for the sorting-key expression; see spec section 9). - [ ] **Step 2: Validate** with the tinybird toolchain if available locally (`tb check` / project tests under `tinybird/tests`); otherwise assert the file diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1..2892386f4 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -276,7 +276,8 @@ Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the `mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the -snapshot), `sample_rate`, `event_date`, 30-day TTL. +snapshot), `sample_rate`, 30-day TTL. (`event_date` was later dropped for the +`toDate(event_ts)` sorting-key expression; see section 9's schema note.) Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality @@ -290,8 +291,11 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. - Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 - ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + when it fails the charset allowlist, exceeds 32 characters, carries more than 7 + ASCII digits, or is the only segment in the path. Depth is what makes a first + segment a section name: single-segment paths are documents (WordPress + `/%postname%/` puts every article at depth 1), so they reject wholesale, root + landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -352,7 +356,7 @@ pop, status)`. Every column carries a `json:$.` path (the Events API rejec NDJSON into a datasource without JSONPaths, discovered live); `event_date` was dropped in favor of the sorting-key expression because a DEFAULT column cannot carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query -also carries an `event_date` predicate so the primary index prunes; rollout validates +also carries a `toDate(event_ts)` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the reserved datasource was ever deployed to the remote workspace; if it was, this schema @@ -409,8 +413,8 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 11. Dashboard and query model No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the -ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, -matching the auction dashboards. +ClickHouse connector with `$__timeFilter(event_ts)` plus a `toDate(event_ts)` +predicate, matching the auction dashboards. Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with From c6235b91fcd247a57d27fa8703734340d85d0c22 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 31 Aug 2026 14:09:26 -0500 Subject: [PATCH 029/230] Enforce access telemetry body limit The access emitter carried the configured body limit without enforcing it, allowing oversized rows to bypass the intended transport safeguard. --- .../src/tinybird.rs | 42 +++++++++++++++++-- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 97a1ffd3e..84dfc37d8 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -319,14 +319,25 @@ fn build_access_events_request( /// /// # Errors /// -/// Returns `Err` when the access-log APPEND token cannot be loaded, the -/// backend cannot be registered, the request cannot be built or sent, or the -/// Tinybird Events API responds with a non-2xx status. +/// Returns `Err` when the row exceeds the configured request-body limit, the +/// access-log APPEND token cannot be loaded, the backend cannot be registered, +/// the request cannot be built or sent, or the Tinybird Events API responds +/// with a non-2xx status. pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, row: String, ) -> Result<(), Report> { + let body_len = row.len(); + if body_len > target.max_body_bytes { + return Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request body has {body_len} bytes, exceeding {} byte limit", + target.max_body_bytes + ), + })); + } + let token = load_access_token(target)?; let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; let backend_name = FastlyPlatformBackend @@ -884,6 +895,31 @@ mod tests { ); } + #[test] + fn access_emitter_rejects_oversized_row_before_sending() { + let mut config = enabled_config(); + config.max_body_bytes = 1024; + let target = TinybirdEventsTarget::from_access_config(config); + let http_client = RecordingHttpClient::respond_with(202); + let row = "x".repeat(1025); + + let result = futures::executor::block_on(emit_access_event(&http_client, &target, row)); + + let error = result.expect_err("should reject a row above the configured body limit"); + assert!( + error.to_string().contains("1024"), + "error should name the configured body limit: {error}" + ); + assert!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .is_empty(), + "should not send an oversized access row" + ); + } + #[test] fn access_emitter_posts_ndjson_and_validates_2xx() { // `ts_secrets`/`tinybird_access_append_token` is seeded in From fde55b1d828037ab5f437aa70df83ca663e607fe Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 14:45:53 +0530 Subject: [PATCH 030/230] docs: design mobile ad-render trace endpoint --- ...-mobile-ad-render-trace-endpoint-design.md | 733 ++++++++++++++++++ 1 file changed, 733 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md new file mode 100644 index 000000000..e46f9892b --- /dev/null +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -0,0 +1,733 @@ +# Mobile Ad-Rendering Trace Endpoint Design + +**Status:** Proposed + +**Issue:** [#1050 — Create debug endpoint for mobile user to trace ad rendering](https://github.com/IABTechLab/trusted-server/issues/1050) + +**Related work:** + +- [#1081 — Improvements to TS_CONSOLE for ad observability](https://github.com/IABTechLab/trusted-server/issues/1081) +- [#1074 — Request-phase timing](https://github.com/IABTechLab/trusted-server/pull/1074) +- [#1076 — Auction timing milestones](https://github.com/IABTechLab/trusted-server/pull/1076) +- [#974 — GPT runtime diagnostics](https://github.com/IABTechLab/trusted-server/pull/974) +- [#997 — Trusted Server delivery attribution](https://github.com/IABTechLab/trusted-server/pull/997) + +## 1. Summary + +Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a +mobile end user who needs to reproduce an ad-rendering problem and give support +an exportable diagnostic report. + +The endpoint is both a setup page and a report viewer. On the first visit it +enables the existing GPT diagnostics browser session and explains how to +reproduce the problem. The user then returns to the real publisher page and +reloads it. Trusted Server supplies redacted request context, while the existing +TS Console records GPT, auction, and render evidence. A `View trace results` +action creates one bounded, allowlisted snapshot in same-tab `sessionStorage` +and navigates to `/_ts/trace`. The endpoint reads that snapshot, presents a +mobile-first HTML report, and offers JSON export, copy, and progressive Web +Share actions. + +The design introduces no report database, server-side trace store, report ID, +target URL parameter, telemetry query, or publisher-origin change. It cannot +recover an event that happened before tracing was enabled; the user must +reproduce the problem. + +## 2. Problem and product interpretation + +Issue #1050 names three required data groups: + +1. Network information inspired by `fastly-debug.com`. +2. Auction information per ad slot, similar to the TS tracer and auction + telemetry. +3. End-user cookie information. + +The title additionally establishes two product constraints: the experience is +for a mobile user, and it is reached through an endpoint. A mobile user should +not need browser developer tools, Basic Authentication, a copied trace ID, or a +second copy of the affected page URL. + +A standalone request cannot know what occurred in a previous document. Exact +render evidence exists only while the publisher page is running in the browser. +The design therefore separates two responsibilities without separating the user +experience: + +- `/_ts/trace` owns setup, consolidated presentation, and export. +- TS Console owns observation of the real publisher page. + +The browser-local handoff joins them without introducing a backend report +service. + +## 3. Goals + +- Give a non-technical mobile user one memorable URL: + `https:///_ts/trace`. +- Capture evidence from a real publisher-page reproduction, not a synthetic + auction. +- Display a Fastly-inspired network summary for the traced publisher request. +- Report health for an explicit allowlist of Trusted Server cookies without + exposing their values. +- Present the versioned, allowlisted TS Console evidence for every retained GPT + slot and request cycle. +- Support a full report in a narrow mobile viewport without developer tools. +- Export the same allowlisted model as formatted JSON. +- Keep capture bounded, same-tab, temporary, and inactive by default. +- Preserve normal auction, GPT, rendering, origin, and caching behavior whenever + diagnostics is inactive. +- Keep core behavior platform-neutral while allowing Fastly to provide richer + optional request fields. + +## 4. Non-goals + +- Recovering a failure that occurred before tracing was enabled. +- Permanent history, cross-device retrieval, server upload, or support-ticket + integration. +- A database, distributed trace store, report token, or telemetry lookup. +- A target URL such as `/_ts/trace?target=/article`. +- Replaying an auction or treating a synthetic auction as evidence about a + publisher page. +- Exact parity with every field or active measurement on `fastly-debug.com`. +- Reading the browser's complete cookie jar, third-party cookies, cookie + attributes, or cookies withheld from the request. +- Exposing raw cookie values, EC IDs, EIDs, consent strings, unmasked IP + addresses, internal auction request IDs, creative markup, targeting maps, + cache URLs, or stack traces. +- Reimplementing TS Console auction and creative observability requested by + #1081. +- Querying Tinybird to build an interactive report. +- Adding exact provider-by-slot no-bid explanations before the auction model can + observe those dispositions. +- Direct `POST /auction` browser diagnostics in the first release. + +## 5. Decisions + +### 5.1 Public, redacted endpoint + +`/_ts/trace` is public when explicitly enabled by deployment configuration. It +is not placed under `/_ts/admin`, because the intended user is a layperson on a +phone and the existing Basic Authentication flow is unsuitable for that +journey. + +Public access is safe only because both the page and export use a strict +allowlist. The activation cookie is a feature toggle, not authentication. No +field becomes eligible merely because tracing is active. + +### 5.2 Reuse the existing diagnostics session + +The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics +activation semantics rather than creating a second `ts-trace` session. The +cookie remains host-only, `Secure`, `HttpOnly`, `SameSite=Lax`, and +browser-session scoped. + +`/_ts/trace?enabled=false` clears the activation cookie and browser snapshot. +Other values, duplicate `enabled` parameters, and malformed directives fail +closed and do not mutate session state. + +### 5.3 Browser-local, explicit handoff + +TS Console remains memory-only during observation. It writes a report to +`sessionStorage` only after the user selects `View trace results`. The action: + +1. Builds the same versioned allowlisted snapshot used by export. +2. Adds the redacted request-context envelope. +3. Serializes and validates the size. +4. Stores it under one versioned key in the current tab. +5. Navigates the same tab to `/_ts/trace`. + +Continuous persistence is prohibited. Opening the endpoint in another tab does +not retrieve the snapshot. Closing the tab deletes it according to browser +session-storage semantics. + +### 5.4 Forward reproduction, not historical diagnosis + +The first endpoint visit enables tracing for subsequent eligible document +navigations. The setup page must say plainly that the user needs to return to +the affected page, reload it, and reproduce the problem. + +If the user replaced the affected URL in the address bar with `/_ts/trace`, the +page offers a `Return to previous page` action backed by browser history and +then instructs the user to reload once. The design does not claim that +back-forward-cache restoration caused a new server request. + +Support should preferably give the user the trace URL before reproduction. The +product does not attempt to discover the previous URL through `Referer`, because +address-bar navigation commonly omits it and relying on it would create +inconsistent behavior. + +### 5.5 Separate issue ownership + +#1050 defines the report shell, request context, mobile flow, browser-local +handoff, and export. #1081 remains the owner of creative numbering, auction +classification, bidder/price policy, terminology, and normalized auction/render +timing. + +The trace report consumes TS Console's public versioned export contract. It +does not read TS Console internals or create an alternate slot correlation +engine. + +## 6. User experience + +### 6.1 First visit: no captured report + +`GET /_ts/trace` returns a mobile-first HTML page with: + +- Title: `Trusted Server ad diagnostics`. +- State: `Trace ready` after the response establishes the session cookie. +- A short explanation that no previous ad failure can be recovered. +- Network and cookie health for the setup request, labeled `Setup request`. +- Primary action: `Return to previous page` when browser history permits. +- Secondary instructions: return to the affected page, reload once, reproduce + the problem, then select `View trace results`. +- Action to disable tracing. + +The page must not imply that setup-request network facts or an empty auction +section describe the affected page. + +### 6.2 Active publisher page + +The existing TS Console remains available. On mobile it gains a prominent +`View trace results` action. Selecting it never changes ad behavior; it only +snapshots retained observations and navigates after serialization succeeds. + +If the snapshot cannot be stored, the page remains in place, announces the +failure, and keeps the existing direct JSON export available. + +### 6.3 Report visit + +When a valid snapshot exists, `/_ts/trace` renders: + +1. Report summary and capture time. +2. Network and request section for the traced publisher document. +3. Trusted Server cookie-health section. +4. Auction and rendering section grouped by numbered slot. +5. Coverage and ambiguity section. +6. Export actions. +7. `Clear report and end tracing` action. + +The setup request's facts are not merged into or substituted for missing traced +page facts. Missing fields display `Unavailable`; missing evidence displays +`Not observed` or `Unknown`, following TS Console terminology. + +### 6.4 Mobile and accessibility requirements + +- Support viewport widths down to 320 CSS pixels without horizontal page + scrolling. +- Use a full-document report rather than the current floating 460-pixel panel. +- Use at least 44-by-44 CSS pixel primary touch targets. +- Keep export/end-trace actions reachable without covering report content. +- Use semantic headings, lists, tables only where they remain readable on a + narrow viewport, visible focus styles, and an `aria-live` status region. +- Do not rely on hover, color alone, badges alone, or precise pointer input. +- Preserve browser zoom and safe-area insets. +- Prefer native text and controls over a framework or new UI dependency. + +## 7. Architecture + +```text +First GET /_ts/trace + | + |-- core route builds setup request context + |-- response sets __Host-ts-console + |-- HTML explains forward reproduction + v +Real publisher document reload + | + |-- adapter supplies optional network facts + |-- core computes allowlisted cookie health + |-- core injects redacted TraceRequestContextV1 + |-- existing TS Console observes GPT and TS delivery + v +User selects "View trace results" + | + |-- JS builds bounded TraceReportV1 + |-- same-tab sessionStorage write + |-- location.assign('/_ts/trace') + v +Second GET /_ts/trace + | + |-- static report shell reads and validates TraceReportV1 + |-- mobile HTML renders sections + |-- local JSON/copy/share actions +``` + +### 7.1 Core responsibilities + +- Define configuration and route behavior. +- Register the route before publisher fallback on every supported adapter. +- Define the platform-neutral request-context and report-envelope schemas. +- Build cookie-health facts through read-only parsing. +- Convert `ClientInfo` and available geo data into the public network allowlist. +- Inject request context only into an active private diagnostics document. +- Apply response privacy and security headers. +- Ensure trace requests never reach the publisher origin. + +### 7.2 Adapter responsibilities + +- Register the named route with exact method handling. +- Populate optional `ClientInfo` fields available on the platform. +- Fastly may supply POP, HTTP version, TLS, JA4, H2 fingerprint, and edge + server data when the SDK exposes them. +- Other adapters return the same schema with unsupported fields absent. +- Adapter-specific errors omit optional facts rather than failing publisher + delivery. + +### 7.3 JavaScript responsibilities + +- Accept the immutable redacted request context at initialization. +- Preserve the existing bounded TS Console observation store. +- Build and validate `TraceReportV1` on explicit user action. +- Store only one report in same-tab `sessionStorage`. +- Render the report shell from the validated model. +- Implement download, copy, progressive Web Share, clearing, expiry, and + accessible status reporting. +- Never upload diagnostic data or issue a telemetry query. + +## 8. Route and configuration contract + +Add an explicit default-off option to the existing integration: + +```toml +[integrations.gpt_diagnostics] +enabled = true +trace_page_enabled = false +``` + +Rules: + +- `trace_page_enabled = true` requires `enabled = true`; invalid combinations + fail configuration validation. +- `GET /_ts/trace` returns the setup/report HTML and establishes the session. +- `GET /_ts/trace?enabled=false` returns the shell, clears the cookie, and asks + the client to clear the stored snapshot. +- `HEAD /_ts/trace` returns the same status and headers without a body but does + not mutate the cookie. +- All other methods return a local `405 Method Not Allowed` with `Allow: GET, +HEAD`. +- Disabled deployments return a local `404` for the exact route and never fall + through to the publisher origin. +- Extra path segments, encoded separators, duplicate parameters, and lookalike + paths do not match. +- The route never creates or refreshes an EC, ingests EIDs, runs an auction, + fetches the publisher origin, or emits auction telemetry. + +The current `?ts_console=1` and `?ts_console=0` activation flow remains +supported for technical users. Both activation surfaces drive the same cookie +and runtime; they must not create two concurrent diagnostic modes. + +## 9. Data contracts + +### 9.1 Request context + +The server injects one immutable `TraceRequestContextV1` into active diagnostic +documents: + +```text +TraceRequestContextV1 + schema_version: 1 + captured_at: RFC 3339 UTC timestamp + page: + origin: publisher origin + path: normalized path + network: + masked_client_ip?: string + country?: string + region?: string + asn?: u32 + http_version?: string + tls_protocol?: string + tls_cipher?: string + tls_ja4?: string + h2_fingerprint?: string + edge_hostname?: string + edge_region?: string + edge_pop?: string + cookies: + ts_ec: CookieHealth + ts_eids: CookieHealth + ts_tester: CookieHealth + diagnostics_session: CookieHealth +``` + +The page field omits query and fragment data. It does not contain origin-facing +URLs, referrers, or arbitrary headers. + +`masked_client_ip` uses a deterministic display-only mask for the current +request: IPv4 keeps at most the first 24 bits and IPv6 keeps at most the first +48 bits. The full address never enters HTML, JavaScript, browser storage, or +export. + +JA4 and H2 fingerprints are optional probabilistic identifiers. They are +included only when the deployment has separately enabled the existing +fingerprint diagnostic capability. Their absence is not an error. + +### 9.2 Cookie health + +```text +CookieHealth + state: + absent | present_valid | present_invalid | duplicate | unavailable + source: request + detail?: allowlisted enum +``` + +Allowed details describe shape, not value, for example `valid_ec_format`, +`malformed`, `oversized`, or `activation_pending_response`. + +The parser must inspect the incoming request before any diagnostics-cookie +sanitization, while preserving existing authoritative-cookie and consent +semantics. Inspection is read-only: it must not generate an EC, touch the +identity graph, sync partner IDs, or extend any cookie lifetime. + +Only Trusted Server-owned cookie names are reported. Arbitrary cookie names and +values are excluded. The endpoint cannot claim knowledge of browser attributes, +expiry, or cookies the browser withheld from the request. + +### 9.3 Report envelope + +```text +TraceReportV1 + schema_version: 1 + captured_at: RFC 3339 UTC timestamp + request_context: TraceRequestContextV1 + gpt_diagnostics: GptDiagnosticsExportV1-or-successor +``` + +The trace envelope owns request context and transport. TS Console continues to +own its nested schema. Compatibility is explicit: the viewer supports a small +documented set of TS Console schema versions and rejects unknown versions with +an actionable message rather than guessing. + +### 9.4 Storage limits and expiry + +- Storage key: a namespaced, versioned constant owned by the diagnostics + module. +- Maximum encoded report size: 512 KiB. +- Maximum report age: 15 minutes from `captured_at`. +- One report per tab; a new explicit snapshot replaces the old report. +- Invalid, oversized, expired, or unsupported reports are removed immediately. +- `Clear report and end tracing` removes the storage entry and clears the + activation cookie. + +These are product limits, not assumptions about browser quota. A storage write +failure is handled even when the report is below the application limit. + +## 10. Auction and rendering evidence + +The report uses TS Console's evidence model. It must preserve the distinction +between: + +- A Trusted Server opportunity. +- A provider response. +- A selected Trusted Server candidate. +- A GPT request and response. +- A non-empty GPT render. +- Trusted Server creative-bridge evidence. +- Creative load and viewability. +- A publisher or client-side refresh. + +The viewer must not infer that Trusted Server rendered an ad merely because GPT +reported a filled slot. Ambiguous and unattributed cycles remain explicit. + +Current diagnostics tokens exist only on delivered winning bids. No-bid, +failed, skipped, hidden, unresolved, and direct `/auction` paths can lack server +correlation. The report displays the available observed facts and `Unknown` +rather than manufacturing a correlation. + +Provider-call telemetry is auction-wide, while bid rows exist only for returned +bids. Exact `provider X was asked for slot Y` and exact no-bid causality require +a future provider-impression disposition model. That instrumentation is not +silently assumed by this design. + +Timing fields introduced by #1074/#1076 are consumed only after they merge and +are propagated through the live diagnostics contract. The report never queries +Tinybird, and it does not combine browser `performance.now()` values with +server-relative timing as though they were one clock. + +Bidder and winning price are included only if #1081 approves them in the public +TS Console export contract. #1050 does not independently weaken the existing +privacy policy. + +## 11. Network scope + +The report is inspired by Fastly Debug, not a clone. Version one uses facts +already present or reasonably addable to the platform request abstraction. + +Supported categories: + +- Masked client address. +- Country, region, and ASN when available. +- HTTP version. +- TLS protocol and cipher. +- Optional JA4 and H2 fingerprints. +- Edge hostname, region, and POP. +- Capture time. + +Explicitly excluded: + +- DNS resolver address and resolver ASN. +- Active bandwidth or speed tests. +- TCP congestion window, next hop, RTT, and retransmit counters. +- DDoS/internal Fastly classifications. +- Arbitrary request headers. +- Full client IP in HTML or export. + +Unsupported optional fields are omitted rather than populated with fabricated +fallbacks. + +## 12. Security and privacy + +### 12.1 Allowlist boundary + +The report serializer constructs a new public model field by field. It never +serializes request structs, cookie parsers, auction requests, telemetry rows, or +browser objects wholesale. + +Forbidden data includes: + +- Raw `Cookie` and `Set-Cookie` headers. +- EC IDs, EIDs, bidder user IDs, and consent strings. +- Unmasked client IP. +- Query strings and fragments. +- Fastly or internal request identifiers that can join to user-bearing logs. +- Internal `AuctionRequest.id`. +- Bid requests/responses, losing-bid payloads, targeting, creative markup, + cache URLs, and stack traces. + +### 12.2 Same-origin script visibility + +Publisher and third-party scripts running on the publisher origin can access +`sessionStorage`. Therefore the stored model must be safe even if read by any +same-origin script. A random storage key, closed shadow root, or public endpoint +does not change this requirement. + +### 12.3 Response hardening + +Both the endpoint and every active diagnostic publisher response are terminally +`private, no-store`. The endpoint also sends: + +- `Content-Type: text/html; charset=utf-8` +- `X-Content-Type-Options: nosniff` +- `Referrer-Policy: no-referrer` +- `Content-Security-Policy` restricting content to the endpoint's own static + assets and prohibiting framing +- A restrictive `Permissions-Policy` + +The endpoint makes no third-party requests. Dynamic JSON embedded in HTML uses +the repository's script-safe serializer and is never concatenated into +executable JavaScript. + +### 12.4 Shared templates and ESI + +Per-request trace context must never enter a shared template or ESI fragment. +The existing diagnostics private/no-store decision remains a load-bearing gate. +Tests must prove that late response-header handlers cannot make traced content +publicly cacheable. + +## 13. Failure handling + +- Disabled route: local privacy-safe `404`. +- Unsupported method: local `405`; never publisher fallback. +- Optional platform fact unavailable: omit the field and continue. +- Cookie parser failure: report `present_invalid` without the value. +- Diagnostics context serialization failure: omit the context, log a bounded + server error, and preserve publisher delivery. +- TS Console capture failure: fail open for advertising and show incomplete + coverage in diagnostics. +- Storage unavailable, quota exceeded, or serialization oversized: remain on + the publisher page, announce the error, and offer direct download. +- Missing snapshot on endpoint: show setup state, not an empty successful + report. +- Expired, malformed, or unknown report schema: clear it and explain that the + user must reproduce again. +- Clipboard or Web Share unavailable: keep JSON download available. +- Export failure: retain the on-screen report and show an accessible error. + +Diagnostic failures must never suppress, delay, add, remove, or reorder GPT +requests, auctions, targeting, or creative rendering. + +## 14. Testing strategy + +### 14.1 Core unit tests + +- Configuration defaults off and rejects trace-page enablement without GPT + diagnostics. +- Exact route, query, method, encoded-path, and fallback behavior. +- Session cookie set/clear attributes and duplicate-directive fail-closed + behavior. +- Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, + and origin fetch. +- Cookie-health parser covers absent, valid, malformed, duplicate, non-UTF-8, + and oversized inputs without retaining values. +- Request-context serializer masks IPv4/IPv6 and omits query, raw headers, IDs, + and unsupported fields. +- Active responses remain terminally private/no-store under hostile late header + overrides. +- Dynamic HTML/JSON values cannot close elements or create executable script. + +### 14.2 Adapter parity tests + +- Fastly route registration and optional field mapping. +- Axum, Cloudflare, and Spin return the common route/schema with unavailable + fields omitted. +- Named route failures never fall through to publisher origin. +- HEAD and unsupported methods behave identically across adapters. +- Fastly fingerprint fields respect the existing fingerprint-debug gate. + +### 14.3 JavaScript unit tests + +- Explicit snapshot only; no continuous `sessionStorage` writes. +- Size limit, schema validation, expiry, replacement, clearing, and storage + exceptions. +- Same-tab navigation occurs only after a successful write. +- Viewer handles absent optional network facts and every cookie-health state. +- Forbidden fields never enter storage or export fixtures. +- Download filename and MIME type are deterministic. +- Copy and Web Share success, rejection, absence, and fallback behavior. +- 320-pixel layout, keyboard navigation, focus handling, and accessible status + announcements. + +### 14.4 Browser integration tests + +- First endpoint visit sets the session and shows setup state. +- A real fixture reload activates diagnostics and captures multiple slots. +- `View trace results` navigates in the same tab and renders the captured + request context and slot evidence. +- Empty, filled, ambiguous, no-candidate, and unattributed slot states remain + distinct. +- Reloading the trace page retains an unexpired same-tab report. +- A new tab cannot access the original tab's report. +- Disabling clears both cookie and report. +- Back-forward-cache restoration is not described as a fresh traced request; + the setup page tells the user to reload. +- Export JSON matches the displayed versioned model. +- Inactive publisher traffic has no trace assets, storage access, listeners, or + cache-policy change. + +### 14.5 Manual acceptance + +Verify on current iOS Safari and Android Chrome using a representative publisher +fixture: + +- A layperson can follow the page instructions without developer tools. +- Touch targets, scrolling, zoom, safe areas, download, copy, and native share + behavior are usable. +- The user can distinguish setup information from captured-page information. +- A failed share or download does not lose the visible report. + +## 15. Rollout and observability + +- Ship default-off. +- Enable first in a controlled staging publisher configuration. +- Validate response cache headers and CDN behavior before production use. +- Validate with redacted fixtures before real publisher traffic. +- Log only route outcome, schema version, report-present boolean, and bounded + error category. Never log report contents or cookie/network values. +- Roll back by disabling `trace_page_enabled`; existing `?ts_console=1` + diagnostics remain independently configurable. + +## 16. Acceptance criteria + +1. With the feature disabled, exact trace routes return local `404` and ordinary + traffic is unchanged. +2. A mobile user can enable tracing by opening only `/_ts/trace`; no target URL, + credentials, or trace ID is required. +3. The setup page accurately explains that the problem must be reproduced after + activation. +4. A subsequent real publisher-page reload captures redacted request context + and existing TS Console evidence without altering ad behavior. +5. `View trace results` transfers one bounded snapshot in the same tab and opens + the report page without server-side storage. +6. The report separates network, cookie health, auction/render evidence, and + coverage/unknowns. +7. JSON export contains the same versioned allowlisted information shown on the + page. +8. No raw cookies, user IDs, full IPs, consent strings, query strings, internal + auction IDs, targeting, or creative payloads appear in HTML, browser storage, + logs, or export. +9. Trace HTML and active publisher pages remain terminally private/no-store. +10. Missing platform fields, incomplete auction correlation, storage failure, + and unavailable share APIs degrade honestly without affecting advertising. +11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader + navigation. +12. Auction fields owned by #1081 are consumed through its versioned public + contract rather than duplicated in #1050. + +## 17. Implementation sequencing + +This design is one product flow but should be implemented in dependency order: + +1. Core request-context schema, cookie-health classification, configuration, + and endpoint shell. +2. Adapter route parity and Fastly optional network enrichment. +3. TS Console request-context envelope and explicit same-tab snapshot handoff. +4. Mobile viewer, export/copy/share, expiry, and clearing. +5. Integration with the current TS Console schema. +6. Additive adoption of #1081 and #1074/#1076 fields after their contracts + merge. +7. Browser, privacy, cache, and real-device acceptance. + +The implementation plan must not claim completion of #1081 or the open timing +PRs as part of #1050. If those dependencies are unavailable, the report ships +only with current observed auction/render evidence and labels unavailable fields +honestly. + +## 18. Rejected alternatives + +### `/_ts/admin/trace?target=/article` + +Rejected because it requires the user to supply the affected URL twice, adds +target validation and open-redirect risk, and is unsuitable for a layperson. + +### Basic Authentication + +Rejected for the mobile end-user workflow. Authentication also would not make +it safe to inject raw secrets into a publisher page containing third-party +JavaScript. + +### Server-managed trace sessions + +Rejected because they require shared storage, report authorization, expiry, +deletion, and operational infrastructure beyond the issue's needs. + +### Synthetic auction on the endpoint + +Rejected because it does not reproduce the real page's DOM, GPT lifecycle, +consent context, refresh path, or auction timing and could produce misleading +results. + +### Endpoint-only report with no publisher-page integration + +Rejected because a request to `/_ts/trace` cannot observe rendering that +occurred in another document. + +### Query-only in-page console + +The existing `?ts_console=1` flow remains supported, but it is not the complete +answer to #1050: the issue asks for a memorable mobile endpoint and a +Fastly-style consolidated HTML report. The endpoint/viewer builds on rather +than replaces the console. + +### Cross-page URL payload + +Rejected because fragments or query strings containing the report create URL +length, history, logging, referrer, and accidental-sharing risks. + +## 19. Known limitations + +- The user must reproduce the problem after enabling tracing. +- Same-tab storage prevents cross-device and cross-tab sharing; JSON export is + the handoff artifact. +- Publisher-origin scripts can read the stored public-safe report. +- Browser privacy settings may disable storage, clipboard, download, or share + capabilities. +- Current server/browser correlation does not cover every no-bid, skipped, + failed, hidden, unresolved, or direct-auction path. +- Fastly-only transport details do not exist on every adapter. +- The current 30-pixel TS Console controls are not sufficient for this mobile + report; the endpoint uses independent 44-pixel touch targets. +- `fastly-debug.com` fields that require resolver, TCP, or active speed probes + remain out of scope. + +These limitations are displayed in operator documentation and, where relevant, +in the report itself. They are not hidden behind apparently successful empty +states. From 8899dcadc89d1c4db3c5902078fa3a9e2cbf0c42 Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 15:33:05 +0530 Subject: [PATCH 031/230] docs: harden mobile ad trace design --- ...-mobile-ad-render-trace-endpoint-design.md | 754 ++++++++++++++---- 1 file changed, 591 insertions(+), 163 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e46f9892b..6ed9f8c48 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -16,16 +16,17 @@ Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a mobile end user who needs to reproduce an ad-rendering problem and give support -an exportable diagnostic report. +an exportable diagnostic report. Visiting the page is read-only. The user +intentionally enables or ends tracing with a same-origin POST action. The endpoint is both a setup page and a report viewer. On the first visit it -enables the existing GPT diagnostics browser session and explains how to -reproduce the problem. The user then returns to the real publisher page and -reloads it. Trusted Server supplies redacted request context, while the existing -TS Console records GPT, auction, and render evidence. A `View trace results` -action creates one bounded, allowlisted snapshot in same-tab `sessionStorage` -and navigates to `/_ts/trace`. The endpoint reads that snapshot, presents a -mobile-first HTML report, and offers JSON export, copy, and progressive Web +offers a large `Enable tracing` action and explains how to reproduce the +problem. The user then returns to the real publisher page and reloads it. +Trusted Server supplies redacted request context, while the existing TS Console +records GPT, auction, and render evidence. A `View trace results` action creates +one bounded, allowlisted snapshot in same-tab `sessionStorage` and navigates to +`/_ts/trace`. The endpoint reads that untrusted snapshot, validates it, presents +a mobile-first HTML report, and offers JSON export, copy, and progressive Web Share actions. The design introduces no report database, server-side trace store, report ID, @@ -68,10 +69,12 @@ service. - Report health for an explicit allowlist of Trusted Server cookies without exposing their values. - Present the versioned, allowlisted TS Console evidence for every retained GPT - slot and request cycle. + slot and request cycle that fits the public report bounds, with explicit + omission counts when deterministic size truncation is required. - Support a full report in a narrow mobile viewport without developer tools. - Export the same allowlisted model as formatted JSON. -- Keep capture bounded, same-tab, temporary, and inactive by default. +- Keep the supported capture journey bounded, same-tab, temporary, and inactive + by default without treating browser storage as a security boundary. - Preserve normal auction, GPT, rendering, origin, and caching behavior whenever diagnostics is inactive. - Keep core behavior platform-neutral while allowing Fastly to provide richer @@ -101,7 +104,7 @@ service. ## 5. Decisions -### 5.1 Public, redacted endpoint +### 5.1 Public, redacted endpoint with intentional activation `/_ts/trace` is public when explicitly enabled by deployment configuration. It is not placed under `/_ts/admin`, because the intended user is a layperson on a @@ -112,16 +115,27 @@ Public access is safe only because both the page and export use a strict allowlist. The activation cookie is a feature toggle, not authentication. No field becomes eligible merely because tracing is active. +`GET /_ts/trace` is read-only and never activates or ends tracing. Activation +and deactivation use an in-page same-origin `fetch` POST accepted only when its +fixed custom action header, `Origin`, and Fetch Metadata identify the publisher +origin. Requests with a conflicting or missing signal fail closed. The POST +updates the existing page rather than adding a history entry, so browser Back +can still reach the article. This prevents an unrelated site from silently +toggling diagnostics through a top-level GET while preserving a one-URL, +one-tap mobile workflow. This control does not defend against code already +executing on the publisher origin. + ### 5.2 Reuse the existing diagnostics session The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics activation semantics rather than creating a second `ts-trace` session. The -cookie remains host-only, `Secure`, `HttpOnly`, `SameSite=Lax`, and -browser-session scoped. - -`/_ts/trace?enabled=false` clears the activation cookie and browser snapshot. -Other values, duplicate `enabled` parameters, and malformed directives fail -closed and do not mutate session state. +cookie remains host-only, `Secure`, `HttpOnly`, and `SameSite=Lax`. +`POST /_ts/trace/enable` sets it with a fixed 30-minute `Max-Age` and does not +refresh that lifetime on publisher requests; `POST /_ts/trace/end` clears it. +Neither action accepts state-changing query parameters. The shorter endpoint +lifetime bounds accidental private/no-store operation if a user forgets to end +tracing; the existing technical query flow keeps its existing session-cookie +semantics. ### 5.3 Browser-local, explicit handoff @@ -134,20 +148,26 @@ TS Console remains memory-only during observation. It writes a report to 4. Stores it under one versioned key in the current tab. 5. Navigates the same tab to `/_ts/trace`. -Continuous persistence is prohibited. Opening the endpoint in another tab does -not retrieve the snapshot. Closing the tab deletes it according to browser -session-storage semantics. +Continuous persistence is prohibited. Same-tab navigation is the supported +handoff, not an isolation guarantee: a browser may copy session storage into an +opener-created tab or preserve it during session restore. Same-origin scripts +and service workers can read, replace, or forge the snapshot. The viewer +therefore treats it as untrusted, applies an application-level expiry, and +labels it browser-observed rather than authoritative. ### 5.4 Forward reproduction, not historical diagnosis -The first endpoint visit enables tracing for subsequent eligible document +The user's explicit activation enables tracing for subsequent eligible document navigations. The setup page must say plainly that the user needs to return to the affected page, reload it, and reproduce the problem. If the user replaced the affected URL in the address bar with `/_ts/trace`, the page offers a `Return to previous page` action backed by browser history and -then instructs the user to reload once. The design does not claim that -back-forward-cache restoration caused a new server request. +then instructs the user to reload once. History is only a convenience: it may +lead to a messaging app, search page, or unrelated site. The page includes a +fallback instruction to reopen the affected article on the same hostname and +in the same tab. The design does not claim that back-forward-cache restoration +caused a new server request. Support should preferably give the user the trace URL before reproduction. The product does not attempt to discover the previous URL through `Referer`, because @@ -161,9 +181,11 @@ handoff, and export. #1081 remains the owner of creative numbering, auction classification, bidder/price policy, terminology, and normalized auction/render timing. -The trace report consumes TS Console's public versioned export contract. It +Version one consumes `GptDiagnosticsExportV1` through TS Console's public export +contract and projects it into a distinct redacted `TraceGptDiagnosticsV1`. It does not read TS Console internals or create an alternate slot correlation -engine. +engine. #1081 and #1074/#1076 are additive follow-up work and are not release +gates for this version. ## 6. User experience @@ -172,13 +194,22 @@ engine. `GET /_ts/trace` returns a mobile-first HTML page with: - Title: `Trusted Server ad diagnostics`. -- State: `Trace ready` after the response establishes the session cookie. +- State derived from the setup request: `Tracing is off` unless the server + observed a valid existing diagnostics cookie, including one activated through + the technical query flow. - A short explanation that no previous ad failure can be recovered. - Network and cookie health for the setup request, labeled `Setup request`. -- Primary action: `Return to previous page` when browser history permits. +- Primary action: `Enable tracing`, implemented as an in-page same-origin fetch + POST that does not add a history entry. +- After activation and a successful state-verification request, state: `Tracing +is on — cookie observed by server`. +- After activation, primary action: `Return to previous page` when browser + history permits. - Secondary instructions: return to the affected page, reload once, reproduce the problem, then select `View trace results`. -- Action to disable tracing. +- A recovery instruction to reopen the affected article on the exact same + hostname and in the same tab if browser history is not useful. +- Action to end tracing when it is active. The page must not imply that setup-request network facts or an empty auction section describe the affected page. @@ -189,8 +220,12 @@ The existing TS Console remains available. On mobile it gains a prominent `View trace results` action. Selecting it never changes ad behavior; it only snapshots retained observations and navigates after serialization succeeds. -If the snapshot cannot be stored, the page remains in place, announces the -failure, and keeps the existing direct JSON export available. +If a valid bounded snapshot is built but browser storage rejects it, the page +remains in place, announces the storage failure, and offers a direct download +of that same combined `TraceReportV1` envelope. If projection, validation, or +size bounding fails before a valid report exists, the page reports capture +failure and does not mislabel the existing GPT-only export as an equivalent +fallback. ### 6.3 Report visit @@ -208,6 +243,16 @@ The setup request's facts are not merged into or substituted for missing traced page facts. Missing fields display `Unavailable`; missing evidence displays `Not observed` or `Unknown`, following TS Console terminology. +The report begins with `Browser-observed, unverified diagnostic data`. It does +not claim that the snapshot is authentic or suitable as forensic or security +evidence. + +`Copy` copies formatted JSON. `Share` supplies the same JSON file to the native +Web Share sheet only after an explicit tap and tells the user that the selected +app will receive it. If file sharing is unsupported or rejected, the viewer +keeps Copy and Download available; it does not silently share a URL or upload +the report. + ### 6.4 Mobile and accessibility requirements - Support viewport widths down to 320 CSS pixels without horizontal page @@ -224,11 +269,17 @@ page facts. Missing fields display `Unavailable`; missing evidence displays ## 7. Architecture ```text -First GET /_ts/trace +GET /_ts/trace | - |-- core route builds setup request context + |-- early reserved-route classifier terminates locally + |-- HTML explains forward reproduction; no state mutation + v +POST /_ts/trace/enable after explicit user action + | + |-- validates same-origin request signals |-- response sets __Host-ts-console - |-- HTML explains forward reproduction + |-- client requests /_ts/trace/state + |-- server reports whether the new request carried a valid cookie v Real publisher document reload | @@ -243,7 +294,7 @@ User selects "View trace results" |-- same-tab sessionStorage write |-- location.assign('/_ts/trace') v -Second GET /_ts/trace +Report GET /_ts/trace | |-- static report shell reads and validates TraceReportV1 |-- mobile HTML renders sections @@ -253,7 +304,8 @@ Second GET /_ts/trace ### 7.1 Core responsibilities - Define configuration and route behavior. -- Register the route before publisher fallback on every supported adapter. +- Provide a shared exact-path reserved-route classifier that runs before event + context, filters, auctions, named routes, or publisher fallback. - Define the platform-neutral request-context and report-envelope schemas. - Build cookie-health facts through read-only parsing. - Convert `ClientInfo` and available geo data into the public network allowlist. @@ -263,10 +315,11 @@ Second GET /_ts/trace ### 7.2 Adapter responsibilities -- Register the named route with exact method handling. +- Invoke the reserved-route classifier at the earliest adapter dispatch point + with exact path and method handling. - Populate optional `ClientInfo` fields available on the platform. -- Fastly may supply POP, HTTP version, TLS, JA4, H2 fingerprint, and edge - server data when the SDK exposes them. +- Fastly may supply bounded POP, HTTP version, TLS, and edge-server data when + the SDK exposes them. JA4 and H2 fingerprints are excluded from version one. - Other adapters return the same schema with unsupported fields absent. - Adapter-specific errors omit optional facts rather than failing publisher delivery. @@ -275,11 +328,14 @@ Second GET /_ts/trace - Accept the immutable redacted request context at initialization. - Preserve the existing bounded TS Console observation store. -- Build and validate `TraceReportV1` on explicit user action. -- Store only one report in same-tab `sessionStorage`. +- Build and validate `TraceReportV1` with a redacted + `TraceGptDiagnosticsV1` projection on explicit user action. +- Store only one supported report for the same-tab workflow in + `sessionStorage`, while treating its contents as untrusted. - Render the report shell from the validated model. -- Implement download, copy, progressive Web Share, clearing, expiry, and - accessible status reporting. +- Implement equivalent combined-report download, formatted-JSON copy, + progressive JSON-file Web Share, clearing, expiry, and accessible status + reporting. - Never upload diagnostic data or issue a telemetry query. ## 8. Route and configuration contract @@ -296,23 +352,102 @@ Rules: - `trace_page_enabled = true` requires `enabled = true`; invalid combinations fail configuration validation. -- `GET /_ts/trace` returns the setup/report HTML and establishes the session. -- `GET /_ts/trace?enabled=false` returns the shell, clears the cookie, and asks - the client to clear the stored snapshot. -- `HEAD /_ts/trace` returns the same status and headers without a body but does - not mutate the cookie. -- All other methods return a local `405 Method Not Allowed` with `Allow: GET, -HEAD`. -- Disabled deployments return a local `404` for the exact route and never fall - through to the publisher origin. -- Extra path segments, encoded separators, duplicate parameters, and lookalike - paths do not match. -- The route never creates or refreshes an EC, ingests EIDs, runs an auction, - fetches the publisher origin, or emits auction telemetry. +- Operator documentation beside this option states that the public page makes + the allowlisted presence/validity of four HttpOnly Trusted Server cookies + visible to same-origin JavaScript whenever the feature is enabled. It also + states that masked IP prefixes and coarse geo remain potentially personal or + pseudonymous network data. Enabling the option is the deployment's explicit + acceptance of those bounded disclosures. +- `GET /_ts/trace` returns the setup/report shell without changing cookies or + browser storage at the HTTP layer. After load, the explicitly included viewer + script may remove a rejected or expired local entry. Unrelated query + parameters do not activate or deactivate tracing and are not reflected into + the page or export. +- `HEAD /_ts/trace` returns the GET status and headers without a body or state + mutation. +- `GET /_ts/trace/state` returns private/no-store JSON containing only + `observed_active: true|false`, determined from whether that request carried + exactly one valid diagnostics cookie. `HEAD` returns the same status and + headers without a body. Other methods return local 405 responses. +- `GET` and `HEAD` on exactly `/_ts/trace/assets/v1.js` and + `/_ts/trace/assets/v1.css` return fixed versioned assets. They contain no + request or report data and may use immutable public caching. Other methods + return local 405 responses. These v1 URLs are immutable byte contracts: any + JS or CSS byte change requires a new asset-set URL such as `v2.js`/`v2.css` + and an updated shell reference; a release never replaces bytes at a published + immutable URL. +- `POST /_ts/trace/enable` accepts no query parameters, validates an empty body, + the exact `X-TS-Trace-Action: enable` header, and same-origin request signals; + sets the diagnostics cookie; and returns a small local JSON result. A success + response means only that the server requested the cookie change. +- `POST /_ts/trace/end` applies the same validation, clears the diagnostics + cookie using `X-TS-Trace-Action: end`, and returns a small local JSON result. + After explicit user confirmation, client JavaScript independently attempts + local report deletion and the end POST. Neither result gates the other. +- For both POST paths, absent or exactly-zero `Content-Length` is accepted, + `Transfer-Encoding` is rejected, and the adapter reads at most one byte when + it must verify an absent length. Any body byte or positive/invalid length + returns local `413 Payload Too Large` without draining or processing an + unbounded body. The one-byte read inherits a maximum two-second adapter + request-body deadline; timeout returns local `408 Request Timeout` with no + mutation. +- State-changing POSTs require an `Origin` exactly matching the canonical + request origin and `Sec-Fetch-Site: same-origin`. Missing, conflicting, + malformed, cross-site, or duplicate control values return local `403` without + mutation. This deliberately targets current supported mobile browsers rather + than weakening the check for legacy clients. +- The canonical request origin comes from adapter-owned inbound URL/scheme and + validated authority data, never an arbitrary forwarded header. Both it and + the single parsed `Origin` header are serialized with lowercase host and + default ports removed before exact comparison. Invalid or multi-valued host, + authority, scheme, or origin input fails closed. +- Unsupported methods on a shell or state-changing path return a local 405 + Method Not Allowed response with the path-specific `Allow` header. +- Disabled deployments return a local `404` for the complete trace route set, + including assets, and never fall through to the publisher origin. +- The `/_ts/trace` namespace is reserved. A trailing slash, extra path segment, + unsupported asset name, repeated separator, or lookalike beneath that + namespace returns a local `404`; an encoded separator or ambiguous dot + segment returns a local `400`. None falls through to the publisher origin. + The adapter classifies from its canonical parsed path while retaining enough + raw-path information to reject ambiguous encodings consistently. + +Every adapter implements the following order: + +1. Parse the method, canonical host/origin, path, query, and bounded headers + required for route safety. +2. Classify an exact Trusted Server reserved path. +3. For a trace path, terminate locally after only trace-specific validation and + bounded request-context inspection, including an optional read-only platform + geo lookup used solely for the displayed setup request. +4. For all other paths, continue through the adapter's ordinary event context, + authentication, request filters, geo enrichment, EC/EID processing, named + routes, auction handling, telemetry, and publisher fallback. + +Consequently a trace route never creates or finalizes an ordinary event +context, invokes publisher-configured filters, creates or refreshes an EC, +ingests EIDs, runs an auction, fetches the publisher origin, or emits auction +telemetry. Tests must verify ordering in Fastly, Axum, Cloudflare, and Spin; +ordinary named-route registration alone does not satisfy this contract. + +After an enable or end POST succeeds, the client performs a no-store state GET. +It claims `Tracing is on — cookie observed by server` only when that separate +request reports active, and `Tracing is off — cookie absent on server request` +only when it reports inactive. A mismatch or failed verification is +`Activation unconfirmed` or `Deactivation unconfirmed` and offers an idempotent +retry. These are server-observation statements, not proof that browser state is +authentic: same-origin service workers can forge or suppress the whole exchange. + +Versioned assets may remain in a CDN or browser cache after the feature is +disabled. Cache misses return the configured local 404, but rollback relies on +the uncached shell, state, and action routes being disabled; inert cached assets +alone cannot activate tracing or access a report page. The current `?ts_console=1` and `?ts_console=0` activation flow remains supported for technical users. Both activation surfaces drive the same cookie -and runtime; they must not create two concurrent diagnostic modes. +and runtime; they must not create two concurrent diagnostic modes. That +pre-existing query flow has its existing top-level-navigation activation risk; +#1050 neither expands it to the new trace GET nor claims to remediate it. ## 9. Data contracts @@ -325,9 +460,6 @@ documents: TraceRequestContextV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp - page: - origin: publisher origin - path: normalized path network: masked_client_ip?: string country?: string @@ -336,8 +468,6 @@ TraceRequestContextV1 http_version?: string tls_protocol?: string tls_cipher?: string - tls_ja4?: string - h2_fingerprint?: string edge_hostname?: string edge_region?: string edge_pop?: string @@ -348,17 +478,27 @@ TraceRequestContextV1 diagnostics_session: CookieHealth ``` -The page field omits query and fragment data. It does not contain origin-facing -URLs, referrers, or arbitrary headers. +The request-context envelope intentionally contains no page URL, path, +referrer, query, or fragment. During the field-by-field trace projection, +`GptDiagnosticsExportV1.page.origin` is retained after validation and its +`pathname` is replaced with the literal `/[redacted]`. The trace viewer accepts +only that literal. Version one therefore does not store or export an exact page +path. Any future route-template policy requires a new schema and privacy review +because paths can contain accounts, emails, preview tokens, and other secrets. `masked_client_ip` uses a deterministic display-only mask for the current request: IPv4 keeps at most the first 24 bits and IPv6 keeps at most the first 48 bits. The full address never enters HTML, JavaScript, browser storage, or -export. +export. These prefixes can still be personal or pseudonymous network data; the +report labels them as approximate network identifiers and the operator privacy +decision covers them explicitly. -JA4 and H2 fingerprints are optional probabilistic identifiers. They are -included only when the deployment has separately enabled the existing -fingerprint diagnostic capability. Their absence is not an error. +All platform strings are normalized to printable characters and bounded before +they enter logs, HTML, storage, or export. Country uses at most 2 ASCII +characters; region and POP 32 UTF-8 bytes; HTTP/TLS enumerations 32 bytes; and +edge hostname/region 128 bytes. Values that fail their field contract are +omitted and produce only a bounded error category. JA4 and H2 fingerprints are +not members of `TraceRequestContextV1`. ### 9.2 Cookie health @@ -367,20 +507,62 @@ CookieHealth state: absent | present_valid | present_invalid | duplicate | unavailable source: request - detail?: allowlisted enum + detail?: + valid_ec_format | valid_eids_format | valid_tester_value + | valid_diagnostics_value | malformed | oversized + | unsupported_value | multiple_values + | header_too_large | header_not_utf8 ``` -Allowed details describe shape, not value, for example `valid_ec_format`, -`malformed`, `oversized`, or `activation_pending_response`. - -The parser must inspect the incoming request before any diagnostics-cookie -sanitization, while preserving existing authoritative-cookie and consent -semantics. Inspection is read-only: it must not generate an EC, touch the -identity graph, sync partner IDs, or extend any cookie lifetime. - -Only Trusted Server-owned cookie names are reported. Arbitrary cookie names and -values are excluded. The endpoint cannot claim knowledge of browser attributes, -expiry, or cookies the browser withheld from the request. +Details describe shape, never value. `absent` has no detail; `duplicate` uses +`multiple_values`; `unavailable` uses `header_too_large` or `header_not_utf8`; +and a valid state uses its cookie-specific valid detail. + +The classifier uses this deterministic contract: + +- Inspect all `Cookie` header fields in wire order, up to a combined 16 KiB. + Exceeding the cap or encountering any non-UTF-8 header makes all four states + `unavailable`; no partial result is presented as authoritative. +- Split each readable header on semicolons and trim optional ASCII whitespace. + A valid pair contains a non-empty RFC 6265 token name, one `=`, and the + remaining bytes as its value; additional `=` bytes belong to the value. Empty + segments and malformed pairs with an unrelated name are ignored. A segment + with no `=` counts as one malformed reserved occurrence only when its first + whitespace-delimited token is exactly a reserved name; a name such as + `ts-ec-extra` remains unrelated. No malformed unrelated pair poisons a + reserved-cookie result. +- Count exact, case-sensitive reserved names before passing values to existing + parsers. Zero occurrences is `absent`; more than one is `duplicate`, + regardless of whether one value would otherwise be valid. Duplicate + precedence is therefore diagnostic rather than first- or last-value + selection. +- Per-value limits are 512 bytes for `ts-ec`, 8 KiB for `ts-eids`, and 16 bytes + each for `ts-tester` and `__Host-ts-console`. A single value beyond its limit + is `present_invalid/oversized`; it does not change the other three states. +- One `ts-ec` occurrence is valid only when the canonical EC cookie validator + accepts its complete value. +- One `ts-eids` occurrence is valid only when the existing bounded Base64/JSON + EID parser accepts its complete value, including its current 8 KiB value cap. +- One `ts-tester` occurrence is valid only when its value is exactly `true`. +- One `__Host-ts-console` occurrence is valid only when its value is exactly + `1`. +- A single rejected value is `present_invalid` with exactly one of the public + details `malformed`, `oversized`, or `unsupported_value`. Parser error text + and the value itself never enter the report or logs. + +The parser inspects the incoming request before diagnostics-cookie sanitation, +while preserving existing authoritative-cookie and consent semantics. It must +scan without using the current lossy `CookieJar` representation, which skips +malformed pairs and cannot preserve duplicate evidence. Inspection is +read-only: it must not generate an EC, touch the identity graph, sync partner +IDs, or extend any cookie lifetime. + +Only those four Trusted Server-owned cookie names are reported. Arbitrary +cookie names and values are excluded. The endpoint cannot claim knowledge of +browser attributes, expiry, or cookies the browser withheld from the request. +Because the result reveals presence and validity of HttpOnly cookies to +same-origin JavaScript, enabling this public feature requires an explicit +operator privacy decision documented beside `trace_page_enabled`. ### 9.3 Report envelope @@ -389,28 +571,132 @@ TraceReportV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp request_context: TraceRequestContextV1 - gpt_diagnostics: GptDiagnosticsExportV1-or-successor + gpt_diagnostics: TraceGptDiagnosticsV1 + truncation: + omitted_request_cycles: u16 + omitted_callback_issues: u16 + omitted_attribution_issues: u16 + omitted_nested_values: u16 ``` -The trace envelope owns request context and transport. TS Console continues to -own its nested schema. Compatibility is explicit: the viewer supports a small -documented set of TS Console schema versions and rejects unknown versions with -an actionable message rather than guessing. +`TraceGptDiagnosticsV1` is a trace-owned projection sourced only from +`GptDiagnosticsExportV1`. It contains: + +- `schema_version: 1` and `source_schema_version: 1`; +- the source `capturedAt` value; +- `page.origin` after validation and `page.pathname` fixed to `/[redacted]`; +- field-for-field allowlisted copies of the current v1 slots, requests, + callback issues, attribution issues, coverage, and metadata, subject to the + bounds and truncation below. + +It is deliberately not named or represented as `GptDiagnosticsExportV1`, +because the fixed pathname and trace-level bounds change the source field +semantics. TS Console continues to own the source schema; the trace envelope +owns its public projection and transport. The initial compatibility matrix is +exactly `TraceReportV1` plus `TraceGptDiagnosticsV1`, sourced from +`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, trace-auction, +or source version with an actionable message rather than guessing. A future TS +Console successor requires an additive source compatibility change and, if the +public projection changes, a new trace-envelope version. + +Origin validation requires a parseable HTTP(S) origin whose canonical +serialization exactly equals `window.location.origin`; credentials, paths, +queries, and fragments are rejected. Outer and source capture times require +strict RFC 3339 UTC strings and must be within 60 seconds of `stored_at_ms`. +`TraceRequestContextV1.captured_at` requires strict RFC 3339 UTC but may be older +because it represents the publisher document request rather than snapshot time. ### 9.4 Storage limits and expiry - Storage key: a namespaced, versioned constant owned by the diagnostics module. -- Maximum encoded report size: 512 KiB. -- Maximum report age: 15 minutes from `captured_at`. -- One report per tab; a new explicit snapshot replaces the old report. -- Invalid, oversized, expired, or unsupported reports are removed immediately. -- `Clear report and end tracing` removes the storage entry and clears the - activation cookie. +- Stored value: `{ stored_at_ms, report }`, where `stored_at_ms` is generated by + the capture code and is not taken from report content. +- Maximum encoded size: 512 KiB, defined as the byte length of the complete + compact UTF-8 `{ stored_at_ms, report }` JSON measured with `TextEncoder` + before storage. Formatted download size and JavaScript UTF-16 string length + are not used for enforcement. +- Maximum age: 15 minutes from `stored_at_ms`. Non-finite, negative, malformed, + more than 60 seconds in the future, or older values are rejected. A backward + wall-clock jump that places the timestamp beyond the tolerated future skew + also invalidates the entry. Expiry is exposure reduction, not a security + guarantee. +- One supported report for the current browsing context; a new explicit + snapshot replaces the old report. Browser opener cloning and session restore + may copy or retain it. +- The runtime validator accepts only the exact outer and trace-auction v1 + schemas, rejects unknown fields, applies the limits below, and checks compact + UTF-8 size before rendering. +- Invalid, oversized, expired, unsupported, or hostile reports are removed when + possible and otherwise ignored. Rendering uses DOM properties and + `textContent`, never report-derived HTML. +- After confirmation, `Clear report and end tracing` always attempts local + deletion, the validated end POST, and state verification as independent + retry-safe steps. Offline or server failure cannot prevent local deletion. + The UI reports server-observed cookie state and local-report state separately. + A distinct `Delete local report` action remains available whenever a report + is displayed, including after an earlier local-deletion failure. These are product limits, not assumptions about browser quota. A storage write failure is handled even when the report is below the application limit. +Runtime limits are part of the v1 contract: + +| Value | Limit | +| --------------------------------------------- | ------------------------------------------------------------ | +| Container nesting | 8 levels | +| Slots | 64 | +| Request cycles | 10 per slot before total-size truncation | +| Callback issues | 128 | +| Attribution issues | 128 | +| Requested slot sizes | 16 per cycle | +| Ad Manager yield-group or company IDs | 8 of each per cycle | +| Creative-failure enums | 16 per cycle | +| Origin | 255 UTF-8 bytes | +| GPT pathname in trace projection | Exact literal `/[redacted]` | +| Slot element ID and ad-unit path | 512 UTF-8 bytes each | +| Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | +| Any other string | 128 UTF-8 bytes | +| Enum | Exact documented value only | +| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | +| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | +| Visibility percentage | Finite number from 0 through 100 | +| Slot dimension | Finite integer from 1 through 100,000 | + +Every accepted string must be valid Unicode and must not contain C0/C1 control +characters or bidirectional override/isolate controls. This applies to browser +source fields as well as platform fields and precedes rendering or export. + +The snapshot builder creates a new field-by-field projection and rejects an +invalid source value rather than stringifying it. It retains only the first +documented number of requested sizes, yield-group IDs, company IDs, and creative +failure enums, recording discarded entries in `omitted_nested_values`; strings +are never silently shortened. It then measures the complete compact UTF-8 +storage wrapper. If it exceeds 512 KiB, it removes the globally oldest request +cycles first while retaining the newest cycle for each slot, then the oldest +callback issues, then the oldest attribution issues, and finally the oldest +remaining request cycles until the report fits. It records every removal in +`truncation`. +A report that still cannot fit after this bounded procedure fails snapshot +creation. The implementation must include a worst-case fixture proving the +result is bounded. + +All omission counters use checked addition. If any source collection would +make a counter exceed `u16::MAX`, projection rejects the source instead of +wrapping or saturating the count. + +For depth accounting, the `TraceReportV1` object—not its storage wrapper—is +level 1; entering either an object or an array increments the level by one; +primitives do not. No accepted report value may enter a ninth container level. +The storage wrapper is validated separately as the exact two-field object +`{ stored_at_ms, report }`. + +For deterministic ordering, a request cycle with no `requestedAtMs` sorts +before a cycle with a timestamp; otherwise cycles sort by `requestedAtMs`, then +`runtimeSlotNumber`, then `requestNumber`. Callback and attribution issues sort +by `timestampMs`, then their original array index. The builder preserves the +relative order of all retained records. + ## 10. Auction and rendering evidence The report uses TS Console's evidence model. It must preserve the distinction @@ -438,19 +724,21 @@ bids. Exact `provider X was asked for slot Y` and exact no-bid causality require a future provider-impression disposition model. That instrumentation is not silently assumed by this design. -Timing fields introduced by #1074/#1076 are consumed only after they merge and -are propagated through the live diagnostics contract. The report never queries -Tinybird, and it does not combine browser `performance.now()` values with -server-relative timing as though they were one clock. +Timing fields introduced by #1074/#1076 are outside the v1 compatibility +matrix. They may be consumed in a later version only after they merge and are +propagated through the public live-diagnostics contract. The report never +queries Tinybird, and it does not combine browser `performance.now()` values +with server-relative timing as though they were one clock. -Bidder and winning price are included only if #1081 approves them in the public -TS Console export contract. #1050 does not independently weaken the existing -privacy policy. +Bidder and winning price are not added by version one. A later version may +consume them only if #1081 approves them in the public TS Console export +contract. #1050 does not independently weaken the existing privacy policy. ## 11. Network scope -The report is inspired by Fastly Debug, not a clone. Version one uses facts -already present or reasonably addable to the platform request abstraction. +The report is inspired by Fastly Debug, not a clone. Version one exposes only +facts with a defined source and privacy boundary. Every field is optional; an +adapter must omit a value it cannot obtain directly and safely. Supported categories: @@ -458,10 +746,28 @@ Supported categories: - Country, region, and ASN when available. - HTTP version. - TLS protocol and cipher. -- Optional JA4 and H2 fingerprints. - Edge hostname, region, and POP. - Capture time. +Initial provenance and adapter support are: + +| Public field | Source | Fastly | Axum | Cloudflare | Spin | +| ------------------------------ | ---------------------------------------------------------------------------------------------- | --------------------------- | ---------------------- | ------------------------------------ | ----------- | +| `masked_client_ip` | `RuntimeServices.client_info.client_ip`, after trusted-client-IP resolution, then core masking | expected | expected | expected | expected | +| `country`, `region` | `RuntimeServices.geo.lookup(client_info.client_ip)` projected to `GeoInfo.country/region` | expected | unavailable by default | country expected, region unavailable | unavailable | +| `asn` | `GeoInfo.asn` | unavailable until populated | unavailable | unavailable until populated | unavailable | +| `http_version` | new bounded adapter mapping from inbound protocol metadata | optional | expected | optional | optional | +| `tls_protocol`, `tls_cipher` | `ClientInfo.tls_protocol/tls_cipher` | expected | unavailable | unavailable | unavailable | +| `edge_hostname`, `edge_region` | `ClientInfo.server_hostname/server_region` | expected | unavailable | unavailable | unavailable | +| `edge_pop` | new bounded adapter mapping from documented runtime metadata | optional | unavailable | optional | unavailable | + +`expected` means the implementation plan must map and test an existing source; +`optional` means the adapter includes it only when its supported SDK exposes a +stable value; `unavailable` means v1 intentionally omits it. In particular, +ASN is currently not populated by the Fastly or Cloudflare geo adapters and +must not be claimed until a concrete source is implemented. New HTTP-version or +POP mappings must be confirmed against the pinned adapter SDK before addition. + Explicitly excluded: - DNS resolver address and resolver ASN. @@ -470,6 +776,7 @@ Explicitly excluded: - DDoS/internal Fastly classifications. - Arbitrary request headers. - Full client IP in HTML or export. +- JA4, H2, or other probabilistic client fingerprints. Unsupported optional fields are omitted rather than populated with fabricated fallbacks. @@ -482,6 +789,12 @@ The report serializer constructs a new public model field by field. It never serializes request structs, cookie parsers, auction requests, telemetry rows, or browser objects wholesale. +The deserializer is an equally strict boundary. It validates the complete +outer and nested schema at runtime before any display, export, copy, or share +operation. Unknown properties, overlong strings, non-finite numbers, excessive +arrays, excessive depth, unsupported versions, and invalid timestamps reject +the report. Validation errors expose only bounded categories. + Forbidden data includes: - Raw `Cookie` and `Set-Cookie` headers. @@ -496,25 +809,52 @@ Forbidden data includes: ### 12.2 Same-origin script visibility Publisher and third-party scripts running on the publisher origin can access -`sessionStorage`. Therefore the stored model must be safe even if read by any -same-origin script. A random storage key, closed shadow root, or public endpoint -does not change this requirement. +`sessionStorage`. They can also replace it, opener-created tabs may receive a +copy, browser session restore may preserve it, and a same-origin service worker +may intercept navigation. Therefore the stored model must be safe even if read +or forged by any same-origin code. A random storage key, closed shadow root, or +public endpoint does not change this requirement. + +Every report view and exported artifact is labeled `Browser-observed, +unverified diagnostic data`. Support documentation says that it helps +troubleshoot rendering but is not proof of a server event, user identity, or +security incident. ### 12.3 Response hardening -Both the endpoint and every active diagnostic publisher response are terminally -`private, no-store`. The endpoint also sends: +The HTML shell, enable/end responses, and every active diagnostic publisher +response are terminally `private, no-store`. The fixed versioned JS/CSS assets +are the sole exception and may be publicly cached because they contain no +request or report data. HTML and JSON endpoint responses also send: -- `Content-Type: text/html; charset=utf-8` +- Path-appropriate `Content-Type`: `text/html; charset=utf-8` for the shell and + `application/json; charset=utf-8` for enable, end, and state results. - `X-Content-Type-Options: nosniff` - `Referrer-Policy: no-referrer` -- `Content-Security-Policy` restricting content to the endpoint's own static - assets and prohibiting framing -- A restrictive `Permissions-Policy` +- The Content Security Policy specified below. +- `Permissions-Policy: camera=(), microphone=(), geolocation=(), payment=(), +usb=()` + +```text +default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; +object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self' +``` -The endpoint makes no third-party requests. Dynamic JSON embedded in HTML uses -the repository's script-safe serializer and is never concatenated into -executable JavaScript. +The endpoint makes no third-party requests. Its script and stylesheet are fixed +same-origin static assets. Setup-request values are server-rendered as escaped +text nodes, and the viewer obtains the report only from browser storage. Active +publisher-page context continues to use the repository's script-safe serializer +and is never concatenated into executable JavaScript. If inline executable +assets become necessary, they require a per-response nonce or fixed build-time +hash and a corresponding CSP change. Validated report strings enter the +document through `textContent` or equivalent DOM properties, never `innerHTML`. + +The JS asset uses `application/javascript; charset=utf-8`; the CSS asset uses +`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff`, +`Cache-Control: public, max-age=31536000, immutable`, and a strong ETag derived +from their build bytes. They accept no dynamic input. `script-src 'self'` is an +origin-level CSP permission, not a path restriction; same-origin script +interference remains inside the stated trust limitation. ### 12.4 Shared templates and ESI @@ -527,20 +867,34 @@ publicly cacheable. - Disabled route: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. +- Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. -- Cookie parser failure: report `present_invalid` without the value. +- Bounded cookie inspection failure: report the contract-defined invalid or + unavailable state without a value or parser message. - Diagnostics context serialization failure: omit the context, log a bounded server error, and preserve publisher delivery. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. -- Storage unavailable, quota exceeded, or serialization oversized: remain on - the publisher page, announce the error, and offer direct download. +- Storage unavailable or quota exceeded after a valid bounded report exists: + remain on the publisher page, announce the error, and offer direct download + of that same combined `TraceReportV1`. +- Invalid projection or a report that remains oversized after deterministic + truncation: remain on the publisher page, show a bounded capture-failure + category, and do not claim that a combined trace report exists. - Missing snapshot on endpoint: show setup state, not an empty successful report. - Expired, malformed, or unknown report schema: clear it and explain that the user must reproduce again. - Clipboard or Web Share unavailable: keep JSON download available. - Export failure: retain the on-screen report and show an accessible error. +- End POST failure: report that tracing may remain active and offer an + idempotent server retry; do not undo or block the independent local-deletion + attempt. +- State verification failure or mismatch: use `unconfirmed` wording and offer + an idempotent server retry independently of local report state. +- Local deletion failure: report separately that saved browser data could not be + removed and retain the always-available deletion retry, regardless of the + server end result. Diagnostic failures must never suppress, delay, add, remove, or reorder GPT requests, auctions, targeting, or creative rendering. @@ -551,55 +905,96 @@ requests, auctions, targeting, or creative rendering. - Configuration defaults off and rejects trace-page enablement without GPT diagnostics. -- Exact route, query, method, encoded-path, and fallback behavior. -- Session cookie set/clear attributes and duplicate-directive fail-closed - behavior. +- Exact reserved-route classification, canonical-path, query, method, encoded + path, and fallback behavior. +- Exact versioned asset routes are local and contain no dynamic data; lookalike + asset paths never reach the publisher origin. +- Same-origin POST validation, cross-site/missing signal rejection, cookie + set/clear attributes, fixed 30-minute endpoint activation without request + refresh, and idempotent enable/end behavior. +- Enable/end success requires a separate state request to observe the resulting + cookie; failed and mismatched verification never displays confirmed state. +- Empty-body enforcement rejects positive/invalid lengths, transfer encoding, + the first unexpected body byte, and the two-second deadline without an + unbounded read. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, - and origin fetch. -- Cookie-health parser covers absent, valid, malformed, duplicate, non-UTF-8, - and oversized inputs without retaining values. -- Request-context serializer masks IPv4/IPv6 and omits query, raw headers, IDs, - and unsupported fields. + configured filters, ordinary event context, and origin fetch. +- Cookie-health scanner covers multiple header fields; zero, one, and duplicate + occurrences; mixed valid/invalid duplicates; non-UTF-8; malformed pairs; and + per-value and total-header limits without retaining values. +- Request-context serializer masks IPv4/IPv6; enforces every string bound; and + omits page paths, fingerprints, query, raw headers, IDs, and unsupported + fields. - Active responses remain terminally private/no-store under hostile late header overrides. - Dynamic HTML/JSON values cannot close elements or create executable script. ### 14.2 Adapter parity tests -- Fastly route registration and optional field mapping. +- Fastly early-route ordering and optional field mapping from documented + sources. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. -- Named route failures never fall through to publisher origin. -- HEAD and unsupported methods behave identically across adapters. -- Fastly fingerprint fields respect the existing fingerprint-debug gate. +- Trace-route failures never fall through to publisher origin. +- GET, HEAD, state-changing POST, and unsupported methods obey the same + lifecycle contract across adapters. +- Every adapter omits JA4/H2 and rejects control characters or overlong platform + strings. ### 14.3 JavaScript unit tests - Explicit snapshot only; no continuous `sessionStorage` writes. -- Size limit, schema validation, expiry, replacement, clearing, and storage +- Compact UTF-8 size measurement; exact outer/nested schema validation; unknown + fields; per-string/array/numeric/depth caps; hostile mutation; expiry; + future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. +- Omission counters use checked arithmetic and reject overflow. +- Trace projection replaces the nested GPT pathname with `/[redacted]`, rejects + any other stored value, emits `TraceGptDiagnosticsV1`, applies deterministic + ordering/truncation, and records exact omission counts in a worst-case 512 KiB + fixture. - Same-tab navigation occurs only after a successful write. - Viewer handles absent optional network facts and every cookie-health state. - Forbidden fields never enter storage or export fixtures. - Download filename and MIME type are deterministic. -- Copy and Web Share success, rejection, absence, and fallback behavior. +- Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and + download/copy fallback behavior. - 320-pixel layout, keyboard navigation, focus handling, and accessible status announcements. ### 14.4 Browser integration tests -- First endpoint visit sets the session and shows setup state. +- First endpoint GET is read-only and shows setup state; a user-initiated, + same-origin enable POST sets the session. +- Successful in-page activation adds no history entry, so Back can return to + the article when it was the prior same-tab page. +- Cross-site top-level GET, form POST, and fetch attempts cannot enable or end + tracing. - A real fixture reload activates diagnostics and captures multiple slots. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain distinct. - Reloading the trace page retains an unexpired same-tab report. -- A new tab cannot access the original tab's report. -- Disabling clears both cookie and report. +- Opener-cloned tabs and browser session restore never bypass validation or + application expiry; the UI does not promise tab isolation. +- Ending tracing covers successful clearing, offline POST failure, idempotent + retry, verification mismatch, successful local deletion while offline, and + local-storage deletion failure without false success messaging. - Back-forward-cache restoration is not described as a fresh traced request; the setup page tells the user to reload. - Export JSON matches the displayed versioned model. +- A storage-failure direct export matches the combined displayed model rather + than the GPT-only export. +- Hostname changes between apex, `www`, or another subdomain show the recovery + guidance rather than claiming the session followed the user. +- A fixture service worker interception is recognized as a same-origin trust + limitation, and the server endpoint remains correct when the request reaches + it. +- The delivered CSP blocks inline injection, framing, third-party connections, + and report-derived executable HTML. +- Immutable asset fixtures prove published v1 bytes never change; changed bytes + require a new URL referenced by the shell. - Inactive publisher traffic has no trace assets, storage access, listeners, or cache-policy change. @@ -620,58 +1015,84 @@ fixture: - Enable first in a controlled staging publisher configuration. - Validate response cache headers and CDN behavior before production use. - Validate with redacted fixtures before real publisher traffic. -- Log only route outcome, schema version, report-present boolean, and bounded - error category. Never log report contents or cookie/network values. +- Log only server-observable route outcome, served shell/schema version, and + bounded error category. The server cannot know whether a browser-local report + exists and must not add an upload or beacon merely to learn that fact. Never + log report contents or cookie/network values. - Roll back by disabling `trace_page_enabled`; existing `?ts_console=1` diagnostics remain independently configurable. ## 16. Acceptance criteria -1. With the feature disabled, exact trace routes return local `404` and ordinary - traffic is unchanged. -2. A mobile user can enable tracing by opening only `/_ts/trace`; no target URL, - credentials, or trace ID is required. +1. With the feature disabled, trace-route origin requests return local `404` + and ordinary traffic is unchanged. Previously cached inert versioned assets + may remain until cache eviction, but cannot activate tracing or load a shell. +2. A mobile user can enable tracing by opening only `/_ts/trace` and selecting + one prominent action; no target URL, credentials, or trace ID is required, + and a cross-site GET cannot activate tracing. 3. The setup page accurately explains that the problem must be reproduced after activation. 4. A subsequent real publisher-page reload captures redacted request context and existing TS Console evidence without altering ad behavior. -5. `View trace results` transfers one bounded snapshot in the same tab and opens - the report page without server-side storage. +5. `View trace results` transfers one bounded, runtime-validated snapshot in + the supported same-tab journey and opens the report page without server-side + storage or claims of browser-storage isolation. 6. The report separates network, cookie health, auction/render evidence, and coverage/unknowns. 7. JSON export contains the same versioned allowlisted information shown on the page. -8. No raw cookies, user IDs, full IPs, consent strings, query strings, internal - auction IDs, targeting, or creative payloads appear in HTML, browser storage, - logs, or export. +8. No raw cookies, user IDs, full IPs, consent strings, exact page paths, query + strings, fingerprints, internal auction IDs, targeting, or creative payloads + appear in trace HTML, browser storage, logs, or export. 9. Trace HTML and active publisher pages remain terminally private/no-store. 10. Missing platform fields, incomplete auction correlation, storage failure, and unavailable share APIs degrade honestly without affecting advertising. 11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader navigation. -12. Auction fields owned by #1081 are consumed through its versioned public - contract rather than duplicated in #1050. +12. Version one accepts exactly `TraceReportV1` with + `TraceGptDiagnosticsV1`, sourced only from `GptDiagnosticsExportV1`; #1081 + and #1074/#1076 are optional additive follow-ups rather than release gates. +13. Every rendered and exported report is identified as browser-observed and + unverified, and hostile storage content cannot create executable HTML or + unbounded DOM output. +14. Enable/end operations expose partial failure honestly and are safe to + retry; the UI does not claim server-observed cookie state without the + follow-up state request or claim that local data was cleared when deletion + fails. ## 17. Implementation sequencing -This design is one product flow but should be implemented in dependency order: +This design is one product flow, but its implementation is split into three +independently reviewable plans and preferably three PRs: + +1. **Reserved route and privacy foundation:** configuration, shared early-route + classification, same-origin enable/end lifecycle, bounded cookie-health + inspection, base request-context schema, projection of already populated + `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not + add speculative new platform fields in this change. +2. **Browser handoff and viewer:** integrate current + `GptDiagnosticsExportV1`, project `TraceGptDiagnosticsV1`, construct and + strictly validate `TraceReportV1`, implement the same-tab workflow, combined + direct/storage exports, mobile viewer, copy/share, expiry, clearing, and + browser/accessibility tests. +3. **Optional network enrichment and future schemas:** add HTTP-version, POP, + ASN, or other fields only from SDK-verified platform sources with explicit + bounds. Adopt #1081 or #1074/#1076 later through a separately reviewed + versioned compatibility change. + +Each plan must include its own adapter, privacy, cache, and failure tests. The +implementation must not claim completion of #1081 or the timing work as part of +#1050. Version one ships with current observed auction/render evidence and +labels unavailable fields honestly. -1. Core request-context schema, cookie-health classification, configuration, - and endpoint shell. -2. Adapter route parity and Fastly optional network enrichment. -3. TS Console request-context envelope and explicit same-tab snapshot handoff. -4. Mobile viewer, export/copy/share, expiry, and clearing. -5. Integration with the current TS Console schema. -6. Additive adoption of #1081 and #1074/#1076 fields after their contracts - merge. -7. Browser, privacy, cache, and real-device acceptance. +## 18. Rejected alternatives -The implementation plan must not claim completion of #1081 or the open timing -PRs as part of #1050. If those dependencies are unavailable, the report ships -only with current observed auction/render evidence and labels unavailable fields -honestly. +### Automatic activation on `GET /_ts/trace` -## 18. Rejected alternatives +Rejected because a cross-site top-level navigation can trigger a public GET and +`SameSite=Lax` does not make that activation intentional. A same-origin fetch +POST after one explicit button press preserves the simple mobile journey and +the useful Back history entry without requiring server-side session storage. ### `/_ts/admin/trace?target=/article` @@ -715,9 +1136,16 @@ length, history, logging, referrer, and accidental-sharing risks. ## 19. Known limitations - The user must reproduce the problem after enabling tracing. -- Same-tab storage prevents cross-device and cross-tab sharing; JSON export is - the handoff artifact. -- Publisher-origin scripts can read the stored public-safe report. +- Same-tab navigation is the supported workflow, but opener-created tabs and + browser session restore may copy or retain session storage. JSON export is + the intentional support handoff artifact. +- Publisher-origin scripts and service workers can read or forge the stored + public-safe report. A same-origin service worker can also intercept or fake + the shell, enable/end/state requests, assets, and report navigation. The + experience is diagnostic evidence, not an authenticity boundary. +- The activation cookie is host-only and session storage is origin-scoped, so + the workflow does not follow the user across apex, `www`, or other + subdomains. - Browser privacy settings may disable storage, clipboard, download, or share capabilities. - Current server/browser correlation does not cover every no-bid, skipped, From 534a691e2e64852c28728be80fb90e44167c22ce Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 1 Sep 2026 16:04:45 +0530 Subject: [PATCH 032/230] Add server auction evidence to mobile trace design --- ...-mobile-ad-render-trace-endpoint-design.md | 726 +++++++++++++++--- 1 file changed, 616 insertions(+), 110 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 6ed9f8c48..092466a6d 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -22,12 +22,12 @@ intentionally enables or ends tracing with a same-origin POST action. The endpoint is both a setup page and a report viewer. On the first visit it offers a large `Enable tracing` action and explains how to reproduce the problem. The user then returns to the real publisher page and reloads it. -Trusted Server supplies redacted request context, while the existing TS Console -records GPT, auction, and render evidence. A `View trace results` action creates -one bounded, allowlisted snapshot in same-tab `sessionStorage` and navigates to -`/_ts/trace`. The endpoint reads that untrusted snapshot, validates it, presents -a mobile-first HTML report, and offers JSON export, copy, and progressive Web -Share actions. +Trusted Server supplies redacted request context and a minimal live summary of +each server auction, while the existing TS Console records GPT and render +evidence. A `View trace results` action creates one bounded, allowlisted +snapshot in same-tab `sessionStorage` and navigates to `/_ts/trace`. The +endpoint reads that untrusted snapshot, validates it, presents a mobile-first +HTML report, and offers JSON export, copy, and progressive Web Share actions. The design introduces no report database, server-side trace store, report ID, target URL parameter, telemetry query, or publisher-origin change. It cannot @@ -57,7 +57,10 @@ experience: - TS Console owns observation of the real publisher page. The browser-local handoff joins them without introducing a backend report -service. +service. The report keeps three evidence layers separate: what the server +auction observed, what GPT observed in the page, and what the creative bridge +observed during rendering. Correlation is displayed only when an ephemeral +diagnostic token connects those layers. ## 3. Goals @@ -68,6 +71,9 @@ service. - Display a Fastly-inspired network summary for the traced publisher request. - Report health for an explicit allowlist of Trusted Server cookies without exposing their values. +- Report each observed server auction as initial-navigation SSAT, SPA page-bids, + or the Trusted Server `/auction` API, including bounded auction-local timing, + provider-call outcomes, and per-slot candidate outcomes. - Present the versioned, allowlisted TS Console evidence for every retained GPT slot and request cycle that fits the public report bounds, with explicit omission counts when deterministic size truncation is required. @@ -97,10 +103,15 @@ service. cache URLs, or stack traces. - Reimplementing TS Console auction and creative observability requested by #1081. +- Naming bidders, exposing winning price, assigning final creative numbers, or + asserting which demand path ultimately won when the available evidence does + not prove it; those remain #1081 concerns. - Querying Tinybird to build an interactive report. - Adding exact provider-by-slot no-bid explanations before the auction model can observe those dispositions. -- Direct `POST /auction` browser diagnostics in the first release. +- Introspection into third-party client-side auction internals. Version one can + report that a publisher/Prebid refresh was observed, but cannot identify its + participants or winner unless Trusted Server itself handled that auction. ## 5. Decisions @@ -137,6 +148,13 @@ lifetime bounds accidental private/no-store operation if a user forgets to end tracing; the existing technical query flow keeps its existing session-cookie semantics. +The new trace capture is active only when both `trace_page_enabled = true` and +the request carries exactly one valid diagnostics cookie. The shared cookie by +itself continues to activate the existing TS Console but does not authorize +trace tokens, server-auction projection, trace response extensions, or +correlation sidecars on a deployment whose trace page is disabled. This makes +the configuration flag the disclosure and rollback boundary. + ### 5.3 Browser-local, explicit handoff TS Console remains memory-only during observation. It writes a report to @@ -177,15 +195,25 @@ inconsistent behavior. ### 5.5 Separate issue ownership #1050 defines the report shell, request context, mobile flow, browser-local -handoff, and export. #1081 remains the owner of creative numbering, auction -classification, bidder/price policy, terminology, and normalized auction/render -timing. - -Version one consumes `GptDiagnosticsExportV1` through TS Console's public export -contract and projects it into a distinct redacted `TraceGptDiagnosticsV1`. It -does not read TS Console internals or create an alternate slot correlation -engine. #1081 and #1074/#1076 are additive follow-up work and are not release -gates for this version. +handoff, export, and the minimum server-auction facts needed to answer whether +SSAT or the Trusted Server auction API ran. Core produces a new redacted +`TraceAuctionEvidenceV1`; TS Console consumes it as immutable evidence rather +than reconstructing server behavior from GPT callbacks. + +#1081 remains the owner of bidder/price disclosure policy, creative numbering, +final user-facing terminology, and any richer cross-demand winner +classification. #1074/#1076 remain the owners of request-phase and +request-relative auction milestones. Version one may ship without those +follow-ups because it exposes already-available auction-local total and provider +durations and labels each clock explicitly. + +Version one also consumes `GptDiagnosticsExportV1` through TS Console's public +export contract and projects it into a distinct redacted +`TraceGptDiagnosticsV1`. It does not read TS Console internals or create a +second GPT attribution engine. The existing recorder emits an exact-token +`TraceSlotCorrelationV1` sidecar when it binds an opportunity to a request +cycle. The server-auction, correlation, and GPT projections are separate sibling +contracts in `TraceReportV1`; none is treated as a substitute for another. ## 6. User experience @@ -234,18 +262,43 @@ When a valid snapshot exists, `/_ts/trace` renders: 1. Report summary and capture time. 2. Network and request section for the traced publisher document. 3. Trusted Server cookie-health section. -4. Auction and rendering section grouped by numbered slot. -5. Coverage and ambiguity section. -6. Export actions. -7. `Clear report and end tracing` action. +4. Server-auction section grouped by auction and numbered slot. +5. GPT delivery and creative-rendering section grouped by numbered slot. +6. Coverage and ambiguity section. +7. Export actions. +8. `Clear report and end tracing` action. The setup request's facts are not merged into or substituted for missing traced page facts. Missing fields display `Unavailable`; missing evidence displays `Not observed` or `Unknown`, following TS Console terminology. -The report begins with `Browser-observed, unverified diagnostic data`. It does -not claim that the snapshot is authentic or suitable as forensic or security -evidence. +The viewer presents an evidence chain rather than one overloaded status: + +```text +Server auction -> GPT request/response -> creative render/load/viewability +``` + +For each step it shows the source, observed outcome, and correlation state. +`Client-side refresh observed` describes browser intent only; it must never be +rendered as `client-side auction won`. Likewise, a filled GPT slot does not +prove that the Trusted Server candidate rendered. + +Internal enums are exported for machines, but the page uses plain labels: + +| Evidence enum | Mobile label | +| ------------------------------------------ | ------------------------------------------------- | +| `initial_navigation_ssat` | `Initial-page server auction (SSAT)` | +| `spa_page_bids` | `Trusted Server page-refresh auction` | +| `auction_api` | `Trusted Server auction API` | +| GPT `prebid_refresh`/`publisher_refresh` | `Browser refresh observed; winner not determined` | +| GPT `competing`/`unattributed` | `Multiple or unknown delivery paths` | +| matched non-empty creative-bridge evidence | `Trusted Server creative rendered` | + +The report begins with `Browser-carried, unverified diagnostic data`. Server +auction entries are labeled `Produced by Trusted Server; copied through an +untrusted browser snapshot`, while GPT and creative entries are labeled +`Browser observed`. The viewer does not claim that the stored snapshot is +authentic or suitable as forensic or security evidence. `Copy` copies formatted JSON. `Share` supplies the same JSON file to the native Web Share sheet only after an explicit tap and tells the user that the selected @@ -286,7 +339,8 @@ Real publisher document reload |-- adapter supplies optional network facts |-- core computes allowlisted cookie health |-- core injects redacted TraceRequestContextV1 - |-- existing TS Console observes GPT and TS delivery + |-- core observes live server auctions and emits TraceAuctionEvidenceV1 + |-- existing TS Console observes GPT and creative delivery v User selects "View trace results" | @@ -307,6 +361,10 @@ Report GET /_ts/trace - Provide a shared exact-path reserved-route classifier that runs before event context, filters, auctions, named routes, or publisher fallback. - Define the platform-neutral request-context and report-envelope schemas. +- Build `TraceAuctionEvidenceV1` directly from the live auction observation and + orchestration result; never query telemetry or serialize telemetry rows. +- Mint and thread public diagnostic auction/slot tokens independently of + internal request IDs, including zero-bid and failed auctions. - Build cookie-health facts through read-only parsing. - Convert `ClientInfo` and available geo data into the public network allowlist. - Inject request context only into an active private diagnostics document. @@ -327,9 +385,14 @@ Report GET /_ts/trace ### 7.3 JavaScript responsibilities - Accept the immutable redacted request context at initialization. +- Accept immutable server-auction evidence delivered through the supported + initial-navigation, page-bids, and `/auction` transports. +- Emit the bounded `TraceSlotCorrelationV1` sidecar at the existing GPT + recorder's opportunity-to-cycle binding point. - Preserve the existing bounded TS Console observation store. -- Build and validate `TraceReportV1` with a redacted - `TraceGptDiagnosticsV1` projection on explicit user action. +- Build and validate `TraceReportV1` with redacted + `TraceAuctionEvidenceV1` and `TraceGptDiagnosticsV1` projections on explicit + user action. - Store only one supported report for the same-tab workflow in `sessionStorage`, while treating its contents as untrusted. - Render the report shell from the validated model. @@ -356,8 +419,9 @@ Rules: the allowlisted presence/validity of four HttpOnly Trusted Server cookies visible to same-origin JavaScript whenever the feature is enabled. It also states that masked IP prefixes and coarse geo remain potentially personal or - pseudonymous network data. Enabling the option is the deployment's explicit - acceptance of those bounded disclosures. + pseudonymous network data, and that opaque server-auction outcomes become + visible to same-origin JavaScript. Enabling the option is the deployment's + explicit acceptance of those bounded disclosures. - `GET /_ts/trace` returns the setup/report shell without changing cookies or browser storage at the HTTP layer. After load, the explicitly included viewer script may remove a rejected or expired local entry. Unrelated query @@ -571,16 +635,28 @@ TraceReportV1 schema_version: 1 captured_at: RFC 3339 UTC timestamp request_context: TraceRequestContextV1 + server_auctions: TraceAuctionEvidenceV1[] + slot_correlations: TraceSlotCorrelationV1[] gpt_diagnostics: TraceGptDiagnosticsV1 + auction_coverage: + capture_status: complete | partial | unavailable | not_observed + issues: + evidence_projection_failed | evidence_transport_failed + | evidence_validation_failed | record_evicted + | correlation_unavailable | external_client_side_unobservable truncation: + omitted_server_auctions: u16 + omitted_slot_correlations: u16 omitted_request_cycles: u16 omitted_callback_issues: u16 omitted_attribution_issues: u16 omitted_nested_values: u16 ``` -`TraceGptDiagnosticsV1` is a trace-owned projection sourced only from -`GptDiagnosticsExportV1`. It contains: +`TraceAuctionEvidenceV1` is a server-produced public model defined in section +9.4. `TraceSlotCorrelationV1` is the browser-produced exact-token sidecar +defined in section 9.4.1. `TraceGptDiagnosticsV1` is a separate trace-owned +projection sourced only from `GptDiagnosticsExportV1`. It contains: - `schema_version: 1` and `source_schema_version: 1`; - the source `capturedAt` value; @@ -593,11 +669,24 @@ It is deliberately not named or represented as `GptDiagnosticsExportV1`, because the fixed pathname and trace-level bounds change the source field semantics. TS Console continues to own the source schema; the trace envelope owns its public projection and transport. The initial compatibility matrix is -exactly `TraceReportV1` plus `TraceGptDiagnosticsV1`, sourced from -`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, trace-auction, -or source version with an actionable message rather than guessing. A future TS -Console successor requires an additive source compatibility change and, if the -public projection changes, a new trace-envelope version. +exactly `TraceReportV1`, `TraceAuctionEvidenceV1`, `TraceSlotCorrelationV1`, and +`TraceGptDiagnosticsV1`, with the GPT projection sourced from +`GptDiagnosticsExportV1`. The viewer rejects every unknown outer, +server-auction, correlation, GPT-projection, or GPT-source version with an +actionable message rather than guessing. A future TS Console successor requires +an additive source compatibility change and, if the public projection changes, +a new trace-envelope version. + +`auction_coverage.capture_status` describes only what reached the browser +collector: `not_observed` means no valid server-auction record arrived, not that +no server auction ran. `partial` requires at least one retained record plus a +projection, transport, validation, or eviction issue; `unavailable` requires no +retained records plus a known projection, transport, or validation issue; and +`complete` requires at least one retained record without those capture issues. +`correlation_unavailable` and `external_client_side_unobservable` describe +interpretation limits and do not change an otherwise complete capture status. +The issue array is deduplicated, sorted in enum order, bounded to 16 values, and +contains no error text. Origin validation requires a parseable HTTP(S) origin whose canonical serialization exactly equals `window.location.origin`; credentials, paths, @@ -606,7 +695,268 @@ strict RFC 3339 UTC strings and must be within 60 seconds of `stored_at_ms`. `TraceRequestContextV1.captured_at` requires strict RFC 3339 UTC but may be older because it represents the publisher document request rather than snapshot time. -### 9.4 Storage limits and expiry +### 9.4 Server-auction evidence + +Core creates one `TraceAuctionEvidenceV1` at the live auction boundary. It is +not derived from the browser, reconstructed from winning-bid targeting, or +loaded from auction telemetry: + +```text +TraceAuctionEvidenceV1 + schema_version: 1 + diagnostic_auction_id: string + source: + initial_navigation_ssat | spa_page_bids | auction_api + terminal_status: + completed | execution_failed | dispatch_failed | abandoned | skipped + terminal_reason?: + policy_skipped | no_eligible_slots | no_provider_launched + | provider_execution_failed | collection_failed | unknown + total_time_ms?: u32 + provider_calls: + - provider_number: u16 + role: bidder | mediator | unknown + status: success | no_bid | error | pending | abandoned | unknown + response_time_ms?: u32 + returned_bid_count: u16 + slots: + - slot_number: u16 + slot_ref: string + requested_sizes: [u32, u32][] + returned_bid_count: u16 + candidate: + selected | no_candidate | selected_unrenderable | unknown + selected_creative_size?: [u32, u32] + truncation: + omitted_provider_calls: u16 + omitted_slots: u16 + omitted_nested_values: u16 + coverage: + provider_to_slot_no_bid: unavailable +``` + +The model has deliberately lower cardinality and sensitivity than the existing +telemetry and OpenRTB objects: + +- `diagnostic_auction_id` is a fresh opaque `ts-auc-...` correlation token. When + trace capture is active under the two-part gate in section 5.2, it is minted + once when an eligible auction is observed, before dispatch, and is retained + for zero-bid, skipped, dispatch-failed, execution-failed, and abandoned + outcomes. It is never `AuctionRequest.id`, the telemetry UUID, a provider + request ID, or an identifier joinable to user-bearing logs. +- Auction and slot tokens are the fixed prefixes `ts-auc-` and `ts-slot-` + followed by a canonical lowercase hyphenated UUID v4. Validators reject every + other shape; tokens are not silently shortened or normalized. +- `slot_number` is a one-based ordinal over the exact post-conversion + `AuctionRequest.slots` sequence observed by orchestration. It is display-only + and is never used to map a response back to pre-conversion client input. + `slot_ref` is a fresh auction-local opaque token carried with that slot. Core + creates it for initial-navigation and SPA auctions. For a TSJS `/auction` + request, TSJS creates it only after `buildAdRequest` has finished grouping and + deduplicating the final `adUnits` array, attaches it to that exact outgoing + unit as `adUnits[].ext.trusted_server.trace_slot_ref`, and retains the + request-scoped token-to-unit mapping. Core accepts that member only under the + two-part trace gate, validates and echoes the token for accepted converted + slots, and strips it before every provider or mediator request. A missing or + invalid client token causes core to mint a server token with no browser + correlation; it never changes ordinary auction acceptance. TSJS uses + `crypto.randomUUID()` and, if unavailable, omits the client token rather than + using weak randomness. Raw publisher slot IDs, ad-unit paths, and internal + impression IDs are not copied into this model. Neither the ordinal nor token + is a creative number. +- `source` is assigned by the server call site: initial document auction is + `initial_navigation_ssat`, `/_ts/page-bids` is `spa_page_bids`, and + `POST /auction` is `auction_api`. Browser `requestPath` does not determine or + override this value. +- `provider_number` is assigned deterministically in provider dispatch order + and is stable only within one auction. Provider names, bidder/seat names, and + provider metadata are omitted. `returned_bid_count` is a count, not a bid + payload. +- `terminal_reason` is mapped to the allowlisted category at the observation + boundary. Raw error messages, parser errors, URLs, and provider text never + enter the model. +- Slot candidate state is computed from the requested slots, returned bids, + winner selection, and final response-conversion disposition. A winner that + cannot safely enter the bid map/OpenRTB response is + `selected_unrenderable`, not `selected` and not `no_candidate`. +- Provider calls are auction-wide. The model does not claim that a provider was + called, timed out, or returned no bid for a particular slot. Only returned + bids can contribute to a slot's `returned_bid_count`; the fixed coverage + value makes the missing provider-to-slot no-bid relation explicit. +- `total_time_ms` and `response_time_ms` use the server's auction-local monotonic + durations. They are not request-relative milestones and are never + arithmetically combined with browser timestamps. +- Core applies the provider, slot, size, string, and numeric limits before the + model crosses into HTML or JSON. It retains request/dispatch order and records + every discarded nested entry in the auction-local `truncation` object. An + omission-counter overflow rejects that auction evidence rather than wrapping + or saturating. A duration that cannot convert to its optional public integer + type is omitted and counted; a required count or ordinal conversion failure + rejects that auction evidence. Values are never clamped to a plausible value. + +The server evidence and GPT evidence retain separate meanings: + +| Question | Authoritative v1 source | +| --------------------------------------------------- | ------------------------------------------------------------- | +| Did initial-navigation SSAT run? | `server_auctions[].source = initial_navigation_ssat` | +| Did the Trusted Server auction API run? | `server_auctions[].source = auction_api` | +| Was a publisher/Prebid refresh observed? | GPT `requestPath`, labeled as browser intent | +| Did a server auction select a slot candidate? | matching server-auction slot `candidate` | +| Did GPT request, fill, and render the slot? | `TraceGptDiagnosticsV1` request-cycle evidence | +| Did the Trusted Server creative bridge participate? | `TraceGptDiagnosticsV1` creative-delivery evidence | +| Which path ultimately won? | only when existing correlation proves it; otherwise `Unknown` | + +Absence is not converted into a negative assertion. If transport or +correlation failed, the viewer shows `Server auction evidence unavailable` or +`Correlation unknown`, not `SSAT did not run`. + +#### 9.4.1 Live transport and correlation + +Evidence is transported only while `trace_page_enabled` is true and the +diagnostics cookie is valid. Every response carrying it is terminally +`private, no-store`: + +```text +TraceAuctionTransportV1 + schema_version: 1 + evidence?: TraceAuctionEvidenceV1 + unavailable_reason?: evidence_projection_failed +``` + +Exactly one of `evidence` and `unavailable_reason` is present. This small +transport envelope lets core report a safe projection failure without exposing +the raw error. A malformed envelope is rejected as a whole. A network failure +before an envelope arrives is recorded separately by the browser as +`evidence_transport_failed`. + +Initial-navigation and SPA slot definitions carry their token on the exact slot +object TSJS already consumes: + +```text +AuctionSlot.ext.trusted_server.trace_slot_ref: string +``` + +Core adds that optional nested member only under the two-part trace gate. It +assigns the token while constructing the request-scoped slot definitions and +threads the same token into the corresponding `AuctionRequest` observation, so +neither side needs to recover the relationship from an ordinal or raw slot ID. +The ordinary `AuctionSlot.id`, `gam_unit_path`, `div_id`, formats, targeting, +ordering, and bid-map keys remain unchanged. The extension is absent when the +gate is false and is never copied into `TraceAuctionEvidenceV1` except as its +already-allowlisted opaque `slot_ref`. + +TSJS accepts a slot extension only when its canonical token occurs exactly once +in both the delivered slot list and the matching auction evidence. A missing, +malformed, duplicate, or conflicting token prevents only that sidecar join, +adds `evidence_validation_failed` and `correlation_unavailable`, and does not +drop, reorder, or mutate the ordinary slot or bid. TSJS reads no other extension +property. This validation occurs before the slot is handed to the existing GPT +initialization path. + +The three transport call shapes are exact v1 contracts: + +```text +Initial seam: + scheduleInitialAdInit(bids, slots?, traceAuctionTransport?) + +SPA JSON: + { slots, bids, trace_auction?: TraceAuctionTransportV1 } + +/auction request unit: + adUnits[].ext.trusted_server.trace_slot_ref?: string + +/auction OpenRTB response: + ext.trusted_server.trace_auction?: TraceAuctionTransportV1 +``` + +The initial scheduler validates/records the optional third argument before it +runs `adInit`; cached older bundles may ignore the extra argument without +affecting ads, in which case evidence remains `not_observed`. The SPA parser +accepts only the exact optional top-level member and preserves its existing +`slots` and `bids` behavior. These trace members never become required for a +successful advertising response. + +When the existing GPT recorder consumes the matching Trusted Server opportunity +for a concrete request cycle, it emits this trace-owned sidecar: + +```text +TraceSlotCorrelationV1 + schema_version: 1 + diagnostic_auction_id: string + slot_ref: string + runtime_slot_number: safe positive integer + request_number: safe positive integer +``` + +The recorder already decides which pending opportunity belongs to which GPT +slot/request cycle. The sidecar records that exact decision; it does not rerun +attribution or read a private store during export. It is emitted only when both +opaque server tokens and the concrete GPT cycle are present. The current +`GptDiagnosticsExportV1` remains unchanged and the sidecar contains no slot +element ID or ad-unit path. + +- **Initial-navigation SSAT:** core builds the evidence when the split auction + is collected at the held body tail. It injects the script-safe public model + beside the winning-bid map before initial ad initialization; the corresponding + request-scoped slot definitions already carry + `ext.trusted_server.trace_slot_ref`. Failed, abandoned, skipped, and zero-bid + outcomes still inject their bounded evidence when the publisher document can + be delivered. +- **SPA page-bids:** `/_ts/page-bids` adds an optional, namespaced + `trace_auction` transport envelope beside its existing bid result. TSJS + validates and records it and consumes each returned slot's + `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both + the envelope and slot extensions are absent when the trace gate is inactive. +- **Trusted Server `/auction` API:** the existing OpenRTB response adds a + namespaced `ext.trusted_server.trace_auction` transport envelope only for an + active diagnostics request. After producing the final grouped `AdRequest`, + both TSJS callers assign one fresh token to each outgoing unit and retain that + exact request-scoped mapping. They validate the echoed evidence and record it + before parsing bids. A converted or skipped unit therefore cannot shift + another slot's correlation. The response member does not replace or expose + the existing orchestrator extension, and the trace projection must not copy + that extension's provider names, bidder names, metadata, price, creative IDs, + domains, or markup. HTTP/transport failures with no readable response are + browser-observed failures only; no successful server evidence is + manufactured. + +The direct TSJS caller records `evidence_transport_failed` from its existing +non-OK, unreadable-JSON, and rejected-`fetch` paths. The Prebid adapter +creates one bounded pending transport record after `buildRequests`, keyed by the +request's original bid IDs and its normalized unit tokens. `interpretResponse` +consumes it on a readable response; the pinned Prebid `onTimeout` and +`onBidderError` bidder-spec hooks consume it and record +`evidence_transport_failed` otherwise. Repeated hooks are idempotent. Pending +records are capped at 128 and expire after the configured bid timeout plus five +seconds. Expiry without any supported success/error/timeout hook proves no +transport outcome, so it removes the marker and leaves evidence `not_observed` +rather than inventing a failure. These hooks collect only bounded categories and +opaque tokens, never XHR error text or response bodies. + +The diagnostic auction token is also attached to the existing GPT opportunity +marker, and the opaque slot token is carried through the corresponding +winning-bid/slot initialization path. The numeric ordinal is never a +correlation key. The viewer joins a server slot to a GPT cycle only when one validated +`TraceSlotCorrelationV1` exactly matches both tokens and the exported +runtime/request numbers. It displays unmatched, duplicate, or conflicting +records independently, preserves competing paths, and never joins by +timestamps, implicit array position, ad-unit path, or a best-effort heuristic. + +TSJS retains at most the newest 16 validated server-auction records and 128 +correlation sidecars in memory. It increments checked eviction counters for +older records; the snapshot adds those counts to the matching truncation fields +and emits `record_evicted` with `partial`. It performs no storage write until +the explicit snapshot action. +Requests for which either side of the trace-capture gate is false do not mint +trace tokens, build trace evidence, add response members, emit sidecars, or +install auction-evidence listeners. + +No database, server-side report store, Tinybird query, beacon, or follow-up +network request is required. Evidence already available at the live auction +boundary is projected and carried forward in the response that the browser is +already receiving. + +### 9.5 Storage limits and expiry - Storage key: a namespaced, versioned constant owned by the diagnostics module. @@ -624,9 +974,10 @@ because it represents the publisher document request rather than snapshot time. - One supported report for the current browsing context; a new explicit snapshot replaces the old report. Browser opener cloning and session restore may copy or retain it. -- The runtime validator accepts only the exact outer and trace-auction v1 - schemas, rejects unknown fields, applies the limits below, and checks compact - UTF-8 size before rendering. +- The runtime validator accepts only the exact outer, server-auction, + slot-correlation, auction-coverage, and GPT-projection v1 schemas, rejects + unknown fields, applies the limits below, and checks compact UTF-8 size before + rendering. - Invalid, oversized, expired, unsupported, or hostile reports are removed when possible and otherwise ignored. Rendering uses DOM properties and `textContent`, never report-derived HTML. @@ -645,6 +996,11 @@ Runtime limits are part of the v1 contract: | Value | Limit | | --------------------------------------------- | ------------------------------------------------------------ | | Container nesting | 8 levels | +| Server auctions | 16 | +| Slot correlations | 128 | +| Provider calls | 16 per server auction | +| Auction slots | 64 per server auction | +| Auction coverage issues | 16 | | Slots | 64 | | Request cycles | 10 per slot before total-size truncation | | Callback issues | 128 | @@ -656,6 +1012,7 @@ Runtime limits are part of the v1 contract: | GPT pathname in trace projection | Exact literal `/[redacted]` | | Slot element ID and ad-unit path | 512 UTF-8 bytes each | | Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | +| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | | Any other string | 128 UTF-8 bytes | | Enum | Exact documented value only | | Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | @@ -668,15 +1025,24 @@ characters or bidirectional override/isolate controls. This applies to browser source fields as well as platform fields and precedes rendering or export. The snapshot builder creates a new field-by-field projection and rejects an -invalid source value rather than stringifying it. It retains only the first -documented number of requested sizes, yield-group IDs, company IDs, and creative -failure enums, recording discarded entries in `omitted_nested_values`; strings -are never silently shortened. It then measures the complete compact UTF-8 -storage wrapper. If it exceeds 512 KiB, it removes the globally oldest request -cycles first while retaining the newest cycle for each slot, then the oldest -callback issues, then the oldest attribution issues, and finally the oldest -remaining request cycles until the report fits. It records every removal in -`truncation`. +invalid source value rather than stringifying it. Server auctions are already +bounded by core; the browser rejects an invalid inner model rather than +truncating it. The builder retains the newest 16 server auctions in observation +order, the newest 128 correlations in recorder emission order, and only the +first documented number of GPT requested sizes, yield-group IDs, company IDs, +and creative failure enums. It records each discard in +`omitted_server_auctions`, `omitted_slot_correlations`, or +`omitted_nested_values`; strings are never silently shortened. It then measures +the complete compact UTF-8 storage wrapper. If it exceeds 512 KiB, it removes +the globally oldest GPT +request cycles first while retaining the newest cycle for each GPT slot, then +the oldest callback issues, then the oldest attribution issues, then the oldest +uncorrelated server auctions, and finally the oldest remaining GPT cycles and +server auctions until the report fits. It records every removal in +`truncation`. A correlated auction and the newest GPT cycle that references it +are retained or removed together once the algorithm reaches correlated server +auctions; every sidecar referencing a removed auction or cycle is removed and +counted. The report must not retain a dangling token while claiming a join. A report that still cannot fit after this bounded procedure fails snapshot creation. The implementation must include a worst-case fixture proving the result is bounded. @@ -691,48 +1057,61 @@ primitives do not. No accepted report value may enter a ninth container level. The storage wrapper is validated separately as the exact two-field object `{ stored_at_ms, report }`. -For deterministic ordering, a request cycle with no `requestedAtMs` sorts -before a cycle with a timestamp; otherwise cycles sort by `requestedAtMs`, then -`runtimeSlotNumber`, then `requestNumber`. Callback and attribution issues sort -by `timestampMs`, then their original array index. The builder preserves the -relative order of all retained records. +For deterministic ordering, retained server auctions remain in observation +order, provider calls and auction slots retain their server-assigned numeric +order, and correlations retain recorder emission order. A request cycle with no +`requestedAtMs` sorts before a cycle with a timestamp; otherwise cycles sort by +`requestedAtMs`, then `runtimeSlotNumber`, then `requestNumber`. Callback and +attribution issues sort by `timestampMs`, then their original array index. The +builder preserves the relative order of all retained records. ## 10. Auction and rendering evidence -The report uses TS Console's evidence model. It must preserve the distinction -between: +The report combines, but never conflates, the server model from section 9.4 and +TS Console's browser model. It must preserve the distinction between: -- A Trusted Server opportunity. -- A provider response. -- A selected Trusted Server candidate. +- A server-observed auction and its source. +- An auction-wide provider call and response. +- A server-selected candidate for one opaque slot reference. +- A browser-observed Trusted Server opportunity. - A GPT request and response. - A non-empty GPT render. - Trusted Server creative-bridge evidence. - Creative load and viewability. - A publisher or client-side refresh. -The viewer must not infer that Trusted Server rendered an ad merely because GPT -reported a filled slot. Ambiguous and unattributed cycles remain explicit. - -Current diagnostics tokens exist only on delivered winning bids. No-bid, -failed, skipped, hidden, unresolved, and direct `/auction` paths can lack server -correlation. The report displays the available observed facts and `Unknown` -rather than manufacturing a correlation. +The diagnostic auction token is created before dispatch rather than only on a +delivered winner, so zero-bid and terminal failure states have an identity when +a response can carry evidence. The viewer still must not infer that Trusted +Server rendered an ad merely because the server selected a candidate or GPT +reported a filled slot. Ambiguous, competing, unmatched, and unattributed +cycles remain explicit. Provider-call telemetry is auction-wide, while bid rows exist only for returned bids. Exact `provider X was asked for slot Y` and exact no-bid causality require a future provider-impression disposition model. That instrumentation is not silently assumed by this design. -Timing fields introduced by #1074/#1076 are outside the v1 compatibility -matrix. They may be consumed in a later version only after they merge and are -propagated through the public live-diagnostics contract. The report never -queries Tinybird, and it does not combine browser `performance.now()` values -with server-relative timing as though they were one clock. +The UI groups timing into three labeled clocks: + +1. **Server auction-local:** v1 `total_time_ms` and provider + `response_time_ms`, measured from the live orchestration result. +2. **Request-relative server milestones:** dispatched, resolved, and committed + milestones from #1076, unavailable in this schema and adoptable later. +3. **Browser/GPT:** TS Console request, response, render, load, and viewability + timings. -Bidder and winning price are not added by version one. A later version may -consume them only if #1081 approves them in the public TS Console export -contract. #1050 does not independently weaken the existing privacy policy. +The report never queries Tinybird and never subtracts or combines values from +different clocks. #1074/#1076 may add request-relative fields only through a +separately reviewed compatibility change. + +Bidder identity, provider identity, winning price, currency, creative numbering, +and a final `SSAT/TS/client-side winner` label are not added by version one. A +later version may consume them only if #1081 approves their meaning and public +disclosure policy. #1050 does not independently weaken the existing privacy +policy. Version one can nevertheless answer the narrower, evidence-based +questions: which Trusted Server entry point ran, what bounded server outcome it +reported, what GPT did afterward, and where correlation is missing. ## 11. Network scope @@ -798,13 +1177,15 @@ the report. Validation errors expose only bounded categories. Forbidden data includes: - Raw `Cookie` and `Set-Cookie` headers. -- EC IDs, EIDs, bidder user IDs, and consent strings. +- EC IDs, EIDs, bidder user IDs, provider/bidder/seat names, and consent + strings. - Unmasked client IP. - Query strings and fragments. - Fastly or internal request identifiers that can join to user-bearing logs. - Internal `AuctionRequest.id`. -- Bid requests/responses, losing-bid payloads, targeting, creative markup, - cache URLs, and stack traces. +- Bid requests/responses, bid prices/currency, losing-bid payloads, provider + metadata, targeting, creative IDs/domains/markup, cache URLs, and stack + traces. ### 12.2 Same-origin script visibility @@ -815,15 +1196,17 @@ may intercept navigation. Therefore the stored model must be safe even if read or forged by any same-origin code. A random storage key, closed shadow root, or public endpoint does not change this requirement. -Every report view and exported artifact is labeled `Browser-observed, -unverified diagnostic data`. Support documentation says that it helps -troubleshoot rendering but is not proof of a server event, user identity, or -security incident. +Every report view and exported artifact is labeled `Browser-carried, +unverified diagnostic data`. Individual server entries retain their +server-produced provenance, but support documentation says the browser-carried +copy helps troubleshoot rendering and is not cryptographic proof of a server +event, user identity, or security incident. ### 12.3 Response hardening -The HTML shell, enable/end responses, and every active diagnostic publisher -response are terminally `private, no-store`. The fixed versioned JS/CSS assets +The HTML shell, enable/end responses, every active diagnostic publisher +response, and every dynamic page-bids or `/auction` response carrying trace +evidence are terminally `private, no-store`. The fixed versioned JS/CSS assets are the sole exception and may be publicly cached because they contain no request or report data. HTML and JSON endpoint responses also send: @@ -859,6 +1242,9 @@ interference remains inside the stated trust limitation. ### 12.4 Shared templates and ESI Per-request trace context must never enter a shared template or ESI fragment. +This includes diagnostic auction/slot tokens and the optional `AuctionSlot` +extension; active responses add them only in request-scoped injection or the +request-scoped body seam. The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content publicly cacheable. @@ -873,6 +1259,15 @@ publicly cacheable. unavailable state without a value or parser message. - Diagnostics context serialization failure: omit the context, log a bounded server error, and preserve publisher delivery. +- Server-auction evidence construction or serialization failure: omit only the + affected evidence, retain a bounded `unavailable` coverage marker when safe, + log no report content, and preserve the normal auction/result path. +- Initial-navigation evidence cannot be injected because the body tail is not + reached: preserve publisher delivery. Because the browser received no safe + marker, display `not_observed` rather than claiming a known server failure. +- Page-bids or `/auction` evidence is absent or rejected by its strict client + validator: parse the ordinary bid response exactly as before, discard the + diagnostic member, and show an unmatched/invalid-evidence coverage category. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. - Storage unavailable or quota exceeded after a valid bounded report exists: @@ -896,8 +1291,10 @@ publicly cacheable. removed and retain the always-available deletion retry, regardless of the server end result. -Diagnostic failures must never suppress, delay, add, remove, or reorder GPT -requests, auctions, targeting, or creative rendering. +Diagnostic failures must never suppress, delay, add, remove, reorder, or change +the success status of GPT requests, auctions, bid responses, targeting, or +creative rendering. The evidence projection is a side effect of already-known +results, never a prerequisite for returning them. ## 14. Testing strategy @@ -905,6 +1302,10 @@ requests, auctions, targeting, or creative rendering. - Configuration defaults off and rejects trace-page enablement without GPT diagnostics. +- With GPT diagnostics enabled but `trace_page_enabled = false`, a valid + console cookie still enables the existing console but never mints trace + tokens, builds auction evidence, adds response extensions, or emits + correlation sidecars. - Exact reserved-route classification, canonical-path, query, method, encoded path, and fallback behavior. - Exact versioned asset routes are local and contain no dynamic data; lookalike @@ -925,6 +1326,38 @@ requests, auctions, targeting, or creative rendering. - Request-context serializer masks IPv4/IPv6; enforces every string bound; and omits page paths, fingerprints, query, raw headers, IDs, and unsupported fields. +- Server-auction projection maps initial navigation, SPA page-bids, and auction + API call sites to the exact public source enums without using a browser hint. +- The diagnostic auction token is minted before dispatch and remains identical + across completed, zero-bid, skipped, failed, and abandoned evidence and the + corresponding browser opportunity marker. It never equals or contains the + internal auction ID or telemetry UUID. +- Server-auction projection covers every terminal status/reason mapping, + auction-local total duration, provider role/status/duration/count, per-slot + requested sizes/bid count/candidate disposition, and checked numeric + conversion. +- Provider numbering and opaque slot references are deterministic and bounded; + provider names, bidder/seat names, prices, currency, publisher slot IDs, + creative identifiers/domains/markup, metadata, raw errors, and internal IDs + are absent from serialized fixtures. +- Auction-wide provider no-bid evidence is never projected as a per-slot + disposition, and `provider_to_slot_no_bid` remains `unavailable`. +- Evidence projection/serialization failure leaves the ordinary bid map or + OpenRTB response unchanged apart from the bounded unavailable transport + envelope. +- `/auction` request tokens survive the exact AdRequest-to-AuctionRequest slot + conversion, are stripped before provider dispatch, and remain correctly + associated across grouped multi-bidder units, duplicate codes, skipped + non-banner units, and mixed accepted/filtered inputs. Numeric ordinals are + never used for client correlation. +- Token tests cover canonical UUID-v4 shape, missing Web Crypto, malformed or + duplicate request extensions, the disabled trace gate, and proof that invalid + tokens neither fail nor otherwise alter the ordinary auction. +- Initial and SPA slot JSON attaches the exact + `ext.trusted_server.trace_slot_ref` token that appears in server evidence; + inactive responses omit it. Missing, duplicate, conflicting, malformed, and + evidence-mismatched slot tokens suppress only correlation and produce the + specified coverage issues without changing slot/bid order or contents. - Active responses remain terminally private/no-store under hostile late header overrides. - Dynamic HTML/JSON values cannot close elements or create executable script. @@ -949,6 +1382,31 @@ requests, auctions, targeting, or creative rendering. future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. - Omission counters use checked arithmetic and reject overflow. +- Strict validation of every server-auction enum, token, numeric bound, array + bound, nesting level, and unknown property; invalid evidence is discarded + without changing ordinary bid parsing. +- Transport envelopes require exactly one of evidence/unavailable reason; + projection, transport, validation, eviction, correlation, and external-client + coverage states produce the specified complete/partial/unavailable/not-observed + result without treating absence as proof that no auction ran. +- Direct fetch failures and Prebid `interpretResponse`, `onTimeout`, and + `onBidderError` paths consume their pending transport record exactly once; + capped/expired records and absent hooks follow the specified `not_observed` + behavior without retaining error text or bodies. +- Exact-token correlation joins matching server auctions, GPT opportunities, + and slot references; unmatched, duplicated, conflicting, missing, and + forged tokens stay separate and produce explicit coverage states. No + timestamp, index, or ad-unit-path heuristic is used. +- `TraceSlotCorrelationV1` is emitted only at the existing recorder's exact + opportunity-to-cycle binding, is capped and evicted deterministically, + contains only opaque tokens plus runtime/request numbers, and does not alter + `GptDiagnosticsExportV1`. +- Source presentation distinguishes server-owned SSAT/page-bids/auction API + facts from browser-observed publisher refresh, Prebid refresh, competing, and + unattributed request paths. No fixture turns intent or a GPT fill into a + winner assertion. +- Server auction-local, request-relative unavailable, and browser/GPT timings + render in separate labeled groups and are never combined arithmetically. - Trace projection replaces the nested GPT pathname with `/[redacted]`, rejects any other stored value, emits `TraceGptDiagnosticsV1`, applies deterministic ordering/truncation, and records exact omission counts in a worst-case 512 KiB @@ -971,8 +1429,20 @@ requests, auctions, targeting, or creative rendering. - Cross-site top-level GET, form POST, and fetch attempts cannot enable or end tracing. - A real fixture reload activates diagnostics and captures multiple slots. +- Initial-navigation SSAT fixtures cover selected, no-candidate, + selected-unrenderable, skipped, dispatch-failed, execution-failed, and + abandoned outcomes without requiring a winning bid. +- SPA page-bids and both TSJS `/auction` callers consume the optional + `trace_auction` member before ad initialization/bid parsing; inactive + responses have no member, and malformed members do not affect bids. +- Initial seams pass the optional transport as the scheduler's third argument, + SPA JSON uses the exact optional top-level member, and old-scheduler/absent + transport fixtures preserve ad initialization while reporting `not_observed`. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. +- A correlated fixture renders the chain `server auction -> GPT -> creative`, + while unmatched server, client-side refresh, competing, and transport-failure + fixtures show honest independent evidence and `Unknown` where appropriate. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain distinct. - Reloading the trace page retains an unexpired same-tab report. @@ -996,7 +1466,10 @@ requests, auctions, targeting, or creative rendering. - Immutable asset fixtures prove published v1 bytes never change; changed bytes require a new URL referenced by the shell. - Inactive publisher traffic has no trace assets, storage access, listeners, or - cache-policy change. + cache-policy change, diagnostic token generation, or trace-auction response + extension. +- Disabling `trace_page_enabled` removes every new capture behavior even when a + technical `?ts_console=1` session leaves a valid diagnostics cookie present. ### 14.5 Manual acceptance @@ -1007,6 +1480,9 @@ fixture: - Touch targets, scrolling, zoom, safe areas, download, copy, and native share behavior are usable. - The user can distinguish setup information from captured-page information. +- The user can distinguish `SSAT/Trusted Server auction ran`, `browser refresh +observed`, `GPT filled/rendered`, and `Unknown` without understanding internal + request-path names. - A failed share or download does not lose the visible report. ## 15. Rollout and observability @@ -1032,58 +1508,74 @@ fixture: and a cross-site GET cannot activate tracing. 3. The setup page accurately explains that the problem must be reproduced after activation. -4. A subsequent real publisher-page reload captures redacted request context - and existing TS Console evidence without altering ad behavior. +4. A subsequent real publisher-page reload captures redacted request context, + live server-auction evidence, and existing TS Console evidence without + altering ad behavior. 5. `View trace results` transfers one bounded, runtime-validated snapshot in the supported same-tab journey and opens the report page without server-side storage or claims of browser-storage isolation. -6. The report separates network, cookie health, auction/render evidence, and - coverage/unknowns. +6. The report separates network, cookie health, server auction, GPT delivery, + creative rendering, and coverage/unknowns. 7. JSON export contains the same versioned allowlisted information shown on the page. 8. No raw cookies, user IDs, full IPs, consent strings, exact page paths, query - strings, fingerprints, internal auction IDs, targeting, or creative payloads - appear in trace HTML, browser storage, logs, or export. + strings, fingerprints, internal auction IDs, provider/bidder/seat names, + prices, targeting, or creative identifiers/payloads appear in trace HTML, + browser storage, trace-specific logs, or export. 9. Trace HTML and active publisher pages remain terminally private/no-store. -10. Missing platform fields, incomplete auction correlation, storage failure, - and unavailable share APIs degrade honestly without affecting advertising. +10. Missing platform fields, failed evidence projection/transport, incomplete + auction correlation, storage failure, and unavailable share APIs degrade + honestly without affecting advertising. 11. The full report is usable at 320 CSS pixels and with keyboard/screen-reader navigation. -12. Version one accepts exactly `TraceReportV1` with - `TraceGptDiagnosticsV1`, sourced only from `GptDiagnosticsExportV1`; #1081 - and #1074/#1076 are optional additive follow-ups rather than release gates. -13. Every rendered and exported report is identified as browser-observed and - unverified, and hostile storage content cannot create executable HTML or +12. Version one accepts exactly `TraceReportV1` with server-produced + `TraceAuctionEvidenceV1`, browser-produced `TraceSlotCorrelationV1`, and + `TraceGptDiagnosticsV1`; only the latter is sourced from + `GptDiagnosticsExportV1`. #1081 and #1074/#1076 are optional additive + follow-ups rather than release gates. +13. Every rendered and exported report is identified as browser-carried and + unverified; server-produced and browser-observed entries retain distinct + provenance, and hostile storage content cannot create executable HTML or unbounded DOM output. 14. Enable/end operations expose partial failure honestly and are safe to retry; the UI does not claim server-observed cookie state without the follow-up state request or claim that local data was cleared when deletion fails. +15. A report can show that initial-navigation SSAT, SPA page-bids, or the + Trusted Server auction API ran; show its bounded provider and per-slot + outcome; show subsequent GPT/creative evidence; and display `Unknown` + rather than inventing a client-side winner or an unsupported correlation. ## 17. Implementation sequencing -This design is one product flow, but its implementation is split into three -independently reviewable plans and preferably three PRs: +This design is one product flow, but its implementation is split into four +independently reviewable plans and preferably four PRs: 1. **Reserved route and privacy foundation:** configuration, shared early-route classification, same-origin enable/end lifecycle, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not add speculative new platform fields in this change. -2. **Browser handoff and viewer:** integrate current +2. **Live server-auction evidence:** introduce the public diagnostic auction and + slot tokens, project `TraceAuctionEvidenceV1` at the live observation + boundary, transport it through initial navigation, page-bids, and both + TSJS `/auction` callers, emit `TraceSlotCorrelationV1` from the existing GPT + recorder binding, and prove ordinary bid behavior is unchanged on + absent/invalid evidence. +3. **Browser handoff and viewer:** integrate current `GptDiagnosticsExportV1`, project `TraceGptDiagnosticsV1`, construct and strictly validate `TraceReportV1`, implement the same-tab workflow, combined - direct/storage exports, mobile viewer, copy/share, expiry, clearing, and - browser/accessibility tests. -3. **Optional network enrichment and future schemas:** add HTTP-version, POP, + server/GPT/creative correlation, direct/storage exports, mobile viewer, + copy/share, expiry, clearing, and browser/accessibility tests. +4. **Optional network enrichment and future schemas:** add HTTP-version, POP, ASN, or other fields only from SDK-verified platform sources with explicit bounds. Adopt #1081 or #1074/#1076 later through a separately reviewed versioned compatibility change. Each plan must include its own adapter, privacy, cache, and failure tests. The -implementation must not claim completion of #1081 or the timing work as part of -#1050. Version one ships with current observed auction/render evidence and -labels unavailable fields honestly. +implementation must not claim completion of #1081 or request-relative timing as +part of #1050. Version one ships with minimal live server-auction evidence plus +current GPT/render evidence and labels unavailable fields honestly. ## 18. Rejected alternatives @@ -1148,8 +1640,22 @@ length, history, logging, referrer, and accidental-sharing risks. subdomains. - Browser privacy settings may disable storage, clipboard, download, or share capabilities. -- Current server/browser correlation does not cover every no-bid, skipped, - failed, hidden, unresolved, or direct-auction path. +- A server can report an auction failure only when it can still deliver a + response containing evidence. Network termination or an unreadable `/auction` + response remains a browser-observed transport failure with no server outcome. +- Provider calls are auction-wide. Version one cannot attribute a provider + no-bid, timeout, or error to a specific slot. +- Third-party client-side auction participants, bids, and winners are not + observable. Publisher/Prebid refresh is browser intent, not proof that a + client-side bidder won. +- `/auction` evidence requires the TSJS request to reach the same Trusted Server + host with the active diagnostics cookie. A custom cross-origin auction + endpoint does not inherit this trace session and is shown as unavailable. +- Exact-token correlation can remain unavailable for hidden, unresolved, + competing, or independently initiated GPT cycles. The report preserves both + sides instead of guessing. +- Version one has auction-local server durations and browser/GPT timings, but + request-relative dispatched/resolved/committed milestones await #1076. - Fastly-only transport details do not exist on every adapter. - The current 30-pixel TS Console controls are not sufficient for this mobile report; the endpoint uses independent 44-pixel touch targets. From dc34278c8552d2782b22ab23b4bdae09bdea19e9 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 20 Aug 2026 10:26:06 -0500 Subject: [PATCH 033/230] Report a reason when an APS creative render fails An APS bid that wins Prebid targeting is served by Ad Manager as a 1x1 universal creative that resizes itself only after the creative draws. Every guard on that render path returned silently, so a slot that never drew was indistinguishable from one that did: Ad Manager reports a non-empty 1x1 render either way, and the tester framework reports "filled". Name the guard that stopped the render instead. The sandboxed renderer document now reports bad_hash, source_mismatch, nonce_mismatch, descriptor_keys, descriptor_fields, descriptor_envelope, and amazon_script_error on the existing failure message. Reporting is one-shot and answers through the parent, never the sender, so an unrelated sender cannot consume the frame's single report or learn anything from it. Traffic that is not shaped like the render handshake stays silent as before. The Universal Creative source labels its own frame_timeout and frame_load_error, and relays whichever reason it holds to the top window. That relay crosses an origin boundary, so reasons resolve through a null-prototype allowlist that drops anything unlisted and leaves a hostile __proto__ or constructor as undefined. Reasons are fixed categories. A descriptor is never echoed back. --- .../src/integrations/aps.rs | 89 ++++++++++---- .../trusted-server-js/lib/src/core/types.ts | 29 ++++- .../lib/src/integrations/aps/render.ts | 67 ++++++++++- .../lib/test/integrations/aps/render.test.ts | 109 ++++++++++++++++++ 4 files changed, 266 insertions(+), 28 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/aps.rs b/crates/trusted-server-core/src/integrations/aps.rs index 581e5c200..23f092b48 100644 --- a/crates/trusted-server-core/src/integrations/aps.rs +++ b/crates/trusted-server-core/src/integrations/aps.rs @@ -57,42 +57,54 @@ const APS_RENDERER_DOCUMENT: &str = r#" var match=/^#tsaps=([A-Za-z0-9_-]{22,128})$/.exec(location.hash); var expected=match&&match[1]; try{history.replaceState(null,'',location.pathname+location.search);}catch(_error){} -if(!expected)return; +var reported=false; +function report(reason,nonce){ + if(reported)return; + reported=true; + try{parent.postMessage({message:'trusted-server/aps/renderer-failed',nonce:nonce,reason:reason},'*');}catch(_error){} +} +if(!expected){report('bad_hash');return;} function keys(value,expectedKeys){ if(!value||typeof value!=='object'||Array.isArray(value))return false; var actual=Object.keys(value).sort(); return actual.length===expectedKeys.length&&actual.every(function(key,index){return key===expectedKeys[index];}); } -function validRenderer(renderer){ +function rendererProblem(renderer){ if(!keys(renderer,['aaxResponse','accountId','bidId','creativeId','creativeUrl','height','tagType','type','version','width'])&& - !keys(renderer,['aaxResponse','accountId','bidId','creativeUrl','height','tagType','type','version','width']))return false; - if(renderer.type!=='aps'||renderer.version!==1||typeof renderer.accountId!=='string'||!renderer.accountId||new TextEncoder().encode(renderer.accountId).length>1024)return false; - if(typeof renderer.bidId!=='string'||!renderer.bidId||!Number.isInteger(renderer.width)||renderer.width<=0||!Number.isInteger(renderer.height)||renderer.height<=0)return false; - if(Object.prototype.hasOwnProperty.call(renderer,'creativeId')&&(typeof renderer.creativeId!=='string'||!renderer.creativeId||new TextEncoder().encode(renderer.creativeId).length>1024))return false; - if(renderer.tagType!=='iframe'&&renderer.tagType!=='script')return false; - if(typeof renderer.creativeUrl!=='string'||new TextEncoder().encode(renderer.creativeUrl).length>4096)return false; - if(typeof renderer.aaxResponse!=='string'||!renderer.aaxResponse||renderer.aaxResponse.length>349528)return false; + !keys(renderer,['aaxResponse','accountId','bidId','creativeUrl','height','tagType','type','version','width']))return 'descriptor_keys'; + if(renderer.type!=='aps'||renderer.version!==1||typeof renderer.accountId!=='string'||!renderer.accountId||new TextEncoder().encode(renderer.accountId).length>1024)return 'descriptor_fields'; + if(typeof renderer.bidId!=='string'||!renderer.bidId||!Number.isInteger(renderer.width)||renderer.width<=0||!Number.isInteger(renderer.height)||renderer.height<=0)return 'descriptor_fields'; + if(Object.prototype.hasOwnProperty.call(renderer,'creativeId')&&(typeof renderer.creativeId!=='string'||!renderer.creativeId||new TextEncoder().encode(renderer.creativeId).length>1024))return 'descriptor_fields'; + if(renderer.tagType!=='iframe'&&renderer.tagType!=='script')return 'descriptor_fields'; + if(typeof renderer.creativeUrl!=='string'||new TextEncoder().encode(renderer.creativeUrl).length>4096)return 'descriptor_fields'; + if(typeof renderer.aaxResponse!=='string'||!renderer.aaxResponse||renderer.aaxResponse.length>349528)return 'descriptor_fields'; try{ var url=new URL(renderer.creativeUrl); - if(url.protocol!=='https:'||url.username||url.password)return false; + if(url.protocol!=='https:'||url.username||url.password)return 'descriptor_envelope'; var binary=atob(renderer.aaxResponse); - if(binary.length>262144||btoa(binary)!==renderer.aaxResponse)return false; + if(binary.length>262144||btoa(binary)!==renderer.aaxResponse)return 'descriptor_envelope'; var bytes=Uint8Array.from(binary,function(character){return character.charCodeAt(0);}); var decoded=JSON.parse(new TextDecoder('utf-8',{fatal:true}).decode(bytes)); - if(!keys(decoded,['seatbid'])||!Array.isArray(decoded.seatbid)||decoded.seatbid.length!==1)return false; + if(!keys(decoded,['seatbid'])||!Array.isArray(decoded.seatbid)||decoded.seatbid.length!==1)return 'descriptor_envelope'; var seat=decoded.seatbid[0]; - if(!keys(seat,['bid'])||!Array.isArray(seat.bid)||seat.bid.length!==1)return false; + if(!keys(seat,['bid'])||!Array.isArray(seat.bid)||seat.bid.length!==1)return 'descriptor_envelope'; var bid=seat.bid[0]; - if(!keys(bid,['ext','h','id','price','w'])||!keys(bid.ext,['creativeurl','tagtype']))return false; - return bid.id===renderer.bidId&&bid.w===renderer.width&&bid.h===renderer.height&& + if(!keys(bid,['ext','h','id','price','w'])||!keys(bid.ext,['creativeurl','tagtype']))return 'descriptor_envelope'; + if(bid.id===renderer.bidId&&bid.w===renderer.width&&bid.h===renderer.height&& bid.ext.creativeurl===renderer.creativeUrl&&bid.ext.tagtype===renderer.tagType&& - typeof bid.price==='number'&&Number.isFinite(bid.price)&&bid.price>=0; - }catch(_error){return false;} + typeof bid.price==='number'&&Number.isFinite(bid.price)&&bid.price>=0)return undefined; + return 'descriptor_envelope'; + }catch(_error){return 'descriptor_envelope';} } function receive(event){ - if(event.source!==parent)return; var message=event.data; - if(!keys(message,['nonce','renderer'])||message.nonce!==expected||!validRenderer(message.renderer))return; + // Stay silent for traffic that is not shaped like the render handshake, so an + // unrelated sender cannot consume this frame's single report. + if(!keys(message,['nonce','renderer']))return; + if(event.source!==parent){report('source_mismatch');return;} + if(message.nonce!==expected){report('nonce_mismatch');return;} + var problem=rendererProblem(message.renderer); + if(problem){report(problem,message.nonce);return;} removeEventListener('message',receive); var acceptedNonce=expected; expected=''; @@ -107,7 +119,7 @@ function receive(event){ var script=document.createElement('script'); script.src='https://client.aps.amazon-adsystem.com/prebid-creative.js'; script.onload=function(){parent.postMessage({message:'trusted-server/aps/renderer-ready',nonce:acceptedNonce},'*');}; - script.onerror=function(){parent.postMessage({message:'trusted-server/aps/renderer-failed',nonce:acceptedNonce},'*');}; + script.onerror=function(){report('amazon_script_error',acceptedNonce);}; document.head.appendChild(script); } addEventListener('message',receive); @@ -2613,4 +2625,41 @@ mod tests { assert!(APS_RENDERER_CSP.contains("sandbox allow-forms")); assert!(!APS_RENDERER_CSP.contains("allow-same-origin")); } + + #[test] + fn renderer_document_reports_a_reason_for_every_silent_guard() { + for reason in [ + "bad_hash", + "source_mismatch", + "nonce_mismatch", + "descriptor_keys", + "descriptor_fields", + "descriptor_envelope", + "amazon_script_error", + ] { + assert!( + APS_RENDERER_DOCUMENT.contains(reason), + "renderer document should report a `{reason}` reason instead of returning silently" + ); + } + + // Reasons travel on the existing failure message rather than a new channel. + assert!( + APS_RENDERER_DOCUMENT.contains("reason:reason"), + "should attach the reason to the failure message" + ); + + // A reason is a fixed category, never a copy of the rejected descriptor. + assert!(!APS_RENDERER_DOCUMENT.contains("JSON.stringify(renderer)")); + assert!(!APS_RENDERER_DOCUMENT.contains("reason:message")); + + // Reporting is one-shot so a hostile sender cannot flood the parent. + assert!( + APS_RENDERER_DOCUMENT.contains("if(reported)return"), + "should report at most one reason per frame" + ); + + // A foreign sender is answered through the parent, never the sender. + assert!(!APS_RENDERER_DOCUMENT.contains("event.source.postMessage")); + } } diff --git a/crates/trusted-server-js/lib/src/core/types.ts b/crates/trusted-server-js/lib/src/core/types.ts index 03ff0aca2..05af3fff6 100644 --- a/crates/trusted-server-js/lib/src/core/types.ts +++ b/crates/trusted-server-js/lib/src/core/types.ts @@ -181,12 +181,37 @@ export type GptDiagnosticsTrustedServerOpportunity = | 'unrenderable_candidate' | 'no_candidate'; -/** A safe failure category observed while obtaining or posting creative markup. */ +/** + * A safe failure category observed while obtaining or posting creative markup. + * + * The `aps_` members cover the APS Universal Creative render path, where a + * blank slot is otherwise indistinguishable from a filled one: Ad Manager + * reports a non-empty 1x1 render whether or not the creative ever drew. Each + * member names the exact guard that stopped the render. + */ export type GptDiagnosticsCreativeFailure = | 'missing_render_source' | 'cache_fetch_failed' | 'invalid_cache_payload' - | 'response_post_failed'; + | 'response_post_failed' + // Reported by the sandboxed renderer document and relayed by the creative. + | 'aps_bad_hash' + | 'aps_nonce_mismatch' + | 'aps_source_mismatch' + | 'aps_descriptor_keys' + | 'aps_descriptor_fields' + | 'aps_descriptor_envelope' + | 'aps_runner_script_error' + // Observed by the Universal Creative source around its renderer frame. + | 'aps_frame_timeout' + | 'aps_frame_load_error' + | 'aps_frame_reported_failure' + | 'aps_unknown' + // Observed on the Trusted Server side of the capability handshake. + | 'aps_consumed_tombstone' + | 'aps_source_not_in_ad_unit' + | 'aps_missing_renderer_url' + | 'aps_tombstone_capacity'; /** Delivery evidence derived for a GPT request cycle. */ export type GptDiagnosticsDelivery = diff --git a/crates/trusted-server-js/lib/src/integrations/aps/render.ts b/crates/trusted-server-js/lib/src/integrations/aps/render.ts index adec0b036..4d20d23a7 100644 --- a/crates/trusted-server-js/lib/src/integrations/aps/render.ts +++ b/crates/trusted-server-js/lib/src/integrations/aps/render.ts @@ -1,6 +1,11 @@ import { log } from '../../core/log'; import { findSlot } from '../../core/render'; -import type { ApsPrebidRendererEntry, ApsRendererV1, TsjsApi } from '../../core/types'; +import type { + ApsPrebidRendererEntry, + ApsRendererV1, + GptDiagnosticsCreativeFailure, + TsjsApi, +} from '../../core/types'; export const APS_RENDERER_PATH = '/integrations/aps/renderer'; export const APS_RENDERING_MODE_ATTRIBUTE_NAME = 'data-ts-aps-rendering-mode'; @@ -32,6 +37,55 @@ const activeFrames = new WeakMap(); const pendingFrameCancels = new WeakMap void>(); const RENDERER_READY_MESSAGE = 'trusted-server/aps/renderer-ready'; const RENDERER_FAILED_MESSAGE = 'trusted-server/aps/renderer-failed'; +/** + * Message the Universal Creative frame relays to the top window when an APS + * render never completes. + * + * The creative frame is cross-origin, so the top-window listener treats every + * field as untrusted and validates the reason against + * [`APS_RENDER_FAILURE_REASONS`] before recording it. The relay is + * diagnostics-only and never influences creative delivery. + */ +export const APS_RENDER_FAILED_MESSAGE = 'trusted-server/aps/render-failed'; + +/** + * Wire reasons the render path can emit, mapped onto safe diagnostic categories. + * + * Built on a null prototype so a hostile `__proto__`, `constructor`, or + * `toString` relayed by the cross-origin creative frame resolves to `undefined` + * rather than an inherited member. + */ +const APS_RENDER_FAILURE_REASONS: Readonly> = + Object.freeze( + Object.assign(Object.create(null) as Record, { + bad_hash: 'aps_bad_hash', + nonce_mismatch: 'aps_nonce_mismatch', + source_mismatch: 'aps_source_mismatch', + descriptor_keys: 'aps_descriptor_keys', + descriptor_fields: 'aps_descriptor_fields', + descriptor_envelope: 'aps_descriptor_envelope', + amazon_script_error: 'aps_runner_script_error', + frame_timeout: 'aps_frame_timeout', + frame_load_error: 'aps_frame_load_error', + frame_reported_failure: 'aps_frame_reported_failure', + unknown: 'aps_unknown', + } as const) + ); + +/** + * Resolve a relayed render failure reason to a safe diagnostic category. + * + * Returns `undefined` for anything not on the allowlist, so an unrecognized or + * hostile value from the cross-origin creative frame is dropped instead of + * being recorded. + * + * @example + * apsRenderFailureReason('frame_timeout'); // 'aps_frame_timeout' + * apsRenderFailureReason('__proto__'); // undefined + */ +export function apsRenderFailureReason(value: unknown): GptDiagnosticsCreativeFailure | undefined { + return typeof value === 'string' ? APS_RENDER_FAILURE_REASONS[value] : undefined; +} const RENDERER_READY_TIMEOUT_MS = 10_000; const MAX_PREBID_RENDERER_ENTRIES = 256; const DEFAULT_PREBID_RENDERER_TTL_SECONDS = 300; @@ -711,12 +765,13 @@ var b=new Uint8Array(16);c.getRandomValues(b);var s="";for(var i=0;i { }); }); +describe('APS render failure reason allowlist', () => { + it('maps every reason the renderer frame and creative source can emit', () => { + expect(apsRenderFailureReason('descriptor_envelope')).toBe('aps_descriptor_envelope'); + expect(apsRenderFailureReason('source_mismatch')).toBe('aps_source_mismatch'); + expect(apsRenderFailureReason('bad_hash')).toBe('aps_bad_hash'); + expect(apsRenderFailureReason('frame_timeout')).toBe('aps_frame_timeout'); + expect(apsRenderFailureReason('amazon_script_error')).toBe('aps_runner_script_error'); + }); + + it('rejects unlisted, inherited, and non-string reasons from the cross-origin frame', () => { + expect(apsRenderFailureReason('not_a_real_reason')).toBeUndefined(); + expect(apsRenderFailureReason('__proto__')).toBeUndefined(); + expect(apsRenderFailureReason('constructor')).toBeUndefined(); + expect(apsRenderFailureReason('toString')).toBeUndefined(); + expect(apsRenderFailureReason(42)).toBeUndefined(); + expect(apsRenderFailureReason(undefined)).toBeUndefined(); + expect(apsRenderFailureReason({ toString: () => 'frame_timeout' })).toBeUndefined(); + }); +}); + describe('Universal Creative APS source', () => { it('uses the deployed dynamic renderer protocol and only creates the opaque route frame', () => { expect(APS_UNIVERSAL_CREATIVE_RENDERER_VERSION).toBeGreaterThanOrEqual(4); @@ -803,4 +825,91 @@ describe('Universal Creative APS source', () => { document.body.innerHTML = ''; } }); + + it('relays the frame failure reason to the top window for diagnostics', async () => { + const dynamicWindow = window as unknown as { + render?: (data: Record, helper: unknown, target: Window) => Promise; + }; + const relayed: unknown[] = []; + const capture = (event: MessageEvent): void => { + const data = event.data as { message?: unknown } | undefined; + if (data && data.message === APS_RENDER_FAILED_MESSAGE) relayed.push(data); + }; + window.addEventListener('message', capture); + window.eval(APS_UNIVERSAL_CREATIVE_RENDERER); + + try { + const rendered = dynamicWindow.render!( + { adId: 'example-ad-id', apsRenderer: descriptor(), rendererUrl: apsRendererUrl() }, + undefined, + window + ); + const iframe = document.body.querySelector('iframe')!; + const postMessage = vi.spyOn(iframe.contentWindow!, 'postMessage'); + iframe.dispatchEvent(new Event('load')); + const sent = postMessage.mock.calls[0][0] as { nonce: string }; + + window.dispatchEvent( + new MessageEvent('message', { + data: { + message: 'trusted-server/aps/renderer-failed', + nonce: sent.nonce, + reason: 'descriptor_envelope', + }, + source: iframe.contentWindow, + }) + ); + + await expect(rendered).rejects.toThrow(); + // `postMessage` is delivered on a later task than the rejection microtask. + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(relayed).toEqual([ + { + message: APS_RENDER_FAILED_MESSAGE, + adId: 'example-ad-id', + reason: 'descriptor_envelope', + }, + ]); + } finally { + window.removeEventListener('message', capture); + delete dynamicWindow.render; + document.body.innerHTML = ''; + } + }); + + it('reports a frame timeout when the renderer frame never acknowledges', async () => { + const dynamicWindow = window as unknown as { + render?: (data: Record, helper: unknown, target: Window) => Promise; + }; + const relayed: unknown[] = []; + const capture = (event: MessageEvent): void => { + const data = event.data as { message?: unknown } | undefined; + if (data && data.message === APS_RENDER_FAILED_MESSAGE) relayed.push(data); + }; + window.addEventListener('message', capture); + vi.useFakeTimers(); + window.eval(APS_UNIVERSAL_CREATIVE_RENDERER); + + try { + const rendered = dynamicWindow.render!( + { adId: 'timeout-ad-id', apsRenderer: descriptor(), rendererUrl: apsRendererUrl() }, + undefined, + window + ); + rendered.catch(() => {}); + document.body.querySelector('iframe')!.dispatchEvent(new Event('load')); + + // Async advance so the queued `postMessage` dispatch task also runs. + await vi.advanceTimersByTimeAsync(10_001); + + expect(relayed).toEqual([ + { message: APS_RENDER_FAILED_MESSAGE, adId: 'timeout-ad-id', reason: 'frame_timeout' }, + ]); + } finally { + vi.useRealTimers(); + window.removeEventListener('message', capture); + delete dynamicWindow.render; + document.body.innerHTML = ''; + } + }); }); From fae9e357ce88cf7de8140f3fc442482f41807103 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 20 Aug 2026 10:32:12 -0500 Subject: [PATCH 034/230] Attribute APS creative delivery in GPT diagnostics The APS capability handshake never told diagnostics anything, so every request cycle on that path reported `delivery: unknown` and no creative failures at all. On a live page that meant 24 of 24 cycles were unattributed while APS bids were winning and rendering blank, which is the state that made this hard to diagnose from the outside. Record the attempt around the handshake. The path runs on the publisher's own Prebid ad units, which never pass through Trusted Server slot mapping, so no creative opportunity exists for them and the store would reject the attempt as `creative_request_without_slot`. Resolve the GPT slot by element ID and record the opportunity first. Each silent return that ends in a blank now names itself: aps_consumed_tombstone, aps_source_not_in_ad_unit, aps_descriptor_fields, aps_tombstone_capacity, and aps_missing_renderer_url. A successful post records a response. Consumed ad IDs carry the attempt they were served under, so a replay, or a failure the creative frame relays after the fact, is attributed to the render it belongs to rather than guessed at. The relay listener treats the creative as untrusted: the reason must resolve through the allowlist, the attempt comes from our own tombstone rather than the message, and it never answers the sender. --- .../lib/src/integrations/aps/render.ts | 29 ++--- .../lib/src/integrations/gpt/index.ts | 97 +++++++++++++++-- .../lib/test/integrations/gpt/ad_init.test.ts | 102 ++++++++++++++++++ 3 files changed, 205 insertions(+), 23 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/aps/render.ts b/crates/trusted-server-js/lib/src/integrations/aps/render.ts index 4d20d23a7..cfe93502c 100644 --- a/crates/trusted-server-js/lib/src/integrations/aps/render.ts +++ b/crates/trusted-server-js/lib/src/integrations/aps/render.ts @@ -57,19 +57,22 @@ export const APS_RENDER_FAILED_MESSAGE = 'trusted-server/aps/render-failed'; */ const APS_RENDER_FAILURE_REASONS: Readonly> = Object.freeze( - Object.assign(Object.create(null) as Record, { - bad_hash: 'aps_bad_hash', - nonce_mismatch: 'aps_nonce_mismatch', - source_mismatch: 'aps_source_mismatch', - descriptor_keys: 'aps_descriptor_keys', - descriptor_fields: 'aps_descriptor_fields', - descriptor_envelope: 'aps_descriptor_envelope', - amazon_script_error: 'aps_runner_script_error', - frame_timeout: 'aps_frame_timeout', - frame_load_error: 'aps_frame_load_error', - frame_reported_failure: 'aps_frame_reported_failure', - unknown: 'aps_unknown', - } as const) + Object.assign( + Object.create(null) as Record, + { + bad_hash: 'aps_bad_hash', + nonce_mismatch: 'aps_nonce_mismatch', + source_mismatch: 'aps_source_mismatch', + descriptor_keys: 'aps_descriptor_keys', + descriptor_fields: 'aps_descriptor_fields', + descriptor_envelope: 'aps_descriptor_envelope', + amazon_script_error: 'aps_runner_script_error', + frame_timeout: 'aps_frame_timeout', + frame_load_error: 'aps_frame_load_error', + frame_reported_failure: 'aps_frame_reported_failure', + unknown: 'aps_unknown', + } as const + ) ); /** diff --git a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts index 89b480c6f..cf49d654d 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts @@ -8,8 +8,10 @@ import type { TsjsApi, } from '../../core/types'; import { + APS_RENDER_FAILED_MESSAGE, APS_UNIVERSAL_CREATIVE_RENDERER, APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, + apsRenderFailureReason, apsRendererUrl, dispatchApsRendering, consumeApsPrebidRenderer, @@ -1597,11 +1599,49 @@ function safelyRecordCreativeFailure( } } +/** + * Open a diagnostics attempt for an APS capability handshake. + * + * The APS path runs on the publisher's own Prebid ad units, which never pass + * through Trusted Server slot mapping, so no creative opportunity has been + * recorded for them. Without one the store rejects the attempt as + * `creative_request_without_slot` and the request cycle stays `unknown`, + * leaving a blank APS render indistinguishable from a delivered one. + */ +function beginApsCreativeAttempt(adUnitCode: string): number | undefined { + try { + const pubads = window.googletag?.pubads?.(); + const slot = pubads ? findGptSlotByElementId(pubads, adUnitCode) : undefined; + if (slot) { + window.tsjs?.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( + slot, + adUnitCode, + 'renderable_candidate' + ); + } + } catch { + // Diagnostics must not alter creative delivery. + } + return safelyRecordCreativeRequest(adUnitCode); +} + +/** + * A consumed APS ad ID, retained as a security tombstone. + * + * `attemptId` carries the diagnostics attempt the capability was served under so + * a later replay, or a failure relayed by the creative frame, is attributed to + * the render it belongs to. + */ +interface ApsConsumedTombstone { + expiresAt: number; + attemptId?: number; +} + /** Maximum number of consumed APS Prebid IDs retained as security tombstones. */ const MAX_CONSUMED_PREBID_APS_IDS = 256; function pruneConsumedPrebidApsIds( - consumedIds: Map, + consumedIds: Map, now: number ): void { for (const [adId, consumed] of consumedIds) { @@ -1610,7 +1650,7 @@ function pruneConsumedPrebidApsIds( } function hasConsumedPrebidApsIdCapacity( - consumedIds: Map, + consumedIds: Map, adId: string ): boolean { if (consumedIds.has(adId) || consumedIds.size < MAX_CONSUMED_PREBID_APS_IDS) return true; @@ -1620,11 +1660,12 @@ function hasConsumedPrebidApsIdCapacity( } function recordConsumedPrebidApsId( - consumedIds: Map, + consumedIds: Map, adId: string, - expiresAt: number + expiresAt: number, + attemptId: number | undefined ): void { - consumedIds.set(adId, { expiresAt }); + consumedIds.set(adId, { expiresAt, attemptId }); } /** @@ -1658,7 +1699,7 @@ export function installTsRenderBridge(): void { // is scoped to the slot, not the bare adId: hb_adid is not unique per bid, so // keying on it alone would let one slot block a distinct slot's render. const renderingKeys = new Set(); - const consumedPrebidApsIds = new Map(); + const consumedPrebidApsIds = new Map(); // One consumed APS ad ID per slot is sufficient: a newer bid replaces the // slot's old ad ID in `window.tsjs.bids`, so the ownership guard rejects it. const consumedServerApsBySlot = new Map(); @@ -1674,6 +1715,20 @@ export function installTsRenderBridge(): void { return; } + // Diagnostics relayed by the APS Universal Creative frame. The creative is + // cross-origin, so every field is untrusted: the reason must resolve through + // the allowlist and the attempt comes from our own tombstone, never the + // message. Recording only, and it never answers the sender. + if (data['message'] === APS_RENDER_FAILED_MESSAGE) { + const failedAdId = data['adId']; + const reason = apsRenderFailureReason(data['reason']); + if (typeof failedAdId === 'string' && reason !== undefined) { + pruneConsumedPrebidApsIds(consumedPrebidApsIds, Date.now()); + safelyRecordCreativeFailure(consumedPrebidApsIds.get(failedAdId)?.attemptId, reason); + } + return; + } + if (data['message'] !== 'Prebid Request') return; const adId = data['adId'] as string | undefined; if (!adId) return; @@ -1689,6 +1744,7 @@ export function installTsRenderBridge(): void { // other iframe. Letting Prebid's global handler answer a foreign source // would expose the creative despite the slot-bound capability check. e.stopImmediatePropagation(); + safelyRecordCreativeFailure(consumedPrebidAps.attemptId, 'aps_consumed_tombstone'); return; } @@ -1698,11 +1754,27 @@ export function installTsRenderBridge(): void { // Prebid handles ad IDs globally and would otherwise answer a request from // an unrelated iframe when this slot-bound capability rejects it. e.stopImmediatePropagation(); - if (!messageSourceBelongsToAdUnit(e.source, prebidRendererEntry.adUnitCode)) return; + const attemptId = beginApsCreativeAttempt(prebidRendererEntry.adUnitCode); + if (!messageSourceBelongsToAdUnit(e.source, prebidRendererEntry.adUnitCode)) { + safelyRecordCreativeFailure(attemptId, 'aps_source_not_in_ad_unit'); + return; + } const renderer = validateApsRenderer(prebidRendererEntry.renderer); - if (!renderer || !hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) return; + if (!renderer) { + safelyRecordCreativeFailure(attemptId, 'aps_descriptor_fields'); + return; + } + if (!hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) { + safelyRecordCreativeFailure(attemptId, 'aps_tombstone_capacity'); + return; + } if (!consumeApsPrebidRenderer(adId, prebidRendererEntry)) return; - recordConsumedPrebidApsId(consumedPrebidApsIds, adId, prebidRendererEntry.expiresAt); + recordConsumedPrebidApsId( + consumedPrebidApsIds, + adId, + prebidRendererEntry.expiresAt, + attemptId + ); const markUsed = (): void => { try { @@ -1717,7 +1789,10 @@ export function installTsRenderBridge(): void { source: e.source, trustedServer: (validatedRenderer) => { const rendererUrl = apsRendererUrl(); - if (!rendererUrl) return false; + if (!rendererUrl) { + safelyRecordCreativeFailure(attemptId, 'aps_missing_renderer_url'); + return false; + } try { port.postMessage( JSON.stringify({ @@ -1731,9 +1806,11 @@ export function installTsRenderBridge(): void { height: validatedRenderer.height, }) ); + safelyRecordCreativeResponse(attemptId); return true; } catch (err) { log.warn(`[tsjs-gpt] APS Prebid response post failed for '${adId}'`, err); + safelyRecordCreativeFailure(attemptId, 'response_post_failed'); return false; } }, diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts index b7186518b..c04409faf 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts @@ -3034,6 +3034,13 @@ describe('installTsRenderBridge', () => { return iframe.contentWindow!; } + // The APS capability path runs on publisher Prebid ad units, so diagnostics + // resolve the GPT slot by element ID rather than through TS slot mapping. + function stubGoogletagSlot(elementId: string): void { + const slot = { getSlotElementId: () => elementId }; + vi.stubGlobal('googletag', { pubads: () => ({ getSlots: () => [slot] }) }); + } + async function captureBridgeListener(): Promise<(e: MessageEvent) => unknown> { let bridgeListener: ((e: MessageEvent) => unknown) | undefined; const origAdd = window.addEventListener.bind(window); @@ -3339,6 +3346,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3378,6 +3386,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3457,6 +3466,96 @@ describe('installTsRenderBridge', () => { foreignIframe.remove(); }); + it('records a creative attempt for a registered APS renderer so delivery is attributable', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'prebid-diagnostics-ad-id'; + const recordTrustedServerOpportunity = vi.fn(); + const recordTrustedServerCreativeRequest = vi.fn(() => 7); + const recordTrustedServerCreativeResponse = vi.fn(); + const recordTrustedServerCreativeFailure = vi.fn(); + (window as TestWindow).tsjs.gptDiagnosticsRecorder = { + recordTrustedServerOpportunity, + recordTrustedServerCreativeRequest, + recordTrustedServerCreativeResponse, + recordTrustedServerCreativeFailure, + } as never; + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed: vi.fn(), + }, + }; + + try { + stubGoogletagSlot('div-header'); + const bridgeListener = await captureBridgeListener(); + const portMessages: string[] = []; + bridgeListener( + Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source: createTrustedSlotIframe(), + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent + ); + + expect(portMessages).toHaveLength(1); + expect(recordTrustedServerOpportunity.mock.calls[0]?.slice(1, 3)).toEqual([ + 'div-header', + 'renderable_candidate', + ]); + expect(recordTrustedServerCreativeRequest).toHaveBeenCalledWith('div-header'); + expect(recordTrustedServerCreativeResponse).toHaveBeenCalledWith(7); + expect(recordTrustedServerCreativeFailure).not.toHaveBeenCalled(); + } finally { + delete (window as TestWindow).tsjs.gptDiagnosticsRecorder; + } + }); + + it('records the tombstone reason when a consumed APS ad ID is replayed', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'prebid-replayed-ad-id'; + const recordTrustedServerCreativeFailure = vi.fn(); + (window as TestWindow).tsjs.gptDiagnosticsRecorder = { + recordTrustedServerOpportunity: vi.fn(), + recordTrustedServerCreativeRequest: vi.fn(() => 11), + recordTrustedServerCreativeResponse: vi.fn(), + recordTrustedServerCreativeFailure, + } as never; + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed: vi.fn(), + }, + }; + + try { + stubGoogletagSlot('div-header'); + const bridgeListener = await captureBridgeListener(); + const request = (): MessageEvent => + Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: () => {} }], + source: createTrustedSlotIframe(), + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request()); + recordTrustedServerCreativeFailure.mockClear(); + bridgeListener(request()); + + expect(recordTrustedServerCreativeFailure).toHaveBeenCalledWith(11, 'aps_consumed_tombstone'); + } finally { + delete (window as TestWindow).tsjs.gptDiagnosticsRecorder; + } + }); + it('contract test: fails a registered APS runner without a Universal Creative response or markUsed', async () => { const renderer = apsRenderer(); const prebidAdId = 'native-prebid-decline-ad-id'; @@ -3473,6 +3572,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -3515,6 +3615,7 @@ describe('installTsRenderBridge', () => { const marker = enablePublisherNativeMode(); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const portMessages: string[] = []; @@ -4033,6 +4134,7 @@ describe('installTsRenderBridge', () => { }); try { + stubGoogletagSlot('div-header'); const bridgeListener = await captureBridgeListener(); const source = createTrustedSlotIframe(); const postMessage = vi.fn(); From ca46e4f7393826f0e33725299bb08f8bc57ad998 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 4 Sep 2026 12:38:55 -0500 Subject: [PATCH 035/230] Format auction timeline implementation plan --- .../superpowers/plans/2026-08-26-auction-timeline-offsets.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md index 08e14dabc..6dfff6c8c 100644 --- a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -23,9 +23,11 @@ ### Task 1: RequestTimings marks and snapshot fields **Files:** + - Modify: `crates/trusted-server-core/src/request_timing.rs` **Interfaces:** + - Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` - [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. @@ -37,9 +39,11 @@ ### Task 2: Publisher call sites **Files:** + - Modify: `crates/trusted-server-core/src/publisher.rs` **Interfaces:** + - Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. - [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` @@ -50,6 +54,7 @@ ### Task 3: Row columns and datasource **Files:** + - Modify: `crates/trusted-server-core/src/access_telemetry.rs` - Modify: `tinybird/datasources/access_logs_raw.datasource` From c59611ec086b8f64782c16e271f9e4637c0a8720 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 7 Sep 2026 16:56:55 -0400 Subject: [PATCH 036/230] Persist the tester cookie for 30 days so it survives browser restarts The set-tester endpoint minted the ts-tester cookie without Max-Age or Expires, making it a session cookie. Safari deletes session cookies when the browser quits, so Safari testers silently fell back to the baseline arm on every restart. Add a 30-day Max-Age; /_ts/clear-tester already expires the cookie explicitly and is unchanged. --- .../trusted-server-adapter-fastly/src/app.rs | 5 +++-- .../trusted-server-core/src/tester_cookie.rs | 19 ++++++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index cf19518ff..000f7f06e 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -2108,8 +2108,9 @@ mod tests { .to_str() .expect("should render set-cookie as utf-8"); assert_eq!( - set_cookie, "ts-tester=true; Domain=.test-publisher.com; Path=/; Secure; SameSite=Lax", - "tester cookie should use publisher.cookie_domain" + set_cookie, + "ts-tester=true; Domain=.test-publisher.com; Path=/; Secure; SameSite=Lax; Max-Age=2592000", + "tester cookie should use publisher.cookie_domain and persist across browser restarts" ); } diff --git a/crates/trusted-server-core/src/tester_cookie.rs b/crates/trusted-server-core/src/tester_cookie.rs index 58de3d2de..18f2a896e 100644 --- a/crates/trusted-server-core/src/tester_cookie.rs +++ b/crates/trusted-server-core/src/tester_cookie.rs @@ -14,11 +14,18 @@ use crate::constants::COOKIE_TS_TESTER; use crate::error::TrustedServerError; use crate::settings::Settings; +/// Lifetime of the tester cookie in seconds (30 days). +/// +/// Without an explicit lifetime the cookie is session-scoped, and Safari +/// deletes session cookies when the browser quits, so testers silently fall +/// back to the baseline arm on their next visit. +const TESTER_COOKIE_MAX_AGE_SECONDS: u32 = 2_592_000; + /// Formats the tester cookie `Set-Cookie` header value. fn format_tester_cookie(domain: &str) -> String { format!( - "{}=true; Domain={}; Path=/; Secure; SameSite=Lax", - COOKIE_TS_TESTER, domain, + "{}=true; Domain={}; Path=/; Secure; SameSite=Lax; Max-Age={}", + COOKIE_TS_TESTER, domain, TESTER_COOKIE_MAX_AGE_SECONDS, ) } @@ -34,7 +41,8 @@ fn format_clear_tester_cookie(domain: &str) -> String { /// /// Returns `404 Not Found` while `[tester_cookie].enabled` is false. When the /// feature is enabled, returns `204 No Content` with `Set-Cookie: ts-tester=true` -/// scoped to `publisher.cookie_domain`. +/// scoped to `publisher.cookie_domain` and persisted for +/// [`TESTER_COOKIE_MAX_AGE_SECONDS`]. /// /// # Errors /// @@ -138,8 +146,9 @@ mod tests { .to_str() .expect("should render set-cookie as utf-8"); assert_eq!( - set_cookie, "ts-tester=true; Domain=.tester.example; Path=/; Secure; SameSite=Lax", - "tester cookie should use publisher.cookie_domain" + set_cookie, + "ts-tester=true; Domain=.tester.example; Path=/; Secure; SameSite=Lax; Max-Age=2592000", + "tester cookie should use publisher.cookie_domain and persist across browser restarts" ); } From bcc1475f7d458addd2c21745f85456302a9c8ed3 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 037/230] Bound route templates by allowlist and harden the telemetry config surface Review round 3, the blocking route finding plus the config items: - Publisher route templates now come from an operator allowlist (observability.route_sections, default empty): a first segment that matches an entry and has further depth emits the lowercased allowlist entry as /{section}/*; everything else emits /other/*. The shape heuristics (charset, length, digit bounds) are gone because they could not bound identity: depth-2 first segments are usernames under /{username}/posts shapes and single-segment paths are documents. The emitted value set is now fixed by configuration, so no request-derived byte reaches the row. - Integration-proxy responses carry the registered route pattern verbatim (bounded, integration-defined) instead of a classifier output; the registry stores the pattern at registration. - auction_enabled serializes only when false, so a pushed config cannot silently re-enable auction telemetry on rollback; with a serialization test alongside the observability one. - The secret-store validator is renamed to validate_secret_store_key_name with a key_name parameter: it validates an identifier, never a credential, and the old name tainted the key name as a secret value in CodeQL, lighting up eleven pre-existing log sites. - Docs: the tinybird table gains its three missing rows, max_body_bytes states the 1024 floor the code enforces, the rollback guidance now describes the real compatibility boundary (push a compat config first: drop [observability], access_enabled = false, and enabled = false for access-only deployments), and the fixture uses the example-domain convention. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-fastly/src/app.rs | 41 ++- .../src/access_telemetry.rs | 248 +++++++----------- .../src/integrations/registry.rs | 35 ++- crates/trusted-server-core/src/settings.rs | 83 +++++- docs/guide/configuration.md | 51 ++-- .../2026-08-24-request-phase-timing-design.md | 33 ++- tinybird/fixtures/access_logs_raw.ndjson | 2 +- 7 files changed, 289 insertions(+), 204 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index a12484ec5..b2221de8d 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -881,9 +881,16 @@ async fn dispatch_fallback( // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + // The matched route pattern is an integration-defined literal + // (bounded and content-free by construction), so telemetry keeps it + // verbatim instead of running the request path through the lossy + // publisher classifier. route_metadata = Some(RouteMetadata { route_class: RouteClass::IntegrationProxy, - route_template: publisher_route_template(&path), + route_template: state + .registry + .matched_route_pattern(&method, &path) + .map_or_else(|| "/other/*".to_owned(), str::to_owned), }); state .registry @@ -934,7 +941,10 @@ async fn dispatch_fallback( route_metadata = Some(RouteMetadata { route_class: RouteClass::PublisherHtml, - route_template: publisher_route_template(&path), + route_template: publisher_route_template( + &path, + &state.settings.observability.route_sections, + ), }); // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -1517,7 +1527,7 @@ mod tests { NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, build_per_request_services, build_state_from_settings, handle_publisher_request, - publisher_response_into_streaming_response, publisher_route_template, startup_error_router, + publisher_response_into_streaming_response, startup_error_router, }; use base64::Engine as _; use bytes::Bytes; @@ -2451,8 +2461,8 @@ mod tests { .expect("integration-proxy fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); assert_eq!( - metadata.route_template, - publisher_route_template("/integrations/prebid/bundle.js") + metadata.route_template, "/integrations/prebid/bundle.js", + "should carry the registered route pattern verbatim, not a classifier output" ); } @@ -2466,7 +2476,10 @@ mod tests { .get::() .expect("publisher fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::PublisherHtml); - assert_eq!(metadata.route_template, "/news/*"); + assert_eq!( + metadata.route_template, "/other/*", + "the default empty section allowlist should collapse publisher paths" + ); } #[test] @@ -3853,6 +3866,22 @@ mod tests { ); } + #[test] + fn server_timing_absent_when_no_cache_control_header_exists() { + // The fail-closed case: absence of Cache-Control is not evidence of + // privacy, so emission must be suppressed rather than defaulted on. + let mut response = response_builder() + .body(Body::empty()) + .expect("should build a response with no cache-control header"); + + crate::apply_server_timing_header(&mut response, &RequestTimings::new(), true); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit when the response carries no Cache-Control at all" + ); + } + #[test] fn preexisting_server_timing_values_survive() { let mut response = response_builder() diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index aaad46633..e468b109c 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,20 +12,6 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum length of a publisher path's first segment before -/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments -/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a -/// truncated prefix of either would still be identifying, so the segment -/// is rejected whole rather than truncated. -const MAX_SEGMENT_LEN: usize = 32; - -/// Maximum number of ASCII digits in a publisher path's first segment -/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, -/// base36 ids, and reset tokens are digit-heavy; real section names carry -/// at most a year (`2026`) or a small version number, so a segment with -/// more digits than this is treated as an identifier, not a name. -const MAX_SEGMENT_DIGITS: usize = 7; - /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -127,46 +113,43 @@ pub struct RouteMetadata { } /// Normalizes a publisher-fallback request path into a bounded, -/// content-free route template. -/// -/// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: -/// depth is what makes the first segment a section name (`/news/*`) rather -/// than the document itself. The root path `/` maps to itself. Everything -/// else is rejected to `/other/*` — outright, never filtered or truncated, -/// so no fragment of a rejected path ever reaches the row: +/// content-free route template using an operator-configured allowlist of +/// section names. /// -/// - single-segment paths (`/my-post-title` under a `/%postname%/` -/// permalink structure is a per-article slug that no shape heuristic -/// can separate from a section name); -/// - an empty first segment, or one containing any character outside the -/// allowlist after lowercasing (an email address, a search phrase); -/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a -/// truncated prefix of a UUID or token would still be identifying); or -/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits -/// (hex ids, base36 ids, and reset tokens are digit-heavy; section -/// names carry at most a year or a version number). +/// The root path `/` maps to itself. A path whose first segment matches an +/// entry in `sections` (ASCII case-insensitive) and that has at least one +/// further segment maps to `/{section}/*`, emitting the lowercased +/// allowlist entry rather than anything taken from the request. Every +/// other path maps to `/other/*`. /// -/// This is deliberately coarser than the auction-telemetry path -/// normalizer, which redacts long tokens but preserves short identifiers -/// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// This construction makes the template content-free by definition: the +/// set of emitted values is exactly `{"/", "/other/*"}` plus one +/// `/{section}/*` per configured entry, so no request-derived byte ever +/// reaches the row and cardinality is bounded by operator configuration. +/// With the default empty allowlist, every publisher path collapses to +/// `/other/*`. Depth alone was rejected as a signal in review: under +/// `/{username}/posts` shapes the first segment is user data, and under +/// single-segment permalink structures it is the document itself. /// /// # Examples /// /// ``` /// use trusted_server_core::access_telemetry::publisher_route_template; /// -/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); -/// assert_eq!(publisher_route_template("/"), "/"); -/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// let sections = vec!["news".to_owned()]; +/// assert_eq!( +/// publisher_route_template("/news/some-article-slug", §ions), +/// "/news/*" +/// ); +/// assert_eq!(publisher_route_template("/", §ions), "/"); /// assert_eq!( -/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// publisher_route_template("/alice/orders", §ions), /// "/other/*" /// ); +/// assert_eq!(publisher_route_template("/news/x", &[]), "/other/*"); /// ``` #[must_use] -pub fn publisher_route_template(path: &str) -> String { +pub fn publisher_route_template(path: &str, sections: &[String]) -> String { if path == "/" { return "/".to_owned(); } @@ -178,31 +161,18 @@ pub fn publisher_route_template(path: &str) -> String { }; let has_more_depth = !rest.is_empty(); - let lowered = first_segment.to_ascii_lowercase(); - let is_allowlisted = !lowered.is_empty() - && lowered - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); - let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; - let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - - if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { - return "/other/*".to_owned(); - } - - if has_more_depth { - format!("/{lowered}/*") - } else { - // A single-segment path's first segment is the document, not a - // section: `/my-post-title` under WordPress `/%postname%/` is a - // per-article slug, and no shape heuristic can separate it from a - // section name. Only depth >= 2 makes the first segment a section. - "/other/*".to_owned() + let matched_section = sections + .iter() + .find(|section| section.eq_ignore_ascii_case(first_segment)); + match matched_section { + Some(section) if has_more_depth => format!("/{}/*", section.to_ascii_lowercase()), + _ => "/other/*".to_owned(), } } /// A point-in-time view of the access-log dimensions for one response, -/// captured unconditionally at the `Server-Timing` freeze point. +/// captured at the `Server-Timing` freeze point when access telemetry is +/// enabled (a disabled deployment skips the build entirely). /// /// Built from typed response extensions ([`RouteMetadata`], the geo /// lookup state, and the template-cache response state) plus adapter-owned @@ -358,120 +328,84 @@ mod tests { } #[test] - fn publisher_paths_normalize_to_coarse_templates() { + fn publisher_route_template_emits_only_allowlisted_sections() { + let sections = vec!["news".to_owned(), "Sports".to_owned()]; assert_eq!( - publisher_route_template("/news/some-article-slug"), + publisher_route_template("/news/some-article-slug", §ions), "/news/*" ); - assert_eq!(publisher_route_template("/"), "/"); - assert_eq!( - publisher_route_template("/user@example.com/profile"), - "/other/*", - "should reject non-allowlisted characters" - ); - assert_eq!( - publisher_route_template(&format!("/{}", "a".repeat(500))), - "/other/*", - "should reject overlong segments whole rather than truncate" - ); - assert_eq!(publisher_route_template("/search terms here"), "/other/*"); - } - - #[test] - fn publisher_route_template_rejects_opaque_identifier_segments() { - // Every row here passes the character allowlist (`[a-z0-9_-]` is - // exactly what UUIDs, hex ids, and tokens are built from) and must - // be caught by the length and digit-count bounds instead. A - // truncated prefix of any of these would still be identifying, so - // rejection must be whole-segment. - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), - "/other/*", - "should reject a UUID (36 chars) by length" - ); - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), - "/other/*", - "should reject a UUID first segment on deeper paths too" - ); - assert_eq!( - publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), - "/other/*", - "should reject a 32-char hex id by digit count" - ); assert_eq!( - publisher_route_template(&format!("/{}", "a1".repeat(32))), - "/other/*", - "should reject a 64-char token by length" + publisher_route_template("/NEWS/some-article-slug", §ions), + "/news/*", + "matching should be case-insensitive on the request side" ); assert_eq!( - publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), - "/other/*", - "should reject a reset token by length" + publisher_route_template("/sports/scores/today", §ions), + "/sports/*", + "the emitted value should be the lowercased allowlist entry" ); + assert_eq!(publisher_route_template("/", §ions), "/"); assert_eq!( - publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + publisher_route_template("/news", §ions), "/other/*", - "should reject a full article slug by length" + "an allowlisted section with no further depth is the document, not a section" ); - } - - #[test] - fn publisher_route_template_keeps_digit_light_section_names() { - assert_eq!( - publisher_route_template("/2026/08/some-article"), - "/2026/*", - "a year archive segment should pass the digit bound" - ); - assert_eq!( - publisher_route_template("/wp-content/themes/site/app.css"), - "/wp-content/*", - "a hyphenated section name should pass" - ); - } - - #[test] - fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( - publisher_route_template("//double-slash"), + publisher_route_template("/opinion/some-slug", §ions), "/other/*", - "an empty first segment should not be treated as allowlisted" + "a section absent from the allowlist should collapse" ); } #[test] - fn publisher_route_template_rejects_single_segment_paths() { - // A single-segment path's first segment is the document itself - // (WordPress `/%postname%/` puts every article at depth 1), so no - // shape heuristic can separate a slug from a section name; depth - // is the only safe signal. Root-level landing pages pay for this - // deliberately. - assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); - assert_eq!( - publisher_route_template("/my-post-title/"), - "/other/*", - "a trailing slash should not count as depth" - ); - assert_eq!( - publisher_route_template("/1234567"), - "/other/*", - "a numeric post id at the digit boundary should still reject" - ); - assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); - assert_eq!( - publisher_route_template("/about"), - "/other/*", - "root-level landing pages reject too; only depth makes a section" - ); + fn publisher_route_template_collapses_everything_by_default() { + // The default allowlist is empty, so no request-derived byte can + // reach the row: the only emitted values are `/` and `/other/*`. + let empty: Vec = Vec::new(); + for path in [ + "/news/some-article-slug", + "/alice/orders", + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/my-post-title", + "/user@example.com/profile", + "/search terms here", + "//double-slash", + "/550e8400-e29b-41d4-a716-446655440000/profile", + ] { + assert_eq!( + publisher_route_template(path, &empty), + "/other/*", + "should collapse with an empty allowlist: {path}" + ); + } + assert_eq!(publisher_route_template("/", &empty), "/"); } #[test] - fn publisher_route_template_lowercases_before_allowlisting() { - assert_eq!( - publisher_route_template("/News/Article"), - "/news/*", - "should lowercase before validating" - ); + fn publisher_route_template_never_emits_user_data_even_when_allowlisted() { + // Adversarial depth-2 shapes from review: even with sections + // configured, a first segment that is not an exact allowlist match + // collapses, and the emitted value on a match is the allowlist + // entry itself, never request bytes. + let sections = vec!["news".to_owned()]; + for path in [ + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/newsy/article", + ] { + assert_eq!( + publisher_route_template(path, §ions), + "/other/*", + "should collapse non-allowlisted first segments: {path}" + ); + } } #[test] diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 1acb6fdb6..6d65afef3 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -688,7 +688,17 @@ impl IntegrationRegistrationBuilder { } } -type RouteValue = (Arc, &'static str); +/// Proxy handler, integration id, and the registered route pattern (kept so +/// telemetry can label responses with the integration-defined literal, e.g. +/// `/integrations/prebid/*`, instead of deriving anything from the request +/// path). +type RouteValue = (Arc, &'static str, String); + +/// A test-constructor route entry: method, path pattern, and the proxy with +/// its integration id ([`IntegrationRegistry::from_routes`] fills the +/// pattern into [`RouteValue`] itself). +#[cfg(test)] +type RouteEntry<'a> = (Method, &'a str, (Arc, &'static str)); struct IntegrationRegistryInner { // Method-specific routers for O(log n) lookups @@ -805,7 +815,11 @@ impl IntegrationRegistry { for proxy in registration.proxies { for route in proxy.routes() { - let value = (proxy.clone(), registration.integration_id); + let value = ( + proxy.clone(), + registration.integration_id, + route.path.clone(), + ); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if route.path.ends_with("/*") { @@ -894,6 +908,16 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// The registered route pattern matched by `method` and `path`, if any. + /// + /// Patterns are integration-defined literals (for example + /// `/integrations/prebid/*`), so they are bounded and content-free and + /// safe to store as a telemetry dimension, unlike the request path. + #[must_use] + pub fn matched_route_pattern(&self, method: &Method, path: &str) -> Option<&str> { + self.find_route(method, path).map(|value| value.2.as_str()) + } + /// Return true when at least one integration request filter is /// registered. /// @@ -980,7 +1004,7 @@ impl IntegrationRegistry { services, mut req, } = input; - if let Some((proxy, _)) = self.find_route(method, path) { + if let Some((proxy, _, _)) = self.find_route(method, path) { // Organic proxy handler: generate if needed (best effort). // Only generate for document navigations — subresource requests // may lack consent signals such as the Sec-GPC header. @@ -1296,7 +1320,7 @@ impl IntegrationRegistry { /// # Panics /// /// Panics if route registration fails due to duplicate or invalid paths. - pub fn from_routes(routes: Vec<(Method, &str, RouteValue)>) -> Self { + pub fn from_routes(routes: Vec>) -> Self { let mut get_router = Router::new(); let mut post_router = Router::new(); let mut put_router = Router::new(); @@ -1305,7 +1329,8 @@ impl IntegrationRegistry { let mut head_router = Router::new(); let mut options_router = Router::new(); - for (method, path, value) in routes { + for (method, path, (proxy, integration_id)) in routes { + let value: RouteValue = (proxy, integration_id, path.to_owned()); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if path.ends_with("/*") { format!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 216a7fd7a..19951b8d4 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1795,7 +1795,11 @@ pub struct TinybirdSettings { /// configs preserve their current auction-emission behavior after /// upgrading; set `false` to silence auction events while keeping /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). - #[serde(default = "default_true")] + /// Serialized only when `false`: older binaries ignore the key rather + /// than reject it, so writing the default `true` into every pushed + /// config would let a rollback silently resume auction telemetry after + /// an operator disabled it. + #[serde(default = "default_true", skip_serializing_if = "is_true")] pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] @@ -1924,11 +1928,17 @@ impl TinybirdSettings { } if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; - validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; + validate_secret_store_key_name( + &self.auction_token_secret, + "tinybird.auction_token_secret", + )?; } if self.access_enabled { validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; - validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + validate_secret_store_key_name( + &self.access_token_secret, + "tinybird.access_token_secret", + )?; if self.access_sample_rate <= 0.0 { return Err(Report::new(TrustedServerError::Configuration { message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), @@ -1973,8 +1983,15 @@ fn validate_tinybird_dataset(value: &str, setting: &str) -> Result<(), Report Result<(), Report> { - if value.is_empty() || value.chars().any(char::is_control) { +// Named to make the key-name-vs-secret-value distinction legible to static +// analysis: the argument is a Secret Store KEY NAME (an identifier such as +// `tinybird_access_append_token`), never a credential value, so formatting +// it into an error message discloses nothing. +fn validate_secret_store_key_name( + key_name: &str, + setting: &str, +) -> Result<(), Report> { + if key_name.is_empty() || key_name.chars().any(char::is_control) { return Err(Report::new(TrustedServerError::Configuration { message: format!("{setting} must be a non-empty Secret Store key"), })); @@ -2554,6 +2571,14 @@ pub(crate) const AUCTION_DEBUG_UPSTREAM_METADATA_KEYS: &[&str] = &[ "upstream_message_truncated", ]; +/// `skip_serializing_if` helper: true is the serde default for the fields +/// that use it, so serializing it would only widen the pushed config's +/// rollback surface. +#[allow(clippy::trivially_copy_pass_by_ref)] +fn is_true(value: &bool) -> bool { + *value +} + fn default_true() -> bool { true } @@ -2714,6 +2739,13 @@ pub struct ObservabilitySettings { /// timing. Defaults to `false` (off). #[serde(default)] pub server_timing_enabled: bool, + /// Section names whose publisher paths keep a named route template in + /// access telemetry (`/{section}/*`); everything else collapses to + /// `/other/*`. Matching is ASCII case-insensitive on the first path + /// segment, and a match requires at least one further segment. Defaults + /// to empty, which collapses every publisher path. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub route_sections: Vec, } impl ObservabilitySettings { @@ -4448,6 +4480,47 @@ mod tests { ); } + #[test] + fn auction_enabled_serializes_only_when_disabled() { + let mut settings = create_test_settings(); + assert!(settings.tinybird.auction_enabled, "should default on"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("auction_enabled"), + "should omit the default-true key: a rollback must not silently \ + re-enable auction telemetry an operator disabled" + ); + + settings.tinybird.auction_enabled = false; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("auction_enabled = false"), + "should serialize the operator's explicit disable" + ); + } + + #[test] + fn route_sections_serialize_only_when_configured() { + let mut settings = create_test_settings(); + assert!( + settings.observability.route_sections.is_empty(), + "should default to the collapse-everything allowlist" + ); + settings.observability.server_timing_enabled = true; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("route_sections"), + "should omit the empty allowlist so a prior binary can parse the config" + ); + + settings.observability.route_sections = vec!["news".to_owned()]; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("route_sections"), + "should serialize a configured allowlist" + ); + } + #[test] fn test_settings_from_valid_toml() { let toml_str = crate_test_settings_str(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 2b154b8a3..8cc3f7324 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1926,9 +1926,10 @@ enabling one does not enable the other. ### `[observability]` -| Field | Type | Required | Default | Description | -| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | -| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| Field | Type | Required | Default | Description | +| ----------------------- | -------- | -------- | ------- | --------------------------------------------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| `route_sections` | String[] | No | `[]` | Section names kept as publisher route templates (`/{section}/*`) in access telemetry; empty collapses all | **Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and @@ -1957,6 +1958,12 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: tip The Axum dev server reads this flag once at startup +Unlike the Fastly adapter, which reads settings per request, the Axum dev +server bakes `server_timing_enabled` into its service when it starts. +Flipping the flag there requires a restart to take effect. +::: + ::: warning Client-visible latency disclosure The `Server-Timing` header is sent to every client on eligible responses, not only to operators: browsers expose the values to same-origin JavaScript @@ -1993,12 +2000,15 @@ independent emitters: auction telemetry (`auction_dataset`, `auction_token_secret`) and access telemetry. The keys below cover the access-telemetry sink and the shared enable flags. -| Field | Type | Required | Default | Description | -| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | -| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | -| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | -| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | -| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | +| Field | Type | Required | Default | Description | +| --------------------- | ------- | ------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_dataset` | String | Yes, when `access_enabled` | `access_logs_raw` | Access-log Events API datasource name | +| `access_token_secret` | String | Yes, when `access_enabled` | `tinybird_access_append_token` | Secret Store key holding the access APPEND token | +| `max_body_bytes` | Integer | No | `1048576` | Maximum NDJSON request body size; must be at least 1024 | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | **Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird sinks separately so that turning on one does not silently turn on (or leave @@ -2007,8 +2017,8 @@ Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at config load as an armed-but-silent configuration; use `access_enabled` itself to turn the sink off, not the sample rate. Enabling `access_enabled` also requires the shared transport fields (`enabled`, non-empty `api_host`, -`secret_store`, `access_dataset`, `access_token_secret`, and a positive -`max_body_bytes`) to already be set. +`secret_store`, `access_dataset`, `access_token_secret`, and a +`max_body_bytes` of at least 1024) to already be set. **Example**: @@ -2041,18 +2051,23 @@ the response the reader sees. ### Deploy and rollback ordering -::: warning `Settings` rejects unknown fields; order matters -Both `[observability].server_timing_enabled` and the new `[tinybird]` access -keys are new fields on a config schema that uses `deny_unknown_fields`, so an -older binary fails to load a config that carries them. +::: warning Push a compatibility config before rolling back +The compatibility boundary is uneven. The top-level `Settings` schema uses +`deny_unknown_fields`, so an older binary rejects a config carrying the +`[observability]` table. The nested `[tinybird]` table does not: an older +binary accepts unknown keys there, rejects `access_enabled = true` through +validation, ignores `auction_enabled` entirely, and reads `enabled = true` +as "auction telemetry on". **Deploying**: upgrade the binary first, then push a config containing the new fields second. Never push a config with these fields while a pre-observability binary can still receive it. -**Rolling back**: reverse the order. Remove the `[observability]` table and -any new `[tinybird]` access keys from the config and push that first, then -roll back the binary second. +**Rolling back**: push a compatibility config first, then roll the binary +back. The compatibility config removes the `[observability]` table, sets +`access_enabled = false`, and, for a deployment that only used the access +sink, sets `enabled = false` as well; otherwise the older binary would +interpret the leftover `enabled = true` as enabling auction telemetry. ::: ## Validation diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 2892386f4..3833e81eb 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -85,7 +85,7 @@ auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in th | send_edgezero_response, immediately before into_parts(): mark_headers_ready() snapshot (unconditional) - build AccessTelemetrySnapshot (unconditional) + build AccessTelemetrySnapshot (only when tinybird.access_enabled) append Server-Timing header (flag-gated, only on conclusively private responses) | headers committed; body streams @@ -125,9 +125,11 @@ New module `crates/trusted-server-core/src/request_timing.rs`. - Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses three boundaries: adapter entry to core handlers, the streaming body closure (records body-phase spans after the response object has been handed off), and the adapter's - post-send emission read. Access is exclusively `try_lock()`: a contended or - poisoned lock drops the sample immediately rather than waiting, so recording can - never delay a request. + post-send emission read. Access is exclusively `try_lock()`: a contended lock + drops the one sample rather than waiting, so recording can never delay a + request, and a poisoned lock is recovered (the guarded data are plain counters + with no invariant a panic can break) so one panic cannot silence the rest of + the request's timing. - Recording API: `timings.record(Phase::Geo, dur)` and a scope guard `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration math; a non-monotonic reading records zero rather than panicking. The auction-wait @@ -214,7 +216,8 @@ marks middleware finalization, not header commitment, and must not be treated as timing boundary. At the freeze point, in order: `mark_headers_ready()` (unconditional), the -`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`AccessTelemetrySnapshot` build (gated on `tinybird.access_enabled`, +section 10), then, gated on `observability.server_timing_enabled`, append one `Server-Timing` header from `server_timing_value()`. Append semantics, never insert: an origin-supplied Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, @@ -290,12 +293,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. -- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, carries more than 7 - ASCII digits, or is the only segment in the path. Depth is what makes a first - segment a section name: single-segment paths are documents (WordPress - `/%postname%/` puts every article at depth 1), so they reject wholesale, root - landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` +- Publisher templates come from an operator-configured allowlist of section names + (`observability.route_sections`, default empty), not from the request: a path + whose first segment matches an allowlist entry and that has at least one further + segment emits `/{section}/*` (the lowercased allowlist entry itself); everything + else emits `/other/*`, and the root path emits `/`. This replaces the earlier + shape heuristics (charset, length, digit bounds), which review showed cannot + bound identity: depth-2 first segments are usernames on `/{username}/posts` + shapes, and single-segment paths are documents under `/%postname%/` permalinks. + With the allowlist, the emitted value set is fixed by configuration, so no + request-derived byte ever reaches the row. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -364,7 +371,9 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 10. Emission mechanics -- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before +- `AccessTelemetrySnapshot`: built at the freeze point when + `tinybird.access_enabled` is set (revised in review from the original + unconditional build, so a disabled deployment pays nothing here), before `into_parts()` consumes the response. It captures method, status, route metadata (from the `RouteMetadata` extension), and typed dimension states (`env`, `template_cache_state`, geo country). It exists because nothing else survives to diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson index 3c82c5ca6..983b3acd1 100644 --- a/tinybird/fixtures/access_logs_raw.ndjson +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -1 +1 @@ -{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.example","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From a1523675a21f5bbd3a327de45a95385d26716fc2 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 038/230] Address round-3 telemetry robustness findings - The access sink streams the Tinybird response (the body is never consumed; buffered conversion materializes chunked bodies before the limit check) and newline-terminates rows to match the auction sink's NDJSON framing, with the recording client now asserting both. - Poisoned RequestTimings locks recover via into_inner instead of silently dropping every subsequent sample: the guarded data are plain counters, so one panic cannot blank the header and row for the rest of the request. Module and spec docs updated to stop conflating poisoning with contention. - The geo write-back skips 401 responses through a shared helper: resolve_geo_for_response short-circuits on 401 before consulting the carried state, so the old unconditional write downgraded a carried Resolved to Attempted and cost the row its country. - TimedKvStore forwards exists, so decorating a store with a cheap metadata probe (Spin) no longer downgrades it to the get-and-discard default body; with a contradiction-stub delegation test. - Post-send ordering is owned by run_post_send_steps, which both production sites route through, and the instrumented sequence test drives the real seam: elapsed stamped by send, then pull-sync, then telemetry. - Axum: dev_server_service remains the standard path; new tests pin flag-off suppression and the extension round trip (a phase recorded in the handler must surface as ts-filter in the header); the outer-wrapper rationale is reworded to the terminal-freeze-point argument; the configuration guide notes the flag is read once at startup. - The no-Cache-Control fail-closed case is pinned by a test, and t0's boundary (constructed after the adapter prologue) is documented. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-axum/src/timing.rs | 82 +++++++++++++++-- .../wrangler.integration.generated.toml | 16 ++++ .../trusted-server-adapter-fastly/src/main.rs | 91 ++++++++++++++----- .../src/middleware.rs | 80 +++++++++++++--- .../src/tinybird.rs | 26 +++++- .../src/platform/timed_kv.rs | 68 ++++++++++++++ .../trusted-server-core/src/request_timing.rs | 89 +++++++++++++----- .../plans/2026-08-24-request-phase-timing.md | 3 +- 8 files changed, 385 insertions(+), 70 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs index 832823c15..315ebb5ef 100644 --- a/crates/trusted-server-adapter-axum/src/timing.rs +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -9,13 +9,15 @@ //! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). //! //! This wraps *outside* `RouterService` rather than registering as -//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits -//! `RouterInner::dispatch` before its middleware chain ever runs, so -//! middleware never sees those responses. By the time a response reaches -//! this layer -- after `RouterService::oneshot` inside -//! `EdgeZeroAxumService::call` has already converted any dispatch error into -//! a plain response -- every response is covered uniformly, router-generated -//! or not. +//! `RouterBuilder::middleware` because the tower boundary is the terminal +//! freeze point: by the time a response reaches this layer -- after +//! `RouterService::oneshot` inside `EdgeZeroAxumService::call` has +//! converted any dispatch error into a plain response -- every response is +//! covered uniformly regardless of how routing produced it, and the +//! position survives future routing changes. (In this application's router +//! a catch-all fallback spans every path and publisher method, so +//! router-generated 404/405s that bypass middleware are close to +//! unreachable today; the outer position does not depend on them.) //! //! `/health` is excluded by path match before a //! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) @@ -163,6 +165,72 @@ mod tests { ); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_suppresses_header_when_flag_is_off() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), false); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "should not emit server-timing when server_timing_enabled is false" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_round_trips_phase_timings_recorded_in_the_handler() { + // The collector crosses the adapter boundary as a request extension; + // this pins the round trip end to end: a phase recorded inside a + // core-style handler must come back out in the rendered header, so + // a future adapter conversion that drops request extensions fails + // here instead of silently losing every phase. + let router = RouterService::builder() + .get("/private", |ctx: RequestContext| async move { + if let Some(timings) = ctx.request().extensions().get::() { + timings.record( + trusted_server_core::request_timing::Phase::Filter, + std::time::Duration::from_millis(7), + ); + } + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-filter;dur=7.0"), + "a phase recorded in the handler should survive the adapter round trip: {server_timing}" + ); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn axum_404_carries_header_when_private() { // An empty router has no routes at all, so any path dispatches diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 000000000..263403193 --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 333df736a..8cd2b39d0 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -290,8 +290,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); - emit_access_telemetry_after_send(settings, &outcome, &timings); + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + settings, + &partner_registry, + &ec_state, + ) + }, + || emit_access_telemetry_after_send(settings, &outcome, &timings), + ); return; } Err(e) => { @@ -322,12 +330,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send( - &settings, - &partner_registry, - &ec_state, + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + &settings, + &partner_registry, + &ec_state, + ); + }, + || emit_access_telemetry_after_send(&settings, &outcome, &timings), ); - emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -408,13 +420,9 @@ fn apply_entry_point_finalize_headers( // router-level 404/405 for an unregistered method), so `geo_state` may // still be `NotAttempted` even after a fresh lookup just ran above. // Write the resolved outcome back so the access-telemetry snapshot built - // later in `send_edgezero_response` sees what was actually looked up, - // not the stale carried-in state. - let resolved_state = match &geo_info { - Some(info) => GeoLookupState::Resolved(info.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // later in `send_edgezero_response` sees what was actually looked up; + // 401 handling lives in the shared helper. + middleware::write_back_geo_lookup_state(response, geo_info.as_ref()); } fn apply_edgezero_ec_finalize( @@ -453,6 +461,19 @@ fn run_edgezero_pull_sync_after_send( } } +/// Runs the post-send steps in their contract order: EC identity pull-sync +/// first, then access-telemetry emission. +/// +/// Every `edgezero_main` site that has both steps routes through this +/// function, so the ordering is owned in exactly one place and the +/// sequence test can instrument it; `request_elapsed` is already stamped +/// before either step because `send_edgezero_response` stamps it before +/// returning. +fn run_post_send_steps(pull_sync: impl FnOnce(), emit_access_telemetry: impl FnOnce()) { + pull_sync(); + emit_access_telemetry(); +} + /// Builds and emits the access-telemetry row for one delivered response, /// when access telemetry is enabled and this request is sampled in. /// @@ -1003,6 +1024,8 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { + use std::sync::Mutex; + use super::*; use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; @@ -1733,13 +1756,13 @@ mod tests { } #[test] - fn request_elapsed_is_stamped_when_send_returns() { - // `edgezero_main`'s post-send ordering (pull-sync before telemetry) - // is a source-order invariant with no injectable seam, so this test - // deliberately proves only the leg that has one: by the time - // `send_edgezero_response` returns, `request_elapsed` is already - // stamped, so everything `edgezero_main` runs afterwards (pull-sync, - // telemetry emission) is excluded from `request_elapsed_ms`. + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // The full contract sequence, instrumented through the real seams: + // `send_edgezero_response` stamps `request_elapsed` before + // returning, and `run_post_send_steps` (which every production + // site with both steps routes through) owns pull-sync-then- + // telemetry ordering. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1757,14 +1780,36 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should be stamped by the time send returns" + "request_elapsed should be stamped before any post-send step runs" ); assert!( outcome.snapshot.is_some(), "the access snapshot should exist for the enabled context" ); + + let pull_log = Arc::clone(&log); + let emit_log = Arc::clone(&log); + run_post_send_steps( + move || { + pull_log + .lock() + .expect("should lock order log") + .push("pull_sync") + }, + move || { + emit_log + .lock() + .expect("should lock order log") + .push("telemetry") + }, + ); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 79ded8362..7bd577363 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -100,17 +100,10 @@ impl Middleware for FinalizeResponseMiddleware { }) }); - // Write the resolved outcome back so a downstream access-telemetry - // snapshot (built from response extensions after finalize) sees - // what was actually looked up here rather than the stale carried-in - // state — mirrors the entry-point finalize site in `main.rs` - // (`apply_entry_point_finalize_headers`), which writes back for the - // same reason. - let resolved_state = match &geo_info { - Some(geo) => GeoLookupState::Resolved(geo.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // Mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`); 401 handling lives in the + // shared helper. + write_back_geo_lookup_state(&mut response, geo_info.as_ref()); apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response @@ -193,6 +186,26 @@ impl Middleware for AuthMiddleware { /// is intentionally more conservative: geo data is not sent to any /// unauthenticated caller regardless of whether the 401 originated from this /// server or the upstream origin. +/// Writes the resolved geo outcome back onto the response as a +/// [`GeoLookupState`] extension, so a downstream access-telemetry snapshot +/// sees what was actually looked up rather than the stale carried-in state. +/// +/// Skips the write on a 401: [`resolve_geo_for_response`] returns `None` +/// for unauthorized responses before consulting the carried state, so +/// writing `Attempted` there would overwrite a carried `Resolved` with a +/// value that was never looked up, and the row would lose a country it +/// legitimately had. +pub(crate) fn write_back_geo_lookup_state(response: &mut Response, geo_info: Option<&GeoInfo>) { + if response.status() == StatusCode::UNAUTHORIZED { + return; + } + let resolved_state = match geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); +} + pub(crate) fn resolve_geo_for_response( response: &Response, carried: &GeoLookupState, @@ -285,6 +298,51 @@ pub(crate) use trusted_server_core::response_privacy::{ mod tests { use super::*; + #[test] + fn geo_write_back_preserves_resolved_state_on_401() { + // A 401 short-circuits geo resolution before the carried state is + // consulted, so the write-back must not downgrade a carried + // Resolved to Attempted (which would cost the row its country). + let mut response = response_builder() + .status(StatusCode::UNAUTHORIZED) + .body(Body::empty()) + .expect("should build a 401 response"); + response + .extensions_mut() + .insert(GeoLookupState::Resolved(sample_geo_info())); + + write_back_geo_lookup_state(&mut response, None); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(geo)) => { + assert_eq!( + geo.country, + sample_geo_info().country, + "should keep the carried country" + ); + } + other => panic!("should keep the Resolved state on a 401, got {other:?}"), + } + } + + #[test] + fn geo_write_back_records_attempted_on_non_401_miss() { + let mut response = response_builder() + .status(StatusCode::OK) + .body(Body::empty()) + .expect("should build a 200 response"); + + write_back_geo_lookup_state(&mut response, None); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should record an attempted-but-missed lookup on ordinary responses" + ); + } + use std::collections::HashMap; use std::net::IpAddr; use std::sync::{Arc, Mutex}; diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 84dfc37d8..dfcf65f98 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -326,8 +326,13 @@ fn build_access_events_request( pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, - row: String, + mut row: String, ) -> Result<(), Report> { + // Match the auction sink's NDJSON framing: every row is + // newline-terminated, and the terminator counts toward the body limit. + if !row.ends_with('\n') { + row.push('\n'); + } let body_len = row.len(); if body_len > target.max_body_bytes { return Err(Report::new(TrustedServerError::Proxy { @@ -354,8 +359,11 @@ pub(crate) async fn emit_access_event( backend_name ); + // The response body is never consumed, so stream it: buffered + // conversion on Fastly materializes the body before the size limit is + // enforced, which a chunked response could abuse. let response = client - .send(PlatformHttpRequest::new(request, backend_name)) + .send(PlatformHttpRequest::new(request, backend_name).with_stream_response()) .await .change_context(TrustedServerError::Proxy { message: "failed to send Tinybird access telemetry request".to_owned(), @@ -490,6 +498,7 @@ mod tests { uri: String, headers: Vec<(String, String)>, body: Vec, + stream_response: bool, } /// Records outbound requests and, for [`PlatformHttpClient::send`] (the @@ -515,6 +524,7 @@ mod tests { fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; + let stream_response = request.stream_response; let (parts, body) = request.request.into_parts(); let headers = parts .headers @@ -532,6 +542,7 @@ mod tests { uri: parts.uri.to_string(), headers, body: body.into_bytes().unwrap_or_default().to_vec(), + stream_response, }; self.requests .lock() @@ -949,10 +960,15 @@ mod tests { header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), Some("Bearer test-tinybird-access-append-token") ); + let body = std::str::from_utf8(&requests[0].body).expect("should record utf8 body"); assert_eq!( - std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), - row, - "should send the row verbatim as the request body" + body, + format!("{row}\n"), + "should send newline-delimited JSON, matching the auction sink's framing" + ); + assert!( + requests[0].stream_response, + "should stream the Tinybird response: the body is never consumed" ); } diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs index 0594b7cef..faac0ad2c 100644 --- a/crates/trusted-server-core/src/platform/timed_kv.rs +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -53,6 +53,15 @@ impl PlatformKvStore for TimedKvStore> { self.inner.put_bytes(key, value).await } + // Forwarded explicitly: the trait's default body falls back to + // `get_bytes`, which would silently downgrade a backend's cheap + // metadata-only existence probe (the Spin adapter has one) into a full + // value transfer just because the store was decorated. + async fn exists(&self, key: &str) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.exists(key).await + } + async fn put_bytes_with_ttl( &self, key: &str, @@ -145,6 +154,65 @@ mod tests { ); } + #[test] + fn exists_delegates_to_the_inner_store_not_get_bytes() { + // A stub whose `exists` answer contradicts its `get_bytes` answer: + // if the decorator fell back to the trait's get-and-discard default + // body, this would return `false`. + struct ExistsOnlyStore; + + #[async_trait::async_trait(?Send)] + impl PlatformKvStore for ExistsOnlyStore { + async fn get_bytes(&self, _key: &str) -> Result, KvError> { + Ok(None) + } + async fn put_bytes(&self, _key: &str, _value: Bytes) -> Result<(), KvError> { + Ok(()) + } + async fn put_bytes_with_ttl( + &self, + _key: &str, + _value: Bytes, + _ttl: StdDuration, + ) -> Result<(), KvError> { + Ok(()) + } + async fn delete(&self, _key: &str) -> Result<(), KvError> { + Ok(()) + } + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + Ok(KvPage { + keys: Vec::new(), + cursor: None, + }) + } + async fn exists(&self, _key: &str) -> Result { + Ok(true) + } + } + + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(ExistsOnlyStore); + let store = TimedKvStore::new(inner, timings.clone()); + + let exists = futures::executor::block_on(store.exists("key")) + .expect("should forward the existence probe"); + assert!( + exists, + "should delegate to the inner exists, not the get_bytes default body" + ); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should time the existence probe like any other store operation" + ); + } + #[test] fn store_name_is_not_timed() { let timings = RequestTimings::new(); diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index cbb269b1e..3b26c3ca2 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -1,10 +1,12 @@ //! Per-request phase timing collection and Server-Timing rendering. //! -//! Collection is always-on and infallible: saturating math, lock failure -//! drops the sample, no panics. See the design spec +//! Collection is always-on and infallible: saturating math, no panics. A +//! contended lock drops the one sample; a poisoned lock recovers (the +//! guarded data are plain counters), so a panic elsewhere cannot silence +//! the rest of the request's timing. See the design spec //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, TryLockError}; use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; @@ -91,7 +93,12 @@ pub enum AuctionWaitPlacement { /// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. struct Inner { - /// Instant the request started; the reference point for elapsed marks. + /// Instant the collector was constructed; the reference point for + /// elapsed marks. Adapters construct the collector after their own + /// prologue (client-request acquisition, logger init, and any + /// short-circuit routes on Fastly), so `ts-total` and + /// `time_elapsed_ms` exclude that prologue rather than measuring + /// wall-clock-from-accept. t0: Instant, /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. phases: [Option; PHASE_COUNT], @@ -111,9 +118,9 @@ struct Inner { /// Per-request phase timing collector. /// /// Cheap to clone (an [`Arc`] handle) and safe to share across threads and -/// async tasks handling the same request. Every method is infallible: lock -/// contention or poisoning silently drops the sample rather than blocking or -/// panicking. +/// async tasks handling the same request. Every method is infallible: +/// contention silently drops the one sample rather than blocking, and a +/// poisoned lock is recovered rather than treated as permanent loss. #[derive(Clone)] pub struct RequestTimings(Arc>); @@ -134,10 +141,16 @@ impl RequestTimings { /// Accumulates `dur` into `phase`'s running total. /// /// Repeated calls for the same phase saturate-add rather than overwrite. - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record(&self, phase: Phase, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = phase.index(); let accumulated = inner.phases[index] @@ -149,10 +162,16 @@ impl RequestTimings { /// Records an auction wait duration under [`Phase::AuctionWait`] and /// stores its placement. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = Phase::AuctionWait.index(); let accumulated = inner.phases[index] @@ -178,8 +197,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_headers_ready(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.headers_ready_total.is_none() { inner.headers_ready_total = Some(inner.t0.elapsed()); @@ -192,8 +217,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_request_elapsed(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.request_elapsed.is_none() { inner.request_elapsed = Some(inner.t0.elapsed()); @@ -202,10 +233,16 @@ impl RequestTimings { /// Records the response body size in bytes. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn set_resp_bytes(&self, bytes: u64) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; inner.resp_bytes = Some(bytes); } @@ -221,7 +258,11 @@ impl RequestTimings { /// silently (returning `None`) on lock contention or poisoning. #[must_use] pub fn server_timing_value(&self) -> Option { - let inner = self.0.try_lock().ok()?; + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return None, + }; let total = inner.headers_ready_total?; let mut entries = vec![format_entry("ts-total", total)]; for phase in HEADER_PHASES { @@ -241,8 +282,10 @@ impl RequestTimings { /// consistent with the infallibility of every other method. #[must_use] pub fn snapshot(&self) -> TimingSnapshot { - let Ok(inner) = self.0.try_lock() else { - return TimingSnapshot::default(); + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return TimingSnapshot::default(), }; TimingSnapshot { time_elapsed_ms: duration_ms(inner.headers_ready_total), diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 02cda64d8..1b25b70d1 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -819,7 +819,8 @@ plus the coarse template; the freeze point consumes the extension (no `RouteClas column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set at every point that writes `x-ts-template-cache` so header and extension cannot -drift; the row reads the extension. The snapshot is built unconditionally in +drift; the row reads the extension. The snapshot is built (when access +telemetry is enabled) in `send_edgezero_response` right after `mark_headers_ready()` and returned inside `DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). From 1476137e794e95f43d36a7531d59a082d640ca8f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 15 Sep 2026 17:29:49 +0530 Subject: [PATCH 039/230] Show help for a bare ts dev proxy invocation Signed-off-by: dhruv8sh --- .../src/commands/dev/mod.rs | 2 +- .../src/commands/dev/proxy/config.rs | 11 ++-------- .../src/commands/dev/proxy/mod.rs | 1 + crates/trusted-server-cli/src/run.rs | 21 +++++++++++++++++++ 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/mod.rs b/crates/trusted-server-cli/src/commands/dev/mod.rs index 7a61d769b..822d99fd1 100644 --- a/crates/trusted-server-cli/src/commands/dev/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/mod.rs @@ -31,6 +31,6 @@ pub enum DevCommand { pub fn run(command: DevCommand) -> Result<(), String> { match command { #[cfg(target_os = "macos")] - DevCommand::Proxy(args) => proxy::run(&args).map_err(|report| format!("{report:?}")), + DevCommand::Proxy(args) => proxy::run(&args).map_err(|report| format!("{report:#}")), } } diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index d831735c2..9d0ef2380 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -359,14 +359,7 @@ mod tests { }; fn base_args() -> crate::commands::dev::proxy::ProxyArgs { - // Construct via clap so defaults match the real surface. - use clap::Parser; - #[derive(clap::Parser)] - struct W { - #[command(flatten)] - a: crate::commands::dev::proxy::ProxyArgs, - } - W::parse_from(["ts"]).a + parse_args(&["ts", "--listen", "127.0.0.1:18080"]) } fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { @@ -382,7 +375,7 @@ mod tests { #[test] fn clap_parses_rewrite_host_as_a_bool() { assert!( - !parse_args(&["ts"]).rewrite_host, + !parse_args(&["ts", "--listen", "127.0.0.1:18080"]).rewrite_host, "absent --rewrite-host is false" ); assert!( diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs index 1bebd872d..6b6d2b748 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs @@ -78,6 +78,7 @@ async fn finish_interrupted_run( /// `ts dev proxy [OPTIONS]` — see the design spec §4. #[derive(Debug, clap::Args)] +#[command(arg_required_else_help = true)] pub struct ProxyArgs { /// Rewrite rule `FROM=TO` (repeatable). #[arg(long = "map", value_name = "FROM=TO")] diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 13009d448..e5079ebb8 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -678,4 +678,25 @@ mod tests { "error should explain unsupported option" ); } + + #[test] + #[cfg(target_os = "macos")] + fn dev_proxy_bare_invocation_shows_help_before_running() { + let error = Args::try_parse_from(["ts", "dev", "proxy"]) + .expect_err("a bare `ts dev proxy` should short-circuit to help, not run"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::DisplayHelpOnMissingArgumentOrSubcommand, + "should print help instead of touching system proxy state or attempting sudo" + ); + parse(&["ts", "dev", "proxy", "ca", "path"]); + } + + #[test] + #[cfg(target_os = "macos")] + fn dev_proxy_partial_rule_parses_instead_of_showing_help() { + // An explicit but incomplete rule (`--from` with no `--to`) must reach + // `run` and surface the concise no-rule error there, not clap help. + parse(&["ts", "dev", "proxy", "--from", "a.example.com"]); + } } From ecc962fc550001f2b6d4363f8e0d49f816c164f6 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 15 Sep 2026 18:27:28 +0530 Subject: [PATCH 040/230] Enforce formatting on root level markdown files Signed-off-by: dhruv8sh --- .github/workflows/format.yml | 4 ++++ AGENTS.md | 38 ++++++++++++++++++------------------ FAQ_POC.md | 18 ++++++++--------- ProjectGovernance.md | 12 ++++++------ 4 files changed, 38 insertions(+), 34 deletions(-) diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 7253b5ea2..c844e95d3 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,5 +149,9 @@ jobs: - name: Run Prettier (check) run: npm run format + - name: Run Prettier (check) — root Markdown + working-directory: . + run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + - name: Build with VitePress (fails on dead links) run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 3b7189204..88bd7595a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -323,14 +323,14 @@ IntegrationRegistration::builder(ID) ## Configuration Files -| File | Purpose | -| --------------------- | ---------------------------------------------------------- | -| `edgezero.toml` | EdgeZero app/platform manifest and logical stores | -| `fastly.toml` | Fastly service configuration and build settings | -| `trusted-server.example.toml` | Source-controlled Trusted Server app-config template | -| `trusted-server.toml` | Operator-owned app config; gitignored; `ts config push` publishes it as an EdgeZero blob envelope | -| `rust-toolchain.toml` | Pins Rust version to 1.95.0 | -| `.env.dev` | Local development environment variables | +| File | Purpose | +| ----------------------------- | ------------------------------------------------------------------------------------------------- | +| `edgezero.toml` | EdgeZero app/platform manifest and logical stores | +| `fastly.toml` | Fastly service configuration and build settings | +| `trusted-server.example.toml` | Source-controlled Trusted Server app-config template | +| `trusted-server.toml` | Operator-owned app config; gitignored; `ts config push` publishes it as an EdgeZero blob envelope | +| `rust-toolchain.toml` | Pins Rust version to 1.95.0 | +| `.env.dev` | Local development environment variables | --- @@ -433,18 +433,18 @@ both runtime behavior and build/tooling changes. ## Key Files -| File | Purpose | -| -------------------------------------------- | ------------------------------------------------- | -| `crates/trusted-server-core/src/integrations/registry.rs` | IntegrationRegistry, `js_module_ids()` | -| `crates/trusted-server-core/src/tsjs.rs` | Script tag generation with module IDs | -| `crates/trusted-server-core/src/html_processor.rs` | Injects `` sequences inside the string. pub(crate) fn build_bids_script(bid_map: &serde_json::Map) -> String { + build_bids_script_with_diagnostics(bid_map, None) +} + +fn build_bids_script_with_diagnostics( + bid_map: &serde_json::Map, + auction_diagnostics: Option<&BrowserAuctionDiagnostics>, +) -> String { let json = serde_json::to_string(bid_map) .expect("serde_json::to_string of Map should be infallible"); let escaped = html_escape_for_script(&json); @@ -5614,6 +5706,23 @@ pub(crate) fn build_bids_script(bid_map: &serde_json::Map(function(){{\ +var t=window.tsjs=window.tsjs||{{}};\ +var b=JSON.parse(\"{}\");\ +var d=JSON.parse(\"{}\");\ +var s=t.scheduleInitialAdInit;\ +if(typeof s===\"function\")s(b,void 0,d);\ +else{{t.bids=b;t.auctionDiagnostics=d;}}\ +}})();", + escaped, + html_escape_for_script(&diagnostics) + ); + } + format!( "", + html_escape_for_script(slots_json), + html_escape_for_script(&bids), + html_escape_for_script(&diagnostics) + ); + } + format!( ""#.to_vec(), + vec![("content-type", "text/html"), ("cache-control", "no-store")], + ); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req(Method::GET, "https://publisher.example.com/integrations/sourcepoint/cdn/us_pm/index.html"), + ) + .await + .expect("should proxy HTML without Content-Length"); + + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve upstream HTML cache policy" + ); + assert_eq!( + response.into_body().into_bytes_bounded(1024).await.expect("should collect HTML response").as_ref(), + br#""#, + "should rewrite streamed privacy-manager assets without Content-Length" + ); + }); + } + + #[test] + fn handle_accepts_exact_rewrite_limit_and_stops_reading_on_overflow() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + let limit = MAX_REWRITE_BODY_SIZE as usize; + for content_type in ["application/javascript", "text/html"] { + for declared_length in [None, Some("1")] { + for extra_bytes in [0, 1, TEST_CHUNK_SIZE * 2] { + let client = Arc::new(StreamingHttpClient::new()); + let mut headers = vec![("content-type", content_type)]; + if let Some(length) = declared_length { + headers.push(("content-length", length)); + } + client.stub.push_response_with_headers( + 200, + vec![b' '; limit + extra_bytes], + headers, + ); + let services = build_services_with_http_client(client.clone()); + + let result = integration.handle( + &settings, + &services, + make_req(Method::GET, "https://publisher.example.com/integrations/sourcepoint/cdn/asset"), + ).await; + + if extra_bytes == 0 { + let response = result.expect("should accept exactly 5 MiB"); + assert!( + response.headers().get(header::CONTENT_LENGTH).is_none(), + "should remove the advisory length after rewriting" + ); + assert_eq!( + take_body_bytes(response).len(), + limit, + "should retain the entire body at the limit" + ); + } else { + let error = + result.expect_err("should reject an oversized streamed body"); + assert_eq!( + error.current_context().status_code(), + StatusCode::BAD_GATEWAY, + "should report upstream overflow as 502" + ); + assert!( + matches!(error.current_context(), TrustedServerError::Integration { integration, message } if integration == SOURCEPOINT_INTEGRATION_ID && message.contains("exceeds")), + "should identify Sourcepoint response overflow" + ); + } + assert_eq!( + client.reads.load(Ordering::Relaxed), + limit / TEST_CHUNK_SIZE + 1, + "should stop at EOF or the first overflowing chunk without draining the stream" + ); + } + } + } + }); + } + + #[test] + fn handle_passes_through_declared_oversize_without_reading() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for content_type in ["application/javascript", "text/html"] { + let client = Arc::new(StreamingHttpClient::new()); + let length = (MAX_REWRITE_BODY_SIZE + 1).to_string(); + client.stub.push_response_with_headers( + 200, + vec![b' '; MAX_REWRITE_BODY_SIZE as usize + 1], + vec![ + ("content-type", content_type), + ("content-length", &length), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/asset", + ), + ) + .await + .expect("should pass through a declared oversized response"); + + assert!( + matches!(response.body(), EdgeBody::Stream(_)), + "should retain the original stream" + ); + assert_eq!( + client.reads.load(Ordering::Relaxed), + 0, + "should not poll a declared oversized body" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_LENGTH), + Some(length.as_str()), + "should preserve the pass-through length" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve upstream cache policy" + ); + } + }); + } + + #[test] + fn handle_keeps_ineligible_responses_streaming() { + futures::executor::block_on(async { + let settings = create_test_settings(); + for (content_type, method, status, rewrite_sdk) in [ + ("application/json", Method::GET, 200, true), + ("text/css", Method::GET, 200, true), + ("image/png", Method::GET, 200, true), + ("application/javascript", Method::GET, 200, false), + ("text/html", Method::GET, 200, false), + ("application/javascript", Method::HEAD, 200, true), + ("text/html", Method::POST, 200, true), + ("application/javascript", Method::GET, 206, true), + ("text/html", Method::GET, 404, true), + ] { + let mut cfg = config(true); + cfg.rewrite_sdk = rewrite_sdk; + let integration = SourcepointIntegration::new(Arc::new(cfg)); + let client = Arc::new(StreamingHttpClient::new()); + client.stub.push_response_with_headers( + status, + b"unchanged".to_vec(), + vec![ + ("content-type", content_type), + ("content-encoding", "gzip"), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client.clone()); + let mut request = make_req( + method, + "https://publisher.example.com/integrations/sourcepoint/cdn/asset", + ); + set_req_header(&mut request, header::ACCEPT_ENCODING, "gzip, br"); + + let response = integration + .handle(&settings, &services, request) + .await + .expect("should pass through an ineligible response"); + + assert!( + matches!(response.body(), EdgeBody::Stream(_)), + "should leave ineligible bodies streaming" + ); + assert_eq!( + client.reads.load(Ordering::Relaxed), + 0, + "should not poll an ineligible body" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_ENCODING), + Some("gzip"), + "should preserve pass-through encoding" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve pass-through cache policy" + ); + assert_eq!( + response + .into_body() + .into_bytes_bounded(1024) + .await + .expect("should collect pass-through body") + .as_ref(), + b"unchanged", + "should preserve pass-through bytes" + ); + assert!( + client.stub.recorded_request_headers()[0] + .iter() + .any(|(name, value)| name == "accept-encoding" && value == "gzip, br"), + "should forward the client's encoding for non-script paths" + ); + } + }); + } + + #[test] + fn handle_rewrites_on_buffered_adapters_with_or_without_content_length() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for has_length in [false, true] { + let client = Arc::new(StubHttpClient::new()); + let input = format!(r#"var api="https://{SOURCEPOINT_CDN_HOST}/consent/tcfv2";"#); + let length = input.len().to_string(); + let mut headers = vec![ + ("content-type", "application/javascript"), + ("content-encoding", "identity"), + ("vary", "Accept-Encoding, Origin"), + ]; + if has_length { + headers.push(("content-length", &length)); + } + client.push_response_with_headers(200, input.into_bytes(), headers); + let services = build_services_with_http_client(client.clone()); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/wrapper.js", + ), + ) + .await + .expect("should rewrite a buffered response"); + + assert_eq!( + client.recorded_stream_response_flags(), + vec![false], + "should respect adapters without streaming support" + ); + assert!( + response.headers().get(header::CONTENT_LENGTH).is_none(), + "should not forward a stale upstream length" + ); + assert!( + response.headers().get(header::CONTENT_ENCODING).is_none(), + "should remove upstream encoding after rewriting" + ); + assert_eq!( + get_header_str(&response, header::VARY), + Some("Origin"), + "should remove only Accept-Encoding from Vary" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("public, max-age=3600"), + "should retain the static JavaScript cache policy" + ); + assert_eq!( + get_header_str(&response, header::CONTENT_TYPE), + Some("application/javascript; charset=utf-8"), + "should identify rewritten JavaScript" + ); + assert_eq!( + take_body_bytes(response), + br#"var api="/integrations/sourcepoint/cdn/consent/tcfv2";"#, + "should rewrite with either header state" + ); + } + }); + } + + #[test] + fn handle_site_data_requests_identity_and_preserves_dynamic_cache_policy() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + for (upstream_cache, forwarded_cookies, sets_cookie, expected_cache) in [ + (Some("no-store"), false, false, "no-store"), + ( + Some("private, max-age=60"), + true, + false, + "private, max-age=60", + ), + (None, true, false, "private, max-age=0"), + (None, false, false, "public, max-age=3600"), + ( + Some("public, max-age=3600"), + false, + true, + "private, no-store", + ), + ] { + let client = Arc::new(StreamingHttpClient::new()); + let input = format!(r#"var api="https://{SOURCEPOINT_CDN_HOST}/consent/tcfv2";"#); + let mut headers = vec![("content-type", "application/javascript")]; + if let Some(cache) = upstream_cache { + headers.push(("cache-control", cache)); + } + if sets_cookie { + headers.push(("set-cookie", "consentUUID=example; Path=/")); + } + client + .stub + .push_response_with_headers(200, input.into_bytes(), headers); + let services = build_services_with_http_client(client.clone()); + let mut request = make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/mms/v2/get_site_data?account_id=123", + ); + set_req_header(&mut request, header::ACCEPT_ENCODING, "gzip, br"); + if forwarded_cookies { + set_req_header(&mut request, header::COOKIE, "consentUUID=example"); + } + + let response = integration + .handle(&settings, &services, request) + .await + .expect("should rewrite dynamic site data"); + + assert!( + client.stub.recorded_request_headers()[0] + .iter() + .any(|(name, value)| name == "accept-encoding" && value == "identity"), + "should request uncompressed site data despite its extensionless path" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some(expected_cache), + "should use the dynamic endpoint's cache policy" + ); + assert_eq!( + take_body_bytes(response), + br#"var api="/integrations/sourcepoint/cdn/consent/tcfv2";"#, + "should rewrite unknown-length site data" + ); + } + }); + } + + #[test] + fn handle_preserves_invalid_utf8_bytes_and_headers() { + futures::executor::block_on(async { + let settings = create_test_settings(); + let integration = SourcepointIntegration::new(Arc::new(config(true))); + let client = Arc::new(StreamingHttpClient::new()); + let bytes = vec![0x1f, 0x8b, 0xff]; + client.stub.push_response_with_headers( + 200, + bytes.clone(), + vec![ + ("content-type", "application/javascript"), + ("content-encoding", "gzip"), + ("cache-control", "no-store"), + ], + ); + let services = build_services_with_http_client(client); + + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/wrapper.js", + ), + ) + .await + .expect("should retain non-UTF-8 content unchanged"); + + assert_eq!( + get_header_str(&response, header::CONTENT_ENCODING), + Some("gzip"), + "should preserve encoding when no rewrite occurs" + ); + assert_eq!( + get_header_str(&response, header::CACHE_CONTROL), + Some("no-store"), + "should preserve cache policy when no rewrite occurs" + ); + assert_eq!( + take_body_bytes(response), + bytes, + "should preserve invalid UTF-8 bytes" + ); + }); + } fn config(enabled: bool) -> SourcepointConfig { SourcepointConfig { @@ -1437,9 +1984,12 @@ mod tests { assert!(SourcepointIntegration::is_likely_javascript_path( "/module/sourcepoint.mjs" )); - assert!(!SourcepointIntegration::is_likely_javascript_path( + assert!(SourcepointIntegration::is_likely_javascript_path( "/mms/v2/get_site_data" )); + assert!(!SourcepointIntegration::is_likely_javascript_path( + "/mms/v2/get_site_data/other" + )); assert!(!SourcepointIntegration::is_likely_javascript_path( "/consent/tcfv2" )); @@ -1943,7 +2493,12 @@ mod tests { set_header(&mut response, header::CACHE_CONTROL, "no-store"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert_eq!(response.status(), StatusCode::OK); assert_eq!( @@ -1982,7 +2537,12 @@ mod tests { set_header(&mut response, header::CACHE_CONTROL, "public, max-age=3600"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert_eq!( get_header_str(&response, header::CACHE_CONTROL), @@ -2002,7 +2562,12 @@ mod tests { set_header(&mut response, header::VARY, "Accept-Encoding"); *response.body_mut() = EdgeBody::from(b"payload".to_vec()); - integration.rewrite_javascript_response(&mut response, "rewritten".to_string()); + integration.rewrite_javascript_response( + &mut response, + "rewritten".to_string(), + "/wrapper.js", + false, + ); assert!( response.headers().get(header::VARY).is_none(), diff --git a/docs/guide/integrations/sourcepoint.md b/docs/guide/integrations/sourcepoint.md index 7b9ce9435..22437dd2b 100644 --- a/docs/guide/integrations/sourcepoint.md +++ b/docs/guide/integrations/sourcepoint.md @@ -60,6 +60,21 @@ When `rewrite_sdk = true`, Trusted Server rewrites matching Sourcepoint URLs in If a publisher uses a Content Security Policy, `script-src` must allow the first-party Trusted Server host after rewriting. A policy that only allows `https://cdn.privacy-mgmt.com` can block the rewritten first-party script URLs. +## Response body rewriting + +With `rewrite_sdk = true`, successful `GET` responses with status 200 and a JavaScript or HTML content type are eligible for body rewriting. JavaScript CDN URLs and privacy-manager HTML root-relative `src` / `href` assets are routed through the first-party proxy. + +The input body limit is 5 MiB: + +- Bodies at or below the limit are rewritten, including chunked or HTTP/2 responses without `Content-Length`. +- A declared `Content-Length` above the limit skips rewriting and leaves the body unchanged. +- If collection exceeds the limit, Trusted Server stops reading and returns `502 Bad Gateway`. This also applies when `Content-Length` understates the size. The partial body is discarded, not returned to the browser. +- Other content types, disabled rewriting, and ineligible methods or statuses do not collect the body for rewriting. + +On Fastly, upstream responses remain streaming until the bounded collector reads them. It retains at most 5 MiB of input plus the current transport chunk, with additional bounded allocations for rewriting. Non-rewritten responses remain streaming. Adapters without streaming support still buffer upstream responses before this check; this limit is not an adapter-level memory guarantee on Cloudflare or Spin. + +Likely JavaScript and HTML paths, including `/mms/v2/get_site_data`, request `Accept-Encoding: identity`. Rewritten bodies discard the upstream `Content-Length` and `Content-Encoding`, and remove `Accept-Encoding` from `Vary`. Non-UTF-8 bodies pass through unchanged after bounded collection. + ## Runtime Config Rewriting Trusted Server also injects a `window._sp_` property trap that rewrites known Sourcepoint URL-bearing config fields (`baseEndpoint`, `mmsDomain`, `wrapperAPIOrigin`, `cmpOrigin`, and `metricUrl`). If Sourcepoint introduces another URL-bearing field and third-party `cdn.privacy-mgmt.com` requests remain after enabling this integration, report the missed field so it can be added to the trap. @@ -74,6 +89,8 @@ Trusted Server forwards only Sourcepoint's documented cookie names upstream, plu Responses that include `Set-Cookie` are forced to `Cache-Control: private, no-store` so cookie-bearing Sourcepoint traffic is never marked as publicly cacheable content by the proxy. +Rewritten `/mms/v2/get_site_data` responses preserve upstream cache policy rather than receiving the static JavaScript cache policy. When upstream omits `Cache-Control`, they use `private, max-age=0` if Sourcepoint cookies were forwarded, or the configured public TTL otherwise. Rewritten HTML uses the same policy. Other rewritten JavaScript retains the existing configured public TTL, except for responses that set cookies. + ## Notes - This version scopes the integration to `cdn.privacy-mgmt.com`. Additional Sourcepoint domains (e.g., `geo.privacymanager.io`) can be added later if publishers require them. From 713b27134d13262ee5728dff6ca92634f89d1305 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:02:14 -0700 Subject: [PATCH 069/230] Move the EdgeZero pin to the PR 381 branch head 4531aeec Upstream now parses the live Fastly resource-link types (`config`, `kv-store`, `secret-store`), documents the custom entry point migration, and moves the release verifier into edgezero-adapter, which adds serde, serde_json, sha2, and walkdir edges under its cli feature. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 285104697..e5cb56241 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,15 +1427,19 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ + "serde", + "serde_json", + "sha2 0.10.9", "toml", + "walkdir", ] [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1463,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1486,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-stream", @@ -1515,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-trait", @@ -1542,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "chrono", "clap", @@ -1567,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "anyhow", "async-compression", @@ -1598,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#c40ae8d0164e9adabe521bceeebd95ec492095dd" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" dependencies = [ "log", "proc-macro2", From 3c9d8c05a30d01ab194f6f76840ce3f76f66d850 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:16:51 -0700 Subject: [PATCH 070/230] Move the EdgeZero pin to the PR 381 branch head 7162b7e2 The two upstream commits since 4531aeec touch only the deploy action script and a demo lockfile. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e5cb56241..c9b545ca6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#4531aeecfb5cc7ba57997a6b907f402c90f88eed" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" dependencies = [ "log", "proc-macro2", From 9226cfbba9672d52d0c3d63fe0b754b08511f7d9 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:41:13 -0700 Subject: [PATCH 071/230] Move the EdgeZero pin to the PR 381 branch head 8b506782 The two upstream commits since 7162b7e2 only reorganize the EdgeZero CI test workflows. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c9b545ca6..283e99dc0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#7162b7e2b2a7d6e2dba7e02488881e6c873143b7" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" dependencies = [ "log", "proc-macro2", From 7d212b7d3182162e881e7208ac37abf92f945188 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 15:31:21 -0700 Subject: [PATCH 072/230] Move the EdgeZero pin to the PR 381 branch head fd45db1f Upstream replaces the Compute-unsupported version diff snapshot with per-collection reads and drops the parent service ID comparison from the staged-source guard and staging rollback. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 283e99dc0..742351842 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#8b5067821b0556c5e8df6d06a6bc2a2f07ec6c78" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" dependencies = [ "log", "proc-macro2", From c7483aa9245e3dea2bf16d197f40d2ca7b08b8a3 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 16:21:33 -0700 Subject: [PATCH 073/230] Move the EdgeZero pin to the PR 381 branch head 6258b6b2 The only upstream change since fd45db1f corrects the Google Pub/Sub logging snapshot path to the Fastly API's `logging/pubsub` and adds `logentries` to the swept endpoint kinds. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 742351842..5c1e1a412 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#fd45db1f8dbac02759333ff2364abd1c546a5722" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" dependencies = [ "log", "proc-macro2", From 9380c12aff3b22be401a41d13302506c00f34747 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 17:03:58 -0700 Subject: [PATCH 074/230] Move the EdgeZero pin to the PR 381 branch head bb4e0040 The only upstream change since 6258b6b2 switches the Fastly deploy to the current `service resource-link` and `service version` command spellings, removing the CLI deprecation notices. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 5c1e1a412..d655fc4ed 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#6258b6b2f9b577079eeb78e5d2ecfe629320a262" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" dependencies = [ "log", "proc-macro2", From b1d41c541e6e089764ef4b9bce404f56b3fd5e3b Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:01:20 -0700 Subject: [PATCH 075/230] Move the EdgeZero pin to the PR 381 branch head cf9a96a0 The only crate change since bb4e0040 canonicalizes the Fastly configuration snapshot by sorting object keys before ordering rows, so the drift check no longer fails on Fastly's random JSON key order. No public API used by Trusted Server changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d655fc4ed..229bd7191 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#bb4e0040e4f2af5a70e061660438099e3e12c691" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" dependencies = [ "log", "proc-macro2", From 429503979dc81a16e19e1b27b48eae79cf8c6c17 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:41:47 -0700 Subject: [PATCH 076/230] Move the EdgeZero pin to the PR 381 branch head 119fbf80 The only upstream commit since cf9a96a0 adds a Cargo target cache to EdgeZero's CI. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 229bd7191..c2d6c9f17 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#cf9a96a03d9bb3a7e902d2341185e7def6a00b00" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" dependencies = [ "log", "proc-macro2", From 1e6fc674f27e4110866776469d9d197c9e49550d Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:39:33 -0700 Subject: [PATCH 077/230] Watch only bundle inputs in the trusted-server-js build script The build script asked Cargo to rerun on every file under lib/, including node_modules, which emitted 33,932 rerun-if-changed directives in a 2.8 MB output that Cargo re-checked on every build and invalidated the crate after any npm install. Watch the TypeScript sources, package manifests, and bundler configuration explicitly instead. Measured on an Apple Silicon host with a warm cache: a no-op `cargo check -p trusted-server-js` drops from 1.1s to 0.15s, the script emits 7 directives, touching a node_modules file no longer reruns it, and touching lib/src or package-lock.json still does. --- crates/trusted-server-js/build.rs | 40 ++++++++++++------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index ba6cd88f2..519f0bd71 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -15,9 +15,21 @@ use build_print::{info, warn}; use sha2::{Digest as _, Sha256}; fn main() { - // Rebuild if TS sources change (belt-and-suspenders): enumerate every file under lib/ - println!("cargo:rerun-if-changed=lib"); - watch_dir_recursively(Path::new("lib")); + // Rebuild when the TypeScript sources or anything that shapes the bundle + // output changes. `lib` as a whole is deliberately not watched: it holds + // `node_modules`, and enumerating that tree emitted tens of thousands of + // directives that Cargo re-checked on every build. + for watched in [ + "lib/src", + "lib/package.json", + "lib/package-lock.json", + "lib/build-all.mjs", + "lib/build-prebid-external.mjs", + "lib/tsconfig.json", + "lib/vite.config.ts", + ] { + println!("cargo:rerun-if-changed={watched}"); + } // Allow opt-out or force via env let skip = env::var("TSJS_SKIP_BUILD").is_ok_and(|value| value == "1"); @@ -197,25 +209,3 @@ fn copy_bundle(filename: &str, required: bool, dist_dir: &Path, out_dir: &Path) fs::write(&target, "").expect("should write optional empty bundle placeholder"); } - -fn watch_dir_recursively(root: &Path) { - if !root.exists() { - return; - } - let mut stack = vec![root.to_path_buf()]; - while let Some(dir) = stack.pop() { - let Ok(read) = fs::read_dir(&dir) else { - continue; - }; - for entry in read.flatten() { - let path = entry.path(); - // Always ask Cargo to rerun if this path changes - if let Some(path_str) = path.to_str() { - println!("cargo:rerun-if-changed={path_str}"); - } - if path.is_dir() { - stack.push(path); - } - } - } -} From ae47b4350b0297be935f258c9e9d81d22426bf4c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 18 Sep 2026 18:50:52 -0700 Subject: [PATCH 078/230] Move the EdgeZero pin to the PR 381 branch head 5878eb74 The only upstream commit since 119fbf80 fixes a shellcheck finding in EdgeZero's CI cache action. No crate source changed, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c2d6c9f17..25197f4e6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#119fbf803f77e26fe23cbc595f00db408e5460a8" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" dependencies = [ "log", "proc-macro2", From e3f371f8cb102d4b7f713d375f8587a34c6f01fd Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Sat, 19 Sep 2026 11:34:08 +0530 Subject: [PATCH 079/230] Reconcile mobile trace design contracts --- ...-mobile-ad-render-trace-endpoint-design.md | 419 +++++++++++++----- 1 file changed, 302 insertions(+), 117 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 092466a6d..5d86b9092 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -14,7 +14,8 @@ ## 1. Summary -Add a deployment-controlled, public, privacy-safe `GET /_ts/trace` page for a +Add a deployment-controlled, privacy-safe `GET /_ts/trace` page, public subject +to operator authentication rules, for a mobile end user who needs to reproduce an ad-rendering problem and give support an exportable diagnostic report. Visiting the page is read-only. The user intentionally enables or ends tracing with a same-origin POST action. @@ -44,9 +45,10 @@ Issue #1050 names three required data groups: 3. End-user cookie information. The title additionally establishes two product constraints: the experience is -for a mobile user, and it is reached through an endpoint. A mobile user should -not need browser developer tools, Basic Authentication, a copied trace ID, or a -second copy of the affected page URL. +for a mobile user, and it is reached through an endpoint. On a deployment whose +authentication rules leave trace routes public, a mobile +user should not need browser developer tools, credentials, a copied trace ID, +or a second copy of the affected page URL. A standalone request cannot know what occurred in a previous document. Exact render evidence exists only while the publisher page is running in the browser. @@ -117,10 +119,13 @@ diagnostic token connects those layers. ### 5.1 Public, redacted endpoint with intentional activation -`/_ts/trace` is public when explicitly enabled by deployment configuration. It -is not placed under `/_ts/admin`, because the intended user is a layperson on a -phone and the existing Basic Authentication flow is unsuitable for that -journey. +`/_ts/trace` is available when explicitly enabled by deployment configuration +and is public only when no operator authentication rule covers it. It is not +placed under `/_ts/admin`, because the intended user is a layperson on a phone. +Existing Basic Authentication rules still apply to every trace path, including +assets and actions; the early dispatcher must not carve out an exemption. +Operators offering the credential-free journey must scope authentication to +the paths they intend to protect, for example `^/_ts/admin` (see section 12.5). Public access is safe only because both the page and export use a strict allowlist. The activation cookie is a feature toggle, not authentication. No @@ -141,19 +146,51 @@ executing on the publisher origin. The endpoint reuses `__Host-ts-console` and the existing GPT diagnostics activation semantics rather than creating a second `ts-trace` session. The cookie remains host-only, `Secure`, `HttpOnly`, and `SameSite=Lax`. -`POST /_ts/trace/enable` sets it with a fixed 30-minute `Max-Age` and does not -refresh that lifetime on publisher requests; `POST /_ts/trace/end` clears it. -Neither action accepts state-changing query parameters. The shorter endpoint -lifetime bounds accidental private/no-store operation if a user forgets to end -tracing; the existing technical query flow keeps its existing session-cookie -semantics. - -The new trace capture is active only when both `trace_page_enabled = true` and -the request carries exactly one valid diagnostics cookie. The shared cookie by +Both `POST /_ts/trace/enable` and the existing `?ts_console=1` writer must +set `Max-Age=1800` through one shared cookie policy, including when +`trace_page_enabled` is false. This explicitly changes the technical query flow +from a browser-session cookie to a 30-minute cookie. An explicit activation +restarts the 30-minute lifetime; ordinary publisher requests do not refresh it. +`POST /_ts/trace/end` and `?ts_console=0` clear the same cookie. Neither POST +action accepts state-changing query parameters. A later query activation must +never replace the bounded cookie with a session cookie. + +This bounds browser-managed persistence from the most recent activation by an +updated writer, not total session duration or server-enforced authorization. +Pre-existing session cookies carry no expiry in the request and cannot be +retroactively aged; users with those cookies must end or re-enable diagnostics +to adopt the new lifetime. Cookie expiry also does not unload a running page +or clear an existing report; report expiry is defined separately in section 9.5. + +The base trace gate requires both `trace_page_enabled = true` and exactly one +valid incoming diagnostics cookie, inspected before cookie sanitation. +Publisher-document tracing additionally requires the existing effective +`GptDiagnosticsRequestDecision.active` decision. Query disable or invalid +directives, prefetches, bots, and other ineligible navigations therefore +suppress document trace context, auction evidence, and browser activation even +when an incoming cookie is valid. A query activation without an incoming valid +cookie enables the existing console; trace capture starts on the next eligible +reload carrying that cookie. Page-bids and `/auction` requests use the base +configuration-plus-cookie gate without requiring a document-navigation +decision. References below to the trace gate include the additional effective +diagnostics decision for publisher documents. The shared cookie by itself continues to activate the existing TS Console but does not authorize trace tokens, server-auction projection, trace response extensions, or correlation sidecars on a deployment whose trace page is disabled. This makes -the configuration flag the disclosure and rollback boundary. +the configuration flag the disclosure and rollback boundary for new requests. + +Core exposes the evaluated document trace gate as the literal boolean +`window.__tsjs_trace_active` before TSJS initializes on the publisher document; +it is true only when the base gate and effective document diagnostics decision +are both active. TSJS requires +`window.__tsjs_trace_active === true` before minting slot tokens, retaining +request mappings, installing trace listeners, collecting transport evidence, +emitting sidecars, or offering the trace handoff action. Missing or non-boolean +values mean inactive; `window.__tsjs_gpt_diagnostics_active` alone is +insufficient. Core independently rechecks the gate on each request and never +trusts the browser flag as authorization. Configuration changes or cookie expiry +cannot revoke code already loaded in a document; rollback takes effect on its +next reload, and subsequent server requests stop returning evidence immediately. ### 5.3 Browser-local, explicit handoff @@ -324,7 +361,7 @@ the report. ```text GET /_ts/trace | - |-- early reserved-route classifier terminates locally + |-- early reserved-route classifier applies configured auth, then terminates locally |-- HTML explains forward reproduction; no state mutation v POST /_ts/trace/enable after explicit user action @@ -411,7 +448,7 @@ enabled = true trace_page_enabled = false ``` -Rules: +Rules (route responses below apply after configured authentication): - `trace_page_enabled = true` requires `enabled = true`; invalid combinations fail configuration validation. @@ -481,9 +518,15 @@ Every adapter implements the following order: 1. Parse the method, canonical host/origin, path, query, and bounded headers required for route safety. 2. Classify an exact Trusted Server reserved path. -3. For a trace path, terminate locally after only trace-specific validation and - bounded request-context inspection, including an optional read-only platform - geo lookup used solely for the displayed setup request. +3. For a trace path, enforce the existing configured Basic Authentication + rules before any trace response, cookie mutation, body inspection, or setup + context projection. A matching rule challenges missing/invalid credentials + locally with the ordinary `401` and `WWW-Authenticate` behavior. After + authentication succeeds or no rule matches, apply the trace-specific + validation and bounded request-context inspection, including an optional + read-only platform geo lookup, and terminate locally. This applies to the + entire reserved namespace, assets, unsupported methods, and disabled routes; + the route statuses below authentication are never an auth exemption. 4. For all other paths, continue through the adapter's ordinary event context, authentication, request filters, geo enrichment, EC/EID processing, named routes, auction handling, telemetry, and publisher fallback. @@ -496,8 +539,9 @@ ordinary named-route registration alone does not satisfy this contract. After an enable or end POST succeeds, the client performs a no-store state GET. It claims `Tracing is on — cookie observed by server` only when that separate -request reports active, and `Tracing is off — cookie absent on server request` -only when it reports inactive. A mismatch or failed verification is +request reports active, and `Tracing is off — no valid diagnostics session +observed` only when it reports inactive. An inactive result covers absent, +invalid, duplicate, or uninspectable cookies; it does not prove cookie absence. A mismatch or failed verification is `Activation unconfirmed` or `Deactivation unconfirmed` and offers an idempotent retry. These are server-observation statements, not proof that browser state is authentic: same-origin service workers can forge or suppress the whole exchange. @@ -508,7 +552,8 @@ the uncached shell, state, and action routes being disabled; inert cached assets alone cannot activate tracing or access a report page. The current `?ts_console=1` and `?ts_console=0` activation flow remains -supported for technical users. Both activation surfaces drive the same cookie +supported for technical users with the shared 30-minute cookie policy in +section 5.2. Both activation surfaces drive the same cookie and runtime; they must not create two concurrent diagnostic modes. That pre-existing query flow has its existing top-level-navigation activation risk; #1050 neither expands it to the new trace GET nor claims to remediate it. @@ -517,8 +562,8 @@ pre-existing query flow has its existing top-level-navigation activation risk; ### 9.1 Request context -The server injects one immutable `TraceRequestContextV1` into active diagnostic -documents: +The server injects one immutable `TraceRequestContextV1` only into documents +that satisfy the applicable trace gate in section 5.2: ```text TraceRequestContextV1 @@ -546,8 +591,12 @@ The request-context envelope intentionally contains no page URL, path, referrer, query, or fragment. During the field-by-field trace projection, `GptDiagnosticsExportV1.page.origin` is retained after validation and its `pathname` is replaced with the literal `/[redacted]`. The trace viewer accepts -only that literal. Version one therefore does not store or export an exact page -path. Any future route-template policy requires a new schema and privacy review +only that literal. The projection also omits `slotElementId` and `adUnitPath` +from slots and `slotElementId` from callback and attribution issues, because +these values can embed the same page path. They are not hashed or truncated. +Numbered slots and request cycles retain grouping and exact-token correlation. +Version one therefore does not intentionally retain an exact page path in any +of these source fields. Any future route-template policy requires a new schema and privacy review because paths can contain accounts, emails, preview tokens, and other secrets. `masked_client_ip` uses a deterministic display-only mask for the current @@ -661,13 +710,26 @@ projection sourced only from `GptDiagnosticsExportV1`. It contains: - `schema_version: 1` and `source_schema_version: 1`; - the source `capturedAt` value; - `page.origin` after validation and `page.pathname` fixed to `/[redacted]`; -- field-for-field allowlisted copies of the current v1 slots, requests, - callback issues, attribution issues, coverage, and metadata, subject to the - bounds and truncation below. +- explicit field-by-field projections of current v1 slots, requests, callback + issues, attribution issues, coverage, and metadata, subject to the following + exclusions and the bounds/truncation below. + +The trace schema omits the entire request-cycle `adManager` object, including +`lineItemId`, `creativeId`, `campaignId`, `advertiserId`, +`sourceAgnosticLineItemId`, `sourceAgnosticCreativeId`, `yieldGroupIds`, and +`companyIds`, and omits `previousCreativeId`. Derived `responseClass` and +`creativeChanged` facts remain eligible without their underlying identifiers. +It also omits slot `slotElementId` and `adUnitPath` and callback/attribution +issue `slotElementId`. These properties are forbidden in the trace schema, +not optional passthrough fields: the builder never copies them and the viewer +rejects a stored report that supplies them. The remaining current v1 fields +retain their source meaning and require explicit allowlisting; future source +fields are not inherited automatically. `trustedServerAuctionId`, when +present, must satisfy the diagnostic auction-token contract in section 9.4. It is deliberately not named or represented as `GptDiagnosticsExportV1`, -because the fixed pathname and trace-level bounds change the source field -semantics. TS Console continues to own the source schema; the trace envelope +because redaction, excluded identifiers, and trace-level bounds change the +source field semantics. TS Console continues to own the source schema; the trace envelope owns its public projection and transport. The initial compatibility matrix is exactly `TraceReportV1`, `TraceAuctionEvidenceV1`, `TraceSlotCorrelationV1`, and `TraceGptDiagnosticsV1`, with the GPT projection sourced from @@ -678,13 +740,17 @@ an additive source compatibility change and, if the public projection changes, a new trace-envelope version. `auction_coverage.capture_status` describes only what reached the browser -collector: `not_observed` means no valid server-auction record arrived, not that -no server auction ran. `partial` requires at least one retained record plus a -projection, transport, validation, or eviction issue; `unavailable` requires no -retained records plus a known projection, transport, or validation issue; and +collector: `not_observed` means no valid server-auction record arrived and no +capture issue is known, not that no server auction ran. `partial` requires at +least one retained record plus a projection, transport, validation, or eviction +issue; `unavailable` requires no retained records plus a known projection, +transport, validation, or eviction issue; and `complete` requires at least one retained record without those capture issues. `correlation_unavailable` and `external_client_side_unobservable` describe interpretation limits and do not change an otherwise complete capture status. +Evicting all received records therefore yields `unavailable` with +`record_evicted`, never `not_observed`. Recompute status after both in-memory +eviction and snapshot size truncation. The issue array is deduplicated, sorted in enum order, bounded to 16 values, and contains no error text. @@ -739,24 +805,29 @@ The model has deliberately lower cardinality and sensitivity than the existing telemetry and OpenRTB objects: - `diagnostic_auction_id` is a fresh opaque `ts-auc-...` correlation token. When - trace capture is active under the two-part gate in section 5.2, it is minted + trace capture is active under the applicable trace gate in section 5.2, it is minted once when an eligible auction is observed, before dispatch, and is retained for zero-bid, skipped, dispatch-failed, execution-failed, and abandoned outcomes. It is never `AuctionRequest.id`, the telemetry UUID, a provider request ID, or an identifier joinable to user-bearing logs. -- Auction and slot tokens are the fixed prefixes `ts-auc-` and `ts-slot-` - followed by a canonical lowercase hyphenated UUID v4. Validators reject every - other shape; tokens are not silently shortened or normalized. +- Auction tokens are `ts-auc-` followed by a lowercase UUID v4 in the existing + producer's 32-hex-digit unhyphenated form (`Uuid::new_v4().simple()`). Slot + tokens are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, + matching `crypto.randomUUID()`. Both validators enforce UUID version 4 and + the RFC variant and reject every other shape. Tokens are compared verbatim, + never normalized during validation or correlation; no producer format change + is required for the existing GPT auction opportunity marker. - `slot_number` is a one-based ordinal over the exact post-conversion `AuctionRequest.slots` sequence observed by orchestration. It is display-only and is never used to map a response back to pre-conversion client input. `slot_ref` is a fresh auction-local opaque token carried with that slot. Core creates it for initial-navigation and SPA auctions. For a TSJS `/auction` - request, TSJS creates it only after `buildAdRequest` has finished grouping and + request, TSJS creates it only when `window.__tsjs_trace_active === true` and + only after `buildAdRequest` has finished grouping and deduplicating the final `adUnits` array, attaches it to that exact outgoing unit as `adUnits[].ext.trusted_server.trace_slot_ref`, and retains the request-scoped token-to-unit mapping. Core accepts that member only under the - two-part trace gate, validates and echoes the token for accepted converted + applicable trace gate, validates and echoes the token for accepted converted slots, and strips it before every provider or mediator request. A missing or invalid client token causes core to mint a server token with no browser correlation; it never changes ordinary auction acceptance. TSJS uses @@ -812,8 +883,8 @@ correlation failed, the viewer shows `Server auction evidence unavailable` or #### 9.4.1 Live transport and correlation -Evidence is transported only while `trace_page_enabled` is true and the -diagnostics cookie is valid. Every response carrying it is terminally +Evidence is transported only when the applicable trace gate in section 5.2 +is active, including the effective diagnostics decision for publisher documents. Every response carrying it is terminally `private, no-store`: ```text @@ -836,7 +907,7 @@ object TSJS already consumes: AuctionSlot.ext.trusted_server.trace_slot_ref: string ``` -Core adds that optional nested member only under the two-part trace gate. It +Core adds that optional nested member only under the applicable trace gate. It assigns the token while constructing the request-scoped slot definitions and threads the same token into the corresponding `AuctionRequest` observation, so neither side needs to recover the relationship from an ordinal or raw slot ID. @@ -845,13 +916,15 @@ ordering, and bid-map keys remain unchanged. The extension is absent when the gate is false and is never copied into `TraceAuctionEvidenceV1` except as its already-allowlisted opaque `slot_ref`. -TSJS accepts a slot extension only when its canonical token occurs exactly once +When valid auction evidence is supplied, TSJS accepts a slot extension only +when its canonical token occurs exactly once in both the delivered slot list and the matching auction evidence. A missing, malformed, duplicate, or conflicting token prevents only that sidecar join, adds `evidence_validation_failed` and `correlation_unavailable`, and does not drop, reorder, or mutate the ordinary slot or bid. TSJS reads no other extension property. This validation occurs before the slot is handed to the existing GPT -initialization path. +initialization path. An absent optional transport member does not trigger +missing-token validation; no sidecar is emitted and no capture issue is added. The three transport call shapes are exact v1 contracts: @@ -876,8 +949,9 @@ accepts only the exact optional top-level member and preserves its existing `slots` and `bids` behavior. These trace members never become required for a successful advertising response. -When the existing GPT recorder consumes the matching Trusted Server opportunity -for a concrete request cycle, it emits this trace-owned sidecar: +For initial-navigation SSAT and SPA page-bids only, when the existing GPT +recorder consumes the matching Trusted Server opportunity for a concrete request +cycle, it emits this trace-owned sidecar: ```text TraceSlotCorrelationV1 @@ -895,6 +969,19 @@ opaque server tokens and the concrete GPT cycle are present. The current `GptDiagnosticsExportV1` remains unchanged and the sidecar contains no slot element ID or ad-unit path. +Version one does not correlate either `/auction` caller to a GPT cycle. Prebid +refresh records only `prebid_refresh` intent and has no existing token-bearing +opportunity binding; passing its tokens through `recordTrustedServerOpportunity` +would incorrectly introduce `trusted_server_direct` attribution. The direct +`requestAds` caller renders outside GPT and supplies no GPT request cycle. Both +callers retain server evidence and the request-unit mapping, but emit no +`TraceSlotCorrelationV1`. The viewer displays API evidence independently with +`correlation_unavailable`; this interpretation limit does not downgrade an +otherwise complete server capture. A sidecar referencing an `auction_api` +record is invalid in v1. A future Prebid join requires a separately designed +association between request-unit tokens and exact GPT slot/request identities +that preserves existing request-path attribution. + - **Initial-navigation SSAT:** core builds the evidence when the split auction is collected at the held body tail. It injects the script-safe public model beside the winning-bid map before initial ad initialization; the corresponding @@ -908,12 +995,17 @@ element ID or ad-unit path. `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both the envelope and slot extensions are absent when the trace gate is inactive. - **Trusted Server `/auction` API:** the existing OpenRTB response adds a - namespaced `ext.trusted_server.trace_auction` transport envelope only for an - active diagnostics request. After producing the final grouped `AdRequest`, - both TSJS callers assign one fresh token to each outgoing unit and retain that - exact request-scoped mapping. They validate the echoed evidence and record it + namespaced `ext.trusted_server.trace_auction` transport envelope only for a + request satisfying the base trace gate in section 5.2. After producing the + final grouped `AdRequest`, + both TSJS callers check `window.__tsjs_trace_active === true` before + assigning one fresh token to each outgoing unit and retaining that exact + request-scoped mapping. An inactive caller creates neither tokens nor pending + transport records and installs no trace-specific timeout/error hooks. Active + callers validate the echoed evidence and record it before parsing bids. A converted or skipped unit therefore cannot shift - another slot's correlation. The response member does not replace or expose + another request unit's server-evidence association. This mapping does not + imply a GPT-cycle join. The response member does not replace or expose the existing orchestrator extension, and the trace projection must not copy that extension's provider names, bidder names, metadata, price, creative IDs, domains, or markup. HTTP/transport failures with no readable response are @@ -933,8 +1025,9 @@ transport outcome, so it removes the marker and leaves evidence `not_observed` rather than inventing a failure. These hooks collect only bounded categories and opaque tokens, never XHR error text or response bodies. -The diagnostic auction token is also attached to the existing GPT opportunity -marker, and the opaque slot token is carried through the corresponding +For SSAT and SPA page-bids, the diagnostic auction token is also attached to +the existing GPT opportunity marker, and the opaque slot token is carried +through the corresponding winning-bid/slot initialization path. The numeric ordinal is never a correlation key. The viewer joins a server slot to a GPT cycle only when one validated `TraceSlotCorrelationV1` exactly matches both tokens and the exported @@ -945,9 +1038,10 @@ timestamps, implicit array position, ad-unit path, or a best-effort heuristic. TSJS retains at most the newest 16 validated server-auction records and 128 correlation sidecars in memory. It increments checked eviction counters for older records; the snapshot adds those counts to the matching truncation fields -and emits `record_evicted` with `partial`. It performs no storage write until +and emits `record_evicted`, with `partial` if server records remain or +`unavailable` if none remain after snapshot truncation. It performs no storage write until the explicit snapshot action. -Requests for which either side of the trace-capture gate is false do not mint +Requests that fail the applicable trace gate do not mint trace tokens, build trace evidence, add response members, emit sidecars, or install auction-evidence listeners. @@ -993,32 +1087,40 @@ failure is handled even when the report is below the application limit. Runtime limits are part of the v1 contract: -| Value | Limit | -| --------------------------------------------- | ------------------------------------------------------------ | -| Container nesting | 8 levels | -| Server auctions | 16 | -| Slot correlations | 128 | -| Provider calls | 16 per server auction | -| Auction slots | 64 per server auction | -| Auction coverage issues | 16 | -| Slots | 64 | -| Request cycles | 10 per slot before total-size truncation | -| Callback issues | 128 | -| Attribution issues | 128 | -| Requested slot sizes | 16 per cycle | -| Ad Manager yield-group or company IDs | 8 of each per cycle | -| Creative-failure enums | 16 per cycle | -| Origin | 255 UTF-8 bytes | -| GPT pathname in trace projection | Exact literal `/[redacted]` | -| Slot element ID and ad-unit path | 512 UTF-8 bytes each | -| Trusted Server auction ID and callback reason | 256 UTF-8 bytes each | -| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | -| Any other string | 128 UTF-8 bytes | -| Enum | Exact documented value only | -| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | -| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | -| Visibility percentage | Finite number from 0 through 100 | -| Slot dimension | Finite integer from 1 through 100,000 | +| Value | Limit | +| ----------------------------------------------- | ------------------------------------------------------------ | +| Container nesting | 8 levels | +| Server auctions | 16 | +| Slot correlations | 128 | +| Provider calls | 16 per server auction | +| Auction slots | 64 per server auction | +| Auction coverage issues | 16 | +| Slots | 64 | +| Request cycles | 10 per slot before total-size truncation | +| Callback issues | 128 | +| Attribution issues | 128 | +| Requested slot sizes | 16 per cycle | +| Creative-failure enums | 16 per cycle | +| Origin | 255 UTF-8 bytes | +| GPT pathname in trace projection | Exact literal `/[redacted]` | +| Slot element ID and ad-unit path | Forbidden, including callback/attribution issue copies | +| Trusted Server auction ID | Exact diagnostic auction-token shape | +| Callback reason | Exact documented value only | +| Diagnostic auction ID and opaque slot ref | 128 UTF-8 bytes each | +| Any other string | 128 UTF-8 bytes | +| Enum | Exact documented value only | +| Identifier, sequence, or counter | Finite safe integer from 0 through `Number.MAX_SAFE_INTEGER` | +| Browser-relative timestamp or duration | Finite number from 0 through `Number.MAX_SAFE_INTEGER` | +| Visibility percentage | Finite number from 0 through 100 | +| Requested or selected creative dimension | Finite integer from 1 through 100,000 | +| GPT-reported fill dimension (`size`) | Finite integer from 1 through 100,000 | +| Observed CSS box dimension (`observedSlotSize`) | Finite integer from 0 through 100,000 | + +`observedSlotSize` preserves zero dimensions, including `[0, 0]` for a +hidden or collapsed element after a filled GPT render. Zero is a measured box +size, not missing data or evidence of an empty GPT response; do not omit it or +change `isEmpty`. Positive dimensions remain required for requested and +selected creative sizes and GPT-reported fill sizes. Every accepted string must be valid Unicode and must not contain C0/C1 control characters or bidirectional override/isolate controls. This applies to browser @@ -1029,8 +1131,7 @@ invalid source value rather than stringifying it. Server auctions are already bounded by core; the browser rejects an invalid inner model rather than truncating it. The builder retains the newest 16 server auctions in observation order, the newest 128 correlations in recorder emission order, and only the -first documented number of GPT requested sizes, yield-group IDs, company IDs, -and creative failure enums. It records each discard in +first documented number of GPT requested sizes and creative failure enums. It records each discard in `omitted_server_auctions`, `omitted_slot_correlations`, or `omitted_nested_values`; strings are never silently shortened. It then measures the complete compact UTF-8 storage wrapper. If it exceeds 512 KiB, it removes @@ -1180,7 +1281,9 @@ Forbidden data includes: - EC IDs, EIDs, bidder user IDs, provider/bidder/seat names, and consent strings. - Unmasked client IP. -- Query strings and fragments. +- Exact page paths, slot element IDs, ad-unit paths (including issue copies), + query strings, and fragments. +- The entire GPT `adManager` identity object and `previousCreativeId`. - Fastly or internal request identifiers that can join to user-bearing logs. - Internal `AuctionRequest.id`. - Bid requests/responses, bid prices/currency, losing-bid payloads, provider @@ -1233,9 +1336,11 @@ hash and a corresponding CSP change. Validated report strings enter the document through `textContent` or equivalent DOM properties, never `innerHTML`. The JS asset uses `application/javascript; charset=utf-8`; the CSS asset uses -`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff`, -`Cache-Control: public, max-age=31536000, immutable`, and a strong ETag derived -from their build bytes. They accept no dynamic input. `script-src 'self'` is an +`text/css; charset=utf-8`. Both send `X-Content-Type-Options: nosniff` and a strong ETag derived +from their build bytes. They use +`Cache-Control: public, max-age=31536000, immutable` only when no configured +authentication rule covers the asset; otherwise they use `private, no-store` +as specified in section 12.5. They accept no dynamic input. `script-src 'self'` is an origin-level CSP permission, not a path restriction; same-origin script interference remains inside the stated trust limitation. @@ -1249,9 +1354,29 @@ The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content publicly cacheable. +### 12.5 Operator authentication policy + +Trace routing preserves the existing `auth.rs` namespace contract: every +matching operator Basic Authentication rule is enforced, including `^/_ts` or +`^/`. Classification may run early, but does not authorize a request. Only +authentication is factored ahead of trace handling; ordinary event, identity, +filter, and auction processing stays outside trace routes. A challenge exposes +no setup context, changes no cookie, and is terminally private/no-store. + +The credential-free mobile journey requires deployment rules that leave the +trace namespace public. Document this beside `trace_page_enabled`; do not +silently narrow an operator rule. On origin requests, authentication precedes +disabled-route `404` and unsupported-method `405` responses as well. Asset +responses covered by an authentication rule must use `private, no-store` +instead of public immutable caching. Previously public cached inert assets may +remain available, but contain no report data and cannot bypass the uncached +shell or actions. + ## 13. Failure handling -- Disabled route: local privacy-safe `404`. +- Configured authentication failure: local private/no-store `401` challenge + before trace handling, including on disabled routes. +- Disabled route after authentication: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. - Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. @@ -1265,9 +1390,14 @@ publicly cacheable. - Initial-navigation evidence cannot be injected because the body tail is not reached: preserve publisher delivery. Because the browser received no safe marker, display `not_observed` rather than claiming a known server failure. -- Page-bids or `/auction` evidence is absent or rejected by its strict client - validator: parse the ordinary bid response exactly as before, discard the - diagnostic member, and show an unmatched/invalid-evidence coverage category. +- A successful page-bids or `/auction` response omits the optional evidence + member: parse ordinary bids unchanged and add no capture issue. With no other + records or known capture issues, coverage is `not_observed`; absence does not + reset previously collected evidence or failures. +- A supplied evidence member fails strict validation: discard that member, + parse ordinary bids unchanged, and add `evidence_validation_failed`. Coverage + is `partial` when valid records remain or `unavailable` when none remain, + following section 9.3. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. - Storage unavailable or quota exceeded after a valid bounded report exists: @@ -1306,15 +1436,24 @@ results, never a prerequisite for returning them. console cookie still enables the existing console but never mints trace tokens, builds auction evidence, adds response extensions, or emits correlation sidecars. +- With a valid incoming cookie, `?ts_console=0`, invalid/duplicate directives, + prefetches, bots, and other ineligible navigations suppress all document trace + capture through the effective diagnostics decision. Query enable without a + cookie activates only the console until the next eligible cookie-bearing + reload. Page-bids and `/auction` use the base gate without navigation checks. - Exact reserved-route classification, canonical-path, query, method, encoded path, and fallback behavior. - Exact versioned asset routes are local and contain no dynamic data; lookalike asset paths never reach the publisher origin. - Same-origin POST validation, cross-site/missing signal rejection, cookie - set/clear attributes, fixed 30-minute endpoint activation without request - refresh, and idempotent enable/end behavior. + set/clear attributes, shared `Max-Age=1800` for endpoint and query activation + without ordinary-request refresh, and idempotent enable/end behavior. Cover + enable then query activation, query then enable, repeated explicit activation, + clearing through either surface, and the pre-existing session-cookie caveat. - Enable/end success requires a separate state request to observe the resulting cookie; failed and mismatched verification never displays confirmed state. + Absent, invalid, duplicate, and uninspectable cookies all report inactive, + with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, the first unexpected body byte, and the two-second deadline without an unbounded read. @@ -1330,7 +1469,8 @@ results, never a prerequisite for returning them. API call sites to the exact public source enums without using a browser hint. - The diagnostic auction token is minted before dispatch and remains identical across completed, zero-bid, skipped, failed, and abandoned evidence and the - corresponding browser opportunity marker. It never equals or contains the + corresponding SSAT/SPA browser opportunity marker. API tokens remain + server-evidence identifiers without GPT sidecars. No token equals or contains the internal auction ID or telemetry UUID. - Server-auction projection covers every terminal status/reason mapping, auction-local total duration, provider role/status/duration/count, per-slot @@ -1350,13 +1490,15 @@ results, never a prerequisite for returning them. associated across grouped multi-bidder units, duplicate codes, skipped non-banner units, and mixed accepted/filtered inputs. Numeric ordinals are never used for client correlation. -- Token tests cover canonical UUID-v4 shape, missing Web Crypto, malformed or +- Token tests cover the existing simple auction UUID-v4 shape and hyphenated + slot UUID-v4 shape, exact producer-to-sidecar joins, missing Web Crypto, malformed or duplicate request extensions, the disabled trace gate, and proof that invalid tokens neither fail nor otherwise alter the ordinary auction. - Initial and SPA slot JSON attaches the exact `ext.trusted_server.trace_slot_ref` token that appears in server evidence; - inactive responses omit it. Missing, duplicate, conflicting, malformed, and - evidence-mismatched slot tokens suppress only correlation and produce the + inactive responses omit it. With valid matching auction evidence supplied, + missing, duplicate, conflicting, malformed, and evidence-mismatched slot + tokens suppress only correlation and produce the specified coverage issues without changing slot/bid order or contents. - Active responses remain terminally private/no-store under hostile late header overrides. @@ -1369,6 +1511,11 @@ results, never a prerequisite for returning them. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. - Trace-route failures never fall through to publisher origin. +- Broad `^/_ts` and `^/` authentication rules challenge every trace path + (shell/state/actions/assets, disabled routes, and unsupported methods); valid + credentials proceed to trace handling, while `^/_ts/admin` leaves trace + routes public. Challenges expose no context or cookie mutation, and + protected assets remain private/no-store. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. - Every adapter omits JA4/H2 and rejects control characters or overlong platform @@ -1377,10 +1524,19 @@ results, never a prerequisite for returning them. ### 14.3 JavaScript unit tests - Explicit snapshot only; no continuous `sessionStorage` writes. +- Both TSJS auction callers require the literal `__tsjs_trace_active === true` + before token generation or trace collection. Cover missing/false/non-boolean + flags with GPT diagnostics active: no tokens, mappings, listeners, pending + transport records, sidecars, or handoff action are created. - Compact UTF-8 size measurement; exact outer/nested schema validation; unknown fields; per-string/array/numeric/depth caps; hostile mutation; expiry; future-clock skew; wall-clock rollback; replacement; clearing; and storage exceptions. +- A filled GPT cycle with `observedSlotSize: [0, 0]`, `[0, 250]`, or `[300, 0]` + survives projection, storage validation, rendering, and export unchanged. + Negative, fractional, non-finite, and over-limit observed dimensions fail + validation; requested/selected creative and GPT fill dimensions retain their + positive bounds. - Omission counters use checked arithmetic and reject overflow. - Strict validation of every server-auction enum, token, numeric bound, array bound, nesting level, and unknown property; invalid evidence is discarded @@ -1388,12 +1544,18 @@ results, never a prerequisite for returning them. - Transport envelopes require exactly one of evidence/unavailable reason; projection, transport, validation, eviction, correlation, and external-client coverage states produce the specified complete/partial/unavailable/not-observed - result without treating absence as proof that no auction ran. + result without treating absence as proof that no auction ran. A fixture that + receives valid evidence and then evicts every server record during size + truncation must yield `unavailable` with `record_evicted` and exact omission + counts; partial eviction remains `partial`. +- Successful responses without an optional transport member add no validation + issue; malformed supplied members add `evidence_validation_failed`. Cover + both an empty collector and one with retained evidence or earlier failures. - Direct fetch failures and Prebid `interpretResponse`, `onTimeout`, and `onBidderError` paths consume their pending transport record exactly once; capped/expired records and absent hooks follow the specified `not_observed` behavior without retaining error text or bodies. -- Exact-token correlation joins matching server auctions, GPT opportunities, +- Exact-token correlation joins matching SSAT/SPA server auctions, GPT opportunities, and slot references; unmatched, duplicated, conflicting, missing, and forged tokens stay separate and produce explicit coverage states. No timestamp, index, or ad-unit-path heuristic is used. @@ -1401,6 +1563,11 @@ results, never a prerequisite for returning them. opportunity-to-cycle binding, is capped and evicted deterministically, contains only opaque tokens plus runtime/request numbers, and does not alter `GptDiagnosticsExportV1`. +- Prebid `/auction` evidence remains independent of its `prebid_refresh` GPT + cycle, and direct `requestAds` evidence remains independent of GPT. Neither + path emits a sidecar or introduces `trusted_server_direct`/`competing` + attribution. Both show `correlation_unavailable` without downgrading complete + server capture; supplied API sidecars are rejected by the viewer. - Source presentation distinguishes server-owned SSAT/page-bids/auction API facts from browser-observed publisher refresh, Prebid refresh, competing, and unattributed request paths. No fixture turns intent or a GPT fill into a @@ -1413,7 +1580,12 @@ results, never a prerequisite for returning them. fixture. - Same-tab navigation occurs only after a successful write. - Viewer handles absent optional network facts and every cookie-health state. -- Forbidden fields never enter storage or export fixtures. +- Populate every excluded `adManager` field, `previousCreativeId`, slot + `slotElementId`/`adUnitPath`, and callback/attribution issue `slotElementId` + with distinct sentinel values, including a synthetic secret-bearing path. + Verify none enter trace HTML, storage, copy/share, or direct export; hostile + stored copies containing any excluded property are rejected. Numbered + slot/cycle correlation still joins after redaction. - Download filename and MIME type are deterministic. - Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and download/copy fallback behavior. @@ -1440,7 +1612,7 @@ results, never a prerequisite for returning them. transport fixtures preserve ad initialization while reporting `not_observed`. - `View trace results` navigates in the same tab and renders the captured request context and slot evidence. -- A correlated fixture renders the chain `server auction -> GPT -> creative`, +- A correlated SSAT/SPA fixture renders the chain `server auction -> GPT -> creative`, while unmatched server, client-side refresh, competing, and transport-failure fixtures show honest independent evidence and `Unknown` where appropriate. - Empty, filled, ambiguous, no-candidate, and unattributed slot states remain @@ -1468,8 +1640,10 @@ results, never a prerequisite for returning them. - Inactive publisher traffic has no trace assets, storage access, listeners, or cache-policy change, diagnostic token generation, or trace-auction response extension. -- Disabling `trace_page_enabled` removes every new capture behavior even when a - technical `?ts_console=1` session leaves a valid diagnostics cookie present. +- After disabling `trace_page_enabled`, a fresh publisher load removes every + new capture behavior even when a technical `?ts_console=1` session leaves a + valid diagnostics cookie present. Already-loaded documents cannot be remotely + deactivated, but subsequent server responses contain no trace evidence. ### 14.5 Manual acceptance @@ -1500,12 +1674,17 @@ observed`, `GPT filled/rendered`, and `Unknown` without understanding internal ## 16. Acceptance criteria -1. With the feature disabled, trace-route origin requests return local `404` - and ordinary traffic is unchanged. Previously cached inert versioned assets +1. With the feature disabled, trace-route origin requests that pass configured + authentication return local `404` + and ordinary requests without explicit diagnostics activation are unchanged. + Query activation adopts the shared cookie lifetime in section 5.2. + Previously cached inert versioned assets may remain until cache eviction, but cannot activate tracing or load a shell. -2. A mobile user can enable tracing by opening only `/_ts/trace` and selecting +2. On a deployment whose authentication rules leave trace routes public, a + mobile user can enable tracing by opening only `/_ts/trace` and selecting one prominent action; no target URL, credentials, or trace ID is required, - and a cross-site GET cannot activate tracing. + and a cross-site GET cannot activate tracing. Matching operator auth rules + remain enforced on all trace paths. 3. The setup page accurately explains that the problem must be reproduced after activation. 4. A subsequent real publisher-page reload captures redacted request context, @@ -1552,7 +1731,8 @@ This design is one product flow, but its implementation is split into four independently reviewable plans and preferably four PRs: 1. **Reserved route and privacy foundation:** configuration, shared early-route - classification, same-origin enable/end lifecycle, bounded cookie-health + classification with operator authentication, same-origin enable/end lifecycle, + shared 30-minute endpoint/query cookie policy, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not add speculative new platform fields in this change. @@ -1593,9 +1773,11 @@ target validation and open-redirect risk, and is unsuitable for a layperson. ### Basic Authentication -Rejected for the mobile end-user workflow. Authentication also would not make -it safe to inject raw secrets into a publisher page containing third-party -JavaScript. +Rejected as a new mandatory prerequisite for the mobile end-user workflow. +Existing operator authentication rules remain enforced; deployment configuration +must leave trace routes public to offer the credential-free journey. +Authentication also would not make it safe to inject raw secrets into a +publisher page containing third-party JavaScript. ### Server-managed trace sessions @@ -1651,6 +1833,9 @@ length, history, logging, referrer, and accidental-sharing risks. - `/auction` evidence requires the TSJS request to reach the same Trusted Server host with the active diagnostics cookie. A custom cross-origin auction endpoint does not inherit this trace session and is shown as unavailable. +- Version one has no GPT correlation for `/auction`: the Prebid caller lacks + a token-bearing GPT binding, and direct `requestAds` renders outside GPT. + Their server evidence is displayed independently with correlation unavailable. - Exact-token correlation can remain unavailable for hidden, unresolved, competing, or independently initiated GPT cycles. The report preserves both sides instead of guessing. From 6e9c50ca51d8fd3a9295f134d95e92fddc128e7a Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 20 Sep 2026 00:32:31 -0700 Subject: [PATCH 080/230] Select the release manifest in the CLI deploy examples `ts deploy --application-release` does not choose the release's manifest. The CLI loads `edgezero.toml` from the working directory unless `EDGEZERO_MANIFEST` names another file, and it rejects a manifest outside the release root, so both documented commands failed from the checkout root with "loaded application manifest ... is outside application release root" before reaching Fastly. Set `EDGEZERO_MANIFEST` to the release's manifest in both examples, as the EdgeZero deploy action does, and say where the release root comes from. --- docs/guide/cli.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 04f8ef3f0..7231e3d82 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -122,13 +122,24 @@ adapter flags directly need the `--` added. Trusted Server declares Config, KV, and Secret Stores, so EdgeZero treats every Fastly deploy as managed and requires a verified application release root via `--application-release`; a bare `ts deploy --adapter fastly` without a release is accepted only for -store-free applications: +store-free applications. The CLI loads `edgezero.toml` from the working +directory unless `EDGEZERO_MANIFEST` names another file, and it rejects a +manifest outside the release root, so select the release's own manifest: ```bash -ts deploy --adapter fastly --service-id --application-release --staging -ts deploy --adapter fastly --service-id --application-release -- --comment "release" +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" --staging +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" -- --comment "release" ``` +`` is the extracted immutable application release that EdgeZero's +`package-fastly-application-release` action produces from a `ts build`. The +EdgeZero `deploy-fastly` action performs both steps, selects the manifest, and +supplies the release root; see EdgeZero's GitHub Actions deployment guide +(`docs/guide/deploy-github-actions.md` in the EdgeZero repository) for the +producer and consumer workflow. + A staged deploy selects the physical Config Store from the staging environment and links it to the staged version under the logical store ID. The staged runtime reads the `` key from that store. It does not copy From ed82795d1d4989a61e467473f57d423a70d1c81d Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Sun, 20 Sep 2026 13:53:32 +0530 Subject: [PATCH 081/230] Derive ALL_MODULE_IDS from TSJS_MODULES to prevent the two lists drifting Fix a compile error in the const-derived array and add a regression test pinning all_module_ids() to the generated TSJS_MODULES table. Signed-off-by: dhruv8sh --- crates/trusted-server-js/build.rs | 7 +------ crates/trusted-server-js/src/bundle.rs | 10 ++++++++++ 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index 784d87494..8fbd34c9e 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -131,7 +131,6 @@ fn main() { // Generate tsjs_modules.rs with include_str!() for each module let mut codegen = String::new(); - let mut ids_block = String::new(); codegen.push_str("// Auto-generated by build.rs - DO NOT EDIT\n\n"); writeln!( @@ -141,8 +140,6 @@ fn main() { ) .expect("should write generated module header"); for (id, filename) in &modules { - writeln!(ids_block, " \"{id}\",").expect("should write generated module ID"); - let sha256 = bundle_sha256(&out_dir.join(filename)); writeln!( codegen, @@ -153,12 +150,10 @@ fn main() { codegen.push_str("];\n"); writeln!( codegen, - "pub(crate) const ALL_MODULE_IDS: [&str; {}] = [", + "pub(crate) const ALL_MODULE_IDS: [&str; {0}] = {{\n let mut ids = [\"\"; {0}];\n let mut index = 0;\n while index < {0} {{\n ids[index] = TSJS_MODULES[index].id;\n index += 1;\n }}\n ids\n}};", modules.len() ) .expect("should write generated module IDs"); - codegen.push_str(&ids_block); - codegen.push_str("];\n"); codegen.push_str("\npub(crate) struct TsjsModuleMeta {\n"); codegen.push_str(" pub bundle: &'static str,\n"); codegen.push_str(" pub id: &'static str,\n"); diff --git a/crates/trusted-server-js/src/bundle.rs b/crates/trusted-server-js/src/bundle.rs index 762d51cf4..267f74495 100644 --- a/crates/trusted-server-js/src/bundle.rs +++ b/crates/trusted-server-js/src/bundle.rs @@ -138,6 +138,16 @@ mod tests { encode(Sha256::digest(bytes)) } + #[test] + fn all_module_ids_matches_generated_module_list() { + let from_modules: Vec<&str> = TSJS_MODULES.iter().map(|module| module.id).collect(); + assert_eq!( + all_module_ids(), + from_modules.as_slice(), + "the generated ID list should match the generated module table" + ); + } + #[test] fn generated_single_module_hashes_match_bundle_contents() { for id in all_module_ids() { From 95bd289427cf161442fbda484b160e6127f93d71 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Sun, 20 Sep 2026 14:05:06 +0530 Subject: [PATCH 082/230] Reorder root Markdown check, document npm ci prereq Signed-off-by: dhruv8sh --- .github/workflows/format.yml | 6 +++--- AGENTS.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 4df61cf25..c844e95d3 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,9 +149,9 @@ jobs: - name: Run Prettier (check) run: npm run format - - name: Build with VitePress (fails on dead links) - run: npm run build - - name: Run Prettier (check) — root Markdown working-directory: . run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + + - name: Build with VitePress (fails on dead links) + run: npm run build diff --git a/AGENTS.md b/AGENTS.md index ac9e3c80c..aab22cae5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -345,7 +345,7 @@ Every PR must pass: 5. JS build and test (`cd crates/trusted-server-js/lib && npx vitest run`) 6. JS format (`cd crates/trusted-server-js/lib && npm run format`) 7. Docs format (`cd docs && npm run format`) -8. Root Markdown format (`docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check`) +8. Root Markdown format (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check` --- From 3e8cde2b74e5328330c71f8b45a54ad5f0014241 Mon Sep 17 00:00:00 2001 From: Christian Pavilonis Date: Sun, 20 Sep 2026 11:09:09 -0500 Subject: [PATCH 083/230] Update crates/trusted-server-js/lib/src/integrations/prebid/index.ts Co-authored-by: prk-Jr <49094961+prk-Jr@users.noreply.github.com> --- .../trusted-server-js/lib/src/integrations/prebid/index.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 51be7d665..df589150d 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -518,7 +518,12 @@ function installPrebidWinDiagnostics(): void { const bid = rawBid as Record; const auctionId = typeof bid.auctionId === 'string' ? bid.auctionId : undefined; const adUnitCode = typeof bid.adUnitCode === 'string' ? bid.adUnitCode : undefined; - if (!auctionId || !adUnitCode) return; + if ( + !auctionId || + !adUnitCode || + (bid.latestTargetedAuctionId !== undefined && bid.latestTargetedAuctionId !== auctionId) + ) + return; const key = prebidDiagnosticKey(auctionId, adUnitCode); const attempt = prebidDiagnosticAttempts.get(key); prebidDiagnosticAttempts.delete(key); From d6fecdf8a24fba7a5a9b916cd0f3dfc6a6876877 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 20 Sep 2026 10:56:57 -0700 Subject: [PATCH 084/230] Move the EdgeZero pin to the PR 381 branch head 12c3215c Upstream merged main into the branch, which renames the staged-deploy identifiers to the staging naming. Trusted Server references none of the renamed variants, so this is a lockfile-only update. --- Cargo.lock | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 25197f4e6..62bca5610 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#5878eb740897749e4cfab7b410fa28791a371602" +source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "log", "proc-macro2", From 14c274cbae3e33b338fe6119e98ec4470d4b075d Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Mon, 21 Sep 2026 10:50:53 +0530 Subject: [PATCH 085/230] Clarify mobile trace design contracts --- ...-mobile-ad-render-trace-endpoint-design.md | 190 +++++++++++++++--- 1 file changed, 160 insertions(+), 30 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index 5d86b9092..e3aaca4c6 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -451,7 +451,17 @@ trace_page_enabled = false Rules (route responses below apply after configured authentication): - `trace_page_enabled = true` requires `enabled = true`; invalid combinations - fail configuration validation. + fail configuration validation, including when `enabled` is omitted (defaults + to false). Add a raw-config validation hook in both deploy and runtime + validation, following `validate_js_asset_proxy_config` in + `crates/trusted-server-core/src/config.rs`. It must deserialize and validate + this combination outside the enabled gate: `IntegrationSettings::get_typed` + returns `Ok(None)` before `validate()` for disabled integrations, so a schema + validator alone cannot enforce this rule. Preserve unknown-field rejection + on disabled configurations. This spec explicitly requires invalid enabled + configuration to fail rather than be logged and disabled; the prior rationale + is the HIGH-severity finding in + `2026-03-11-production-readiness-report-design.md`, not a contributor-doc rule. - Operator documentation beside this option states that the public page makes the allowlisted presence/validity of four HttpOnly Trusted Server cookies visible to same-origin JavaScript whenever the feature is enabled. It also @@ -465,7 +475,12 @@ Rules (route responses below apply after configured authentication): parameters do not activate or deactivate tracing and are not reflected into the page or export. - `HEAD /_ts/trace` returns the GET status and headers without a body or state - mutation. + mutation. Explicitly handle `HEAD` on every trace path in all four adapters, + returning a bodyless local 405 on POST-only paths. Fastly's + `publisher_fallback_methods()` includes `HEAD`; a GET registration alone + sends HEAD to the publisher fallback, as pinned by + `dispatch_head_on_named_get_route_falls_through_to_publisher_fallback` in + `crates/trusted-server-adapter-fastly/src/app.rs`. - `GET /_ts/trace/state` returns private/no-store JSON containing only `observed_active: true|false`, determined from whether that request carried exactly one valid diagnostics cookie. `HEAD` returns the same status and @@ -485,13 +500,20 @@ Rules (route responses below apply after configured authentication): cookie using `X-TS-Trace-Action: end`, and returns a small local JSON result. After explicit user confirmation, client JavaScript independently attempts local report deletion and the end POST. Neither result gates the other. -- For both POST paths, absent or exactly-zero `Content-Length` is accepted, - `Transfer-Encoding` is rejected, and the adapter reads at most one byte when - it must verify an absent length. Any body byte or positive/invalid length - returns local `413 Payload Too Large` without draining or processing an - unbounded body. The one-byte read inherits a maximum two-second adapter - request-body deadline; timeout returns local `408 Request Timeout` with no - mutation. +- For both POST paths, accept absent or exactly-zero `Content-Length` only + when the body is empty; reject `Transfer-Encoding`, positive/invalid lengths, + and any actual body bytes with local `413 Payload Too Large` and no mutation. + Precheck headers, then use `Body::into_bytes_bounded(0)` to check emptiness, + following the header-precheck/body-size-check pattern in + `crates/trusted-server-core/src/auction/endpoints.rs`. This is an application + acceptance limit, not a transport read or allocation limit. Pinned EdgeZero + v0.0.8 buffers the Fastly body with blocking `read_to_end` and the Cloudflare + body with `req.bytes().await` before core handling. Spin also buffers the + body; Axum buffers JSON bodies but can stream other content types. `Body` + exposes no read deadline, and Fastly uses `futures::executor::block_on` without a timer. + Therefore v1 promises neither a one-byte transport read, a two-second timeout, + nor a local 408. Transport-level size/deadline protection requires a separate + adapter/upstream change; document the pre-buffering limitation at deployment. - State-changing POSTs require an `Origin` exactly matching the canonical request origin and `Sec-Fetch-Site: same-origin`. Missing, conflicting, malformed, cross-site, or duplicate control values return local `403` without @@ -503,7 +525,16 @@ Rules (route responses below apply after configured authentication): default ports removed before exact comparison. Invalid or multi-valued host, authority, scheme, or origin input fails closed. - Unsupported methods on a shell or state-changing path return a local 405 - Method Not Allowed response with the path-specific `Allow` header. + Method Not Allowed response with the path-specific `Allow` header: + `GET, HEAD` for shell/state/assets and `POST` for enable/end. Classification + must intercept unsupported methods before router dispatch. The router's + `MethodNotAllowed` response has no `Allow` header and bypasses + `FinalizeResponseMiddleware`, as pinned by + `dispatch_unregistered_method_returns_405_at_router_level` in the Fastly + adapter. The trace responder itself supplies `Allow` and all section 12.3 + error-response hardening; it must not rely on router-generated errors. + Fastly entry-point finalization can add ordinary headers later, but does not + establish this trace-specific contract on behalf of the router. - Disabled deployments return a local `404` for the complete trace route set, including assets, and never fall through to the publisher origin. - The `/_ts/trace` namespace is reserved. A trailing slash, extra path segment, @@ -513,6 +544,13 @@ Rules (route responses below apply after configured authentication): The adapter classifies from its canonical parsed path while retaining enough raw-path information to reject ambiguous encodings consistently. +Reuse the bounded percent-decode-to-fixed-point classification pattern from +`deny_admin_diagnostic_fallback` in `crates/trusted-server-core/src/ec/admin.rs` +(`MAX_PERCENT_DECODE_ROUNDS = 4`), moving trace classification before dispatch +rather than relying on fallback. Register and intercept trace paths on Fastly, +Axum, Cloudflare, and Spin from the first implementation PR; existing Fastly-only +`/_ts/*` routes are not a parity precedent. + Every adapter implements the following order: 1. Parse the method, canonical host/origin, path, query, and bounded headers @@ -650,7 +688,11 @@ The classifier uses this deterministic contract: precedence is therefore diagnostic rather than first- or last-value selection. - Per-value limits are 512 bytes for `ts-ec`, 8 KiB for `ts-eids`, and 16 bytes - each for `ts-tester` and `__Host-ts-console`. A single value beyond its limit + each for `ts-tester` and `__Host-ts-console`. Only the 8 KiB EID limit is + inherited (`MAX_EIDS_COOKIE_BYTES` in `ec/prebid_eids.rs`); the other limits + are new trace-inspection bounds. The 512-byte EC limit is an outer guard, + above the exact 71-character format accepted by `is_valid_ec_id` in + `ec/generation.rs`, and does not broaden EC validity. A single value beyond its limit is `present_invalid/oversized`; it does not change the other three states. - One `ts-ec` occurrence is valid only when the canonical EC cookie validator accepts its complete value. @@ -727,6 +769,40 @@ retain their source meaning and require explicit allowlisting; future source fields are not inherited automatically. `trustedServerAuctionId`, when present, must satisfy the diagnostic auction-token contract in section 9.4. +The following is the exhaustive v1 property allowlist derived from the current +interfaces in `crates/trusted-server-js/lib/src/core/types.ts`, after the above +exclusions. Preserve source optionality and validate each source enum against its +explicit current members; do not spread source objects or dynamically inherit +later fields. Section 9.5 supplies numeric, string, array, and depth bounds. + +| Object | Allowed properties | +| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| GPT projection | `schema_version`, `source_schema_version`, `capturedAt`, `page`, `slots`, `callbackIssues`, `attributionIssues`, `coverage`, `metadata` | +| `page` | `origin`, `pathname` (fixed redacted literal) | +| Slot | `runtimeSlotNumber`, `binding`, `currentVisibilityPercentage`, `maximumVisibilityPercentage`, `requests` | +| `binding` | `status`, `reason` (current `GptDiagnosticsBindingReason` enum) | +| Request cycle | `requestNumber`, `requestedAtMs`, `responseAtMs`, `renderAtMs`, `loadAtMs`, `viewableAtMs`, `durations`, `isEmpty`, `requestedSlotSizes`, `size`, `observedSlotSize`, `isBackfill`, `slotContentChanged`, `incompleteSequence`, `responseClass`, `requestPath`, `requestIntentId`, `trustedServerAuctionId`, `opportunityToRequestMs`, `replacedRequestNumber`, `previousRenderToRequestMs`, `creativeChanged`, `loadObservedBeforeRender`, `trustedServerOpportunity`, `trustedServerCreativeRequestAtMs`, `trustedServerCreativeResponseAtMs`, `trustedServerCreativeFailures`, `delivery` | +| `durations` | `requestToResponseMs`, `responseToRenderMs`, `requestToRenderMs`, `renderToLoadMs`, `renderToViewableMs` | +| Callback issue | `kind`, `runtimeSlotNumber`, `timestampMs`, `disposition`, `reason` (only the documented values below) | +| Attribution issue | `reason` (current `GptDiagnosticsAttributionIssueReason` enum), `timestampMs`, `runtimeSlotNumber` | +| `coverage` | Exactly the six current callback-kind keys: `slotRequested`, `slotResponseReceived`, `slotRenderEnded`, `slotOnload`, `impressionViewable`, `slotVisibilityChanged` | +| Each coverage counter | `observed`, `matched`, `unmatched`, `ambiguous` | +| `metadata` | `droppedCallbacks`, `droppedAttributionIssues`, `evictedSlots`, `evictedRequestCycles` | + +Callback `reason` accepts only the current store's six emitted literals: +`invalid_event_order`, `missing_response_before_render`, +`invalid_visibility_percentage`, `evicted_slot`, `no_compatible_request_cycle`, +and `overlapping_request_cycles`. +Although the source interface types it as `string`, the trace validator rejects +all other values rather than copying arbitrary text. + +`requestPath` is the bounded `GptDiagnosticsRequestPath` enum, not a URL or +pathname; include it as browser intent only. `requestIntentId` is a local numeric +sequence, not an external request identifier. Source `version` becomes +`source_schema_version`; no other source properties pass through. Tests must +classify every current source member as copied, transformed, or excluded, and +reject extra keys at every object boundary. + It is deliberately not named or represented as `GptDiagnosticsExportV1`, because redaction, excluded identifiers, and trace-level bounds change the source field semantics. TS Console continues to own the source schema; the trace envelope @@ -812,11 +888,16 @@ telemetry and OpenRTB objects: request ID, or an identifier joinable to user-bearing logs. - Auction tokens are `ts-auc-` followed by a lowercase UUID v4 in the existing producer's 32-hex-digit unhyphenated form (`Uuid::new_v4().simple()`). Slot - tokens are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, + tokens are new: neither a `ts-slot-` producer nor a TSJS `crypto.randomUUID()` + call exists today. This design introduces tokens that are `ts-slot-` followed by a canonical lowercase hyphenated UUID v4, matching `crypto.randomUUID()`. Both validators enforce UUID version 4 and the RFC variant and reject every other shape. Tokens are compared verbatim, never normalized during validation or correlation; no producer format change - is required for the existing GPT auction opportunity marker. + is required for the existing GPT auction opportunity marker. For the GPT + projection, validation and comparison start from the marker returned by + `normalizedAuctionId` in `gpt_diagnostics/store.ts`, which already trims + whitespace and caps the stored value. Trace validation performs no further + normalization; invalid shapes are not repaired into valid tokens. - `slot_number` is a one-based ordinal over the exact post-conversion `AuctionRequest.slots` sequence observed by orchestration. It is display-only and is never used to map a response back to pre-conversion client input. @@ -838,7 +919,10 @@ telemetry and OpenRTB objects: - `source` is assigned by the server call site: initial document auction is `initial_navigation_ssat`, `/_ts/page-bids` is `spa_page_bids`, and `POST /auction` is `auction_api`. Browser `requestPath` does not determine or - override this value. + override this value. These are intentionally trace-owned public names: + existing `AuctionSource` in `auction/telemetry.rs` uses `initial_navigation`, + `spa_navigation`, and `auction_api`, respectively. Map these explicitly; + do not change telemetry vocabulary or serialize it directly into the report. - `provider_number` is assigned deterministically in provider dispatch order and is stable only within one auction. Provider names, bidder/seat names, and provider metadata are omitted. `returned_bid_count` is a count, not a bid @@ -901,7 +985,10 @@ before an envelope arrives is recorded separately by the browser as `evidence_transport_failed`. Initial-navigation and SPA slot definitions carry their token on the exact slot -object TSJS already consumes: +object TSJS already consumes. This is a two-sided addition: add optional `ext?` +to the TypeScript `AuctionSlot` interface in `core/types.ts` and the matching +nested key to Rust's `build_slot_json` in `publisher.rs`. Neither has this member +today; Rust builds free-form JSON, and TSJS currently retains the slot objects: ```text AuctionSlot.ext.trusted_server.trace_slot_ref: string @@ -989,7 +1076,8 @@ that preserves existing request-path attribution. `ext.trusted_server.trace_slot_ref`. Failed, abandoned, skipped, and zero-bid outcomes still inject their bounded evidence when the publisher document can be delivered. -- **SPA page-bids:** `/_ts/page-bids` adds an optional, namespaced +- **SPA page-bids:** `/_ts/page-bids` and its still-live deprecated alias + `/__ts/page-bids` both add an optional, namespaced `trace_auction` transport envelope beside its existing bid result. TSJS validates and records it and consumes each returned slot's `ext.trusted_server.trace_slot_ref` before triggering ad initialization. Both @@ -1089,7 +1177,7 @@ Runtime limits are part of the v1 contract: | Value | Limit | | ----------------------------------------------- | ------------------------------------------------------------ | -| Container nesting | 8 levels | +| Container nesting | 10 levels | | Server auctions | 16 | | Slot correlations | 128 | | Provider calls | 16 per server auction | @@ -1116,6 +1204,12 @@ Runtime limits are part of the v1 contract: | GPT-reported fill dimension (`size`) | Finite integer from 1 through 100,000 | | Observed CSS box dimension (`observedSlotSize`) | Finite integer from 0 through 100,000 | +The depth cap includes two levels of headroom above the current deepest valid +GPT path: report (1), GPT projection (2), slots array (3), slot (4), requests +array (5), cycle (6), `requestedSlotSizes` array (7), size tuple (8). Headroom +does not permit unknown fields; future schema additions still require explicit +compatibility review. Test the complete current projection and over-depth input. + `observedSlotSize` preserves zero dimensions, including `[0, 0]` for a hidden or collapsed element after a filled GPT render. Zero is a measured box size, not missing data or evidence of an empty GPT response; do not omit it or @@ -1154,7 +1248,7 @@ wrapping or saturating the count. For depth accounting, the `TraceReportV1` object—not its storage wrapper—is level 1; entering either an object or an array increments the level by one; -primitives do not. No accepted report value may enter a ninth container level. +primitives do not. No accepted report value may enter an eleventh container level. The storage wrapper is validated separately as the exact two-field object `{ stored_at_ms, report }`. @@ -1311,7 +1405,10 @@ The HTML shell, enable/end responses, every active diagnostic publisher response, and every dynamic page-bids or `/auction` response carrying trace evidence are terminally `private, no-store`. The fixed versioned JS/CSS assets are the sole exception and may be publicly cached because they contain no -request or report data. HTML and JSON endpoint responses also send: +request or report data. HTML and JSON endpoint responses, including state +results and all local authentication, routing, validation, and disabled-route errors, also send the +following headers. Every such error is `private, no-store`; only successful +fixed-asset responses qualify for the cache exception: - Path-appropriate `Content-Type`: `text/html; charset=utf-8` for the shell and `application/json; charset=utf-8` for enable, end, and state results. @@ -1323,9 +1420,19 @@ usb=()` ```text default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; -object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self' +object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self'; +img-src data: ``` +The shell supplies a fixed data-URL favicon; `img-src data:` allows it without +an automatic publisher `/favicon.ico` fetch. All styles live in the fixed CSS +asset: toggle classes or the `hidden` attribute, with no inline style attributes, +style blocks, or JavaScript style-property writes. JSON download uses a Blob +object URL assigned directly to an `` followed by a click, then +revokes the URL after the download has started. Do not fetch the blob URL or +embed it in a frame. This fixes the download mechanism without widening +`connect-src` or enabling frames; browser tests exercise it under this exact CSP. + The endpoint makes no third-party requests. Its script and stylesheet are fixed same-origin static assets. Setup-request values are server-rendered as escaped text nodes, and the viewer obtains the report only from browser storage. Active @@ -1352,13 +1459,26 @@ extension; active responses add them only in request-scoped injection or the request-scoped body seam. The existing diagnostics private/no-store decision remains a load-bearing gate. Tests must prove that late response-header handlers cannot make traced content -publicly cacheable. +publicly cacheable. Reuse +`apply_response_headers_with_cache_privacy` in `response_privacy.rs`, which +skips operator cache-header overrides on already uncacheable responses. Fastly +also re-runs privacy guards in `apply_terminal_response_effects` after late EC +and filter effects; retain its +`late_filter_effects_cannot_make_an_assembled_response_public` regression. +`apply_finalize_headers` is terminal on Axum, Cloudflare, and Spin, but not on +Fastly; trace handling must preserve the appropriate terminal protection. ### 12.5 Operator authentication policy -Trace routing preserves the existing `auth.rs` namespace contract: every -matching operator Basic Authentication rule is enforced, including `^/_ts` or -`^/`. Classification may run early, but does not authorize a request. Only +Trace routing preserves existing first-match-wins operator authentication: +`Settings::handler_for_path` selects one handler, and trace handling enforces +that handler's Basic Authentication policy. Rules do not compose; a preceding +narrow rule can shadow `^/_ts` or `^/`. With no matching handler, trace remains +public: the fail-closed unmatched-path backstop in `enforce_basic_auth` applies +only to `/_ts/admin`, not trace. Call the synchronous `enforce_basic_auth` with +settings and the request before serving any trace response. Do not copy the +Fastly `/_ts/debug/ja4` early return, which bypasses `AuthMiddleware`. +Classification may run early, but does not authorize a request. Only authentication is factored ahead of trace handling; ordinary event, identity, filter, and auction processing stays outside trace routes. A challenge exposes no setup context, changes no cookie, and is terminally private/no-store. @@ -1431,7 +1551,8 @@ results, never a prerequisite for returning them. ### 14.1 Core unit tests - Configuration defaults off and rejects trace-page enablement without GPT - diagnostics. + diagnostics, with `enabled` both explicitly false and omitted, on both + deploy and runtime validation paths. - With GPT diagnostics enabled but `trace_page_enabled = false`, a valid console cookie still enables the existing console but never mints trace tokens, builds auction evidence, adds response extensions, or emits @@ -1455,8 +1576,9 @@ results, never a prerequisite for returning them. Absent, invalid, duplicate, and uninspectable cookies all report inactive, with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, - the first unexpected body byte, and the two-second deadline without an - unbounded read. + nonempty bodies even with absent/zero lengths, and verifies no mutation on + rejection. Tests must not claim a transport bound or timeout that the pinned + adapters cannot enforce. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, configured filters, ordinary event context, and origin fetch. - Cookie-health scanner covers multiple header fields; zero, one, and duplicate @@ -1467,6 +1589,8 @@ results, never a prerequisite for returning them. fields. - Server-auction projection maps initial navigation, SPA page-bids, and auction API call sites to the exact public source enums without using a browser hint. + Canonical and legacy page-bids routes produce equivalent gated evidence and + slot extensions, including the TSJS retry against the legacy alias. - The diagnostic auction token is minted before dispatch and remains identical across completed, zero-bid, skipped, failed, and abandoned evidence and the corresponding SSAT/SPA browser opportunity marker. API tokens remain @@ -1510,12 +1634,16 @@ results, never a prerequisite for returning them. sources. - Axum, Cloudflare, and Spin return the common route/schema with unavailable fields omitted. -- Trace-route failures never fall through to publisher origin. +- Trace-route failures never fall through to publisher origin, including HEAD + on each exact path, malformed reserved paths, and arbitrary unsupported + methods intercepted before router dispatch. Assert path-specific `Allow`, + bodyless HEAD errors, and hardening headers on local errors. - Broad `^/_ts` and `^/` authentication rules challenge every trace path (shell/state/actions/assets, disabled routes, and unsupported methods); valid credentials proceed to trace handling, while `^/_ts/admin` leaves trace routes public. Challenges expose no context or cookie mutation, and - protected assets remain private/no-store. + protected assets remain private/no-store. Also test an earlier narrow handler + shadowing a broad rule to pin first-match-wins behavior. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. - Every adapter omits JA4/H2 and rejects control characters or overlong platform @@ -1735,7 +1863,9 @@ independently reviewable plans and preferably four PRs: shared 30-minute endpoint/query cookie policy, bounded cookie-health inspection, base request-context schema, projection of already populated `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not - add speculative new platform fields in this change. + add speculative new platform fields in this change. Include the raw-config + validation hook and method-independent dispatch on all four adapters in + this first PR; Fastly-only route registration does not satisfy the contract. 2. **Live server-auction evidence:** introduce the public diagnostic auction and slot tokens, project `TraceAuctionEvidenceV1` at the live observation boundary, transport it through initial navigation, page-bids, and both From a7b19ca2881eef16bcf04a78ce75ec839f89307e Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 21 Sep 2026 09:50:54 -0500 Subject: [PATCH 086/230] Address remaining GPT diagnostics review feedback --- .../integrations/gpt_diagnostics/overlay.ts | 2 + .../src/integrations/gpt_diagnostics/store.ts | 10 +-- .../lib/src/integrations/prebid/index.ts | 3 + .../gpt_diagnostics/overlay.test.ts | 62 +++++++++++++++++++ .../test/integrations/prebid/index.test.ts | 11 ++++ .../gpt-diagnostics-dictionary.md | 32 +++++----- 6 files changed, 99 insertions(+), 21 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index b69b40194..fe064064e 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -761,6 +761,8 @@ export class GptDiagnosticsOverlay { note.setAttribute('role', 'status'); note.textContent = `Ad #${this.selectedRequest.runtimeSlotNumber}, Request #${this.selectedRequest.requestNumber} is no longer retained.`; panel.append(note); + this.selectedRequest = undefined; + this.selectedRequestHasFocus = false; } const content = this.document.createElement('div'); diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index 344e20221..fd32b9c26 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -730,11 +730,11 @@ export class GptDiagnosticsStore { const requestPath = this.requestPath(intent); const auctionType = this.auctionType(intent, trustedServerEvidence); const serverAuctionTimingOrigin: GptDiagnosticsServerAuctionTimingOrigin | undefined = - trustedServerEvidence?.serverAuctionTimings === undefined - ? undefined - : trustedServerEvidence.auctionType === 'trusted_server' - ? 'spa_auction' - : 'navigation'; + trustedServerEvidence?.auctionType === 'trusted_server' + ? 'spa_auction' + : trustedServerEvidence?.auctionType === 'ssat' + ? 'navigation' + : undefined; record.requests.push({ requestNumber, requestedAtMs: timestampMs, diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index df589150d..9764dcb97 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -484,6 +484,9 @@ function recordCompletedPrebidAuction( const counts = new Map(); for (const code of adUnitCodes) counts.set(code, (counts.get(code) ?? 0) + 1); const nowMs = performance.now(); + for (const [key, attempt] of prebidDiagnosticAttempts) { + if (nowMs > attempt.expiresAtMs) prebidDiagnosticAttempts.delete(key); + } const generation = window.tsjs?.navGeneration ?? 0; for (let index = 0; index < auctionSlots.length; index += 1) { const slot = auctionSlots[index]; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index a5c4bf812..6f81692bc 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -575,6 +575,68 @@ describe('GptDiagnosticsOverlay', () => { overlay.destroy(); }); + it('labels unavailable SPA auction timing with the SPA request clock', () => { + const frames: Array<() => void> = []; + const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); + const diagnosticSlot = slot('spa-without-timings'); + store.recordTrustedServerOpportunity( + diagnosticSlot, + 'auction-spa-without-timings', + 'renderable_candidate', + undefined, + undefined, + { + auctionType: 'trusted_server', + winner: { bidder: 'example-bidder', priceBucket: '1.20' }, + } + ); + store.recordSlotRequested(diagnosticSlot); + let root: ShadowRoot | undefined; + const overlay = new GptDiagnosticsOverlay(store, new FakeBindings(), { + scheduleFrame: (callback) => frames.push(callback), + onShadowRoot: (createdRoot) => { + root = createdRoot; + }, + }); + runNextFrame(frames); + runNextFrame(frames); + + expect(root?.textContent).toContain('SPA page-bids T0 → auction dispatched Unavailable'); + expect(root?.textContent).not.toContain('Edge request T0 → auction dispatched Unavailable'); + overlay.destroy(); + }); + + it('announces an evicted request selection once and then clears it', () => { + const frames: Array<() => void> = []; + const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); + const diagnosticSlot = slot('evicted-selection'); + store.recordSlotRequested(diagnosticSlot); + let root: ShadowRoot | undefined; + const overlay = new GptDiagnosticsOverlay(store, new FakeBindings(), { + scheduleFrame: (callback) => frames.push(callback), + onShadowRoot: (createdRoot) => { + root = createdRoot; + }, + }); + runNextFrame(frames); + runNextFrame(frames); + + overlay.selectRequest(1, 1); + runNextFrame(frames); + let requestsToRecord = 10; + while (requestsToRecord > 0) { + store.recordSlotRequested(diagnosticSlot); + requestsToRecord -= 1; + } + runNextFrame(frames); + + expect(root?.textContent).toContain('Ad #1, Request #1 is no longer retained.'); + store.recordSlotResponseReceived(diagnosticSlot); + runNextFrame(frames); + expect(root?.textContent).not.toContain('Ad #1, Request #1 is no longer retained.'); + overlay.destroy(); + }); + it('reveals an exact request and locates it without mutating publisher markup', () => { const frames: Array<() => void> = []; const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index a5b9a756c..08ff55085 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -2619,6 +2619,9 @@ describe('prebid publisher snapshots and delivery refreshes', () => { bidder: 'example-client', priceBucket: '2.40', }); + expect(recordPrebidRefresh.mock.invocationCallOrder[0]).toBeLessThan( + recordPrebidAuction.mock.invocationCallOrder[0] + ); expect(getTargeting).not.toHaveBeenCalledWith('hb_cur'); const bidWon = mockOnEvent.mock.calls.find(([event]) => event === 'bidWon')?.[1]; expect(bidWon).toBeTypeOf('function'); @@ -2630,6 +2633,14 @@ describe('prebid publisher snapshots and delivery refreshes', () => { expect(recordPrebidWin).not.toHaveBeenCalled(); bidWon?.({ auctionId: 'example-client-auction', + latestTargetedAuctionId: 'later-client-auction', + adUnitCode: 'example-client-slot', + adserverTargeting: { hb_bidder: 'stale-client', hb_pb: '9.99' }, + }); + expect(recordPrebidWin).not.toHaveBeenCalled(); + bidWon?.({ + auctionId: 'example-client-auction', + latestTargetedAuctionId: 'example-client-auction', adUnitCode: 'example-client-slot', adserverTargeting: { hb_bidder: 'example-client', hb_pb: '2.40' }, }); diff --git a/docs/guide/integrations/gpt-diagnostics-dictionary.md b/docs/guide/integrations/gpt-diagnostics-dictionary.md index 896fc0cce..e0dc433a6 100644 --- a/docs/guide/integrations/gpt-diagnostics-dictionary.md +++ b/docs/guide/integrations/gpt-diagnostics-dictionary.md @@ -86,21 +86,21 @@ Path markers live for five seconds, are consumed once, and are keyed by GPT slot ## Timing -All values are milliseconds. Missing timing that should apply is `Unavailable`, never zero. Server timing is `Not applicable` when no completed server auction was observed. A displayed zero is a valid immediate observation. - -| Label | Origin and boundaries | Raw field | -| ------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------ | -| `Server request start → auction dispatched` | Server request `RequestTimings` T0 to successful `dispatch_auction` outcome | `serverAuctionTimings.auctionDispatchedMs` | -| `Server request start → auction collected` | Same T0 to completion of `collect_dispatched_auction` | `auctionResolvedMs` | -| `Server request start → bids ready` | Same T0 to winning-bid map commit | `auctionCommittedMs` | -| `Auction collection wait` | Actual duration blocked in collect; placement is `pre-header` or `in stream` | `auctionWaitMs`, `auctionWaitPlacement` | -| `Opportunity → request` | Browser `performance.now()`: recorder observation to matched GPT `slotRequested` | `opportunityToRequestMs` | -| `GAM request → response` | Browser `slotRequested` to `slotResponseReceived` | `durations.requestToResponseMs` | -| `GAM response → render` | Browser `slotResponseReceived` to `slotRenderEnded` | `responseToRenderMs` | -| `GAM request → render` | Browser `slotRequested` to `slotRenderEnded` | `requestToRenderMs` | -| `Render → load` | Browser `slotRenderEnded` to `slotOnload` | `renderToLoadMs` | -| `Render → viewable` | Browser `slotRenderEnded` to `impressionViewable` | `renderToViewableMs` | -| `Replaced rendered request` | Earlier browser render callback to later request callback | `previousRenderToRequestMs` | +All values are milliseconds. Missing timing that should apply is `Unavailable`, never zero. Server timing is `Not applicable` when no completed server auction was observed. A displayed zero is a valid immediate observation. The three server rows below are rendered with their T0 anchor substituted: `Edge request T0` for a navigation-origin auction and `SPA page-bids T0` for an SPA-origin one. + +| Label | Origin and boundaries | Raw field | +| ---------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------ | +| ` → auction dispatched` | Server request `RequestTimings` T0 to successful `dispatch_auction` outcome | `serverAuctionTimings.auctionDispatchedMs` | +| ` → auction collected` | Same T0 to completion of `collect_dispatched_auction` | `auctionResolvedMs` | +| ` → bids ready` | Same T0 to winning-bid map commit | `auctionCommittedMs` | +| `Auction collection wait` | Actual duration blocked in collect; placement is `pre-header` or `in stream` | `auctionWaitMs`, `auctionWaitPlacement` | +| `Opportunity → request` | Browser `performance.now()`: recorder observation to matched GPT `slotRequested` | `opportunityToRequestMs` | +| `GAM request → response` | Browser `slotRequested` to `slotResponseReceived` | `durations.requestToResponseMs` | +| `GAM response → render` | Browser `slotResponseReceived` to `slotRenderEnded` | `responseToRenderMs` | +| `GAM request → render` | Browser `slotRequested` to `slotRenderEnded` | `requestToRenderMs` | +| `Render → load` | Browser `slotRenderEnded` to `slotOnload` | `renderToLoadMs` | +| `Render → viewable` | Browser `slotRenderEnded` to `impressionViewable` | `renderToViewableMs` | +| `Replaced rendered request` | Earlier browser render callback to later request callback | `previousRenderToRequestMs` | Server offsets are not browser timestamps. `auctionResolvedMs` means collection completed (including timeout handling), not that a network byte arrived at that exact instant. @@ -109,7 +109,7 @@ Server offsets are not browser timestamps. `auctionResolvedMs` means collection | Label | Raw field / source | Meaning and limits | | ------------------------------------------------------------------------------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `Requested sizes` | `requestedSlotSizes` | Configured sizes supplied to GPT; ordinary sizes such as 300×250 remain visible. Missing data is shown as `Requested sizes: Not observed`. | -| `GPT-reported size` | `size` | Exact `slotRenderEnded.size`. A 1×1 placeholder is shown as `GPT-reported size: 1×1 placeholder hidden` and retained unchanged in V1 JSON. | +| `GPT-reported size` | `size` | Exact `slotRenderEnded.size`. A 1×1 placeholder is shown as `GPT-reported size: placeholder hidden` and retained unchanged in V1 JSON. | | `Size filled` / `Measured outer slot size` | `observedSlotSize` | CSS outer box of the exact uniquely bound slot after fill. Missing data is shown as `Size filled: Not observed · Measured outer slot size`. | | `GPT visibility` | current/maximum visibility percentage | Values from GPT visibility callbacks; absence is `GPT visibility: Not observed`. | | `Binding: Bound` / `Bound` | `binding.status=bound` | Exactly one connected publisher element matched. | From f951955f537b89392dcb863fca7a01a5f6fa4846 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 21 Sep 2026 10:21:17 -0500 Subject: [PATCH 087/230] Fix cross-adapter GPT diagnostic timing origins --- .../src/app.rs | 5 +- .../src/middleware.rs | 77 +++++++++++++++++-- crates/trusted-server-adapter-spin/src/app.rs | 4 +- .../src/middleware.rs | 77 +++++++++++++++++-- crates/trusted-server-core/src/publisher.rs | 76 +++++++++++++++--- .../integrations/gpt_diagnostics/overlay.ts | 3 +- .../gpt_diagnostics/overlay.test.ts | 10 +-- docs/guide/integrations/gpt-diagnostics.md | 11 +-- .../2026-08-24-request-phase-timing-design.md | 13 ++-- 9 files changed, 238 insertions(+), 38 deletions(-) diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index a3457ef7c..9aba604b0 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -36,7 +36,9 @@ use trusted_server_core::request_signing::{ }; use trusted_server_core::settings::Settings; -use crate::middleware::{AuthMiddleware, FinalizeResponseMiddleware, SanitizeRequestMiddleware}; +use crate::middleware::{ + AuthMiddleware, FinalizeResponseMiddleware, RequestTimingMiddleware, SanitizeRequestMiddleware, +}; use crate::platform::build_runtime_services; // --------------------------------------------------------------------------- @@ -467,6 +469,7 @@ fn build_router(state: &Arc) -> RouterService { // any middleware registered ahead of it would observe the // shared-secret authentication header. .middleware(SanitizeRequestMiddleware::new(Arc::clone(&state.settings))) + .middleware(RequestTimingMiddleware::new()) .middleware(FinalizeResponseMiddleware::new(Arc::clone(&state.settings))) .middleware(AuthMiddleware::new(Arc::clone(&state.settings))) .get( diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 14efed56a..6b418688b 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -8,6 +8,7 @@ use edgezero_core::middleware::{Middleware, Next}; use trusted_server_core::auth::enforce_basic_auth; use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; use trusted_server_core::http_util::sanitize_trusted_client_ip_headers; +use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; // --------------------------------------------------------------------------- @@ -46,6 +47,40 @@ impl Middleware for SanitizeRequestMiddleware { } } +// --------------------------------------------------------------------------- +// RequestTimingMiddleware +// --------------------------------------------------------------------------- + +/// Attaches the server request clock consumed by core timing instrumentation. +/// +/// This adapter does not emit `Server-Timing`; the collector keeps timing +/// origins consistent for request-scoped consumers such as GPT diagnostics. +/// Health checks remain outside timing collection on every adapter. +#[derive(Default)] +pub struct RequestTimingMiddleware; + +impl RequestTimingMiddleware { + /// Creates a new [`RequestTimingMiddleware`]. + #[must_use] + pub fn new() -> Self { + Self + } +} + +#[async_trait(?Send)] +impl Middleware for RequestTimingMiddleware { + async fn handle(&self, mut ctx: RequestContext, next: Next<'_>) -> Result { + if ctx.request().uri().path() != "/health" + && ctx.request().extensions().get::().is_none() + { + ctx.request_mut() + .extensions_mut() + .insert(RequestTimings::new()); + } + next.run(ctx).await + } +} + // --------------------------------------------------------------------------- // FinalizeResponseMiddleware // --------------------------------------------------------------------------- @@ -56,9 +91,9 @@ impl Middleware for SanitizeRequestMiddleware { /// (injected by the Cloudflare Workers runtime). On the native host target the /// header is absent, so `X-Geo-Info-Available: false` is emitted. /// -/// Registered directly inside [`SanitizeRequestMiddleware`] and ahead of -/// [`AuthMiddleware`] so that every outgoing response — including auth-rejected -/// ones — carries a consistent set of headers. +/// Registered inside [`RequestTimingMiddleware`] and ahead of [`AuthMiddleware`] +/// so that every outgoing response — including auth-rejected ones — carries a +/// consistent set of headers. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -180,10 +215,10 @@ mod tests { .expect("should build empty test response") } - fn empty_ctx() -> RequestContext { + fn ctx_for_path(path: &str) -> RequestContext { let req = request_builder() .method(Method::GET) - .uri("/test") + .uri(path) .header("x-reader-ip", "198.51.100.7") .header("x-reader-ip-auth", "fictional-shared-secret-0123456789") .body(Body::empty()) @@ -191,6 +226,10 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn empty_ctx() -> RequestContext { + ctx_for_path("/test") + } + fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings { // Build from explicit test settings: the settings baked into the // binary contain placeholder secrets that `get_settings()` rejects @@ -289,6 +328,34 @@ mod tests { ); } + #[test] + fn request_timing_middleware_attaches_a_collector_except_for_health() { + for (path, expected) in [("/test", true), ("/health", false)] { + let observed = Arc::new(Mutex::new(None)); + let handler_observed = Arc::clone(&observed); + let handler = Arc::new(move |ctx: RequestContext| { + let handler_observed = Arc::clone(&handler_observed); + async move { + *handler_observed.lock().expect("should lock observation") = + Some(ctx.request().extensions().get::().is_some()); + Ok::(empty_response()) + } + }); + + block_on( + RequestTimingMiddleware::new() + .handle(ctx_for_path(path), Next::new(&[], &*handler)), + ) + .expect("should run timing middleware"); + + assert_eq!( + *observed.lock().expect("should lock observation"), + Some(expected), + "collector presence should match timing policy for {path}" + ); + } + } + #[test] fn sanitize_middleware_strips_configured_trust_headers_before_routing() { let mut settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 8917f2a65..59e7734f2 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -36,7 +36,8 @@ use trusted_server_core::request_signing::{ use trusted_server_core::settings::Settings; use crate::middleware::{ - AuthMiddleware, FinalizeResponseMiddleware, NormalizeMiddleware, SanitizeRequestMiddleware, + AuthMiddleware, FinalizeResponseMiddleware, NormalizeMiddleware, RequestTimingMiddleware, + SanitizeRequestMiddleware, }; use crate::platform::build_runtime_services; @@ -773,6 +774,7 @@ fn build_router(state: &Arc) -> RouterService { // any middleware registered ahead of it would observe the // shared-secret authentication header. .middleware(SanitizeRequestMiddleware::new(Arc::clone(&state.settings))) + .middleware(RequestTimingMiddleware::new()) .middleware(FinalizeResponseMiddleware::new(Arc::clone(&state.settings))) .middleware(AuthMiddleware::new(Arc::clone(&state.settings))) // Innermost middleware: normalize every routed request (strip diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index d7a09987a..d31c6ad7e 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -8,6 +8,7 @@ use edgezero_core::middleware::{Middleware, Next}; use trusted_server_core::auth::enforce_basic_auth; use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; use trusted_server_core::http_util::sanitize_trusted_client_ip_headers; +use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; // --------------------------------------------------------------------------- @@ -46,6 +47,40 @@ impl Middleware for SanitizeRequestMiddleware { } } +// --------------------------------------------------------------------------- +// RequestTimingMiddleware +// --------------------------------------------------------------------------- + +/// Attaches the server request clock consumed by core timing instrumentation. +/// +/// This adapter does not emit `Server-Timing`; the collector keeps timing +/// origins consistent for request-scoped consumers such as GPT diagnostics. +/// Health checks remain outside timing collection on every adapter. +#[derive(Default)] +pub struct RequestTimingMiddleware; + +impl RequestTimingMiddleware { + /// Creates a new [`RequestTimingMiddleware`]. + #[must_use] + pub fn new() -> Self { + Self + } +} + +#[async_trait(?Send)] +impl Middleware for RequestTimingMiddleware { + async fn handle(&self, mut ctx: RequestContext, next: Next<'_>) -> Result { + if ctx.request().uri().path() != "/health" + && ctx.request().extensions().get::().is_none() + { + ctx.request_mut() + .extensions_mut() + .insert(RequestTimings::new()); + } + next.run(ctx).await + } +} + // --------------------------------------------------------------------------- // FinalizeResponseMiddleware // --------------------------------------------------------------------------- @@ -55,9 +90,9 @@ impl Middleware for SanitizeRequestMiddleware { /// Spin does not expose geo headers to the application, so /// `X-Geo-Info-Available: false` is emitted for every response. /// -/// Registered directly inside [`SanitizeRequestMiddleware`] and ahead of -/// [`AuthMiddleware`] so that every outgoing response — including auth-rejected -/// ones — carries a consistent set of headers. +/// Registered inside [`RequestTimingMiddleware`] and ahead of [`AuthMiddleware`] +/// so that every outgoing response — including auth-rejected ones — carries a +/// consistent set of headers. pub struct FinalizeResponseMiddleware { settings: Arc, } @@ -207,10 +242,10 @@ mod tests { .expect("should build empty test response") } - fn empty_ctx() -> RequestContext { + fn ctx_for_path(path: &str) -> RequestContext { let req = request_builder() .method(Method::GET) - .uri("/test") + .uri(path) .header("x-reader-ip", "198.51.100.7") .header("x-reader-ip-auth", "fictional-shared-secret-0123456789") .body(Body::empty()) @@ -218,6 +253,10 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn empty_ctx() -> RequestContext { + ctx_for_path("/test") + } + fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings { // Build from explicit test settings: the settings baked into the // binary contain placeholder secrets that `get_settings()` rejects @@ -316,6 +355,34 @@ mod tests { ); } + #[test] + fn request_timing_middleware_attaches_a_collector_except_for_health() { + for (path, expected) in [("/test", true), ("/health", false)] { + let observed = Arc::new(Mutex::new(None)); + let handler_observed = Arc::clone(&observed); + let handler = Arc::new(move |ctx: RequestContext| { + let handler_observed = Arc::clone(&handler_observed); + async move { + *handler_observed.lock().expect("should lock observation") = + Some(ctx.request().extensions().get::().is_some()); + Ok::(empty_response()) + } + }); + + block_on( + RequestTimingMiddleware::new() + .handle(ctx_for_path(path), Next::new(&[], &*handler)), + ) + .expect("should run timing middleware"); + + assert_eq!( + *observed.lock().expect("should lock observation"), + Some(expected), + "collector presence should match timing policy for {path}" + ); + } + } + #[test] fn sanitize_middleware_strips_configured_trust_headers_before_routing() { let mut settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index c2b6bc6be..52cbc39a8 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -6681,13 +6681,13 @@ pub async fn handle_page_bids( ec_context: &mut EcContext, mut req: Request, ) -> Result, Report> { - // Same defaulted-handle rule as `handle_publisher_request`: a request - // without the extension records into a collector nothing reads. - let timings = req - .extensions() - .get::() - .cloned() - .unwrap_or_default(); + // Keep a local collector for direct-handler callers, but only expose + // browser timings from an adapter-attached request clock. Otherwise an + // adapter that forgets the extension would silently report a different + // handler-entry timing origin. + let request_timings = req.extensions().get::().cloned(); + let has_request_timings = request_timings.is_some(); + let timings = request_timings.unwrap_or_default(); // Adapter fallbacks prepare this before routing. Keep this idempotent call as // a direct-handler safety net and retain the session decision after the @@ -6925,7 +6925,7 @@ pub async fn handle_page_bids( // A successful result proves at least one pending or immediate // provider outcome. Failures can occur before any request leaves // the edge, so they must not fabricate dispatch timing evidence. - if gpt_diagnostics.browser_session_active() { + if gpt_diagnostics.browser_session_active() && has_request_timings { let timing_snapshot = timings.snapshot(); auction_diagnostics = Some(BrowserAuctionDiagnostics { auction_dispatched_ms: timing_snapshot.auction_dispatched_ms, @@ -21087,6 +21087,52 @@ mod tests { ); } + #[tokio::test] + async fn active_page_bids_omits_timings_without_adapter_collector() { + let mut settings = settings_with_co(); + settings.auction.providers = vec![AUCTION_ID_TEST_PROVIDER.to_string()]; + settings + .integrations + .insert_config("gpt_diagnostics", &serde_json::json!({ "enabled": true })) + .expect("should enable diagnostics"); + let slots = article_slot(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"winner".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let orchestrator = + auction_id_test_orchestrator(&settings, Arc::new(Mutex::new(None)), true); + let mut ec_context = consent_allowing_ec_context(); + + let response = handle_page_bids( + &settings, + &services, + None, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &slots, + registry: None, + }, + &mut ec_context, + make_active_page_bids_request("/2024/01/my-article/"), + ) + .await + .expect("should return page-bids response"); + let body: serde_json::Value = serde_json::from_slice( + &response + .into_body() + .into_bytes() + .expect("should read page-bids response body"), + ) + .expect("should serialize page-bids response as JSON"); + + assert!( + body.get("auctionDiagnostics").is_none(), + "missing adapter timing context must not emit handler-local timing facts" + ); + } + #[tokio::test] async fn page_bids_response_includes_auction_id_only_for_winning_bids() { let mut settings = settings_with_co(); @@ -21113,6 +21159,14 @@ mod tests { }, ); + let mut winning_page_bids_request = + make_active_page_bids_request("/2024/01/my-article/"); + let request_timings = RequestTimings::new(); + winning_page_bids_request + .extensions_mut() + .insert(request_timings); + std::thread::sleep(std::time::Duration::from_millis(2)); + let winning_response = handle_page_bids( &settings, &winning_services, @@ -21123,7 +21177,7 @@ mod tests { registry: None, }, &mut ec_context, - make_active_page_bids_request("/2024/01/my-article/"), + winning_page_bids_request, ) .await .expect("should return winning page-bids response"); @@ -21161,6 +21215,10 @@ mod tests { let resolved_ms = timing("auctionResolvedMs"); let committed_ms = timing("auctionCommittedMs"); let wait_ms = timing("auctionWaitMs"); + assert!( + dispatched_ms > 0, + "page-bids dispatch should include time since adapter request entry" + ); assert!( dispatched_ms <= resolved_ms && resolved_ms <= committed_ms, "page-bids auction milestones should be monotonic" diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index 127317035..35aed65a7 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -243,7 +243,8 @@ function cycleFacts(cycle: GptDiagnosticsRequestCycle): string[] { const timingOrigin = cycle.serverAuctionTimingOrigin ?? (cycle.auctionType === 'trusted_server' ? 'spa_auction' : 'navigation'); - const timingAnchor = timingOrigin === 'spa_auction' ? 'SPA page-bids T0' : 'Edge request T0'; + const timingAnchor = + timingOrigin === 'spa_auction' ? 'SPA page-bids request T0' : 'Initial document request T0'; const serverTimings = [ [`${timingAnchor} → auction dispatched`, cycle.serverAuctionTimings.auctionDispatchedMs], [`${timingAnchor} → auction resolved`, cycle.serverAuctionTimings.auctionResolvedMs], diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index cdc4e4aef..c20d6d7f5 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -275,9 +275,9 @@ describe('GptDiagnosticsOverlay', () => { expect(responseSentArticle).toContain('Auction type: SSAT'); expect(responseSentArticle).toContain('Winning bidder: example-bidder'); expect(responseSentArticle).toContain('Winning bid price bucket: 1.20'); - expect(responseSentArticle).toContain('Edge request T0 → auction dispatched 4 ms'); - expect(responseSentArticle).toContain('Edge request T0 → auction resolved 84 ms'); - expect(responseSentArticle).toContain('Edge request T0 → bids committed 85 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → auction dispatched 4 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → auction resolved 84 ms'); + expect(responseSentArticle).toContain('Initial document request T0 → bids committed 85 ms'); expect(responseSentArticle).toContain('Auction wait (in stream) 80 ms'); expect(responseSentArticle).toContain('Opportunity → request 0 ms'); expect(responseSentArticle).toContain('Direct opportunity: Renderable candidate'); @@ -303,8 +303,8 @@ describe('GptDiagnosticsOverlay', () => { const selectedArticle = slotArticle(root!, 'selected-slot').textContent ?? ''; expect(selectedArticle).toContain('Request path: Competing paths'); expect(selectedArticle).toContain('Auction type: Competing auctions'); - expect(selectedArticle).toContain('SPA page-bids T0 → auction dispatched 0 ms'); - expect(selectedArticle).toContain('SPA page-bids T0 → auction resolved 40 ms'); + expect(selectedArticle).toContain('SPA page-bids request T0 → auction dispatched 0 ms'); + expect(selectedArticle).toContain('SPA page-bids request T0 → auction resolved 40 ms'); expect(selectedArticle).toContain('Direct opportunity: Unrenderable candidate'); expect(selectedArticle).toContain('Trusted Server creative request observed at 23 ms'); expect(selectedArticle).not.toContain('Trusted Server markup response sent'); diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md index 0f6c81ce8..d8273174e 100644 --- a/docs/guide/integrations/gpt-diagnostics.md +++ b/docs/guide/integrations/gpt-diagnostics.md @@ -153,11 +153,12 @@ unattributed requests have no auction label because the available evidence does establish an auction implementation. Server auction timing and browser GPT timing use separate clocks and are never -subtracted from each other. Initial SSAT offsets use edge-request T0. SPA TS auction -offsets use a local server clock started when the page-bids handler begins, not the -browser's navigation clock or the edge's request-receipt time. Diagnostics retain that -timing origin separately from the aggregate auction classification, so a request marked -`competing` still labels SPA offsets from SPA page-bids T0. The server facts are: +subtracted from each other. Both initial SSAT and SPA TS auction offsets use the +adapter's server-request clock. Initial offsets belong to the document request, while +SPA offsets belong to the later `/_ts/page-bids` request. Diagnostics retain that +request origin separately from the aggregate auction classification, so a request marked +`competing` still labels SPA offsets from the SPA page-bids request T0. Neither server +clock is the browser's navigation clock. The server facts are: - `auctionDispatchedMs`: bid dispatch offset from that timing origin. - `auctionResolvedMs`: final bid or timeout offset from the same timing origin. diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 61d43e273..0d510971c 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -271,7 +271,8 @@ structurally and its emissions are defined accordingly rather than pretending pa by path match inside the wrapper. - Axum emits the header only; no Tinybird rows in v1 (unchanged). -Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). +Cloudflare and Spin attach the per-request collector so core request-scoped consumers +share the adapter clock, but have no header or telemetry emission wiring in v1. ## 9. Access telemetry row @@ -750,11 +751,11 @@ the auction dataset as the authority on per-bidder duration. ### Scope -- Fastly emits. Axum attaches a collector and records the marks but does not - emit them. Cloudflare and Spin attach no collector at all: `handle_publisher_request` - falls back to `RequestTimings::default()`, so the marks land in a throwaway - handle and are dropped with it. That is pre-existing for every phase, not new - to these marks, and it matches section 8a adapter semantics. +- Fastly emits. Axum, Cloudflare, and Spin attach a collector and record the marks but + do not emit them. Cloudflare and Spin added collection when GPT diagnostics began + exposing request-scoped auction offsets; without an adapter collector those offsets + silently used a different handler-entry clock. This matches section 8a adapter + semantics: collection is portable, while emission remains adapter-specific. - No header emission for any of these values: they are post-hoc analysis fields, and two of the three are typically unknown at the header freeze point in streaming mode. From 58963a80e3e5cad029294c52409e9e4e4b4fbfd1 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 21 Sep 2026 21:26:02 +0530 Subject: [PATCH 088/230] Strengthen the oversized-dimension regression test The fixture used u32::MAX + 1, which truncates to zero under the original as u32 cast and is already caught by the separate zero-dimension check, so the test passed even without the fix. Use u32::MAX + 101 so the test actually fails against the old truncating cast, and log the offending raw value when a width or height is rejected. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 96a2654d2..a2a47c9b1 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -294,13 +294,15 @@ impl AdServerMockProvider { let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid width, skipping" + "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", + bid["w"] ); continue; }; let Some(height) = bid["h"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid height, skipping" + "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", + bid["h"] ); continue; }; @@ -1336,11 +1338,13 @@ mod tests { #[test] fn test_parse_mediation_response_skips_oversized_dimensions() { // A dimension above u32::MAX must be rejected rather than silently - // wrapped into a small, plausible-looking value. + // wrapped into a small, plausible-looking value. The offset of 101 is + // load-bearing: u32::MAX + 1 truncates to 0, which the zero-check + // below would already have caught, so it would not pin this fix. let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); - let oversized_width = u64::from(u32::MAX) + 1; + let oversized_width = u64::from(u32::MAX) + 101; let mediation_response = json!({ "id": "test-auction-123", "seatbid": [ From b70cb57a0af89cd9e6f8400a19a17e965bb3825c Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 00:42:52 +0530 Subject: [PATCH 089/230] Guard the IntegrationSettings debug boundary and correct the test comment claims Adds a canary that pins IntegrationSettings hand-written Debug impl, which is the only thing keeping resolved DataDome credentials out of Settings debug output. Rewrites the leading comment to state what the canary list actually guarantees instead of an enforcement it cannot provide, and adds a positive assertion that a non-secret field stays visible so a blanket-redacting Debug impl would not pass unnoticed. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 46 +++++++++++++++++----- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index d6581f466..dc0504398 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3807,16 +3807,20 @@ mod tests { } // One distinctive canary per `Redacted` field reachable from - // `Settings`'s derived `Debug` impl. A new secret field added without the - // `Redacted` wrapper should be caught here by adding its own canary; - // re-run `rg 'Redacted<' crates/trusted-server-core/src` when touching - // this test to check the field list is still complete. + // `Settings`'s derived `Debug` impl. This is a regression guard over the + // field list below, not a completeness guarantee: a new secret field + // added without the `Redacted` wrapper has no canary here and will pass + // this test while leaking. Adding the canary is a manual step. // - // Do not use `..Struct::default()` anywhere in this function. This test's - // entire purpose is exhaustive field coverage, and a default spread would - // silently swallow any field added to `Handler`, `TinybirdSettings`, or - // any other struct built here, defeating that coverage. List every field - // explicitly instead. + // Integration configs are deliberately out of scope. They reach + // `Settings` as opaque JSON under `IntegrationSettings`, whose + // hand-written `Debug` impl prints only integration IDs, never values. + // + // Do not use `..Struct::default()` anywhere in this function. A default + // spread would let a new secret field be added to `Handler`, + // `TinybirdSettings`, or any other struct built here without forcing + // anyone to consider it. The compile break is the prompt; the canary + // list below is still maintained by hand. List every field explicitly. #[test] fn settings_debug_output_redacts_every_secret_field() { const CANARY_PROXY_SECRET: &str = "CANARY-PROXY-SECRET-0123456789"; @@ -3832,6 +3836,7 @@ mod tests { const CANARY_S3_SESSION_TOKEN: &str = "CANARY-S3-SESSION-TOKEN-0123456789"; const CANARY_TINYBIRD_AUCTION_TOKEN: &str = "CANARY-TINYBIRD-AUCTION-TOKEN-0123456789"; const CANARY_TINYBIRD_ACCESS_TOKEN: &str = "CANARY-TINYBIRD-ACCESS-TOKEN-0123456789"; + const CANARY_DATADOME_SERVER_SIDE_KEY: &str = "CANARY-DATADOME-SERVER-SIDE-KEY-0123456789"; let mut settings = create_test_settings(); @@ -3890,12 +3895,31 @@ mod tests { max_body_bytes: 0, }; + // `IntegrationSettings` stores integration configs as opaque JSON and + // relies on a hand-written `Debug` impl to suppress their values. That + // impl is the only thing keeping resolved DataDome credentials out of + // this output, so pin it here. + settings + .integrations + .insert_config( + "datadome", + &json!({ + "enabled": true, + "server_side_key_secret_name": CANARY_DATADOME_SERVER_SIDE_KEY, + }), + ) + .expect("should insert datadome integration config"); + let debug = format!("{settings:?}"); assert!( debug.contains("[REDACTED]"), "should redact secret fields in Settings debug output" ); + assert!( + debug.contains("^/secure"), + "should leave non-secret handler path visible in debug output" + ); let canaries = [ ("publisher.proxy_secret", CANARY_PROXY_SECRET), @@ -3928,6 +3952,10 @@ mod tests { CANARY_TINYBIRD_AUCTION_TOKEN, ), ("tinybird.access_token_secret", CANARY_TINYBIRD_ACCESS_TOKEN), + ( + "integrations.datadome.server_side_key_secret_name", + CANARY_DATADOME_SERVER_SIDE_KEY, + ), ]; for (field, canary) in canaries { From 57eab4aa28d886736714975f2899952e0b9d04f3 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 01:22:04 +0530 Subject: [PATCH 090/230] Fix placeholder partner secret check to compare resolved values, not key names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit partner_api_token and partner_ts_pull_token are secret-store key names shipped as examples in trusted-server.example.toml, not resolved secret values — reject_placeholder_secrets runs after secret resolution and only ever sees resolved values, matching every other secret field in the template. Removes the two key-name entries from EcPartner::API_TOKEN_PLACEHOLDERS, retargets the affected test at an existing value placeholder, and drops the drift-guard test that asserted the incorrect invariant. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 39 +++------------------- 1 file changed, 5 insertions(+), 34 deletions(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 48560bc57..47f614387 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -443,15 +443,14 @@ pub struct EcPartner { } impl EcPartner { - /// Known partner API token placeholders that must not be used in deployments. + /// Known partner secret placeholders (`api_token` and `ts_pull_token`) that + /// must not be used in deployments. pub const API_TOKEN_PLACEHOLDERS: &[&str] = &[ "partner-api-token-32-bytes-minimum", "replace-with-partner-api-token-32-bytes-minimum", "sharedid-internal-token-32-bytes", "inttest-api-key-1-32-bytes-minimum", "inttest2-api-key-2-32-bytes-minimum", - "partner_api_token", - "partner_ts_pull_token", ]; /// Returns `true` if `api_token` matches a known placeholder value @@ -5441,7 +5440,9 @@ source_domain = "partner.example.com" Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); settings.publisher.proxy_secret = Redacted::new("unit-test-proxy-secret".to_owned()); settings.ec.passphrase = Redacted::new("test-secret-key-32-bytes-minimum".to_owned()); - settings.ec.partners = vec![test_partner_with_pull_token("partner_ts_pull_token")]; + settings.ec.partners = vec![test_partner_with_pull_token( + "partner-api-token-32-bytes-minimum", + )]; let err = settings .reject_placeholder_secrets() @@ -5467,36 +5468,6 @@ source_domain = "partner.example.com" .expect("should accept a realistic partner pull token"); } - /// Guards against the placeholder lists drifting away from the example config - /// Example value shipped in `trusted-server.example.toml` must be recognized - /// as a placeholder - #[test] - fn example_toml_partner_secret_examples_are_recognized_placeholders() { - const EXAMPLE_TOML: &str = include_str!("../../../trusted-server.example.toml"); - const EXAMPLE_PARTNER_KEYS: &[&str] = &["api_token", "ts_pull_token"]; - let pattern = format!( - r#"(?m)^\s*#?\s*(?:{})\s*=\s*"([^"]+)""#, - EXAMPLE_PARTNER_KEYS.join("|") - ); - let example_value = - Regex::new(&pattern).expect("should compile example partner secret regex"); - - let mut checked_any = false; - for capture in example_value.captures_iter(EXAMPLE_TOML) { - checked_any = true; - let value = &capture[1]; - assert!( - EcPartner::is_placeholder_api_token(value), - "example config partner secret '{value}' should be a recognized placeholder \ - — add it to EcPartner::API_TOKEN_PLACEHOLDERS" - ); - } - assert!( - checked_any, - "should have found at least one partner secret example value in the template" - ); - } - #[test] fn is_unusable_store_id_rejects_placeholders_empty_and_padded_values() { for placeholder in RequestSigning::STORE_ID_PLACEHOLDERS { From ffd5bf8ffa8d9e1170362f646f7f94637b368139 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 22 Sep 2026 09:46:25 +0530 Subject: [PATCH 091/230] Rework proxy CLI arg-parsing tests to cover the real clap defaults Removes base_args, the shared test helper that let every test skip the real --listen default; each test now passes its args explicitly through parse_args, so nothing can mask default_value drifting from DEFAULT_LISTEN. Splits the former no_rule_passed_is_a_no_rule_error into bare_invocation_is_rejected_at_parse_time (proves arg_required_else_help rejects a fully-bare ts before resolve runs) and a fixed no_rule_passed_is_a_no_rule_error that uses --insecure so it still exercises the NoRule path, restoring coverage the rewrite had silently dropped. Also fixes two invalid --rewrite-host true/false assertions (that flag takes no value) and corrects the dev-proxy guide, which still described the pre-fix bare-invocation error. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/config.rs | 178 ++++++++++++------ docs/guide/ts-dev-proxy.md | 3 +- 2 files changed, 122 insertions(+), 59 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index 6f4ada02d..f097c8b71 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -359,14 +359,6 @@ mod tests { AddressPolicy, OriginKey, ReferenceIdentity, Transport, VerifyMode, }; - fn base_args() -> crate::commands::dev::proxy::ProxyArgs { - parse_args(&[ - "ts", - "--listen", - crate::commands::dev::proxy::DEFAULT_LISTEN, - ]) - } - fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { #[derive(clap::Parser)] struct W { @@ -378,18 +370,35 @@ mod tests { #[test] fn clap_parses_rewrite_host_as_a_bool() { - assert!(!base_args().rewrite_host, "absent --rewrite-host is false"); + assert!( + !parse_args(&["ts", "--from", "a.example.com", "--to", "b.example.com"]).rewrite_host, + "absent --rewrite-host is false" + ); assert!( parse_args(&["ts", "--rewrite-host"]).rewrite_host, "present --rewrite-host is true" ); } + #[test] + fn clap_applies_the_real_listen_default() { + let args = parse_args(&["ts", "--rewrite-host"]); + assert_eq!( + args.listen, + crate::commands::dev::proxy::DEFAULT_LISTEN, + "should apply the real clap --listen default" + ); + } + #[test] fn single_rule_from_to_keeps_from_host_by_default() { - let mut args = base_args(); - args.from = Some("www.example-publisher.com".into()); - args.to = Some("to.edgecompute.app".into()); + let args = parse_args(&[ + "ts", + "--from", + "www.example-publisher.com", + "--to", + "to.edgecompute.app", + ]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg .rules @@ -405,9 +414,12 @@ mod tests { #[test] fn rewrite_host_uses_to() { - let mut args = base_args(); - args.map = vec!["www.example-publisher.com=to.edgecompute.app".into()]; - args.rewrite_host = true; + let args = parse_args(&[ + "ts", + "--map", + "www.example-publisher.com=to.edgecompute.app", + "--rewrite-host", + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( rewrite_for( @@ -423,10 +435,13 @@ mod tests { #[test] fn resolve_pins_host_to_ip() { - let mut args = base_args(); - args.map = vec!["www.example-publisher.com=ts.edgecompute.app".into()]; - // Mixed case to confirm the host key is lowercased. - args.resolve = vec!["TS.EdgeCompute.app:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "www.example-publisher.com=ts.edgecompute.app", + "--resolve", + "TS.EdgeCompute.app:192.0.2.10", // Mixed case to confirm the host key is lowercased. + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( cfg.resolve.get("ts.edgecompute.app"), @@ -437,10 +452,13 @@ mod tests { #[test] fn resolve_accepts_ipv6_target() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - // Split-on-first-colon must keep the colon-bearing IPv6 address intact. - args.resolve = vec!["b.edgecompute.app:::1".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--resolve", + "b.edgecompute.app:::1", // Split-on-first-colon must keep the colon-bearing IPv6 address intact. + ]); let cfg = resolve(&args).expect("should resolve"); assert_eq!( cfg.resolve.get("b.edgecompute.app"), @@ -451,11 +469,15 @@ mod tests { #[test] fn resolve_host_not_matching_any_rule_warns_but_succeeds() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - // A pin for a host that is no rule's TO is a likely typo: it should warn - // (not error) and still be recorded. - args.resolve = vec!["typo.edgecompute.app:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + // A pin for a host that is no rule's TO is a likely typo: it should warn + // (not error) and still be recorded. + "--resolve", + "typo.edgecompute.app:192.0.2.10", + ]); let cfg = resolve(&args).expect("an unmatched --resolve host should warn, not error"); assert!( cfg.resolve.contains_key("typo.edgecompute.app"), @@ -465,9 +487,13 @@ mod tests { #[test] fn resolve_rejects_malformed_value() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.resolve = vec!["b.edgecompute.app:not-an-ip".into()]; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--resolve", + "b.edgecompute.app:not-an-ip", + ]); let err = resolve(&args).expect_err("a non-IP --resolve target should error"); assert!( matches!(err.current_context(), ConfigError::Resolve { .. }), @@ -477,8 +503,7 @@ mod tests { #[test] fn map_value_must_be_from_equals_to() { - let mut args = base_args(); - args.map = vec!["not-a-map".into()]; + let args = parse_args(&["ts", "--map", "not-a-map"]); assert!(resolve(&args).is_err(), "malformed --map errors"); } @@ -486,11 +511,16 @@ mod tests { fn basic_auth_on_non_loopback_listen_is_rejected() { // Injected Basic auth on a non-loopback bind would expose the upstream // credentials to any reachable network client. - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.listen = "0.0.0.0:18080".into(); - args.allow_non_loopback = true; - args.basic_auth = Some("dev:secret".into()); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + "--basic-auth", + "dev:secret", + ]); let err = resolve(&args).expect_err("non-loopback listen with --basic-auth should be rejected"); assert!( @@ -502,7 +532,14 @@ mod tests { ); // The same non-loopback bind without credentials is allowed. - args.basic_auth = None; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + ]); assert!( resolve(&args).is_ok(), "non-loopback without --basic-auth is allowed" @@ -512,8 +549,7 @@ mod tests { #[test] fn invalid_from_host_is_rejected() { // A FROM with characters that would break the PAC JS / Host header. - let mut args = base_args(); - args.map = vec!["bad\"host=to.edgecompute.app".into()]; + let args = parse_args(&["ts", "--map", "bad\"host=to.edgecompute.app"]); let err = resolve(&args).expect_err("a malformed FROM host should error"); assert!( matches!(err.current_context(), ConfigError::InvalidFrom { .. }), @@ -523,14 +559,25 @@ mod tests { #[test] fn non_loopback_listen_requires_flag() { - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.listen = "0.0.0.0:18080".into(); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + ]); assert!( resolve(&args).is_err(), "non-loopback without flag is rejected" ); - args.allow_non_loopback = true; + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--listen", + "0.0.0.0:18080", + "--allow-non-loopback", + ]); assert!(resolve(&args).is_ok(), "non-loopback allowed with flag"); } @@ -553,11 +600,15 @@ mod tests { #[test] fn resolve_precomputes_typed_rule_identity_and_headers() { - let mut args = base_args(); - args.map = vec!["www.example.com=TO.Example.com:8443".into()]; - args.rewrite_host = true; - args.insecure = true; - args.resolve = vec!["to.example.com:192.0.2.10".into()]; + let args = parse_args(&[ + "ts", + "--map", + "www.example.com=TO.Example.com:8443", + "--rewrite-host", + "--insecure", + "--resolve", + "to.example.com:192.0.2.10", + ]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg @@ -596,10 +647,7 @@ mod tests { #[test] fn resolve_keeps_ip_reference_identities_http1_only() { - let mut args = base_args(); - args.map = vec!["www.example.com=127.0.0.1".into()]; - args.rewrite_host = true; - + let args = parse_args(&["ts", "--map", "www.example.com=127.0.0.1", "--rewrite-host"]); let cfg = resolve(&args).expect("should resolve"); let rule = cfg .rules @@ -649,9 +697,13 @@ mod tests { let dir = tempfile::tempdir().expect("should create temp dir"); let missing = dir.path().join("no-such-file.txt"); - let mut args = base_args(); - args.map = vec!["a.example.com=b.edgecompute.app".into()]; - args.basic_auth_file = Some(missing.to_string_lossy().into_owned()); + let args = parse_args(&[ + "ts", + "--map", + "a.example.com=b.edgecompute.app", + "--basic-auth-file", + &missing.to_string_lossy(), + ]); let err = resolve(&args).expect_err("should fail when file is missing"); assert!( @@ -660,9 +712,19 @@ mod tests { ); } + #[test] + #[should_panic(expected = "DisplayHelpOnMissingArgumentOrSubcommand")] + fn bare_invocation_is_rejected_at_parse_time() { + // `arg_required_else_help` makes a fully-bare `ts` fail to parse at all, + // before `resolve` (and its `NoRule` check) ever runs. + parse_args(&["ts"]); + } + #[test] fn no_rule_passed_is_a_no_rule_error() { - let args = base_args(); + // An invocation with some other flag but no rule still reaches + // `resolve`: `arg_required_else_help` only rejects a fully-bare `ts`. + let args = parse_args(&["ts", "--insecure"]); let err = resolve(&args).expect_err("should error when no rule is passed"); assert!( matches!(err.current_context(), ConfigError::NoRule), diff --git a/docs/guide/ts-dev-proxy.md b/docs/guide/ts-dev-proxy.md index b99603b9e..234385f3d 100644 --- a/docs/guide/ts-dev-proxy.md +++ b/docs/guide/ts-dev-proxy.md @@ -77,7 +77,8 @@ shorthand, or one or more `--map FROM=TO` rules: ts dev proxy -f www.example-publisher.com -t trusted-server-example.edgecompute.app ``` -With no `--map`/`-f`/`-t`, the proxy exits with +A bare `ts dev proxy` prints help and exits before proxy startup. An +invocation with explicit options but no complete rewrite rule reports `no rewrite rule: pass --map FROM=TO (or -f/--from with -t/--to)`. Connection options — `--rewrite-host`, `--basic-auth`/`--basic-auth-file`, From 155d3b09ffb0fe637cf59f6972cf327be5ce7cc9 Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Thu, 24 Sep 2026 10:34:10 +0530 Subject: [PATCH 092/230] Clarify trace body validation and browser cleanup --- ...-mobile-ad-render-trace-endpoint-design.md | 45 +++++++++++++++---- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e3aaca4c6..e12d79760 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -503,9 +503,20 @@ Rules (route responses below apply after configured authentication): - For both POST paths, accept absent or exactly-zero `Content-Length` only when the body is empty; reject `Transfer-Encoding`, positive/invalid lengths, and any actual body bytes with local `413 Payload Too Large` and no mutation. - Precheck headers, then use `Body::into_bytes_bounded(0)` to check emptiness, - following the header-precheck/body-size-check pattern in - `crates/trusted-server-core/src/auction/endpoints.rs`. This is an application + Precheck headers, then explicitly match both body variants: accept + `Body::Once` only when its bytes are empty; for `Body::Stream`, consume chunks + until EOF, skipping empty chunks and rejecting the first non-empty chunk + with a local `413`. A stream read error returns local `400 Bad Request` + without mutation; only a clean EOF proves a streamed body empty. Build the + `StatusCode::PAYLOAD_TOO_LARGE` response explicitly, as in the + header-precheck/body-size-check pattern in + `crates/trusted-server-core/src/auction/endpoints.rs`. Do not copy that + handler's `into_bytes().unwrap_or_default()`: `Body::into_bytes` returns + `None` for streams, which would incorrectly treat a non-empty streamed body + as empty. Nor should `Body::into_bytes_bounded(0)` errors be propagated as + the response: overflow is `EdgeError::bad_request`, which maps to `400`, + and `EdgeError` has no `413` variant. All local body-validation errors use + the section 12.3 response hardening. This is an application acceptance limit, not a transport read or allocation limit. Pinned EdgeZero v0.0.8 buffers the Fastly body with blocking `read_to_end` and the Cloudflare body with `req.bytes().await` before core handling. Spin also buffers the @@ -1424,12 +1435,16 @@ object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self img-src data: ``` -The shell supplies a fixed data-URL favicon; `img-src data:` allows it without -an automatic publisher `/favicon.ico` fetch. All styles live in the fixed CSS -asset: toggle classes or the `hidden` attribute, with no inline style attributes, +The shell declares a fixed data-URL favicon with `` to suppress +the implicit publisher `/favicon.ico` fallback; `img-src data:` permits that +declared data URL to load despite `default-src 'none'`. All styles live in the +fixed CSS asset: toggle classes or the `hidden` attribute, with no inline style attributes, style blocks, or JavaScript style-property writes. JSON download uses a Blob object URL assigned directly to an `` followed by a click, then -revokes the URL after the download has started. Do not fetch the blob URL or +schedules `URL.revokeObjectURL` with `setTimeout` for 1000 ms after the click. +Never revoke synchronously after `click()`; the delay gives the browser time +to acquire the Blob and is not a download-completion signal. Each download +schedules cleanup of its own object URL. Do not fetch the blob URL or embed it in a frame. This fixes the download mechanism without widening `connect-src` or enabling frames; browser tests exercise it under this exact CSP. @@ -1577,8 +1592,11 @@ results, never a prerequisite for returning them. with no claim that inactive proves the cookie is absent. - Empty-body enforcement rejects positive/invalid lengths, transfer encoding, nonempty bodies even with absent/zero lengths, and verifies no mutation on - rejection. Tests must not claim a transport bound or timeout that the pinned - adapters cannot enforce. + rejection. Cover both `Body::Once` and `Body::Stream`, including clean EOF, + empty chunks before EOF or a non-empty chunk, and stream read errors. Assert + local `413` for body bytes and local `400` for read errors, with section 12.3 + hardening and no cookie mutation. Tests must not claim a transport bound or + timeout that the pinned adapters cannot enforce. - Endpoint skips EC generation/finalization, EID ingestion, auction, telemetry, configured filters, ordinary event context, and origin fetch. - Cookie-health scanner covers multiple header fields; zero, one, and duplicate @@ -1646,6 +1664,8 @@ results, never a prerequisite for returning them. shadowing a broad rule to pin first-match-wins behavior. - GET, HEAD, state-changing POST, and unsupported methods obey the same lifecycle contract across adapters. +- Enable/end POSTs without `Content-Type` accept empty bodies and reject actual + body bytes without cookie mutation, including Axum's streaming path. - Every adapter omits JA4/H2 and rejects control characters or overlong platform strings. @@ -1715,6 +1735,9 @@ results, never a prerequisite for returning them. stored copies containing any excluded property are rejected. Numbered slot/cycle correlation still joins after redaction. - Download filename and MIME type are deterministic. +- Fake timers verify that each download clicks its Blob-backed anchor before + scheduling cleanup, never revokes synchronously, and revokes its own URL + when the 1000 ms timer fires, including repeated downloads. - Formatted-JSON copy and JSON-file Web Share success, rejection, absence, and download/copy fallback behavior. - 320-pixel layout, keyboard navigation, focus handling, and accessible status @@ -1722,6 +1745,10 @@ results, never a prerequisite for returning them. ### 14.4 Browser integration tests +- Under the exact section 12.3 CSP, JSON downloads contain the complete report + with deferred URL cleanup, including the storage-failure recovery path and + clipboard/Web Share fallback. The declared data-URL favicon loads without + an implicit `/favicon.ico` request. - First endpoint GET is read-only and shows setup state; a user-initiated, same-origin enable POST sets the session. - Successful in-page activation adds no history entry, so Back can return to From a1710a059c0401447f4ac7e033799d07cbcf0081 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 23 Sep 2026 23:04:58 -0700 Subject: [PATCH 093/230] Pin EdgeZero to an immutable rev and close the review findings Replace the mutable `branch =` pin on all six EdgeZero dependencies with `rev = "12c3215c2637d961a27eefa12e235033fadb4c4a"`, the commit the lockfile already resolved, so `main` stays reproducible and no `cargo update` or upstream force-push can move it. Issue #1195 tracks replacing the rev with the release tag once the upstream PR merges. Assert the legacy physical `ts_secrets` store is absent from both Viceroy configurations, so a squash merge that resurrects the block fails instead of writing entries into a store nothing opens. Correct the `RuntimeStoreConfig::logical` doc comment: the crate does read the process environment elsewhere, so state the real invariant, that no store selector reaches the runtime. Document the Config Store selector in the Fastly guide alongside the secrets one, and drop the leftover instruction not to create a `trusted_server_secrets` link, which the deployment now creates itself. --- Cargo.lock | 16 ++++++++-------- Cargo.toml | 12 ++++++------ crates/trusted-server-adapter-fastly/src/app.rs | 11 ++++++----- .../tests/common/config.rs | 4 ++++ docs/guide/fastly.md | 14 ++++++++++++-- 5 files changed, 36 insertions(+), 21 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d5b777858..0aadf3761 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "serde", "serde_json", @@ -1439,7 +1439,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1490,7 +1490,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-stream", @@ -1519,7 +1519,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-trait", @@ -1546,7 +1546,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "chrono", "clap", @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "anyhow", "async-compression", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?branch=fix%2Ffastly-environment-store-selectors#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" dependencies = [ "log", "proc-macro2", diff --git a/Cargo.toml b/Cargo.toml index 5476d0760..75060c4c7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -55,12 +55,12 @@ cssparser = "0.36" derive_more = { version = "2.0", features = ["display", "error"] } directories = "5" ed25519-dalek = { version = "2.2", features = ["rand_core"] } -edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } -edgezero-cli = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors" } -edgezero-core = { git = "https://github.com/stackpop/edgezero", branch = "fix/fastly-environment-store-selectors", default-features = false } +edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a" } +edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } env_logger = "0.11" error-stack = "0.6" esi = "0.7.2" diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 9f538ad1c..0829045ff 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -162,11 +162,12 @@ pub(crate) struct RuntimeStoreConfig { impl RuntimeStoreConfig { /// Store bindings for the Fastly runtime. /// - /// Fastly Compute has no process environment. `EdgeZero` links each selected - /// physical store to the service version under its logical ID, so the - /// runtime opens stores by logical ID and reads the config entry under - /// that same ID for every publication target. Staging isolation comes from - /// linking a different physical store, never from a different key. + /// Store selection is a deploy-time input; no store selector reaches the + /// Fastly runtime. `EdgeZero` links each selected physical store to the + /// service version under its logical ID, so the runtime opens stores by + /// logical ID and reads the config entry under that same ID for every + /// publication target. Staging isolation comes from linking a different + /// physical store, never from a different key. pub(crate) fn logical() -> Self { Self { config_store_name: StoreName::from(DEFAULT_CONFIG_STORE_ID), diff --git a/crates/trusted-server-integration-tests/tests/common/config.rs b/crates/trusted-server-integration-tests/tests/common/config.rs index 1e83da363..b8eeb0783 100644 --- a/crates/trusted-server-integration-tests/tests/common/config.rs +++ b/crates/trusted-server-integration-tests/tests/common/config.rs @@ -86,6 +86,10 @@ mod tests { local_server["secret_stores"][LOGICAL_SECRET_STORE_ID].is_array(), "{name} should expose the secret store under its logical ID" ); + assert!( + local_server["secret_stores"].get("ts_secrets").is_none(), + "{name} should not define the legacy physical secret store" + ); assert!( local_server["config_stores"] .get("edgezero_runtime_env") diff --git a/docs/guide/fastly.md b/docs/guide/fastly.md index cfd3df15a..4ec64ebef 100644 --- a/docs/guide/fastly.md +++ b/docs/guide/fastly.md @@ -271,6 +271,16 @@ Used for storing public configuration (e.g., public keys, key metadata): fastly config-store create --name jwks_store ``` +Trusted Server's app config lives under logical store ID +`trusted_server_config`. Select its physical store in each deployment +environment, the same way as the secret store below. Production and staging +isolate their app config by selecting different physical stores, because the +config entry key is the logical ID on every target: + +```bash +export EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME= +``` + ### Secret Stores Trusted Server keeps static app-config credentials under logical store ID @@ -312,8 +322,8 @@ Create the separate request-signing store when that feature is enabled: fastly secret-store create --name signing_keys ``` -Do not copy the same app credential store under a second hardcoded -`trusted_server_secrets` Fastly link. Configure the mapping instead. +The deployment creates the `trusted_server_secrets` link itself from the +selected physical store. Do not create that link by hand. ## Create EC KV Store From bf96305b9d327b7927eb918361f9eb30018e0d1e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 13:50:42 +0530 Subject: [PATCH 094/230] Pin adserver mock bid dimension boundaries in tests Cover zero height separately from missing dimensions, accept u32::MAX exactly, skip negative dimensions, rename the shared oversized dimension variable, and document why the raw upstream value is logged with Debug formatting. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 130 +++++++++++++++++- 1 file changed, 124 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index a2a47c9b1..066e32e28 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -292,6 +292,8 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); + // Keep `{:?}` for the raw upstream values below: Debug escapes + // newlines and quotes, which prevents log injection. let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", @@ -1344,7 +1346,7 @@ mod tests { let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); - let oversized_width = u64::from(u32::MAX) + 101; + let oversized_dimension = u64::from(u32::MAX) + 101; let mediation_response = json!({ "id": "test-auction-123", "seatbid": [ @@ -1356,7 +1358,7 @@ mod tests { "impid": "header-banner", "price": 3.50, "adm": "
Oversized width
", - "w": oversized_width, + "w": oversized_dimension, "h": 90, }, { @@ -1365,7 +1367,7 @@ mod tests { "price": 1.25, "adm": "
Oversized height
", "w": 300, - "h": oversized_width, + "h": oversized_dimension, }, { "id": "bid-valid", @@ -1415,9 +1417,17 @@ mod tests { "h": 90, }, { - "id": "bid-missing-dimensions", + "id": "bid-zero-height", "impid": "sidebar", "price": 1.25, + "adm": "
Zero height
", + "w": 300, + "h": 0, + }, + { + "id": "bid-missing-dimensions", + "impid": "skyscraper", + "price": 1.10, "adm": "
Missing dimensions
", }, { @@ -1434,15 +1444,123 @@ mod tests { "cur": "USD" }); + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + let slots: Vec<&str> = auction_response + .bids + .iter() + .map(|bid| bid.slot_id.as_str()) + .collect(); + assert_eq!( + slots, + ["footer"], + "should drop the zero-width, zero-height, and missing-dimension bids" + ); + } + + #[test] + fn test_parse_mediation_response_accepts_u32_max_dimensions() { + // u32::MAX is the largest representable dimension and must be accepted, + // pinning the upper boundary of the oversized-dimension check. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-max-dimensions", + "impid": "header-banner", + "price": 3.50, + "adm": "
Max dimensions
", + "w": u32::MAX, + "h": u32::MAX, + } + ] + } + ], + "cur": "USD" + }); + let auction_response = provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); assert_eq!( auction_response.bids.len(), 1, - "Bids with zero or missing w/h should be skipped, only the valid bid should remain" + "should accept a bid whose w/h equal u32::MAX" + ); + assert_eq!( + auction_response.bids[0].width, + u32::MAX, + "should keep width at u32::MAX" + ); + assert_eq!( + auction_response.bids[0].height, + u32::MAX, + "should keep height at u32::MAX" + ); + } + + #[test] + fn test_parse_mediation_response_skips_negative_dimensions() { + // Negative dimensions are not valid u64 values and must be skipped. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-negative-width", + "impid": "header-banner", + "price": 3.50, + "adm": "
Negative width
", + "w": -1, + "h": 90, + }, + { + "id": "bid-negative-height", + "impid": "sidebar", + "price": 1.25, + "adm": "
Negative height
", + "w": 300, + "h": -1, + }, + { + "id": "bid-valid", + "impid": "footer", + "price": 2.00, + "adm": "
Valid Ad
", + "w": 728, + "h": 90, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + let slots: Vec<&str> = auction_response + .bids + .iter() + .map(|bid| bid.slot_id.as_str()) + .collect(); + assert_eq!( + slots, + ["footer"], + "should drop the negative-width and negative-height bids" ); - assert_eq!(auction_response.bids[0].slot_id, "footer"); } #[test] From 8dd9713b3fcf30cba7b9615e6486b8a9c199e8a0 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 14:01:15 +0530 Subject: [PATCH 095/230] Assert bare proxy invocation via clap's typed error kind Add a fallible try_parse_args test helper and check error.kind() against DisplayHelpOnMissingArgumentOrSubcommand instead of matching a should_panic substring from clap's private ErrorInner Debug output. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/config.rs | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs index f097c8b71..86ffc9d0f 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/config.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/config.rs @@ -360,12 +360,18 @@ mod tests { }; fn parse_args(argv: &[&str]) -> crate::commands::dev::proxy::ProxyArgs { + try_parse_args(argv).expect("should parse proxy args") + } + + fn try_parse_args( + argv: &[&str], + ) -> Result { #[derive(clap::Parser)] struct W { #[command(flatten)] a: crate::commands::dev::proxy::ProxyArgs, } - W::try_parse_from(argv).expect("should parse proxy args").a + W::try_parse_from(argv).map(|w| w.a) } #[test] @@ -713,11 +719,16 @@ mod tests { } #[test] - #[should_panic(expected = "DisplayHelpOnMissingArgumentOrSubcommand")] fn bare_invocation_is_rejected_at_parse_time() { // `arg_required_else_help` makes a fully-bare `ts` fail to parse at all, // before `resolve` (and its `NoRule` check) ever runs. - parse_args(&["ts"]); + let error = try_parse_args(&["ts"]) + .expect_err("a fully-bare invocation should short-circuit to help"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::DisplayHelpOnMissingArgumentOrSubcommand, + "should short-circuit to help rather than reaching resolve" + ); } #[test] From d100d1b3c87755d5af42ab106a904ebe565e2698 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 24 Sep 2026 14:01:15 +0530 Subject: [PATCH 096/230] Resolve proxy rules before restoring pending system proxy state An invocation with flags but no usable rewrite rule (for example ts dev proxy --insecure) previously ran the Safari system-proxy restore, and could attempt sudo, before failing on the missing rule. Resolve the config first so it fails without touching system proxy state; a proxy stranded by a hard-killed run is still restored on the next valid run. Add a regression test using a malformed restore file, which the restore path deletes without running networksetup, and soften the guide so it no longer promises the concise one-line error output. Signed-off-by: dhruv8sh --- .../src/commands/dev/proxy/browser.rs | 2 +- .../src/commands/dev/proxy/mod.rs | 50 ++++++++++++++++--- docs/guide/ts-dev-proxy.md | 4 +- 3 files changed, 47 insertions(+), 9 deletions(-) diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs b/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs index 4af9b2d3d..e61f92a01 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/browser.rs @@ -18,7 +18,7 @@ use crate::output; /// is `on` or `off`. A missing third line is tolerated when reading (treated as /// `on` if a URL is present, else `off`) for forward-compatibility with the /// earlier two-line format. -const SAFARI_RESTORE_FILE: &str = "safari-proxy-restore"; +pub(super) const SAFARI_RESTORE_FILE: &str = "safari-proxy-restore"; /// Generates a PAC script that proxies only `https://` requests for matched FROM hosts. /// diff --git a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs index 6ec9e497d..8bb28118a 100644 --- a/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs +++ b/crates/trusted-server-cli/src/commands/dev/proxy/mod.rs @@ -252,14 +252,19 @@ pub fn run(args: &ProxyArgs) -> core::result::Result<(), error_stack::Report::try_parse_from(["ts", "--insecure", "--ca-dir", &ca_dir]) + .expect("should parse proxy args") + .a; + + let err = run(&args).expect_err("should fail without a rewrite rule"); + + assert!( + matches!(err.current_context(), ProxyError::Config), + "should fail with a config error" + ); + assert!( + restore_path.exists(), + "should fail before attempting to restore the system proxy" + ); + } } diff --git a/docs/guide/ts-dev-proxy.md b/docs/guide/ts-dev-proxy.md index 234385f3d..789393e92 100644 --- a/docs/guide/ts-dev-proxy.md +++ b/docs/guide/ts-dev-proxy.md @@ -78,8 +78,8 @@ ts dev proxy -f www.example-publisher.com -t trusted-server-example.edgecompute. ``` A bare `ts dev proxy` prints help and exits before proxy startup. An -invocation with explicit options but no complete rewrite rule reports -`no rewrite rule: pass --map FROM=TO (or -f/--from with -t/--to)`. +invocation with explicit options but no complete rewrite rule fails with a +`no rewrite rule` error before touching system proxy state. Connection options — `--rewrite-host`, `--basic-auth`/`--basic-auth-file`, `--insecure`, and `--upstream-plaintext` — apply to every mapping, not per-rule. From 0fa7dfb649da04050b9185bde34137a1e0696b2e Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 24 Sep 2026 19:38:49 -0500 Subject: [PATCH 097/230] Clarify Prebid diagnostics export boundaries --- .../src/integrations/gpt_diagnostics/store.ts | 2 ++ .../gpt_diagnostics/store.test.ts | 24 +++++++++++++++++++ .../gpt-diagnostics-dictionary.md | 10 ++++++++ docs/guide/integrations/gpt-diagnostics.md | 22 +++++++++++++---- 4 files changed, 54 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index fd32b9c26..69c87abea 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -1101,6 +1101,8 @@ export class GptDiagnosticsStore { this.nextRequestIntentId += 1; this.pendingRequestIntents.set(slot, intent); } + // Replace rather than merge: a later bare refresh must not carry an + // earlier auction's evidence into the next GPT request. intent.sources.set(source, { observedAtMs, ...facts }); } diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts index 22c9760c1..64b4ecfa4 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts @@ -710,6 +710,30 @@ describe('GptDiagnosticsStore', () => { }); }); + it('does not carry an earlier auction into a later bare Prebid refresh', () => { + let now = 10; + const store = new GptDiagnosticsStore({ now: () => now, defer: () => undefined }); + const slot = fakeSlot('prebid-replaced-intent'); + store.recordPrebidRefresh([slot]); + store.recordPrebidAuction(slot, 'auction-client-1', { + bidder: 'example-client', + priceBucket: '2.40', + }); + + now = 20; + store.recordPrebidRefresh([slot]); + store.recordSlotRequested(slot); + store.recordPrebidWin(slot, 'auction-client-1', { + bidder: 'example-client', + priceBucket: '2.40', + }); + + const cycle = store.snapshot().slots[0].requests[0]; + expect(cycle.requestPath).toBe('prebid_refresh'); + expect(cycle.auctionType).toBeUndefined(); + expect(cycle.prebidAuction).toBeUndefined(); + }); + it('retains bounded winner and server timing facts for a Trusted Server auction', () => { const store = new GptDiagnosticsStore({ now: () => 10, defer: () => undefined }); const slot = fakeSlot('auction-facts'); diff --git a/docs/guide/integrations/gpt-diagnostics-dictionary.md b/docs/guide/integrations/gpt-diagnostics-dictionary.md index e0dc433a6..2901cbb64 100644 --- a/docs/guide/integrations/gpt-diagnostics-dictionary.md +++ b/docs/guide/integrations/gpt-diagnostics-dictionary.md @@ -65,6 +65,16 @@ Panel status is `GPT observed` or `Waiting for GPT`. Filter values are `All`, `V Bidder names are limited to 128 UTF-8 bytes, numeric bucket strings to 64 bytes, currencies to three ASCII letters, and auction IDs to 256 UTF-8 bytes. Prebid supplies its own auction ID through `bidsBackHandler`; diagnostics does not override Prebid auction identity. Duplicate, ambiguous, expired, late, prior-navigation, and malformed observations are rejected. Only an active diagnostics recorder installs the `bidWon` listener or retains candidate/win state, which is bounded to 128 pending attempts and 30 seconds. No raw CPM, creative markup, targeting dump, or losing bids are retained. +First-party GPT and Prebid integrations never populate `currency`. The optional field +accepts currency supplied through the internal `gptDiagnosticsRecorder` channel; that +channel is not a supported operator API. Diagnostics does not infer a currency or read +`hb_cur`, and missing currency adds no suffix to the price bucket. + +`Auction not observed` also covers the Prebid watchdog fallback: targeting can be +applied even when Prebid never invokes `bidsBackHandler`, so no completed-auction ID +is available to correlate with the GPT request. Targeting alone is not proof of a +completed auction. + ## Request paths and opportunities | Label | Raw value | Meaning | diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md index 7b8345400..1928c1d6c 100644 --- a/docs/guide/integrations/gpt-diagnostics.md +++ b/docs/guide/integrations/gpt-diagnostics.md @@ -494,6 +494,13 @@ The allowlisted export contains: timestamps, and safe failure enums. - The bounded winning bidder and bucketed price plus server auction timing fields and their `navigation` or `spa_auction` origin when direct auction evidence was observed. +- Completed client-side Prebid evidence (`prebidAuction`): Prebid's opaque `auctionId`, + plus bounded bidder and bucketed price fields in `targetingCandidate` and `win`, + when a completed attempt and any documented `bidWon` observation were correlated + to that exact request. Neither candidate nor win evidence proves the final served bidder. +- Optional `currency` on server winner, Prebid candidate, and Prebid win evidence, + when supplied through the internal recorder. Values are normalized to three uppercase + ASCII letters; first-party integrations do not populate this field. - The per-auction diagnostics token (`trustedServerAuctionId`) and the opportunity-to-request duration, when a direct opportunity was observed. - Replacement facts for a re-rendered slot: `replacedRequestNumber`, @@ -504,10 +511,11 @@ The allowlisted export contains: - Separate callback issues, attribution issues, coverage counters, and retention counters. -It does not contain raw targeting, bid IDs, exact unbucketed bid prices, losing bidder -identity, creative markup, cache URLs, cache payloads, cache or bridge error details, -cookies, user identifiers, query strings, or URL fragments. It does contain the winning -bidder and bucketed `hb_pb` value described above. The exported `trustedServerAuctionId` +It does not contain raw targeting, bid IDs, exact unbucketed bid prices, losing bid +lists, creative markup, cache URLs, cache payloads, cache or bridge error details, +cookies, user identifiers, query strings, or URL fragments. It does contain the server +winner and the Prebid candidate and win bidder names with their bucketed `hb_pb` values +as described above, even when the final served bidder is unconfirmed. The exported `trustedServerAuctionId` is the `hb_auction_id` value described in [Auction correlation token](#auction-correlation-token): minted fresh for each server-side auction, not derived from the Edge Cookie ID or any other visitor @@ -515,6 +523,12 @@ identifier, and never repeated across auctions, so it cannot be joined back to a visitor. Diagnostics retain it only after trimming to a non-empty value of at most 256 UTF-8 bytes. +The exported `prebidAuction.auctionId` is supplied by Prebid for auction correlation, +not visitor identity. Diagnostics does not generate or replace it and retains only +non-empty values of at most 256 UTF-8 bytes. Bidder names are limited to 128 UTF-8 +bytes and numeric price bucket strings to 64 bytes. These bounds limit retained data; +they do not establish how a publisher's Prebid configuration generated its auction ID. + Captured records are memory-only. Diagnostics do not add an upload, diagnostics network request, `localStorage`, `sessionStorage`, IndexedDB, or other persistence. `export()` creates only the user-requested local JSON download; it sends nothing to a From 64eb5f69f4350846e5b74006a4a306457f1fb03e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:18 +0530 Subject: [PATCH 098/230] Add design spec and plan for git version in x-ts-version Signed-off-by: dhruv8sh --- .../plans/2026-09-25-git-version-header.md | 959 ++++++++++++++++++ .../2026-09-25-git-version-header-design.md | 385 +++++++ 2 files changed, 1344 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-25-git-version-header.md create mode 100644 docs/superpowers/specs/2026-09-25-git-version-header-design.md diff --git a/docs/superpowers/plans/2026-09-25-git-version-header.md b/docs/superpowers/plans/2026-09-25-git-version-header.md new file mode 100644 index 000000000..43b666403 --- /dev/null +++ b/docs/superpowers/plans/2026-09-25-git-version-header.md @@ -0,0 +1,959 @@ +# Trusted Server: `x-ts-version` = git version, `x-ts-fastly-version` = Fastly version — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Responses carry `x-ts-version` = the deployed git tag, else branch, else +6-char commit hash. The Fastly service version moves to `x-ts-fastly-version`. + +**Architecture:** `trusted-server-core/build.rs` compiles the version in as +`TS_GIT_VERSION`. It takes the value from the build-time env +`TRUSTED_SERVER_GIT_VERSION` (set by the deploy pipeline), or falls back to local `git`. +The choice itself is a pure function in `build_support/git_version.rs`, shared by +`build.rs` and an integration test through `#[path]`. A core module +`version_header` owns the header value and the write. Every adapter's +`apply_finalize_headers` calls it before operator `response_headers` are +applied, and the Fastly `/health` fast path sets it directly. + +**Tech Stack:** Rust 2024, Cargo build scripts, `edgezero_core::http`, `fastly` 0.12, Viceroy (`cargo test-fastly`). + +**Spec:** [`docs/superpowers/specs/2026-09-25-git-version-header-design.md`](../specs/2026-09-25-git-version-header-design.md) + +## Global Constraints + +- Build env var: exactly `TRUSTED_SERVER_GIT_VERSION`. Compiled-in rustc env: exactly `TS_GIT_VERSION`. +- Resolution order: usable env override → exact tag → branch → first **6** chars of the commit → none. +- Usable = trimmed, non-empty, and only bytes `0x21`–`0x7E`. An unusable override prints a `cargo:warning` and falls back to local git. +- No version known: **omit** `x-ts-version`. Never emit `unknown`. +- `build.rs` must never fail the build over version metadata. +- `build.rs` must print `cargo:rerun-if-env-changed=TRUSTED_SERVER_GIT_VERSION`, and `rerun-if-changed` only for git paths that exist (`/HEAD`, the current branch's ref file under ``, `/refs/tags`, `/packed-refs`). +- Header names: `x-ts-version` (git), `x-ts-fastly-version` (the `FASTLY_SERVICE_VERSION` value). Fastly `/health` gets `x-ts-version` only. +- Operator `settings.response_headers` still apply last. +- Test messages use the repo's `"should …"` style. Run the relevant `cargo test-*` / `cargo clippy-*` alias per task. +- Commits: sentence case, imperative, no prefixes. + +--- + +### Task 1: Version resolution in `build.rs` + +**Files:** + +- Create: `crates/trusted-server-core/build_support/git_version.rs` +- Modify: `crates/trusted-server-core/build.rs` +- Test: `crates/trusted-server-core/tests/git_version_resolve.rs` + +**Interfaces:** + +- Produces: `git_version::is_usable(value: &str) -> bool` and + `git_version::resolve_git_version(candidates: &Candidates<'_>) -> Option`, + where `Candidates { override_value, exact_tag, branch, commit }` are all `Option<&str>`. +- Produces: the compile-time env `TS_GIT_VERSION`, set only when a version is known. + +- [ ] **Step 1: Write the failing test** + +`crates/trusted-server-core/tests/git_version_resolve.rs`: + +```rust +//! The version-resolution rule `build.rs` uses for `TS_GIT_VERSION`. + +#[path = "../build_support/git_version.rs"] +mod git_version; + +use git_version::{Candidates, resolve_git_version}; + +const COMMIT: &str = "abcdef0123456789abcdef0123456789abcdef01"; + +fn local(exact_tag: Option<&'static str>, branch: Option<&'static str>) -> Candidates<'static> { + Candidates { + override_value: None, + exact_tag, + branch, + commit: Some(COMMIT), + } +} + +#[test] +fn override_wins_over_local_git() { + let candidates = Candidates { + override_value: Some("v1.2.3"), + ..local(Some("v9.9.9"), Some("main")) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("v1.2.3"), + "should prefer the pipeline-supplied TRUSTED_SERVER_GIT_VERSION" + ); +} + +#[test] +fn override_is_trimmed() { + let candidates = Candidates { + override_value: Some(" feature/x\n"), + ..local(None, None) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("feature/x"), + "should trim surrounding whitespace from the override" + ); +} + +#[test] +fn tag_wins_over_branch() { + assert_eq!( + resolve_git_version(&local(Some("v1.2.3"), Some("main"))).as_deref(), + Some("v1.2.3"), + "should prefer an exact tag over the branch" + ); +} + +#[test] +fn branch_when_no_tag() { + assert_eq!( + resolve_git_version(&local(None, Some("feature/x"))).as_deref(), + Some("feature/x"), + "should use the branch when HEAD is not tagged" + ); +} + +#[test] +fn six_char_hash_when_no_tag_or_branch() { + assert_eq!( + resolve_git_version(&local(None, None)).as_deref(), + Some("abcdef"), + "should use exactly the first 6 characters of the commit" + ); +} + +#[test] +fn blank_candidates_are_skipped() { + let candidates = Candidates { + override_value: Some(" "), + exact_tag: Some(""), + branch: Some(" "), + commit: Some(COMMIT), + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("abcdef"), + "should skip empty or whitespace-only candidates" + ); +} + +#[test] +fn non_ascii_override_falls_back_to_local_git() { + let candidates = Candidates { + override_value: Some("v1-été"), + ..local(None, None) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("abcdef"), + "should skip a non-ASCII override and fall back to local git" + ); +} + +#[test] +fn override_with_inner_space_falls_back_to_local_git() { + let candidates = Candidates { + override_value: Some("v1 2"), + ..local(None, Some("main")) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("main"), + "should skip an override containing a space" + ); +} + +#[test] +fn none_when_nothing_is_known() { + let candidates = Candidates { + override_value: None, + exact_tag: None, + branch: None, + commit: None, + }; + assert_eq!( + resolve_git_version(&candidates), + None, + "should leave the version unset so the header is omitted" + ); +} + +#[test] +fn is_usable_accepts_only_visible_ascii() { + assert!(git_version::is_usable("v1.2.3"), "should accept a tag"); + assert!(git_version::is_usable(" main "), "should accept after trimming"); + assert!(!git_version::is_usable(""), "should reject empty"); + assert!(!git_version::is_usable("a\tb"), "should reject a control character"); + assert!(!git_version::is_usable("v1-été"), "should reject non-ASCII"); +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p trusted-server-core --test git_version_resolve --target wasm32-wasip1` +Expected: FAIL to compile, with `couldn't read …/build_support/git_version.rs`. + +- [ ] **Step 3: Implement the pure function** + +`crates/trusted-server-core/build_support/git_version.rs`: + +```rust +//! Resolution rule for the deployed git version reported in `x-ts-version`. +//! +//! Shared by `build.rs` and `tests/git_version_resolve.rs` via `#[path]`, so it +//! must stay dependency-free. + +/// Raw candidates for the deployed git version, in priority order. +pub struct Candidates<'a> { + /// `TRUSTED_SERVER_GIT_VERSION`, supplied by the deploy pipeline. + pub override_value: Option<&'a str>, + /// `git describe --tags --exact-match`. + pub exact_tag: Option<&'a str>, + /// `git symbolic-ref --short -q HEAD`. + pub branch: Option<&'a str>, + /// `git rev-parse HEAD`. + pub commit: Option<&'a str>, +} + +/// Whether `value`, once trimmed, is non-empty visible ASCII (`0x21`–`0x7E`). +/// +/// Stricter than git, which also allows non-ASCII UTF-8 in ref names. Such a +/// ref is skipped so every compiled-in value is a `to_str()`-able header value. +pub fn is_usable(value: &str) -> bool { + let trimmed = value.trim(); + !trimmed.is_empty() && trimmed.bytes().all(|b| (0x21..=0x7E).contains(&b)) +} + +/// Picks the first usable candidate: override, tag, branch, then the first 6 +/// characters of the commit. `None` when nothing usable is known. +pub fn resolve_git_version(candidates: &Candidates<'_>) -> Option { + let usable = |value: Option<&str>| value.filter(|v| is_usable(v)).map(str::trim); + + usable(candidates.override_value) + .or_else(|| usable(candidates.exact_tag)) + .or_else(|| usable(candidates.branch)) + .map(str::to_owned) + .or_else(|| usable(candidates.commit).map(|c| c.chars().take(6).collect())) +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cargo test -p trusted-server-core --test git_version_resolve --target wasm32-wasip1` +Expected: 10 passed. + +- [ ] **Step 5: Wire up `build.rs`** + +Replace `crates/trusted-server-core/build.rs` with: + +```rust +//! Compiles the deployed git version into `trusted-server-core` as `TS_GIT_VERSION`. + +#[path = "build_support/git_version.rs"] +mod git_version; + +use std::path::Path; +use std::process::Command; + +use git_version::{Candidates, is_usable, resolve_git_version}; + +/// Set by the deploy pipeline. Its CI checkout is shallow and detached, so local git +/// cannot see the tag or branch there. +const OVERRIDE_ENV: &str = "TRUSTED_SERVER_GIT_VERSION"; + +/// Runs `git` in the crate directory; `None` if git is missing or fails. +fn git(args: &[&str]) -> Option { + let output = Command::new("git").args(args).output().ok()?; + if !output.status.success() { + return None; + } + String::from_utf8(output.stdout) + .ok() + .map(|s| s.trim().to_owned()) +} + +/// Re-runs this script when `path` changes. Skips missing paths: Cargo treats +/// them as always changed, which would rebuild core on every build. +fn rerun_if_exists(path: &Path) { + if path.exists() { + println!("cargo:rerun-if-changed={}", path.display()); + } +} + +/// Resolves the version from local git, watching the paths that move with it. +fn resolve_from_local_git() -> Option { + // HEAD is per-worktree; refs are shared in the common dir. They differ in a + // linked worktree and coincide in a plain clone. + if let Some(git_dir) = git(&["rev-parse", "--absolute-git-dir"]) { + rerun_if_exists(&Path::new(&git_dir).join("HEAD")); + } + if let Some(common_dir) = git(&["rev-parse", "--path-format=absolute", "--git-common-dir"]) { + let common_dir = Path::new(&common_dir); + if let Some(branch_ref) = git(&["symbolic-ref", "-q", "HEAD"]) { + rerun_if_exists(&common_dir.join(branch_ref)); + } + rerun_if_exists(&common_dir.join("refs/tags")); + rerun_if_exists(&common_dir.join("packed-refs")); + } + + let exact_tag = git(&["describe", "--tags", "--exact-match"]); + let branch = git(&["symbolic-ref", "--short", "-q", "HEAD"]); + let commit = git(&["rev-parse", "HEAD"]); + resolve_git_version(&Candidates { + override_value: None, + exact_tag: exact_tag.as_deref(), + branch: branch.as_deref(), + commit: commit.as_deref(), + }) +} + +fn main() { + println!("cargo:rerun-if-changed=build.rs"); + println!("cargo:rerun-if-changed=build_support/git_version.rs"); + println!("cargo:rerun-if-env-changed={OVERRIDE_ENV}"); + + let override_value = std::env::var(OVERRIDE_ENV).ok(); + let resolved = match override_value.as_deref() { + Some(value) if is_usable(value) => resolve_git_version(&Candidates { + override_value: Some(value), + exact_tag: None, + branch: None, + commit: None, + }), + Some(value) => { + if !value.trim().is_empty() { + println!( + "cargo:warning={OVERRIDE_ENV}={value:?} is not visible ASCII; \ + falling back to local git for x-ts-version" + ); + } + resolve_from_local_git() + } + None => resolve_from_local_git(), + }; + + if let Some(version) = resolved { + println!("cargo:rustc-env=TS_GIT_VERSION={version}"); + } +} +``` + +- [ ] **Step 6: Verify the override reaches the compile and re-runs on change** + +Run: +`TRUSTED_SERVER_GIT_VERSION=v0.0.0-check cargo build -p trusted-server-core --target wasm32-wasip1 -vv 2>&1 | grep -F 'TS_GIT_VERSION=v0.0.0-check'` +Expected: one matching `cargo:rustc-env=TS_GIT_VERSION=v0.0.0-check` line. + +Run the same command again with `v0.0.0-check2`. Expected: the build script +re-runs and prints `…check2`. + +Run: `cargo build -p trusted-server-core --target wasm32-wasip1 -vv 2>&1 | grep -F 'cargo:rerun-if-changed='` +Expected: `…/.git/worktrees//HEAD` and `…/.git/refs` (when in a linked +worktree), no `packed-refs` line unless that file exists. + +- [ ] **Step 7: Clippy and commit** + +Run: `cargo clippy-fastly` +Expected: no warnings. + +```bash +git add crates/trusted-server-core/build.rs crates/trusted-server-core/build_support/git_version.rs crates/trusted-server-core/tests/git_version_resolve.rs +git commit --signoff -S -m "Compile deployed git version into trusted-server-core as TS_GIT_VERSION" +``` + +--- + +### Task 2: Core constants and the `version_header` module + +**Files:** + +- Modify: `crates/trusted-server-core/src/constants.rs` (the `// Staging / version identification headers` block) +- Create: `crates/trusted-server-core/src/version_header.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` (add `pub mod version_header;` after `pub mod tsjs;`) + +**Interfaces:** + +- Consumes: `TS_GIT_VERSION` compile-time env (Task 1). +- Produces: `constants::HEADER_X_TS_FASTLY_VERSION: HeaderName`, `constants::TS_GIT_VERSION: Option<&'static str>`. +- Produces: `version_header::git_version_header_value() -> Option`, + `version_header::header_value_from(version: Option<&str>) -> Option`, + `version_header::apply_git_version_header(response: &mut Response)`, + `version_header::apply_git_version_header_from(version: Option<&str>, response: &mut Response)`. + `HeaderValue`/`Response` are `edgezero_core::http`'s. + +- [ ] **Step 1: Write the failing tests** + +In `constants.rs`, replace the block under `// Staging / version identification headers` with: + +```rust +// Staging / version identification headers +/// Deployed git version: tag, else branch, else 6-char commit (see [`TS_GIT_VERSION`]). +pub const HEADER_X_TS_VERSION: HeaderName = HeaderName::from_static("x-ts-version"); +/// Fastly service version (`FASTLY_SERVICE_VERSION`), formerly sent as `x-ts-version`. +pub const HEADER_X_TS_FASTLY_VERSION: HeaderName = HeaderName::from_static("x-ts-fastly-version"); +pub const HEADER_X_TS_ENV: HeaderName = HeaderName::from_static("x-ts-env"); + +/// Deployed git version compiled in by `build.rs`, from the deploy pipeline's +/// `TRUSTED_SERVER_GIT_VERSION` or local git. `None` when unknown. +pub const TS_GIT_VERSION: Option<&str> = option_env!("TS_GIT_VERSION"); +``` + +Create `crates/trusted-server-core/src/version_header.rs`: + +```rust +//! `x-ts-version`: the deployed git version compiled in by `build.rs`. + +use edgezero_core::http::{HeaderValue, Response}; + +use crate::constants::{HEADER_X_TS_VERSION, TS_GIT_VERSION}; + +/// Returns the compiled-in git version as a header value. +/// +/// `None` when no version is known or the value is not a valid header value. +#[must_use] +pub fn git_version_header_value() -> Option { + header_value_from(TS_GIT_VERSION) +} + +/// Converts `version` to a header value, logging and returning `None` when it +/// is not a valid header value. +#[must_use] +pub fn header_value_from(version: Option<&str>) -> Option { + todo!() +} + +/// Sets `x-ts-version` to the compiled-in git version, if one is known. +pub fn apply_git_version_header(response: &mut Response) { + apply_git_version_header_from(TS_GIT_VERSION, response); +} + +/// Sets `x-ts-version` to `version`, or leaves it unset when `version` is +/// unknown or invalid. +pub fn apply_git_version_header_from(version: Option<&str>, response: &mut Response) { + todo!() +} + +#[cfg(test)] +mod tests { + use super::*; + use edgezero_core::body::Body; + use edgezero_core::http::response_builder; + + fn empty_response() -> Response { + response_builder() + .body(Body::empty()) + .expect("should build empty test response") + } + + fn version_of(response: &Response) -> Option<&str> { + response + .headers() + .get(HEADER_X_TS_VERSION) + .and_then(|v| v.to_str().ok()) + } + + #[test] + fn header_value_from_valid_version() { + assert_eq!( + header_value_from(Some("v1.2.3")), + Some(HeaderValue::from_static("v1.2.3")), + "should convert a valid version" + ); + } + + #[test] + fn header_value_from_unknown_or_invalid_version() { + assert_eq!(header_value_from(None), None, "should be None when unknown"); + assert_eq!( + header_value_from(Some("bad\nvalue")), + None, + "should be None for a non-header-safe version" + ); + } + + #[test] + fn git_version_header_value_matches_compiled_in_version() { + assert_eq!( + git_version_header_value() + .as_ref() + .and_then(|v| v.to_str().ok()), + TS_GIT_VERSION, + "should convert the compiled-in TS_GIT_VERSION" + ); + } + + #[test] + fn sets_header_from_version() { + let mut response = empty_response(); + apply_git_version_header_from(Some("v1.2.3"), &mut response); + assert_eq!(version_of(&response), Some("v1.2.3"), "should set x-ts-version"); + } + + #[test] + fn omits_header_when_version_unknown() { + let mut response = empty_response(); + apply_git_version_header_from(None, &mut response); + assert_eq!(version_of(&response), None, "should omit x-ts-version when unknown"); + } + + #[test] + fn skips_invalid_header_value() { + let mut response = empty_response(); + apply_git_version_header_from(Some("bad\nvalue"), &mut response); + assert_eq!(version_of(&response), None, "should skip a non-header-safe version"); + } + + #[test] + fn default_uses_compiled_in_version() { + let mut response = empty_response(); + apply_git_version_header(&mut response); + assert_eq!( + version_of(&response), + TS_GIT_VERSION, + "should report the compiled-in TS_GIT_VERSION" + ); + } +} +``` + +Add `pub mod version_header;` to `lib.rs` after `pub mod tsjs;`. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cargo test -p trusted-server-core --lib --target wasm32-wasip1 version_header` +Expected: the 7 tests FAIL, panicking with `not yet implemented`. + +- [ ] **Step 3: Implement** + +Replace the two `todo!()` bodies: + +```rust +pub fn header_value_from(version: Option<&str>) -> Option { + let version = version?; + match HeaderValue::from_str(version) { + Ok(value) => Some(value), + Err(_) => { + log::warn!("Skipping invalid TS_GIT_VERSION response header value"); + None + } + } +} +``` + +```rust +pub fn apply_git_version_header_from(version: Option<&str>, response: &mut Response) { + if let Some(value) = header_value_from(version) { + response.headers_mut().insert(HEADER_X_TS_VERSION, value); + } +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cargo test -p trusted-server-core --lib --target wasm32-wasip1 version_header` +Expected: 7 passed. + +- [ ] **Step 5: Clippy and commit** + +Run: `cargo clippy-fastly` +Expected: no warnings. + +```bash +git add crates/trusted-server-core/src/constants.rs crates/trusted-server-core/src/version_header.rs crates/trusted-server-core/src/lib.rs +git commit --signoff -S -m "Add x-ts-fastly-version constant and git version header helpers" +``` + +--- + +### Task 3: Fastly adapter: split the headers and cover `/health` + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/middleware.rs` (imports, the two header-order doc comments, the `FASTLY_SERVICE_VERSION` block in `apply_finalize_headers`, tests) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (`health_response`, tests) + +**Interfaces:** + +- Consumes: `HEADER_X_TS_FASTLY_VERSION`, `HEADER_X_TS_VERSION`, `TS_GIT_VERSION`, + `version_header::{apply_git_version_header, git_version_header_value}` (Task 2). + +- [ ] **Step 1: Write the failing tests** + +Add to `mod tests` in `middleware.rs`: + +```rust + #[test] + fn version_headers_split_git_and_fastly_versions() { + let settings = settings_with_response_headers(vec![]); + let mut response = empty_response(); + + apply_finalize_headers(&settings, None, &mut response); + + let header = |name: &str| response.headers().get(name).and_then(|v| v.to_str().ok()); + assert_eq!( + header("x-ts-version"), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + assert_eq!( + header("x-ts-fastly-version"), + std::env::var(ENV_FASTLY_SERVICE_VERSION).ok().as_deref(), + "should report FASTLY_SERVICE_VERSION as x-ts-fastly-version" + ); + } +``` + +Add to `mod tests` in `main.rs`: + +```rust + #[test] + fn health_response_reports_git_version_only() { + let req = FastlyRequest::get("https://example.com/health"); + + let response = health_response(&req).expect("should build health response"); + + assert_eq!( + response.get_header_str("x-ts-version"), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version on /health" + ); + assert!( + response.get_header("x-ts-fastly-version").is_none(), + "should keep x-ts-fastly-version off the /health fast path" + ); + } +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `cargo test-fastly version_headers_split_git_and_fastly_versions health_response_reports_git_version_only` +Expected: both FAIL on the `x-ts-version` assertion. Local builds have git, so +`TS_GIT_VERSION` is `Some(…)`, but neither path sets it yet. + +If `cargo test` rejects two filters in this toolchain, run each name separately. + +- [ ] **Step 3: Implement** + +`middleware.rs` imports: + +```rust +use trusted_server_core::constants::{ + ENV_FASTLY_IS_STAGING, ENV_FASTLY_SERVICE_VERSION, HEADER_X_GEO_INFO_AVAILABLE, + HEADER_X_TS_ENV, HEADER_X_TS_FASTLY_VERSION, +}; +use trusted_server_core::version_header::apply_git_version_header; +``` + +Replace the `FASTLY_SERVICE_VERSION` block in `apply_finalize_headers` with: + +```rust + apply_git_version_header(response); + + if let Ok(v) = std::env::var(ENV_FASTLY_SERVICE_VERSION) { + if let Ok(value) = HeaderValue::from_str(&v) { + response + .headers_mut() + .insert(HEADER_X_TS_FASTLY_VERSION, value); + } else { + log::warn!("Skipping invalid FASTLY_SERVICE_VERSION response header value"); + } + } +``` + +In **both** header-order doc comments (on `FinalizeResponseMiddleware` and on +`apply_finalize_headers`), replace +``2. `X-TS-Version` from `FASTLY_SERVICE_VERSION` env var`` with: + +```rust +/// 2. `X-TS-Version` from the compiled-in git version (`TS_GIT_VERSION`), and +/// `X-TS-Fastly-Version` from the `FASTLY_SERVICE_VERSION` env var +``` + +`main.rs` — add imports: + +```rust +use trusted_server_core::constants::HEADER_X_TS_VERSION; +use trusted_server_core::version_header::git_version_header_value; +``` + +and replace `health_response` with: + +```rust +fn health_response(req: &FastlyRequest) -> Option { + if req.get_method() == FastlyMethod::GET && req.get_path() == "/health" { + let mut response = FastlyResponse::from_status(200).with_body_text_plain("ok"); + // Compiled-in constant: keeps the probe free of settings and app construction. + if let Some(version) = git_version_header_value() { + response.set_header(HEADER_X_TS_VERSION, version); + } + return Some(response); + } + + None +} +``` + +- [ ] **Step 4: Run the adapter tests and clippy** + +Run: `cargo test-fastly && cargo clippy-fastly` +Expected: all tests pass. Clippy reports no warnings. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/middleware.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit --signoff -S -m "Send git version as x-ts-version and Fastly version as x-ts-fastly-version on Fastly" +``` + +--- + +### Task 4: Axum, Cloudflare, Spin: emit `x-ts-version` + +**Files:** + +- Modify: `crates/trusted-server-adapter-axum/src/middleware.rs` (the `FinalizeResponseMiddleware` and `apply_finalize_headers` doc comments, `apply_finalize_headers`, tests) +- Modify: `crates/trusted-server-adapter-axum/tests/routes.rs` (new `/health` test) +- Modify: `crates/trusted-server-adapter-cloudflare/src/middleware.rs` (`apply_finalize_headers`, tests) +- Modify: `crates/trusted-server-adapter-spin/src/middleware.rs` (`apply_finalize_headers`, tests) +- Modify: `crates/trusted-server-adapter-spin/src/app.rs` (`startup_error_router_answers_health_with_200`) + +**Interfaces:** + +- Consumes: `version_header::apply_git_version_header`, `constants::TS_GIT_VERSION` (Task 2). + +- [ ] **Step 1: Write the failing tests** + +Axum `middleware.rs` `mod tests` (signature `apply_finalize_headers(&Settings, &mut Response)`): + +```rust + #[test] + fn emits_git_version_header() { + let mut response = empty_response(); + apply_finalize_headers(&settings_with_response_headers(vec![]), &mut response); + assert_eq!( + response.headers().get("x-ts-version").and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + } +``` + +Cloudflare and Spin `middleware.rs` `mod tests` (signature +`apply_finalize_headers(&Settings, bool, &mut Response)`), in each: + +```rust + #[test] + fn emits_git_version_header() { + let mut response = empty_response(); + apply_finalize_headers(&settings_with_response_headers(vec![]), false, &mut response); + assert_eq!( + response.headers().get("x-ts-version").and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + } +``` + +Axum `tests/routes.rs`: + +```rust +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn health_reports_git_version() { + let mut service = make_service(); + let request = Request::builder() + .method("GET") + .uri("/health") + .body(AxumBody::empty()) + .expect("should build health request"); + + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should serve health"); + + assert_eq!(response.status().as_u16(), 200, "should return 200 on /health"); + assert_eq!( + response + .headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version on /health" + ); +} +``` + +Spin `app.rs`, at the end of `startup_error_router_answers_health_with_200`, +capture the header before `resp.into_body()` consumes the response. Insert +directly after the status assertion: + +```rust + assert_eq!( + resp.headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "startup-fallback /health should report the compiled-in git version" + ); +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `cargo test-axum emits_git_version_header; cargo test-axum --test routes health_reports_git_version; cargo test-cloudflare emits_git_version_header; cargo test-spin emits_git_version_header; cargo test-spin startup_error_router_answers_health_with_200` +Expected: each FAILs, with left `None` and right `Some(…)`. + +- [ ] **Step 3: Implement** + +In each of the three `apply_finalize_headers` functions, directly after the +`HEADER_X_GEO_INFO_AVAILABLE` insert and before +`apply_response_headers_with_cache_privacy`, add: + +```rust + trusted_server_core::version_header::apply_git_version_header(response); +``` + +In the Axum `FinalizeResponseMiddleware` doc comment, replace +``is always emitted. Fastly-specific headers (`X-TS-Version`, `X-TS-ENV`) are`` +and the following line with: + +```rust +/// is always emitted. `X-TS-Version` carries the compiled-in git version. The +/// Fastly-specific headers (`X-TS-Fastly-Version`, `X-TS-ENV`) are skipped +/// because the corresponding env vars are not set in a local dev context. +``` + +In the Axum `apply_finalize_headers` doc comment, replace +`is unconditionally emitted. Fastly-specific headers are omitted.` with: + +```rust +/// is unconditionally emitted, followed by the compiled-in `X-TS-Version`. +/// Fastly-specific headers are omitted. +``` + +- [ ] **Step 4: Run the tests and clippy** + +Run: `cargo test-axum && cargo test-cloudflare && cargo test-spin && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasm` +Expected: all pass, with no warnings. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-axum crates/trusted-server-adapter-cloudflare/src/middleware.rs crates/trusted-server-adapter-spin/src +git commit --signoff -S -m "Emit x-ts-version git version on Axum, Cloudflare, and Spin" +``` + +--- + +### Task 5: Docs and changelog + +**Files:** + +- Modify: `docs/guide/first-party-proxy.md` (`[response_headers]` example, around line 719) +- Modify: `CHANGELOG.md` (`## [Unreleased]` → `### Changed`) + +- [ ] **Step 1: Stop the docs example from overriding the managed header** + +In `docs/guide/first-party-proxy.md`, replace `X-TS-Version = "1.0"` with: + +```toml +X-Debug-Build = "canary" +``` + +- [ ] **Step 2: Changelog entry** + +Add as the first bullet under `## [Unreleased]` → `### Changed`: + +```markdown +- **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER_GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. +``` + +- [ ] **Step 3: Format checks** + +Run: `cd docs && npm run format` then `git diff --stat` to confirm only intended files changed. +Expected: no unrelated reformatting. + +- [ ] **Step 4: Commit** + +```bash +git add docs/guide/first-party-proxy.md CHANGELOG.md +git commit --signoff -S -m "Document the x-ts-version and x-ts-fastly-version split" +``` + +--- + +### Task 6: Offline end-to-end verification under Viceroy + +No repository changes. Record every command and its observed output for the +PR test plan. Work in the scratchpad directory (`$SCRATCH` below). + +- [ ] **Step 1: Override build, served under Viceroy** + +```bash +TRUSTED_SERVER_GIT_VERSION=v9.9.9-test cargo build --release -p trusted-server-adapter-fastly --target wasm32-wasip1 +``` + +Serve it with a pushed local config, reusing the setup in +`scripts/smoke-fastly.sh` (`smoke-common.sh`: stub origin, +`smoke_initialize_config`, `ts config push --adapter fastly --local`, the three +`ts_secrets` entries, then `fastly compute serve --dir --file `). +Then: + +```bash +curl -s -o /dev/null -D - http://127.0.0.1:$PORT/health | grep -iE '^HTTP|^x-ts-' +curl -s -o /dev/null -D - http://127.0.0.1:$PORT/ | grep -iE '^HTTP|^x-ts-|^x-geo-info' +``` + +Expected: both `200`. `/health` shows only `x-ts-version: v9.9.9-test`. `/` +shows `x-ts-version: v9.9.9-test`, `x-ts-fastly-version: `, and +`x-geo-info-available` (proof it went through `apply_finalize_headers`). Record +Viceroy's `FASTLY_SERVICE_VERSION`. No `x-ts-version` is numeric. + +- [ ] **Step 2: Warm-cache rebuild with a changed value** + +```bash +TRUSTED_SERVER_GIT_VERSION=v9.9.9-test2 cargo build --release -p trusted-server-adapter-fastly --target wasm32-wasip1 -vv 2>&1 | grep -F 'TS_GIT_VERSION=' +``` + +Expected: `cargo:rustc-env=TS_GIT_VERSION=v9.9.9-test2`. Re-serve and curl +`/health`: `x-ts-version: v9.9.9-test2`. + +- [ ] **Step 3: No-env fallback** + +Rebuild with `TRUSTED_SERVER_GIT_VERSION` unset on branch +`feat/git-version-header`: `/health` shows `x-ts-version: feat/git-version-header`. +Then `git switch --detach` and rebuild: `x-ts-version` is the first 6 chars of +`git rev-parse HEAD`. Switch back to the branch afterwards. + +- [ ] **Step 4: Non-git build** + +```bash +git archive --format=tar HEAD | (mkdir -p "$SCRATCH/nogit" && tar -x -C "$SCRATCH/nogit") +cd "$SCRATCH/nogit" && env -u TRUSTED_SERVER_GIT_VERSION cargo build --release -p trusted-server-adapter-fastly --target wasm32-wasip1 +``` + +Confirm `$SCRATCH` is not inside a git repository first +(`git -C "$SCRATCH/nogit" rev-parse` fails). Expected: the build succeeds; +serving it, `/health` has no `x-ts-version`. + +--- + +### Task 7: Full CI gate + +- [ ] **Step 1: Run every gate from `CLAUDE.md`** + +```bash +cargo fmt --all -- --check +cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasm && cargo clippy-cli && cargo clippy-codegen +cargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spin +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +(cd crates/trusted-server-js/lib && npx vitest run && npm run format) +(cd docs && npm run format) +``` + +Expected: all green, and `git status` clean afterwards (format scripts may +rewrite files; commit any intended changes). diff --git a/docs/superpowers/specs/2026-09-25-git-version-header-design.md b/docs/superpowers/specs/2026-09-25-git-version-header-design.md new file mode 100644 index 000000000..67678e4d1 --- /dev/null +++ b/docs/superpowers/specs/2026-09-25-git-version-header-design.md @@ -0,0 +1,385 @@ +# Deployed git version in `x-ts-version` + +**Issues:** not filed yet + +**Date:** 2026-09-25 + +**Plan:** [../plans/2026-09-25-git-version-header.md](../plans/2026-09-25-git-version-header.md) + +**Status:** Pending maintainer review + +## Problem + +Nothing in a Trusted Server response says which Trusted Server code is +running. The Fastly adapter does send `x-ts-version`, but it fills it from +`FASTLY_SERVICE_VERSION`. That is the Fastly service version number (for +example `42`). It changes on every activation, config-only or not, and it +cannot be mapped back to a commit without the Fastly console. The name suggests +a Trusted Server version, but the value is a Fastly one. + +This matters for deploy pipelines that restrict production to published +releases. Operators and +monitors need to see from a response that production runs a release, and which +one. Staging keeps deploying branches and commits, so it needs to be +identifiable too. + +## Goals + +- Report the deployed git version in `x-ts-version`: the tag when the deployed + ref is a tag, else the branch name, else the first 6 characters of the commit + hash. +- Keep the Fastly service version available under a correct name, + `x-ts-fastly-version`. +- Make the value follow the Wasm binary. A Fastly rollback to an earlier + version must report that version's git version, with no extra step. +- Accept the value from the deploy pipeline, and fall back to local git so + local and ad-hoc builds still report something useful. +- Send `x-ts-version` from every adapter, since it does not depend on the + platform, including on the Fastly `GET /health` fast path that deploy health + checks probe right after a deploy. + +## Non-goals + +- Do not change `x-ts-env` or how staging is detected. +- Do not mark dirty working trees. The local fallback reports the tag or branch + even when the tree has uncommitted changes, so a local build of a modified + `v1.3.0` checkout reports `v1.3.0`. CI checkouts are always clean; a local + header is not proof of a clean release build. +- Do not normalize tag names (for example to semver). Report the ref as + deployed, subject only to the header-safety rule in §2. +- Do not add a runtime or versionless source for the version, such as the + `edgezero_runtime_env` Config Store or a config-blob field. +- Do not change the precedence of operator `settings.response_headers`. Whether + managed `x-ts-*` headers should be protected from operator overrides is a + separate decision (see Open Questions). +- Do not add `x-ts-fastly-version` to the Fastly `/health` fast path. It stays + minimal and only gains the compiled-in `x-ts-version`. +- Do not implement a production release gate or the pipeline-side version + resolution here. Those belong to the deploy pipeline; this repository only + defines the `TRUSTED_SERVER_GIT_VERSION` build input. + +## Current behavior + +`crates/trusted-server-core/src/constants.rs` defines `HEADER_X_TS_VERSION` +(`x-ts-version`), `HEADER_X_TS_ENV`, and the env names +`ENV_FASTLY_SERVICE_VERSION` and `ENV_FASTLY_IS_STAGING`. + +`apply_finalize_headers` in +`crates/trusted-server-adapter-fastly/src/middleware.rs` reads +`FASTLY_SERVICE_VERSION` from the Compute runtime env and inserts it as +`x-ts-version`. It skips an invalid value with a warning. Two doc comments on the +header write order describe this (on `FinalizeResponseMiddleware` and on +`apply_finalize_headers`). + +`health_response` in `crates/trusted-server-adapter-fastly/src/main.rs` answers +`GET /health` before logging, settings, app construction, and middleware, so it +carries no `x-ts-*` header at all. + +The Axum, Cloudflare, and Spin `apply_finalize_headers` functions do not send +any version header. The Axum middleware doc comment calls `X-TS-Version` +Fastly-specific. Axum and Spin register `/health` as router routes, so their +`FinalizeResponseMiddleware` already runs on it. Cloudflare has no `/health` +route; the path falls through to the publisher fallback, which also runs +through its middleware. + +`crates/trusted-server-core/build.rs` only prints `rerun-if-changed=build.rs`. +The workspace compiles in no git metadata. + +`docs/guide/first-party-proxy.md` shows `X-TS-Version = "1.0"` as an example +`[response_headers]` entry. Operator headers apply last, so copying that example +overwrites the managed value. + +## Design + +### 1. The value is compiled in + +Fastly Compute only exposes platform variables (`FASTLY_SERVICE_VERSION`, +`FASTLY_IS_STAGING`, and so on) through the process env at runtime. There is no +way to add a deploy-time variable. The only EdgeZero channel for runtime values +is the `edgezero_runtime_env` Config Store. It is versionless and holds only +`EDGEZERO__*` store mappings, so a Fastly rollback would leave the newer +deploy's version in it. + +So the git version is compiled into the binary. A compiled-in value is +immutable per Fastly version, so it always matches the code being served, +including after a rollback. + +### 2. Resolution order + +`crates/trusted-server-core/build.rs` resolves the version once per build and +emits it as the rustc env `TS_GIT_VERSION`: + +1. `TRUSTED_SERVER_GIT_VERSION` from the build environment, if it is usable + (see below). The deploy pipeline sets this. +2. Otherwise the exact tag at `HEAD`, from `git describe --tags --exact-match`. +3. Otherwise the current branch, from `git symbolic-ref --short -q HEAD`. +4. Otherwise the first 6 characters of `git rev-parse HEAD`. Take exactly 6 + characters. Do not use `--short`, which may lengthen the hash to keep it + unique. +5. Otherwise leave it unset. + +A candidate is **usable** when, after trimming surrounding whitespace, it is +non-empty and consists only of visible ASCII (bytes `0x21`–`0x7E`). An +unusable candidate is skipped and the next step is tried. + +This is stricter than git. `git check-ref-format` forbids control characters, +space, and `~^:?*[\`, but it allows non-ASCII UTF-8, so a tag such as `v1-été` +is a valid ref, and a pipeline may pass such a ref through. Under this rule it +is skipped: `build.rs` prints a `cargo:warning` naming +`TRUSTED_SERVER_GIT_VERSION`, and resolution falls back to local git. In a CI +checkout that yields the 6-char hash, which still identifies the code. Falling +back was preferred over omitting the header, because a missing header is harder +to diagnose. The visible-ASCII rule keeps every compiled-in value a valid +`HeaderValue` whose `to_str()` succeeds, so consumers never see opaque bytes. + +The deploy pipeline must supply the value because CI checkouts are shallow and +detached (`actions/checkout`, `fetch-depth: 1`, no tags). In such a checkout +steps 2 and 3 always fail, and only the hash is available. The pipeline knows +the ref the operator asked for and can resolve the tag or branch against +`origin` before the build. + +The choice between these candidates is a pure function in +`crates/trusted-server-core/build_support/git_version.rs`. `build.rs` and an +integration test both include it through `#[path]`, so the rule is unit-tested +without running a build script. + +### 3. Build caching + +`build.rs` prints `cargo:rerun-if-env-changed=TRUSTED_SERVER_GIT_VERSION`. +CI deploys commonly restore a cached `target/` (EdgeZero's `deploy-fastly` +action does). Without this line a warm cache could keep the previous deploy's +version. + +It also prints `cargo:rerun-if-changed` for `build.rs` and +`build_support/git_version.rs`, so editing the resolver re-runs the script. + +When it uses the local git fallback, `build.rs` also prints `rerun-if-changed` +for the paths that move when the checkout does, so local branch switches and +new tags show up without `cargo clean`: + +- `/HEAD`, from `git rev-parse --absolute-git-dir`; +- the current branch's ref file, `/` (for + example `refs/heads/feature/x`), only when `HEAD` is on a branch; +- `/refs/tags` and `/packed-refs`, from + `git rev-parse --git-common-dir`. + +The current branch's ref matters because a new commit on a tagged branch must +switch the result from the tag to the branch name, so a commit on the current +branch re-runs the script and recompiles core in local builds. That is +inherent to exact-tag detection. Commits on other branches and `refs/remotes` +are deliberately not watched: they never change the result, and every +`git fetch` rewrites `refs/remotes`. + +The two directories differ in a linked worktree: `HEAD` lives in +`.git/worktrees/`, while refs live in the main `.git`. In a plain clone +they are the same directory. + +Only paths that exist are printed. Cargo treats a missing `rerun-if-changed` +path as changed, so printing an absent `packed-refs` would re-run the script +and recompile core on every build. The accepted cost: a `packed-refs` file +created after the last run is not noticed until something else triggers a +re-run. Watching the branch ref and `refs/tags` still catches new loose refs. Likewise, if the current branch's ref exists only in `packed-refs` (after `git pack-refs`), the loose ref written by the next commit is not watched. The tag-to-branch switch after committing on a tagged commit then waits for the next re-run. + +### 4. Headers + +Constants in `trusted-server-core::constants`: + +- `HEADER_X_TS_VERSION` keeps the name `x-ts-version`, which now means the git + version. +- Add `HEADER_X_TS_FASTLY_VERSION`, named `x-ts-fastly-version`. +- Add `TS_GIT_VERSION: Option<&str> = option_env!("TS_GIT_VERSION")`. + +A new module, `trusted-server-core::version_header`, owns the value and the +write: + +- `git_version_header_value() -> Option` converts + `TS_GIT_VERSION`. It returns `None` when no version is known, or when the + value is not a valid header value (logged at `warn`). §2 makes the latter + unreachable for values from `build.rs`, but the check stays as a backstop. +- `apply_git_version_header(response)` inserts `x-ts-version` from + `git_version_header_value()`. +- Testable variants take the version as an argument. + +Keeping this in core means all four adapters, and the Fastly health fast path, +use the same rule. + +Fastly `apply_finalize_headers` calls `apply_git_version_header` and writes +`FASTLY_SERVICE_VERSION` to `x-ts-fastly-version` instead of `x-ts-version`. +Axum, Cloudflare, and Spin `apply_finalize_headers` call +`apply_git_version_header` right after the geo-availability header. In every +adapter the call runs before `apply_response_headers_with_cache_privacy`, so the +existing order is kept: operator headers still apply last. + +Fastly `health_response` sets `x-ts-version` from `git_version_header_value()`. +The `fastly` crate's `set_header` accepts the `http` 1.x `HeaderValue` that +`edgezero_core::http` re-exports, so no conversion is needed and nothing can +panic. The fast path stays free of settings, logging, and app construction. + +A Fastly production response from release `v1.3.0`, served as Fastly version 42: + +```text +x-ts-version: v1.3.0 +x-ts-fastly-version: 42 +``` + +A staged deploy of branch `feature/x`: + +```text +x-ts-version: feature/x +x-ts-fastly-version: 43 +x-ts-env: staging +``` + +`GET /health` on either: + +```text +x-ts-version: v1.3.0 +``` + +## Error Handling + +`build.rs` never fails the build over version metadata. If git is missing, the +directory is not a repository (for example an exported source tree), or git +fails, the build leaves `TS_GIT_VERSION` unset and `x-ts-version` is omitted. +Sending a placeholder such as `unknown` was rejected: a missing header is easier +to tell apart from a real ref. + +An unusable `TRUSTED_SERVER_GIT_VERSION` (§2) produces a `cargo:warning` and +falls back to local git. It does not fail the build. + +No new public error type is introduced. + +## Compatibility and Rollout + +This is a **breaking change** for anything that reads `x-ts-version` as the +Fastly version number. Dashboards, monitors, and scripts must switch to +`x-ts-fastly-version`. The CHANGELOG entry calls this out. + +The rollout does not depend on the deploy pipeline: + +- A pipeline that does not set `TRUSTED_SERVER_GIT_VERSION`: CI deploys report + the 6-char hash from the local fallback. This is still correct, just less + readable. +- A pipeline that sets it before this change ships: the variable is in the + build env but not read, so behavior is unchanged. + +Rollback needs nothing extra. Reactivating an earlier Fastly version serves that +version's compiled-in value. + +## Alternatives and Decision + +- **`edgezero_runtime_env` Config Store.** Rejected. It is versionless, so it is + wrong after a rollback. It holds only `EDGEZERO__*` mappings, and writing it + per deploy would need EdgeZero changes. +- **Operator `[response_headers]` in the pushed config blob.** Rejected. Config + push is versionless and not reverted by rollback, and it depends on every + operator keeping it in sync. +- **Local git only, in `build.rs`.** Rejected as the only source. CI checkouts + are shallow and detached, so it would always produce the hash and never the + tag or branch. It stays as the fallback. +- **A new header, keeping `x-ts-version` as the Fastly version.** Rejected. The + existing name is the one people expect to carry the Trusted Server version, + and keeping it wrong would keep misleading them. +- **Omitting the header on an unusable override.** Rejected in favor of falling + back to local git (§2). + +Decision: compile in the pipeline-supplied value, with a local git fallback. + +## Testing + +Unit and integration tests follow red-green-refactor and cover: + +- resolution: override wins; tag beats branch; branch when untagged; exactly 6 + characters of the commit; blank candidates skipped; a non-ASCII override + (`v1-été`) and one with an inner space are skipped in favor of local git; + `None` when nothing is known; +- `version_header`: value from a version; `None` when unknown; `None` for an + invalid value; the header is set, omitted, or skipped accordingly; the + default path reports the compiled-in `TS_GIT_VERSION`; +- Fastly `apply_finalize_headers`: `x-ts-version` equals `TS_GIT_VERSION`, and + `x-ts-fastly-version` equals `FASTLY_SERVICE_VERSION` when set; +- Fastly `health_response`: `x-ts-version` equals `TS_GIT_VERSION`, and + `x-ts-fastly-version` is absent; +- Axum, Cloudflare, and Spin `apply_finalize_headers`: `x-ts-version` equals + `TS_GIT_VERSION`; +- Axum and Spin `GET /health` through the router: `x-ts-version` equals + `TS_GIT_VERSION`. + +End-to-end, offline, with Viceroy and no live Fastly service: + +1. Build the release Wasm with `TRUSTED_SERVER_GIT_VERSION=v9.9.9-test`, run it + under `fastly compute serve --file `, and `curl -sI` both `GET /health` + and a route that passes through `FinalizeResponseMiddleware`, with a local + `trusted_server_config` pushed as `scripts/smoke-fastly.sh` does so the route + loads settings. Assert the status codes (`200` for both) so the check cannot + pass on an error page. Both carry + `x-ts-version: v9.9.9-test`. The finalized route carries + `x-ts-fastly-version` with whatever Viceroy reports, and no response carries + a numeric `x-ts-version`. +2. Warm-cache rebuild, without cleaning, with `v9.9.9-test2`. The build script + re-runs (visible in `cargo build -vv`), and the header changes. +3. Rebuild with the variable unset, on a branch and then on a detached `HEAD`. + The header reports the branch, then the 6-char hash. +4. `git archive` the tree into a scratch directory outside any repository and + build there with the variable unset. The build succeeds and `x-ts-version` + is absent. + +Final verification runs every CI gate in `CLAUDE.md`: `cargo fmt --all -- +--check`; all eight clippy aliases; the Fastly, Axum, Cloudflare, and Spin test +aliases; the parity integration test; and the JS and docs format checks. The +handoff reports the commands run and their observed results. + +## Documentation + +- In `docs/guide/first-party-proxy.md`, replace the `X-TS-Version = "1.0"` + `[response_headers]` example with a header that is not managed + (`X-Debug-Build = "canary"`). +- Add a breaking-change entry to `CHANGELOG.md` under `## [Unreleased]` → + `### Changed`. +- Update the header write-order doc comments in the Fastly middleware, and the + Axum middleware comment that calls `X-TS-Version` Fastly-specific. + +## Acceptance Criteria + +- A Fastly deploy built with `TRUSTED_SERVER_GIT_VERSION=v1.3.0` returns + `x-ts-version: v1.3.0` and `x-ts-fastly-version: ` on + finalized responses, and `x-ts-version: v1.3.0` on `GET /health`. +- A staging deploy of branch `feature/x` returns `x-ts-version: feature/x`. A + deploy of a bare SHA returns the first 6 characters of that commit. +- A local build without the override reports the local tag, branch, or 6-char + hash, in that order of preference, including from a linked worktree. +- A build with neither the override nor git omits `x-ts-version` and still + succeeds. +- An override that is not visible ASCII produces a `cargo:warning` and the + local git fallback. +- Changing `TRUSTED_SERVER_GIT_VERSION` between builds changes the header even + with a warm `target/` cache. +- Axum, Cloudflare, and Spin responses carry `x-ts-version`. +- No response carries the Fastly service version as `x-ts-version`. + +## Open Questions + +- Should operator `settings.response_headers` be able to override + `x-ts-version` and `x-ts-fastly-version`? This spec keeps current precedence. + Protecting managed diagnostic headers, as cache-control is already protected + on uncacheable responses, can follow separately. + +## Expected Files + +- `crates/trusted-server-core/build.rs` +- `crates/trusted-server-core/build_support/git_version.rs` +- `crates/trusted-server-core/tests/git_version_resolve.rs` +- `crates/trusted-server-core/src/constants.rs` +- `crates/trusted-server-core/src/version_header.rs` +- `crates/trusted-server-core/src/lib.rs` +- `crates/trusted-server-adapter-fastly/src/main.rs` +- `crates/trusted-server-adapter-fastly/src/middleware.rs` +- `crates/trusted-server-adapter-axum/src/middleware.rs` +- `crates/trusted-server-adapter-axum/tests/routes.rs` +- `crates/trusted-server-adapter-cloudflare/src/middleware.rs` +- `crates/trusted-server-adapter-spin/src/middleware.rs` +- `crates/trusted-server-adapter-spin/src/app.rs` +- `docs/guide/first-party-proxy.md` +- `CHANGELOG.md` +- `docs/superpowers/specs/2026-09-25-git-version-header-design.md` +- `docs/superpowers/plans/2026-09-25-git-version-header.md` From e326f5acfbf11008fe938613dedd1964849f7831 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:19 +0530 Subject: [PATCH 099/230] Compile deployed git version into trusted-server-core as TS_GIT_VERSION Signed-off-by: dhruv8sh --- crates/trusted-server-core/build.rs | 89 +++++++++++ .../build_support/git_version.rs | 42 ++++++ .../tests/git_version_resolve.rs | 141 ++++++++++++++++++ 3 files changed, 272 insertions(+) create mode 100644 crates/trusted-server-core/build_support/git_version.rs create mode 100644 crates/trusted-server-core/tests/git_version_resolve.rs diff --git a/crates/trusted-server-core/build.rs b/crates/trusted-server-core/build.rs index c2bce4fe2..c02a88d1e 100644 --- a/crates/trusted-server-core/build.rs +++ b/crates/trusted-server-core/build.rs @@ -1,3 +1,92 @@ +//! Compiles the deployed git version into `trusted-server-core` as `TS_GIT_VERSION`. + +#[path = "build_support/git_version.rs"] +mod git_version; + +use std::path::Path; +use std::process::Command; + +use git_version::{Candidates, is_usable, resolve_git_version}; + +/// Set by the deploy pipeline. Its CI checkout is shallow and detached, so local git +/// cannot see the tag or branch there. +const OVERRIDE_ENV: &str = "TRUSTED_SERVER_GIT_VERSION"; + +/// Runs `git` in the crate directory; `None` if git is missing or fails. +fn git(args: &[&str]) -> Option { + let output = Command::new("git").args(args).output().ok()?; + if !output.status.success() { + return None; + } + String::from_utf8(output.stdout) + .ok() + .map(|s| s.trim().to_owned()) +} + +/// Re-runs this script when `path` changes. Skips missing paths: Cargo treats +/// them as always changed, which would rebuild core on every build. +fn rerun_if_exists(path: &Path) { + if path.exists() { + println!("cargo:rerun-if-changed={}", path.display()); + } +} + +/// Resolves the version from local git, watching the paths that move with it. +fn resolve_from_local_git() -> Option { + // HEAD is per-worktree; refs are shared in the common dir. They differ in a + // linked worktree and coincide in a plain clone. Only the current branch's + // ref (its commit decides the exact-tag match) and tags can change the + // result, so other branches and `refs/remotes` are not watched: a commit + // elsewhere or a fetch must not rebuild core. + if let Some(git_dir) = git(&["rev-parse", "--absolute-git-dir"]) { + rerun_if_exists(&Path::new(&git_dir).join("HEAD")); + } + if let Some(common_dir) = git(&["rev-parse", "--path-format=absolute", "--git-common-dir"]) { + let common_dir = Path::new(&common_dir); + if let Some(branch_ref) = git(&["symbolic-ref", "-q", "HEAD"]) { + rerun_if_exists(&common_dir.join(branch_ref)); + } + rerun_if_exists(&common_dir.join("refs/tags")); + rerun_if_exists(&common_dir.join("packed-refs")); + } + + let exact_tag = git(&["describe", "--tags", "--exact-match"]); + let branch = git(&["symbolic-ref", "--short", "-q", "HEAD"]); + let commit = git(&["rev-parse", "HEAD"]); + resolve_git_version(&Candidates { + override_value: None, + exact_tag: exact_tag.as_deref(), + branch: branch.as_deref(), + commit: commit.as_deref(), + }) +} + fn main() { println!("cargo:rerun-if-changed=build.rs"); + println!("cargo:rerun-if-changed=build_support/git_version.rs"); + println!("cargo:rerun-if-env-changed={OVERRIDE_ENV}"); + + let override_value = std::env::var(OVERRIDE_ENV).ok(); + let resolved = match override_value.as_deref() { + Some(value) if is_usable(value) => resolve_git_version(&Candidates { + override_value: Some(value), + exact_tag: None, + branch: None, + commit: None, + }), + Some(value) => { + if !value.trim().is_empty() { + println!( + "cargo:warning={OVERRIDE_ENV}={value:?} is not visible ASCII; \ + falling back to local git for x-ts-version" + ); + } + resolve_from_local_git() + } + None => resolve_from_local_git(), + }; + + if let Some(version) = resolved { + println!("cargo:rustc-env=TS_GIT_VERSION={version}"); + } } diff --git a/crates/trusted-server-core/build_support/git_version.rs b/crates/trusted-server-core/build_support/git_version.rs new file mode 100644 index 000000000..981434b1b --- /dev/null +++ b/crates/trusted-server-core/build_support/git_version.rs @@ -0,0 +1,42 @@ +//! Resolution rule for the deployed git version reported in `x-ts-version`. +//! +//! Shared by `build.rs` and `tests/git_version_resolve.rs` via `#[path]`, so it +//! must stay dependency-free. + +/// Raw candidates for the deployed git version, in priority order. +pub struct Candidates<'a> { + /// `TRUSTED_SERVER_GIT_VERSION`, supplied by the deploy pipeline. + pub override_value: Option<&'a str>, + /// `git describe --tags --exact-match`. + pub exact_tag: Option<&'a str>, + /// `git symbolic-ref --short -q HEAD`. + pub branch: Option<&'a str>, + /// `git rev-parse HEAD`. + pub commit: Option<&'a str>, +} + +/// Whether `value`, once trimmed, is non-empty visible ASCII (`0x21`–`0x7E`). +/// +/// Stricter than git, which also allows non-ASCII UTF-8 in ref names. Such a +/// ref is skipped so every compiled-in value is a `to_str()`-able header value. +#[must_use] +pub fn is_usable(value: &str) -> bool { + let trimmed = value.trim(); + !trimmed.is_empty() && trimmed.bytes().all(|b| (0x21..=0x7E).contains(&b)) +} + +/// Picks the first usable candidate: override, tag, branch, then the first 6 +/// characters of the commit. `None` when nothing usable is known. +#[must_use] +pub fn resolve_git_version(candidates: &Candidates<'_>) -> Option { + usable(candidates.override_value) + .or_else(|| usable(candidates.exact_tag)) + .or_else(|| usable(candidates.branch)) + .map(str::to_owned) + .or_else(|| usable(candidates.commit).map(|c| c.chars().take(6).collect())) +} + +/// The trimmed candidate, if it is usable. +fn usable(value: Option<&str>) -> Option<&str> { + value.filter(|v| is_usable(v)).map(str::trim) +} diff --git a/crates/trusted-server-core/tests/git_version_resolve.rs b/crates/trusted-server-core/tests/git_version_resolve.rs new file mode 100644 index 000000000..d65cd4d42 --- /dev/null +++ b/crates/trusted-server-core/tests/git_version_resolve.rs @@ -0,0 +1,141 @@ +//! The version-resolution rule `build.rs` uses for `TS_GIT_VERSION`. + +#[path = "../build_support/git_version.rs"] +mod git_version; + +use git_version::{Candidates, resolve_git_version}; + +const COMMIT: &str = "abcdef0123456789abcdef0123456789abcdef01"; + +fn local(exact_tag: Option<&'static str>, branch: Option<&'static str>) -> Candidates<'static> { + Candidates { + override_value: None, + exact_tag, + branch, + commit: Some(COMMIT), + } +} + +#[test] +fn override_wins_over_local_git() { + let candidates = Candidates { + override_value: Some("v1.2.3"), + ..local(Some("v9.9.9"), Some("main")) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("v1.2.3"), + "should prefer the pipeline-supplied TRUSTED_SERVER_GIT_VERSION" + ); +} + +#[test] +fn override_is_trimmed() { + let candidates = Candidates { + override_value: Some(" feature/x\n"), + ..local(None, None) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("feature/x"), + "should trim surrounding whitespace from the override" + ); +} + +#[test] +fn tag_wins_over_branch() { + assert_eq!( + resolve_git_version(&local(Some("v1.2.3"), Some("main"))).as_deref(), + Some("v1.2.3"), + "should prefer an exact tag over the branch" + ); +} + +#[test] +fn branch_when_no_tag() { + assert_eq!( + resolve_git_version(&local(None, Some("feature/x"))).as_deref(), + Some("feature/x"), + "should use the branch when HEAD is not tagged" + ); +} + +#[test] +fn six_char_hash_when_no_tag_or_branch() { + assert_eq!( + resolve_git_version(&local(None, None)).as_deref(), + Some("abcdef"), + "should use exactly the first 6 characters of the commit" + ); +} + +#[test] +fn blank_candidates_are_skipped() { + let candidates = Candidates { + override_value: Some(" "), + exact_tag: Some(""), + branch: Some(" "), + commit: Some(COMMIT), + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("abcdef"), + "should skip empty or whitespace-only candidates" + ); +} + +#[test] +fn non_ascii_override_falls_back_to_local_git() { + let candidates = Candidates { + override_value: Some("v1-été"), + ..local(None, None) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("abcdef"), + "should skip a non-ASCII override and fall back to local git" + ); +} + +#[test] +fn override_with_inner_space_falls_back_to_local_git() { + let candidates = Candidates { + override_value: Some("v1 2"), + ..local(None, Some("main")) + }; + assert_eq!( + resolve_git_version(&candidates).as_deref(), + Some("main"), + "should skip an override containing a space" + ); +} + +#[test] +fn none_when_nothing_is_known() { + let candidates = Candidates { + override_value: None, + exact_tag: None, + branch: None, + commit: None, + }; + assert_eq!( + resolve_git_version(&candidates), + None, + "should leave the version unset so the header is omitted" + ); +} + +#[test] +fn is_usable_accepts_only_visible_ascii() { + assert!(git_version::is_usable("v1.2.3"), "should accept a tag"); + assert!( + git_version::is_usable(" main "), + "should accept after trimming" + ); + assert!(!git_version::is_usable(""), "should reject empty"); + assert!( + !git_version::is_usable("a\tb"), + "should reject a control character" + ); + assert!(!git_version::is_usable("v1-été"), "should reject non-ASCII"); +} From 976cdea16c2785d8aaa8900f432d0d0be077d8e5 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:20 +0530 Subject: [PATCH 100/230] Add x-ts-fastly-version constant and git version header helpers Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/constants.rs | 7 + crates/trusted-server-core/src/lib.rs | 1 + .../trusted-server-core/src/version_header.rs | 134 ++++++++++++++++++ 3 files changed, 142 insertions(+) create mode 100644 crates/trusted-server-core/src/version_header.rs diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index d444238e8..633c0a57b 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -30,9 +30,16 @@ pub const HEADER_X_COMPRESS_HINT: HeaderName = HeaderName::from_static("x-compre pub const HEADER_X_DEBUG_FASTLY_POP: HeaderName = HeaderName::from_static("x-debug-fastly-pop"); // Staging / version identification headers +/// Deployed git version: tag, else branch, else 6-char commit (see [`TS_GIT_VERSION`]). pub const HEADER_X_TS_VERSION: HeaderName = HeaderName::from_static("x-ts-version"); +/// Fastly service version (`FASTLY_SERVICE_VERSION`), formerly sent as `x-ts-version`. +pub const HEADER_X_TS_FASTLY_VERSION: HeaderName = HeaderName::from_static("x-ts-fastly-version"); pub const HEADER_X_TS_ENV: HeaderName = HeaderName::from_static("x-ts-env"); +/// Deployed git version compiled in by `build.rs`, from the deploy pipeline's +/// `TRUSTED_SERVER_GIT_VERSION` or local git. `None` when unknown. +pub const TS_GIT_VERSION: Option<&str> = option_env!("TS_GIT_VERSION"); + // Fastly environment variables pub const ENV_FASTLY_SERVICE_VERSION: &str = "FASTLY_SERVICE_VERSION"; pub const ENV_FASTLY_IS_STAGING: &str = "FASTLY_IS_STAGING"; diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 76621baf7..704903848 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -73,6 +73,7 @@ pub mod streaming_replacer; pub mod test_support; pub mod tester_cookie; pub mod tsjs; +pub mod version_header; #[cfg(test)] mod migration_guards; diff --git a/crates/trusted-server-core/src/version_header.rs b/crates/trusted-server-core/src/version_header.rs new file mode 100644 index 000000000..f2587cbb8 --- /dev/null +++ b/crates/trusted-server-core/src/version_header.rs @@ -0,0 +1,134 @@ +//! `x-ts-version`: the deployed git version compiled in by `build.rs`. + +use edgezero_core::http::{HeaderValue, Response}; + +use crate::constants::{HEADER_X_TS_VERSION, TS_GIT_VERSION}; + +/// Returns the compiled-in git version as a header value. +/// +/// `None` when no version is known or the value is not a valid header value. +#[must_use] +pub fn git_version_header_value() -> Option { + header_value_from(TS_GIT_VERSION) +} + +/// Converts `version` to a header value, logging and returning `None` when it +/// is not a valid header value. +#[must_use] +pub fn header_value_from(version: Option<&str>) -> Option { + let version = version?; + match HeaderValue::from_str(version) { + Ok(value) => Some(value), + Err(_) => { + log::warn!("Skipping invalid TS_GIT_VERSION response header value"); + None + } + } +} + +/// Sets `x-ts-version` to the compiled-in git version, if one is known. +pub fn apply_git_version_header(response: &mut Response) { + apply_git_version_header_from(TS_GIT_VERSION, response); +} + +/// Sets `x-ts-version` to `version`, or leaves it unset when `version` is +/// unknown or invalid. +pub fn apply_git_version_header_from(version: Option<&str>, response: &mut Response) { + if let Some(value) = header_value_from(version) { + response.headers_mut().insert(HEADER_X_TS_VERSION, value); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use edgezero_core::body::Body; + use edgezero_core::http::response_builder; + + fn empty_response() -> Response { + response_builder() + .body(Body::empty()) + .expect("should build empty test response") + } + + fn version_of(response: &Response) -> Option<&str> { + response + .headers() + .get(HEADER_X_TS_VERSION) + .and_then(|v| v.to_str().ok()) + } + + #[test] + fn header_value_from_valid_version() { + assert_eq!( + header_value_from(Some("v1.2.3")), + Some(HeaderValue::from_static("v1.2.3")), + "should convert a valid version" + ); + } + + #[test] + fn header_value_from_unknown_or_invalid_version() { + assert_eq!(header_value_from(None), None, "should be None when unknown"); + assert_eq!( + header_value_from(Some("bad\nvalue")), + None, + "should be None for a non-header-safe version" + ); + } + + #[test] + fn git_version_header_value_matches_compiled_in_version() { + assert_eq!( + git_version_header_value() + .as_ref() + .and_then(|v| v.to_str().ok()), + TS_GIT_VERSION, + "should convert the compiled-in TS_GIT_VERSION" + ); + } + + #[test] + fn sets_header_from_version() { + let mut response = empty_response(); + apply_git_version_header_from(Some("v1.2.3"), &mut response); + assert_eq!( + version_of(&response), + Some("v1.2.3"), + "should set x-ts-version" + ); + } + + #[test] + fn omits_header_when_version_unknown() { + let mut response = empty_response(); + apply_git_version_header_from(None, &mut response); + assert_eq!( + version_of(&response), + None, + "should omit x-ts-version when unknown" + ); + } + + #[test] + fn skips_invalid_header_value() { + let mut response = empty_response(); + apply_git_version_header_from(Some("bad\nvalue"), &mut response); + assert_eq!( + version_of(&response), + None, + "should skip a non-header-safe version" + ); + } + + #[test] + fn default_uses_compiled_in_version() { + let mut response = empty_response(); + apply_git_version_header(&mut response); + assert_eq!( + version_of(&response), + TS_GIT_VERSION, + "should report the compiled-in TS_GIT_VERSION" + ); + } +} From a93ac53808e47828956d32b56cb67d0a4f3d348b Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:21 +0530 Subject: [PATCH 101/230] Send git version as x-ts-version and Fastly version as x-ts-fastly-version on Fastly Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/main.rs | 26 +++++++++++++- .../src/middleware.rs | 35 ++++++++++++++++--- 2 files changed, 56 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 367a3e33f..798db7925 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -14,6 +14,7 @@ use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::cache_policy::EdgeCacheHeader; +use trusted_server_core::constants::HEADER_X_TS_VERSION; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -28,6 +29,7 @@ use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; +use trusted_server_core::version_header::git_version_header_value; mod app; mod backend; @@ -64,7 +66,12 @@ fn open_trusted_server_config_store(store_name: &str) -> Result Option { if req.get_method() == FastlyMethod::GET && req.get_path() == "/health" { - return Some(FastlyResponse::from_status(200).with_body_text_plain("ok")); + let mut response = FastlyResponse::from_status(200).with_body_text_plain("ok"); + // Compiled-in constant: keeps the probe free of settings and app construction. + if let Some(version) = git_version_header_value() { + response.set_header(HEADER_X_TS_VERSION, version); + } + return Some(response); } None @@ -558,6 +565,23 @@ mod tests { ); } + #[test] + fn health_response_reports_git_version_only() { + let req = FastlyRequest::get("https://example.com/health"); + + let response = health_response(&req).expect("should build health response"); + + assert_eq!( + response.get_header_str("x-ts-version"), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version on /health" + ); + assert!( + response.get_header("x-ts-fastly-version").is_none(), + "should keep x-ts-fastly-version off the /health fast path" + ); + } + #[test] fn health_response_ignores_non_health_paths() { let req = FastlyRequest::get("https://example.com/auction"); diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 283f16255..e0b1791b4 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -22,11 +22,12 @@ use std::net::IpAddr; use trusted_server_core::auth::enforce_basic_auth; use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_SERVICE_VERSION, HEADER_X_GEO_INFO_AVAILABLE, - HEADER_X_TS_ENV, HEADER_X_TS_VERSION, + HEADER_X_TS_ENV, HEADER_X_TS_FASTLY_VERSION, }; use trusted_server_core::geo::GeoInfo; use trusted_server_core::platform::{ClientInfo, PlatformGeo}; use trusted_server_core::settings::Settings; +use trusted_server_core::version_header::apply_git_version_header; pub(crate) const HEADER_X_TS_FINALIZED: &str = "x-ts-finalized"; @@ -49,7 +50,8 @@ pub(crate) const HEADER_X_TS_FINALIZED: &str = "x-ts-finalized"; /// /// Headers are written in this order (last write wins): /// 1. Geo headers (or `X-Geo-Info-Available: false` when geo is unavailable) -/// 2. `X-TS-Version` from `FASTLY_SERVICE_VERSION` env var +/// 2. `X-TS-Version` from the compiled-in git version (`TS_GIT_VERSION`), and +/// `X-TS-Fastly-Version` from the `FASTLY_SERVICE_VERSION` env var /// 3. `X-TS-ENV: staging` when `FASTLY_IS_STAGING == "1"` /// 4. Operator-configured `settings.response_headers` (can override any managed header) pub struct FinalizeResponseMiddleware { @@ -187,7 +189,8 @@ where /// /// Header write order (last write wins): /// 1. Geo headers (`x-geo-*`) — or `X-Geo-Info-Available: false` when absent -/// 2. `X-TS-Version` from `FASTLY_SERVICE_VERSION` env var +/// 2. `X-TS-Version` from the compiled-in git version (`TS_GIT_VERSION`), and +/// `X-TS-Fastly-Version` from the `FASTLY_SERVICE_VERSION` env var /// 3. `X-TS-ENV: staging` when `FASTLY_IS_STAGING == "1"` /// 4. Set-Cookie cache privacy — strip surrogate cache headers and downgrade /// `Cache-Control` to `private, max-age=0` on cookie-bearing responses @@ -208,9 +211,13 @@ pub(crate) fn apply_finalize_headers( ); } + apply_git_version_header(response); + if let Ok(v) = std::env::var(ENV_FASTLY_SERVICE_VERSION) { if let Ok(value) = HeaderValue::from_str(&v) { - response.headers_mut().insert(HEADER_X_TS_VERSION, value); + response + .headers_mut() + .insert(HEADER_X_TS_FASTLY_VERSION, value); } else { log::warn!("Skipping invalid FASTLY_SERVICE_VERSION response header value"); } @@ -813,4 +820,24 @@ mod tests { "should reach the handler when auth is not required" ); } + + #[test] + fn version_headers_split_git_and_fastly_versions() { + let settings = settings_with_response_headers(vec![]); + let mut response = empty_response(); + + apply_finalize_headers(&settings, None, &mut response); + + let header = |name: &str| response.headers().get(name).and_then(|v| v.to_str().ok()); + assert_eq!( + header("x-ts-version"), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + assert_eq!( + header("x-ts-fastly-version"), + std::env::var(ENV_FASTLY_SERVICE_VERSION).ok().as_deref(), + "should report FASTLY_SERVICE_VERSION as x-ts-fastly-version" + ); + } } From 0090f728ada73c9dda22aad254fca791e9a27a08 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:22 +0530 Subject: [PATCH 102/230] Emit x-ts-version git version on Axum, Cloudflare, and Spin Signed-off-by: dhruv8sh --- .../src/middleware.rs | 24 ++++++++++++-- .../tests/routes.rs | 32 +++++++++++++++++++ .../src/middleware.rs | 20 ++++++++++++ crates/trusted-server-adapter-spin/src/app.rs | 7 ++++ .../src/middleware.rs | 20 ++++++++++++ 5 files changed, 100 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/middleware.rs b/crates/trusted-server-adapter-axum/src/middleware.rs index fd11d7728..f099bc15b 100644 --- a/crates/trusted-server-adapter-axum/src/middleware.rs +++ b/crates/trusted-server-adapter-axum/src/middleware.rs @@ -53,8 +53,9 @@ impl Middleware for SanitizeRequestMiddleware { /// Response-finalization middleware: injects all standard TS response headers. /// /// Geo lookup is unavailable in the Axum dev server — `X-Geo-Info-Available: false` -/// is always emitted. Fastly-specific headers (`X-TS-Version`, `X-TS-ENV`) are -/// skipped because the corresponding env vars are not set in a local dev context. +/// is always emitted. `X-TS-Version` carries the compiled-in git version. The +/// Fastly-specific headers (`X-TS-Fastly-Version`, `X-TS-ENV`) are skipped +/// because the corresponding env vars are not set in a local dev context. /// /// Registered directly inside [`SanitizeRequestMiddleware`] and ahead of /// [`AuthMiddleware`] so that every outgoing response — including auth-rejected @@ -125,7 +126,8 @@ impl Middleware for AuthMiddleware { /// Applies standard Trusted Server response headers to the given response. /// /// Unlike the Fastly variant, geo is always unavailable so `X-Geo-Info-Available: false` -/// is unconditionally emitted. Fastly-specific headers are omitted. +/// is unconditionally emitted, followed by the compiled-in `X-TS-Version`. +/// Fastly-specific headers are omitted. /// Operator-configured `settings.response_headers` are applied last (with the /// shared cookie cache-privacy hardening) and can override any managed header. pub(crate) fn apply_finalize_headers(settings: &Settings, response: &mut Response) { @@ -134,6 +136,8 @@ pub(crate) fn apply_finalize_headers(settings: &Settings, response: &mut Respons HeaderValue::from_static("false"), ); + trusted_server_core::version_header::apply_git_version_header(response); + // Cookie-bearing responses stay private to shared caches and operator // headers cannot re-enable caching for uncacheable per-user payloads. trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( @@ -287,4 +291,18 @@ mod tests { "should remove both configured trust headers before the handler" ); } + + #[test] + fn emits_git_version_header() { + let mut response = empty_response(); + apply_finalize_headers(&settings_with_response_headers(vec![]), &mut response); + assert_eq!( + response + .headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + } } diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 7126b2a71..bcb31b7e4 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -938,3 +938,35 @@ async fn nextjs_auction_output_holds_until_the_structural_body_close() { "should not leak generated placeholders: {html}" ); } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn health_reports_git_version() { + let mut service = make_service(); + let request = Request::builder() + .method("GET") + .uri("/health") + .body(AxumBody::empty()) + .expect("should build health request"); + + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should serve health"); + + assert_eq!( + response.status().as_u16(), + 200, + "should return 200 on /health" + ); + assert_eq!( + response + .headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version on /health" + ); +} diff --git a/crates/trusted-server-adapter-cloudflare/src/middleware.rs b/crates/trusted-server-adapter-cloudflare/src/middleware.rs index 14efed56a..a22b51982 100644 --- a/crates/trusted-server-adapter-cloudflare/src/middleware.rs +++ b/crates/trusted-server-adapter-cloudflare/src/middleware.rs @@ -146,6 +146,8 @@ pub(crate) fn apply_finalize_headers( HeaderValue::from_static(if geo_available { "true" } else { "false" }), ); + trusted_server_core::version_header::apply_git_version_header(response); + // Cloudflare is a real shared cache: cookie-bearing responses must stay // private and operator headers must not re-enable caching for uncacheable // per-user payloads. @@ -320,4 +322,22 @@ mod tests { "should remove both configured trust headers before the handler" ); } + + #[test] + fn emits_git_version_header() { + let mut response = empty_response(); + apply_finalize_headers( + &settings_with_response_headers(vec![]), + false, + &mut response, + ); + assert_eq!( + response + .headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + } } diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 821b52646..d248f2f15 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -1367,6 +1367,13 @@ mod tests { 200, "GET /health must return 200 from the startup fallback" ); + assert_eq!( + resp.headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "startup-fallback /health should report the compiled-in git version" + ); let body = resp.into_body().into_bytes().unwrap_or_default(); assert_eq!( &body[..], diff --git a/crates/trusted-server-adapter-spin/src/middleware.rs b/crates/trusted-server-adapter-spin/src/middleware.rs index d7a09987a..fc1b939bd 100644 --- a/crates/trusted-server-adapter-spin/src/middleware.rs +++ b/crates/trusted-server-adapter-spin/src/middleware.rs @@ -174,6 +174,8 @@ pub(crate) fn apply_finalize_headers( HeaderValue::from_static(if geo_available { "true" } else { "false" }), ); + trusted_server_core::version_header::apply_git_version_header(response); + // Cookie-bearing responses stay private to shared caches and operator // headers cannot re-enable caching for uncacheable per-user payloads. trusted_server_core::response_privacy::apply_response_headers_with_cache_privacy( @@ -347,4 +349,22 @@ mod tests { "should remove both configured trust headers before the handler" ); } + + #[test] + fn emits_git_version_header() { + let mut response = empty_response(); + apply_finalize_headers( + &settings_with_response_headers(vec![]), + false, + &mut response, + ); + assert_eq!( + response + .headers() + .get("x-ts-version") + .and_then(|v| v.to_str().ok()), + trusted_server_core::constants::TS_GIT_VERSION, + "should report the compiled-in git version as x-ts-version" + ); + } } From cd704a7aacb4df866c471cfafe3cf6d130ad5fcb Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 25 Sep 2026 17:01:23 +0530 Subject: [PATCH 103/230] Document the x-ts-version and x-ts-fastly-version split Signed-off-by: dhruv8sh --- CHANGELOG.md | 1 + docs/guide/first-party-proxy.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 083d6e746..847914204 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER_GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** Auction providers and bidder routes now use the configuration-first `[auction.providers.]` and `[auction.bidders.]` maps. The removed `[auction].providers = [...]` list and removed server fields under `[integrations.prebid]` and `[integrations.aps]` are rejected even when those integrations are disabled, and `ts config push` rejects the old shape before publication. Move PBS `server_url` to provider `endpoint`, server timeout to provider `timeout_ms`, request controls and bidder-parameter overrides to the `prebid-server` `profile_config`, notification suppression to `notifications`, and each former server bidder to an `[auction.bidders.]` route. Move APS endpoint, timeout, account, inventory, debug, and creative controls to an `aps` provider and its `profile_config`. Browser Prebid settings remain under `[integrations.prebid]`; values such as timeout and debug that previously affected both browser and server behavior must now be configured for each owner. Provider endpoints must be absolute HTTPS URLs. Only bidder codes present in `[auction.bidders]` are folded into Trusted Server requests; unlisted publisher bids remain native browser demand. Provider response names now use the configured provider ID, such as `pbs-main`, instead of the legacy literal `prebid`; audit consumers that match `AuctionResponse.provider`. This schema has no mixed-version-safe deployment order: old binaries reject the maps and new binaries reject the retired fields, so activate the new binary and config blob together. Rollbacks must restore an old-schema blob together with the old binary. - **Breaking** — Admin Basic-auth coverage now includes `GET /_ts/admin/ec`, `GET /_ts/admin/ec/{id}`, and `GET /_ts/admin/eids`. Existing configurations whose `[[handlers]]` patterns protect only the key-management endpoints now fail startup; broaden coverage before deploying, preferably with a namespace-boundary pattern such as `^/_ts/admin(?:/|$)`. Coverage of the dynamic `/_ts/admin/ec/{id}` route is no longer inferred from ID-shaped samples: the router accepts any segment after `/_ts/admin/ec/` and Basic Auth runs on the raw path before routing, so patterns anchored to the EC ID grammar (for example `^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$`) are rejected in favor of a prefix-level matcher. Placeholder and well-known weak handler passwords (`changeme`, `password`, `admin`, `replace-with-…`) now fail startup on every handler rather than only on handlers inferred to cover an admin endpoint, because first-match-wins handler selection lets a narrow handler shadow the admin namespace. - Prebid Server provider endpoints now normalize origin-only legacy `server_url` values to `/openrtb2/auction`. Query parameters are preserved, the canonical path loses a trailing slash, and configured non-root custom paths remain exact. diff --git a/docs/guide/first-party-proxy.md b/docs/guide/first-party-proxy.md index 8b9056f0e..69d4ed7c6 100644 --- a/docs/guide/first-party-proxy.md +++ b/docs/guide/first-party-proxy.md @@ -716,7 +716,7 @@ Add custom headers for debugging: ```toml [response_headers] X-Proxy-Mode = "rewrite" -X-TS-Version = "1.0" +X-Debug-Build = "canary" ``` ### Metrics to Track From 819ac714f3c2f16e267ce91f4e8c8f06ebd99af7 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:13:15 +0530 Subject: [PATCH 104/230] Parse mock bid dimensions with the shared OpenRTB helper Reuse parse_optional_bid_dimension so integral float dimensions are accepted like in the other auction providers, and pin that behavior with a test. Signed-off-by: dhruv8sh --- .../src/integrations/adserver_mock.rs | 79 ++++++++++++++++--- 1 file changed, 67 insertions(+), 12 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 066e32e28..21cb41341 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -16,6 +16,7 @@ use std::time::Duration; use validator::Validate; use crate::auction::context::{ContextQueryParams, build_url_with_context_params}; +use crate::auction::openrtb::parse_optional_bid_dimension; use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; use crate::auction::types::{ AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType, @@ -292,28 +293,25 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); - // Keep `{:?}` for the raw upstream values below: Debug escapes - // newlines and quotes, which prevents log injection. - let Some(width) = bid["w"].as_u64().and_then(|v| u32::try_from(v).ok()) else { + // Reuse the shared `OpenRTB` dimension parser so integral floats + // (`300.0`) are accepted and zero, negative, or oversized values + // are rejected, matching the other auction providers. Keep `{:?}` + // for the raw upstream values below: Debug escapes newlines and + // quotes, which prevents log injection. + let Ok(Some(width)) = parse_optional_bid_dimension(bid, "w") else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", bid["w"] ); continue; }; - let Some(height) = bid["h"].as_u64().and_then(|v| u32::try_from(v).ok()) else { + let Ok(Some(height)) = parse_optional_bid_dimension(bid, "h") else { log::debug!( "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", bid["h"] ); continue; }; - if width == 0 || height == 0 { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has zero dimension ({width}×{height}), skipping" - ); - continue; - } all_bids.push(Bid { slot_id, @@ -1341,8 +1339,8 @@ mod tests { fn test_parse_mediation_response_skips_oversized_dimensions() { // A dimension above u32::MAX must be rejected rather than silently // wrapped into a small, plausible-looking value. The offset of 101 is - // load-bearing: u32::MAX + 1 truncates to 0, which the zero-check - // below would already have caught, so it would not pin this fix. + // load-bearing: u32::MAX + 1 truncates to 0, which is already rejected + // as a zero dimension, so it would not pin this fix. let config = AdServerMockConfig::default(); let provider = AdServerMockProvider::new(config); @@ -1563,6 +1561,63 @@ mod tests { ); } + #[test] + fn test_parse_mediation_response_accepts_integral_float_dimensions() { + // Integral floats (`300.0`) are a legitimate dimension encoding, matching + // the shared `OpenRTB` parser; fractional floats are still skipped. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-float-dimensions", + "impid": "header-banner", + "price": 3.50, + "adm": "
Float dimensions
", + "w": 300.0, + "h": 250.0, + }, + { + "id": "bid-fractional-width", + "impid": "sidebar", + "price": 1.25, + "adm": "
Fractional width
", + "w": 300.5, + "h": 250, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + + assert_eq!( + auction_response.bids.len(), + 1, + "should accept the integral-float bid and skip the fractional one" + ); + assert_eq!( + auction_response.bids[0].slot_id, "header-banner", + "should keep the integral-float bid" + ); + assert_eq!( + ( + auction_response.bids[0].width, + auction_response.bids[0].height + ), + (300, 250), + "should convert integral floats to u32 dimensions" + ); + } + #[test] fn test_build_endpoint_url_with_context_query_params() { let config = AdServerMockConfig { From 7eed6a3f2a4bfa0a25abd7c738087162f16f738d Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:39:04 +0530 Subject: [PATCH 105/230] Validate mock mediated bid dimensions against requested slots Build the requested-dimension index from the auction request and run mediated bids through resolve_bid_dimensions, so the mock provider rejects unrequested impressions and mismatched sizes and infers missing dimensions like the other auction providers. Signed-off-by: dhruv8sh --- .../src/auction/openrtb.rs | 17 +- .../src/integrations/adserver_mock.rs | 164 ++++++++++++++---- 2 files changed, 142 insertions(+), 39 deletions(-) diff --git a/crates/trusted-server-core/src/auction/openrtb.rs b/crates/trusted-server-core/src/auction/openrtb.rs index cf657a1ec..d2a988db3 100644 --- a/crates/trusted-server-core/src/auction/openrtb.rs +++ b/crates/trusted-server-core/src/auction/openrtb.rs @@ -16,7 +16,7 @@ use super::profile::{ use super::routing::{ PrebidTransportHeaders, ProviderAuctionInput, ProviderSlotInput, RoutedAuction, }; -use super::types::{AdFormat, AuctionResponse, Bid}; +use super::types::{AdFormat, AdSlot, AuctionResponse, Bid}; use crate::consent::ConsentSource; use crate::error::TrustedServerError; use crate::openrtb::{ @@ -154,11 +154,20 @@ pub(crate) type BidDimensionIndex = BTreeMap; /// Build the requested-dimension index once for one provider response. pub(crate) fn build_bid_dimension_index(input: &ProviderAuctionInput) -> BidDimensionIndex { + build_bid_dimension_index_from_slots(input.slots().iter().map(ProviderSlotInput::slot)) +} + +/// Build the requested-dimension index from plain [`AdSlot`]s. +/// +/// The first slot wins when several share an ID. +pub(crate) fn build_bid_dimension_index_from_slots<'a>( + slots: impl IntoIterator, +) -> BidDimensionIndex { let mut index = BidDimensionIndex::new(); - for slot in input.slots() { + for slot in slots { index - .entry(slot.slot().id.clone()) - .or_insert_with(|| SlotBidDimensions::from_formats(slot.slot().formats.as_slice())); + .entry(slot.id.clone()) + .or_insert_with(|| SlotBidDimensions::from_formats(slot.formats.as_slice())); } index } diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 21cb41341..640574e03 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -16,7 +16,10 @@ use std::time::Duration; use validator::Validate; use crate::auction::context::{ContextQueryParams, build_url_with_context_params}; -use crate::auction::openrtb::parse_optional_bid_dimension; +use crate::auction::openrtb::{ + BidDimensionIndex, BidRejectionReason, build_bid_dimension_index_from_slots, + parse_optional_bid_dimension, resolve_bid_dimensions, +}; use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; use crate::auction::types::{ AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType, @@ -265,6 +268,7 @@ impl AdServerMockProvider { json: &Json, response_time_ms: u64, bid_index: &BidIndex, + dimensions_by_slot: Option<&BidDimensionIndex>, ) -> AuctionResponse { let empty_array = vec![]; let seatbid = json["seatbid"].as_array().unwrap_or(&empty_array); @@ -293,24 +297,31 @@ impl AdServerMockProvider { let restored_bidder = original.map_or_else(|| seat_name.to_string(), |b| b.bidder.clone()); - // Reuse the shared `OpenRTB` dimension parser so integral floats - // (`300.0`) are accepted and zero, negative, or oversized values - // are rejected, matching the other auction providers. Keep `{:?}` - // for the raw upstream values below: Debug escapes newlines and - // quotes, which prevents log injection. - let Ok(Some(width)) = parse_optional_bid_dimension(bid, "w") else { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid width {:?}, skipping", - bid["w"] - ); - continue; + // Reuse the shared `OpenRTB` dimension parser and slot-format + // validation so this provider admits the same dimensions as the + // other auction providers. Keep `{:?}` for the raw upstream + // values below: Debug escapes newlines and quotes, which + // prevents log injection. + let dimensions = match ( + parse_optional_bid_dimension(bid, "w"), + parse_optional_bid_dimension(bid, "h"), + ) { + (Ok(width), Ok(height)) => match dimensions_by_slot { + Some(index) => resolve_bid_dimensions(index, &slot_id, width, height), + None => width.zip(height).ok_or(BidRejectionReason::InvalidBid), + }, + _ => Err(BidRejectionReason::InvalidBid), }; - let Ok(Some(height)) = parse_optional_bid_dimension(bid, "h") else { - log::debug!( - "adserver_mock: bid for slot '{slot_id}' has invalid height {:?}, skipping", - bid["h"] - ); - continue; + let (width, height) = match dimensions { + Ok(dimensions) => dimensions, + Err(reason) => { + log::debug!( + "adserver_mock: bid for slot '{slot_id}' has unusable dimensions {:?}×{:?} ({reason:?}), skipping", + bid["w"], + bid["h"] + ); + continue; + } }; all_bids.push(Bid { @@ -372,6 +383,7 @@ impl AdServerMockProvider { response: PlatformResponse, response_time_ms: u64, bid_index: &BidIndex, + dimensions_by_slot: Option<&BidDimensionIndex>, ) -> Result> { let response = response.response; @@ -397,8 +409,12 @@ impl AdServerMockProvider { log::trace!("AdServer Mock response: {:?}", response_json); - let auction_response = - self.parse_mediation_response(&response_json, response_time_ms, bid_index); + let auction_response = self.parse_mediation_response( + &response_json, + response_time_ms, + bid_index, + dimensions_by_slot, + ); log::info!( "AdServer Mock returned {} bids in {}ms", @@ -517,7 +533,7 @@ impl AuctionProvider for AdServerMockProvider { // [`parse_response_with_context`], so this path only serves callers // outside the orchestration flow. log::debug!("adserver_mock: parsing without context — SSP bid metadata unavailable"); - self.parse_response_inner(response, response_time_ms, &BidIndex::new()) + self.parse_response_inner(response, response_time_ms, &BidIndex::new(), None) .await } @@ -525,15 +541,21 @@ impl AuctionProvider for AdServerMockProvider { &self, response: PlatformResponse, response_time_ms: u64, - _request: &AuctionRequest, + request: &AuctionRequest, context: &AuctionContext<'_>, ) -> Result> { // Rebuild the SSP-bid lookup from the orchestrator-provided bidder // responses so nurl/burl/ad_id survive mediation. Request-scoped data // travels on the context instead of provider-instance state. let bid_index = build_bid_index(context.provider_responses.unwrap_or(&[])); - self.parse_response_inner(response, response_time_ms, &bid_index) - .await + let dimensions_by_slot = build_bid_dimension_index_from_slots(&request.slots); + self.parse_response_inner( + response, + response_time_ms, + &bid_index, + Some(&dimensions_by_slot), + ) + .await } fn supports_media_type(&self, media_type: &MediaType) -> bool { @@ -801,7 +823,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!(auction_response.provider, "adserver_mock"); assert_eq!(auction_response.status, BidStatus::Success); @@ -848,7 +870,8 @@ mod tests { ] }); - let response = provider.parse_mediation_response(&mediation_response, 10, &BidIndex::new()); + let response = + provider.parse_mediation_response(&mediation_response, 10, &BidIndex::new(), None); assert_eq!(response.bids.len(), 2); assert_eq!(response.bids[0].bidder, "provider-instance"); @@ -924,7 +947,7 @@ mod tests { ); let auction_response = - provider.parse_mediation_response(&mediation_response, 42, &bid_index); + provider.parse_mediation_response(&mediation_response, 42, &bid_index, None); assert_eq!(auction_response.status, BidStatus::Success); assert_eq!(auction_response.bids.len(), 1); @@ -1028,7 +1051,7 @@ mod tests { ); let auction_response = - provider.parse_mediation_response(&mediation_response, 42, &bid_index); + provider.parse_mediation_response(&mediation_response, 42, &bid_index, None); assert_eq!( auction_response.bids[0].bid_id.as_deref(), @@ -1092,6 +1115,7 @@ mod tests { }), 2, &reduced_index, + None, ); let winner = mediated .bids @@ -1123,7 +1147,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 100, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 100, &BidIndex::new(), None); assert_eq!(auction_response.status, BidStatus::NoBid); assert_eq!(auction_response.bids.len(), 0); @@ -1316,7 +1340,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!(auction_response.status, BidStatus::Success); assert_eq!(auction_response.bids.len(), 2); @@ -1382,7 +1406,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1443,7 +1467,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); let slots: Vec<&str> = auction_response .bids @@ -1485,7 +1509,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1547,7 +1571,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); let slots: Vec<&str> = auction_response .bids @@ -1597,7 +1621,7 @@ mod tests { }); let auction_response = - provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new()); + provider.parse_mediation_response(&mediation_response, 200, &BidIndex::new(), None); assert_eq!( auction_response.bids.len(), @@ -1618,6 +1642,76 @@ mod tests { ); } + #[test] + fn test_parse_mediation_response_validates_dimensions_against_slots() { + // With the request slots available, mediated bids must match a + // requested format, and missing dimensions are inferred from the + // slot's single format, matching the other auction providers. + let config = AdServerMockConfig::default(); + let provider = AdServerMockProvider::new(config); + let request = create_test_auction_request(); + let dimensions_by_slot = build_bid_dimension_index_from_slots(&request.slots); + + let mediation_response = json!({ + "id": "test-auction-123", + "seatbid": [ + { + "seat": "test-bidder", + "bid": [ + { + "id": "bid-exact", + "impid": "header-banner", + "price": 3.50, + "w": 728, + "h": 90, + }, + { + "id": "bid-mismatch", + "impid": "header-banner", + "price": 3.00, + "w": 300, + "h": 250, + }, + { + "id": "bid-inferred", + "impid": "header-banner", + "price": 2.50, + }, + { + "id": "bid-unrequested", + "impid": "sidebar", + "price": 1.25, + "w": 728, + "h": 90, + } + ] + } + ], + "cur": "USD" + }); + + let auction_response = provider.parse_mediation_response( + &mediation_response, + 200, + &BidIndex::new(), + Some(&dimensions_by_slot), + ); + + let admitted: Vec<(Option<&str>, u32, u32)> = auction_response + .bids + .iter() + .map(|bid| (bid.bid_id.as_deref(), bid.width, bid.height)) + .collect(); + assert_eq!( + admitted, + [ + (Some("bid-exact"), 728, 90), + (Some("bid-inferred"), 728, 90) + ], + "should keep the exact and inferred bids and drop the mismatched and unrequested ones" + ); + } + #[test] fn test_build_endpoint_url_with_context_query_params() { let config = AdServerMockConfig { From 44fb9c2dd4c8644c5e21e9bbb9436307b5a3eca4 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 13:40:47 +0530 Subject: [PATCH 106/230] Extend the Markdown format check beyond the repo root Check every tracked Markdown file outside docs/ with Prettier, and reformat the two files that failed. In pr-reviewer.md, escape BRANCH-* so Prettier does not read it as emphasis, and keep long code spans on one line so they are not split across list-item wraps. Signed-off-by: dhruv8sh --- .claude/agents/pr-reviewer.md | 74 ++++++++++--------- .../SKILL.md | 6 +- .github/workflows/format.yml | 6 +- AGENTS.md | 2 +- 4 files changed, 46 insertions(+), 42 deletions(-) diff --git a/.claude/agents/pr-reviewer.md b/.claude/agents/pr-reviewer.md index 4f65a426f..66bc88c10 100644 --- a/.claude/agents/pr-reviewer.md +++ b/.claude/agents/pr-reviewer.md @@ -27,13 +27,13 @@ The agent resolves the input into exactly one of three **modes** before starting any fetch. Modes determine the per-invocation variables defined in step 1; later steps reference those variables and don't restate mode logic. -| Input | Mode (after resolution) | -|---|---| -| A PR number (e.g. `#165`) | **PR** | -| A branch name, and PR lookup returns exactly one matching PR | **PR** (via lookup) | -| A branch name, no PR exists | **BRANCH-REMOTE** — always. No probe on the current checkout. | -| User explicitly says "review my local working tree" | **BRANCH-LOCAL** — current checkout only. If the user also names a branch, the agent verifies it matches `git branch --show-current`; otherwise it stops and asks the user to either check out that branch first or drop the name. The agent does **not** silently review whatever HEAD happens to be. | -| No input | Run the PR lookup probe with `$(git branch --show-current)`. If it returns a PR → PR mode. Otherwise apply the no-input rule below. | +| Input | Mode (after resolution) | +| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| A PR number (e.g. `#165`) | **PR** | +| A branch name, and PR lookup returns exactly one matching PR | **PR** (via lookup) | +| A branch name, no PR exists | **BRANCH-REMOTE** — always. No probe on the current checkout. | +| User explicitly says "review my local working tree" | **BRANCH-LOCAL** — current checkout only. If the user also names a branch, the agent verifies it matches `git branch --show-current`; otherwise it stops and asks the user to either check out that branch first or drop the name. The agent does **not** silently review whatever HEAD happens to be. | +| No input | Run the PR lookup probe with `$(git branch --show-current)`. If it returns a PR → PR mode. Otherwise apply the no-input rule below. | **Branch-to-PR lookup rule.** `gh pr list --head ` does not support `:` syntax, so fork PRs with the same branch name can collide. @@ -52,7 +52,7 @@ matches=$(gh pr list --head "$REQUESTED_HEAD" \ - `>1` matches → stop and ask the user which PR number to review; do not pick the first row. -**No-input / no-PR rule** (the only place an inferred BRANCH-* mode happens +**No-input / no-PR rule** (the only place an inferred BRANCH-\* mode happens — a named branch never triggers this probe; the user said the name, the agent honours it): @@ -60,7 +60,7 @@ agent honours it): configured **and** `git rev-list --left-right --count "@{upstream}...HEAD"` returns `0 0` → resolve to **BRANCH-REMOTE**, with `` bound to the branch name the upstream points at, not to the local branch name. The - probe approves the *upstream* state; the agent must fetch that exact + probe approves the _upstream_ state; the agent must fetch that exact branch, not `origin/$(git branch --show-current)` which could be a different ref (the local branch might track `origin/main-fork`). Because BRANCH-REMOTE fetches from `origin`, this inference only applies when the @@ -87,6 +87,7 @@ agent honours it): When `$REQUESTED_HEAD` is bound, BRANCH-REMOTE proceeds with that name. When it's not bound (upstream on a non-`origin` remote), the agent asks the user instead. + - Anything else → **ask the user** which mode they want. The probe uses only `git status` / `git rev-parse` on existing local refs — it does **not** fetch, so the choice is made before any network or worktree side effect. @@ -283,7 +284,7 @@ fi This block adds the worktree **at most once** per invocation, and reuses the one from a prior invocation rather than adding a second. The decision keys off -git's worktree *registry* — the source of truth — not just the directory's +git's worktree _registry_ — the source of truth — not just the directory's existence, because the two can disagree (a registered worktree whose directory was manually `rm -rf`'d, or a leftover directory git never registered). Keying off the directory alone would send a dangling-but-registered path into @@ -345,9 +346,9 @@ After this step every later step uses `${WT:-.}` for cwd (so BRANCH-LOCAL implicitly runs from the project root) and `$DIFF_RANGE` for diffs. There are no more per-mode forks until step 7e (which checks `[ -n "$WT" ]` for scratch verification) and step 8, where only **step 8b (the GitHub review -submission)** is skipped for BRANCH-* modes — no PR to submit a review to. +submission)** is skipped for BRANCH-\* modes — no PR to submit a review to. Step 8a still runs in every mode to compose the review artifact and, in -BRANCH-* modes, render the would-have-been verdict and findings into chat. +BRANCH-\* modes, render the would-have-been verdict and findings into chat. Stash the `HEAD_OID_EXPECTED` value — step 8 re-checks it immediately before submission and pins it into the review payload as `commit_id`. @@ -449,7 +450,7 @@ fi If the PR has passing CI checks, report them as PASS in the review. Only run CI locally if checks haven't run yet or if you need to verify a specific failure. Note any CI failures in the review but continue with the code review -regardless. (This governs CI *status reporting* only — the suggestion +regardless. (This governs CI _status reporting_ only — the suggestion scratch-verification in step 7e is independent and runs regardless of what GitHub's checks say.) @@ -647,7 +648,7 @@ numbers it covers; pick `line` (and `start_line` when multi-line) from inside the same hunk; don't span hunk boundaries. **Deleted lines and base-side context.** Findings about something the PR -*removed* don't live on the RIGHT side at all — there are no new-file lines +_removed_ don't live on the RIGHT side at all — there are no new-file lines to anchor to. Two options: - Anchor the inline comment on the LEFT side: `"side": "LEFT"` (and @@ -662,7 +663,7 @@ to anchor to. Two options: see step 1's "base-side reads" note.) Same rule for renamed files: the file's new path can carry a `suggestion` -block normally; comments about content the rename *also dropped* anchor on +block normally; comments about content the rename _also dropped_ anchor on the old path with `side: "LEFT"`. Use a `suggestion` block when: @@ -693,12 +694,12 @@ fenced code block when: cleanly revised. In all of those cases, give the proposed code in a plain fenced block (e.g. -```` ```rust ````) and end with a short "Apply manually — can't be auto-applied +` ```rust `) and end with a short "Apply manually — can't be auto-applied as a suggestion because …" sentence. ##### Fence length when the replacement itself contains backticks -The default `` ```suggestion `` fence is three backticks. If the replacement +The default ` ```suggestion ` fence is three backticks. If the replacement bytes contain a line that is itself a run of three-or-more backticks — common for Markdown/docs suggestions that include a nested code fence — that inner run closes the outer `suggestion` block early, and the rendered one-click @@ -706,14 +707,14 @@ suggestion is truncated or malformed rather than matching the bytes the user approved. Before displaying or submitting any suggestion: 1. Scan the replacement for the longest run of consecutive backticks, `N`. -2. If `N < 3`, use the normal three-backtick `` ```suggestion `` fence. +2. If `N < 3`, use the normal three-backtick ` ```suggestion ` fence. 3. If `N >= 3`, open and close the block with a fence of `N + 1` backticks - (e.g. ` ````suggestion ` for an inner ```` ``` ````), so the outer fence is + (e.g. ` ````suggestion ` for an inner ` ``` `), so the outer fence is strictly longer than any inner run — GitHub follows the CommonMark rule that a fence closes only on a run of **at least** as many backticks. This rule is deterministic; whether GitHub renders the widened fence as a one-click suggestion is a server-side property that cannot be checked locally (7e's - scratch pass verifies replacement *bytes*, not rendering). When in doubt — + scratch pass verifies replacement _bytes_, not rendering). When in doubt — e.g. an unusually exotic replacement — **demote the finding to prose-only** (a plain fenced block plus the "Apply manually …" sentence) rather than risk posting a malformed suggestion. @@ -753,7 +754,7 @@ For a multi-line suggestion, add `start_line` and `start_side`: **Indentation matters**: the block replaces the original lines verbatim, so leading whitespace must match exactly what the file expects after the fix. -**Fence length matters too**: the `` ```suggestion `` fences above use three +**Fence length matters too**: the ` ```suggestion ` fences above use three backticks, which only holds when the replacement contains no three-or-more backtick run of its own. When it does (e.g. a docs suggestion with a nested code fence), widen the outer fence per step 7a's fence-length rule or demote @@ -776,7 +777,7 @@ the comment body and tell the author it has to be applied manually: #### 7c-bis. Inline comment on a removed (LEFT-side) line -A finding about a line the PR *removed* has no RIGHT-side anchor — pin it +A finding about a line the PR _removed_ has no RIGHT-side anchor — pin it on the LEFT (base) side instead. `suggestion` blocks aren't applicable (GitHub only commits suggestions from the RIGHT side), so the body uses a plain code block: @@ -805,8 +806,8 @@ can't straddle sides. - The total number of inline comments has a soft cap of ~30. If you would exceed that, consolidate the lowest-severity findings into the review body with file/line references but no inline comment. -- A given inline comment may contain at most one ```` ```suggestion ```` block. - Prose context blocks (e.g. ```` ```rust ````) are fine alongside it. +- A given inline comment may contain at most one ` ```suggestion ` block. + Prose context blocks (e.g. ` ```rust `) are fine alongside it. - If the user changed an emoji tag during triage, the comment uses the new tag. - Don't post suggestions on lines outside the RIGHT side of the diff — they'll fail GitHub's "position could not be resolved" check. Comments about @@ -838,7 +839,7 @@ suggestion A only compiles because suggestion B was also applied, the agent has labelled A as verified but A-alone can break the build. So the inner loop tests each suggestion against a clean worktree first; a final batch pass (all approved suggestions applied together) is a nice-to-have to -catch *interactions*, but the per-suggestion runs are the real gate: +catch _interactions_, but the per-suggestion runs are the real gate: ```bash # Confirm clean starting state at $WT (HEAD = $HEAD_REF, status empty). @@ -931,8 +932,9 @@ gate when **any** of these is true: - The suggestion touches a `#[cfg(test)]` module, a test, or a feature gate. - The finding is 🔧 wrench (blocking) — release-blocking fixes must clear the release gate. -- The touched code is shared (`crates/trusted-server-core/src/{auction,ec, - http_util,publisher,html_processor,settings,constants}` and similar). +- The touched code is shared + (`crates/trusted-server-core/src/{auction,ec,http_util,publisher,html_processor,settings,constants}` + and similar). - The suggestion changes program behaviour and the agent prefers to ship it **without** the compile-verified-only disclaimer. @@ -967,11 +969,11 @@ at build time. Run the build whenever the suggestion touches files under **Post-verify drift check (snapshot the approved patch, hard-fail on any deviation).** Filename-level comparison isn't enough — a formatter or -codegen step can change a different range *inside* an approved file, while +codegen step can change a different range _inside_ an approved file, while the posted GitHub suggestion still contains only the originally-approved range. The correct check is byte-exact: snapshot the full patch immediately after applying the approved suggestions but **before** running any -verification command, then compare with the patch *after* verification. Any +verification command, then compare with the patch _after_ verification. Any delta — different range in the same file, an extra tracked file, a whitespace change in `Cargo.toml` from a build script — means verification mutated the tree beyond what the agent approved, and the suggestion as @@ -1060,7 +1062,7 @@ After cleanup, the worktree's HEAD must be at `$HEAD_REF` Step 8 splits into two halves. **8a** is mode-agnostic: determine the verdict and compose the review body + inline comments. **8b** is PR-only: post the -review to GitHub. BRANCH-* modes still produce the artifact in 8a (so step 10 +review to GitHub. BRANCH-\* modes still produce the artifact in 8a (so step 10 has a "would-have-been verdict" to report) but skip 8b — the artifact is rendered into the chat instead. @@ -1164,14 +1166,14 @@ inline comment", not "this finding is out of scope". Omit any section that has no findings — don't include empty headings. -In BRANCH-* modes (`[ -z "$NUMBER" ]`) the artifact is now complete: render it +In BRANCH-\* modes (`[ -z "$NUMBER" ]`) the artifact is now complete: render it into the chat exactly as the GitHub UI would have shown it (body markdown followed by each inline comment, labelled with the file:line it would have anchored to). Stop after rendering — there is no review to submit. #### 8b. Submit the GitHub review (PR mode only — `[ -n "$NUMBER" ]`) -Skip this entire sub-step in BRANCH-* modes. +Skip this entire sub-step in BRANCH-\* modes. ##### Re-check the PR head before submission @@ -1197,7 +1199,7 @@ fi If `SKIP_SUBMISSION` is set, the agent **must skip every command in the "Submit the review" sub-section below**, render the artifact in chat (the -same way BRANCH-* mode would in 8a), and report `submission skipped: +same way BRANCH-\* mode would in 8a), and report `submission skipped: $SKIP_REASON` as the stop reason in step 10. The agent does not restart inside the same invocation — single-invocation = single pass; the user re-invokes if they want another pass against the new head. @@ -1218,7 +1220,7 @@ fi ``` When `STOP_SUBMISSION=1`, render the composed artifact into chat exactly the -way BRANCH-* mode does in 8a, report `submission skipped: $SKIP_REASON` per +way BRANCH-\* mode does in 8a, report `submission skipped: $SKIP_REASON` per step 10, and do not run any of the `gh api` submit/delete commands below. Use the GitHub API to submit. Handle these known issues: @@ -1307,7 +1309,7 @@ Use the GitHub API to submit. Handle these known issues: fallthrough, no `exit 1`. **Re-gate after the pending-review check.** The `SKIP_SUBMISSION` guard - at the top of "Submit the review" runs *before* the pending-review check, + at the top of "Submit the review" runs _before_ the pending-review check, so a `keep` decision must be caught by a second gate immediately after the case block — otherwise the JSON-assembly / `gh api … -X POST` below would still run: @@ -1349,7 +1351,7 @@ head-recheck above. **Invariant — never heredoc-interpolate user content into the payload.** Inline-comment bodies routinely contain `$`, backticks, backslashes, -quoted code, and the literal `` ```suggestion `` fence. A `cat <` (binary name `ts`), or install it with - `cargo install --path crates/trusted-server-cli`. +- The `ts` CLI runs from source without installing: + `cargo run -p trusted-server-cli -- config ` (binary name `ts`), or + install it with `cargo install --path crates/trusted-server-cli`. - The app-config store's logical id and blob key are `trusted_server_config` (`settings_data.rs` `DEFAULT_CONFIG_STORE_ID`, `config_payload.rs` `CONFIG_BLOB_KEY`); older builds used `app_config`. The override env-var key diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index c844e95d3..93bc99229 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -149,9 +149,11 @@ jobs: - name: Run Prettier (check) run: npm run format - - name: Run Prettier (check) — root Markdown + - name: Run Prettier (check) — Markdown outside docs/ working-directory: . - run: docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" + run: >- + docs/node_modules/.bin/prettier --config docs/.prettierrc --check + "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md" - name: Build with VitePress (fails on dead links) run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 9641e79fc..738fcab34 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -351,7 +351,7 @@ Every PR must pass: 5. JS build and test (`cd crates/trusted-server-js/lib && npx vitest run`) 6. JS format (`cd crates/trusted-server-js/lib && npm run format`) 7. Docs format (`cd docs && npm run format`) -8. Root Markdown format (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md"`; fix with `--write` in place of `--check` +8. Markdown format outside `docs/` (requires `cd docs && npm ci` first): `docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md"`; fix with `--write` in place of `--check` --- From d7dcb183a70e4d51379d0ff942e9785cbc16cbf4 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 28 Sep 2026 14:13:08 +0530 Subject: [PATCH 107/230] Test that both partner token placeholders are reported together Let the partner test helper take an api_token, and assert that one partner with placeholder api_token and ts_pull_token values reports both fields in a single error. Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/settings.rs | 31 +++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 47f614387..5eb2e070b 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -5418,12 +5418,16 @@ source_domain = "partner.example.com" } fn test_partner_with_pull_token(ts_pull_token: &str) -> EcPartner { + test_partner_with_tokens(None, ts_pull_token) + } + + fn test_partner_with_tokens(api_token: Option<&str>, ts_pull_token: &str) -> EcPartner { EcPartner { name: "Test Partner".to_owned(), source_domain: "partner.example.com".to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), bidstream_enabled: false, - api_token: None, + api_token: api_token.map(|token| Redacted::new(token.to_owned())), batch_rate_limit: EcPartner::default_batch_rate_limit(), pull_sync_enabled: true, pull_sync_url: Some("https://partner.example.com/sync".to_owned()), @@ -5468,6 +5472,31 @@ source_domain = "partner.example.com" .expect("should accept a realistic partner pull token"); } + #[test] + fn reject_placeholder_secrets_reports_both_partner_tokens() { + let mut settings = + Settings::from_toml(&crate_test_settings_str()).expect("should parse test settings"); + settings.publisher.proxy_secret = Redacted::new("unit-test-proxy-secret".to_owned()); + settings.ec.passphrase = Redacted::new("test-secret-key-32-bytes-minimum".to_owned()); + settings.ec.partners = vec![test_partner_with_tokens( + Some("partner-api-token-32-bytes-minimum"), + "replace-with-partner-api-token-32-bytes-minimum", + )]; + + let err = settings + .reject_placeholder_secrets() + .expect_err("should reject placeholder partner tokens"); + let message = format!("{err:?}"); + assert!( + message.contains("ec.partners[partner.example.com].api_token"), + "error should mention the partner API token field" + ); + assert!( + message.contains("ec.partners[partner.example.com].ts_pull_token"), + "error should also mention the partner pull token field on the same partner" + ); + } + #[test] fn is_unusable_store_id_rejects_placeholders_empty_and_padded_values() { for placeholder in RequestSigning::STORE_ID_PLACEHOLDERS { From 4552fcfa621231789bb0830518091995ada52ace Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 28 Sep 2026 17:51:32 -0500 Subject: [PATCH 108/230] Fix Sourcepoint HEAD pass-through test --- .../trusted-server-core/src/integrations/sourcepoint.rs | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/sourcepoint.rs b/crates/trusted-server-core/src/integrations/sourcepoint.rs index 1bdc6aa6f..d22cc4ffe 100644 --- a/crates/trusted-server-core/src/integrations/sourcepoint.rs +++ b/crates/trusted-server-core/src/integrations/sourcepoint.rs @@ -1400,6 +1400,11 @@ mod tests { ], ); let services = build_services_with_http_client(client.clone()); + let expected_body: &[u8] = if method == Method::HEAD { + b"" + } else { + b"unchanged" + }; let mut request = make_req( method, "https://publisher.example.com/integrations/sourcepoint/cdn/asset", @@ -1437,8 +1442,8 @@ mod tests { .await .expect("should collect pass-through body") .as_ref(), - b"unchanged", - "should preserve pass-through bytes" + expected_body, + "should preserve pass-through bytes or omit them for HEAD" ); assert!( client.stub.recorded_request_headers()[0] From 4e64fbfb0fbd7f54fce813350fe2133499546786 Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 30 Sep 2026 16:29:27 -0500 Subject: [PATCH 109/230] Compose streamed Next.js and GTM script rewrites Apply matching script stages in sequence and emit each fragment once. Keep GTM buffering response-local and bounded, and rewrite captured Flight payloads only after reading their original record boundaries. Preserve unsafe originals and recompute decoded record lengths. Cover Pages and App Router data, bootstrap controls, escaped queries, source-order fallback, and complete identity/gzip adapter responses. Refs #1206, #1207, #1208 --- .../tests/routes.rs | 45 + .../trusted-server-adapter-fastly/src/app.rs | 46 + .../trusted-server-core/src/html_processor.rs | 150 +++- .../src/integrations/google_tag_manager.rs | 383 +++++++- .../src/integrations/nextjs/mod.rs | 823 +++++++++++++++++- .../src/integrations/nextjs/rsc.rs | 156 +++- .../integrations/nextjs/rsc_placeholders.rs | 595 +++++++++++-- .../src/integrations/nextjs/rsc_stream.rs | 79 +- .../integrations/nextjs/script_rewriter.rs | 8 +- .../src/integrations/registry.rs | 10 +- .../trusted-server-core/src/test_support.rs | 172 +++- 11 files changed, 2276 insertions(+), 191 deletions(-) diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 7126b2a71..42d1b1b6b 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -7,8 +7,10 @@ use axum::body::Body as AxumBody; use axum::http::Request; use edgezero_adapter_axum::service::EdgeZeroAxumService; +use std::sync::Arc; use tower::{Service as _, ServiceExt as _}; use trusted_server_adapter_axum::app::TrustedServerApp; +use trusted_server_core::test_support::nextjs_auction; const LEGACY_ADMIN_DENY_METHODS: &[&str] = &["GET", "POST", "HEAD", "OPTIONS", "PUT", "PATCH", "DELETE"]; @@ -938,3 +940,46 @@ async fn nextjs_auction_output_holds_until_the_structural_body_close() { "should not leak generated placeholders: {html}" ); } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn nextjs_gtm_html_response_is_complete() { + for gzip in [false, true] { + let origin = Arc::new(nextjs_auction::NextJsAuctionOrigin::with_html_response( + &nextjs_auction::script_composition_html(), + gzip, + false, + )); + let router = TrustedServerApp::routes_with_settings_and_services( + nextjs_auction::script_composition_settings(), + nextjs_auction::services(origin), + ) + .expect("should build mixed integration router"); + let mut service = EdgeZeroAxumService::new(router); + let request = Request::builder() + .method("GET") + .uri("/article") + .header("host", nextjs_auction::PUBLISHER_HOST) + .header("accept", "text/html") + .body(AxumBody::empty()) + .expect("should build navigation"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should serve HTML"); + assert_eq!(response.status().as_u16(), 200); + assert_eq!( + response + .headers() + .get("content-encoding") + .map(|value| value.to_str().expect("should have ASCII coding")), + gzip.then_some("gzip") + ); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("should consume entire adapter body"); + nextjs_auction::assert_script_composition_response(&body, gzip); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index fcb4c2856..0e7eb0836 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -1352,6 +1352,7 @@ mod tests { use std::net::{IpAddr, Ipv4Addr}; use error_stack::Report; + use futures::StreamExt as _; use futures::executor::block_on; use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; use trusted_server_core::ec::device::DeviceSignals; @@ -1368,6 +1369,7 @@ mod tests { TemplateCacheMiss, TemplateCacheReservation, TemplateEntry, TemplateMetadata, }; use trusted_server_core::settings::Settings; + use trusted_server_core::test_support::nextjs_auction; #[test] fn hooks_store_metadata_matches_edgezero_manifest() { @@ -3092,6 +3094,50 @@ mod tests { ); } + #[test] + fn dispatch_fallback_nextjs_gtm_response_is_complete() { + for gzip in [false, true] { + let state = build_state_from_settings(nextjs_auction::script_composition_settings()) + .expect("should build mixed state"); + let origin = Arc::new(nextjs_auction::NextJsAuctionOrigin::with_html_response( + &nextjs_auction::script_composition_html(), + gzip, + true, + )); + let services = nextjs_auction::services(origin); + let request = request_builder() + .method(Method::GET) + .uri(format!( + "https://{}/article", + nextjs_auction::PUBLISHER_HOST + )) + .header("host", nextjs_auction::PUBLISHER_HOST) + .header("accept", "text/html") + .body(Body::empty()) + .expect("should build navigation"); + let response = block_on(super::dispatch_fallback(&state, &services, request)); + assert_eq!(response.status(), StatusCode::OK); + assert!(!response.headers().contains_key(header::CONTENT_LENGTH)); + assert_eq!( + response + .headers() + .get(header::CONTENT_ENCODING) + .map(|value| value.to_str().expect("should have ASCII coding")), + gzip.then_some("gzip") + ); + let Body::Stream(mut stream) = response.into_body() else { + panic!("should retain lazy Fastly body"); + }; + let mut bytes = Vec::new(); + block_on(async { + while let Some(chunk) = stream.next().await { + bytes.extend_from_slice(&chunk.expect("should process every stream item")); + } + }); + nextjs_auction::assert_script_composition_response(&bytes, gzip); + } + } + #[test] fn dispatch_fallback_streams_publisher_body_without_buffering() { // Regression guard for the publisher streaming cutover (#849): a diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 3889d7d59..79a2bfaa6 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -1,7 +1,8 @@ //! Simplified HTML processor that combines URL replacement and integration injection //! //! This module provides a `StreamProcessor` implementation for HTML content. -use std::cell::Cell; +use std::borrow::Cow; +use std::cell::{Cell, RefCell}; use std::io; use std::rc::Rc; use std::sync::Arc; @@ -661,38 +662,62 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso })); } - for script_rewriter in script_rewriters { - let selector = script_rewriter.selector(); - let rewriter = script_rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - element_content_handlers.push(text!(selector, { - let rewriter = rewriter.clone(); - let patterns = patterns.clone(); - let document_state = document_state.clone(); - move |text| { - let ctx = IntegrationScriptContext { - selector, - request_host: &patterns.request_host, - request_scheme: &patterns.request_scheme, - origin_host: &patterns.origin_host, - is_last_in_text_node: text.last_in_text_node(), - max_buffered_script_bytes: config.max_buffered_body_bytes, - document_state: &document_state, - }; - match rewriter.rewrite(text.as_str(), &ctx) { - ScriptRewriteAction::Keep => {} - ScriptRewriteAction::Replace(rewritten) => { - text.replace(&rewritten, ContentType::Text); - } - ScriptRewriteAction::RemoveNode => { - text.remove(); - } - } - Ok(()) - } + // Selector callbacks retain lol_html's CSS matching; only the composed text + // callback mutates a fragment. Start each script with fresh match state. + let script_matches = Rc::new(RefCell::new(vec![false; script_rewriters.len()])); + let reset_matches = script_matches.clone(); + element_content_handlers.push(element!("script", move |_el| { + reset_matches.borrow_mut().fill(false); + Ok(()) + })); + for (index, rewriter) in script_rewriters.iter().enumerate() { + let matches = script_matches.clone(); + element_content_handlers.push(element!(rewriter.selector(), move |_el| { + matches.borrow_mut()[index] = true; + Ok(()) })); } + let script_patterns = patterns.clone(); + let script_document_state = document_state.clone(); + element_content_handlers.push(text!("script", move |text| { + let mut working = Cow::Borrowed(text.as_str()); + let mut changed = false; + for (rewriter, matched) in script_rewriters.iter().zip(script_matches.borrow().iter()) { + if !matched { + continue; + } + let ctx = IntegrationScriptContext { + selector: rewriter.selector(), + request_host: &script_patterns.request_host, + request_scheme: &script_patterns.request_scheme, + origin_host: &script_patterns.origin_host, + is_last_in_text_node: text.last_in_text_node(), + max_buffered_script_bytes: config.max_buffered_body_bytes, + document_state: &script_document_state, + }; + match rewriter.rewrite(&working, &ctx) { + ScriptRewriteAction::Keep => {} + ScriptRewriteAction::Replace(rewritten) => { + working = Cow::Owned(rewritten); + changed = true; + } + ScriptRewriteAction::RemoveNode => { + working = Cow::Borrowed(""); + changed = true; + } + } + } + if changed { + if working.is_empty() { + text.remove(); + } else if let Cow::Owned(rewritten) = working { + // Script is raw text: HTML entities would change JS/JSON data. + // Emit composed source verbatim rather than escaping it again. + text.replace(&rewritten, ContentType::Html); + } + } + Ok(()) + })); let rewriter_settings = RewriterSettings { document_content_handlers, @@ -723,7 +748,7 @@ mod tests { use super::*; use crate::integrations::{ AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, - IntegrationHeadInjector, IntegrationHtmlContext, + IntegrationHeadInjector, IntegrationHtmlContext, IntegrationScriptRewriter, }; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use crate::test_support::tests::create_test_settings; @@ -752,6 +777,67 @@ mod tests { } } + struct SuppressingScript; + impl IntegrationScriptRewriter for SuppressingScript { + fn integration_id(&self) -> &'static str { + "suppress" + } + fn selector(&self) -> &'static str { + "script[data-suppress='yes']" + } + fn rewrite(&self, _text: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + assert_eq!(ctx.selector, self.selector()); + ScriptRewriteAction::RemoveNode + } + } + + struct FlushingScript; + impl IntegrationScriptRewriter for FlushingScript { + fn integration_id(&self) -> &'static str { + "flush" + } + fn selector(&self) -> &'static str { + "script#flush" + } + fn rewrite(&self, text: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + assert_eq!(ctx.selector, self.selector()); + assert!(text.is_empty(), "should see upstream suppression"); + if ctx.is_last_in_text_node { + ScriptRewriteAction::Replace("flushed-once".into()) + } else { + ScriptRewriteAction::Keep + } + } + } + + #[test] + fn script_actions_compose_with_selector_reset_and_empty_final_flush() { + let mut config = create_test_config(); + config.integrations = IntegrationRegistry::from_rewriters( + Vec::new(), + vec![Arc::new(SuppressingScript), Arc::new(FlushingScript)], + ); + let mut processor = create_html_processor(config); + let mut output = processor + .process_chunk(b"", true) + .expect("should finalize all matching stages"), + ); + let output = String::from_utf8(output).expect("should emit UTF-8"); + assert!( + output.contains(""), + "should flush downstream even on suppressed final: {output}" + ); + assert!( + output.contains(""), + "should reset CSS match state between scripts" + ); + assert_eq!(output.matches("flushed-once").count(), 1); + } + #[test] fn integration_attribute_rewriter_can_remove_elements() { struct RemovingLinkRewriter; diff --git a/crates/trusted-server-core/src/integrations/google_tag_manager.rs b/crates/trusted-server-core/src/integrations/google_tag_manager.rs index ae9ceb7cb..6661df1a6 100644 --- a/crates/trusted-server-core/src/integrations/google_tag_manager.rs +++ b/crates/trusted-server-core/src/integrations/google_tag_manager.rs @@ -143,6 +143,91 @@ static GTM_QUOTED_URL_PATTERNS: LazyLock> = LazyLock::new(|| { .collect() }); +// Flight spans are JS string source, not decoded JSON. Match only whole or +// quoted routed URL tokens and preserve their delimiters, suffix and slash layer. +static GTM_RSC_WHOLE_URL_PATTERN: LazyLock = LazyLock::new(|| { + Regex::new( + r"^(?:https?:)?(?P\\{0,8}/)(?:\\{0,8}/)(?:www\.(?:googletagmanager|google-analytics)\.com|analytics\.google\.com)(?P(?:\\{0,8}/)(?:gtm\.js|gtag(?:\\{0,8}/|\.)js|g(?:\\{0,8}/)collect|collect)|)(?P[?#].*)?$" + ).expect("should compile whole Flight URL pattern") +}); +static GTM_RSC_QUOTED_URL_START: LazyLock = LazyLock::new(|| { + Regex::new(r#"(?P\\{0,8}["'`])(?:https?:)?\\{0,8}/\\{0,8}/(?:www\.(?:googletagmanager|google-analytics)\.com|analytics\.google\.com)"#) + .expect("should compile quoted Flight URL start") +}); + +// Match the opening quote's escape layer. Higher-layer instances of that +// delimiter belong to the URL (e.g. escaped quotes inside a JSON query). +fn flight_url_token_end( + content: &str, + start: usize, + delimiter: u8, + layer: usize, +) -> Option<(usize, usize)> { + let bytes = content.as_bytes(); + let mut slash_run = 0; + for index in start..bytes.len() { + let byte = bytes[index]; + if byte == delimiter { + if slash_run == layer { + return Some((index - slash_run, index + 1)); + } + if slash_run < layer { + return None; + } + } else if delimiter == b'`' && byte == b'$' { + return (bytes.get(index + 1) == Some(&b'{')).then_some((index, index + 2)); + } + slash_run = if byte == b'\\' { slash_run + 1 } else { 0 }; + } + None +} + +/// Rewrite supported GTM/GA tokens in a captured Flight source span. +/// +/// Call only after original T-record boundaries are known, before lengths are +/// recomputed. Originals used for fallback must never pass through this function. +pub(crate) fn rewrite_gtm_rsc_span(content: &str) -> String { + fn route(token: &str) -> Option { + let captures = GTM_RSC_WHOLE_URL_PATTERN.captures(token)?; + let slash = &captures["slash"]; + let path = &captures["path"]; + let suffix = captures.name("suffix").map_or("", |value| value.as_str()); + Some(format!( + "{slash}integrations{slash}google_tag_manager{path}{suffix}" + )) + } + if let Some(rewritten) = route(content) { + return rewritten; + } + let mut rewritten = String::with_capacity(content.len()); + let mut cursor = 0; + let mut search = 0; + while let Some(captures) = GTM_RSC_QUOTED_URL_START.captures(&content[search..]) { + let open = captures.name("open").expect("should capture quote layer"); + let open_start = search + open.start(); + let url_start = search + open.end(); + let delimiter = content.as_bytes()[url_start - 1]; + let layer = open.len() - 1; + search = url_start; + // Never accept a suffix of an unsupported, deeper opening layer. + if open_start > 0 && content.as_bytes()[open_start - 1] == b'\\' { + continue; + } + let Some((url_end, token_end)) = flight_url_token_end(content, url_start, delimiter, layer) + else { + continue; + }; + search = token_end; + if let Some(url) = route(&content[url_start..url_end]) { + rewritten.push_str(&content[cursor..url_start]); + rewritten.push_str(&url); + cursor = url_end; + } + } + rewritten.push_str(&content[cursor..]); + rewritten +} + #[derive(Debug, Clone, Deserialize, Serialize, Validate)] pub struct GoogleTagManagerConfig { #[serde(default = "default_enabled")] @@ -356,14 +441,12 @@ pub struct GoogleTagManagerIntegration { /// allowlist as "any host", so without this a 3xx from the upstream would /// let an arbitrary origin's body be re-served as first-party JavaScript. proxy_allowed_domains: Vec, - /// Accumulates text fragments when `lol_html` splits a text node across - /// chunk boundaries. Drained on `is_last_in_text_node`. - /// - /// Uses `Mutex` to satisfy the `Sync` bound on `IntegrationScriptRewriter`. - /// The pipeline is single-threaded (`lol_html::HtmlRewriter` is `!Send`), - /// so the lock is uncontended. `lol_html` delivers text chunks sequentially - /// per element — the buffer is always empty when a new element's text begins. - accumulated_text: Mutex, +} + +#[derive(Default)] +struct GtmScriptState { + held: String, + bypass_until_last: bool, } impl GoogleTagManagerIntegration { @@ -372,7 +455,6 @@ impl GoogleTagManagerIntegration { Arc::new(Self { config, proxy_allowed_domains, - accumulated_text: Mutex::new(String::new()), }) } @@ -1029,39 +1111,57 @@ impl IntegrationScriptRewriter for GoogleTagManagerIntegration { } fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { - let mut buf = self - .accumulated_text + let state = ctx + .document_state + .get_or_insert_with(GTM_INTEGRATION_ID, || Mutex::new(GtmScriptState::default())); + let mut state = state .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); - - // Cheap gate: only engage the accumulation path for scripts whose - // running text could plausibly contain a GTM/GA domain. Unrelated - // scripts (the vast majority) return Keep on every fragment so they - // stay visible in `lol_html`'s output unchanged — and, critically, - // so a Replace from this rewriter does not clobber a Replace from - // another rewriter on the same element (e.g., `NextJsNextDataRewriter` - // on `script#__NEXT_DATA__`). See `might_contain_gtm_prefix` for the - // boundary-safe substring check. - let prior_and_current_might_match = - might_contain_gtm_prefix(&buf) || might_contain_gtm_prefix(content); - - if !ctx.is_last_in_text_node { - // Intermediate fragment. Only accumulate + suppress when the script - // might still resolve to a GTM match. Otherwise return Keep so the - // fragment is emitted as-is by `lol_html` and untouched by us. - if prior_and_current_might_match { - buf.push_str(content); - return ScriptRewriteAction::RemoveNode; + if crate::integrations::nextjs::protects_current_flight_fragment(ctx.document_state) { + if ctx.is_last_in_text_node { + state.bypass_until_last = false; + } + if state.held.is_empty() { + return ScriptRewriteAction::Keep; } - return ScriptRewriteAction::keep(); + let mut restored = std::mem::take(&mut state.held); + restored.push_str(content); + return ScriptRewriteAction::Replace(restored); } - - // Last fragment. If we accumulated prior fragments, combine them. - let full_content: Option = if buf.is_empty() { + if state.bypass_until_last { + if ctx.is_last_in_text_node { + state.bypass_until_last = false; + } + return ScriptRewriteAction::Keep; + } + // Once holding begins, retain every following fragment in source order, + // even if the tentative hostname prefix turns out not to match. + if state.held.is_empty() && !might_contain_gtm_prefix(content) { + return ScriptRewriteAction::Keep; + } + if state + .held + .len() + .checked_add(content.len()) + .is_none_or(|length| length > ctx.max_buffered_script_bytes) + { + state.bypass_until_last = !ctx.is_last_in_text_node; + if state.held.is_empty() { + return ScriptRewriteAction::Keep; + } + let mut restored = std::mem::take(&mut state.held); + restored.push_str(content); + return ScriptRewriteAction::Replace(restored); + } + if !ctx.is_last_in_text_node { + state.held.push_str(content); + return ScriptRewriteAction::RemoveNode; + } + let full_content = if state.held.is_empty() { None } else { - buf.push_str(content); - Some(std::mem::take(&mut *buf)) + state.held.push_str(content); + Some(std::mem::take(&mut state.held)) }; let text = full_content.as_deref().unwrap_or(content); @@ -1098,6 +1198,7 @@ mod tests { }; use crate::platform::test_support::{StubHttpClient, build_services_with_http_client}; use crate::settings::Settings; + use crate::streaming_processor::StreamProcessor as _; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; #[test] @@ -3414,6 +3515,216 @@ container_id = "GTM-DEFAULT" } } + #[test] + fn captured_flight_preserves_quoted_query_delimiters_and_template_interpolation() { + for (source, expected) in [ + ( + r#"\"http://www.googletagmanager.com/gtm.js?q='text'\""#, + r#"\"/integrations/google_tag_manager/gtm.js?q='text'\""#, + ), + ( + r#"\"http://www.googletagmanager.com/gtm.js?q=`text`\""#, + r#"\"/integrations/google_tag_manager/gtm.js?q=`text`\""#, + ), + ( + r#"'http://www.googletagmanager.com/gtm.js?q=\"text\"'"#, + r#"'/integrations/google_tag_manager/gtm.js?q=\"text\"'"#, + ), + ( + r#"\"http://www.googletagmanager.com/gtm.js?q=\\\"text\\\"\""#, + r#"\"/integrations/google_tag_manager/gtm.js?q=\\\"text\\\"\""#, + ), + ( + r"'http://www.googletagmanager.com/gtm.js?q=\'text\''", + r"'/integrations/google_tag_manager/gtm.js?q=\'text\''", + ), + ( + "`http://www.googletagmanager.com/gtm.js?q=$literal`", + "`http://www.googletagmanager.com/gtm.js?q=$literal`", + ), + ( + "`http://www.googletagmanager.com/gtm.js${id}`", + "`/integrations/google_tag_manager/gtm.js${id}`", + ), + ] { + assert_eq!(rewrite_gtm_rsc_span(source), expected); + } + } + + #[test] + fn captured_flight_token_rewrite_preserves_escape_layers_and_allowed_paths() { + for slash in ["/", r"\/", r"\\/"] { + for quote in ["", "'", "\"", r#"\""#, "`"] { + let source = format!( + "{quote}http:{slash}{slash}www.googletagmanager.com{slash}gtm.js{quote}" + ); + let expected = format!( + "{quote}{slash}integrations{slash}google_tag_manager{slash}gtm.js{quote}" + ); + assert_eq!( + rewrite_gtm_rsc_span(&source), + expected, + "should preserve source escape layer: {source}" + ); + } + } + for source in [ + r#""https://www.googletagmanager.com/ns.html?id=GTM-MIX1""#, + "text http://www.googletagmanager.com/gtm.js text", + "'http://www.googletagmanager.com/gtm.js/path'", + "'http://www.googletagmanager.com/collect;matrix'", + "'http://www.googletagmanager.com.evil/gtm.js'", + "'http://www.googletagmanager.com/gtm.js\"", + ] { + assert_eq!( + rewrite_gtm_rsc_span(source), + source, + "should not route unsupported or unbounded token" + ); + } + } + + #[test] + fn inline_gtm_split_after_underscore_emits_once() { + let mut settings = make_settings(); + settings + .integrations + .insert_config( + "google_tag_manager", + &serde_json::json!({"enabled": true, "container_id": "GTM-MIX1"}), + ) + .expect("should enable GTM"); + settings + .integrations + .insert_config("nextjs", &serde_json::json!({"enabled": true})) + .expect("should enable Next.js"); + let registry = IntegrationRegistry::with_plan( + &settings, + Arc::new(crate::auction::compile_auction_plan(&settings).expect("should compile plan")), + ) + .expect("should build registry"); + let mut processor = create_html_processor(config_from_settings(&settings, ®istry)); + let mut bytes = processor + .process_chunk( + b"", true) + .expect("should consume final"), + ); + let html = String::from_utf8(bytes).expect("should emit UTF-8"); + assert!(html.contains("var url='/integrations/google_tag_manager/gtm.js';var tail_=1;")); + assert_eq!( + html.matches("var url=").count(), + 1, + "should emit script text once" + ); + } + + #[test] + fn sticky_gtm_four_fragments_preserve_order_after_false_google_prefix() { + let integration = GoogleTagManagerIntegration::new(tag_config("GTM-MIX1", &[])); + let state = IntegrationDocumentState::default(); + let fragments = [ + "var falsePrefix='google", + "X';first=1;", + "second=2;var tag='http://www.googletagmanager.com/gtm.js", + "';last=3;", + ]; + let mut emitted = String::new(); + for (index, fragment) in fragments.iter().enumerate() { + let context = IntegrationScriptContext { + selector: "script", + request_host: "test.example.com", + request_scheme: "https", + origin_host: "origin.example.com", + is_last_in_text_node: index == 3, + max_buffered_script_bytes: 1000, + document_state: &state, + }; + match IntegrationScriptRewriter::rewrite(&*integration, fragment, &context) { + ScriptRewriteAction::Keep => emitted.push_str(fragment), + ScriptRewriteAction::Replace(value) => emitted.push_str(&value), + ScriptRewriteAction::RemoveNode => {} + } + } + assert_eq!( + emitted, + "var falsePrefix='googleX';first=1;second=2;var tag='/integrations/google_tag_manager/gtm.js';last=3;" + ); + } + + #[test] + fn sticky_gtm_holding_is_bounded_and_document_local() { + let integration = GoogleTagManagerIntegration::new(tag_config("GTM-MIX1", &[])); + let a = IntegrationDocumentState::default(); + let b = IntegrationDocumentState::default(); + let context = |last, budget, state| IntegrationScriptContext { + selector: "script", + request_host: "test.example.com", + request_scheme: "https", + origin_host: "origin.example.com", + is_last_in_text_node: last, + max_buffered_script_bytes: budget, + document_state: state, + }; + let fragments = ["prefix google", "X-middle", "later", "-tail"]; + let mut emitted = String::new(); + for (i, fragment) in fragments.iter().enumerate() { + let action = IntegrationScriptRewriter::rewrite( + &*integration, + fragment, + &context(i == 3, 100, &a), + ); + match action { + ScriptRewriteAction::Keep => emitted.push_str(fragment), + ScriptRewriteAction::Replace(value) => emitted.push_str(&value), + ScriptRewriteAction::RemoveNode => {} + } + if i == 0 { + assert_eq!( + IntegrationScriptRewriter::rewrite( + &*integration, + "other", + &context(true, 100, &b) + ), + ScriptRewriteAction::Keep, + "should isolate documents" + ); + } + } + assert_eq!( + emitted, + fragments.concat(), + "should preserve source order after false prefix" + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*integration, "google", &context(false, 6, &a)), + ScriptRewriteAction::RemoveNode + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*integration, "X", &context(false, 6, &a)), + ScriptRewriteAction::Replace("googleX".into()), + "should restore on overflow" + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*integration, "tail", &context(true, 6, &a)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*integration, "google", &context(false, 6, &a)), + ScriptRewriteAction::RemoveNode + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*integration, "", &context(true, 6, &a)), + ScriptRewriteAction::Replace("google".into()), + "should flush empty final at exact limit" + ); + } + #[test] fn fragmented_gtm_snippet_is_accumulated_and_rewritten() { let config = tag_config("GTM-FRAG1", &[]); diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index f6f36f67c..4036e5e76 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -63,6 +63,21 @@ pub(super) fn configuration_error(message: impl Into) -> Report bool { + state + .get::>(NEXTJS_INTEGRATION_ID) + .is_some_and(|state| { + state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .current_fragment_protected + }) +} + /// Register the Next.js integration when enabled. /// /// # Errors @@ -91,7 +106,18 @@ pub fn register( // then substitute them through the bounded output stream processor. let placeholders = Arc::new(NextJsRscPlaceholderRewriter::new(config.clone())); - let stream_processor = Arc::new(NextJsRscStreamProcessorFactory::new(config.clone())); + let gtm_transform = settings + .integration_config::( + "google_tag_manager", + )? + .filter(|config| config.enabled) + .map(|_| { + crate::integrations::google_tag_manager::rewrite_gtm_rsc_span as fn(&str) -> String + }); + let stream_processor = Arc::new(NextJsRscStreamProcessorFactory::new( + config.clone(), + gtm_transform, + )); let builder = IntegrationRegistration::builder(NEXTJS_INTEGRATION_ID) .with_script_rewriter(structured) @@ -115,10 +141,13 @@ mod tests { use super::*; use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; use crate::integrations::IntegrationRegistry; - use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; + use crate::streaming_processor::{ + Compression, PipelineConfig, StreamProcessor as _, StreamingPipeline, + }; use crate::test_support::tests::create_test_settings; + use flate2::{read::GzDecoder, write::GzEncoder}; use serde_json::json; - use std::io::Cursor; + use std::io::{Cursor, Read as _, Write as _}; fn config_from_settings( settings: &Settings, @@ -133,6 +162,794 @@ mod tests { ) } + fn mixed_output(html: &str, chunk_size: usize) -> String { + mixed_output_with_budget(html, chunk_size, 10 * 1024 * 1024, 16 * 1024 * 1024) + } + + fn mixed_config(group_limit: usize, script_limit: usize) -> HtmlProcessorConfig { + let mut settings = create_test_settings(); + for (id, config) in [ + ( + "nextjs", + json!({"enabled": true, "max_combined_payload_bytes": group_limit}), + ), + ( + "google_tag_manager", + json!({"enabled": true, "container_id": "GTM-MIX1"}), + ), + ] { + settings + .integrations + .insert_config(id, &config) + .expect("should configure integration"); + } + let registry = IntegrationRegistry::with_plan( + &settings, + Arc::new(crate::auction::compile_auction_plan(&settings).expect("should compile plan")), + ) + .expect("should create registry"); + let mut config = config_from_settings(&settings, ®istry); + config.max_buffered_body_bytes = script_limit; + config + } + + fn mixed_output_with_budget( + html: &str, + chunk_size: usize, + group_limit: usize, + script_limit: usize, + ) -> String { + let config = mixed_config(group_limit, script_limit); + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size, + }, + create_html_processor(config), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .expect("should consume complete mixed response"); + String::from_utf8(output).expect("should emit UTF-8") + } + + #[test] + fn mixed_next_data_flight_mentions_do_not_claim_json_or_hide_gtm() { + for text in [ + "self.__next_f", + "window.__next_f", + "self.__next_f.push([1,content])", + ] { + for chunk in [32, 64, 1000, 8192] { + let data = json!({"props": {"pageProps": { + "text": text, "href": "https://origin.example.com/page", + "tag": "http://www.googletagmanager.com/gtm.js", "padding": "_".repeat(10000), + }}}); + let output = mixed_output( + &format!(r#""#), + chunk, + ); + let script = output + .split(r#"") + .next() + .expect("should close Pages script"); + let mut expected = data; + expected["props"]["pageProps"]["href"] = json!("https://test.example.com/page"); + expected["props"]["pageProps"]["tag"] = + json!("/integrations/google_tag_manager/gtm.js"); + assert_eq!( + serde_json::from_str::(script) + .expect("should preserve Pages JSON"), + expected, + "should not claim quoted Flight mention: {text}, chunk={chunk}" + ); + } + } + } + + #[test] + fn protected_flight_then_next_data_then_ordinary_gtm_resets_ownership() { + let payload = "1:Tffff,'http://www.googletagmanager.com/gtm.js'"; + let pages = json!({"text": "self.__next_f.push([1,content])", "href": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js"}); + // Keep the ordinary URL's opening quote and >=6 marker bytes together; + // standalone GTM deliberately does not hold shorter tentative prefixes. + let alignment = " ".repeat(23); + let html = format!( + r#"{alignment}"#, + serde_json::to_string(payload).expect("should encode raw Flight") + ); + for chunk in [32, 8192] { + let output = mixed_output_with_budget(&html, chunk, 24, 10000); + assert_eq!( + flight_payloads(&output), + [payload], + "should preserve raw Flight" + ); + let source = output + .split(r#"") + .next() + .expect("should close Pages script"); + let mut expected = pages.clone(); + expected["href"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js"); + assert_eq!( + serde_json::from_str::(source).expect("should retain JSON"), + expected + ); + assert!( + output.ends_with( + "" + ), + "should rewrite later ordinary GTM at chunk={chunk}: {output}" + ); + } + } + + #[test] + fn push_head_after_tentative_reference_overflow_protects_raw_continuation() { + let payload = "1:Tffff,'http://www.googletagmanager.com/gtm.js'"; + let html = format!( + r#""#, + " ".repeat(512), + serde_json::to_string(payload).expect("should encode raw Flight") + ); + for chunk in [32, 8192] { + assert_eq!( + mixed_output_with_budget(&html, chunk, 100, 10000), + html, + "should protect a call confirmed after tentative probe overflow and preserve its later raw continuation" + ); + } + } + + #[test] + fn ordinary_next_f_reference_does_not_bypass_later_flight() { + let body = "http://www.googletagmanager.com/gtm.js"; + let payload = format!("1:T{:x},{body}", body.len()); + for padding in [0, 400] { + for chunk in [32, 8192] { + let html = format!( + "{}", + "_".repeat(padding), + flight_html(&[&payload]) + ); + let output = mixed_output_with_budget(&html, chunk, 100, 10000); + assert_eq!( + flight_payloads(&output), + [format!( + "1:T{:x},/integrations/google_tag_manager/gtm.js", + "/integrations/google_tag_manager/gtm.js".len() + )], + "should not bypass captured Flight because of a prior ordinary reference: padding={padding}, chunk={chunk}" + ); + } + } + } + + #[test] + fn ordinary_next_f_reference_does_not_hide_gtm() { + let script = "var ref=self.__next_f;var tag='http://www.googletagmanager.com/gtm.js';"; + for chunk in [32, 64, 1000, 8192] { + let output = mixed_output(&format!(""), chunk); + assert_eq!( + output, + "", + "should not protect a non-push reference, chunk={chunk}" + ); + } + } + + #[test] + fn mixed_next_data_preserves_json_and_both_rewrites() { + for chunk in [32, 64, 1000, 8192] { + for marker in ["_", "__", "__n", "__ne", "__nex", "__next", "__next_"] { + for (url, routed) in [ + ( + "https://www.googletagmanager.com/gtm.js?id=GTM-MIX1", + "/integrations/google_tag_manager/gtm.js?id=GTM-MIX1", + ), + ( + "http://www.google-analytics.com/g/collect?v=2", + "/integrations/google_tag_manager/g/collect?v=2", + ), + ] { + for late in [false, true] { + let padding = json!(format!("{}{marker}", "_".repeat(10000))); + let source = if late { + format!( + r#"{{"padding":{padding},"href":"https://origin.example.com/page","tag":"{url}"}}"# + ) + } else { + format!( + r#"{{"tag":"{url}","padding":{padding},"href":"https://origin.example.com/page"}}"# + ) + }; + let html = format!( + r#"

suffix

"# + ); + let output = mixed_output(&html, chunk); + let script = output + .split(r#"") + .next() + .expect("should close script"); + let expected = json!({"padding": format!("{}{marker}", "_".repeat(10000)), "href": "https://test.example.com/page", "tag": routed}); + assert_eq!( + serde_json::from_str::(script) + .expect("should preserve valid JSON"), + expected, + "chunk={chunk}, marker={marker}, late={late}" + ); + assert!(output.ends_with(""), "should finish response"); + } + } + } + } + } + + fn flight_payloads(output: &str) -> Vec { + let mut payloads = Vec::new(); + for part in output.split("").next().expect("should close script"); + while let Some((start, end)) = shared::find_rsc_push_payload_range(script) { + payloads.push( + serde_json::from_str::(&script[start - 1..=end]) + .expect("should retain valid push string JSON"), + ); + script = &script[end + 1..]; + } + } + payloads + } + + fn assert_mixed_t_model(payload: &str, expected: &serde_json::Value) { + let (header, body) = payload.split_once(',').expect("should retain T record"); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should retain hex length"), + body.len(), + "should count decoded transformed T bytes" + ); + assert_eq!( + &serde_json::from_str::(body).expect("should retain model JSON"), + expected + ); + } + + #[test] + fn normal_flight_bootstrap_controls_allow_later_payload_rewriting() { + let controls = "(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])"; + let data = json!({"href": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js", "ga": "//www.google-analytics.com/g/collect?v=2", "text": "é😀"}); + let body = data.to_string(); + let push = format!( + "self.__next_f.push([1,{}])", + serde_json::to_string(&format!("1:T{:x},{body}", body.len())) + .expect("should encode Flight") + ); + let mut expected = data; + expected["href"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js"); + expected["ga"] = json!("/integrations/google_tag_manager/g/collect?v=2"); + for together in [false, true] { + let scripts = if together { + format!("") + } else { + format!("") + }; + let html = format!("{scripts}

suffix

"); + for compression in [Compression::None, Compression::Gzip] { + let input = if compression == Compression::Gzip { + let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(html.as_bytes()) + .expect("should encode gzip bootstrap document"); + encoder.finish().expect("should finish gzip input") + } else { + html.as_bytes().to_vec() + }; + for chunk_size in [32, 64, 1000, 8192] { + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: compression, + output_compression: compression, + chunk_size, + }, + create_html_processor(mixed_config(100000, 100000)), + ); + let mut bytes = Vec::new(); + pipeline + .process(Cursor::new(&input), &mut bytes) + .expect("should consume bootstrap document"); + let mut decoded = Vec::new(); + if compression == Compression::Gzip { + GzDecoder::new(bytes.as_slice()) + .read_to_end(&mut decoded) + .expect("should decode gzip bootstrap response"); + } else { + decoded = bytes; + } + let output = String::from_utf8(decoded).expect("should emit UTF-8"); + assert!( + output.contains(controls), + "should preserve inert bootstrap controls" + ); + let payloads = flight_payloads(&output); + assert_eq!(payloads.len(), 1, "should emit payload exactly once"); + assert_mixed_t_model(&payloads[0], &expected); + assert!(!output.contains("__ts_rsc_")); + assert!(output.ends_with("

suffix

")); + } + } + } + } + + #[test] + fn same_delimiter_escaped_flight_queries_preserve_data_and_t_lengths() { + let data = json!({"tag": "http://www.googletagmanager.com/gtm.js?q=\"x\"&v=é", "href": "https://origin.example.com/page", "url": "http://origin.example.com/other"}); + let body = data.to_string(); + let payload = format!("1:T{:x},{body}", body.len()); + let mut expected = data; + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js?q=\"x\"&v=é"); + expected["href"] = json!("https://test.example.com/page"); + expected["url"] = json!("https://test.example.com/other"); + for chunk in [32, 64, 1000, 8192] { + let output = mixed_output(&flight_html(&[&payload]), chunk); + let payloads = flight_payloads(&output); + assert_mixed_t_model(&payloads[0], &expected); + let body = r"'http://www.googletagmanager.com/gtm.js?q=\'x\''"; + let payload = format!("1:T{:x},{body}", body.len()); + let output = mixed_output(&flight_html(&[&payload]), chunk); + let payloads = flight_payloads(&output); + let (header, transformed) = payloads[0].split_once(',').expect("should retain T"); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should parse hex"), + transformed.len() + ); + assert_eq!( + transformed, + r"'/integrations/google_tag_manager/gtm.js?q=\'x\''" + ); + } + } + + #[test] + fn sticky_false_google_prefix_preserves_raw_script_metacharacters() { + let fragments = [ + "", + ]; + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let mut bytes = Vec::new(); + for (index, fragment) in fragments.iter().enumerate() { + bytes.extend( + processor + .process_chunk(fragment.as_bytes(), index == fragments.len() - 1) + .expect("should consume raw script fragment"), + ); + } + assert_eq!( + String::from_utf8(bytes).expect("should emit UTF-8"), + fragments.concat(), + "holding/restoration must preserve raw JS source, not HTML-escape it" + ); + } + + #[test] + fn mixed_flight_first_party_urls_preserve_t_lengths() { + for scheme in ["https:", "http:", ""] { + for host_path in [ + "www.googletagmanager.com/gtm.js", + "www.googletagmanager.com/gtag/js?id=G-MIX1", + "www.google-analytics.com/g/collect", + "analytics.google.com/collect?v=2", + ] { + for quoted in [false, true] { + let url = format!("{scheme}//{host_path}"); + let body = if quoted { format!("\"{url}\"") } else { url }; + let payload = format!("1:T{:x},{body}", body.len()); + let encoded = serde_json::to_string(&payload).expect("should encode payload"); + let html = format!( + "

suffix

" + ); + for chunk in [32, 64, 1000, 8192] { + let output = mixed_output(&html, chunk); + let payloads = flight_payloads(&output); + assert_eq!(payloads.len(), 1); + let (header, body) = + payloads[0].split_once(',').expect("should retain T record"); + assert!( + body.contains("/integrations/google_tag_manager/"), + "should rewrite captured GTM-only flight: {body}" + ); + assert_eq!( + usize::from_str_radix(&header[3..], 16) + .expect("should keep hex length"), + body.len(), + "should count transformed decoded bytes" + ); + } + } + } + } + let payload = "0:{\"url\":\"https://www.googletagmanager.com/gtm.js\",\"href\":\"https://origin.example.com/page\"}\n"; + let encoded = serde_json::to_string(&payload).expect("should encode payload"); + let output = mixed_output( + &format!(""), + 32, + ); + let payloads = flight_payloads(&output); + assert!(payloads[0].contains("test.example.com/page")); + assert!( + payloads[0].contains("/integrations/google_tag_manager/gtm.js"), + "should rewrite queryless escaped quote URL" + ); + } + + #[test] + fn mixed_flight_batch_rewrites_every_payload() { + let bodies = [ + "http://www.googletagmanager.com/gtm.js", + "//www.google-analytics.com/collect?v=2", + ]; + let pushes: Vec = bodies + .iter() + .enumerate() + .map(|(index, body)| { + let payload = format!("{index}:T{:x},{body}", body.len()); + format!( + "self.__next_f.push([1,{}])", + serde_json::to_string(&payload).expect("should encode payload") + ) + }) + .collect(); + let output = mixed_output(&format!("", pushes.join(";")), 8192); + let script = output + .split("") + .next() + .expect("should close script"); + for push in script.split(';') { + let json = push + .strip_prefix("self.__next_f.push(") + .expect("should retain push") + .strip_suffix(')') + .expect("should close push"); + let value: serde_json::Value = + serde_json::from_str(json).expect("should preserve push JSON"); + let (header, body) = value[1] + .as_str() + .expect("should retain payload") + .split_once(',') + .expect("should retain T header"); + assert!( + body.contains("/integrations/google_tag_manager/"), + "should rewrite every captured push" + ); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should parse hex"), + body.len(), + "should recount every payload" + ); + } + } + + #[test] + fn mixed_next_data_prefix_tails_and_false_google_preserve_payload() { + for chunk in [32, 64, 1000, 8192] { + for prefix in [ + "_", "__", "__n", "__ne", "__nex", "__next", "__next_", "google", + ] { + for alignment in [0, 1, 13, 1023, 1024] { + let data = json!({"padding": format!("{}{prefix}", "_".repeat(12000)), "href": "https://origin.example.com/page"}); + let start = format!( + "{}"), chunk); + let script = output + .split("") + .next() + .expect("should close script"); + data["href"] = json!("https://test.example.com/page"); + assert_eq!( + serde_json::from_str::(script) + .expect("should keep valid JSON"), + data, + "prefix={prefix}, chunk={chunk}, alignment={alignment}" + ); + } + } + } + } + + #[test] + fn mixed_next_data_identity_and_gzip_chunk_sweep() { + let data = json!({"href": "https://origin.example.com/page", "tag": "https://www.google-analytics.com/collect?v=2", "padding": "_".repeat(12000)}); + let html = format!( + "

suffix

" + ); + for compression in [Compression::None, Compression::Gzip] { + let input = if compression == Compression::Gzip { + let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(html.as_bytes()) + .expect("should encode actual gzip input"); + encoder.finish().expect("should finish gzip input") + } else { + html.as_bytes().to_vec() + }; + for chunk_size in [32, 64, 1000, 8192] { + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: compression, + output_compression: compression, + chunk_size, + }, + create_html_processor(mixed_config(100000, 100000)), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(&input), &mut output) + .expect("should process complete compressed document"); + let mut decoded = Vec::new(); + if compression == Compression::Gzip { + GzDecoder::new(output.as_slice()) + .read_to_end(&mut decoded) + .expect("should decode actual gzip output"); + } else { + decoded = output; + } + let output = String::from_utf8(decoded).expect("should emit UTF-8"); + let script = output + .split("") + .next() + .expect("should close script"); + let mut expected = data.clone(); + expected["href"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/collect?v=2"); + assert_eq!( + serde_json::from_str::(script) + .expect("should retain valid JSON"), + expected + ); + assert!(output.ends_with("")); + } + } + } + + #[test] + fn mixed_registry_isolates_interleaved_html_documents() { + let config = mixed_config(10000, 10000); + let mut a = create_html_processor(config.clone()); + let mut b = create_html_processor(config); + let mut output_a = a + .process_chunk(b"", true) + .expect("should finish document A"), + ); + output_b.extend( + b.process_chunk(b"-analytics.com/collect?v=2';", true) + .expect("should finish document B"), + ); + assert_eq!( + String::from_utf8(output_a).expect("should emit UTF-8"), + "" + ); + assert_eq!( + String::from_utf8(output_b).expect("should emit UTF-8"), + "" + ); + } + + #[test] + fn structured_and_probe_composition_at_every_identifier_prefix_tail() { + for prefix in ["_", "__", "__n", "__ne", "__nex", "__next", "__next_"] { + let mut processor = create_html_processor(mixed_config(100000, 100000)); + let first = format!(""; + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept explicit prefix tail"); + bytes.extend( + processor + .process_chunk(second.as_bytes(), true) + .expect("should finish suppressed text"), + ); + let html = String::from_utf8(bytes).expect("should emit UTF-8"); + let script = html + .split("") + .next() + .expect("should close script"); + assert_eq!( + serde_json::from_str::(script).expect("should keep valid JSON"), + json!({"padding": prefix, "href": "https://test.example.com/page"}) + ); + } + } + + #[test] + fn mixed_existing_app_fixture_completes_with_both_enabled() { + let fixture = include_str!("../../html_processor.test.html"); + let data = json!({"href": "https://origin.example.com/fixture-proof", "tag": "http://www.googletagmanager.com/gtm.js", "ga": "//www.google-analytics.com/g/collect?v=2"}); + let body = data.to_string(); + let push = format!( + "", + serde_json::to_string(&format!("f:T{:x},{body}", body.len())) + .expect("should encode fixture proof") + ); + let (before, after) = fixture + .rsplit_once("") + .expect("should retain structural body close"); + let fixture_copy = format!("{before}{push}{after}"); + assert!( + fixture_copy.contains( + "(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])" + ), + "should retain real bootstrap sequence" + ); + let mut expected = data; + expected["href"] = json!("https://test.example.com/fixture-proof"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js"); + expected["ga"] = json!("/integrations/google_tag_manager/g/collect?v=2"); + for chunk in [32, 8192] { + let output = mixed_output(&fixture_copy, chunk); + let payloads = flight_payloads(&output); + let proof = payloads + .iter() + .find(|payload| payload.starts_with("f:T")) + .expect("should retain designated Flight proof"); + assert_mixed_t_model(proof, &expected); + assert!(!output.contains("__ts_rsc_")); + assert!( + output.contains("googletagmanager.com/ns.html"), + "should retain unsupported iframe routing" + ); + assert!(output.trim_end().ends_with("")); + } + } + + fn flight_html(payloads: &[&str]) -> String { + let scripts: String = payloads + .iter() + .map(|payload| { + format!( + "", + serde_json::to_string(payload).expect("should encode Flight") + ) + }) + .collect(); + format!("{scripts}

suffix

") + } + + #[test] + fn mixed_flight_cross_script_lengths_and_split_hostname_safety() { + let body = r#"{"tag":"http://www.googletagmanager.com/gtm.js","url":"https://origin.example.com/page","text":"é😀"}"#; + let payload = format!("1:T{:x},{body}", body.len()); + for split in [ + (payload.find("http:").expect("should locate URL") - 1), + payload + .find("tagmanager") + .expect("should locate hostname split"), + ] { + for chunk in [32, 64, 1000, 8192] { + let output = + mixed_output(&flight_html(&[&payload[..split], &payload[split..]]), chunk); + let combined = flight_payloads(&output).concat(); + let (header, rewritten) = combined.split_once(',').expect("should retain T header"); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should parse length"), + rewritten.len() + ); + let data: serde_json::Value = + serde_json::from_str(rewritten).expect("should preserve model JSON"); + if split == (payload.find("http:").expect("should locate URL") - 1) { + assert_eq!(data["tag"], "/integrations/google_tag_manager/gtm.js"); + } else { + assert_eq!( + data["tag"], "http://www.googletagmanager.com/gtm.js", + "should safely preserve URL split across actual payload scripts" + ); + } + assert_eq!(data["url"], "https://test.example.com/page"); + assert_eq!(data["text"], "é😀"); + } + } + let escaped = body.replace('/', r"\/"); + let payload = format!("1:T{:x},{escaped}", escaped.len()); + let output = mixed_output(&flight_html(&[&payload]), 32); + let payloads = flight_payloads(&output); + let (header, body) = payloads[0].split_once(',').expect("should retain T"); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should parse length"), + body.len() + ); + let data: serde_json::Value = + serde_json::from_str(body).expect("should preserve escaped model JSON"); + assert_eq!(data["tag"], "/integrations/google_tag_manager/gtm.js"); + } + + #[test] + fn mixed_flight_fallback_preserves_original_payloads_and_later_scripts() { + let url = "'http://www.googletagmanager.com/gtm.js?id=GTM-MIX1'"; + let incomplete = format!("1:Tffff,{url}"); + let oversized = format!("1:T{:x},{url}", url.len()); + for (payloads, group_limit, script_limit) in [ + (vec![incomplete.as_str()], 10000, 10000), + (vec!["1:T", url], 10000, 10000), + (vec![oversized.as_str()], 24, 10000), + (vec![oversized.as_str()], 10000, 10), + ] { + for chunk in [32, 64, 1000, 8192] { + let html = flight_html(&payloads).replace("

suffix

", "

suffix

"); + let output = mixed_output_with_budget(&html, chunk, group_limit, script_limit); + assert_eq!( + flight_payloads(&output), + payloads, + "should preserve fallback payload bytes: group={group_limit}, script={script_limit}, chunk={chunk}" + ); + if script_limit > 10 { + assert!( + output.contains("var ordinary='/integrations/google_tag_manager/gtm.js';"), + "should rewrite ordinary script after Flight fallback" + ); + } + assert!(output.ends_with("")); + } + } + } + + #[test] + fn mixed_flight_response_completes() { + let payload = r#"0:{\"href\":\"https://origin.example.com/page\",\"url\":\"https://www.googletagmanager.com/gtm.js?id=GTM-MIX1\"}\n"#; + let html = format!( + r#"

suffix

"# + ); + let output = mixed_output(&html, 8192); + assert!( + output.contains("test.example.com/page"), + "should rewrite origin" + ); + assert!(!output.contains("__ts_rsc_"), "should resolve placeholders"); + assert!(output.ends_with(""), "should finish response"); + } + #[test] fn html_processor_rewrites_nextjs_script_when_enabled() { let html = r#" diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc.rs index ab144083c..9a53d5e8d 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc.rs @@ -398,12 +398,46 @@ fn find_tchunks_with_markers(content: &str) -> Option> { // Single-script T-chunk processing // ============================================================================= -pub(crate) fn rewrite_rsc_tchunks_with_rewriter( +fn rewrite_span( content: &str, rewriter: &RscUrlRewriter, origin_host: &str, request_host: &str, request_scheme: &str, + transform: Option String>, +) -> String { + let rewritten = rewriter.rewrite_to_string(content, origin_host, request_host, request_scheme); + match transform { + Some(transform) => transform(&rewritten), + None => rewritten, + } +} + +#[cfg(test)] +fn rewrite_rsc_tchunks_with_rewriter( + content: &str, + rewriter: &RscUrlRewriter, + origin_host: &str, + request_host: &str, + request_scheme: &str, +) -> String { + rewrite_rsc_tchunks_with_transform( + content, + rewriter, + origin_host, + request_host, + request_scheme, + None, + ) +} + +fn rewrite_rsc_tchunks_with_transform( + content: &str, + rewriter: &RscUrlRewriter, + origin_host: &str, + request_host: &str, + request_scheme: &str, + transform: Option String>, ) -> String { let Some(chunks) = find_tchunks(content) else { log::warn!( @@ -413,7 +447,14 @@ pub(crate) fn rewrite_rsc_tchunks_with_rewriter( }; if chunks.is_empty() { - return rewriter.rewrite_to_string(content, origin_host, request_host, request_scheme); + return rewrite_span( + content, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + ); } let mut result = String::with_capacity(content.len()); @@ -421,15 +462,24 @@ pub(crate) fn rewrite_rsc_tchunks_with_rewriter( for chunk in &chunks { let before = &content[last_end..chunk.match_start]; - result.push_str( - rewriter - .rewrite(before, origin_host, request_host, request_scheme) - .as_ref(), - ); + result.push_str(&rewrite_span( + before, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + )); let chunk_content = &content[chunk.header_end..chunk.content_end]; - let rewritten_content = - rewriter.rewrite_to_string(chunk_content, origin_host, request_host, request_scheme); + let rewritten_content = rewrite_span( + chunk_content, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + ); let new_length = calculate_unescaped_byte_length(&rewritten_content); let new_length_hex = format!("{new_length:x}"); @@ -444,11 +494,14 @@ pub(crate) fn rewrite_rsc_tchunks_with_rewriter( } let remaining = &content[last_end..]; - result.push_str( - rewriter - .rewrite(remaining, origin_host, request_host, request_scheme) - .as_ref(), - ); + result.push_str(&rewrite_span( + remaining, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + )); result } @@ -519,23 +572,44 @@ pub(crate) fn rewrite_rsc_scripts_combined_with_limit( request_host: &str, request_scheme: &str, max_combined_payload_bytes: usize, +) -> Vec { + rewrite_rsc_scripts_combined_with_transform( + payloads, + rewriter, + origin_host, + request_host, + request_scheme, + max_combined_payload_bytes, + None, + ) +} + +pub(super) fn rewrite_rsc_scripts_combined_with_transform( + payloads: &[&str], + rewriter: &RscUrlRewriter, + origin_host: &str, + request_host: &str, + request_scheme: &str, + max_combined_payload_bytes: usize, + transform: Option String>, ) -> Vec { let Some((_, preceding_payloads)) = payloads.split_last() else { return Vec::new(); }; // Early exit if no payload contains the origin host - avoids regex compilation - if !payloads.iter().any(|p| p.contains(origin_host)) { + if transform.is_none() && !payloads.iter().any(|p| p.contains(origin_host)) { return payloads.iter().map(|p| (*p).to_owned()).collect(); } if payloads.len() == 1 { - return vec![rewrite_rsc_tchunks_with_rewriter( + return vec![rewrite_rsc_tchunks_with_transform( payloads[0], rewriter, origin_host, request_host, request_scheme, + transform, )]; } @@ -570,12 +644,13 @@ pub(crate) fn rewrite_rsc_scripts_combined_with_limit( return payloads .iter() .map(|p| { - rewrite_rsc_tchunks_with_rewriter( + rewrite_rsc_tchunks_with_transform( p, rewriter, origin_host, request_host, request_scheme, + transform, ) }) .collect(); @@ -615,7 +690,16 @@ pub(crate) fn rewrite_rsc_scripts_combined_with_limit( if chunks.is_empty() { return payloads .iter() - .map(|p| rewriter.rewrite_to_string(p, origin_host, request_host, request_scheme)) + .map(|p| { + rewrite_span( + p, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + ) + }) .collect(); } @@ -624,15 +708,24 @@ pub(crate) fn rewrite_rsc_scripts_combined_with_limit( for chunk in &chunks { let before = &combined[last_end..chunk.match_start]; - result.push_str( - rewriter - .rewrite(before, origin_host, request_host, request_scheme) - .as_ref(), - ); + result.push_str(&rewrite_span( + before, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + )); let chunk_content = &combined[chunk.header_end..chunk.content_end]; - let rewritten_content = - rewriter.rewrite_to_string(chunk_content, origin_host, request_host, request_scheme); + let rewritten_content = rewrite_span( + chunk_content, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + ); let new_length = calculate_unescaped_byte_length_skip_markers(&rewritten_content); let new_length_hex = format!("{new_length:x}"); @@ -647,11 +740,14 @@ pub(crate) fn rewrite_rsc_scripts_combined_with_limit( } let remaining = &combined[last_end..]; - result.push_str( - rewriter - .rewrite(remaining, origin_host, request_host, request_scheme) - .as_ref(), - ); + result.push_str(&rewrite_span( + remaining, + rewriter, + origin_host, + request_host, + request_scheme, + transform, + )); result.split(RSC_MARKER).map(String::from).collect() } diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs index 7fd8d70a8..9b1c78311 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs @@ -1,4 +1,6 @@ -use std::sync::Arc; +use std::sync::{Arc, LazyLock}; + +use regex::Regex; use crate::integrations::{ IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, @@ -12,11 +14,37 @@ use super::rsc_stream::{ capture_fragment, classify_rsc_group, document_state, rsc_payload_placeholder, }; use super::shared::{ - RSC_RECEIVER_CONTEXT_BYTES, find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, + RSC_PUSH_CALL_PATTERN, RSC_PUSH_CALL_PATTERN_TRIMMED, RSC_RECEIVER_CONTEXT_BYTES, + find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, receiver_context_is_flight_push, }; use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; +// Observe actual qualified push calls, not bare property references. Unlike +// capture, ownership also covers unsupported channels and whitespace spellings +// because raw fallback must retain their original T lengths. +static QUALIFIED_FLIGHT_PUSH_PATTERN: LazyLock = LazyLock::new(|| { + Regex::new(r"(?:self|window)\.__next_f\s*(?:\.\s*push\s*\(|=\s*(?:self|window)\.__next_f\s*\|\|\s*\[\]\s*\)\s*\.\s*push\s*\()") + .expect("should compile qualified Flight push observation") +}); + +// The receiver may already have streamed, leaving a verified trimmed head. +static TRIMMED_FLIGHT_PUSH_PATTERN: LazyLock = LazyLock::new(|| { + Regex::new(r"^__next_f\s*(?:\.\s*push\s*\(|=\s*(?:self|window)\.__next_f\s*\|\|\s*\[\]\s*\)\s*\.\s*push\s*\()") + .expect("should compile trimmed Flight push observation") +}); + +// Only completed inert bootstrap controls may be skipped. Opaque channel-2 +// strings and incomplete calls must retain the conservative raw fallback. +static INERT_FLIGHT_CONTROL_ARGUMENTS: LazyLock = LazyLock::new(|| { + Regex::new(r"^\s*\[\s*(?:0|2\s*,\s*null)\s*\]\s*\)") + .expect("should compile inert Flight control arguments") +}); + +// Collapsing whitespace runs bounds either supported call head well below this +// size, while keeping the receiver's preceding boundary for qualification. +const FLIGHT_QUALIFIER_TAIL_BYTES: usize = 96; + pub(super) struct NextJsRscPlaceholderRewriter { config: Arc, } @@ -42,67 +70,103 @@ impl NextJsRscPlaceholderRewriter { }; } - let range = if state.rsc_receiver_trimmed { - find_trimmed_rsc_push_payload_range(content) - } else { - find_rsc_push_payload_range(content) - }; - state.rsc_receiver_trimmed = false; - let Some((payload_start, payload_end)) = range else { - return if was_buffered { - ScriptRewriteAction::replace(content.to_owned()) + // Discover all ranges and validate the queue budget before committing any + // capture. A malformed or unrecognized remainder stays original/protected. + let mut ranges = Vec::new(); + let mut cursor = 0; + let mut trimmed = std::mem::take(&mut state.rsc_receiver_trimmed); + let mut queued_bytes = state.captured_payload_bytes; + while content[cursor..].contains("__next_f") { + let remaining = &content[cursor..]; + let head = if trimmed { + TRIMMED_FLIGHT_PUSH_PATTERN.find(remaining) } else { - ScriptRewriteAction::Keep + first_qualified_flight_push(remaining) }; - }; - - if payload_start > payload_end - || payload_end > content.len() - || !content.is_char_boundary(payload_start) - || !content.is_char_boundary(payload_end) - { - state.bypass_rsc = true; - return if was_buffered { - ScriptRewriteAction::replace(content.to_owned()) + if let Some(head) = head + && let Some(arguments) = + INERT_FLIGHT_CONTROL_ARGUMENTS.find(&remaining[head.end()..]) + { + cursor += head.end() + arguments.end(); + trimmed = false; + continue; + } + let range = if trimmed { + find_trimmed_rsc_push_payload_range(remaining) } else { - ScriptRewriteAction::Keep + find_rsc_push_payload_range(remaining) + }; + let call = if trimmed { + RSC_PUSH_CALL_PATTERN_TRIMMED.find(remaining) + } else { + RSC_PUSH_CALL_PATTERN.find(remaining) + }; + trimmed = false; + let Some((start, end)) = range.filter(|(start, _)| { + call.is_some_and(|call| { + call.end() == *start && !contains_qualified_flight(&remaining[..call.start()]) + }) + }) else { + // A raw qualified push cannot safely continue a pending T group + // or be skipped by later captures. A bare property reference is + // ordinary script text and must not disable later Flight capture. + state.bypass_rsc |= state.flight_node_owned; + return if was_buffered { + ScriptRewriteAction::replace(content.to_owned()) + } else { + ScriptRewriteAction::Keep + }; }; + let start = cursor + start; + let end = cursor + end; + let length = end.saturating_sub(start); + // Group bytes and queued parser-held bytes have distinct limits: + // independent groups parsed in one call must not consume each other's + // group allowance before the output processor can release them. + let total = queued_bytes.checked_add(length); + if start > end + || end >= content.len() + || !content.is_char_boundary(start) + || !content.is_char_boundary(end) + || length > limit + || state.captured_payloads.len() + ranges.len() >= MAX_UNRESOLVED_RSC_PAYLOADS + || total.is_none_or(|bytes| bytes > max_queued_payload_bytes) + { + state.bypass_rsc = true; + return if was_buffered { + ScriptRewriteAction::replace(content.to_owned()) + } else { + ScriptRewriteAction::Keep + }; + } + queued_bytes = total.expect("should have validated queue size"); + ranges.push((start, end)); + cursor = end + 1; } - - let payload = &content[payload_start..payload_end]; - // `limit` bounds one script here and the downstream unresolved group in - // `classify_rsc_group`. It deliberately does not bound this queue in - // aggregate: the processor cannot decrement a resolved group until the - // whole parser call returns, so a shared `limit` budget would make - // independent payloads bypass each other purely because they shared a - // source chunk. The queue is parser-held script text, so it is bounded - // by the parser's own script-buffer budget instead — the same budget - // `NextJsNextDataRewriter` buffers against — plus a payload count. - let exceeds_limit = payload.len() > limit - || state.captured_payloads.len() >= MAX_UNRESOLVED_RSC_PAYLOADS - || state - .captured_payload_bytes - .checked_add(payload.len()) - .is_none_or(|queued| queued > max_queued_payload_bytes); - if exceeds_limit { - state.bypass_rsc = true; + if ranges.is_empty() { return if was_buffered { ScriptRewriteAction::replace(content.to_owned()) } else { ScriptRewriteAction::Keep }; } - - let placeholder = rsc_payload_placeholder(&state.namespace, state.next_placeholder_index); - state.next_placeholder_index = state.next_placeholder_index.saturating_add(1); - state.captured_payload_bytes += payload.len(); - state.captured_payloads.push_back(CapturedPayload { - placeholder: placeholder.clone(), - original: payload.to_owned(), - }); - - let mut rewritten = content.to_owned(); - rewritten.replace_range(payload_start..payload_end, &placeholder); + let mut rewritten = String::with_capacity(content.len()); + cursor = 0; + for (start, end) in ranges { + let placeholder = + rsc_payload_placeholder(&state.namespace, state.next_placeholder_index); + state.next_placeholder_index = state.next_placeholder_index.saturating_add(1); + state.captured_payloads.push_back(CapturedPayload { + placeholder: placeholder.clone(), + original: content[start..end].to_owned(), + }); + rewritten.push_str(&content[cursor..start]); + rewritten.push_str(&placeholder); + cursor = end; + } + state.captured_payload_bytes = queued_bytes; + rewritten.push_str(&content[cursor..]); + state.current_fragment_protected = false; ScriptRewriteAction::replace(rewritten) } @@ -124,12 +188,15 @@ impl NextJsRscPlaceholderRewriter { FragmentCapture::Suppress => ScriptRewriteAction::RemoveNode, FragmentCapture::Restore(restored) => { state.rsc_receiver_trimmed = false; - state.bypass_rsc = true; + state.bypass_rsc |= state.flight_node_owned; ScriptRewriteAction::replace(restored) } FragmentCapture::PassThrough => { let trimmed = state.rsc_receiver_trimmed; state.rsc_receiver_trimmed = false; + if !state.flight_node_owned { + return ScriptRewriteAction::Keep; + } if is_last && content.len() > limit && content.contains("__next_f") { let mut remaining = content; let mut receiver_trimmed = trimmed; @@ -167,26 +234,12 @@ impl NextJsRscPlaceholderRewriter { } } } -} - -impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { - fn integration_id(&self) -> &'static str { - NEXTJS_INTEGRATION_ID - } - - fn selector(&self) -> &'static str { - "script" - } - - fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { - if !self.config.enabled || self.config.rewrite_attributes.is_empty() { - return ScriptRewriteAction::keep(); - } - - let state = document_state(ctx.document_state); - let mut state = state - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); + fn rewrite_fragment( + &self, + content: &str, + ctx: &IntegrationScriptContext<'_>, + state: &mut super::rsc_stream::NextJsDocumentState, + ) -> ScriptRewriteAction { if state.bypass_rsc { // The downstream processor can enter bypass after the parser has // suppressed part of this script. Restore it before passing through @@ -213,7 +266,7 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { return self.rewrite_claimed_fragment( content, ctx.is_last_in_text_node, - &mut state, + state, limit, ctx.max_buffered_script_bytes, ); @@ -299,7 +352,7 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { let action = self.rewrite_claimed_fragment( claimed, ctx.is_last_in_text_node, - &mut state, + state, limit, ctx.max_buffered_script_bytes, ); @@ -315,6 +368,105 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { } } +impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { + fn integration_id(&self) -> &'static str { + NEXTJS_INTEGRATION_ID + } + + fn selector(&self) -> &'static str { + "script" + } + + fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { + let shared = document_state(ctx.document_state); + let mut state = shared + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if std::mem::take(&mut state.next_data_fragment) { + // Pages data is not executable Flight, even when a JSON string + // contains a complete push spelling. Clear the downstream snapshot + // as well so a prior protected script cannot suppress Pages GTM. + state.current_fragment_protected = false; + state.flight_node_owned = false; + state.flight_qualifier_tail.clear(); + return ScriptRewriteAction::Keep; + } + if !self.config.enabled || self.config.rewrite_attributes.is_empty() { + state.current_fragment_protected = false; + return ScriptRewriteAction::Keep; + } + observe_flight_qualifier(&mut state, content); + // A call head can complete after a tentative property probe overflowed. + // Once it proves Flight, preserve its entire raw continuation/group. + if state.flight_node_owned + && matches!( + state.rsc_script, + super::rsc_stream::FragmentState::BypassUntilLast + ) + { + state.bypass_rsc = true; + } + state.current_fragment_protected = state.flight_node_owned; + let action = self.rewrite_fragment(content, ctx, &mut state); + if ctx.is_last_in_text_node { + state.flight_node_owned = false; + state.flight_qualifier_tail.clear(); + state.rsc_probe.clear(); + state.rsc_receiver_context.clear(); + state.rsc_receiver_trimmed = false; + } + action + } +} + +fn contains_qualified_flight(source: &str) -> bool { + first_qualified_flight_push(source).is_some() +} + +fn first_qualified_flight_push(source: &str) -> Option> { + QUALIFIED_FLIGHT_PUSH_PATTERN + .find_iter(source) + .find(|call| { + let identifier = call.start() + + call + .as_str() + .find("__next_f") + .expect("should find identifier in qualified push pattern"); + receiver_context_is_flight_push(&source[..identifier]) + }) +} + +/// Observe a bounded whitespace-normalized call head without buffering script +/// source, including initializer heads split after document bypass. +fn observe_flight_qualifier(state: &mut super::rsc_stream::NextJsDocumentState, content: &str) { + if state.flight_node_owned { + return; + } + for character in content.chars() { + let character = if character.is_whitespace() { + ' ' + } else { + character + }; + if character == ' ' && state.flight_qualifier_tail.ends_with(' ') { + continue; + } + state.flight_qualifier_tail.push(character); + if state.flight_qualifier_tail.len() > FLIGHT_QUALIFIER_TAIL_BYTES { + let start = state.flight_qualifier_tail.len() - FLIGHT_QUALIFIER_TAIL_BYTES; + let start = (start..state.flight_qualifier_tail.len()) + .find(|index| state.flight_qualifier_tail.is_char_boundary(*index)) + .expect("should find bounded qualifier character boundary"); + state.flight_qualifier_tail.drain(..start); + } + if character == '(' && contains_qualified_flight(&state.flight_qualifier_tail) { + state.flight_node_owned = true; + state.flight_qualifier_tail.clear(); + break; + } + } +} + /// Bytes to withhold so a `__next_f` identifier split across text fragments can /// still be recognized once the next fragment arrives. fn longest_identifier_prefix(bytes: &[u8]) -> usize { @@ -370,6 +522,295 @@ mod tests { }) } + #[test] + fn unrecognized_qualified_prefix_makes_the_whole_batch_raw_and_protected() { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + let script = r#"self.__next_f.push([2,"1:T40,'http://www.googletagmanager.com/gtm.js'"]);self.__next_f.push([1,"later"])"#; + assert_eq!( + rewriter.rewrite(script, &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should not partially hide a batch with uncaptured qualified payloads" + ); + let shared = document_state(&state); + let state = shared.lock().expect("should lock state"); + assert!(state.captured_payloads.is_empty()); + assert!(state.current_fragment_protected); + } + + #[test] + fn inert_controls_do_not_make_unsafe_batches_partially_capturable() { + for unsafe_call in [ + r#"self.__next_f.push([2,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + "self.__next_f.push([0,1])", + "self.__next_f.push([2,null", + "self.__next_f.push([0]", + ] { + for unsafe_first in [false, true] { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + let control = "self.__next_f.push([0]);self.__next_f.push([2,null])"; + let supported = r#"self.__next_f.push([1,"later"])"#; + let script = if unsafe_first { + format!("{control};{unsafe_call};{supported}") + } else { + format!("{control};{supported};{unsafe_call}") + }; + assert_eq!( + rewriter.rewrite(&script, &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should preserve unsafe batch: {script}" + ); + let shared = document_state(&state); + let state = shared.lock().expect("should lock state"); + assert!(state.captured_payloads.is_empty()); + assert!(state.current_fragment_protected); + assert!(state.bypass_rsc); + } + } + } + + #[test] + fn inert_controls_are_preserved_without_document_bypass() { + for script in [ + "(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])", + "window.__next_f . push ( [ 0 ] );window.__next_f.push( [ 2 , null ] )", + ] { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + assert_eq!( + rewriter.rewrite(script, &ctx(true, &state)), + ScriptRewriteAction::Keep + ); + let shared = document_state(&state); + let state = shared.lock().expect("should lock state"); + assert!(state.captured_payloads.is_empty()); + assert!(state.current_fragment_protected); + assert!(!state.bypass_rsc); + } + } + + #[test] + fn batched_capture_is_atomic_at_byte_and_count_limits() { + let payload = "x".repeat(40); + let push = format!(r#"self.__next_f.push([1,"{payload}"])"#); + for (count, budget, captured) in + [(2, 80, 2), (2, 79, 0), (256, 20000, 256), (257, 20000, 0)] + { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + let script = vec![push.as_str(); count].join(";"); + let action = rewriter.rewrite( + &script, + &IntegrationScriptContext { + max_buffered_script_bytes: budget, + ..ctx(true, &state) + }, + ); + let shared = document_state(&state); + let state = shared.lock().expect("should lock state"); + assert_eq!( + state.captured_payloads.len(), + captured, + "should commit every push or none" + ); + assert_eq!(state.captured_payload_bytes, captured * payload.len()); + assert_eq!( + matches!(action, ScriptRewriteAction::Replace(_)), + captured > 0 + ); + assert_eq!( + state.current_fragment_protected, + captured == 0, + "should protect rejected raw batch" + ); + } + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + let script = format!("{push};self.__next_f.push([1,\"unfinished"); + assert_eq!( + rewriter.rewrite(&script, &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should leave unsafe remainder original" + ); + assert!( + document_state(&state) + .lock() + .expect("should lock state") + .captured_payloads + .is_empty() + ); + } + + #[test] + fn bypass_ownership_recognizes_every_split_and_resets_on_empty_final() { + let mut settings = crate::test_support::tests::create_test_settings(); + settings + .integrations + .insert_config( + "google_tag_manager", + &serde_json::json!({"enabled": true, "container_id": "GTM-MIX1"}), + ) + .expect("should enable GTM"); + let registry = crate::integrations::IntegrationRegistry::with_plan( + &settings, + Arc::new(crate::auction::compile_auction_plan(&settings).expect("should compile plan")), + ) + .expect("should create registry"); + let gtm = registry + .script_rewriters() + .pop() + .expect("should register GTM"); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + let spaced_initializer = format!( + r#"(self.__next_f{}={}window.__next_f{}||{}[]{}).{}push{}([1,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + " ".repeat(256), + " ".repeat(256), + " ".repeat(256), + " ".repeat(256), + " ".repeat(256), + " ".repeat(256), + " ".repeat(256) + ); + for script in [ + r#"self.__next_f.push([1,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + r#"window.__next_f.push([1,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + r#"(self.__next_f=self.__next_f||[]).push([1,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + r#"(window.__next_f=window.__next_f||[]).push([1,"1:T40,'http://www.googletagmanager.com/gtm.js'"])"#, + &spaced_initializer, + ] { + for split in 1..script.len() { + let state = IntegrationDocumentState::default(); + document_state(&state) + .lock() + .expect("should lock state") + .bypass_rsc = true; + for fragment in [&script[..split], &script[split..], ""] { + let final_fragment = fragment.is_empty(); + assert_eq!( + rewriter.rewrite(fragment, &ctx(final_fragment, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite( + &*gtm, + fragment, + &ctx(final_fragment, &state) + ), + ScriptRewriteAction::Keep, + "should preserve raw Flight at split {split}" + ); + assert!( + document_state(&state) + .lock() + .expect("should lock state") + .flight_qualifier_tail + .len() + <= FLIGHT_QUALIFIER_TAIL_BYTES, + "should bound fallback qualification observation" + ); + } + let ordinary = "var ordinary='http://www.googletagmanager.com/gtm.js';"; + assert_eq!( + rewriter.rewrite(ordinary, &ctx(true, &state)), + ScriptRewriteAction::Keep + ); + assert!( + matches!(IntegrationScriptRewriter::rewrite(&*gtm, ordinary, &ctx(true, &state)), ScriptRewriteAction::Replace(ref text) if text.contains("/integrations/google_tag_manager/gtm.js")) + ); + } + } + } + + #[test] + fn raw_flight_fallback_is_protected_but_later_ordinary_gtm_is_not() { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(Arc::new(NextJsIntegrationConfig { + max_combined_payload_bytes: 24, + ..(*test_config()).clone() + })); + let mut settings = crate::test_support::tests::create_test_settings(); + settings + .integrations + .insert_config( + "google_tag_manager", + &serde_json::json!({"enabled": true, "container_id": "GTM-MIX1"}), + ) + .expect("should configure GTM"); + let registry = crate::integrations::IntegrationRegistry::with_plan( + &settings, + Arc::new(crate::auction::compile_auction_plan(&settings).expect("should compile plan")), + ) + .expect("should create registry"); + let gtm = registry + .script_rewriters() + .pop() + .expect("should register script hook"); + let raw = r#"self.__next_f.push([1,"1:T2,'http://www.googletagmanager.com/gtm.js?id=GTM-MIX1'"] )"#; + assert_eq!( + rewriter.rewrite(raw, &ctx(true, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*gtm, raw, &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should not change original T bytes on overflow" + ); + for part in [ + "self.__n", + "ext_f.push([1,\"", + "1:T2,'http://www.googletagmanager.com/gtm.js?id=GTM-MIX1'\"])", + ] { + let last = part.ends_with(")"); + assert_eq!( + rewriter.rewrite(part, &ctx(last, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*gtm, part, &ctx(last, &state)), + ScriptRewriteAction::Keep, + "should recognize and protect split qualified flight after bypass" + ); + } + let prefix = "var before='google"; + assert_eq!( + rewriter.rewrite(prefix, &ctx(false, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*gtm, prefix, &ctx(false, &state)), + ScriptRewriteAction::RemoveNode + ); + let flight = r#"';self.__next_f.push([1,"1:T2,'http://www.googletagmanager.com/gtm.js'"])"#; + assert_eq!( + rewriter.rewrite(flight, &ctx(false, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*gtm, flight, &ctx(false, &state)), + ScriptRewriteAction::Replace(format!("{prefix}{flight}")), + "should drain GTM-held prefix unchanged before protected raw text" + ); + assert_eq!( + rewriter.rewrite("", &ctx(true, &state)), + ScriptRewriteAction::Keep + ); + assert_eq!( + IntegrationScriptRewriter::rewrite(&*gtm, "", &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should preserve empty final protection" + ); + let ordinary = "var url='http://www.googletagmanager.com/gtm.js';"; + assert_eq!( + rewriter.rewrite(ordinary, &ctx(true, &state)), + ScriptRewriteAction::Keep + ); + assert!( + matches!(IntegrationScriptRewriter::rewrite(&*gtm, ordinary, &ctx(true, &state)), ScriptRewriteAction::Replace(ref value) if value.contains("/integrations/google_tag_manager/gtm.js")), + "should not leak flight ownership to ordinary scripts" + ); + } + #[test] fn inserts_placeholder_and_records_payload() { let state = IntegrationDocumentState::default(); @@ -459,13 +900,13 @@ mod tests { })); assert_eq!( - rewriter.rewrite("self.__next_f", &ctx(false, &state)), + rewriter.rewrite("self.__next_f.push(", &ctx(false, &state)), ScriptRewriteAction::RemoveNode, "should initially suppress the script prefix", ); assert_eq!( rewriter.rewrite("-payload-overflow", &ctx(false, &state)), - ScriptRewriteAction::Replace("self.__next_f-payload-overflow".to_owned()), + ScriptRewriteAction::Replace("self.__next_f.push(-payload-overflow".to_owned()), "should restore suppressed text before overflow", ); assert_eq!( diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs index ab12e13f1..aa3262b37 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs @@ -9,7 +9,7 @@ use crate::streaming_processor::StreamProcessor; use super::rsc::{ DEFAULT_MAX_COMBINED_PAYLOAD_BYTES, PendingTChunk, TChunkStep, next_tchunk, - rewrite_rsc_scripts_combined_with_limit, + rewrite_rsc_scripts_combined_with_transform, }; use super::shared::RscUrlRewriter; use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; @@ -36,6 +36,9 @@ pub(super) struct CapturedPayload { pub(super) struct NextJsDocumentState { pub(super) namespace: String, pub(super) next_data: FragmentState, + /// One-shot discriminator set by the matched Pages stage and consumed by + /// the immediately following Flight stage, including empty/pass-through text. + pub(super) next_data_fragment: bool, pub(super) rsc_script: FragmentState, pub(super) rsc_probe: String, /// Tail of script text already released for the current text node, kept so a @@ -49,6 +52,12 @@ pub(super) struct NextJsDocumentState { pub(super) captured_payload_bytes: usize, pub(super) next_placeholder_index: usize, pub(super) bypass_rsc: bool, + /// Qualified Flight ownership lasts only for the current script text node. + pub(super) flight_node_owned: bool, + /// Snapshot consumed by downstream GTM, including the final raw fragment. + pub(super) current_fragment_protected: bool, + /// Bounded qualifier observation remains active even after document bypass. + pub(super) flight_qualifier_tail: String, } impl Default for NextJsDocumentState { @@ -56,6 +65,7 @@ impl Default for NextJsDocumentState { Self { namespace: uuid::Uuid::new_v4().simple().to_string(), next_data: FragmentState::Idle, + next_data_fragment: false, rsc_script: FragmentState::Idle, rsc_probe: String::new(), rsc_receiver_context: String::new(), @@ -64,6 +74,9 @@ impl Default for NextJsDocumentState { captured_payload_bytes: 0, next_placeholder_index: 0, bypass_rsc: false, + flight_node_owned: false, + current_fragment_protected: false, + flight_qualifier_tail: String::new(), } } } @@ -145,11 +158,15 @@ pub(super) fn capture_fragment<'a>( pub(super) struct NextJsRscStreamProcessorFactory { config: Arc, + transform: Option String>, } impl NextJsRscStreamProcessorFactory { - pub(super) fn new(config: Arc) -> Self { - Self { config } + pub(super) fn new( + config: Arc, + transform: Option String>, + ) -> Self { + Self { config, transform } } } @@ -170,6 +187,7 @@ impl IntegrationHtmlStreamProcessorFactory for NextJsRscStreamProcessorFactory { context.request_host, context.request_scheme, limit, + self.transform, )) } } @@ -185,6 +203,7 @@ pub(super) struct NextJsRscStreamProcessor { group: Vec, classifier: RscGroupClassifier, rewriter: RscUrlRewriter, + transform: Option String>, } impl NextJsRscStreamProcessor { @@ -194,6 +213,7 @@ impl NextJsRscStreamProcessor { request_host: String, request_scheme: String, limit: usize, + transform: Option String>, ) -> Self { Self { state, @@ -206,6 +226,7 @@ impl NextJsRscStreamProcessor { group: Vec::new(), classifier: RscGroupClassifier::new(limit), rewriter: RscUrlRewriter::new(), + transform, } } @@ -325,13 +346,14 @@ impl NextJsRscStreamProcessor { .iter() .map(|payload| payload.original.as_str()) .collect(); - let rewritten = rewrite_rsc_scripts_combined_with_limit( + let rewritten = rewrite_rsc_scripts_combined_with_transform( &payloads, &self.rewriter, &self.origin_host, &self.request_host, &self.request_scheme, self.limit, + self.transform, ); if rewritten.len() != self.group.len() { log::warn!( @@ -1126,11 +1148,60 @@ mod tests { "proxy.example.com".to_owned(), "https".to_owned(), limit, + None, ), placeholders, ) } + #[test] + fn enabled_gtm_transform_never_mutates_fallback_originals() { + let body = "'http://www.googletagmanager.com/gtm.js'"; + let invalid = format!("1:Tzz,{body}"); + let incomplete = format!("1:Tffff,{body}"); + let header_tail = format!("T{:x},{body}", body.len()); + let escape_tail = format!("41{body}"); + let marker_tail = format!("c\n\0SPLIT\0{body}"); + for payloads in [ + vec![invalid.as_str()], + vec![incomplete.as_str()], + vec!["1:", header_tail.as_str()], + vec![r"1:T2,\u00", escape_tail.as_str()], + vec!["1:T3,ab", marker_tail.as_str()], + ] { + let (mut processor, placeholders) = processor_with_payloads(&payloads, 10000); + processor.transform = + Some(crate::integrations::google_tag_manager::rewrite_gtm_rsc_span); + let output = processor + .process_chunk(placeholders.concat().as_bytes(), true) + .expect("should restore fallback group"); + assert_eq!( + output, + payloads.concat().as_bytes(), + "should never transform original fallback bytes" + ); + } + let (mut processor, placeholders) = processor_with_payloads(&[&incomplete, body], 120); + processor.transform = Some(crate::integrations::google_tag_manager::rewrite_gtm_rsc_span); + let interstitial = "x".repeat(100); + let input = format!("{}{interstitial}{}", placeholders[0], placeholders[1]); + assert_eq!( + processor + .process_chunk(input.as_bytes(), true) + .expect("should restore held-output overflow"), + format!("{incomplete}{interstitial}{body}").as_bytes() + ); + let (mut processor, _) = processor_with_payloads(&[body], 1000); + processor.transform = Some(crate::integrations::google_tag_manager::rewrite_gtm_rsc_span); + assert!( + processor + .process_chunk(&[], true) + .expect_err("should reject missing captured placeholder") + .to_string() + .contains("not present") + ); + } + #[test] fn stream_processor_emits_ordinary_html_before_eof() { let (mut processor, _) = processor_with_payloads(&[], 1024); diff --git a/crates/trusted-server-core/src/integrations/nextjs/script_rewriter.rs b/crates/trusted-server-core/src/integrations/nextjs/script_rewriter.rs index b6de84449..e45462620 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/script_rewriter.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/script_rewriter.rs @@ -62,14 +62,14 @@ impl IntegrationScriptRewriter for NextJsNextDataRewriter { } fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { - if self.config.rewrite_attributes.is_empty() { - return ScriptRewriteAction::keep(); - } - let state = document_state(ctx.document_state); let mut state = state .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); + state.next_data_fragment = true; + if self.config.rewrite_attributes.is_empty() { + return ScriptRewriteAction::keep(); + } match capture_fragment( &mut state.next_data, diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 47c521e61..a41f986f1 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -55,11 +55,18 @@ pub enum AttributeRewriteOutcome { RemoveElement, } -/// Action returned by inline script rewriters to describe how to mutate the node. +/// Action on the current working script text fragment, composed in registration order. +/// +/// Every matching stage receives the previous stage's output, including empty +/// suppressed text and the original final-fragment flag. Only the composed result +/// is applied to the parser fragment. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ScriptRewriteAction { + /// Preserve the current working text, not the original parser text. Keep, + /// Pass replacement text to subsequent matching stages. Replace(String), + /// Suppress this fragment; downstream stages still run and may flush held text. RemoveNode, } @@ -93,6 +100,7 @@ pub struct IntegrationAttributeContext<'a> { /// Context passed to script/text rewriters for inline HTML handling. #[derive(Debug)] pub struct IntegrationScriptContext<'a> { + /// The individual rewriter's CSS selector, matched by `lol_html`. pub selector: &'a str, pub request_host: &'a str, pub request_scheme: &'a str, diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index d6e85a167..44889501f 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -66,11 +66,14 @@ pub mod tests { /// drive byte-identical input. #[cfg(any(test, feature = "test-utils"))] pub mod nextjs_auction { + use std::io::{Read as _, Write as _}; use std::net::IpAddr; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; + use bytes::Bytes; use error_stack::Report; + use flate2::{read::GzDecoder, write::GzEncoder}; use crate::geo::GeoInfo; use crate::platform::{ @@ -194,13 +197,150 @@ pub mod nextjs_auction { ) } + /// Isolated Next.js/GTM settings without auction dispatch. + /// + /// # Panics + /// + /// Panics if the static GTM configuration is invalid. + #[must_use] + pub fn script_composition_settings() -> Settings { + let mut settings = settings(); + settings.auction.enabled = false; + settings.creative_opportunities = None; + settings + .integrations + .insert_config( + "google_tag_manager", + &serde_json::json!({ + "enabled": true, "container_id": "GTM-MIX1" + }), + ) + .expect("should enable fixture GTM"); + settings + } + + const SCRIPT_COMPOSITION_BOOTSTRAP: &str = + "(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])"; + + /// HTML with standard bootstrap controls and a mixed Flight T record, + /// crossing parser-internal boundaries with same-delimiter query escapes. + #[must_use] + pub fn script_composition_html() -> String { + let content = serde_json::json!({ + "url": format!("https://{ORIGIN_HOST}/app"), + "tag": "http://www.googletagmanager.com/gtm.js?q=\"x\"&v=é", + "ga": "//www.google-analytics.com/g/collect?v=2", + "text": "é😀", "padding": "_".repeat(11000), + }) + .to_string(); + let payload = serde_json::json!(format!("1:T{:x},{content}", content.len())); + format!( + "

script-composition-suffix

" + ) + } + + /// Assert fully consumed adapter output, decoded JSON and recomputed T length. + /// + /// # Panics + /// + /// Panics if response coding, Flight syntax, rewrites, or completion are wrong. + pub fn assert_script_composition_response(bytes: &[u8], gzip: bool) { + let mut decoded = Vec::new(); + if gzip { + GzDecoder::new(bytes) + .read_to_end(&mut decoded) + .expect("should decode complete gzip response"); + } else { + decoded.extend_from_slice(bytes); + } + let html = String::from_utf8(decoded).expect("should emit UTF-8 HTML"); + assert!( + html.contains("

script-composition-suffix

"), + "should deliver suffix sentinel" + ); + assert!(html.ends_with(""), "should finish HTML"); + assert!( + !html.contains("__ts_rsc_"), + "should resolve every captured payload" + ); + assert!( + html.contains(SCRIPT_COMPOSITION_BOOTSTRAP), + "should preserve inert bootstrap controls" + ); + assert_eq!( + html.matches("self.__next_f.push([1,").count(), + 1, + "should emit Flight once" + ); + let json = html + .rsplit("self.__next_f.push(") + .next() + .expect("should keep push") + .split(")") + .next() + .expect("should close push"); + let push: serde_json::Value = + serde_json::from_str(json).expect("should retain valid push JSON"); + let payload = push[1].as_str().expect("should retain payload"); + let (header, body) = payload.split_once(',').expect("should retain T record"); + assert_eq!( + usize::from_str_radix(&header[3..], 16).expect("should parse hex length"), + body.len(), + "should recount decoded T bytes" + ); + let data: serde_json::Value = + serde_json::from_str(body).expect("should preserve valid model JSON"); + assert!( + data["url"] + .as_str() + .expect("should retain URL") + .contains(&format!("{PUBLISHER_HOST}/app")), + "should rewrite origin" + ); + assert_eq!( + data["tag"], + "/integrations/google_tag_manager/gtm.js?q=\"x\"&v=é" + ); + assert_eq!(data["ga"], "/integrations/google_tag_manager/g/collect?v=2"); + assert_eq!(data["text"], "é😀"); + assert_eq!(data["padding"], "_".repeat(11000)); + } + /// Upstream that serves [`origin_html`] and one deterministic bid. #[derive(Default)] pub struct NextJsAuctionOrigin { auction_requests: AtomicUsize, + html_response: Option>, + gzip: bool, + streaming: bool, } impl NextJsAuctionOrigin { + /// Serve canned HTML instead of the default fixture, optionally gzip-encoded + /// and streamed in small chunks. The default fixture remains unchanged. + /// + /// # Panics + /// + /// Panics if in-memory gzip encoding fails. + #[must_use] + pub fn with_html_response(html: &str, gzip: bool, streaming: bool) -> Self { + let bytes = if gzip { + let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(html.as_bytes()) + .expect("should encode fixture gzip"); + encoder.finish().expect("should finish fixture gzip") + } else { + html.as_bytes().to_vec() + }; + Self { + html_response: Some(bytes), + gzip, + streaming, + ..Self::default() + } + } + /// Number of auction requests this fixture has answered. #[must_use] pub fn auction_requests(&self) -> usize { @@ -215,6 +355,10 @@ pub mod nextjs_auction { #[async_trait::async_trait(?Send)] impl PlatformHttpClient for NextJsAuctionOrigin { + fn supports_streaming_responses(&self) -> bool { + self.streaming + } + async fn send( &self, request: PlatformHttpRequest, @@ -241,11 +385,31 @@ pub mod nextjs_auction { .attach(format!("unexpected fixture upstream: {host:?}"))); } }; + let canned_origin = + request.request.uri().host() == Some(ORIGIN_HOST) && self.html_response.is_some(); + let bytes = if canned_origin { + self.html_response + .as_ref() + .expect("should have canned response") + .clone() + } else { + body.into_bytes() + }; + let mut response = http::Response::builder() + .status(200) + .header("content-type", content_type); + if canned_origin && self.gzip { + response = response.header("content-encoding", "gzip"); + } + let body = if request.stream_response && self.streaming { + let chunks: Vec = bytes.chunks(97).map(Bytes::copy_from_slice).collect(); + edgezero_core::body::Body::stream(futures::stream::iter(chunks)) + } else { + edgezero_core::body::Body::from(bytes) + }; Ok(PlatformResponse::new( - http::Response::builder() - .status(200) - .header("content-type", content_type) - .body(edgezero_core::body::Body::from(body)) + response + .body(body) .expect("should build deterministic upstream response"), ) .with_backend_name(request.backend_name)) From 770f3d62871e95990b9d8a1a911aadd24d9ef168 Mon Sep 17 00:00:00 2001 From: Christian Date: Wed, 30 Sep 2026 17:27:10 -0500 Subject: [PATCH 110/230] Share compressed HTML processing in Next.js tests --- .../src/integrations/nextjs/mod.rs | 99 ++++++++----------- 1 file changed, 39 insertions(+), 60 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index 4036e5e76..07c687745 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -215,6 +215,43 @@ mod tests { String::from_utf8(output).expect("should emit UTF-8") } + fn mixed_output_with_compression( + html: &str, + compression: Compression, + chunk_size: usize, + ) -> String { + let input = if compression == Compression::Gzip { + let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(html.as_bytes()) + .expect("should encode gzip input"); + encoder.finish().expect("should finish gzip input") + } else { + html.as_bytes().to_vec() + }; + let mut pipeline = StreamingPipeline::new( + PipelineConfig { + input_compression: compression, + output_compression: compression, + chunk_size, + }, + create_html_processor(mixed_config(100000, 100000)), + ); + let mut output = Vec::new(); + pipeline + .process(Cursor::new(&input), &mut output) + .expect("should consume complete mixed response"); + let mut decoded = Vec::new(); + if compression == Compression::Gzip { + GzDecoder::new(output.as_slice()) + .read_to_end(&mut decoded) + .expect("should decode gzip output"); + } else { + decoded = output; + } + String::from_utf8(decoded).expect("should emit UTF-8") + } + #[test] fn mixed_next_data_flight_mentions_do_not_claim_json_or_hide_gtm() { for text in [ @@ -447,37 +484,8 @@ mod tests { }; let html = format!("{scripts}

suffix

"); for compression in [Compression::None, Compression::Gzip] { - let input = if compression == Compression::Gzip { - let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); - encoder - .write_all(html.as_bytes()) - .expect("should encode gzip bootstrap document"); - encoder.finish().expect("should finish gzip input") - } else { - html.as_bytes().to_vec() - }; for chunk_size in [32, 64, 1000, 8192] { - let mut pipeline = StreamingPipeline::new( - PipelineConfig { - input_compression: compression, - output_compression: compression, - chunk_size, - }, - create_html_processor(mixed_config(100000, 100000)), - ); - let mut bytes = Vec::new(); - pipeline - .process(Cursor::new(&input), &mut bytes) - .expect("should consume bootstrap document"); - let mut decoded = Vec::new(); - if compression == Compression::Gzip { - GzDecoder::new(bytes.as_slice()) - .read_to_end(&mut decoded) - .expect("should decode gzip bootstrap response"); - } else { - decoded = bytes; - } - let output = String::from_utf8(decoded).expect("should emit UTF-8"); + let output = mixed_output_with_compression(&html, compression, chunk_size); assert!( output.contains(controls), "should preserve inert bootstrap controls" @@ -695,37 +703,8 @@ mod tests { "

suffix

" ); for compression in [Compression::None, Compression::Gzip] { - let input = if compression == Compression::Gzip { - let mut encoder = GzEncoder::new(Vec::new(), flate2::Compression::default()); - encoder - .write_all(html.as_bytes()) - .expect("should encode actual gzip input"); - encoder.finish().expect("should finish gzip input") - } else { - html.as_bytes().to_vec() - }; for chunk_size in [32, 64, 1000, 8192] { - let mut pipeline = StreamingPipeline::new( - PipelineConfig { - input_compression: compression, - output_compression: compression, - chunk_size, - }, - create_html_processor(mixed_config(100000, 100000)), - ); - let mut output = Vec::new(); - pipeline - .process(Cursor::new(&input), &mut output) - .expect("should process complete compressed document"); - let mut decoded = Vec::new(); - if compression == Compression::Gzip { - GzDecoder::new(output.as_slice()) - .read_to_end(&mut decoded) - .expect("should decode actual gzip output"); - } else { - decoded = output; - } - let output = String::from_utf8(decoded).expect("should emit UTF-8"); + let output = mixed_output_with_compression(&html, compression, chunk_size); let script = output .split("{flight}", + &prefix[split..] + ); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept non-code prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should complete ordinary and Flight scripts"), + ); + let output = String::from_utf8(bytes).expect("should emit UTF-8"); + assert!( + output.contains(&format!("")), + "should preserve non-code source and rewrite ordinary GTM at split {split}: {output}" + ); + assert!( + output.contains(r#"\"url\":\"https://test.example.com/page\""#), + "should still rewrite later real Flight at split {split}: {output}" + ); + assert!( + !output.contains("__ts_rsc_"), + "should resolve every placeholder" + ); + } + } + } + + #[test] + fn lexical_filter_captures_only_executable_pushes_in_the_same_script() { + let fake = r#"const demo='self.__next_f.push([1,"0:{\"url\":\"https://origin.example.com/fake\"}\n"])';"#; + let push = r#"self.__next_f.push([1,"0:{\"url\":\"https://origin.example.com/page\"}\n"])"#; + let rewritten_push = push.replace("origin.example.com", "test.example.com"); + for (source, expected) in [ + ( + format!("{fake}{push};/* self.__next_f.push( */"), + format!("{fake}{rewritten_push};/* self.__next_f.push( */"), + ), + ( + format!( + "const out=`text self.__next_f.push( ${{ {{a: {push}, b: `nested ${{{push}}}`}} }} tail`;" + ), + format!( + "const out=`text self.__next_f.push( ${{ {{a: {rewritten_push}, b: `nested ${{{rewritten_push}}}`}} }} tail`;" + ), + ), + ( + format!( + r#"const re=/["'`/]/;if (ready) /["'`/]/.test(input);const ratio=i++ / 2;const value=obj.return / 2;{push};"# + ), + format!( + r#"const re=/["'`/]/;if (ready) /["'`/]/.test(input);const ratio=i++ / 2;const value=obj.return / 2;{rewritten_push};"# + ), + ), + ] { + for split in 0..=source.len() { + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let first = format!("", &source[split..]); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept script prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should complete executable Flight"), + ); + assert_eq!( + String::from_utf8(bytes).expect("should emit UTF-8"), + format!(""), + "should rewrite only executable calls at split {split}" + ); + } + } + } + #[test] fn mixed_next_data_flight_mentions_do_not_claim_json_or_hide_gtm() { for text in [ diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs index 9b1c78311..7abe3aa7c 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs @@ -13,6 +13,7 @@ use super::rsc_stream::{ CapturedPayload, FragmentCapture, MAX_UNRESOLVED_RSC_PAYLOADS, RscGroupStatus, capture_fragment, classify_rsc_group, document_state, rsc_payload_placeholder, }; +use super::script_lexer::ScriptLexer; use super::shared::{ RSC_PUSH_CALL_PATTERN, RSC_PUSH_CALL_PATTERN_TRIMMED, RSC_RECEIVER_CONTEXT_BYTES, find_rsc_push_payload_range, find_trimmed_rsc_push_payload_range, @@ -62,7 +63,8 @@ impl NextJsRscPlaceholderRewriter { limit: usize, max_queued_payload_bytes: usize, ) -> ScriptRewriteAction { - if !content.contains("__next_f") { + let code = state.rsc_lexical_start.clone().mask(content); + if !code.contains("__next_f") { return if was_buffered { ScriptRewriteAction::replace(content.to_owned()) } else { @@ -76,12 +78,13 @@ impl NextJsRscPlaceholderRewriter { let mut cursor = 0; let mut trimmed = std::mem::take(&mut state.rsc_receiver_trimmed); let mut queued_bytes = state.captured_payload_bytes; - while content[cursor..].contains("__next_f") { + while code[cursor..].contains("__next_f") { let remaining = &content[cursor..]; + let remaining_code = &code[cursor..]; let head = if trimmed { - TRIMMED_FLIGHT_PUSH_PATTERN.find(remaining) + TRIMMED_FLIGHT_PUSH_PATTERN.find(remaining_code) } else { - first_qualified_flight_push(remaining) + first_qualified_flight_push(remaining_code) }; if let Some(head) = head && let Some(arguments) = @@ -91,20 +94,17 @@ impl NextJsRscPlaceholderRewriter { trimmed = false; continue; } - let range = if trimmed { - find_trimmed_rsc_push_payload_range(remaining) - } else { - find_rsc_push_payload_range(remaining) - }; + let range = executable_payload_range(remaining, remaining_code, trimmed); let call = if trimmed { - RSC_PUSH_CALL_PATTERN_TRIMMED.find(remaining) + RSC_PUSH_CALL_PATTERN_TRIMMED.find(remaining_code) } else { - RSC_PUSH_CALL_PATTERN.find(remaining) + RSC_PUSH_CALL_PATTERN.find(remaining_code) }; trimmed = false; let Some((start, end)) = range.filter(|(start, _)| { call.is_some_and(|call| { - call.end() == *start && !contains_qualified_flight(&remaining[..call.start()]) + call.end() == *start + && !contains_qualified_flight(&remaining_code[..call.start()]) }) }) else { // A raw qualified push cannot safely continue a pending T group @@ -198,14 +198,13 @@ impl NextJsRscPlaceholderRewriter { return ScriptRewriteAction::Keep; } if is_last && content.len() > limit && content.contains("__next_f") { + let code = state.rsc_lexical_start.clone().mask(content); let mut remaining = content; + let mut remaining_code = code.as_str(); let mut receiver_trimmed = trimmed; let unsafe_continuation = loop { - let range = if receiver_trimmed { - find_trimmed_rsc_push_payload_range(remaining) - } else { - find_rsc_push_payload_range(remaining) - }; + let range = + executable_payload_range(remaining, remaining_code, receiver_trimmed); let Some((start, end)) = range else { break true; }; @@ -219,8 +218,9 @@ impl NextJsRscPlaceholderRewriter { // payload is not classified as another call. Only the // first receiver can have streamed in an earlier fragment. remaining = &remaining[end + 1..]; + remaining_code = &remaining_code[end + 1..]; receiver_trimmed = false; - if !remaining.contains("__next_f") { + if !remaining_code.contains("__next_f") { break false; } }; @@ -237,6 +237,8 @@ impl NextJsRscPlaceholderRewriter { fn rewrite_fragment( &self, content: &str, + code: &str, + lexical_start: ScriptLexer, ctx: &IntegrationScriptContext<'_>, state: &mut super::rsc_stream::NextJsDocumentState, ) -> ScriptRewriteAction { @@ -272,15 +274,15 @@ impl NextJsRscPlaceholderRewriter { ); } - if state.rsc_probe.is_empty() && !content.contains("__next_f") { + if state.rsc_probe.is_empty() && !code.contains("__next_f") { if ctx.is_last_in_text_node { state.rsc_receiver_context.clear(); return ScriptRewriteAction::Keep; } - let probe_length = longest_identifier_prefix(content.as_bytes()); + let probe_length = longest_identifier_prefix(code.as_bytes()); let ready_length = content.len() - probe_length; state.rsc_probe.push_str(&content[ready_length..]); - remember_released(&mut state.rsc_receiver_context, &content[..ready_length]); + remember_released(&mut state.rsc_receiver_context, &code[..ready_length]); return if probe_length == 0 { ScriptRewriteAction::Keep } else if ready_length == 0 { @@ -293,7 +295,8 @@ impl NextJsRscPlaceholderRewriter { let prior_probe = std::mem::take(&mut state.rsc_probe); let mut combined = prior_probe.clone(); combined.push_str(content); - if !combined.contains("__next_f") { + let combined_code = format!("{prior_probe}{code}"); + if !combined_code.contains("__next_f") { if ctx.is_last_in_text_node { state.rsc_receiver_context.clear(); return if prior_probe.is_empty() { @@ -302,10 +305,13 @@ impl NextJsRscPlaceholderRewriter { ScriptRewriteAction::replace(combined) }; } - let probe_length = longest_identifier_prefix(combined.as_bytes()); + let probe_length = longest_identifier_prefix(combined_code.as_bytes()); let ready_length = combined.len() - probe_length; state.rsc_probe.push_str(&combined[ready_length..]); - remember_released(&mut state.rsc_receiver_context, &combined[..ready_length]); + remember_released( + &mut state.rsc_receiver_context, + &combined_code[..ready_length], + ); if prior_probe.is_empty() && probe_length == 0 { return ScriptRewriteAction::Keep; } @@ -316,18 +322,18 @@ impl NextJsRscPlaceholderRewriter { }; } - let identifier_start = combined + let identifier_start = combined_code .find("__next_f") .expect("should find the identifier that selected this branch"); let mut context = state.rsc_receiver_context.clone(); - context.push_str(&combined[..identifier_start]); + context.push_str(&combined_code[..identifier_start]); if !receiver_context_is_flight_push(&context) { // Some other object owns a `__next_f` property. Release the text // unchanged rather than claiming an unrelated publisher script. if ctx.is_last_in_text_node { state.rsc_receiver_context.clear(); } else { - remember_released(&mut state.rsc_receiver_context, &combined); + remember_released(&mut state.rsc_receiver_context, &combined_code); } return if prior_probe.is_empty() { ScriptRewriteAction::Keep @@ -340,7 +346,7 @@ impl NextJsRscPlaceholderRewriter { // one that already streamed leaves a trimmed claim whose receiver this // verified context stands in for. state.rsc_receiver_trimmed = - !receiver_context_is_flight_push(&combined[..identifier_start]); + !receiver_context_is_flight_push(&combined_code[..identifier_start]); state.rsc_receiver_context.clear(); let claimed_start = if state.rsc_receiver_trimmed { identifier_start @@ -349,6 +355,8 @@ impl NextJsRscPlaceholderRewriter { }; let prefix = &combined[..claimed_start]; let claimed = &combined[claimed_start..]; + state.rsc_lexical_start = lexical_start; + state.rsc_lexical_start.mask(&combined[..claimed_start]); let action = self.rewrite_claimed_fragment( claimed, ctx.is_last_in_text_node, @@ -389,13 +397,23 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { state.current_fragment_protected = false; state.flight_node_owned = false; state.flight_qualifier_tail.clear(); + state.flight_lexical = ScriptLexer::default(); return ScriptRewriteAction::Keep; } if !self.config.enabled || self.config.rewrite_attributes.is_empty() { state.current_fragment_protected = false; return ScriptRewriteAction::Keep; } - observe_flight_qualifier(&mut state, content); + let lexical_start = state.flight_lexical.clone(); + let code = state.flight_lexical.mask(content); + if state.flight_lexical.is_opaque() { + // At the lexical nesting limit, retain the existing conservative raw + // qualification. Ordinary scripts without a push do not bypass Flight. + observe_flight_qualifier(&mut state, content); + state.bypass_rsc |= state.flight_node_owned; + } else { + observe_flight_qualifier(&mut state, &code); + } // A call head can complete after a tentative property probe overflowed. // Once it proves Flight, preserve its entire raw continuation/group. if state.flight_node_owned @@ -407,18 +425,35 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { state.bypass_rsc = true; } state.current_fragment_protected = state.flight_node_owned; - let action = self.rewrite_fragment(content, ctx, &mut state); + let action = self.rewrite_fragment(content, &code, lexical_start, ctx, &mut state); if ctx.is_last_in_text_node { state.flight_node_owned = false; state.flight_qualifier_tail.clear(); state.rsc_probe.clear(); state.rsc_receiver_context.clear(); state.rsc_receiver_trimmed = false; + state.flight_lexical = ScriptLexer::default(); + state.rsc_lexical_start = ScriptLexer::default(); } action } } +/// Match a call head only in executable source, then read its original string. +fn executable_payload_range(content: &str, code: &str, trimmed: bool) -> Option<(usize, usize)> { + if trimmed { + RSC_PUSH_CALL_PATTERN_TRIMMED.find(code)?; + return find_trimmed_rsc_push_payload_range(content); + } + let call = RSC_PUSH_CALL_PATTERN.find(code)?; + let identifier = call.start() + call.as_str().find("__next_f")?; + if !receiver_context_is_flight_push(&code[..identifier]) { + return None; + } + let (start, end) = find_rsc_push_payload_range(&content[call.start()..])?; + Some((call.start() + start, call.start() + end)) +} + fn contains_qualified_flight(source: &str) -> bool { first_qualified_flight_push(source).is_some() } @@ -522,6 +557,16 @@ mod tests { }) } + #[test] + fn quoted_flight_spellings_do_not_claim_ownership() { + let state = IntegrationDocumentState::default(); + let rewriter = NextJsRscPlaceholderRewriter::new(test_config()); + rewriter.rewrite(r#"const demo="self.__next_f.push(";"#, &ctx(false, &state)); + let shared = document_state(&state); + let state = shared.lock().expect("should lock state"); + assert!(!state.flight_node_owned, "quoted text must not own Flight"); + } + #[test] fn unrecognized_qualified_prefix_makes_the_whole_batch_raw_and_protected() { let state = IntegrationDocumentState::default(); diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs index aa3262b37..22812c707 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs @@ -11,6 +11,7 @@ use super::rsc::{ DEFAULT_MAX_COMBINED_PAYLOAD_BYTES, PendingTChunk, TChunkStep, next_tchunk, rewrite_rsc_scripts_combined_with_transform, }; +use super::script_lexer::ScriptLexer; use super::shared::RscUrlRewriter; use super::{NEXTJS_INTEGRATION_ID, NextJsIntegrationConfig}; @@ -58,6 +59,10 @@ pub(super) struct NextJsDocumentState { pub(super) current_fragment_protected: bool, /// Bounded qualifier observation remains active even after document bypass. pub(super) flight_qualifier_tail: String, + /// Lexical context advances with original parser fragments, never replacements. + pub(super) flight_lexical: ScriptLexer, + /// Snapshot at the beginning of a claimed buffer for source-offset matching. + pub(super) rsc_lexical_start: ScriptLexer, } impl Default for NextJsDocumentState { @@ -77,6 +82,8 @@ impl Default for NextJsDocumentState { flight_node_owned: false, current_fragment_protected: false, flight_qualifier_tail: String::new(), + flight_lexical: ScriptLexer::default(), + rsc_lexical_start: ScriptLexer::default(), } } } diff --git a/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs new file mode 100644 index 000000000..706144868 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs @@ -0,0 +1,318 @@ +//! Bounded lexical context for recognizing executable Flight call heads. +//! +//! This is not a JavaScript parser. It masks strings, comments and template text +//! without changing byte offsets, and exposes code inside template expressions. + +const MAX_TEMPLATE_DEPTH: usize = 32; + +#[derive(Clone, Debug, Default)] +enum Mode { + #[default] + Code, + Slash { + regex: bool, + }, + Regex { + escaped: bool, + class: bool, + }, + String { + quote: char, + escaped: bool, + }, + LineComment, + BlockComment { + star: bool, + }, + Template { + escaped: bool, + dollar: bool, + }, + Opaque, +} + +#[derive(Clone, Debug)] +pub(super) struct ScriptLexer { + mode: Mode, + template_braces: Vec, + expression_start: bool, + word: String, + word_overflow: bool, + word_property: bool, + after_dot: bool, + control_pending: bool, + parentheses: Vec, + operator: Option<(char, bool)>, +} + +impl Default for ScriptLexer { + fn default() -> Self { + Self { + mode: Mode::Code, + template_braces: Vec::new(), + expression_start: true, + word: String::new(), + word_overflow: false, + word_property: false, + after_dot: false, + control_pending: false, + parentheses: Vec::new(), + operator: None, + } + } +} + +impl ScriptLexer { + /// Mask non-code bytes while preserving source offsets and opening quotes. + /// Opening quotes remain visible so the Flight payload regex can match them. + pub(super) fn mask(&mut self, source: &str) -> String { + let mut code = String::with_capacity(source.len()); + for character in source.chars() { + if self.is_code(character) { + code.push(character); + } else { + // A non-whitespace sentinel prevents matching across inert text. + code.extend(std::iter::repeat_n('#', character.len_utf8())); + } + } + code + } + + /// Excessively nested templates are unsupported rather than unbounded. + pub(super) fn is_opaque(&self) -> bool { + matches!(self.mode, Mode::Opaque) + } + + fn is_code(&mut self, character: char) -> bool { + match &mut self.mode { + Mode::Code => { + if character.is_alphanumeric() || matches!(character, '_' | '$') { + if self.word.is_empty() { + self.word_property = self.after_dot; + self.after_dot = false; + } + self.operator = None; + if self.word.len() < 10 && !self.word_overflow { + self.word.push(character); + } else { + self.word_overflow = true; + } + self.expression_start = false; + return true; + } + if !self.word.is_empty() { + self.expression_start = !self.word_overflow + && !self.word_property + && matches!( + self.word.as_str(), + "return" + | "throw" + | "case" + | "delete" + | "void" + | "typeof" + | "new" + | "yield" + | "await" + | "else" + | "do" + ); + self.control_pending = !self.word_overflow + && !self.word_property + && matches!( + self.word.as_str(), + "if" | "while" | "for" | "with" | "switch" | "catch" + ); + self.word.clear(); + self.word_overflow = false; + } + let prior_operator = self.operator.take(); + if !character.is_whitespace() && character != '/' { + self.after_dot = character == '.'; + } + match character { + '(' => { + if self.parentheses.len() == MAX_TEMPLATE_DEPTH { + self.mode = Mode::Opaque; + } else { + self.parentheses.push(self.control_pending); + } + self.control_pending = false; + self.expression_start = true; + } + ')' => self.expression_start = self.parentheses.pop().unwrap_or(false), + '+' | '-' => { + if let Some((operator, prior_expression)) = prior_operator + && operator == character + { + self.expression_start = prior_expression; + } else { + self.operator = Some((character, self.expression_start)); + self.expression_start = true; + } + } + '\'' | '"' => { + self.mode = Mode::String { + quote: character, + escaped: false, + }; + } + '/' => { + self.mode = Mode::Slash { + regex: self.expression_start, + } + } + '`' => { + if self.template_braces.len() == MAX_TEMPLATE_DEPTH { + self.mode = Mode::Opaque; + } else { + self.template_braces.push(0); + self.mode = Mode::Template { + escaped: false, + dollar: false, + }; + } + } + '{' => { + self.expression_start = true; + if let Some(braces) = self.template_braces.last_mut() { + *braces = braces.saturating_add(1); + } + } + '}' => { + if let Some(braces) = self.template_braces.last_mut() { + if *braces == 0 { + self.mode = Mode::Template { + escaped: false, + dollar: false, + }; + } else { + *braces -= 1; + } + } + } + _ => { + if !character.is_whitespace() { + self.expression_start = !matches!(character, ')' | ']' | '.'); + } + } + } + true + } + Mode::Slash { regex } => { + match character { + '/' => self.mode = Mode::LineComment, + '*' => self.mode = Mode::BlockComment { star: false }, + _ if *regex => { + self.mode = Mode::Regex { + escaped: false, + class: false, + }; + return self.is_code(character); + } + _ => { + self.mode = Mode::Code; + self.expression_start = true; + self.after_dot = false; + return self.is_code(character); + } + } + false + } + Mode::Regex { escaped, class } => { + if *escaped { + *escaped = false; + } else if character == '\\' { + *escaped = true; + } else if character == '[' { + *class = true; + } else if character == ']' { + *class = false; + } else if character == '/' && !*class { + self.mode = Mode::Code; + self.expression_start = false; + } + false + } + Mode::String { quote, escaped } => { + if *escaped { + *escaped = false; + } else if character == '\\' { + *escaped = true; + } else if character == *quote { + self.mode = Mode::Code; + self.expression_start = false; + } + false + } + Mode::LineComment => { + if matches!(character, '\n' | '\r' | '\u{2028}' | '\u{2029}') { + self.mode = Mode::Code; + } + false + } + Mode::BlockComment { star } => { + if *star && character == '/' { + self.mode = Mode::Code; + } else { + *star = character == '*'; + } + false + } + Mode::Template { escaped, dollar } => { + if *escaped { + *escaped = false; + *dollar = false; + } else if character == '\\' { + *escaped = true; + *dollar = false; + } else if character == '`' { + self.template_braces.pop(); + self.mode = Mode::Code; + self.expression_start = false; + } else if *dollar && character == '{' { + self.mode = Mode::Code; + self.expression_start = true; + } else { + *dollar = character == '$'; + } + false + } + Mode::Opaque => false, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn masks_inert_text_at_every_fragment_boundary() { + let source = r#"'é self.__next_f.push('/* self.__next_f.push( */`😀 ${ {nested: `self.__next_f.push(`} } ${self.__next_f.push([1,"data"])} `"#; + let expected = ScriptLexer::default().mask(source); + assert_eq!(expected.len(), source.len(), "should preserve byte offsets"); + assert_eq!(expected.matches("self.__next_f.push(").count(), 1); + for split in (0..=source.len()).filter(|split| source.is_char_boundary(*split)) { + let mut lexer = ScriptLexer::default(); + let actual = format!( + "{}{}", + lexer.mask(&source[..split]), + lexer.mask(&source[split..]) + ); + assert_eq!( + actual, expected, + "should preserve lexical state at split {split}" + ); + } + } + + #[test] + fn bounds_nested_templates() { + let mut lexer = ScriptLexer::default(); + lexer.mask(&"`${".repeat(MAX_TEMPLATE_DEPTH + 1)); + assert!(lexer.is_opaque(), "should refuse excessive nesting"); + assert_eq!(lexer.template_braces.len(), MAX_TEMPLATE_DEPTH); + assert!(!lexer.mask("self.__next_f.push(").contains("__next_f")); + } +} From 4811d98edd316294f7e87cfd909119cf4434c87f Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Thu, 1 Oct 2026 09:47:25 +0530 Subject: [PATCH 113/230] Clarify trace failure and export cleanup contracts --- ...9-01-mobile-ad-render-trace-endpoint-design.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index e12d79760..bc6387914 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -252,6 +252,15 @@ second GPT attribution engine. The existing recorder emits an exact-token cycle. The server-auction, correlation, and GPT projections are separate sibling contracts in `TraceReportV1`; none is treated as a substitute for another. +The deferred object-URL cleanup in section 12.3 deliberately diverges from TS +Console's existing export, which revokes synchronously in a `finally` block +immediately after `anchor.click()` +(`crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts`, +reachable through the console's `export()` action). That older path keeps the +truncation risk this design avoids. Version one does not change it; correcting +it belongs to TS Console under #1081, and until then the two download paths +intentionally differ. + ## 6. User experience ### 6.1 First visit: no captured report @@ -1513,6 +1522,12 @@ shell or actions. before trace handling, including on disabled routes. - Disabled route after authentication: local privacy-safe `404`. - Unsupported method: local `405`; never publisher fallback. +- Reserved-namespace path that is a trailing slash, extra segment, unsupported + asset name, repeated separator, or lookalike: local `404`; an encoded + separator or ambiguous dot segment: local `400`. Never publisher fallback. +- Non-empty or unreadable activation/end body: local `413` for any body bytes, + positive/invalid `Content-Length`, or `Transfer-Encoding`, and local `400` + for a stream read error, both with no cookie mutation. - Rejected activation/end POST: local `403` with no state mutation. - Optional platform fact unavailable: omit the field and continue. - Bounded cookie inspection failure: report the contract-defined invalid or From e6904260ab48917114d9b134bc7f5fbcfe826f6f Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 1 Oct 2026 14:49:04 +0530 Subject: [PATCH 114/230] Remove inherited x-ts-version and tighten local git version resolution Remove an existing x-ts-version when the compiled-in version is unknown or invalid, so a proxied origin header is never reported as the deployed version. Ignore git metadata unless the repository top level is the workspace root, and watch the nearest existing ancestor of a packed current-branch ref so the next commit re-runs the build script. Signed-off-by: dhruv8sh --- crates/trusted-server-core/build.rs | 26 ++++++++++- .../trusted-server-core/src/version_header.rs | 46 ++++++++++++++++++- .../2026-09-25-git-version-header-design.md | 11 ++++- 3 files changed, 77 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-core/build.rs b/crates/trusted-server-core/build.rs index c02a88d1e..e762f66a6 100644 --- a/crates/trusted-server-core/build.rs +++ b/crates/trusted-server-core/build.rs @@ -32,19 +32,41 @@ fn rerun_if_exists(path: &Path) { } /// Resolves the version from local git, watching the paths that move with it. +/// +/// `None` unless git's top level is this workspace, so an exported tree nested +/// in an unrelated repository does not report that repository's version. fn resolve_from_local_git() -> Option { + let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").ok()?; + // This crate lives at `/crates/trusted-server-core`. + let workspace_root = Path::new(&manifest_dir).parent()?.parent()?; + let repository_root = git(&["rev-parse", "--show-toplevel"])?; + if Path::new(&repository_root).canonicalize().ok()? != workspace_root.canonicalize().ok()? { + return None; + } + // HEAD is per-worktree; refs are shared in the common dir. They differ in a // linked worktree and coincide in a plain clone. Only the current branch's // ref (its commit decides the exact-tag match) and tags can change the // result, so other branches and `refs/remotes` are not watched: a commit - // elsewhere or a fetch must not rebuild core. + // elsewhere or a fetch must not rebuild core. When the branch ref is only + // in `packed-refs`, its nearest existing ancestor (usually `refs/heads`) is + // watched instead, so the loose ref the next commit writes is noticed. if let Some(git_dir) = git(&["rev-parse", "--absolute-git-dir"]) { rerun_if_exists(&Path::new(&git_dir).join("HEAD")); } if let Some(common_dir) = git(&["rev-parse", "--path-format=absolute", "--git-common-dir"]) { let common_dir = Path::new(&common_dir); if let Some(branch_ref) = git(&["symbolic-ref", "-q", "HEAD"]) { - rerun_if_exists(&common_dir.join(branch_ref)); + let branch_path = common_dir.join(branch_ref); + // Stop below the common dir: watching all of it would rebuild core + // on every object write. + if let Some(watched_path) = branch_path + .ancestors() + .take_while(|path| *path != common_dir) + .find(|path| path.exists()) + { + rerun_if_exists(watched_path); + } } rerun_if_exists(&common_dir.join("refs/tags")); rerun_if_exists(&common_dir.join("packed-refs")); diff --git a/crates/trusted-server-core/src/version_header.rs b/crates/trusted-server-core/src/version_header.rs index f2587cbb8..658c31f68 100644 --- a/crates/trusted-server-core/src/version_header.rs +++ b/crates/trusted-server-core/src/version_header.rs @@ -31,11 +31,13 @@ pub fn apply_git_version_header(response: &mut Response) { apply_git_version_header_from(TS_GIT_VERSION, response); } -/// Sets `x-ts-version` to `version`, or leaves it unset when `version` is -/// unknown or invalid. +/// Sets `x-ts-version` to `version`, or removes it when `version` is unknown +/// or invalid, so an origin's `x-ts-version` is never reported as ours. pub fn apply_git_version_header_from(version: Option<&str>, response: &mut Response) { if let Some(value) = header_value_from(version) { response.headers_mut().insert(HEADER_X_TS_VERSION, value); + } else { + response.headers_mut().remove(HEADER_X_TS_VERSION); } } @@ -51,6 +53,13 @@ mod tests { .expect("should build empty test response") } + fn response_with_upstream_version() -> Response { + response_builder() + .header(HEADER_X_TS_VERSION, "upstream-v1") + .body(Body::empty()) + .expect("should build test response with an upstream x-ts-version") + } + fn version_of(response: &Response) -> Option<&str> { response .headers() @@ -121,6 +130,39 @@ mod tests { ); } + #[test] + fn replaces_upstream_header_with_version() { + let mut response = response_with_upstream_version(); + apply_git_version_header_from(Some("v2.0.0"), &mut response); + assert_eq!( + version_of(&response), + Some("v2.0.0"), + "should replace an upstream x-ts-version with ours" + ); + } + + #[test] + fn removes_upstream_header_when_version_unknown() { + let mut response = response_with_upstream_version(); + apply_git_version_header_from(None, &mut response); + assert_eq!( + version_of(&response), + None, + "should remove an upstream x-ts-version when ours is unknown" + ); + } + + #[test] + fn removes_upstream_header_when_version_invalid() { + let mut response = response_with_upstream_version(); + apply_git_version_header_from(Some("bad\nvalue"), &mut response); + assert_eq!( + version_of(&response), + None, + "should remove an upstream x-ts-version when ours is invalid" + ); + } + #[test] fn default_uses_compiled_in_version() { let mut response = empty_response(); diff --git a/docs/superpowers/specs/2026-09-25-git-version-header-design.md b/docs/superpowers/specs/2026-09-25-git-version-header-design.md index 67678e4d1..f8e09dd83 100644 --- a/docs/superpowers/specs/2026-09-25-git-version-header-design.md +++ b/docs/superpowers/specs/2026-09-25-git-version-header-design.md @@ -178,7 +178,7 @@ Only paths that exist are printed. Cargo treats a missing `rerun-if-changed` path as changed, so printing an absent `packed-refs` would re-run the script and recompile core on every build. The accepted cost: a `packed-refs` file created after the last run is not noticed until something else triggers a -re-run. Watching the branch ref and `refs/tags` still catches new loose refs. Likewise, if the current branch's ref exists only in `packed-refs` (after `git pack-refs`), the loose ref written by the next commit is not watched. The tag-to-branch switch after committing on a tagged commit then waits for the next re-run. +re-run. Watching the branch ref and `refs/tags` still catches new loose refs. If the current branch's ref exists only in `packed-refs` (after `git pack-refs`), its nearest existing ancestor below the common dir (usually `refs/heads`) is watched instead, so the loose ref written by the next commit still re-runs the script. The tradeoff: until that loose ref exists, commits on sibling branches also re-run it. ### 4. Headers @@ -241,6 +241,11 @@ x-ts-version: v1.3.0 `build.rs` never fails the build over version metadata. If git is missing, the directory is not a repository (for example an exported source tree), or git fails, the build leaves `TS_GIT_VERSION` unset and `x-ts-version` is omitted. +The same applies when git's top level (`git rev-parse --show-toplevel`) is not +the workspace root: git searches parent directories, so an exported tree nested +in an unrelated repository would otherwise report that repository's tag or +branch. When the version is unknown, an `x-ts-version` already on the response +(for example from a proxied origin) is removed rather than passed through. Sending a placeholder such as `unknown` was rejected: a missing header is easier to tell apart from a real ref. @@ -294,7 +299,9 @@ Unit and integration tests follow red-green-refactor and cover: (`v1-été`) and one with an inner space are skipped in favor of local git; `None` when nothing is known; - `version_header`: value from a version; `None` when unknown; `None` for an - invalid value; the header is set, omitted, or skipped accordingly; the + invalid value; the header is set, omitted, or skipped accordingly; an + upstream `x-ts-version` is replaced by a valid version and removed when the + version is unknown or invalid; the default path reports the compiled-in `TS_GIT_VERSION`; - Fastly `apply_finalize_headers`: `x-ts-version` equals `TS_GIT_VERSION`, and `x-ts-fastly-version` equals `FASTLY_SERVICE_VERSION` when set; From 12f1bd9f9dbb106e9db013c7a7f86873b7ec4dca Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 1 Oct 2026 18:03:38 +0530 Subject: [PATCH 115/230] Add documentation proposal automation design and plan Record the design and implementation plan for issue #1105. Signed-off-by: dhruv8sh --- .../2026-10-01-docs-proposal-automation.md | 1500 +++++++++++++++++ ...6-10-01-docs-proposal-automation-design.md | 165 ++ 2 files changed, 1665 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-01-docs-proposal-automation.md create mode 100644 docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md diff --git a/docs/superpowers/plans/2026-10-01-docs-proposal-automation.md b/docs/superpowers/plans/2026-10-01-docs-proposal-automation.md new file mode 100644 index 000000000..a6d67b96d --- /dev/null +++ b/docs/superpowers/plans/2026-10-01-docs-proposal-automation.md @@ -0,0 +1,1500 @@ +# Documentation Proposal Automation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** After a code merge to `main`, run GitHub Copilot CLI to propose +documentation edits, and open one pull request per merge whose hunks each carry +a review comment citing the merged source and a suggestion that reverts the +hunk. + +**Architecture:** A two-job workflow. `propose` (read-only token plus +`copilot-requests: write`) runs Copilot and uploads a patch, rationale, and +per-hunk evidence. `publish` (write token, no Copilot) applies the patch, +enforces a path allowlist, runs the docs gates, pushes `docs/auto/`, +opens or edits the pull request, and posts a review built deterministically +from the diff by a dependency-free Node script. + +**Tech Stack:** GitHub Actions, Copilot CLI (`@github/copilot@1.0.90`), Bash, +Node 24 (`node:test`, no npm dependencies), `gh` CLI, VitePress docs gates. + +**Spec:** `docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md` + +## Global Constraints + +- Copilot may only change `docs/guide/**` and `docs/index.md`; `publish.sh` + enforces it with `^docs/(guide/.+|index\.md)$`. +- Branch name: `docs/auto/`. +- One pull request per merge SHA; a closed or merged proposal is never + reopened or recreated. +- Empty diff: no pull request; an open proposal for that SHA is closed. +- Docs gates before any push: `npm ci`, `npm run lint`, `npm run format`, + `npm run build` in `docs/`. +- Copilot never holds a token with `contents: write` or `pull-requests: write`; + checkout in `propose` uses `persist-credentials: false`. +- Action references match the repository's existing tags (`actions/checkout@v4`, + `actions/setup-node@v4`, `actions/upload-artifact@v4`, + `actions/download-artifact@v4`). Workflow steps are single-line; logic lives + in `scripts/docs-proposal/`. +- Merge SHAs are full 40-character lowercase hex and must be reachable from + `origin/main`. +- Comments above code, never inline. Fictional `example.com` values only. +- Commit messages: sentence case, imperative, no prefixes; `git commit -S --signoff`. + +## File Map + +| File | Responsibility | +| -------------------------------------------------- | ------------------------------------------------------------ | +| `scripts/docs-proposal/hunks.mjs` | Parse a unified diff, list hunks with ids, build the review | +| `scripts/docs-proposal/hunks.test.mjs` | `node:test` coverage for `hunks.mjs` | +| `scripts/docs-proposal/lib.sh` | Shared Bash helpers: SHA validation, branch, allowlist, body | +| `scripts/docs-proposal/prompt.md` | Copilot instructions for proposing edits | +| `scripts/docs-proposal/evidence-prompt.md` | Copilot instructions for per-hunk evidence | +| `scripts/docs-proposal/propose.sh` | Job `propose` entry point | +| `scripts/docs-proposal/publish.sh` | Job `publish` entry point | +| `scripts/docs-proposal/test.sh` | Bash tests for `lib.sh` and `publish.sh` with stubbed `gh` | +| `.github/workflows/docs-proposal.yml` | The proposal workflow | +| `.github/workflows/format.yml` | Add a job that lints and tests the proposal scripts | +| `scripts/README.md`, `.gitignore`, the design spec | Prerequisites, ignored work dir, small spec corrections | + +--- + +### Task 1: Diff parser and hunk listing + +**Files:** + +- Create: `scripts/docs-proposal/hunks.mjs` +- Test: `scripts/docs-proposal/hunks.test.mjs` + +**Interfaces:** + +- Produces: + - `parseDiff(text: string): File[]` where + `File = { path: string, status: "modified"|"added"|"deleted", binary: boolean, hunks: Hunk[] }`, + `Hunk = { lines: Line[] }`, + `Line = { type: " "|"-"|"+", text: string, oldNo: number|null, newNo: number|null }`. + - `collectHunks(files: File[]): { id: string, path: string, status: string, hunk: Hunk }[]` + (binary files skipped; ids are `#<8 hex>`, with `-2`, `-3` … on collision). + - `listHunks(files: File[]): { id, path, status, removed: string[], added: string[] }[]`. + - CLI: `node hunks.mjs list < diff` prints `listHunks` JSON. + +- [ ] **Step 1: Write the failing tests** + +```js +// scripts/docs-proposal/hunks.test.mjs +import assert from 'node:assert/strict' +import { test } from 'node:test' + +import { collectHunks, listHunks, parseDiff } from './hunks.mjs' + +const MODIFIED = [ + 'diff --git a/docs/guide/cli.md b/docs/guide/cli.md', + 'index 1111111..2222222 100644', + '--- a/docs/guide/cli.md', + '+++ b/docs/guide/cli.md', + '@@ -10,4 +10,4 @@ heading', + ' keep one', + '-old flag', + '+new flag', + ' keep two', + '', +].join('\n') + +test('parseDiff numbers context, removed, and added lines', () => { + const [file] = parseDiff(MODIFIED) + assert.equal(file.path, 'docs/guide/cli.md', 'should take the b/ path') + assert.equal(file.status, 'modified', 'should default to modified') + assert.deepEqual( + file.hunks[0].lines, + [ + { type: ' ', text: 'keep one', oldNo: 10, newNo: 10 }, + { type: '-', text: 'old flag', oldNo: 11, newNo: null }, + { type: '+', text: 'new flag', oldNo: null, newNo: 11 }, + { type: ' ', text: 'keep two', oldNo: 12, newNo: 12 }, + ], + 'should track old and new line numbers' + ) +}) + +test('parseDiff records added, deleted, and binary files', () => { + const diff = [ + 'diff --git a/docs/guide/new.md b/docs/guide/new.md', + 'new file mode 100644', + '--- /dev/null', + '+++ b/docs/guide/new.md', + '@@ -0,0 +1 @@', + '+hello', + 'diff --git a/docs/guide/gone.md b/docs/guide/gone.md', + 'deleted file mode 100644', + '--- a/docs/guide/gone.md', + '+++ /dev/null', + '@@ -1 +0,0 @@', + '-bye', + 'diff --git a/docs/guide/logo.png b/docs/guide/logo.png', + 'Binary files a/docs/guide/logo.png and b/docs/guide/logo.png differ', + '', + ].join('\n') + const files = parseDiff(diff) + assert.deepEqual( + files.map(({ path, status, binary }) => ({ path, status, binary })), + [ + { path: 'docs/guide/new.md', status: 'added', binary: false }, + { path: 'docs/guide/gone.md', status: 'deleted', binary: false }, + { path: 'docs/guide/logo.png', status: 'modified', binary: true }, + ], + 'should classify each file' + ) + assert.deepEqual( + files[0].hunks[0].lines, + [{ type: '+', text: 'hello', oldNo: null, newNo: 1 }], + 'should start a new file at line 1' + ) +}) + +test('parseDiff skips no-newline markers', () => { + const diff = MODIFIED.replace( + ' keep two\n', + ' keep two\n\\ No newline at end of file\n' + ) + const [file] = parseDiff(diff) + assert.equal(file.hunks[0].lines.length, 4, 'should ignore the marker line') +}) + +test('collectHunks gives stable ids and skips binary files', () => { + const first = collectHunks(parseDiff(MODIFIED)) + const second = collectHunks( + parseDiff(MODIFIED.replace('@@ -10,4 +10,4', '@@ -20,4 +20,4')) + ) + assert.equal(first.length, 1, 'should collect the single hunk') + assert.match( + first[0].id, + /^docs\/guide\/cli\.md#[0-9a-f]{8}$/, + 'should use path#hash' + ) + assert.equal(first[0].id, second[0].id, 'should not depend on line numbers') +}) + +test('collectHunks suffixes duplicate ids', () => { + const twice = MODIFIED + MODIFIED.split('\n').slice(4).join('\n') + const ids = collectHunks(parseDiff(twice)).map((entry) => entry.id) + assert.equal(ids.length, 2, 'should collect both hunks') + assert.equal(ids[1], `${ids[0]}-2`, 'should suffix the repeated id') +}) + +test('listHunks exposes removed and added text', () => { + const [entry] = listHunks(parseDiff(MODIFIED)) + assert.deepEqual(entry.removed, ['old flag'], 'should list removed lines') + assert.deepEqual(entry.added, ['new flag'], 'should list added lines') + assert.equal(entry.status, 'modified', 'should carry the file status') +}) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `node --test scripts/docs-proposal/hunks.test.mjs` +Expected: FAIL with `Cannot find module` for `hunks.mjs`. + +- [ ] **Step 3: Write the implementation** + +```js +// scripts/docs-proposal/hunks.mjs +// Parses a unified documentation diff into hunks and builds the per-hunk +// review that cites Copilot evidence and offers a suggestion reverting each +// hunk. Dependency-free so it runs on any runner with Node. +import { createHash } from 'node:crypto' +import { readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' + +const FILE_HEADER = /^diff --git a\/(.+) b\/(.+)$/ +const HUNK_HEADER = /^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@/ + +export function parseDiff(text) { + const files = [] + let file = null + let hunk = null + let oldNo = 0 + let newNo = 0 + for (const raw of text.split('\n')) { + const fileHeader = FILE_HEADER.exec(raw) + if (fileHeader) { + file = { + path: fileHeader[2], + status: 'modified', + binary: false, + hunks: [], + } + files.push(file) + hunk = null + continue + } + if (!file) { + continue + } + const hunkHeader = HUNK_HEADER.exec(raw) + if (hunkHeader) { + oldNo = Number(hunkHeader[1]) + newNo = Number(hunkHeader[2]) + hunk = { lines: [] } + file.hunks.push(hunk) + continue + } + if (!hunk) { + if (raw.startsWith('new file mode')) { + file.status = 'added' + } else if (raw.startsWith('deleted file mode')) { + file.status = 'deleted' + } else if (raw.startsWith('Binary files ')) { + file.binary = true + } + continue + } + const type = raw[0] + const lineText = raw.slice(1) + if (type === ' ') { + hunk.lines.push({ type, text: lineText, oldNo: oldNo++, newNo: newNo++ }) + } else if (type === '-') { + hunk.lines.push({ type, text: lineText, oldNo: oldNo++, newNo: null }) + } else if (type === '+') { + hunk.lines.push({ type, text: lineText, oldNo: null, newNo: newNo++ }) + } + } + return files +} + +function hashHunk(path, hunk) { + const hash = createHash('sha256') + hash.update(path) + for (const line of hunk.lines) { + if (line.type !== ' ') { + hash.update(`\n${line.type}${line.text}`) + } + } + return `${path}#${hash.digest('hex').slice(0, 8)}` +} + +export function collectHunks(files) { + const seen = new Map() + const entries = [] + for (const file of files) { + if (file.binary) { + continue + } + for (const hunk of file.hunks) { + const base = hashHunk(file.path, hunk) + const count = (seen.get(base) ?? 0) + 1 + seen.set(base, count) + const id = count === 1 ? base : `${base}-${count}` + entries.push({ id, path: file.path, status: file.status, hunk }) + } + } + return entries +} + +export function listHunks(files) { + return collectHunks(files).map(({ id, path, status, hunk }) => ({ + id, + path, + status, + removed: hunk.lines + .filter((line) => line.type === '-') + .map((line) => line.text), + added: hunk.lines + .filter((line) => line.type === '+') + .map((line) => line.text), + })) +} + +function main([command]) { + const files = parseDiff(readFileSync(0, 'utf8')) + if (command === 'list') { + process.stdout.write(`${JSON.stringify(listHunks(files), null, 2)}\n`) + return + } + process.stderr.write( + 'usage: hunks.mjs list | review \n' + ) + process.exitCode = 2 +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + main(process.argv.slice(2)) +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `node --test scripts/docs-proposal/hunks.test.mjs` +Expected: PASS, 6 tests. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/docs-proposal/hunks.mjs scripts/docs-proposal/hunks.test.mjs +git commit -S --signoff -m "Add documentation diff hunk parser" +``` + +--- + +### Task 2: Per-hunk review builder + +**Files:** + +- Modify: `scripts/docs-proposal/hunks.mjs` +- Test: `scripts/docs-proposal/hunks.test.mjs` + +**Interfaces:** + +- Consumes: `parseDiff`, `collectHunks` from Task 1. +- Produces: + - `revertSuggestion(hunk: Hunk): { startLine: number, line: number, body: string[] } | null`. + - `indexEvidence(raw: unknown): Map`. + - `buildReview(files: File[], evidence: Map, commitId: string): { commit_id, event: "COMMENT", body: string, comments: object[] }`. + - CLI: `node hunks.mjs review < diff` prints the + review payload for `POST /repos/{owner}/{repo}/pulls/{number}/reviews`. A + missing or malformed evidence file is treated as no evidence. + +- [ ] **Step 1: Write the failing tests** (append to `hunks.test.mjs`, and + extend the import to + `import { buildReview, collectHunks, indexEvidence, listHunks, parseDiff, revertSuggestion } from "./hunks.mjs";`) + +`````js +function hunkOf(lines, oldStart = 1, newStart = 1) { + const diff = [ + 'diff --git a/docs/guide/a.md b/docs/guide/a.md', + '--- a/docs/guide/a.md', + '+++ b/docs/guide/a.md', + `@@ -${oldStart} +${newStart} @@`, + ...lines, + '', + ].join('\n') + return parseDiff(diff)[0].hunks[0] +} + +test('revertSuggestion restores replaced lines', () => { + const suggestion = revertSuggestion(hunkOf([' a', '-b', '+B', '+B2', ' c'])) + assert.deepEqual( + suggestion, + { startLine: 2, line: 3, body: ['b'] }, + 'should span the new lines and restore the old one' + ) +}) + +test('revertSuggestion deletes pure insertions', () => { + const suggestion = revertSuggestion(hunkOf([' a', '+new', ' c'])) + assert.deepEqual( + suggestion, + { startLine: 2, line: 2, body: [] }, + 'should suggest an empty body' + ) +}) + +test('revertSuggestion spans several change groups with their context', () => { + const suggestion = revertSuggestion(hunkOf(['-a', '+A', ' mid', '-c', '+C'])) + assert.deepEqual( + suggestion, + { startLine: 1, line: 3, body: ['a', 'mid', 'c'] }, + 'should keep the middle context line' + ) +}) + +test('revertSuggestion anchors pure deletions on the previous context line', () => { + const suggestion = revertSuggestion(hunkOf([' a', '-gone', ' c'], 4, 4)) + assert.deepEqual( + suggestion, + { startLine: 4, line: 4, body: ['a', 'gone'] }, + 'should restore the deleted line after its anchor' + ) +}) + +test('revertSuggestion anchors leading deletions on the next context line', () => { + const suggestion = revertSuggestion(hunkOf(['-gone', ' a'])) + assert.deepEqual( + suggestion, + { startLine: 1, line: 1, body: ['gone', 'a'] }, + 'should restore the deleted line before its anchor' + ) +}) + +test('revertSuggestion returns null without an anchor', () => { + assert.equal( + revertSuggestion(hunkOf(['-only'])), + null, + 'should not invent an anchor' + ) +}) + +test('indexEvidence keeps valid entries and sanitizes text', () => { + const evidence = indexEvidence({ + hunks: [ + { + id: 'x#1', + source: 'crates/a.rs:4', + reason: 'Adds `--flag`\nfor @team', + }, + { id: 7, source: 'ignored' }, + { id: 'x#2' }, + 'junk', + ], + }) + assert.deepEqual( + [...evidence], + [['x#1', { source: 'crates/a.rs:4', reason: 'Adds --flag for @​team' }]], + 'should drop invalid entries and strip backticks, newlines, and mentions' + ) + assert.equal(indexEvidence(null).size, 0, 'should accept missing evidence') +}) + +test('buildReview cites evidence and suggests reverts', () => { + const files = parseDiff( + [ + 'diff --git a/docs/guide/a.md b/docs/guide/a.md', + '--- a/docs/guide/a.md', + '+++ b/docs/guide/a.md', + '@@ -1,3 +1,3 @@', + ' a', + '-uses ```fence```', + '+B', + ' c', + '@@ -10 +10,2 @@', + ' j', + '+k', + '', + ].join('\n') + ) + const [first, second] = collectHunks(files) + const evidence = indexEvidence({ + hunks: [ + { id: first.id, source: 'crates/a.rs:9', reason: 'Renames the flag.' }, + ], + }) + const review = buildReview(files, evidence, 'abc123') + + assert.equal(review.commit_id, 'abc123', 'should pin the reviewed commit') + assert.equal(review.event, 'COMMENT', 'should not approve or request changes') + assert.equal(review.comments.length, 2, 'should comment on every hunk') + assert.deepEqual( + review.comments[0], + { + path: 'docs/guide/a.md', + line: 2, + side: 'RIGHT', + body: [ + '**Source:** `crates/a.rs:9` — Renames the flag.', + '', + 'Apply this suggestion to revert the hunk.', + '', + '````suggestion', + 'uses ```fence```', + '````', + ].join('\n'), + }, + 'should escalate the fence and omit start_line for one line' + ) + assert.match( + review.comments[1].body, + /none cited/, + 'should flag hunks without evidence' + ) + assert.match( + review.comments[1].body, + /```suggestion\n```$/, + 'should suggest deleting an insertion' + ) + assert.equal(second.id, collectHunks(files)[1].id, 'should be deterministic') +}) + +test('buildReview lists binary and unanchored changes in the body', () => { + const files = parseDiff( + [ + 'diff --git a/docs/guide/gone.md b/docs/guide/gone.md', + 'deleted file mode 100644', + '@@ -1 +0,0 @@', + '-bye', + 'diff --git a/docs/guide/logo.png b/docs/guide/logo.png', + 'Binary files a/docs/guide/logo.png and b/docs/guide/logo.png differ', + '', + ].join('\n') + ) + const review = buildReview(files, new Map(), 'abc123') + assert.equal(review.comments.length, 0, 'should not post line comments') + assert.match( + review.body, + /`docs\/guide\/gone\.md`: file deleted/, + 'should note the deletion' + ) + assert.match( + review.body, + /`docs\/guide\/logo\.png`: binary change/, + 'should note the binary file' + ) +}) + +test('buildReview adds start_line for multi-line spans', () => { + const files = parseDiff( + [ + 'diff --git a/docs/index.md b/docs/index.md', + '@@ -1 +1,2 @@', + '-a', + '+b', + '+c', + '', + ].join('\n') + ) + const [comment] = buildReview(files, new Map(), 'abc123').comments + assert.equal(comment.start_line, 1, 'should start at the first new line') + assert.equal( + comment.start_side, + 'RIGHT', + 'should anchor the start on the new side' + ) + assert.equal(comment.line, 2, 'should end at the last new line') +}) +````` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `node --test scripts/docs-proposal/hunks.test.mjs` +Expected: FAIL with `does not provide an export named 'buildReview'`. + +- [ ] **Step 3: Implement** — add to `hunks.mjs` after `listHunks`: + +```js +const MAX_EVIDENCE_LENGTH = 300 + +const REVIEW_INTRO = + 'Copilot evidence for each proposed hunk. Leave a hunk to accept it, or apply its suggestion to revert it. Verify every cited source before merging.' + +export function revertSuggestion(hunk) { + const { lines } = hunk + const first = lines.findIndex((line) => line.type !== ' ') + const last = lines.findLastIndex((line) => line.type !== ' ') + if (first === -1) { + return null + } + const span = lines.slice(first, last + 1) + const newSide = span.filter((line) => line.type !== '-') + const oldText = span + .filter((line) => line.type !== '+') + .map((line) => line.text) + if (newSide.length > 0) { + return { + startLine: newSide[0].newNo, + line: newSide.at(-1).newNo, + body: oldText, + } + } + const before = lines[first - 1] + if (before?.type === ' ') { + return { + startLine: before.newNo, + line: before.newNo, + body: [before.text, ...oldText], + } + } + const after = lines[last + 1] + if (after?.type === ' ') { + return { + startLine: after.newNo, + line: after.newNo, + body: [...oldText, after.text], + } + } + return null +} + +function sanitize(value) { + if (typeof value !== 'string') { + return '' + } + return value + .replace(/[`\r\n]+/g, ' ') + .replace(/@/g, '@​') + .replace(/\s+/g, ' ') + .trim() + .slice(0, MAX_EVIDENCE_LENGTH) +} + +export function indexEvidence(raw) { + const evidence = new Map() + const items = Array.isArray(raw?.hunks) ? raw.hunks : [] + for (const item of items) { + if (typeof item?.id !== 'string') { + continue + } + const source = sanitize(item.source) + const reason = sanitize(item.reason) + if (source || reason) { + evidence.set(item.id, { source, reason }) + } + } + return evidence +} + +function fenceFor(body) { + const longestRun = Math.max( + 0, + ...body.map((text) => + Math.max(0, ...(text.match(/`+/g) ?? []).map((run) => run.length)) + ) + ) + return '`'.repeat(Math.max(3, longestRun + 1)) +} + +function commentBody(cited, body) { + const source = cited?.source + ? `\`${cited.source}\`` + : 'none cited — verify this hunk against the merged code.' + const reason = cited?.reason ? ` — ${cited.reason}` : '' + const fence = fenceFor(body) + const suggestion = + body.length > 0 + ? `${fence}suggestion\n${body.join('\n')}\n${fence}` + : `${fence}suggestion\n${fence}` + return [ + `**Source:** ${source}${reason}`, + '', + 'Apply this suggestion to revert the hunk.', + '', + suggestion, + ].join('\n') +} + +export function buildReview(files, evidence, commitId) { + const comments = [] + const notes = [] + for (const file of files) { + if (file.binary) { + notes.push( + `- \`${file.path}\`: binary change — review the file directly.` + ) + } + } + for (const { id, path, status, hunk } of collectHunks(files)) { + const suggestion = revertSuggestion(hunk) + if (!suggestion) { + const reason = + status === 'deleted' ? 'file deleted' : 'no line to anchor a revert' + notes.push(`- \`${path}\`: ${reason} — revert it manually if needed.`) + continue + } + const comment = { + path, + line: suggestion.line, + side: 'RIGHT', + body: commentBody(evidence.get(id), suggestion.body), + } + if (suggestion.startLine !== suggestion.line) { + comment.start_line = suggestion.startLine + comment.start_side = 'RIGHT' + } + comments.push(comment) + } + const body = + notes.length > 0 + ? `${REVIEW_INTRO}\n\nChanges without a suggestion:\n\n${notes.join('\n')}` + : REVIEW_INTRO + return { commit_id: commitId, event: 'COMMENT', body, comments } +} + +function readEvidence(path) { + try { + return indexEvidence(JSON.parse(readFileSync(path, 'utf8'))) + } catch (error) { + process.stderr.write( + `Ignoring unreadable evidence at ${path}: ${error.message}\n` + ) + return new Map() + } +} +``` + +Note: the `comment` object key order in the test is `path, line, side, body`; +`deepEqual` ignores key order, so the construction order above is fine. + +Then extend `main`: + +```js +function main([command, evidencePath, commitId]) { + const files = parseDiff(readFileSync(0, 'utf8')) + if (command === 'list') { + process.stdout.write(`${JSON.stringify(listHunks(files), null, 2)}\n`) + return + } + if (command === 'review' && evidencePath && commitId) { + const review = buildReview(files, readEvidence(evidencePath), commitId) + process.stdout.write(`${JSON.stringify(review, null, 2)}\n`) + return + } + process.stderr.write( + 'usage: hunks.mjs list | review \n' + ) + process.exitCode = 2 +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `node --test scripts/docs-proposal/hunks.test.mjs` +Expected: PASS, 16 tests. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/docs-proposal/hunks.mjs scripts/docs-proposal/hunks.test.mjs +git commit -S --signoff -m "Build per-hunk revert reviews for documentation proposals" +``` + +--- + +### Task 3: Shared helpers, prompts, and `propose.sh` + +**Files:** + +- Create: `scripts/docs-proposal/lib.sh`, `scripts/docs-proposal/prompt.md`, + `scripts/docs-proposal/evidence-prompt.md`, `scripts/docs-proposal/propose.sh` +- Create: `scripts/docs-proposal/test.sh` (lib tests only in this task) +- Modify: `.gitignore` (add `/.docs-proposal/`) + +**Interfaces:** + +- Consumes: `node hunks.mjs list` from Task 1. +- Produces (sourced by `publish.sh` in Task 4): + - `docs_proposal_require_sha `: exits 1 unless full hex SHA reachable from `origin/main`. + - `docs_proposal_branch ` → prints `docs/auto/`. + - `docs_proposal_disallowed_paths` (stdin: paths) → prints bad paths, returns 1 if any. + - `docs_proposal_pr_body ` → prints body. + - `propose.sh ` writes `/proposal.patch` (possibly empty), + `/rationale.md`, `/evidence.json`. + +- [ ] **Step 1: Write the failing lib tests** + +```bash +#!/usr/bin/env bash +# Tests the documentation proposal helpers and publish flow with stubbed gh +# and npm against a throwaway git repository. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +failures=0 + +assert_eq() { + if [ "$1" != "$2" ]; then + printf 'FAIL: %s\n expected: %s\n actual: %s\n' "$3" "$2" "$1" + failures=$((failures + 1)) + fi +} + +assert_contains() { + if [[ "$1" != *"$2"* ]]; then + printf 'FAIL: %s\n missing: %s\n' "$3" "$2" + failures=$((failures + 1)) + fi +} + +sha=0123456789abcdef0123456789abcdef01234567 + +assert_eq "$(docs_proposal_branch "$sha")" "docs/auto/0123456789ab" "should derive the branch from the SHA" + +bad="$(printf 'docs/guide/a.md\ndocs/index.md\ndocs/superpowers/x.md\nREADME.md\n' | docs_proposal_disallowed_paths || true)" +assert_eq "$bad" $'docs/superpowers/x.md\nREADME.md' "should print only paths outside the allowlist" + +if printf 'docs/guide/a.md\ndocs/guide/integrations/b.md\n' | docs_proposal_disallowed_paths >/dev/null; then + status=0 +else + status=1 +fi +assert_eq "$status" 0 "should accept guide and nested guide paths" + +rationale="$(mktemp)" +printf -- '- docs/guide/cli.md: documents the new flag.\n' > "$rationale" +body="$(docs_proposal_pr_body "$sha" "Add example flag" 42 "$rationale")" +assert_contains "$body" "" "should embed the SHA marker" +assert_contains "$body" "merged change $sha (#42): Add example flag" "should link the merge and its PR" +assert_contains "$body" "documents the new flag" "should include the rationale" +: > "$rationale" +body="$(docs_proposal_pr_body "$sha" "Add example flag" "" "$rationale")" +assert_contains "$body" "merged change $sha: Add example flag" "should omit a missing origin PR" +assert_contains "$body" "did not provide a rationale" "should note an empty rationale" +rm -f "$rationale" + +if (docs_proposal_require_sha "not-a-sha") 2>/dev/null; then + status=0 +else + status=1 +fi +assert_eq "$status" 1 "should reject a malformed SHA" + +# PUBLISH_TESTS + +if [ "$failures" -gt 0 ]; then + printf '%d failure(s)\n' "$failures" + exit 1 +fi +printf 'All documentation proposal script tests passed.\n' +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `chmod +x scripts/docs-proposal/test.sh && scripts/docs-proposal/test.sh` +Expected: FAIL — `lib.sh: No such file or directory`. + +- [ ] **Step 3: Write `lib.sh`** + +```bash +# shellcheck shell=bash +# Shared helpers for the documentation proposal scripts. Source this file; +# do not execute it. + +DOCS_PROPOSAL_ALLOWED_RE='^docs/(guide/.+|index\.md)$' +DOCS_PROPOSAL_MAX_RATIONALE_BYTES=20000 + +# Exits unless the argument is a full commit SHA reachable from origin/main. +docs_proposal_require_sha() { + local sha="$1" + if ! [[ "$sha" =~ ^[0-9a-f]{40}$ ]]; then + printf '::error::Expected a full 40-character commit SHA, got %s\n' "$sha" >&2 + exit 1 + fi + if ! git merge-base --is-ancestor "$sha" origin/main 2>/dev/null; then + printf '::error::Commit %s is not reachable from origin/main\n' "$sha" >&2 + exit 1 + fi +} + +docs_proposal_branch() { + printf 'docs/auto/%s\n' "${1:0:12}" +} + +# Reads changed paths on stdin, prints those outside the allowlist, and +# returns 1 when any were found. +docs_proposal_disallowed_paths() { + local path + local found=0 + while IFS= read -r path; do + if [ -n "$path" ] && ! [[ "$path" =~ $DOCS_PROPOSAL_ALLOWED_RE ]]; then + printf '%s\n' "$path" + found=1 + fi + done + return "$found" +} + +docs_proposal_pr_body() { + local sha="$1" + local subject="$2" + local origin_pr="$3" + local rationale="$4" + local origin="" + if [ -n "$origin_pr" ]; then + origin=" (#$origin_pr)" + fi + printf '\n' "$sha" + printf 'Proposed documentation update for merged change %s%s: %s\n\n' "$sha" "$origin" "$subject" + printf 'Generated by GitHub Copilot CLI in the docs-proposal workflow. A maintainer must verify every hunk against the merged code before merging. Each hunk has a review comment citing its source and a suggestion that reverts it.\n\n' + printf '## Rationale\n\n' + if [ -s "$rationale" ]; then + head -c "$DOCS_PROPOSAL_MAX_RATIONALE_BYTES" "$rationale" + printf '\n' + else + printf '_Copilot did not provide a rationale._\n' + fi +} +``` + +- [ ] **Step 4: Run lib tests to verify they pass** + +Run: `scripts/docs-proposal/test.sh` +Expected: `All documentation proposal script tests passed.` + +- [ ] **Step 5: Write the prompts** + +`scripts/docs-proposal/prompt.md`: + +```markdown +# Documentation proposal + +You are updating the public Trusted Server documentation after a commit merged +to `main`. The merged commit and the table mapping implementation sources to +documentation pages appear at the end of this file. + +## Task + +1. Inspect the merged change with `git show --stat ` and + `git show `. +2. Decide whether it changes behavior that `docs/index.md` or a page under + `docs/guide/` describes: configuration fields and defaults, routes, adapter + support, integrations, auction behavior, CLI commands and flags, browser + bundles, or verification commands. Use the Sources of truth table to find + the affected pages. +3. If no reader-facing documentation is affected, change nothing and write + `.docs-proposal/rationale.md` containing one sentence explaining why. +4. Otherwise edit only the affected pages, then write + `.docs-proposal/rationale.md` with one bullet per edited page naming the + page, what changed, and the merged source file that justifies it. + +## Rules + +- Describe behavior the merged code ships. Do not document intent, roadmap + items, or unmerged work. +- Read the merged source before writing. Every statement you add must be + verifiable in the code at this commit. +- Make the smallest edit that keeps the page accurate. Do not restyle, + reorder, or rewrite unaffected text. +- Use only fictional values: `example.com` domains and placeholder secrets. + Never copy real domains, customer names, credentials, or service IDs. +- Keep existing links and anchors working, and link to other pages with + relative paths. +- Do not edit files outside `docs/guide/`, `docs/index.md`, and + `.docs-proposal/`. Do not commit, push, or create branches. +``` + +`scripts/docs-proposal/evidence-prompt.md`: + +````markdown +# Documentation proposal evidence + +The hunks at the end of this file are proposed edits to the Trusted Server +documentation for the merged commit named there. For each hunk, find the merged +source that justifies it. + +Write `.docs-proposal/evidence.json` with exactly this shape, and change no +other file: + +```json +{ + "hunks": [ + { "id": "", "source": ":", "reason": "" } + ] +} +``` + +- Copy `id` exactly from the hunk list. +- `source` is a repository-relative path and line at the merged commit that + shows the documented behavior. +- `reason` is one sentence a reviewer can check against that source. +- If no source justifies a hunk, omit it. The reviewer will be told the hunk + has no cited source. +```` + +- [ ] **Step 6: Write `propose.sh`** + +````bash +#!/usr/bin/env bash +# Asks Copilot CLI to propose documentation edits for one merged commit. +# +# Usage: scripts/docs-proposal/propose.sh +# Writes proposal.patch (empty when nothing changes), rationale.md, and +# evidence.json to , which must be inside the repository. +set -euo pipefail + +sha="$1" +work_dir="$2" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(git rev-parse --show-toplevel)" +spec="$root/docs/superpowers/specs/2026-08-19-documentation-refresh-design.md" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +cd "$root" +docs_proposal_require_sha "$sha" +mkdir -p "$work_dir" +work_dir="$(cd "$work_dir" && pwd)" +relative_dir="${work_dir#"$root"/}" + +run_copilot() { + copilot -p "Read $relative_dir/$1 and follow its instructions exactly." \ + --no-ask-user \ + --allow-tool=write \ + --allow-tool='shell(git show:*)' \ + --allow-tool='shell(git diff:*)' \ + --allow-tool='shell(git log:*)' \ + --allow-tool='shell(git grep:*)' \ + --deny-tool='shell(git push)' \ + --deny-tool='shell(git commit)' +} + +{ + cat "$here/prompt.md" + printf '\n## Merged change\n\n- Commit: %s\n- Subject: %s\n\n' "$sha" "$(git log -1 --format=%s "$sha")" + printf '## Sources of truth\n\n' + awk '/^## Sources of truth/ { found = 1; next } /^## / { found = 0 } found' "$spec" +} > "$work_dir/prompt.md" +run_copilot prompt.md + +# Normalize Copilot's edits before hunks are identified so ids stay valid +# after the publish job's format check. +(cd docs && npm ci && npm run format:write) + +git add -A -- docs +git diff --cached --no-renames --binary "$sha" -- docs > "$work_dir/proposal.patch" +touch "$work_dir/rationale.md" +if [ ! -s "$work_dir/proposal.patch" ]; then + printf '{}\n' > "$work_dir/evidence.json" + printf 'No documentation changes proposed for %s.\n' "$sha" + exit 0 +fi + +{ + cat "$here/evidence-prompt.md" + printf '\n## Merged change\n\n- Commit: %s\n\n## Hunks\n\n```json\n' "$sha" + git diff --cached --no-renames "$sha" -- docs | node "$here/hunks.mjs" list + printf '```\n' +} > "$work_dir/evidence-prompt.md" +run_copilot evidence-prompt.md +if [ ! -f "$work_dir/evidence.json" ]; then + printf '{}\n' > "$work_dir/evidence.json" +fi +```` + +- [ ] **Step 7: Ignore the work dir, lint, and commit** + +Append `/.docs-proposal/` to `.gitignore` under a comment +`# Documentation proposal workflow scratch space`. + +Run: `chmod +x scripts/docs-proposal/propose.sh && shellcheck scripts/docs-proposal/*.sh` (skip locally if +`shellcheck` is not installed; CI runs it in Task 5). +Run: `scripts/docs-proposal/test.sh` — Expected: pass. + +```bash +git add .gitignore scripts/docs-proposal/lib.sh scripts/docs-proposal/prompt.md scripts/docs-proposal/evidence-prompt.md scripts/docs-proposal/propose.sh scripts/docs-proposal/test.sh +git commit -S --signoff -m "Add Copilot documentation proposal script" +``` + +--- + +### Task 4: `publish.sh` with stubbed integration tests + +**Files:** + +- Create: `scripts/docs-proposal/publish.sh` +- Modify: `scripts/docs-proposal/test.sh` (replace the `# PUBLISH_TESTS` line) + +**Interfaces:** + +- Consumes: `lib.sh` (Task 3), `node hunks.mjs review` (Task 2), files from `propose.sh`. +- Produces: `publish.sh `; env `GH_TOKEN`. `gh` calls used: + - `gh pr list --head --state all --json number,state --jq '.[0] | select(.) | "\(.number) \(.state)"'` + - `gh pr close --comment --delete-branch` + - `gh api repos/{owner}/{repo}/commits//pulls --jq '.[0].number // empty'` + - `gh pr create --base main --head --title --body-file ` (prints URL) + - `gh pr edit --title --body-file ` + - `gh api --method POST repos/{owner}/{repo}/pulls//reviews --input ` + +- [ ] **Step 1: Write the failing publish tests** — replace `# PUBLISH_TESTS` + in `test.sh` with: + +```bash +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 +export GIT_AUTHOR_NAME=Example GIT_AUTHOR_EMAIL=dev@example.com +export GIT_COMMITTER_NAME=Example GIT_COMMITTER_EMAIL=dev@example.com + +mkdir -p "$tmp/bin" +cat > "$tmp/bin/gh" <<'STUB' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$STUB_LOG" +case "$1 $2" in + "pr list") printf '%s' "${STUB_PR:-}" ;; + "pr create") printf 'https://github.com/example/repo/pull/7\n' ;; + "api repos/{owner}/{repo}/commits/"*) printf '%s' "${STUB_ORIGIN_PR:-}" ;; + "api --method") + for arg in "$@"; do last="$arg"; done + cp "$last" "$STUB_REVIEW" + ;; +esac +STUB +printf '#!/usr/bin/env bash\nprintf "npm %%s\\n" "$*" >> "$STUB_LOG"\n' > "$tmp/bin/npm" +chmod +x "$tmp/bin/gh" "$tmp/bin/npm" +export PATH="$tmp/bin:$PATH" STUB_LOG="$tmp/gh.log" STUB_REVIEW="$tmp/review.json" + +git init -q --bare -b main "$tmp/origin.git" +git clone -q "$tmp/origin.git" "$tmp/work" 2>/dev/null +cd "$tmp/work" +mkdir -p docs/guide +printf 'line one\nline two\nline three\n' > docs/guide/cli.md +git add docs && git commit -q -m "Seed docs" && git push -q origin HEAD:main +merge_sha="$(git rev-parse HEAD)" +branch="$(docs_proposal_branch "$merge_sha")" + +# Each publish run leaves HEAD on its proposal commit, so every patch starts +# from a clean checkout of the merge commit. +make_patch() { + git reset -q --hard "$merge_sha" && git clean -qfd + rm -rf "$tmp/proposal" && mkdir -p "$tmp/proposal" + if [ -n "$1" ]; then + mkdir -p "$(dirname "$1")" && printf 'proposed\n' >> "$1" + git add -A && git diff --cached --binary "$merge_sha" > "$tmp/proposal/proposal.patch" + git reset -q --hard "$merge_sha" && git clean -qfd + else + : > "$tmp/proposal/proposal.patch" + fi + printf -- '- docs/guide/cli.md: example rationale.\n' > "$tmp/proposal/rationale.md" + printf '{}\n' > "$tmp/proposal/evidence.json" +} + +run_publish() { + : > "$STUB_LOG" + rm -f "$STUB_REVIEW" + if "$here/publish.sh" "$merge_sha" "$tmp/proposal" > "$tmp/out.log" 2>&1; then + printf '0' + else + printf '1' + fi +} + +remote_has_branch() { + if git ls-remote --exit-code -q origin "refs/heads/$branch" >/dev/null; then printf 'yes'; else printf 'no'; fi +} + +make_patch "" +assert_eq "$(STUB_PR="" run_publish)" 0 "should succeed on an empty proposal" +assert_eq "$(grep -c 'pr create\|pr close' "$STUB_LOG" || true)" 0 "should not open or close a PR for an empty diff" + +make_patch "" +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should succeed when closing a stale proposal" +assert_contains "$(cat "$STUB_LOG")" "pr close 7" "should close the open proposal for an empty diff" + +make_patch docs/superpowers/notes.md +assert_eq "$(STUB_PR="" run_publish)" 1 "should fail for a disallowed path" +assert_contains "$(cat "$tmp/out.log")" "docs/superpowers/notes.md" "should name the disallowed path" +assert_eq "$(remote_has_branch)" no "should not push a disallowed proposal" + +make_patch docs/guide/cli.md +assert_eq "$(STUB_PR="" STUB_ORIGIN_PR=42 run_publish)" 0 "should publish a valid proposal" +assert_eq "$(remote_has_branch)" yes "should push the proposal branch" +assert_contains "$(cat "$STUB_LOG")" "pr create --base main --head $branch" "should open a PR from the proposal branch" +assert_contains "$(cat "$STUB_LOG")" "npm run build" "should run the docs build" +assert_contains "$(cat "$STUB_REVIEW")" '"path": "docs/guide/cli.md"' "should post a review for the hunk" + +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should succeed on an unchanged retry" +assert_eq "$(grep -c 'pr edit\|pr create\|reviews' "$STUB_LOG" || true)" 0 "should not re-post an unchanged proposal" + +make_patch docs/index.md +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should update a changed proposal" +assert_contains "$(cat "$STUB_LOG")" "pr edit 7" "should edit the existing PR" +assert_contains "$(cat "$STUB_REVIEW")" '"path": "docs/index.md"' "should review the new hunk" + +assert_eq "$(STUB_PR="7 CLOSED" run_publish)" 0 "should succeed for a closed proposal" +assert_eq "$(grep -c 'pr edit\|pr create' "$STUB_LOG" || true)" 0 "should never reopen or recreate a closed proposal" + +cd "$here" +``` + +Note: `make_patch docs/index.md` creates `docs/index.md` (an added file) so the +"changed proposal" case diffs a different tree than the pushed branch. + +- [ ] **Step 2: Run to verify it fails** + +Run: `scripts/docs-proposal/test.sh` +Expected: FAIL lines such as `should succeed on an empty proposal` because +`publish.sh` does not exist. + +- [ ] **Step 3: Write `publish.sh`** + +```bash +#!/usr/bin/env bash +# Validates a documentation proposal and opens or updates its pull request. +# +# Usage: scripts/docs-proposal/publish.sh +# holds proposal.patch, rationale.md, and evidence.json from +# propose.sh. Requires GH_TOKEN with contents and pull-requests write access. +set -euo pipefail + +sha="$1" +work_dir="$(cd "$2" && pwd)" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(git rev-parse --show-toplevel)" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +cd "$root" +docs_proposal_require_sha "$sha" +branch="$(docs_proposal_branch "$sha")" + +existing="$(gh pr list --head "$branch" --state all --json number,state --jq '.[0] | select(.) | "\(.number) \(.state)"')" +pr_number="${existing%% *}" +pr_state="${existing#* }" +if [ -n "$existing" ] && [ "$pr_state" != "OPEN" ]; then + printf 'Proposal #%s for %s is %s; not reopening it.\n' "$pr_number" "$sha" "$pr_state" + exit 0 +fi + +if [ ! -s "$work_dir/proposal.patch" ]; then + printf 'No documentation changes proposed for %s.\n' "$sha" + if [ -n "$existing" ]; then + gh pr close "$pr_number" --delete-branch \ + --comment "A rerun for $sha proposed no documentation changes." + fi + exit 0 +fi + +git switch --quiet --detach "$sha" +git apply --index "$work_dir/proposal.patch" +if ! disallowed="$(git diff --cached --name-only --no-renames "$sha" | docs_proposal_disallowed_paths)"; then + printf '::error::Proposal changes paths outside docs/guide/** and docs/index.md:\n%s\n' "$disallowed" + exit 1 +fi + +(cd docs && npm ci && npm run lint && npm run format && npm run build) + +git -c user.name='github-actions[bot]' \ + -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \ + commit --quiet --no-verify \ + -m "Propose documentation updates for ${sha:0:12}" \ + -m "Generated by the docs-proposal workflow from $sha." +head_sha="$(git rev-parse HEAD)" + +remote_tree="" +if git fetch --quiet origin "refs/heads/$branch" 2>/dev/null; then + remote_tree="$(git rev-parse 'FETCH_HEAD^{tree}')" +fi +if [ -n "$existing" ] && [ "$remote_tree" = "$(git rev-parse 'HEAD^{tree}')" ]; then + printf 'Proposal #%s for %s is unchanged.\n' "$pr_number" "$sha" + exit 0 +fi +git push --quiet --force origin "HEAD:refs/heads/$branch" + +subject="$(git log -1 --format=%s "$sha")" +origin_pr="$(gh api "repos/{owner}/{repo}/commits/$sha/pulls" --jq '.[0].number // empty' || true)" +title="$(printf 'Update documentation for %s' "$subject" | cut -c1-200)" +docs_proposal_pr_body "$sha" "$subject" "$origin_pr" "$work_dir/rationale.md" > "$work_dir/pr-body.md" +if [ -n "$existing" ]; then + gh pr edit "$pr_number" --title "$title" --body-file "$work_dir/pr-body.md" +else + pr_url="$(gh pr create --base main --head "$branch" --title "$title" --body-file "$work_dir/pr-body.md")" + pr_number="${pr_url##*/}" +fi + +git diff --no-renames "$sha" "$head_sha" -- docs \ + | node "$here/hunks.mjs" review "$work_dir/evidence.json" "$head_sha" > "$work_dir/review.json" +gh api --method POST "repos/{owner}/{repo}/pulls/$pr_number/reviews" --input "$work_dir/review.json" > /dev/null +printf 'Published proposal #%s for %s.\n' "$pr_number" "$sha" +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `chmod +x scripts/docs-proposal/publish.sh && scripts/docs-proposal/test.sh` +Expected: `All documentation proposal script tests passed.` + +- [ ] **Step 5: Commit** + +```bash +git add scripts/docs-proposal/publish.sh scripts/docs-proposal/test.sh +git commit -S --signoff -m "Publish documentation proposals as per-merge pull requests" +``` + +--- + +### Task 5: Workflow, CI coverage, and operator docs + +**Files:** + +- Create: `.github/workflows/docs-proposal.yml` +- Modify: `.github/workflows/format.yml` (new job), `scripts/README.md` (table rows + prerequisites), + `docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md` (corrections below) + +**Interfaces:** + +- Consumes: `propose.sh .docs-proposal`, `publish.sh .docs-proposal`. + +- [ ] **Step 1: Write `docs-proposal.yml`** + +```yaml +name: Documentation proposal + +# Proposes documentation edits with Copilot CLI after code merges to main and +# opens one reviewable pull request per merge. See +# docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md. +on: + push: + branches: [main] + paths: + - 'crates/**' + - 'trusted-server.example.toml' + - 'edgezero.toml' + - 'fastly.toml' + - '.cargo/config.toml' + - 'scripts/**' + - '!scripts/docs-proposal/**' + workflow_dispatch: + inputs: + sha: + description: Full SHA of a commit on main to propose documentation for + required: true + type: string + +permissions: {} + +concurrency: + group: docs-proposal + cancel-in-progress: false + +env: + MERGE_SHA: ${{ inputs.sha || github.sha }} + +jobs: + propose: + runs-on: ubuntu-latest + permissions: + contents: read + copilot-requests: write + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + ref: ${{ env.MERGE_SHA }} + fetch-depth: 0 + persist-credentials: false + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + cache: 'npm' + cache-dependency-path: docs/package-lock.json + + - name: Install Copilot CLI + run: npm install -g @github/copilot@1.0.90 + + - name: Propose documentation edits + env: + GITHUB_TOKEN: ${{ github.token }} + run: scripts/docs-proposal/propose.sh "$MERGE_SHA" .docs-proposal + + - name: Upload proposal + uses: actions/upload-artifact@v4 + with: + name: docs-proposal + path: .docs-proposal/ + include-hidden-files: true + if-no-files-found: error + + publish: + needs: propose + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + ref: ${{ env.MERGE_SHA }} + fetch-depth: 0 + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + cache: 'npm' + cache-dependency-path: docs/package-lock.json + + - name: Download proposal + uses: actions/download-artifact@v4 + with: + name: docs-proposal + path: .docs-proposal + + - name: Publish proposal pull request + env: + GH_TOKEN: ${{ github.token }} + run: scripts/docs-proposal/publish.sh "$MERGE_SHA" .docs-proposal +``` + +- [ ] **Step 2: Add the CI job to `format.yml`** (after `format-docs`) + +```yaml +docs-proposal-scripts: + name: docs-proposal scripts + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> $GITHUB_OUTPUT + shell: bash + + - name: Use Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + + - name: Run shellcheck + run: shellcheck scripts/docs-proposal/*.sh + + - name: Test hunk review builder + run: node --test scripts/docs-proposal/hunks.test.mjs + + - name: Test proposal scripts + run: scripts/docs-proposal/test.sh +``` + +- [ ] **Step 3: Document in `scripts/README.md`** — add table rows: + +```markdown +| `docs-proposal/propose.sh` | Merge SHA on `main`, work dir; Copilot CLI, Node, `npm` | Runs Copilot CLI to edit `docs/guide/**` or `docs/index.md`, formats docs, and writes a patch, rationale, and evidence into the work dir. Leaves the docs edits staged. | +| `docs-proposal/publish.sh` | Merge SHA, work dir from `propose.sh`; `gh` with write `GH_TOKEN` | Applies the patch, runs docs gates, force-pushes `docs/auto/`, opens or edits its PR, and posts a per-hunk review. Never writes `main`. | +| `docs-proposal/test.sh` | Git, Node | Tests the helpers and publish flow against a temporary repository with stubbed `gh` and `npm`; removes it on exit. | +``` + +Then append after the table: + +```markdown +The `Documentation proposal` workflow runs both proposal scripts after code +merges to `main`. It needs the organization Copilot policy "Allow use of +Copilot CLI billed to the organization" and the repository setting "Allow +GitHub Actions to create and approve pull requests". Pull requests opened with +`GITHUB_TOKEN` do not trigger other workflows, so the publish job runs the docs +gates itself; push to the proposal branch to run regular CI. Retry a merge with +the workflow's manual dispatch and its full commit SHA. +``` + +Run `cd docs && npx prettier --write ../scripts/README.md` only if the docs +Prettier config covers it; otherwise align the table columns by hand. + +- [ ] **Step 4: Correct the spec** in + `docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md`: + +- Per-hunk table, last row: anchor "Listed in the review body", suggestion "None". +- Trigger: add that `scripts/docs-proposal/**` is excluded and that dispatch + requires a full SHA reachable from `origin/main`. +- `publish` step 1: a closed or merged proposal for the SHA is left alone. +- Testing: CI runs the scripts' tests in a new `format.yml` job. + +Run: `cd docs && npx prettier --write superpowers/specs/2026-10-01-docs-proposal-automation-design.md superpowers/plans/2026-10-01-docs-proposal-automation.md` + +- [ ] **Step 5: Validate and commit** + +Run: `actionlint .github/workflows/docs-proposal.yml .github/workflows/format.yml` if +installed, else `python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" .github/workflows/docs-proposal.yml .github/workflows/format.yml`. +Expected: no errors. + +```bash +git add .github/workflows/docs-proposal.yml .github/workflows/format.yml scripts/README.md docs/superpowers +git commit -S --signoff -m "Add documentation proposal workflow" -m "Closes #1105." +``` + +--- + +### Task 6: Verification + +- [ ] `node --test scripts/docs-proposal/hunks.test.mjs` — all pass. +- [ ] `scripts/docs-proposal/test.sh` — all pass. +- [ ] `cd docs && npm run lint && npm run format && npm run build` — pass. +- [ ] `cargo fmt --all -- --check` — unaffected, pass. +- [ ] After merge: dispatch `Documentation proposal` with a recent merge SHA + that changed a CLI flag; confirm a `docs/auto/` PR with per-hunk + review comments, and that a second dispatch with the same SHA does not + re-post. diff --git a/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md b/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md new file mode 100644 index 000000000..545adb58d --- /dev/null +++ b/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md @@ -0,0 +1,165 @@ +# Documentation Proposal Automation Design + +- Status: proposed +- Design date: 2026-10-01 +- Issue: #1105 +- Target branch: `main` + +## Purpose + +When a merge to `main` changes behavior that the public documentation +describes, a reviewable pull request with the proposed documentation diff +should appear automatically. Today `deploy-docs.yml` only builds and deploys +the VitePress site after documentation changes land; nothing proposes edits +after a code merge. + +This design adds a proposal path that never writes to `main`. It is a +deliberate, scoped addition to the documentation refresh design +(`2026-08-19-documentation-refresh-design.md`), which excluded automated +tracked-file write paths. The exclusion still holds for `main`: automation +only ever writes to a dedicated proposal branch and a pull request. + +## Decisions + +| Question | Decision | +| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| How is prose produced? | GitHub Copilot CLI, run headless in GitHub Actions with the workflow `GITHUB_TOKEN` (`copilot-requests: write`), billed to the org. | +| Which code changes are examined? | Pushes to `main` touching `crates/**`, `trusted-server.example.toml`, `edgezero.toml`, `fastly.toml`, `.cargo/config.toml`, `scripts/**`. | +| Which pages may change? | `docs/guide/**` and `docs/index.md` only — the public site validated by the docs gates. | +| How is prose checked against merged code? | Copilot cites the merged source for every hunk; a per-hunk review exposes the evidence and a one-click revert; a human approves. | +| How do sequential or overlapping merges map? | One pull request per merge commit on branch `docs/auto/`; runs are serialized. | +| What happens on an empty diff? | No pull request. An existing proposal pull request for that SHA is closed. | +| What prevents duplicates on retry? | The branch name is derived from the merge SHA; a retry force-updates the same branch and edits the same pull request. | + +## Workflow shape + +`.github/workflows/docs-proposal.yml` has two jobs. Copilot never holds a +token that can push or open pull requests. + +### Trigger + +- `push` to `main`, filtered to the source paths above. `docs/**` and + `scripts/docs-proposal/**` are not included, so a merged documentation + proposal or a change to this tooling never retriggers the workflow. +- `workflow_dispatch` with a required `sha` input, for retries and for the + first end-to-end validation. Both scripts require a full 40-character SHA + reachable from `origin/main`. +- `concurrency: { group: docs-proposal, cancel-in-progress: false }` so + sequential merges are processed one at a time, in order. + +### Job `propose` + +Permissions: `contents: read`, `copilot-requests: write`. + +1. Check out the merge commit with full history. +2. Install Copilot CLI (`@github/copilot`) at a pinned version. +3. `scripts/docs-proposal/propose.sh .docs-proposal`: + 1. Renders `.docs-proposal/prompt.md` from `scripts/docs-proposal/prompt.md`, + the merge SHA, its subject, and the Sources-of-truth table from the + refresh design. + 2. Runs `copilot -p` with `--no-ask-user`, `--allow-tool=write`, and + `git show`, `git diff`, `git log`, and `git grep` shell tools; `git push` + and `git commit` are denied, and file reads need no permission. Copilot + edits documentation only when the merged change alters reader-facing + behavior, and writes `.docs-proposal/rationale.md`. + 3. Runs `npm ci` and `npm run format:write` in `docs/` so Prettier + normalizes the edits before any hunk is identified. + 4. Writes `proposal.patch` (`git diff --cached --binary -- docs`). An + empty patch ends the job here. + 5. Runs `node scripts/docs-proposal/hunks.mjs list` to enumerate the hunks. + A hunk id is the changed path plus a short hash of the hunk's removed + and added lines, so it is stable across reruns. A second Copilot run + writes `evidence.json`: for each hunk id, the merged source `path:line` + that justifies it and a one-sentence reason. +4. Upload the `.docs-proposal/` directory as an artifact. + +### Job `publish` + +Permissions: `contents: write`, `pull-requests: write`. Copilot does not run. + +`scripts/docs-proposal/publish.sh .docs-proposal`: + +1. If a closed or merged pull request already exists for + `docs/auto/`, leave it alone and exit successfully: a + maintainer's decision is never reopened or recreated. +2. Empty diff: close any open pull request for that branch with a comment, + then exit successfully. Otherwise apply `proposal.patch` to the merge + commit. +3. Reject the proposal if any changed path is outside `docs/guide/**` or + `docs/index.md`. +4. In `docs/`: `npm ci`, `npm run lint`, `npm run format`, `npm run build`. + Any failure fails the job; no pull request is offered. +5. Commit as `github-actions[bot]` on `docs/auto/`. If the remote + branch already exists and its tree equals the new tree, stop: the proposal + is unchanged and nothing is re-posted. Otherwise force-push. +6. Find the originating pull request with + `gh api repos/{owner}/{repo}/commits//pulls`. Create the pull request, + or edit the existing one, with a body that links the merge commit and the + originating pull request, includes `rationale.md`, and states that a human + must verify the prose against the code before merging. +7. `node scripts/docs-proposal/hunks.mjs review` builds review comments and + posts one `COMMENT` review through `gh api`. + +## Per-hunk review + +Each hunk in the pull request receives one review comment containing the +Copilot evidence for that hunk and a `suggestion` block that **reverts** the +hunk. Reviewers accept the prose by leaving it, and reject a hunk by applying +its suggestion. Suggestions are built from the diff, never from model output, +so the revert is exact: + +| Hunk kind | Comment anchor | Suggestion body | +| ------------------------- | --------------------------------------- | ---------------------------------------- | +| Changed or inserted lines | The new lines (right side) | The original lines (empty for insertion) | +| Pure deletion | Adjacent context line (prefer previous) | That context line plus deleted lines | +| Deleted file or binary | Listed in the review body | None | + +Hunks without evidence still receive a revert suggestion, marked "none +cited". Copilot output that names unknown hunk ids is ignored, and evidence +text is stripped of backticks, newlines, and `@` mentions before it is posted. + +GitHub only allows suggestions on lines inside the pull request diff, which is +why the edits are commits and suggestions are reverts. + +## Repository prerequisites + +Documented in `scripts/README.md`: + +- The organization Copilot policy "Allow use of Copilot CLI billed to the + organization" is enabled. +- Repository setting "Allow GitHub Actions to create and approve pull + requests" is enabled. +- Pull requests opened with `GITHUB_TOKEN` do not trigger other workflows. + The `publish` job runs the docs gates itself; maintainers re-run regular CI + by pushing to the branch if needed. + +## Security + +- Copilot runs with a read-only `contents` token and an allowlist of read-only + shell tools; it has no network shell, no push, and no pull request + permissions. +- Input is code already merged to `main`, so no untrusted fork content reaches + the agent. +- The path allowlist is enforced by `publish.sh`, not by the prompt. +- Proposed prose must use fictional `example.com` values, per `CLAUDE.md`; + the reviewer verifies this before merging. + +## Testing + +- `node --test scripts/docs-proposal/hunks.test.mjs` covers hunk listing, + every row of the per-hunk table, unknown evidence ids, and the review payload. +- `scripts/docs-proposal/test.sh` stubs `gh` and `npm` and pushes to a + temporary bare repository to cover the empty diff, closing a stale + proposal, a disallowed path, first publish, an unchanged retry, an updated + proposal, a closed proposal, and PR body rendering. +- `shellcheck` passes for every script. +- The `docs-proposal-scripts` job in `format.yml` runs all three on every + pull request. +- Copilot cannot run locally; the first `workflow_dispatch` run against a + recent merge SHA is the end-to-end test. + +## Out of scope + +- Crate READMEs, `docs/internal/**`, and `docs/superpowers/**`. +- Rolling or merged proposals across several merges. +- A GitHub App token to trigger regular CI on proposal pull requests. From f70ce72181e839b0bd0d2c9a93bfb56a66694c21 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Thu, 1 Oct 2026 18:03:38 +0530 Subject: [PATCH 116/230] Propose documentation updates after code merges to main Add a Documentation proposal workflow that runs GitHub Copilot CLI on each code merge to main, proposes edits to docs/guide or docs/index.md, and opens one pull request per merge on docs/auto/. The publish job runs the docs gates first, skips empty diffs, and never reopens a closed proposal. It also posts a review comment on every hunk that cites the merged source and offers a suggestion reverting the hunk. Closes #1105. Signed-off-by: dhruv8sh --- .github/workflows/docs-proposal.yml | 110 ++++++++ .github/workflows/format.yml | 25 ++ .gitignore | 3 + scripts/README.md | 12 + scripts/docs-proposal/evidence-prompt.md | 23 ++ scripts/docs-proposal/hunks.mjs | 276 ++++++++++++++++++++ scripts/docs-proposal/hunks.test.mjs | 307 +++++++++++++++++++++++ scripts/docs-proposal/lib.sh | 58 +++++ scripts/docs-proposal/prompt.md | 35 +++ scripts/docs-proposal/propose.sh | 68 +++++ scripts/docs-proposal/publish.sh | 81 ++++++ scripts/docs-proposal/test.sh | 166 ++++++++++++ 12 files changed, 1164 insertions(+) create mode 100644 .github/workflows/docs-proposal.yml create mode 100644 scripts/docs-proposal/evidence-prompt.md create mode 100644 scripts/docs-proposal/hunks.mjs create mode 100644 scripts/docs-proposal/hunks.test.mjs create mode 100644 scripts/docs-proposal/lib.sh create mode 100644 scripts/docs-proposal/prompt.md create mode 100755 scripts/docs-proposal/propose.sh create mode 100755 scripts/docs-proposal/publish.sh create mode 100755 scripts/docs-proposal/test.sh diff --git a/.github/workflows/docs-proposal.yml b/.github/workflows/docs-proposal.yml new file mode 100644 index 000000000..956a192dd --- /dev/null +++ b/.github/workflows/docs-proposal.yml @@ -0,0 +1,110 @@ +name: Documentation proposal + +# Proposes documentation edits with Copilot CLI after code merges to main and +# opens one reviewable pull request per merge. See +# docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md. +on: + push: + branches: [main] + paths: + - "crates/**" + - "trusted-server.example.toml" + - "edgezero.toml" + - "fastly.toml" + - ".cargo/config.toml" + - "scripts/**" + - "!scripts/docs-proposal/**" + workflow_dispatch: + inputs: + sha: + description: Full SHA of a commit on main to propose documentation for + required: true + type: string + +permissions: {} + +# Sequential merges are proposed one at a time, in order. +concurrency: + group: docs-proposal + cancel-in-progress: false + +env: + MERGE_SHA: ${{ inputs.sha || github.sha }} + +jobs: + # Copilot runs here with a read-only token; it cannot push or open PRs. + propose: + runs-on: ubuntu-latest + permissions: + contents: read + copilot-requests: write + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + ref: ${{ env.MERGE_SHA }} + fetch-depth: 0 + persist-credentials: false + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + cache: "npm" + cache-dependency-path: docs/package-lock.json + + - name: Install Copilot CLI + run: npm install -g @github/copilot@1.0.90 + + - name: Propose documentation edits + env: + GITHUB_TOKEN: ${{ github.token }} + run: scripts/docs-proposal/propose.sh "$MERGE_SHA" .docs-proposal + + - name: Upload proposal + uses: actions/upload-artifact@v4 + with: + name: docs-proposal + path: .docs-proposal/ + include-hidden-files: true + if-no-files-found: error + + # Copilot does not run here; this job validates and publishes the patch. + publish: + needs: propose + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + ref: ${{ env.MERGE_SHA }} + fetch-depth: 0 + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + cache: "npm" + cache-dependency-path: docs/package-lock.json + + - name: Download proposal + uses: actions/download-artifact@v4 + with: + name: docs-proposal + path: .docs-proposal + + - name: Publish proposal pull request + env: + GH_TOKEN: ${{ github.token }} + run: scripts/docs-proposal/publish.sh "$MERGE_SHA" .docs-proposal diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 7253b5ea2..57da08e84 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -151,3 +151,28 @@ jobs: - name: Build with VitePress (fails on dead links) run: npm run build + + docs-proposal-scripts: + name: docs-proposal scripts + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> $GITHUB_OUTPUT + shell: bash + + - name: Use Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + + - name: Run shellcheck + run: shellcheck scripts/docs-proposal/*.sh + + - name: Test hunk review builder + run: node --test scripts/docs-proposal/hunks.test.mjs + + - name: Test proposal scripts + run: scripts/docs-proposal/test.sh diff --git a/.gitignore b/.gitignore index 24b9e06aa..ce2b3a693 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,6 @@ src/*.html # leftover local build artifacts (node_modules, target, dist) that remain on disk. /crates/js/ /crates/integration-tests/ + +# Documentation proposal workflow scratch space +/.docs-proposal/ diff --git a/scripts/README.md b/scripts/README.md index ad4ecf926..8c2081776 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -18,7 +18,19 @@ their documented orchestration handles a probe deliberately. | `smoke-common.sh` | Sourced by the four smoke scripts | Defines bounded port, process, config, secret, and response assertions. Do not invoke it as a standalone smoke. | | `template-cache-local-test.sh` | Viceroy, Node, OpenSSL, `curl`, `lsof`; optional ports and mode | Builds local artifacts, generates certificates and fixtures in a temporary directory, stops owned servers, and removes the directory. | | `test-cli.sh` | Rustup and the host toolchain; optional host triple | May install the selected Rust target, then runs native CLI and browser-audit tests. Cargo artifacts persist. | +| `docs-proposal/propose.sh` | Merge SHA on `main`, work dir; Copilot CLI, Node, `npm` | Runs Copilot CLI to edit `docs/guide/**` or `docs/index.md`, formats docs, and writes a patch, rationale, and evidence into the work dir. Leaves the edits staged. | +| `docs-proposal/publish.sh` | Merge SHA, work dir from `propose.sh`; `gh` with write `GH_TOKEN` | Applies the patch, runs the docs gates, force-pushes `docs/auto/`, opens or edits its PR, and posts a per-hunk review. Never writes `main`. | +| `docs-proposal/test.sh` | Git and Node | Tests the helpers and publish flow against a temporary repository with stubbed `gh` and `npm`; removes it on exit. | The four adapter smoke contracts are documented in the [deployment guides](../docs/guide/integrations-overview.md#adapter-support). Repository-wide verification policy lives in [TESTING.md](../TESTING.md). + +The `Documentation proposal` workflow runs both proposal scripts after code +merges to `main` and opens one pull request per merge on `docs/auto/`. +It needs the organization Copilot policy "Allow use of Copilot CLI billed to +the organization" and the repository setting "Allow GitHub Actions to create +and approve pull requests". Pull requests opened with `GITHUB_TOKEN` do not +trigger other workflows, so the publish job runs the docs gates itself; push to +the proposal branch to run regular CI. Retry a merge from the workflow's manual +dispatch with its full commit SHA. diff --git a/scripts/docs-proposal/evidence-prompt.md b/scripts/docs-proposal/evidence-prompt.md new file mode 100644 index 000000000..9bb51e35b --- /dev/null +++ b/scripts/docs-proposal/evidence-prompt.md @@ -0,0 +1,23 @@ +# Documentation proposal evidence + +The hunks at the end of this file are proposed edits to the Trusted Server +documentation for the merged commit named there. For each hunk, find the merged +source that justifies it. + +Write `.docs-proposal/evidence.json` with exactly this shape, and change no +other file: + +```json +{ + "hunks": [ + { "id": "", "source": ":", "reason": "" } + ] +} +``` + +- Copy `id` exactly from the hunk list. +- `source` is a repository-relative path and line at the merged commit that + shows the documented behavior. +- `reason` is one sentence a reviewer can check against that source. +- If no source justifies a hunk, omit it. The reviewer will be told the hunk + has no cited source. diff --git a/scripts/docs-proposal/hunks.mjs b/scripts/docs-proposal/hunks.mjs new file mode 100644 index 000000000..5ad98d7c4 --- /dev/null +++ b/scripts/docs-proposal/hunks.mjs @@ -0,0 +1,276 @@ +// Parses a unified documentation diff into hunks and builds the per-hunk +// review that cites Copilot evidence and offers a suggestion reverting each +// hunk. Dependency-free so it runs on any runner with Node. +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +const FILE_HEADER = /^diff --git a\/(.+) b\/(.+)$/; +const HUNK_HEADER = /^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@/; + +export function parseDiff(text) { + const files = []; + let file = null; + let hunk = null; + let oldNo = 0; + let newNo = 0; + for (const raw of text.split("\n")) { + const fileHeader = FILE_HEADER.exec(raw); + if (fileHeader) { + file = { + path: fileHeader[2], + status: "modified", + binary: false, + hunks: [], + }; + files.push(file); + hunk = null; + continue; + } + if (!file) { + continue; + } + const hunkHeader = HUNK_HEADER.exec(raw); + if (hunkHeader) { + oldNo = Number(hunkHeader[1]); + newNo = Number(hunkHeader[2]); + hunk = { lines: [] }; + file.hunks.push(hunk); + continue; + } + if (!hunk) { + if (raw.startsWith("new file mode")) { + file.status = "added"; + } else if (raw.startsWith("deleted file mode")) { + file.status = "deleted"; + } else if (raw.startsWith("Binary files ")) { + file.binary = true; + } + continue; + } + const type = raw[0]; + const lineText = raw.slice(1); + if (type === " ") { + hunk.lines.push({ type, text: lineText, oldNo: oldNo++, newNo: newNo++ }); + } else if (type === "-") { + hunk.lines.push({ type, text: lineText, oldNo: oldNo++, newNo: null }); + } else if (type === "+") { + hunk.lines.push({ type, text: lineText, oldNo: null, newNo: newNo++ }); + } + } + return files; +} + +function hashHunk(path, hunk) { + const hash = createHash("sha256"); + hash.update(path); + for (const line of hunk.lines) { + if (line.type !== " ") { + hash.update(`\n${line.type}${line.text}`); + } + } + return `${path}#${hash.digest("hex").slice(0, 8)}`; +} + +export function collectHunks(files) { + const seen = new Map(); + const entries = []; + for (const file of files) { + if (file.binary) { + continue; + } + for (const hunk of file.hunks) { + const base = hashHunk(file.path, hunk); + const count = (seen.get(base) ?? 0) + 1; + seen.set(base, count); + const id = count === 1 ? base : `${base}-${count}`; + entries.push({ id, path: file.path, status: file.status, hunk }); + } + } + return entries; +} + +export function listHunks(files) { + return collectHunks(files).map(({ id, path, status, hunk }) => ({ + id, + path, + status, + removed: hunk.lines + .filter((line) => line.type === "-") + .map((line) => line.text), + added: hunk.lines + .filter((line) => line.type === "+") + .map((line) => line.text), + })); +} + +// Returns the comment anchor and the original text that undoes a hunk, or +// null when the hunk has no new-side line to anchor a suggestion on. +export function revertSuggestion(hunk) { + const { lines } = hunk; + const first = lines.findIndex((line) => line.type !== " "); + const last = lines.findLastIndex((line) => line.type !== " "); + if (first === -1) { + return null; + } + const span = lines.slice(first, last + 1); + const newSide = span.filter((line) => line.type !== "-"); + const oldText = span + .filter((line) => line.type !== "+") + .map((line) => line.text); + if (newSide.length > 0) { + return { + startLine: newSide[0].newNo, + line: newSide.at(-1).newNo, + body: oldText, + }; + } + const before = lines[first - 1]; + if (before?.type === " ") { + return { + startLine: before.newNo, + line: before.newNo, + body: [before.text, ...oldText], + }; + } + const after = lines[last + 1]; + if (after?.type === " ") { + return { + startLine: after.newNo, + line: after.newNo, + body: [...oldText, after.text], + }; + } + return null; +} + +const MAX_EVIDENCE_LENGTH = 300; + +// Model output lands in review comments, so strip anything that could break +// out of the inline code span or ping a user or team. +function sanitize(value) { + if (typeof value !== "string") { + return ""; + } + return value + .replace(/[`\r\n]+/g, " ") + .replace(/@/g, "@​") + .replace(/\s+/g, " ") + .trim() + .slice(0, MAX_EVIDENCE_LENGTH); +} + +export function indexEvidence(raw) { + const evidence = new Map(); + const items = Array.isArray(raw?.hunks) ? raw.hunks : []; + for (const item of items) { + if (typeof item?.id !== "string") { + continue; + } + const source = sanitize(item.source); + const reason = sanitize(item.reason); + if (source || reason) { + evidence.set(item.id, { source, reason }); + } + } + return evidence; +} + +// A suggestion fence must be longer than any backtick run in its body. +function fenceFor(body) { + let longestRun = 0; + for (const text of body) { + for (const run of text.match(/`+/g) ?? []) { + longestRun = Math.max(longestRun, run.length); + } + } + return "`".repeat(Math.max(3, longestRun + 1)); +} + +function commentBody(cited, body) { + const source = cited?.source + ? `\`${cited.source}\`` + : "none cited — verify this hunk against the merged code."; + const reason = cited?.reason ? ` — ${cited.reason}` : ""; + const fence = fenceFor(body); + const content = body.length > 0 ? `${body.join("\n")}\n` : ""; + return [ + `**Source:** ${source}${reason}`, + "", + "Apply this suggestion to revert the hunk.", + "", + `${fence}suggestion\n${content}${fence}`, + ].join("\n"); +} + +const REVIEW_INTRO = + "Copilot evidence for each proposed hunk. Leave a hunk to accept it, or apply its suggestion to revert it. Verify every cited source before merging."; + +export function buildReview(files, evidence, commitId) { + const comments = []; + const notes = []; + for (const file of files) { + if (file.binary) { + notes.push( + `- \`${file.path}\`: binary change — review the file directly.`, + ); + } + } + for (const { id, path, status, hunk } of collectHunks(files)) { + const suggestion = revertSuggestion(hunk); + if (!suggestion) { + const reason = + status === "deleted" ? "file deleted" : "no line to anchor a revert"; + notes.push(`- \`${path}\`: ${reason} — revert it manually if needed.`); + continue; + } + const comment = { + path, + line: suggestion.line, + side: "RIGHT", + body: commentBody(evidence.get(id), suggestion.body), + }; + if (suggestion.startLine !== suggestion.line) { + comment.start_line = suggestion.startLine; + comment.start_side = "RIGHT"; + } + comments.push(comment); + } + const body = + notes.length > 0 + ? `${REVIEW_INTRO}\n\nChanges without a suggestion:\n\n${notes.join("\n")}` + : REVIEW_INTRO; + return { commit_id: commitId, event: "COMMENT", body, comments }; +} + +function readEvidence(path) { + try { + return indexEvidence(JSON.parse(readFileSync(path, "utf8"))); + } catch (error) { + process.stderr.write( + `Ignoring unreadable evidence at ${path}: ${error.message}\n`, + ); + return new Map(); + } +} + +function main([command, evidencePath, commitId]) { + const files = parseDiff(readFileSync(0, "utf8")); + if (command === "list") { + process.stdout.write(`${JSON.stringify(listHunks(files), null, 2)}\n`); + return; + } + if (command === "review" && evidencePath && commitId) { + const review = buildReview(files, readEvidence(evidencePath), commitId); + process.stdout.write(`${JSON.stringify(review, null, 2)}\n`); + return; + } + process.stderr.write( + "usage: hunks.mjs list | review \n", + ); + process.exitCode = 2; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + main(process.argv.slice(2)); +} diff --git a/scripts/docs-proposal/hunks.test.mjs b/scripts/docs-proposal/hunks.test.mjs new file mode 100644 index 000000000..09440708e --- /dev/null +++ b/scripts/docs-proposal/hunks.test.mjs @@ -0,0 +1,307 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; + +import { + buildReview, + collectHunks, + indexEvidence, + listHunks, + parseDiff, + revertSuggestion, +} from "./hunks.mjs"; + +const MODIFIED = [ + "diff --git a/docs/guide/cli.md b/docs/guide/cli.md", + "index 1111111..2222222 100644", + "--- a/docs/guide/cli.md", + "+++ b/docs/guide/cli.md", + "@@ -10,4 +10,4 @@ heading", + " keep one", + "-old flag", + "+new flag", + " keep two", + "", +].join("\n"); + +test("parseDiff numbers context, removed, and added lines", () => { + const [file] = parseDiff(MODIFIED); + assert.equal(file.path, "docs/guide/cli.md", "should take the b/ path"); + assert.equal(file.status, "modified", "should default to modified"); + assert.deepEqual( + file.hunks[0].lines, + [ + { type: " ", text: "keep one", oldNo: 10, newNo: 10 }, + { type: "-", text: "old flag", oldNo: 11, newNo: null }, + { type: "+", text: "new flag", oldNo: null, newNo: 11 }, + { type: " ", text: "keep two", oldNo: 12, newNo: 12 }, + ], + "should track old and new line numbers", + ); +}); + +test("parseDiff records added, deleted, and binary files", () => { + const diff = [ + "diff --git a/docs/guide/new.md b/docs/guide/new.md", + "new file mode 100644", + "--- /dev/null", + "+++ b/docs/guide/new.md", + "@@ -0,0 +1 @@", + "+hello", + "diff --git a/docs/guide/gone.md b/docs/guide/gone.md", + "deleted file mode 100644", + "--- a/docs/guide/gone.md", + "+++ /dev/null", + "@@ -1 +0,0 @@", + "-bye", + "diff --git a/docs/guide/logo.png b/docs/guide/logo.png", + "Binary files a/docs/guide/logo.png and b/docs/guide/logo.png differ", + "", + ].join("\n"); + const files = parseDiff(diff); + assert.deepEqual( + files.map(({ path, status, binary }) => ({ path, status, binary })), + [ + { path: "docs/guide/new.md", status: "added", binary: false }, + { path: "docs/guide/gone.md", status: "deleted", binary: false }, + { path: "docs/guide/logo.png", status: "modified", binary: true }, + ], + "should classify each file", + ); + assert.deepEqual( + files[0].hunks[0].lines, + [{ type: "+", text: "hello", oldNo: null, newNo: 1 }], + "should start a new file at line 1", + ); +}); + +test("parseDiff skips no-newline markers", () => { + const diff = MODIFIED.replace( + " keep two\n", + " keep two\n\\ No newline at end of file\n", + ); + const [file] = parseDiff(diff); + assert.equal(file.hunks[0].lines.length, 4, "should ignore the marker line"); +}); + +test("collectHunks gives stable ids and skips binary files", () => { + const first = collectHunks(parseDiff(MODIFIED)); + const second = collectHunks( + parseDiff(MODIFIED.replace("@@ -10,4 +10,4", "@@ -20,4 +20,4")), + ); + assert.equal(first.length, 1, "should collect the single hunk"); + assert.match( + first[0].id, + /^docs\/guide\/cli\.md#[0-9a-f]{8}$/, + "should use path#hash", + ); + assert.equal(first[0].id, second[0].id, "should not depend on line numbers"); +}); + +test("collectHunks suffixes duplicate ids", () => { + const twice = MODIFIED + MODIFIED.split("\n").slice(4).join("\n"); + const ids = collectHunks(parseDiff(twice)).map((entry) => entry.id); + assert.equal(ids.length, 2, "should collect both hunks"); + assert.equal(ids[1], `${ids[0]}-2`, "should suffix the repeated id"); +}); + +test("listHunks exposes removed and added text", () => { + const [entry] = listHunks(parseDiff(MODIFIED)); + assert.deepEqual(entry.removed, ["old flag"], "should list removed lines"); + assert.deepEqual(entry.added, ["new flag"], "should list added lines"); + assert.equal(entry.status, "modified", "should carry the file status"); +}); + +function hunkOf(lines, oldStart = 1, newStart = 1) { + const diff = [ + "diff --git a/docs/guide/a.md b/docs/guide/a.md", + "--- a/docs/guide/a.md", + "+++ b/docs/guide/a.md", + `@@ -${oldStart} +${newStart} @@`, + ...lines, + "", + ].join("\n"); + return parseDiff(diff)[0].hunks[0]; +} + +test("revertSuggestion restores replaced lines", () => { + const suggestion = revertSuggestion(hunkOf([" a", "-b", "+B", "+B2", " c"])); + assert.deepEqual( + suggestion, + { startLine: 2, line: 3, body: ["b"] }, + "should span the new lines and restore the old one", + ); +}); + +test("revertSuggestion deletes pure insertions", () => { + const suggestion = revertSuggestion(hunkOf([" a", "+new", " c"])); + assert.deepEqual( + suggestion, + { startLine: 2, line: 2, body: [] }, + "should suggest an empty body", + ); +}); + +test("revertSuggestion spans several change groups with their context", () => { + const suggestion = revertSuggestion(hunkOf(["-a", "+A", " mid", "-c", "+C"])); + assert.deepEqual( + suggestion, + { startLine: 1, line: 3, body: ["a", "mid", "c"] }, + "should keep the middle context line", + ); +}); + +test("revertSuggestion anchors pure deletions on the previous context line", () => { + const suggestion = revertSuggestion(hunkOf([" a", "-gone", " c"], 4, 4)); + assert.deepEqual( + suggestion, + { startLine: 4, line: 4, body: ["a", "gone"] }, + "should restore the deleted line after its anchor", + ); +}); + +test("revertSuggestion anchors leading deletions on the next context line", () => { + const suggestion = revertSuggestion(hunkOf(["-gone", " a"])); + assert.deepEqual( + suggestion, + { startLine: 1, line: 1, body: ["gone", "a"] }, + "should restore the deleted line before its anchor", + ); +}); + +test("revertSuggestion returns null without an anchor", () => { + assert.equal( + revertSuggestion(hunkOf(["-only"])), + null, + "should not invent an anchor", + ); +}); + +test("indexEvidence keeps valid entries and sanitizes text", () => { + const evidence = indexEvidence({ + hunks: [ + { + id: "x#1", + source: "crates/a.rs:4", + reason: "Adds `--flag`\nfor @team", + }, + { id: 7, source: "ignored" }, + { id: "x#2" }, + "junk", + ], + }); + assert.deepEqual( + [...evidence], + [["x#1", { source: "crates/a.rs:4", reason: "Adds --flag for @​team" }]], + "should drop invalid entries and strip backticks, newlines, and mentions", + ); + assert.equal(indexEvidence(null).size, 0, "should accept missing evidence"); +}); + +test("buildReview cites evidence and suggests reverts", () => { + const files = parseDiff( + [ + "diff --git a/docs/guide/a.md b/docs/guide/a.md", + "--- a/docs/guide/a.md", + "+++ b/docs/guide/a.md", + "@@ -1,3 +1,3 @@", + " a", + "-uses ```fence```", + "+B", + " c", + "@@ -10 +10,2 @@", + " j", + "+k", + "", + ].join("\n"), + ); + const [first] = collectHunks(files); + const evidence = indexEvidence({ + hunks: [ + { id: first.id, source: "crates/a.rs:9", reason: "Renames the flag." }, + ], + }); + const review = buildReview(files, evidence, "abc123"); + + assert.equal(review.commit_id, "abc123", "should pin the reviewed commit"); + assert.equal( + review.event, + "COMMENT", + "should not approve or request changes", + ); + assert.equal(review.comments.length, 2, "should comment on every hunk"); + assert.deepEqual( + review.comments[0], + { + path: "docs/guide/a.md", + line: 2, + side: "RIGHT", + body: [ + "**Source:** `crates/a.rs:9` — Renames the flag.", + "", + "Apply this suggestion to revert the hunk.", + "", + "````suggestion", + "uses ```fence```", + "````", + ].join("\n"), + }, + "should escalate the fence and omit start_line for one line", + ); + assert.match( + review.comments[1].body, + /none cited/, + "should flag hunks without evidence", + ); + assert.match( + review.comments[1].body, + /```suggestion\n```$/, + "should suggest deleting an insertion", + ); +}); + +test("buildReview lists binary and unanchored changes in the body", () => { + const files = parseDiff( + [ + "diff --git a/docs/guide/gone.md b/docs/guide/gone.md", + "deleted file mode 100644", + "@@ -1 +0,0 @@", + "-bye", + "diff --git a/docs/guide/logo.png b/docs/guide/logo.png", + "Binary files a/docs/guide/logo.png and b/docs/guide/logo.png differ", + "", + ].join("\n"), + ); + const review = buildReview(files, new Map(), "abc123"); + assert.equal(review.comments.length, 0, "should not post line comments"); + assert.match( + review.body, + /`docs\/guide\/gone\.md`: file deleted/, + "should note the deletion", + ); + assert.match( + review.body, + /`docs\/guide\/logo\.png`: binary change/, + "should note the binary file", + ); +}); + +test("buildReview adds start_line for multi-line spans", () => { + const files = parseDiff( + [ + "diff --git a/docs/index.md b/docs/index.md", + "@@ -1 +1,2 @@", + "-a", + "+b", + "+c", + "", + ].join("\n"), + ); + const [comment] = buildReview(files, new Map(), "abc123").comments; + assert.equal(comment.start_line, 1, "should start at the first new line"); + assert.equal( + comment.start_side, + "RIGHT", + "should anchor the start on the new side", + ); + assert.equal(comment.line, 2, "should end at the last new line"); +}); diff --git a/scripts/docs-proposal/lib.sh b/scripts/docs-proposal/lib.sh new file mode 100644 index 000000000..87fb538d7 --- /dev/null +++ b/scripts/docs-proposal/lib.sh @@ -0,0 +1,58 @@ +# shellcheck shell=bash +# Shared helpers for the documentation proposal scripts. Source this file; +# do not execute it. + +DOCS_PROPOSAL_ALLOWED_RE='^docs/(guide/.+|index\.md)$' +DOCS_PROPOSAL_MAX_RATIONALE_BYTES=20000 + +# Exits unless the argument is a full commit SHA reachable from origin/main. +docs_proposal_require_sha() { + local sha="$1" + if ! [[ "$sha" =~ ^[0-9a-f]{40}$ ]]; then + printf '::error::Expected a full 40-character commit SHA, got %s\n' "$sha" >&2 + exit 1 + fi + if ! git merge-base --is-ancestor "$sha" origin/main 2>/dev/null; then + printf '::error::Commit %s is not reachable from origin/main\n' "$sha" >&2 + exit 1 + fi +} + +docs_proposal_branch() { + printf 'docs/auto/%s\n' "${1:0:12}" +} + +# Reads changed paths on stdin, prints those outside the allowlist, and +# returns 1 when any were found. +docs_proposal_disallowed_paths() { + local path + local found=0 + while IFS= read -r path; do + if [ -n "$path" ] && ! [[ "$path" =~ $DOCS_PROPOSAL_ALLOWED_RE ]]; then + printf '%s\n' "$path" + found=1 + fi + done + return "$found" +} + +docs_proposal_pr_body() { + local sha="$1" + local subject="$2" + local origin_pr="$3" + local rationale="$4" + local origin="" + if [ -n "$origin_pr" ]; then + origin=" (#$origin_pr)" + fi + printf '\n' "$sha" + printf 'Proposed documentation update for merged change %s%s: %s\n\n' "$sha" "$origin" "$subject" + printf 'Generated by GitHub Copilot CLI in the docs-proposal workflow. A maintainer must verify every hunk against the merged code before merging. Each hunk has a review comment citing its source and a suggestion that reverts it.\n\n' + printf '## Rationale\n\n' + if [ -s "$rationale" ]; then + head -c "$DOCS_PROPOSAL_MAX_RATIONALE_BYTES" "$rationale" + printf '\n' + else + printf '_Copilot did not provide a rationale._\n' + fi +} diff --git a/scripts/docs-proposal/prompt.md b/scripts/docs-proposal/prompt.md new file mode 100644 index 000000000..691060fb1 --- /dev/null +++ b/scripts/docs-proposal/prompt.md @@ -0,0 +1,35 @@ +# Documentation proposal + +You are updating the public Trusted Server documentation after a commit merged +to `main`. The merged commit and the table mapping implementation sources to +documentation pages appear at the end of this file. + +## Task + +1. Inspect the merged change with `git show --stat ` and + `git show `. +2. Decide whether it changes behavior that `docs/index.md` or a page under + `docs/guide/` describes: configuration fields and defaults, routes, adapter + support, integrations, auction behavior, CLI commands and flags, browser + bundles, or verification commands. Use the Sources of truth table to find + the affected pages. +3. If no reader-facing documentation is affected, change nothing and write + `.docs-proposal/rationale.md` containing one sentence explaining why. +4. Otherwise edit only the affected pages, then write + `.docs-proposal/rationale.md` with one bullet per edited page naming the + page, what changed, and the merged source file that justifies it. + +## Rules + +- Describe behavior the merged code ships. Do not document intent, roadmap + items, or unmerged work. +- Read the merged source before writing. Every statement you add must be + verifiable in the code at this commit. +- Make the smallest edit that keeps the page accurate. Do not restyle, + reorder, or rewrite unaffected text. +- Use only fictional values: `example.com` domains and placeholder secrets. + Never copy real domains, customer names, credentials, or service IDs. +- Keep existing links and anchors working, and link to other pages with + relative paths. +- Do not edit files outside `docs/guide/`, `docs/index.md`, and + `.docs-proposal/`. Do not commit, push, or create branches. diff --git a/scripts/docs-proposal/propose.sh b/scripts/docs-proposal/propose.sh new file mode 100755 index 000000000..6e2b6a0b2 --- /dev/null +++ b/scripts/docs-proposal/propose.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Asks Copilot CLI to propose documentation edits for one merged commit. +# +# Usage: scripts/docs-proposal/propose.sh +# Writes proposal.patch (empty when nothing changes), rationale.md, and +# evidence.json to , which must be inside the repository so Copilot +# can read the prompts and write its outputs there. +set -euo pipefail + +sha="$1" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(git rev-parse --show-toplevel)" +spec="$root/docs/superpowers/specs/2026-08-19-documentation-refresh-design.md" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +cd "$root" +docs_proposal_require_sha "$sha" +mkdir -p "$2" +work_dir="$(cd "$2" && pwd)" +relative_dir="${work_dir#"$root"/}" + +# Reads are always permitted; writes are allowed so Copilot can edit pages, +# and publish.sh enforces which paths may actually change. +run_copilot() { + copilot -p "Read $relative_dir/$1 and follow its instructions exactly." \ + --no-ask-user \ + --allow-tool=write \ + --allow-tool='shell(git show:*)' \ + --allow-tool='shell(git diff:*)' \ + --allow-tool='shell(git log:*)' \ + --allow-tool='shell(git grep:*)' \ + --deny-tool='shell(git push)' \ + --deny-tool='shell(git commit)' +} + +{ + cat "$here/prompt.md" + printf '\n## Merged change\n\n- Commit: %s\n- Subject: %s\n\n' "$sha" "$(git log -1 --format=%s "$sha")" + printf '## Sources of truth\n\n' + awk '/^## Sources of truth/ { found = 1; next } /^## / { found = 0 } found' "$spec" +} > "$work_dir/prompt.md" +run_copilot prompt.md + +# Normalize Copilot's edits before hunks are identified so their ids stay +# valid after the publish job's format check. +(cd docs && npm ci && npm run format:write) + +git add -A -- docs +git diff --cached --no-renames --binary "$sha" -- docs > "$work_dir/proposal.patch" +touch "$work_dir/rationale.md" +if [ ! -s "$work_dir/proposal.patch" ]; then + printf '{}\n' > "$work_dir/evidence.json" + printf 'No documentation changes proposed for %s.\n' "$sha" + exit 0 +fi + +{ + cat "$here/evidence-prompt.md" + printf '\n## Merged change\n\n- Commit: %s\n\n## Hunks\n\n```json\n' "$sha" + git diff --cached --no-renames "$sha" -- docs | node "$here/hunks.mjs" list + printf '```\n' +} > "$work_dir/evidence-prompt.md" +run_copilot evidence-prompt.md +if [ ! -f "$work_dir/evidence.json" ]; then + printf '{}\n' > "$work_dir/evidence.json" +fi +printf 'Proposed documentation changes for %s.\n' "$sha" diff --git a/scripts/docs-proposal/publish.sh b/scripts/docs-proposal/publish.sh new file mode 100755 index 000000000..90fa8c4ff --- /dev/null +++ b/scripts/docs-proposal/publish.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Validates a documentation proposal and opens or updates its pull request. +# +# Usage: scripts/docs-proposal/publish.sh +# holds proposal.patch, rationale.md, and evidence.json from +# propose.sh. Requires GH_TOKEN with contents and pull-requests write access. +set -euo pipefail + +sha="$1" +work_dir="$(cd "$2" && pwd)" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(git rev-parse --show-toplevel)" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +cd "$root" +docs_proposal_require_sha "$sha" +branch="$(docs_proposal_branch "$sha")" + +# A closed or merged proposal is a maintainer decision; never reopen or +# recreate it on a retry. +existing="$(gh pr list --head "$branch" --state all --json number,state --jq '.[0] | select(.) | "\(.number) \(.state)"')" +pr_number="${existing%% *}" +pr_state="${existing#* }" +if [ -n "$existing" ] && [ "$pr_state" != "OPEN" ]; then + printf 'Proposal #%s for %s is %s; not reopening it.\n' "$pr_number" "$sha" "$pr_state" + exit 0 +fi + +if [ ! -s "$work_dir/proposal.patch" ]; then + printf 'No documentation changes proposed for %s.\n' "$sha" + if [ -n "$existing" ]; then + gh pr close "$pr_number" --delete-branch \ + --comment "A rerun for $sha proposed no documentation changes." + fi + exit 0 +fi + +git switch --quiet --detach "$sha" +git apply --index "$work_dir/proposal.patch" +if ! disallowed="$(git diff --cached --name-only --no-renames "$sha" | docs_proposal_disallowed_paths)"; then + printf '::error::Proposal changes paths outside docs/guide/** and docs/index.md:\n%s\n' "$disallowed" + exit 1 +fi + +(cd docs && npm ci && npm run lint && npm run format && npm run build) + +git -c user.name='github-actions[bot]' \ + -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \ + commit --quiet --no-verify \ + -m "Propose documentation updates for ${sha:0:12}" \ + -m "Generated by the docs-proposal workflow from $sha." +head_sha="$(git rev-parse HEAD)" + +# An identical tree means this is a retry of an already-published proposal; +# skip the push and review so reviewers do not see duplicate comments. +remote_tree="" +if git fetch --quiet origin "refs/heads/$branch" 2>/dev/null; then + remote_tree="$(git rev-parse 'FETCH_HEAD^{tree}')" +fi +if [ -n "$existing" ] && [ "$remote_tree" = "$(git rev-parse 'HEAD^{tree}')" ]; then + printf 'Proposal #%s for %s is unchanged.\n' "$pr_number" "$sha" + exit 0 +fi +git push --quiet --force origin "HEAD:refs/heads/$branch" + +subject="$(git log -1 --format=%s "$sha")" +origin_pr="$(gh api "repos/{owner}/{repo}/commits/$sha/pulls" --jq '.[0].number // empty' || true)" +title="$(printf 'Update documentation for %s' "$subject" | cut -c1-200)" +docs_proposal_pr_body "$sha" "$subject" "$origin_pr" "$work_dir/rationale.md" > "$work_dir/pr-body.md" +if [ -n "$existing" ]; then + gh pr edit "$pr_number" --title "$title" --body-file "$work_dir/pr-body.md" +else + pr_url="$(gh pr create --base main --head "$branch" --title "$title" --body-file "$work_dir/pr-body.md")" + pr_number="${pr_url##*/}" +fi + +git diff --no-renames "$sha" "$head_sha" -- docs \ + | node "$here/hunks.mjs" review "$work_dir/evidence.json" "$head_sha" > "$work_dir/review.json" +gh api --method POST "repos/{owner}/{repo}/pulls/$pr_number/reviews" --input "$work_dir/review.json" > /dev/null +printf 'Published proposal #%s for %s.\n' "$pr_number" "$sha" diff --git a/scripts/docs-proposal/test.sh b/scripts/docs-proposal/test.sh new file mode 100755 index 000000000..6a6eb1181 --- /dev/null +++ b/scripts/docs-proposal/test.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# Tests the documentation proposal helpers and publish flow with stubbed gh +# and npm against a throwaway git repository. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +failures=0 + +assert_eq() { + if [ "$1" != "$2" ]; then + printf 'FAIL: %s\n expected: %s\n actual: %s\n' "$3" "$2" "$1" + failures=$((failures + 1)) + fi +} + +assert_contains() { + if [[ "$1" != *"$2"* ]]; then + printf 'FAIL: %s\n missing: %s\n' "$3" "$2" + failures=$((failures + 1)) + fi +} + +sha=0123456789abcdef0123456789abcdef01234567 + +assert_eq "$(docs_proposal_branch "$sha")" "docs/auto/0123456789ab" "should derive the branch from the SHA" + +bad="$(printf 'docs/guide/a.md\ndocs/index.md\ndocs/superpowers/x.md\nREADME.md\n' | docs_proposal_disallowed_paths || true)" +assert_eq "$bad" $'docs/superpowers/x.md\nREADME.md' "should print only paths outside the allowlist" + +if printf 'docs/guide/a.md\ndocs/guide/integrations/b.md\n' | docs_proposal_disallowed_paths >/dev/null; then + status=0 +else + status=1 +fi +assert_eq "$status" 0 "should accept guide and nested guide paths" + +rationale="$(mktemp)" +printf -- '- docs/guide/cli.md: documents the new flag.\n' > "$rationale" +body="$(docs_proposal_pr_body "$sha" "Add example flag" 42 "$rationale")" +assert_contains "$body" "" "should embed the SHA marker" +assert_contains "$body" "merged change $sha (#42): Add example flag" "should link the merge and its PR" +assert_contains "$body" "documents the new flag" "should include the rationale" +: > "$rationale" +body="$(docs_proposal_pr_body "$sha" "Add example flag" "" "$rationale")" +assert_contains "$body" "merged change $sha: Add example flag" "should omit a missing origin PR" +assert_contains "$body" "did not provide a rationale" "should note an empty rationale" +rm -f "$rationale" + +if (docs_proposal_require_sha "not-a-sha") 2>/dev/null; then + status=0 +else + status=1 +fi +assert_eq "$status" 1 "should reject a malformed SHA" + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 +export GIT_AUTHOR_NAME=Example GIT_AUTHOR_EMAIL=dev@example.com +export GIT_COMMITTER_NAME=Example GIT_COMMITTER_EMAIL=dev@example.com + +mkdir -p "$tmp/bin" +cat > "$tmp/bin/gh" <<'STUB' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$STUB_LOG" +case "$1 $2" in + "pr list") printf '%s' "${STUB_PR:-}" ;; + "pr create") printf 'https://github.com/example/repo/pull/7\n' ;; + "api repos/{owner}/{repo}/commits/"*) printf '%s' "${STUB_ORIGIN_PR:-}" ;; + "api --method") + for arg in "$@"; do last="$arg"; done + cp "$last" "$STUB_REVIEW" + ;; +esac +STUB +cat > "$tmp/bin/npm" <<'STUB' +#!/usr/bin/env bash +printf 'npm %s\n' "$*" >> "$STUB_LOG" +STUB +chmod +x "$tmp/bin/gh" "$tmp/bin/npm" +export PATH="$tmp/bin:$PATH" STUB_LOG="$tmp/gh.log" STUB_REVIEW="$tmp/review.json" + +git init -q --bare "$tmp/origin.git" +git clone -q "$tmp/origin.git" "$tmp/work" 2>/dev/null +cd "$tmp/work" +mkdir -p docs/guide +printf 'line one\nline two\nline three\n' > docs/guide/cli.md +git add docs && git commit -q -m "Seed docs" && git push -q origin HEAD:main +merge_sha="$(git rev-parse HEAD)" +branch="$(docs_proposal_branch "$merge_sha")" + +# Each publish run leaves HEAD on its proposal commit, so every patch starts +# from a clean checkout of the merge commit. +make_patch() { + git reset -q --hard "$merge_sha" && git clean -qfd + rm -rf "$tmp/proposal" && mkdir -p "$tmp/proposal" + if [ -n "$1" ]; then + mkdir -p "$(dirname "$1")" && printf 'proposed\n' >> "$1" + git add -A && git diff --cached --binary "$merge_sha" > "$tmp/proposal/proposal.patch" + git reset -q --hard "$merge_sha" && git clean -qfd + else + : > "$tmp/proposal/proposal.patch" + fi + printf -- '- docs/guide/cli.md: example rationale.\n' > "$tmp/proposal/rationale.md" + printf '{}\n' > "$tmp/proposal/evidence.json" +} + +run_publish() { + : > "$STUB_LOG" + rm -f "$STUB_REVIEW" + if "$here/publish.sh" "$merge_sha" "$tmp/proposal" > "$tmp/out.log" 2>&1; then + printf '0' + else + printf '1' + fi +} + +remote_has_branch() { + if git ls-remote --exit-code -q origin "refs/heads/$branch" >/dev/null; then + printf 'yes' + else + printf 'no' + fi +} + +make_patch "" +assert_eq "$(STUB_PR="" run_publish)" 0 "should succeed on an empty proposal" +assert_eq "$(grep -c 'pr create\|pr close' "$STUB_LOG" || true)" 0 "should not open or close a PR for an empty diff" + +make_patch "" +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should succeed when closing a stale proposal" +assert_contains "$(cat "$STUB_LOG")" "pr close 7" "should close the open proposal for an empty diff" + +make_patch docs/superpowers/notes.md +assert_eq "$(STUB_PR="" run_publish)" 1 "should fail for a disallowed path" +assert_contains "$(cat "$tmp/out.log")" "docs/superpowers/notes.md" "should name the disallowed path" +assert_eq "$(remote_has_branch)" no "should not push a disallowed proposal" + +make_patch docs/guide/cli.md +assert_eq "$(STUB_PR="" STUB_ORIGIN_PR=42 run_publish)" 0 "should publish a valid proposal" +assert_eq "$(remote_has_branch)" yes "should push the proposal branch" +assert_contains "$(cat "$STUB_LOG")" "pr create --base main --head $branch" "should open a PR from the proposal branch" +assert_contains "$(cat "$STUB_LOG")" "npm run build" "should run the docs build" +assert_contains "$(cat "$STUB_REVIEW" 2>/dev/null)" '"path": "docs/guide/cli.md"' "should post a review for the hunk" + +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should succeed on an unchanged retry" +assert_eq "$(grep -c 'pr edit\|pr create\|reviews' "$STUB_LOG" || true)" 0 "should not re-post an unchanged proposal" + +make_patch docs/index.md +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should update a changed proposal" +assert_contains "$(cat "$STUB_LOG")" "pr edit 7" "should edit the existing PR" +assert_contains "$(cat "$STUB_REVIEW" 2>/dev/null)" '"path": "docs/index.md"' "should review the new hunk" + +assert_eq "$(STUB_PR="7 CLOSED" run_publish)" 0 "should succeed for a closed proposal" +assert_eq "$(grep -c 'pr edit\|pr create' "$STUB_LOG" || true)" 0 "should never reopen or recreate a closed proposal" + +cd "$here" + +if [ "$failures" -gt 0 ]; then + printf '%d failure(s)\n' "$failures" + exit 1 +fi +printf 'All documentation proposal script tests passed.\n' From d738d93f70e3a1e79bfa9256cc730a317d14381a Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 1 Oct 2026 09:01:30 -0500 Subject: [PATCH 117/230] Distinguish JavaScript blocks from expression-valued braces Keep regex literals after statement blocks inert without hiding Flight calls after object, function, or class expressions. Bound brace and pending-class context alongside existing lexical nesting limits. Add fragment-boundary and identity/gzip regressions for origin/GTM rewrites and decoded Flight T lengths. Both reported brace/slash regressions were reproduced before the fix. --- .../src/integrations/nextjs/mod.rs | 67 ++++++ .../src/integrations/nextjs/script_lexer.rs | 214 +++++++++++++++++- 2 files changed, 270 insertions(+), 11 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index b37b3a8f2..4cf78475f 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -264,6 +264,9 @@ mod tests { "// self.__next_f.push(\n", "/* self.__next_f.push( */", "const demo=/self.__next_f.push()/;", + "if (ready) { work() } /self.__next_f.push()/;", + "function demo() { return 1 } /self.__next_f.push()/;", + "if (ready) { if (nested) { work() } } /self.__next_f.push()/;", r#"if (ready) {} else /self.__next_f.push([2,"opaque"])/;"#, r#"do /self.__next_f.push([2,"opaque"])/; while (ready);"#, "const demo=`self.__next_f.push(`;", @@ -312,6 +315,10 @@ mod tests { format!("{fake}{push};/* self.__next_f.push( */"), format!("{fake}{rewritten_push};/* self.__next_f.push( */"), ), + ( + format!("if (ready) {{ work() }} /self.__next_f.push()/;{push};"), + format!("if (ready) {{ work() }} /self.__next_f.push()/;{rewritten_push};"), + ), ( format!( "const out=`text self.__next_f.push( ${{ {{a: {push}, b: `nested ${{{push}}}`}} }} tail`;" @@ -350,6 +357,66 @@ mod tests { } } + #[test] + fn brace_slash_division_keeps_flight_lengths_and_both_rewrites() { + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js", "text": "é😀"}); + let body = data.to_string(); + let payload = format!("1:T{:x},{body}", body.len()); + let push = format!( + "self.__next_f.push([1,{}])", + serde_json::to_string(&payload).expect("should encode Flight payload") + ); + let mut expected = data; + expected["url"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js"); + for prefix in [ + "const n = {} / 2;", + "const n = {nested: {}} / 2;", + "const n = function() {} / 2;", + "const n = function named() { return 1 } / 2;", + "const n = class { method() { return 1 } } / 2;", + ] { + let source = format!("{prefix}{push}"); + for split in (0..=source.len()).filter(|split| source.is_char_boundary(*split)) { + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let first = format!("", &source[split..]); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept division prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should complete Flight after division"), + ); + let output = String::from_utf8(bytes).expect("should emit UTF-8"); + assert!(output.starts_with(&format!(""); + for compression in [Compression::None, Compression::Gzip] { + for chunk_size in [32, 1000, 8192] { + let output = mixed_output_with_compression(&html, compression, chunk_size); + let payloads = flight_payloads(&output); + assert_eq!( + payloads.len(), + 2, + "should preserve compressed Flight pushes" + ); + for payload in payloads { + assert_mixed_t_model(&payload, &expected); + } + assert!(!output.contains("__ts_rsc_")); + } + } + } + } + #[test] fn mixed_next_data_flight_mentions_do_not_claim_json_or_hide_gtm() { for text in [ diff --git a/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs index 706144868..df1aa278a 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs @@ -31,17 +31,44 @@ enum Mode { Opaque, } +#[derive(Clone, Copy, Debug)] +enum BraceKind { + StatementBlock, + ExpressionBlock, + Object, +} + +#[derive(Clone, Copy, Debug)] +enum ParenthesisKind { + Control, + Expression, + Function(BraceKind), +} + +#[derive(Clone, Copy, Debug)] +struct PendingClass { + body: BraceKind, + parenthesis_depth: usize, + brace_depth: usize, +} + #[derive(Clone, Debug)] pub(super) struct ScriptLexer { mode: Mode, template_braces: Vec, expression_start: bool, + statement_start: bool, + braces: Vec, word: String, + word_statement_start: bool, word_overflow: bool, word_property: bool, after_dot: bool, control_pending: bool, - parentheses: Vec, + parentheses: Vec, + function_pending: Option, + body_pending: Option, + classes: Vec, operator: Option<(char, bool)>, } @@ -51,12 +78,18 @@ impl Default for ScriptLexer { mode: Mode::Code, template_braces: Vec::new(), expression_start: true, + statement_start: true, + braces: Vec::new(), word: String::new(), + word_statement_start: true, word_overflow: false, word_property: false, after_dot: false, control_pending: false, parentheses: Vec::new(), + function_pending: None, + body_pending: None, + classes: Vec::new(), operator: None, } } @@ -78,7 +111,7 @@ impl ScriptLexer { code } - /// Excessively nested templates are unsupported rather than unbounded. + /// Excessive lexical nesting is unsupported rather than unbounded. pub(super) fn is_opaque(&self) -> bool { matches!(self.mode, Mode::Opaque) } @@ -89,7 +122,9 @@ impl ScriptLexer { if character.is_alphanumeric() || matches!(character, '_' | '$') { if self.word.is_empty() { self.word_property = self.after_dot; + self.word_statement_start = self.statement_start; self.after_dot = false; + self.body_pending = None; } self.operator = None; if self.word.len() < 10 && !self.word_overflow { @@ -98,6 +133,7 @@ impl ScriptLexer { self.word_overflow = true; } self.expression_start = false; + self.statement_start = false; return true; } if !self.word.is_empty() { @@ -123,24 +159,80 @@ impl ScriptLexer { self.word.as_str(), "if" | "while" | "for" | "with" | "switch" | "catch" ); + if !self.word_overflow && !self.word_property { + if matches!(self.word.as_str(), "function" | "class") { + let body = if self.word_statement_start { + BraceKind::StatementBlock + } else { + BraceKind::ExpressionBlock + }; + if self.word == "function" { + self.function_pending = Some(body); + } else if self.classes.len() == MAX_TEMPLATE_DEPTH { + self.mode = Mode::Opaque; + return false; + } else { + self.classes.push(PendingClass { + body, + parenthesis_depth: self.parentheses.len(), + brace_depth: self.braces.len(), + }); + } + } + self.statement_start = matches!(self.word.as_str(), "else" | "do") + || (self.word_statement_start + && matches!(self.word.as_str(), "async" | "export" | "default")); + } self.word.clear(); self.word_overflow = false; } let prior_operator = self.operator.take(); + let mut pending_body = None; if !character.is_whitespace() && character != '/' { self.after_dot = character == '.'; + pending_body = self.body_pending.take(); + if !matches!(character, '(' | '*') { + self.function_pending = None; + } } match character { '(' => { if self.parentheses.len() == MAX_TEMPLATE_DEPTH { self.mode = Mode::Opaque; } else { - self.parentheses.push(self.control_pending); + let kind = if self.control_pending { + ParenthesisKind::Control + } else if let Some(kind) = self.function_pending.take() { + ParenthesisKind::Function(kind) + } else { + ParenthesisKind::Expression + }; + self.parentheses.push(kind); } self.control_pending = false; self.expression_start = true; + self.statement_start = false; + } + ')' => { + let kind = self.parentheses.pop(); + self.expression_start = matches!(kind, Some(ParenthesisKind::Control)); + self.statement_start = self.expression_start; + if let Some(ParenthesisKind::Function(kind)) = kind { + self.body_pending = Some(kind); + } + } + '=' => { + self.operator = Some((character, self.expression_start)); + self.expression_start = true; + self.statement_start = false; + } + '>' => { + if prior_operator.is_some_and(|(operator, _)| operator == '=') { + self.body_pending = Some(BraceKind::ExpressionBlock); + } + self.expression_start = true; + self.statement_start = false; } - ')' => self.expression_start = self.parentheses.pop().unwrap_or(false), '+' | '-' => { if let Some((operator, prior_expression)) = prior_operator && operator == character @@ -150,8 +242,10 @@ impl ScriptLexer { self.operator = Some((character, self.expression_start)); self.expression_start = true; } + self.statement_start = false; } '\'' | '"' => { + self.statement_start = false; self.mode = Mode::String { quote: character, escaped: false, @@ -163,6 +257,7 @@ impl ScriptLexer { } } '`' => { + self.statement_start = false; if self.template_braces.len() == MAX_TEMPLATE_DEPTH { self.mode = Mode::Opaque; } else { @@ -174,26 +269,64 @@ impl ScriptLexer { } } '{' => { + // Blocks admit a new statement after closing; objects and + // function expressions finish a value and admit division. + let class_body = self.classes.last().is_some_and(|class| { + class.parenthesis_depth == self.parentheses.len() + && class.brace_depth == self.braces.len() + }); + let kind = if class_body { + self.classes.pop().expect("should have pending class").body + } else { + pending_body.unwrap_or( + if self.statement_start || !self.expression_start { + BraceKind::StatementBlock + } else { + BraceKind::Object + }, + ) + }; + if self.braces.len() == MAX_TEMPLATE_DEPTH { + self.mode = Mode::Opaque; + } else { + self.braces.push(kind); + } self.expression_start = true; + self.statement_start = !matches!(kind, BraceKind::Object); if let Some(braces) = self.template_braces.last_mut() { *braces = braces.saturating_add(1); } } '}' => { - if let Some(braces) = self.template_braces.last_mut() { - if *braces == 0 { - self.mode = Mode::Template { - escaped: false, - dollar: false, - }; - } else { + if self.template_braces.last() == Some(&0) { + self.mode = Mode::Template { + escaped: false, + dollar: false, + }; + } else { + let kind = self.braces.pop(); + self.expression_start = matches!(kind, Some(BraceKind::StatementBlock)); + self.statement_start = self.expression_start; + if let Some(braces) = self.template_braces.last_mut() { *braces -= 1; } } } + ':' => { + // A reserved word used as an object key is not a class head. + if self.classes.last().is_some_and(|class| { + class.parenthesis_depth == self.parentheses.len() + && class.brace_depth == self.braces.len() + }) { + self.classes.pop(); + } + self.expression_start = true; + self.statement_start = false; + } _ => { if !character.is_whitespace() { self.expression_start = !matches!(character, ')' | ']' | '.'); + self.statement_start = character == ';'; } } } @@ -213,6 +346,7 @@ impl ScriptLexer { _ => { self.mode = Mode::Code; self.expression_start = true; + self.statement_start = false; self.after_dot = false; return self.is_code(character); } @@ -273,6 +407,7 @@ impl ScriptLexer { } else if *dollar && character == '{' { self.mode = Mode::Code; self.expression_start = true; + self.statement_start = false; } else { *dollar = character == '$'; } @@ -307,6 +442,63 @@ mod tests { } } + #[test] + fn brace_context_survives_every_fragment_boundary() { + for prefix in [ + "{} /self.__next_f.push()/;", + "if (ready) { work() } /self.__next_f.push()/;", + "if (ready) { if (nested) { work() } } /self.__next_f.push()/;", + "function demo() { return 1 } /self.__next_f.push()/;", + "async function demo() {} /self.__next_f.push()/;", + "const n = {} / 2;", + "const n = {nested: {}} / 2;", + "const n = function() {} / 2;", + "const n = function named() { return {} } / 2;", + "const n = (() => {}) / 2;", + "class Demo { method() { return 1 } } /self.__next_f.push()/;", + "const n = class { method() { return 1 } } / 2;", + "const n = class extends mixin({}) {} / 2;", + "const n = class extends (class {}) {} / 2;", + "const n = {class: {}, method() { return 1 }} / 2;", + "const n = {method() { if (ready) { work() } /self.__next_f.push()/ }} / 2;", + "const n = `value ${ {} / 2 }`;", + ] { + let source = format!("{prefix}self.__next_f.push([1,\"data\"])"); + for split in 0..=source.len() { + let mut lexer = ScriptLexer::default(); + let code = format!( + "{}{}", + lexer.mask(&source[..split]), + lexer.mask(&source[split..]) + ); + assert_eq!(code.len(), source.len(), "should preserve source offsets"); + assert_eq!( + code.matches("self.__next_f.push(").count(), + 1, + "should expose only the executable push: prefix={prefix}, split={split}" + ); + } + } + } + + #[test] + fn bounds_nested_braces() { + let mut lexer = ScriptLexer::default(); + lexer.mask(&"{".repeat(MAX_TEMPLATE_DEPTH + 1)); + assert!(lexer.is_opaque(), "should refuse excessive brace nesting"); + assert_eq!(lexer.braces.len(), MAX_TEMPLATE_DEPTH); + assert!(!lexer.mask("self.__next_f.push(").contains("__next_f")); + } + + #[test] + fn bounds_pending_class_heads() { + let mut lexer = ScriptLexer::default(); + lexer.mask(&"class extends (".repeat(MAX_TEMPLATE_DEPTH + 1)); + assert!(lexer.is_opaque(), "should refuse excessive class nesting"); + assert_eq!(lexer.classes.len(), MAX_TEMPLATE_DEPTH); + assert!(!lexer.mask("self.__next_f.push(").contains("__next_f")); + } + #[test] fn bounds_nested_templates() { let mut lexer = ScriptLexer::default(); From 3f7c3a83ac9caebc784d6ffc63019b7534d62030 Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 1 Oct 2026 10:57:10 -0500 Subject: [PATCH 118/230] Validate request hosts before source rewriting --- .../tests/routes.rs | 101 ++++++ crates/trusted-server-core/src/http_util.rs | 317 +++++++++++++++--- .../src/integrations/prebid.rs | 1 + crates/trusted-server-core/src/proxy.rs | 13 +- crates/trusted-server-core/src/publisher.rs | 15 +- 5 files changed, 396 insertions(+), 51 deletions(-) diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 42d1b1b6b..b9c6379d5 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -941,6 +941,107 @@ async fn nextjs_auction_output_holds_until_the_structural_body_close() { ); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn nextjs_gtm_rejects_malformed_request_hosts() { + for header in ["host", "x-forwarded-host", "forwarded"] { + let origin = Arc::new(nextjs_auction::NextJsAuctionOrigin::with_html_response( + &nextjs_auction::script_composition_html(), + false, + true, + )); + let router = TrustedServerApp::routes_with_settings_and_services( + nextjs_auction::script_composition_settings(), + nextjs_auction::services(origin), + ) + .expect("should build script composition router"); + let mut service = EdgeZeroAxumService::new(router); + let hostile_host = "example.com"; + let value = if header == "forwarded" { + format!("host=\"{hostile_host}\"") + } else { + hostile_host.to_owned() + }; + let mut request = Request::builder() + .uri("/script-composition") + .header("host", nextjs_auction::PUBLISHER_HOST) + .body(AxumBody::empty()) + .expect("should build hostile host request"); + request.headers_mut().insert( + header, + value.parse().expect("should encode hostile host header"), + ); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should respond to hostile host request"); + assert_eq!( + response.status().as_u16(), + 400, + "should reject malformed {header} rather than rewrite script source" + ); + let body = axum::body::to_bytes(response.into_body(), 1024) + .await + .expect("should consume host rejection response"); + assert!( + !String::from_utf8_lossy(&body).contains(hostile_host), + "should not reflect the rejected host in the response" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn nextjs_gtm_accepts_local_request_hosts_and_ports() { + for host in ["localhost:3000", "example.com:8443", "[::1]:3000"] { + let origin = Arc::new(nextjs_auction::NextJsAuctionOrigin::with_html_response( + &nextjs_auction::script_composition_html(), + false, + true, + )); + let router = TrustedServerApp::routes_with_settings_and_services( + nextjs_auction::script_composition_settings(), + nextjs_auction::services(origin), + ) + .expect("should build script composition router"); + let mut service = EdgeZeroAxumService::new(router); + let request = Request::builder() + .uri("/script-composition") + .header("host", host) + .body(AxumBody::empty()) + .expect("should build local host request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should serve local host request"); + assert_eq!( + response.status().as_u16(), + 200, + "should accept host: {host}" + ); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("should consume local host response"); + let html = String::from_utf8(body.to_vec()).expect("should emit UTF-8 HTML"); + assert!( + html.contains(&format!("http://{host}/app")), + "should preserve host and port in rewritten Flight URL: {host}" + ); + assert!( + !html.contains("__ts_rsc_"), + "should resolve every local host payload" + ); + assert!( + html.ends_with(""), + "should complete local response" + ); + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn nextjs_gtm_html_response_is_complete() { for gzip in [false, true] { diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 457e50356..7e3997378 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -4,6 +4,7 @@ use edgezero_core::body::Body as EdgeBody; use error_stack::Report; use http::{Request, Response, StatusCode, header}; use sha2::{Digest as _, Sha256}; +use std::net::Ipv6Addr; use std::time::Duration; use subtle::ConstantTimeEq as _; @@ -134,7 +135,7 @@ pub struct RequestInfo { } impl RequestInfo { - /// Extract request info from a Fastly request. + /// Extract the validated effective request host and scheme. /// /// Host fallback order (first present wins): /// 1. `Forwarded` header (`host=...`) @@ -148,39 +149,108 @@ impl RequestInfo { /// 4. `Fastly-SSL` /// 5. Default `http` /// - /// In production the forwarded headers are stripped by - /// [`sanitize_forwarded_headers`] at the edge, so `Host` and - /// [`ClientInfo`] TLS detection are the only sources that fire. - pub fn from_request(req: &Request, client_info: &ClientInfo) -> Self { - let host = extract_request_host(req); + /// Fastly and Spin strip forwarded headers with [`sanitize_forwarded_headers`] + /// before routing. On those paths, `Host` and [`ClientInfo`] TLS detection + /// are the effective sources. Other adapters retain their existing policy. + /// + /// The selected host must be a DNS name, IPv4 address, or bracketed IPv6 + /// address, optionally followed by a numeric port. Forwarded-header + /// precedence is unchanged. Requests without any host retain an empty host. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::BadRequest`] when the selected host is + /// malformed, rather than falling back to a lower-priority header. + pub fn from_request( + req: &Request, + client_info: &ClientInfo, + ) -> Result> { + let host = extract_request_host(req)?; let scheme = detect_request_scheme( req, client_info.tls_protocol.as_deref(), client_info.tls_cipher.as_deref(), ); - Self { host, scheme } + Ok(Self { host, scheme }) } } -fn extract_request_host(req: &Request) -> String { - req.headers() - .get("forwarded") - .and_then(|h| h.to_str().ok()) - .and_then(|value| parse_forwarded_param(value, "host")) - .or_else(|| { - req.headers() - .get("x-forwarded-host") - .and_then(|h| h.to_str().ok()) - .and_then(parse_list_header_value) - }) - .or_else(|| { - req.headers() - .get(header::HOST) - .and_then(|h| h.to_str().ok()) +/// Select the effective host without allowing malformed input into rewrite data. +/// +/// # Errors +/// +/// Returns a bad-request error for a malformed selected host or a non-text host header. +fn extract_request_host(req: &Request) -> Result> { + let invalid_host = || { + Report::new(TrustedServerError::BadRequest { + message: "Invalid request host".to_owned(), }) - .unwrap_or_default() - .to_owned() + }; + let forwarded_host = req + .headers() + .get("forwarded") + .map(|value| value.to_str().map_err(|_| invalid_host())) + .transpose()? + .and_then(|value| parse_forwarded_param(value, "host")); + let host = if let Some(host) = forwarded_host { + host + } else if let Some(value) = req.headers().get("x-forwarded-host") { + parse_list_header_value(value.to_str().map_err(|_| invalid_host())?) + .ok_or_else(invalid_host)? + } else if let Some(value) = req.headers().get(header::HOST) { + value.to_str().map_err(|_| invalid_host())? + } else { + return Ok(String::new()); + }; + if !is_valid_request_host(host) { + return Err(invalid_host()); + } + Ok(host.to_owned()) +} + +/// Accept only literal host syntax that is safe to insert into script source. +fn is_valid_request_host(value: &str) -> bool { + let port = if let Some(ipv6) = value.strip_prefix('[') { + let Some((address, suffix)) = ipv6.split_once(']') else { + return false; + }; + if address.parse::().is_err() { + return false; + } + if suffix.is_empty() { + None + } else if let Some(port) = suffix.strip_prefix(':') { + Some(port) + } else { + return false; + } + } else { + let (host, port) = value + .split_once(':') + .map_or((value, None), |(host, port)| (host, Some(port))); + let host = host.strip_suffix('.').unwrap_or(host); + if host.is_empty() + || host.len() > 253 + || host.split('.').any(|label| { + label.is_empty() + || label.len() > 63 + || !label.starts_with(|c: char| c.is_ascii_alphanumeric()) + || !label.ends_with(|c: char| c.is_ascii_alphanumeric()) + || !label + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-') + }) + { + return false; + } + port + }; + port.is_none_or(|port| { + !port.is_empty() + && port.bytes().all(|byte| byte.is_ascii_digit()) + && port.parse::().is_ok() + }) } fn parse_forwarded_param<'a>(forwarded: &'a str, param: &str) -> Option<&'a str> { @@ -189,12 +259,11 @@ fn parse_forwarded_param<'a>(forwarded: &'a str, param: &str) -> Option<&'a str> let mut iter = part.splitn(2, '='); let key = iter.next().unwrap_or("").trim(); let value = iter.next().unwrap_or("").trim(); - if key.is_empty() || value.is_empty() { - continue; - } if key.eq_ignore_ascii_case(param) { let value = strip_quotes(value); - if !value.is_empty() { + // A present but empty host must fail validation, not select a + // lower-priority host. Keep the existing empty-proto behavior. + if !value.is_empty() || param.eq_ignore_ascii_case("host") { return Some(value); } } @@ -572,12 +641,175 @@ mod tests { // RequestInfo tests + #[test] + fn request_info_accepts_valid_hosts_without_normalizing_them() { + for host in [ + "example.com", + "Sub-Domain.EXAMPLE.com.", + "xn--bcher-kva.example.com", + "example.com:443", + "localhost:3000", + "internal-proxy:8080", + "192.0.2.1:8080", + "[2001:db8::1]", + "[2001:db8::1]:443", + "[::1]:3000", + "[::ffff:192.0.2.1]:65535", + ] { + for header in ["host", "x-forwarded-host", "forwarded"] { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header(&mut req, "host", "fallback.example.com"); + let value = if header == "forwarded" { + format!("for=192.0.2.2;host=\"{host}\";proto=https") + } else { + host.to_owned() + }; + set_header(&mut req, header, &value); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept valid host syntax"); + assert_eq!(info.host, host, "should preserve {header} host: {host}"); + } + } + } + + #[test] + fn request_info_rejects_malformed_selected_hosts_without_fallback() { + for host in [ + "", + "example.com", + "example.com'", + "example.com\"", + "example.com`", + "example.com\\path", + "example.com/path", + "example.com?query", + "example.com#fragment", + "user@example.com", + "https://example.com", + "example.com%22", + "example.com:invalid", + "example.com:", + "example.com:65536", + "example.com:+443", + "example.com:443:80", + "bad host.example.com", + "bad_label.example.com", + "-bad.example.com", + "bad-.example.com", + "example..com", + ".example.com", + ".", + "::1", + "[not-ipv6]", + "[2001:db8::1", + "[2001:db8::1]suffix", + "[2001:db8::1]:", + ] { + for header in ["host", "x-forwarded-host", "forwarded"] { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header(&mut req, "host", "fallback.example.com"); + let value = if header == "forwarded" { + format!("host=\"{host}\"") + } else { + host.to_owned() + }; + set_header(&mut req, header, &value); + let error = RequestInfo::from_request(&req, &default_client_info()) + .expect_err("should reject malformed selected host"); + assert!( + matches!( + error.current_context(), + TrustedServerError::BadRequest { .. } + ), + "should reject {header} host with a bad-request error: {host}" + ); + assert!( + !format!("{error}").contains(""), + "should not reflect hostile host bytes in the error" + ); + } + } + } + + #[test] + fn request_info_rejects_non_text_selected_host_headers() { + for header in ["host", "x-forwarded-host", "forwarded"] { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header(&mut req, "host", "fallback.example.com"); + req.headers_mut().insert( + HeaderName::from_static(header), + HeaderValue::from_bytes(b"example.com\xff").expect("should build non-text header"), + ); + assert!( + RequestInfo::from_request(&req, &default_client_info()).is_err(), + "should reject non-text {header} rather than falling back" + ); + } + } + + #[test] + fn request_info_rejects_oversized_dns_names() { + for host in [ + format!("{}.example.com", "a".repeat(64)), + format!( + "{}.{}.{}.{}", + "a".repeat(63), + "b".repeat(63), + "c".repeat(63), + "d".repeat(63) + ), + ] { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header(&mut req, "host", &host); + assert!( + RequestInfo::from_request(&req, &default_client_info()).is_err(), + "should reject oversized DNS name: {host}" + ); + } + } + + #[test] + fn request_info_validates_only_the_selected_host() { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header(&mut req, "forwarded", "host=public.example.com"); + set_header(&mut req, "x-forwarded-host", "example.com"); + set_header(&mut req, "host", "example.com'"); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept valid highest-priority host"); + assert_eq!( + info.host, "public.example.com", + "should preserve precedence without using lower-priority hosts" + ); + } + + #[test] + fn request_info_preserves_empty_forwarded_proto_behavior() { + let mut req = build_request(Method::GET, "https://example.com/page"); + set_header( + &mut req, + "forwarded", + "host=example.com;proto=\"\";proto=https", + ); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept valid host and later nonempty proto"); + assert_eq!(info.scheme, "https", "should retain empty-proto fallback"); + } + + #[test] + fn request_info_preserves_absent_host_behavior() { + let req = build_request(Method::GET, "/page"); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should preserve request metadata without host headers"); + assert!(info.host.is_empty(), "should keep absent host empty"); + } + #[test] fn test_request_info_from_host_header() { let mut req = build_request(Method::GET, "https://test.example.com/page"); set_header(&mut req, "host", "test.example.com"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept Host header"); assert_eq!( info.host, "test.example.com", "Host should use Host header when forwarded headers are missing" @@ -599,7 +831,8 @@ mod tests { "public.example.com, proxy.local", ); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept forwarded host"); assert_eq!( info.host, "public.example.com", "Host should prefer X-Forwarded-Host over Host" @@ -612,7 +845,8 @@ mod tests { set_header(&mut req, "host", "test.example.com"); set_header(&mut req, "x-forwarded-proto", "https, http"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept request scheme metadata"); assert_eq!( info.scheme, "https", "Scheme should prefer the first X-Forwarded-Proto value" @@ -623,7 +857,8 @@ mod tests { set_header(&mut req, "host", "test.example.com"); set_header(&mut req, "x-forwarded-proto", "http"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept request scheme metadata"); assert_eq!( info.scheme, "http", "Scheme should use the X-Forwarded-Proto value when present" @@ -643,7 +878,8 @@ mod tests { set_header(&mut req, "x-forwarded-host", "proxy.local"); set_header(&mut req, "x-forwarded-proto", "http"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept forwarded host with port"); assert_eq!( info.host, "public.example.com:443", "Host should prefer Forwarded host over X-Forwarded-Host" @@ -659,7 +895,8 @@ mod tests { let mut req = build_request(Method::GET, "https://test.example.com/page"); set_header(&mut req, "fastly-ssl", "1"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept request without host"); assert_eq!( info.scheme, "https", "Scheme should fall back to Fastly-SSL when other signals are missing" @@ -675,7 +912,8 @@ mod tests { set_header(&mut req, "x-forwarded-host", "public.example.com"); set_header(&mut req, "x-forwarded-proto", "https"); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept chained proxy host"); assert_eq!( info.host, "public.example.com", "Host should use X-Forwarded-Host in chained proxy scenarios" @@ -769,7 +1007,8 @@ mod tests { set_header(&mut req, "x-forwarded-proto", "http"); sanitize_forwarded_headers(&mut req); - let info = RequestInfo::from_request(&req, &default_client_info()); + let info = RequestInfo::from_request(&req, &default_client_info()) + .expect("should accept sanitized Host header"); assert_eq!( info.host, "legit.example.com", @@ -892,7 +1131,8 @@ mod tests { ..ClientInfo::default() }; - let info = RequestInfo::from_request(&req, &client_info); + let info = RequestInfo::from_request(&req, &client_info) + .expect("should accept TLS protocol metadata"); assert_eq!( info.scheme, "https", @@ -908,7 +1148,8 @@ mod tests { ..ClientInfo::default() }; - let info = RequestInfo::from_request(&req, &client_info); + let info = RequestInfo::from_request(&req, &client_info) + .expect("should accept TLS cipher metadata"); assert_eq!( info.scheme, "https", diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index a031c2554..b001b4dce 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -3964,6 +3964,7 @@ mod tests { fn make_request_info(context: &AuctionContext<'_>) -> RequestInfo { RequestInfo::from_request(context.request, context.services.client_info()) + .expect("should extract valid test request info") } fn config_from_settings( diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index 5d659cdbf..b834742a3 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -1621,7 +1621,8 @@ pub async fn handle_first_party_click( /// # Errors /// /// Returns an error if JSON parsing fails, the URL cannot be parsed, the URL uses an -/// unsupported scheme, the URL lacks a host, or the host violates `proxy.allowed_domains`. +/// unsupported scheme, the URL lacks a host, the host violates `proxy.allowed_domains`, +/// or request metadata needed to determine the scheme contains a malformed host. pub async fn handle_first_party_proxy_sign( settings: &Settings, _services: &RuntimeServices, @@ -1639,11 +1640,11 @@ pub async fn handle_first_party_proxy_sign( // adapter forwards verbatim — have no scheme of their own, so parsing would // hand back the placeholder base's `https` and make an HTTP dev server sign // an HTTPS target it then proxies over TLS against a plaintext service. - let request_scheme = req - .uri() - .scheme_str() - .map(str::to_ascii_lowercase) - .unwrap_or_else(|| RequestInfo::from_request(&req, _services.client_info()).scheme); + let request_scheme = if let Some(scheme) = req.uri().scheme_str() { + scheme.to_ascii_lowercase() + } else { + RequestInfo::from_request(&req, _services.client_info())?.scheme + }; let payload = if method == Method::POST { let body_bytes = request_body_bytes(req.into_body(), "first-party sign")?; diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 6f89edcd0..40d46d47d 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -4312,8 +4312,8 @@ pub struct AuctionDispatch<'a> { /// /// # Errors /// -/// Returns a [`TrustedServerError`] if the proxy request fails or the -/// origin backend is unreachable. +/// Returns a [`TrustedServerError`] if the request host is malformed, the proxy +/// request fails, or the origin backend is unreachable. pub async fn handle_publisher_request( settings: &Settings, services: &RuntimeServices, @@ -4333,8 +4333,8 @@ pub async fn handle_publisher_request( // Prebid.js requests are not intercepted here anymore. The HTML processor removes // publisher-supplied Prebid scripts; the unified TSJS bundle includes Prebid.js when enabled. - // Extract request host and scheme (uses Host header and TLS detection after edge sanitization) - let request_info = RequestInfo::from_request(&req, services.client_info()); + // Validate request host before it can enter HTML or Flight rewrite output. + let request_info = RequestInfo::from_request(&req, services.client_info())?; let request_host = &request_info.host; let request_scheme = &request_info.scheme; @@ -6692,7 +6692,8 @@ fn normalize_page_bids_path(raw: &str) -> String { /// /// # Errors /// -/// Returns [`TrustedServerError`] if cookie parsing or EC ID generation fails. +/// Returns [`TrustedServerError`] if the request host is malformed, cookie +/// parsing fails, or EC ID generation fails. pub async fn handle_page_bids( settings: &Settings, services: &RuntimeServices, @@ -6767,7 +6768,7 @@ pub async fn handle_page_bids( // Trusted request origin for absolute inline creative URLs — derived from the // origin the visitor is actually on (scheme, host, port), not the configured // publisher domain, which cannot carry a port and may differ by subdomain. - let request_info = RequestInfo::from_request(&req, services.client_info()); + let request_info = RequestInfo::from_request(&req, services.client_info())?; let page_bids_request_origin = request_origin(&request_info.scheme, &request_info.host); let path_param = req @@ -6799,7 +6800,7 @@ pub async fn handle_page_bids( Vec::new() }; - let request_info = crate::http_util::RequestInfo::from_request(&req, services.client_info()); + let request_info = crate::http_util::RequestInfo::from_request(&req, services.client_info())?; // Owned so the identity-graph snapshot can be stored back on `ec_context` // below without holding a borrow of it across the mutation. let ec_id = ec_context From f2a6a2a28912b8edf83403e8a09986a449e77efb Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 1 Oct 2026 10:57:10 -0500 Subject: [PATCH 119/230] Address script composition review cleanups --- crates/trusted-server-core/src/html_processor.rs | 6 +++++- .../src/integrations/google_tag_manager.rs | 6 +++++- crates/trusted-server-core/src/integrations/mod.rs | 2 ++ crates/trusted-server-core/src/integrations/nextjs/mod.rs | 3 ++- crates/trusted-server-core/src/integrations/registry.rs | 8 +++++--- 5 files changed, 19 insertions(+), 6 deletions(-) diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 79a2bfaa6..87c0991a9 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -835,7 +835,11 @@ mod tests { output.contains(""), "should reset CSS match state between scripts" ); - assert_eq!(output.matches("flushed-once").count(), 1); + assert_eq!( + output.matches("flushed-once").count(), + 1, + "should emit the flushed fragment exactly once" + ); } #[test] diff --git a/crates/trusted-server-core/src/integrations/google_tag_manager.rs b/crates/trusted-server-core/src/integrations/google_tag_manager.rs index abba52c61..cf4f9673b 100644 --- a/crates/trusted-server-core/src/integrations/google_tag_manager.rs +++ b/crates/trusted-server-core/src/integrations/google_tag_manager.rs @@ -3603,7 +3603,11 @@ container_id = "GTM-DEFAULT" "`/integrations/google_tag_manager/gtm.js${id}`", ), ] { - assert_eq!(rewrite_gtm_rsc_span(source), expected); + assert_eq!( + rewrite_gtm_rsc_span(source), + expected, + "should preserve query delimiters and template interpolation: {source}" + ); } } diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index ee883b0d5..2a48e79d4 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -299,6 +299,8 @@ pub(crate) fn builders() -> &'static [IntegrationBuilder] { id: "testlight", build: testlight::register, }, + // Next.js must precede GTM: GTM reads the Flight-protection snapshot + // that the Next.js script stage records for the same fragment. IntegrationBuilder { id: "nextjs", build: nextjs::register, diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index 4cf78475f..d68614881 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -111,7 +111,6 @@ pub fn register( .integration_config::( "google_tag_manager", )? - .filter(|config| config.enabled) .map(|_| { crate::integrations::google_tag_manager::rewrite_gtm_rsc_span as fn(&str) -> String }); @@ -120,6 +119,8 @@ pub fn register( gtm_transform, )); + // Pages data must precede Flight capture: it sets the one-shot discriminator + // that keeps JSON strings from being treated as executable Flight source. let builder = IntegrationRegistration::builder(NEXTJS_INTEGRATION_ID) .with_script_rewriter(structured) .with_script_rewriter(placeholders) diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index a41f986f1..e9001afec 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -540,13 +540,15 @@ pub trait IntegrationAttributeRewriter: Send + Sync { ) -> AttributeRewriteAction; } -/// Trait for integration-provided inline script/text rewrite hooks. +/// Trait for integration-provided inline script rewrite hooks. pub trait IntegrationScriptRewriter: Send + Sync { /// Identifier for logging/diagnostics. fn integration_id(&self) -> &'static str; - /// CSS selector (e.g. `script#__NEXT_DATA__`) that should trigger this rewriter. + /// CSS selector matching `"#.to_vec(), + br#""#, + ) + } else { + ( + format!(r#"var api="https://{SOURCEPOINT_CDN_HOST}/consent/tcfv2";"#) + .into_bytes(), + br#"var api="/integrations/sourcepoint/cdn/consent/tcfv2";"#, + ) + }; + client.stub.push_response_with_headers( + 200, + input, + vec![ + ("content-type", content_type), + ("content-encoding", encoding), + ], + ); + let services = build_services_with_http_client(client.clone()); + let response = integration + .handle( + &settings, + &services, + make_req( + Method::GET, + "https://publisher.example.com/integrations/sourcepoint/cdn/asset", + ), + ) + .await + .expect("should rewrite explicitly identity-encoded responses"); + + assert!(response.headers().get(header::CONTENT_ENCODING).is_none()); + assert!(client.reads.load(Ordering::Relaxed) > 0); + assert_eq!( + take_body_bytes(response), + expected, + "should rewrite identity content regardless of casing and whitespace" + ); + } + } + }); + } + #[test] fn handle_rewrites_streamed_html_without_content_length() { futures::executor::block_on(async { @@ -1599,17 +1763,17 @@ mod tests { let settings = create_test_settings(); let integration = SourcepointIntegration::new(Arc::new(config(true))); let client = Arc::new(StreamingHttpClient::new()); - let bytes = vec![0x1f, 0x8b, 0xff]; + let bytes = vec![b'a', 0xff, b'b']; client.stub.push_response_with_headers( 200, bytes.clone(), vec![ ("content-type", "application/javascript"), - ("content-encoding", "gzip"), + ("content-encoding", "identity"), ("cache-control", "no-store"), ], ); - let services = build_services_with_http_client(client); + let services = build_services_with_http_client(client.clone()); let response = integration .handle( @@ -1623,9 +1787,14 @@ mod tests { .await .expect("should retain non-UTF-8 content unchanged"); + assert!( + client.reads.load(Ordering::Relaxed) > 0, + "should exercise UTF-8 fallback after bounded collection" + ); + assert_eq!( get_header_str(&response, header::CONTENT_ENCODING), - Some("gzip"), + Some("identity"), "should preserve encoding when no rewrite occurs" ); assert_eq!( diff --git a/docs/guide/integrations/sourcepoint.md b/docs/guide/integrations/sourcepoint.md index 22437dd2b..0c71d6563 100644 --- a/docs/guide/integrations/sourcepoint.md +++ b/docs/guide/integrations/sourcepoint.md @@ -62,16 +62,16 @@ If a publisher uses a Content Security Policy, `script-src` must allow the first ## Response body rewriting -With `rewrite_sdk = true`, successful `GET` responses with status 200 and a JavaScript or HTML content type are eligible for body rewriting. JavaScript CDN URLs and privacy-manager HTML root-relative `src` / `href` assets are routed through the first-party proxy. +With `rewrite_sdk = true`, successful `GET` responses with status 200 and a JavaScript or HTML content type are eligible for body rewriting when `Content-Encoding` is absent or explicitly `identity`. JavaScript CDN URLs and privacy-manager HTML root-relative `src` / `href` assets are routed through the first-party proxy. The input body limit is 5 MiB: - Bodies at or below the limit are rewritten, including chunked or HTTP/2 responses without `Content-Length`. - A declared `Content-Length` above the limit skips rewriting and leaves the body unchanged. - If collection exceeds the limit, Trusted Server stops reading and returns `502 Bad Gateway`. This also applies when `Content-Length` understates the size. The partial body is discarded, not returned to the browser. -- Other content types, disabled rewriting, and ineligible methods or statuses do not collect the body for rewriting. +- Compressed responses, unsupported or malformed encoding headers, other content types, disabled rewriting, and ineligible methods or statuses do not collect the body for rewriting. Their bytes and encoding headers pass through unchanged. -On Fastly, upstream responses remain streaming until the bounded collector reads them. It retains at most 5 MiB of input plus the current transport chunk, with additional bounded allocations for rewriting. Non-rewritten responses remain streaming. Adapters without streaming support still buffer upstream responses before this check; this limit is not an adapter-level memory guarantee on Cloudflare or Spin. +On Fastly, upstream responses remain streaming until the bounded collector reads them. It retains at most 5 MiB of input plus the current transport chunk, with additional bounded allocations for rewriting. Non-rewritten responses remain streaming. Adapters without streaming support still buffer upstream responses before this check; this limit is not an adapter-level memory guarantee on Axum, Cloudflare, or Spin. Likely JavaScript and HTML paths, including `/mms/v2/get_site_data`, request `Accept-Encoding: identity`. Rewritten bodies discard the upstream `Content-Length` and `Content-Encoding`, and remove `Accept-Encoding` from `Vary`. Non-UTF-8 bodies pass through unchanged after bounded collection. From 2028bc6e1e523861e01ac5105d93279eb7ca9114 Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Fri, 2 Oct 2026 10:34:03 +0530 Subject: [PATCH 122/230] Clarify mobile trace capture and truncation rules --- ...-mobile-ad-render-trace-endpoint-design.md | 121 +++++++++++++----- 1 file changed, 92 insertions(+), 29 deletions(-) diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md index bc6387914..7b220df24 100644 --- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md +++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md @@ -287,6 +287,9 @@ is on — cookie observed by server`. The page must not imply that setup-request network facts or an empty auction section describe the affected page. +The setup-request projection reuses the section 9.1 `network` block and section +9.2 `CookieHealth` contracts, including their bounds, without requiring a valid +diagnostics cookie or constructing a `TraceReportV1` envelope. ### 6.2 Active publisher page @@ -838,13 +841,14 @@ a new trace-envelope version. `auction_coverage.capture_status` describes only what reached the browser collector: `not_observed` means no valid server-auction record arrived and no capture issue is known, not that no server auction ran. `partial` requires at -least one retained record plus a projection, transport, validation, or eviction -issue; `unavailable` requires no retained records plus a known projection, -transport, validation, or eviction issue; and -`complete` requires at least one retained record without those capture issues. +least one retained server-auction record plus a projection, transport, +validation, or eviction issue; `unavailable` requires no retained server-auction +records plus a known projection, transport, validation, or eviction issue; and +`complete` requires at least one retained server-auction record without those +capture issues. `correlation_unavailable` and `external_client_side_unobservable` describe interpretation limits and do not change an otherwise complete capture status. -Evicting all received records therefore yields `unavailable` with +Evicting all received server-auction records therefore yields `unavailable` with `record_evicted`, never `not_observed`. Recompute status after both in-memory eviction and snapshot size truncation. The issue array is deduplicated, sorted in enum order, bounded to 16 values, and @@ -1027,8 +1031,12 @@ When valid auction evidence is supplied, TSJS accepts a slot extension only when its canonical token occurs exactly once in both the delivered slot list and the matching auction evidence. A missing, malformed, duplicate, or conflicting token prevents only that sidecar join, -adds `evidence_validation_failed` and `correlation_unavailable`, and does not -drop, reorder, or mutate the ordinary slot or bid. TSJS reads no other extension +adds `correlation_unavailable`, and does not +drop, reorder, or mutate the ordinary slot or bid. It does not add +`evidence_validation_failed`: the evidence member itself validated, so this is +a correlation-layer limit that leaves `capture_status` unchanged per section +9.3. `evidence_validation_failed` is reserved for a supplied evidence member +that fails strict validation. TSJS reads no other extension property. This validation occurs before the slot is handed to the existing GPT initialization path. An absent optional transport member does not trigger missing-token validation; no sidecar is emitted and no capture issue is added. @@ -1124,13 +1132,32 @@ The direct TSJS caller records `evidence_transport_failed` from its existing non-OK, unreadable-JSON, and rejected-`fetch` paths. The Prebid adapter creates one bounded pending transport record after `buildRequests`, keyed by the request's original bid IDs and its normalized unit tokens. `interpretResponse` -consumes it on a readable response; the pinned Prebid `onTimeout` and -`onBidderError` bidder-spec hooks consume it and record +consumes it on a readable response. Add `onTimeout` and `onBidderError` to the +existing bidder-spec object passed to +`pbjs.registerBidAdapter(undefined, ADAPTER_CODE, spec)`; neither hook exists in +the TSJS adapter today. Confirm against the pinned Prebid build that this +registration wraps the spec through `newBidder` and routes both callbacks. +The new hooks consume the pending record and record `evidence_transport_failed` otherwise. Repeated hooks are idempotent. Pending -records are capped at 128 and expire after the configured bid timeout plus five -seconds. Expiry without any supported success/error/timeout hook proves no -transport outcome, so it removes the marker and leaves evidence `not_observed` -rather than inventing a failure. These hooks collect only bounded categories and +records are capped at 128. At record creation, read the effective browser +`pbjs.getConfig('bidderTimeout')` value in milliseconds; use it only when it is +a finite integer from 0 through `2^31 - 1 - 5000`, otherwise use the pinned +Prebid 10.26.0 default of 3000 ms (`DEFAULT_BIDDER_TIMEOUT` in its +`src/config.ts`). Each record +expires after that captured timeout plus 5000 ms, so the delay fits the browser +timer's signed 32-bit limit. Compute the expiry with checked safe-integer +addition to the record's finite nonnegative safe-integer creation timestamp. +If the creation clock or the resulting expiry is invalid or unrepresentable, +decline the trace-only pending record without adding a capture issue or changing +previously collected evidence; that attempt remains `not_observed` and ordinary +bidding continues unchanged. +The TSJS integration's +`merged.timeout`, including the injected `[integrations.prebid].timeout_ms`, +sets this browser `bidderTimeout` when supplied; neither `[auction].timeout_ms` +nor `[auction].auction_timeout_ms` is its source. Expiry without any supported +success/error/timeout hook proves no transport outcome, so it removes the +marker and leaves evidence `not_observed` rather than inventing a failure. +These hooks collect only bounded categories and opaque tokens, never XHR error text or response bodies. For SSAT and SPA page-bids, the diagnostic auction token is also attached to @@ -1145,10 +1172,15 @@ timestamps, implicit array position, ad-unit path, or a best-effort heuristic. TSJS retains at most the newest 16 validated server-auction records and 128 correlation sidecars in memory. It increments checked eviction counters for -older records; the snapshot adds those counts to the matching truncation fields -and emits `record_evicted`, with `partial` if server records remain or -`unavailable` if none remain after snapshot truncation. It performs no storage write until -the explicit snapshot action. +older records; the snapshot adds those counts to the matching truncation fields. +Evicting a server-auction record emits `record_evicted`, with `partial` if +server records remain or +`unavailable` if none remain after snapshot truncation. Evicting a correlation +sidecar increments `omitted_slot_correlations` and adds +`correlation_unavailable` only; it never emits `record_evicted` and never +changes `capture_status`, because server capture is unaffected. This distinction +also applies to sidecars removed during snapshot size truncation. It performs +no storage write until the explicit snapshot action. Requests that fail the applicable trace gate do not mint trace tokens, build trace evidence, add response members, emit sidecars, or install auction-evidence listeners. @@ -1252,15 +1284,22 @@ the complete compact UTF-8 storage wrapper. If it exceeds 512 KiB, it removes the globally oldest GPT request cycles first while retaining the newest cycle for each GPT slot, then the oldest callback issues, then the oldest attribution issues, then the oldest -uncorrelated server auctions, and finally the oldest remaining GPT cycles and -server auctions until the report fits. It records every removal in -`truncation`. A correlated auction and the newest GPT cycle that references it -are retained or removed together once the algorithm reaches correlated server -auctions; every sidecar referencing a removed auction or cycle is removed and -counted. The report must not retain a dangling token while claiming a join. +uncorrelated server auctions, and finally the oldest eligible correlated +server auctions until the report fits. The newest cycle for each GPT slot is a +floor for the whole procedure, not only its first stage: no stage removes a +slot's last remaining cycle. When removing a correlated auction in the final +stage, remove every retained GPT cycle that references it together with the +auction; an auction referenced by any floor cycle is therefore protected from +removal. A correlated auction is eligible for removal only when removing all +its referencing cycles preserves every slot's floor. It records every removal +in `truncation`; every sidecar referencing a removed auction or cycle is removed +and counted. The report must not retain a dangling token while claiming a join. A report that still cannot fit after this bounded procedure fails snapshot -creation. The implementation must include a worst-case fixture proving the -result is bounded. +creation, following the oversized case in section 13. The protected floor and +its correlated auctions can exceed the budget on their own. The implementation +must include a worst-case fixture proving successful reports are bounded, and +one proving the floor-exceeds-budget case fails snapshot creation rather than +emptying the GPT section. All omission counters use checked addition. If any source collection would make a counter exceed `u16::MAX`, projection rejects the source instead of @@ -1548,6 +1587,11 @@ shell or actions. parse ordinary bids unchanged, and add `evidence_validation_failed`. Coverage is `partial` when valid records remain or `unavailable` when none remain, following section 9.3. +- Valid evidence cannot join a missing, malformed, duplicate, or conflicting + slot token, or a correlation sidecar is evicted: add only + `correlation_unavailable`, preserve ordinary slots and bids, and leave server + `capture_status` unchanged. Count evicted sidecars in + `omitted_slot_correlations`. - TS Console capture failure: fail open for advertising and show incomplete coverage in diagnostics. - Storage unavailable or quota exceeded after a valid bounded report exists: @@ -1711,13 +1755,26 @@ results, never a prerequisite for returning them. receives valid evidence and then evicts every server record during size truncation must yield `unavailable` with `record_evicted` and exact omission counts; partial eviction remains `partial`. +- Missing, malformed, duplicate, and conflicting slot extensions with valid + evidence add only `correlation_unavailable` and preserve complete server + capture. Evicting correlation sidecars at the 128-record memory cap or during + snapshot truncation increments `omitted_slot_correlations` exactly and never + adds `record_evicted` or changes server `capture_status`. - Successful responses without an optional transport member add no validation issue; malformed supplied members add `evidence_validation_failed`. Cover both an empty collector and one with retained evidence or earlier failures. - Direct fetch failures and Prebid `interpretResponse`, `onTimeout`, and `onBidderError` paths consume their pending transport record exactly once; capped/expired records and absent hooks follow the specified `not_observed` - behavior without retaining error text or bodies. + behavior without retaining error text or bodies. Exercise the newly added + hooks through the pinned Prebid `registerBidAdapter`/`newBidder` registration + path, not only by calling the spec functions directly. Verify expiry uses the + captured browser `bidderTimeout` plus 5000 ms, including configured values, + the 3000 ms default, missing/invalid-value fallback, the maximum accepted + timeout, and fallback for values that would overflow the browser timer. + Invalid creation clocks and unsafe timestamp addition decline the pending + record without changing bids or inventing a transport failure. Later + configuration changes do not alter an existing record's expiry. - Exact-token correlation joins matching SSAT/SPA server auctions, GPT opportunities, and slot references; unmatched, duplicated, conflicting, missing, and forged tokens stay separate and produce explicit coverage states. No @@ -1740,7 +1797,9 @@ results, never a prerequisite for returning them. - Trace projection replaces the nested GPT pathname with `/[redacted]`, rejects any other stored value, emits `TraceGptDiagnosticsV1`, applies deterministic ordering/truncation, and records exact omission counts in a worst-case 512 KiB - fixture. + fixture. An oversized newest-cycle-per-slot floor, including its protected + correlated auctions, fails snapshot creation without dropping a slot's last + cycle or offering a combined-report export. - Same-tab navigation occurs only after a successful write. - Viewer handles absent optional network facts and every cookie-health state. - Populate every excluded `adManager` field, `previousCreativeId`, slot @@ -1805,8 +1864,12 @@ results, never a prerequisite for returning them. it. - The delivered CSP blocks inline injection, framing, third-party connections, and report-derived executable HTML. -- Immutable asset fixtures prove published v1 bytes never change; changed bytes - require a new URL referenced by the shell. +- A committed digest of each published v1 asset is asserted against the bytes + the build produces, so changing those bytes fails the check. Changed bytes + require a new asset-set URL and its + committed digest, preserving the published URL/digest pairs; a fixture also + asserts the shell references the current asset-set URL. This pins the byte + contract at build time rather than claiming a test can observe future releases. - Inactive publisher traffic has no trace assets, storage access, listeners, or cache-policy change, diagnostic token generation, or trace-auction response extension. From 4d737416004758903f330d3cac82ca3cd1defa48 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 2 Oct 2026 12:18:28 -0500 Subject: [PATCH 123/230] Fix remaining streamed Flight composition regressions --- .../src/integrations/nextjs/mod.rs | 166 +++++++++++++++++- .../integrations/nextjs/rsc_placeholders.rs | 70 +++++++- .../src/integrations/nextjs/script_lexer.rs | 19 +- 3 files changed, 238 insertions(+), 17 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index d68614881..d76b82be1 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -360,7 +360,7 @@ mod tests { #[test] fn brace_slash_division_keeps_flight_lengths_and_both_rewrites() { - let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js", "text": "é😀"}); + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js?id=GTM-MIX1", "text": "é😀"}); let body = data.to_string(); let payload = format!("1:T{:x},{body}", body.len()); let push = format!( @@ -369,13 +369,16 @@ mod tests { ); let mut expected = data; expected["url"] = json!("https://test.example.com/page"); - expected["tag"] = json!("/integrations/google_tag_manager/gtm.js"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js?id=GTM-MIX1"); for prefix in [ "const n = {} / 2;", "const n = {nested: {}} / 2;", "const n = function() {} / 2;", "const n = function named() { return 1 } / 2;", "const n = class { method() { return 1 } } / 2;", + "const n = class extends function() {} {} / 2;", + "const n = class extends function Base() {} {} / 2;", + "const n = class extends (function() {}) {} / 2;", ] { let source = format!("{prefix}{push}"); for split in (0..=source.len()).filter(|split| source.is_char_boundary(*split)) { @@ -537,6 +540,58 @@ mod tests { } } + #[test] + fn trailing_next_f_references_preserve_current_and_later_flight() { + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js?id=GTM-MIX1"}); + let body = data.to_string(); + let payload = format!("1:T{:x},{body}", body.len()); + let push = format!( + "self.__next_f.push([1,{}])", + serde_json::to_string(&payload).expect("should encode Flight payload") + ); + let mut expected = data; + expected["url"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js?id=GTM-MIX1"); + for reference in ["self.__next_f", "window.__next_f", "foreign.__next_f"] { + let source = format!("{push};console.log({reference});"); + for split in 0..=source.len() { + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let first = format!("", &source[split..]); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept script prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should complete Flight with trailing reference"), + ); + let output = String::from_utf8(bytes).expect("should emit UTF-8"); + let payloads = flight_payloads(&output); + assert_eq!(payloads.len(), 2, "should emit both Flight pushes once"); + for payload in payloads { + assert_mixed_t_model(&payload, &expected); + } + assert!( + output.contains(&format!(";console.log({reference});")), + "should retain trailing reference at split {split}" + ); + assert!(!output.contains("__ts_rsc_"), "should resolve placeholders"); + } + let html = format!(""); + for compression in [Compression::None, Compression::Gzip] { + for chunk in [32, 1000, 8192] { + let output = mixed_output_with_compression(&html, compression, chunk); + let payloads = flight_payloads(&output); + assert_eq!(payloads.len(), 2, "should emit both compressed pushes once"); + for payload in payloads { + assert_mixed_t_model(&payload, &expected); + } + } + } + } + } + #[test] fn ordinary_next_f_reference_does_not_hide_gtm() { let script = "var ref=self.__next_f;var tag='http://www.googletagmanager.com/gtm.js';"; @@ -719,6 +774,113 @@ mod tests { ); } + #[test] + fn ordinary_gtm_before_captured_flight_is_chunk_invariant() { + let ordinary = "var tag='http://www.googletagmanager.com/gtm.js';"; + let expected_ordinary = "var tag='/integrations/google_tag_manager/gtm.js';"; + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js?id=GTM-MIX1", "text": "é😀"}); + let body = data.to_string(); + let mut expected = data; + expected["url"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js?id=GTM-MIX1"); + for payload in [format!("0:{body}\n"), format!("1:T{:x},{body}", body.len())] { + let push = format!( + "self.__next_f.push([1,{}])", + serde_json::to_string(&payload).expect("should encode Flight payload") + ); + let html = format!(""); + let whole = mixed_output(&html, 8192); + assert!( + whole.starts_with(&format!("", &push[split..]), + ]; + let mut bytes = Vec::new(); + for (index, fragment) in fragments.iter().enumerate() { + bytes.extend( + processor + .process_chunk(fragment.as_bytes(), index == fragments.len() - 1) + .expect("should complete ordinary and captured Flight source"), + ); + } + assert_eq!( + String::from_utf8(bytes).expect("should emit UTF-8"), + whole, + "should preserve both rewrites at Flight split {split}" + ); + } + for compression in [Compression::None, Compression::Gzip] { + for chunk in [128, 1000, 8192] { + assert_eq!( + mixed_output_with_compression(&html, compression, chunk), + whole, + "should preserve both rewrites for {compression:?} at chunk {chunk}" + ); + } + } + } + } + + #[test] + fn ordinary_gtm_before_flight_fallback_preserves_raw_payloads() { + let ordinary = "var tag='http://www.googletagmanager.com/gtm.js';"; + let payload = "1:Tffff,'http://www.googletagmanager.com/gtm.js?id=GTM-MIX1'"; + let encoded = serde_json::to_string(payload).expect("should encode raw Flight payload"); + for (channel, limit, captured) in [(1, 10000, true), (1, 24, false), (2, 10000, false)] { + let push = format!("self.__next_f.push([{channel},{encoded}])"); + let html = format!(""); + let whole = mixed_output_with_budget(&html, 8192, limit, 10000); + let expected_ordinary = if captured { + ordinary.replace( + "http://www.googletagmanager.com", + "/integrations/google_tag_manager", + ) + } else { + ordinary.to_owned() + }; + assert_eq!( + whole, + format!(""), + "should retain original fallback payload bytes" + ); + for split in 0..=push.len() { + let mut processor = create_html_processor(mixed_config(limit, 10000)); + let fragments = [ + format!("".to_owned(), + ]; + let mut bytes = Vec::new(); + for (index, fragment) in fragments.iter().enumerate() { + bytes.extend( + processor + .process_chunk(fragment.as_bytes(), index == fragments.len() - 1) + .expect("should restore raw Flight without changing payload lengths"), + ); + } + assert_eq!( + String::from_utf8(bytes).expect("should emit UTF-8"), + whole, + "should preserve fallback at channel {channel}, limit {limit}, split {split}" + ); + } + } + } + #[test] fn mixed_flight_first_party_urls_preserve_t_lengths() { for scheme in ["https:", "http:", ""] { diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs index 7abe3aa7c..8b596cbdd 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs @@ -78,7 +78,7 @@ impl NextJsRscPlaceholderRewriter { let mut cursor = 0; let mut trimmed = std::mem::take(&mut state.rsc_receiver_trimmed); let mut queued_bytes = state.captured_payload_bytes; - while code[cursor..].contains("__next_f") { + loop { let remaining = &content[cursor..]; let remaining_code = &code[cursor..]; let head = if trimmed { @@ -86,10 +86,12 @@ impl NextJsRscPlaceholderRewriter { } else { first_qualified_flight_push(remaining_code) }; - if let Some(head) = head - && let Some(arguments) = - INERT_FLIGHT_CONTROL_ARGUMENTS.find(&remaining[head.end()..]) - { + // Property references are ordinary source, including after a push. + // Only another qualified call can require capture or raw fallback. + let Some(head) = head else { + break; + }; + if let Some(arguments) = INERT_FLIGHT_CONTROL_ARGUMENTS.find(&remaining[head.end()..]) { cursor += head.end() + arguments.end(); trimmed = false; continue; @@ -185,7 +187,13 @@ impl NextJsRscPlaceholderRewriter { FragmentCapture::CompleteOwned(complete) => { self.rewrite_complete(&complete, true, state, limit, max_queued_payload_bytes) } - FragmentCapture::Suppress => ScriptRewriteAction::RemoveNode, + FragmentCapture::Suppress => { + // Captured Flight bytes are withheld, so downstream stages see + // only empty source or a released receiver prefix, never a raw + // payload. Let GTM retain and rewrite preceding ordinary source. + state.current_fragment_protected = false; + ScriptRewriteAction::RemoveNode + } FragmentCapture::Restore(restored) => { state.rsc_receiver_trimmed = false; state.bypass_rsc |= state.flight_node_owned; @@ -220,7 +228,7 @@ impl NextJsRscPlaceholderRewriter { remaining = &remaining[end + 1..]; remaining_code = &remaining_code[end + 1..]; receiver_trimmed = false; - if !remaining_code.contains("__next_f") { + if !contains_qualified_flight(remaining_code) { break false; } }; @@ -1074,6 +1082,54 @@ mod tests { ); } + #[test] + fn oversized_payload_with_trailing_reference_keeps_later_capture() { + for trimmed in [false, true] { + for reference in ["self.__next_f", "window.__next_f", "foreign.__next_f"] { + let state = IntegrationDocumentState::default(); + let rewriter = + NextJsRscPlaceholderRewriter::new(Arc::new(NextJsIntegrationConfig { + max_combined_payload_bytes: 100, + ..(*test_config()).clone() + })); + let receiver = if trimmed { + assert_eq!( + rewriter.rewrite("self.", &ctx(false, &state)), + ScriptRewriteAction::Keep, + "should release the qualified receiver" + ); + "" + } else { + "self." + }; + let script = format!( + r#"{receiver}__next_f.push([1,"1:T50,{}"]);console.log({reference});"#, + "x".repeat(80) + ); + assert_eq!( + rewriter.rewrite(&script, &ctx(true, &state)), + ScriptRewriteAction::Keep, + "should retain oversized script" + ); + assert!( + !document_state(&state) + .lock() + .expect("should lock document state") + .bypass_rsc, + "should not bypass a complete group with a trailing reference" + ); + assert!( + matches!( + rewriter + .rewrite(r#"self.__next_f.push([1,"1:T3,abc"])"#, &ctx(true, &state)), + ScriptRewriteAction::Replace(_) + ), + "should still capture the next script" + ); + } + } + } + #[test] fn oversized_partial_header_bypasses_later_continuation() { for suffix in ["1", "1:", "1:T", "1:T2"] { diff --git a/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs index df1aa278a..a4d3822cd 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/script_lexer.rs @@ -275,16 +275,16 @@ impl ScriptLexer { class.parenthesis_depth == self.parentheses.len() && class.brace_depth == self.braces.len() }); - let kind = if class_body { + // A superclass function can open at the class head's + // depth. Its body must not consume the pending class. + let kind = if let Some(body) = pending_body { + body + } else if class_body { self.classes.pop().expect("should have pending class").body + } else if self.statement_start || !self.expression_start { + BraceKind::StatementBlock } else { - pending_body.unwrap_or( - if self.statement_start || !self.expression_start { - BraceKind::StatementBlock - } else { - BraceKind::Object - }, - ) + BraceKind::Object }; if self.braces.len() == MAX_TEMPLATE_DEPTH { self.mode = Mode::Opaque; @@ -458,6 +458,9 @@ mod tests { "class Demo { method() { return 1 } } /self.__next_f.push()/;", "const n = class { method() { return 1 } } / 2;", "const n = class extends mixin({}) {} / 2;", + "const n = class extends function() {} {} / 2;", + "const n = class extends function Base() {} {} / 2;", + "const n = class extends (function() {}) {} / 2;", "const n = class extends (class {}) {} / 2;", "const n = {class: {}, method() { return 1 }} / 2;", "const n = {method() { if (ready) { work() } /self.__next_f.push()/ }} / 2;", From 86695c304806e5a512b8f4809027b897e285ed73 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 2 Oct 2026 12:50:23 -0700 Subject: [PATCH 124/230] Address deployment-documentation review and repin EdgeZero Correct the application-release instructions: the action is `package-application-release-fastly`, and `deploy-fastly` consumes a prebuilt archive rather than building one. Warn that `config push --staging` on Fastly selects neither a different store nor a different key. The physical store is whatever EDGEZERO__STORES__CONFIG____NAME resolves to in the shell running the push, so a staged push into the production store overwrites the live entry. Set the selector explicitly in both examples. Add an upgrade section for services deployed before logical store links. Only a managed deploy creates the `trusted_server_secrets` link, so `fastly compute publish` activates a version whose link is missing and every publisher route returns 500 while /health still answers 200. Replace that command in the getting-started guide and AGENTS.md, and annotate the manifest deploy command that EdgeZero never runs for a store-declaring app. Fix the Fastly smoke script. It appended and stripped `ts_secrets` blocks, which nothing opens now, and the base manifest defines the same three keys under indented headers, so removing an appended block still left a resolvable value and no missing-secret case could fail. Match either indentation and strip the base entries on copy. Repin to upstream c2e862f4, eight commits ahead of 12c3215c, and update the test for the key-selector change it brings: with no --key the CLI now writes to EDGEZERO__STORES__CONFIG____KEY instead of ignoring it. Also correct the contradicting staging-key sentence in the EdgeZero guide, the design spec's branch wording, and two stale test messages. --- AGENTS.md | 5 ++- Cargo.lock | 16 ++++---- Cargo.toml | 12 +++--- crates/trusted-server-cli/src/run.rs | 4 +- .../tests/config_store_defaults.rs | 16 ++++++-- docs/guide/cli.md | 41 ++++++++++--------- docs/guide/edgezero.md | 6 ++- docs/guide/fastly.md | 37 +++++++++++++---- docs/guide/getting-started.md | 14 ++++++- ...-edgezero-fastly-store-selectors-design.md | 7 ++-- edgezero.toml | 5 +++ scripts/smoke-fastly.sh | 30 +++++++++++--- 12 files changed, 135 insertions(+), 58 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b6e61ecf9..95cdaa531 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -56,8 +56,9 @@ cargo build --package trusted-server-adapter-fastly --release --target wasm32-wa # Run locally with Fastly simulator fastly compute serve -# Deploy to Fastly -fastly compute publish +# Deploy to Fastly (managed path; links logical stores to the version) +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" # Run Axum dev server (native — no Viceroy) cargo run -p trusted-server-adapter-axum diff --git a/Cargo.lock b/Cargo.lock index 6f18bb329..f5bc29768 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1428,7 +1428,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "serde", "serde_json", @@ -1440,7 +1440,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "anyhow", "async-trait", @@ -1468,7 +1468,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "anyhow", "async-trait", @@ -1491,7 +1491,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "anyhow", "async-stream", @@ -1520,7 +1520,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "anyhow", "async-trait", @@ -1547,7 +1547,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "chrono", "clap", @@ -1572,7 +1572,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "anyhow", "async-compression", @@ -1603,7 +1603,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?rev=12c3215c2637d961a27eefa12e235033fadb4c4a#12c3215c2637d961a27eefa12e235033fadb4c4a" +source = "git+https://github.com/stackpop/edgezero?rev=c2e862f436421cdcfbac620c06995f262d6788c9#c2e862f436421cdcfbac620c06995f262d6788c9" dependencies = [ "log", "proc-macro2", diff --git a/Cargo.toml b/Cargo.toml index 75060c4c7..ec7aab77f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -55,12 +55,12 @@ cssparser = "0.36" derive_more = { version = "2.0", features = ["display", "error"] } directories = "5" ed25519-dalek = { version = "2.2", features = ["rand_core"] } -edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } -edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } -edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } -edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } -edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a" } -edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "12c3215c2637d961a27eefa12e235033fadb4c4a", default-features = false } +edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9", default-features = false } +edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9", default-features = false } +edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9", default-features = false } +edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9", default-features = false } +edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9" } +edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "c2e862f436421cdcfbac620c06995f262d6788c9", default-features = false } env_logger = "0.11" error-stack = "0.6" esi = "0.7.2" diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index e06bb998c..eb0099b65 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -495,7 +495,7 @@ mod tests { let Command::Config(ConfigCommand::Push(push)) = args.command else { panic!("expected config push command"); }; - assert!(push.staging, "should target the derived staging key"); + assert!(push.staging, "should target the staging publication flow"); Args::try_parse_from([ "ts", @@ -518,7 +518,7 @@ mod tests { }; assert!( diff.staging, - "should compare against the derived staging key" + "should compare against the staging publication flow" ); Args::try_parse_from([ diff --git a/crates/trusted-server-cli/tests/config_store_defaults.rs b/crates/trusted-server-cli/tests/config_store_defaults.rs index 23c5f86e9..6b7998dec 100644 --- a/crates/trusted-server-cli/tests/config_store_defaults.rs +++ b/crates/trusted-server-cli/tests/config_store_defaults.rs @@ -133,8 +133,14 @@ fn config_push_resolves_the_physical_name_and_explicit_key() { ); } +// The canonical `__KEY` selector is the push destination when `--key` is absent. +// Under EdgeZero v0.0.8 the CLI ignored it and always wrote the logical id; the +// pinned revision returns the selector verbatim so the pushed entry and the +// runtime that reads it cannot diverge. The Fastly runtime reads the logical id +// for every target, so an operator deploying to Fastly should leave this unset, +// which the Fastly guide now states. #[test] -fn config_push_does_not_use_the_runtime_key_override_without_the_key_flag() { +fn config_push_writes_the_runtime_key_override_without_the_key_flag() { let project = project(); let key_var = format!( "EDGEZERO__STORES__CONFIG__{}__KEY", @@ -145,7 +151,11 @@ fn config_push_does_not_use_the_runtime_key_override_without_the_key_flag() { let entries = stored_entries(&project); assert_eq!(entries.len(), 1); assert!( - entries.contains_key(CONFIG_BLOB_KEY), - "runtime-only key override should not move the CLI's write destination" + entries.contains_key("active_config"), + "the canonical key selector should be the push destination" + ); + assert!( + !entries.contains_key(CONFIG_BLOB_KEY), + "the logical id should not also be written" ); } diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 9fb62380d..2d17a62fa 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -224,12 +224,13 @@ EDGEZERO_MANIFEST="/edgezero.toml" \ ts deploy --adapter fastly --service-id --application-release "" -- --comment "release" ``` -`` is the extracted immutable application release that EdgeZero's -`package-fastly-application-release` action produces from a `ts build`. The -EdgeZero `deploy-fastly` action performs both steps, selects the manifest, and -supplies the release root; see EdgeZero's GitHub Actions deployment guide -(`docs/guide/deploy-github-actions.md` in the EdgeZero repository) for the -producer and consumer workflow. +`` is an extracted immutable application release. Build the +application CLI and Fastly package, then use EdgeZero's +`package-application-release-fastly` action to assemble the release archive. +The `deploy-fastly` action consumes that prebuilt archive, verifies and extracts +it, selects the manifest, and supplies the release root. See EdgeZero's GitHub +Actions deployment guide (`docs/guide/deploy-github-actions.md` in the EdgeZero +repository) for the producer and consumer workflow. A staged deploy selects the physical Config Store from the staging environment and links it to the staged version under the logical store ID. The staged @@ -238,24 +239,26 @@ the production config blob there. Push the staged config before probing the staged version: ```bash -ts config push --adapter fastly --staging -ts config diff --adapter fastly --staging +EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME= \ + ts config push --adapter fastly --staging +EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME= \ + ts config diff --adapter fastly --staging ``` The config key is fixed on Fastly: production, staging, and local Viceroy all -read ``. Staging isolation comes from the physical store the -staging environment selects with `EDGEZERO__STORES__CONFIG____NAME`, never -from a different key. Production and staging may select the same physical -Config Store or different stores. After `ts config push --staging`, the staged -binary reads `` in the store selected by the staging -environment, while the active production version continues to read the same -key in its own store. +read ``. `--staging` does not choose a different store or +key: the physical store is whatever `EDGEZERO__STORES__CONFIG____NAME` +resolves to in the shell or deploy job that runs the push, falling back to the +logical ID when it is unset. If production and staging select the same +physical Config Store, a staged push overwrites the production entry and +production instances load it on their next start. Give staging its own +physical Config Store and set its selector explicitly when pushing, as the +commands above do. `--staging` on `config push` / `config diff` writes and compares the -`` key in the physical store selected by the staging -environment. It is mutually exclusive with `--key`: Fastly accepts only the -logical store ID as the key for every target, so an explicit key would be -written where nothing reads it. +`` key in the physical store selected by the current +environment. It is mutually exclusive with `--key`, and on Fastly any `--key` +other than the logical store ID is rejected for every target, staged or not. Inspect and verify deployments with the deploy lifecycle commands. All three are Fastly-only — the axum, cloudflare, and spin adapters reject them: diff --git a/docs/guide/edgezero.md b/docs/guide/edgezero.md index 30d56856b..a3630763a 100644 --- a/docs/guide/edgezero.md +++ b/docs/guide/edgezero.md @@ -16,8 +16,10 @@ version operations use the adapter commands declared in `edgezero.toml`. Configuration is startup state, not a live control plane. A successful push does not change a running instance until the target's restart or deployment path loads that value. Use `--staging` only with the corresponding staging -deployment flow: it writes `LOGICAL_ID_staging` in the same physical store, -while production continues to read the ordinary logical key. +deployment flow. On Fastly the entry key is always the logical store ID, so +staging isolation comes from selecting a different physical store with +`EDGEZERO__STORES__CONFIG____NAME`; see +[the CLI guide](/guide/cli#lifecycle-commands). ## Logical and physical stores diff --git a/docs/guide/fastly.md b/docs/guide/fastly.md index 05e2ee974..fa433655b 100644 --- a/docs/guide/fastly.md +++ b/docs/guide/fastly.md @@ -319,11 +319,9 @@ declare `trusted_server_secrets`. Local Viceroy configuration exposes the store under the logical ID directly; see `[local_server.secret_stores]` in `fastly.toml`. -The same runtime mapping mechanism applies to app-config stores. See -[Fastly runtime config stores](/guide/configuration#fastly-runtime-config-store) -for the initial provisioning and linking sequence, subsequent CLI pushes, and -precautions when changing a live mapping. A process-environment override alone -does not configure the Fastly runtime. +The app-config store is selected the same way, through +`EDGEZERO__STORES__CONFIG__TRUSTED_SERVER_CONFIG__NAME` in the deployment +environment. No runtime mapping is stored in a Config Store. Create the separate request-signing store when that feature is enabled: @@ -334,6 +332,29 @@ fastly secret-store create --name signing_keys The deployment creates the `trusted_server_secrets` link itself from the selected physical store. Do not create that link by hand. +### Upgrading from EdgeZero v0.0.8 + +Services deployed before this change link the physical store directly, such as +`ts_secrets`, together with an `edgezero_runtime_env` Config Store. This binary +opens `trusted_server_secrets` instead, and only a managed deployment creates +that link. A plain `fastly compute publish` clones the active version's links, +so it activates a version whose link is missing: app config fails to load and +every publisher route returns 500 while `/health` keeps answering 200. + +When upgrading an existing service: + +- Deploy only through the managed path, `ts deploy --adapter fastly +--application-release ` or EdgeZero's `deploy-fastly` action, so + the `trusted_server_secrets` link exists before activation. +- Re-push staging app config under the logical key into the staging Config + Store. Entries previously pushed under `_staging` are + ignored. +- Remove any `EDGEZERO__STORES__CONFIG____KEY` selectors. The CLI now + writes to that selector when `--key` is absent, instead of ignoring it, so a + leftover selector pushes app config to a key the Fastly runtime never reads. +- Optionally delete the stale `edgezero_runtime_env` link and any `*_staging` + config entries once no active version reads them. + ## Create EC KV Store Edge Cookie flows require one KV store: @@ -356,7 +377,8 @@ passphrase = "ec_passphrase" ec_store = "ec_identity_store" ``` -Store the high-entropy passphrase under that key in `ts_secrets`. The resolved +Store the high-entropy passphrase under that key in the physical store +selected for `trusted_server_secrets`. The resolved value, rather than the key name, must contain at least 32 characters. Pipe the value over standard input so it never appears in the process argument list: @@ -394,7 +416,8 @@ Run the repository smoke from a clean shell: The script creates an isolated application config, applies its publisher-origin overrides, and runs strict validation. It then executes `ts config push --adapter fastly --local`, adds all three required entries to -`[local_server.secret_stores.ts_secrets]`, and starts `fastly compute serve` +`[local_server.secret_stores.trusted_server_secrets]`, and starts +`fastly compute serve` through Viceroy. The required keys are `handler_password`, `publisher_proxy_secret`, and `ec_passphrase`. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 83b8ef105..267bdbb1f 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -207,10 +207,22 @@ Restart or redeploy instances after secret rotation. See ## Deploy to Fastly +Trusted Server declares Config, KV, and Secret Stores, so deployment must go +through EdgeZero's managed path. That path links each selected physical store +to the service version under its logical ID; `fastly compute publish` clones +the active version's links instead and activates a version whose logical links +are missing. + ```bash -fastly compute publish +EDGEZERO_MANIFEST="/edgezero.toml" \ + ts deploy --adapter fastly --service-id --application-release "" ``` +See [the Fastly guide](/guide/fastly#upgrading-from-edgezero-v0-0-8) when +upgrading a service that was deployed before logical store links, and +[the CLI guide](/guide/cli#lifecycle-commands) for where `` +comes from. + ## Next Steps - Learn about [Edge Cookies](/guide/edge-cookies) diff --git a/docs/superpowers/specs/2026-09-16-edgezero-fastly-store-selectors-design.md b/docs/superpowers/specs/2026-09-16-edgezero-fastly-store-selectors-design.md index ae4c438e5..428a08078 100644 --- a/docs/superpowers/specs/2026-09-16-edgezero-fastly-store-selectors-design.md +++ b/docs/superpowers/specs/2026-09-16-edgezero-fastly-store-selectors-design.md @@ -19,10 +19,11 @@ remove guidance for the superseded service-scoped and runtime-selector models. ## Design -All EdgeZero workspace dependencies will temporarily use the upstream branch -`fix/fastly-environment-store-selectors`. Keeping the dependencies on one branch +All EdgeZero workspace dependencies are pinned with `rev =` to the immutable +commit `c2e862f436421cdcfbac620c06995f262d6788c9` on the upstream branch +`fix/fastly-environment-store-selectors`. Pinning every crate to one commit ensures the adapter, CLI, core, and platform-specific crates resolve to the same -revision while PR 381 is under review. A follow-up will replace the branch with +revision while PR 381 is under review. Issue #1195 tracks replacing the rev with the release tag that contains the merged change. The Fastly entry point will stop reading `edgezero_runtime_env`. It will open diff --git a/edgezero.toml b/edgezero.toml index b40b20281..54d3578a7 100644 --- a/edgezero.toml +++ b/edgezero.toml @@ -48,6 +48,11 @@ auth-logout = "fastly profile delete" auth-status = "fastly profile list" build = "cargo build --bin trusted-server-adapter-fastly --release --target wasm32-wasip1 --color always" serve = "fastly compute serve" +# Unused by `ts deploy`: this app declares stores, so EdgeZero always takes the +# managed deployment path, which links each selected physical store to the +# service version under its logical ID. Publishing with the Fastly CLI directly +# clones the active version's links and omits the logical links the runtime +# opens. deploy = "fastly compute publish" # -- Axum (native dev server) ----------------------------------------------- diff --git a/scripts/smoke-fastly.sh b/scripts/smoke-fastly.sh index c2e8b01b9..65635c2d2 100755 --- a/scripts/smoke-fastly.sh +++ b/scripts/smoke-fastly.sh @@ -31,7 +31,27 @@ trap cleanup EXIT INT TERM mkdir -p "$FASTLY_PROJECT" cp "$REPO_ROOT/edgezero.toml" "$EDGEZERO_MANIFEST" -cp "$REPO_ROOT/fastly.toml" "$FASTLY_MANIFEST" +# The base manifest defines the same secret keys this smoke appends. Drop its +# entries on copy, or removing an appended block would still leave a resolvable +# value and the missing-secret cases could not fail. +# The base manifest indents its table headers; the blocks this smoke appends +# are unindented. Match either form. +awk ' + /^[[:space:]]*\[\[local_server\.secret_stores\.trusted_server_secrets\]\][[:space:]]*$/ { + if ((getline key_line) <= 0 || (getline data_line) <= 0) { + exit 2 + } + removed++ + next + } + { print } + END { + if (removed < 1) { + print "expected at least one base secret block to strip; found 0" > "/dev/stderr" + exit 1 + } + } +' "$REPO_ROOT/fastly.toml" >"$FASTLY_MANIFEST" ln -s "$REPO_ROOT/crates" "$FASTLY_PROJECT/crates" smoke_resolve_ts_binary "$REPO_ROOT" @@ -90,7 +110,7 @@ write_without_secret() { local destination="$2" local missing_key="$3" awk -v missing_key="$missing_key" ' - $0 == "[[local_server.secret_stores.ts_secrets]]" { + /^[[:space:]]*\[\[local_server\.secret_stores\.trusted_server_secrets\]\][[:space:]]*$/ { header = $0 if ((getline key_line) <= 0 || (getline data_line) <= 0) { exit 2 @@ -131,15 +151,15 @@ run_case missing-config "$BASE_PORT" 500 \ ) cat >>"$FASTLY_MANIFEST" < Date: Sat, 3 Oct 2026 17:52:08 +0530 Subject: [PATCH 125/230] Isolate documentation proposal builds from publication credentials Split the docs gates into a validate job with a read-only token and no persisted credentials, since VitePress executes Vue in generated Markdown. The publish job now re-checks the path allowlist on a fresh checkout without building. Every job runs the scripts from the workflow's revision against a separate checkout of the merge commit, so historical merges can be retried. Queue every pending run instead of cancelling all but one, and have Copilot inspect the full pushed range so merge commits and multi-commit pushes are not missed. A retry with an unchanged tree now reuses the pushed commit and posts the review if an earlier run failed before posting it. Signed-off-by: dhruv8sh --- .github/workflows/docs-proposal.yml | 96 ++++++++++++++++--- ...6-10-01-docs-proposal-automation-design.md | 82 +++++++++++----- scripts/README.md | 45 +++++---- scripts/docs-proposal/lib.sh | 14 +++ scripts/docs-proposal/prompt.md | 10 +- scripts/docs-proposal/propose.sh | 19 ++-- scripts/docs-proposal/publish.sh | 40 ++++---- scripts/docs-proposal/test.sh | 48 ++++++++-- scripts/docs-proposal/validate.sh | 27 ++++++ 9 files changed, 288 insertions(+), 93 deletions(-) create mode 100755 scripts/docs-proposal/validate.sh diff --git a/.github/workflows/docs-proposal.yml b/.github/workflows/docs-proposal.yml index 956a192dd..bb5b9cb3d 100644 --- a/.github/workflows/docs-proposal.yml +++ b/.github/workflows/docs-proposal.yml @@ -23,13 +23,20 @@ on: permissions: {} -# Sequential merges are proposed one at a time, in order. +# Sequential merges are proposed one at a time, in order. The default queue +# keeps only one pending run and cancels older ones, so keep them all. concurrency: group: docs-proposal cancel-in-progress: false + queue: max +# Each job runs the proposal scripts from the workflow's own revision in +# `tools/` against a checkout of the merge commit in `source/`, so retrying a +# merge that predates this tooling, or a later fix to it, still works. A push +# that adds several commits is inspected from the previous main head. env: MERGE_SHA: ${{ inputs.sha || github.sha }} + BASE_SHA: ${{ github.event_name == 'push' && github.event.before || '' }} jobs: # Copilot runs here with a read-only token; it cannot push or open PRs. @@ -39,72 +46,133 @@ jobs: contents: read copilot-requests: write steps: - - name: Checkout + - name: Checkout tools + uses: actions/checkout@v4 + with: + ref: ${{ github.workflow_sha }} + path: tools + persist-credentials: false + + - name: Checkout merge commit uses: actions/checkout@v4 with: ref: ${{ env.MERGE_SHA }} + path: source fetch-depth: 0 persist-credentials: false - name: Retrieve Node.js version id: node-version - run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + run: echo "node-version=$(grep '^nodejs ' tools/.tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: ${{ steps.node-version.outputs.node-version }} cache: "npm" - cache-dependency-path: docs/package-lock.json + cache-dependency-path: source/docs/package-lock.json - name: Install Copilot CLI run: npm install -g @github/copilot@1.0.90 - name: Propose documentation edits + working-directory: source env: GITHUB_TOKEN: ${{ github.token }} - run: scripts/docs-proposal/propose.sh "$MERGE_SHA" .docs-proposal + run: ../tools/scripts/docs-proposal/propose.sh "$MERGE_SHA" .docs-proposal "$BASE_SHA" - name: Upload proposal uses: actions/upload-artifact@v4 with: name: docs-proposal - path: .docs-proposal/ + path: source/.docs-proposal/ include-hidden-files: true if-no-files-found: error - # Copilot does not run here; this job validates and publishes the patch. - publish: + # VitePress executes Vue in Markdown during the build, so the docs gates run + # here with no write token or persisted credentials. + validate: needs: propose runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout tools + uses: actions/checkout@v4 + with: + ref: ${{ github.workflow_sha }} + path: tools + persist-credentials: false + + - name: Checkout merge commit + uses: actions/checkout@v4 + with: + ref: ${{ env.MERGE_SHA }} + path: source + fetch-depth: 0 + persist-credentials: false + + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' tools/.tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + cache: "npm" + cache-dependency-path: source/docs/package-lock.json + + - name: Download proposal + uses: actions/download-artifact@v4 + with: + name: docs-proposal + path: source/.docs-proposal + + - name: Validate proposal + working-directory: source + run: ../tools/scripts/docs-proposal/validate.sh "$MERGE_SHA" .docs-proposal + + # Copilot does not run here, and neither does any proposal content: this job + # re-checks the path allowlist on a fresh checkout and publishes the patch. + publish: + needs: validate + runs-on: ubuntu-latest permissions: contents: write pull-requests: write steps: - - name: Checkout + - name: Checkout tools + uses: actions/checkout@v4 + with: + ref: ${{ github.workflow_sha }} + path: tools + persist-credentials: false + + - name: Checkout merge commit uses: actions/checkout@v4 with: ref: ${{ env.MERGE_SHA }} + path: source fetch-depth: 0 - name: Retrieve Node.js version id: node-version - run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" + run: echo "node-version=$(grep '^nodejs ' tools/.tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT" - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: ${{ steps.node-version.outputs.node-version }} - cache: "npm" - cache-dependency-path: docs/package-lock.json - name: Download proposal uses: actions/download-artifact@v4 with: name: docs-proposal - path: .docs-proposal + path: source/.docs-proposal - name: Publish proposal pull request + working-directory: source env: GH_TOKEN: ${{ github.token }} - run: scripts/docs-proposal/publish.sh "$MERGE_SHA" .docs-proposal + run: ../tools/scripts/docs-proposal/publish.sh "$MERGE_SHA" .docs-proposal diff --git a/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md b/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md index 545adb58d..a8a3c8f53 100644 --- a/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md +++ b/docs/superpowers/specs/2026-10-01-docs-proposal-automation-design.md @@ -33,8 +33,14 @@ only ever writes to a dedicated proposal branch and a pull request. ## Workflow shape -`.github/workflows/docs-proposal.yml` has two jobs. Copilot never holds a -token that can push or open pull requests. +`.github/workflows/docs-proposal.yml` has three jobs. Copilot never holds a +token that can push or open pull requests, and proposal content is never built +or executed in a job that can. + +Every job checks out the workflow's own revision into `tools/` and the merge +commit into `source/`, then runs the scripts from `tools/` against `source/`. +Retrying a merge that predates this tooling therefore works, and a retry runs +the current scripts rather than historical ones. ### Trigger @@ -44,19 +50,24 @@ token that can push or open pull requests. - `workflow_dispatch` with a required `sha` input, for retries and for the first end-to-end validation. Both scripts require a full 40-character SHA reachable from `origin/main`. -- `concurrency: { group: docs-proposal, cancel-in-progress: false }` so - sequential merges are processed one at a time, in order. +- `concurrency` with `group: docs-proposal`, `cancel-in-progress: false`, and + `queue: max` so sequential merges are processed one at a time, in order. + The default queue keeps one pending run and cancels the rest, which would + skip merges. ### Job `propose` Permissions: `contents: read`, `copilot-requests: write`. -1. Check out the merge commit with full history. +1. Check out the tools and the merge commit with full history. 2. Install Copilot CLI (`@github/copilot`) at a pinned version. -3. `scripts/docs-proposal/propose.sh .docs-proposal`: +3. `scripts/docs-proposal/propose.sh .docs-proposal `: 1. Renders `.docs-proposal/prompt.md` from `scripts/docs-proposal/prompt.md`, - the merge SHA, its subject, and the Sources-of-truth table from the - refresh design. + the merge SHA, its base, its subject, and the Sources-of-truth table from + the refresh design. The base is the push's previous `main` head, or the + first parent for a manual dispatch, so Copilot inspects + `git diff `: a multi-commit push is covered in full, and a + clean merge commit is not hidden by `git show`'s empty combined diff. 2. Runs `copilot -p` with `--no-ask-user`, `--allow-tool=write`, and `git show`, `git diff`, `git log`, and `git grep` shell tools; `git push` and `git commit` are denied, and file reads need no permission. Copilot @@ -73,9 +84,24 @@ Permissions: `contents: read`, `copilot-requests: write`. that justifies it and a one-sentence reason. 4. Upload the `.docs-proposal/` directory as an artifact. +### Job `validate` + +Permissions: `contents: read`; checkouts do not persist credentials. +VitePress executes Vue in Markdown during the build, so generated content runs +only here. + +`scripts/docs-proposal/validate.sh .docs-proposal`: + +1. Empty diff: exit successfully. +2. Apply `proposal.patch` to the merge commit and reject the proposal if any + changed path is outside `docs/guide/**` or `docs/index.md`. +3. In `docs/`: `npm ci`, `npm run lint`, `npm run format`, `npm run build`. + Any failure fails the workflow; no pull request is offered. + ### Job `publish` -Permissions: `contents: write`, `pull-requests: write`. Copilot does not run. +Permissions: `contents: write`, `pull-requests: write`. Copilot does not run, +and nothing installs, builds, or executes proposal content. `scripts/docs-proposal/publish.sh .docs-proposal`: @@ -83,22 +109,23 @@ Permissions: `contents: write`, `pull-requests: write`. Copilot does not run. `docs/auto/`, leave it alone and exit successfully: a maintainer's decision is never reopened or recreated. 2. Empty diff: close any open pull request for that branch with a comment, - then exit successfully. Otherwise apply `proposal.patch` to the merge - commit. -3. Reject the proposal if any changed path is outside `docs/guide/**` or + then exit successfully. Otherwise apply `proposal.patch` to a fresh + checkout of the merge commit. +3. Reject the proposal again if any changed path is outside `docs/guide/**` or `docs/index.md`. -4. In `docs/`: `npm ci`, `npm run lint`, `npm run format`, `npm run build`. - Any failure fails the job; no pull request is offered. -5. Commit as `github-actions[bot]` on `docs/auto/`. If the remote - branch already exists and its tree equals the new tree, stop: the proposal - is unchanged and nothing is re-posted. Otherwise force-push. -6. Find the originating pull request with +4. Commit as `github-actions[bot]` on `docs/auto/`. If the remote + branch already exists and its tree equals the new tree, reuse the remote + commit and skip the push, so a retry resumes any later step that failed. + Otherwise force-push. +5. Find the originating pull request with `gh api repos/{owner}/{repo}/commits//pulls`. Create the pull request, or edit the existing one, with a body that links the merge commit and the originating pull request, includes `rationale.md`, and states that a human must verify the prose against the code before merging. -7. `node scripts/docs-proposal/hunks.mjs review` builds review comments and - posts one `COMMENT` review through `gh api`. +6. If `github-actions[bot]` already reviewed the pushed commit, stop: nothing + is re-posted. Otherwise `node scripts/docs-proposal/hunks.mjs review` + builds review comments and posts one `COMMENT` review through `gh api` + against the pushed commit. ## Per-hunk review @@ -130,7 +157,7 @@ Documented in `scripts/README.md`: - Repository setting "Allow GitHub Actions to create and approve pull requests" is enabled. - Pull requests opened with `GITHUB_TOKEN` do not trigger other workflows. - The `publish` job runs the docs gates itself; maintainers re-run regular CI + The `validate` job runs the docs gates itself; maintainers re-run regular CI by pushing to the branch if needed. ## Security @@ -140,7 +167,10 @@ Documented in `scripts/README.md`: permissions. - Input is code already merged to `main`, so no untrusted fork content reaches the agent. -- The path allowlist is enforced by `publish.sh`, not by the prompt. +- The path allowlist is enforced by `validate.sh` and again by `publish.sh`, + not by the prompt. +- Generated Markdown is built only in the `validate` job, which has a + read-only token and no persisted credentials. - Proposed prose must use fictional `example.com` values, per `CLAUDE.md`; the reviewer verifies this before merging. @@ -149,9 +179,11 @@ Documented in `scripts/README.md`: - `node --test scripts/docs-proposal/hunks.test.mjs` covers hunk listing, every row of the per-hunk table, unknown evidence ids, and the review payload. - `scripts/docs-proposal/test.sh` stubs `gh` and `npm` and pushes to a - temporary bare repository to cover the empty diff, closing a stale - proposal, a disallowed path, first publish, an unchanged retry, an updated - proposal, a closed proposal, and PR body rendering. + temporary bare repository to cover validation of empty, disallowed, and + allowed proposals, the empty diff, closing a stale proposal, a disallowed + path, first publish without a build, an unchanged retry, an updated proposal + whose review fails and is resumed without a second push, a closed proposal, + and PR body rendering. - `shellcheck` passes for every script. - The `docs-proposal-scripts` job in `format.yml` runs all three on every pull request. diff --git a/scripts/README.md b/scripts/README.md index 8c2081776..2ce1fc5e2 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -3,24 +3,25 @@ Run these scripts from the repository root. They fail on command errors unless their documented orchestration handles a probe deliberately. -| Script | Inputs and prerequisites | Side effects and cleanup | -| ----------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `batch-sync.sh` | Endpoint, bearer token, EC ID, partner UID; `curl` and Python | Sends one authenticated batch-sync request. Its temporary response file is removed on exit. | -| `benchmark.sh` | A running server; `curl`, `bc`, and optionally `hey` | May install `hey` through Homebrew. `--save` writes under `benchmark-results/`; it does not stop the server. | -| `profile.sh` | Fastly CLI, Rust WASM target, `curl`; endpoint and request options | Builds and starts the Fastly app, stops the owned process, and retains a profile under `benchmark-results/profiles/`. `--open` launches the local viewer. | -| `generate-integration-viceroy-configs.sh` | Rust toolchain; optional origin port and artifact directory | Builds the native generator and writes Viceroy config under `target/integration-test-artifacts/`; generated files persist. | -| `integration-tests.sh` | Docker, Viceroy, Rust WASM target, pinned Node | Builds WASM/native artifacts and two Docker images, generates Viceroy config, and runs native integration tests serially. Build products and images persist. | -| `integration-tests-browser.sh` | The integration prerequisites plus npm and Playwright | Installs package/browser dependencies, builds fixtures and images, runs both browser suites, and stops matching test containers on exit. Build and npm artifacts persist. | -| `smoke-axum.sh` | `cargo`, `curl`, Python | Uses an isolated temporary config/store and stub origin; stops owned processes and removes its workspace. | -| `smoke-fastly.sh` | Fastly CLI, `cargo`, `curl`, Python | Uses an isolated Fastly project and application config; stops owned processes and removes its workspace. | -| `smoke-cloudflare.sh` | Pinned Wrangler, `cargo`, `curl`, `jq`, and Python | Uses isolated Wrangler manifests, KV state, ports, and logs; stops owned processes and removes its workspace. | -| `smoke-spin.sh` | Spin CLI, `cargo`, `curl`, and Python | Uses an isolated Spin manifest and SQLite KV store; stops owned processes and removes its workspace. | -| `smoke-common.sh` | Sourced by the four smoke scripts | Defines bounded port, process, config, secret, and response assertions. Do not invoke it as a standalone smoke. | -| `template-cache-local-test.sh` | Viceroy, Node, OpenSSL, `curl`, `lsof`; optional ports and mode | Builds local artifacts, generates certificates and fixtures in a temporary directory, stops owned servers, and removes the directory. | -| `test-cli.sh` | Rustup and the host toolchain; optional host triple | May install the selected Rust target, then runs native CLI and browser-audit tests. Cargo artifacts persist. | -| `docs-proposal/propose.sh` | Merge SHA on `main`, work dir; Copilot CLI, Node, `npm` | Runs Copilot CLI to edit `docs/guide/**` or `docs/index.md`, formats docs, and writes a patch, rationale, and evidence into the work dir. Leaves the edits staged. | -| `docs-proposal/publish.sh` | Merge SHA, work dir from `propose.sh`; `gh` with write `GH_TOKEN` | Applies the patch, runs the docs gates, force-pushes `docs/auto/`, opens or edits its PR, and posts a per-hunk review. Never writes `main`. | -| `docs-proposal/test.sh` | Git and Node | Tests the helpers and publish flow against a temporary repository with stubbed `gh` and `npm`; removes it on exit. | +| Script | Inputs and prerequisites | Side effects and cleanup | +| ----------------------------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `batch-sync.sh` | Endpoint, bearer token, EC ID, partner UID; `curl` and Python | Sends one authenticated batch-sync request. Its temporary response file is removed on exit. | +| `benchmark.sh` | A running server; `curl`, `bc`, and optionally `hey` | May install `hey` through Homebrew. `--save` writes under `benchmark-results/`; it does not stop the server. | +| `profile.sh` | Fastly CLI, Rust WASM target, `curl`; endpoint and request options | Builds and starts the Fastly app, stops the owned process, and retains a profile under `benchmark-results/profiles/`. `--open` launches the local viewer. | +| `generate-integration-viceroy-configs.sh` | Rust toolchain; optional origin port and artifact directory | Builds the native generator and writes Viceroy config under `target/integration-test-artifacts/`; generated files persist. | +| `integration-tests.sh` | Docker, Viceroy, Rust WASM target, pinned Node | Builds WASM/native artifacts and two Docker images, generates Viceroy config, and runs native integration tests serially. Build products and images persist. | +| `integration-tests-browser.sh` | The integration prerequisites plus npm and Playwright | Installs package/browser dependencies, builds fixtures and images, runs both browser suites, and stops matching test containers on exit. Build and npm artifacts persist. | +| `smoke-axum.sh` | `cargo`, `curl`, Python | Uses an isolated temporary config/store and stub origin; stops owned processes and removes its workspace. | +| `smoke-fastly.sh` | Fastly CLI, `cargo`, `curl`, Python | Uses an isolated Fastly project and application config; stops owned processes and removes its workspace. | +| `smoke-cloudflare.sh` | Pinned Wrangler, `cargo`, `curl`, `jq`, and Python | Uses isolated Wrangler manifests, KV state, ports, and logs; stops owned processes and removes its workspace. | +| `smoke-spin.sh` | Spin CLI, `cargo`, `curl`, and Python | Uses an isolated Spin manifest and SQLite KV store; stops owned processes and removes its workspace. | +| `smoke-common.sh` | Sourced by the four smoke scripts | Defines bounded port, process, config, secret, and response assertions. Do not invoke it as a standalone smoke. | +| `template-cache-local-test.sh` | Viceroy, Node, OpenSSL, `curl`, `lsof`; optional ports and mode | Builds local artifacts, generates certificates and fixtures in a temporary directory, stops owned servers, and removes the directory. | +| `test-cli.sh` | Rustup and the host toolchain; optional host triple | May install the selected Rust target, then runs native CLI and browser-audit tests. Cargo artifacts persist. | +| `docs-proposal/propose.sh` | Merge SHA on `main`, work dir, optional base SHA; Copilot CLI, Node, `npm` | Runs Copilot CLI to edit `docs/guide/**` or `docs/index.md`, formats docs, and writes a patch, rationale, and evidence into the work dir. Leaves the edits staged. | +| `docs-proposal/validate.sh` | Merge SHA, work dir from `propose.sh`; Node, `npm`; no write credentials | Applies the patch, rejects paths outside the allowlist, and runs the docs gates. Leaves the edits staged. | +| `docs-proposal/publish.sh` | Merge SHA, work dir from `propose.sh`; `gh` with write `GH_TOKEN` | Re-checks the allowlist without building, force-pushes `docs/auto/`, opens or edits its PR, and posts a per-hunk review unless that commit already has one. Never writes `main`. | +| `docs-proposal/test.sh` | Git and Node | Tests the helpers, validate flow, and publish flow against a temporary repository with stubbed `gh` and `npm`; removes it on exit. | The four adapter smoke contracts are documented in the [deployment guides](../docs/guide/integrations-overview.md#adapter-support). @@ -31,6 +32,8 @@ merges to `main` and opens one pull request per merge on `docs/auto/`. It needs the organization Copilot policy "Allow use of Copilot CLI billed to the organization" and the repository setting "Allow GitHub Actions to create and approve pull requests". Pull requests opened with `GITHUB_TOKEN` do not -trigger other workflows, so the publish job runs the docs gates itself; push to -the proposal branch to run regular CI. Retry a merge from the workflow's manual -dispatch with its full commit SHA. +trigger other workflows, so the workflow's validate job runs the docs gates +itself, without write credentials; push to the proposal branch to run regular +CI. Retry a merge from the workflow's manual dispatch with its full commit SHA; +the scripts always run from the workflow's revision, so any merge on `main` can +be retried. diff --git a/scripts/docs-proposal/lib.sh b/scripts/docs-proposal/lib.sh index 87fb538d7..52f6d52ce 100644 --- a/scripts/docs-proposal/lib.sh +++ b/scripts/docs-proposal/lib.sh @@ -36,6 +36,20 @@ docs_proposal_disallowed_paths() { return "$found" } +# Applies to the index on a detached checkout of , then exits +# when it changes paths outside the allowlist. +docs_proposal_apply() { + local sha="$1" + local patch="$2" + local disallowed + git switch --quiet --detach "$sha" + git apply --index "$patch" + if ! disallowed="$(git diff --cached --name-only --no-renames "$sha" | docs_proposal_disallowed_paths)"; then + printf '::error::Proposal changes paths outside docs/guide/** and docs/index.md:\n%s\n' "$disallowed" + exit 1 + fi +} + docs_proposal_pr_body() { local sha="$1" local subject="$2" diff --git a/scripts/docs-proposal/prompt.md b/scripts/docs-proposal/prompt.md index 691060fb1..1bde920e8 100644 --- a/scripts/docs-proposal/prompt.md +++ b/scripts/docs-proposal/prompt.md @@ -1,13 +1,15 @@ # Documentation proposal You are updating the public Trusted Server documentation after a commit merged -to `main`. The merged commit and the table mapping implementation sources to -documentation pages appear at the end of this file. +to `main`. The merged commit, its base, and the table mapping implementation +sources to documentation pages appear at the end of this file. ## Task -1. Inspect the merged change with `git show --stat ` and - `git show `. +1. Inspect the merged change from its base to its commit with + `git log --oneline ..`, `git diff --stat `, + and `git diff `. The range covers every commit the push + added; `git show` omits the changes of a clean merge commit. 2. Decide whether it changes behavior that `docs/index.md` or a page under `docs/guide/` describes: configuration fields and defaults, routes, adapter support, integrations, auction behavior, CLI commands and flags, browser diff --git a/scripts/docs-proposal/propose.sh b/scripts/docs-proposal/propose.sh index 6e2b6a0b2..01106a767 100755 --- a/scripts/docs-proposal/propose.sh +++ b/scripts/docs-proposal/propose.sh @@ -1,21 +1,28 @@ #!/usr/bin/env bash # Asks Copilot CLI to propose documentation edits for one merged commit. # -# Usage: scripts/docs-proposal/propose.sh -# Writes proposal.patch (empty when nothing changes), rationale.md, and -# evidence.json to , which must be inside the repository so Copilot -# can read the prompts and write its outputs there. +# Usage: scripts/docs-proposal/propose.sh [] +# Run from a checkout of ; this script may live in a separate +# checkout. Copilot inspects .., where +# defaults to the first parent and is the previous main head when one push +# added several commits. Writes proposal.patch (empty when nothing changes), +# rationale.md, and evidence.json to , which must be inside the +# repository so Copilot can read the prompts and write its outputs there. set -euo pipefail sha="$1" here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" root="$(git rev-parse --show-toplevel)" -spec="$root/docs/superpowers/specs/2026-08-19-documentation-refresh-design.md" +spec="$here/../../docs/superpowers/specs/2026-08-19-documentation-refresh-design.md" # shellcheck source=scripts/docs-proposal/lib.sh source "$here/lib.sh" cd "$root" docs_proposal_require_sha "$sha" +base="${3:-}" +if [ -z "$base" ] || [ "$base" = "$sha" ] || ! git merge-base --is-ancestor "$base" "$sha" 2>/dev/null; then + base="$(git rev-parse "$sha^1")" +fi mkdir -p "$2" work_dir="$(cd "$2" && pwd)" relative_dir="${work_dir#"$root"/}" @@ -36,7 +43,7 @@ run_copilot() { { cat "$here/prompt.md" - printf '\n## Merged change\n\n- Commit: %s\n- Subject: %s\n\n' "$sha" "$(git log -1 --format=%s "$sha")" + printf '\n## Merged change\n\n- Commit: %s\n- Base: %s\n- Subject: %s\n\n' "$sha" "$base" "$(git log -1 --format=%s "$sha")" printf '## Sources of truth\n\n' awk '/^## Sources of truth/ { found = 1; next } /^## / { found = 0 } found' "$spec" } > "$work_dir/prompt.md" diff --git a/scripts/docs-proposal/publish.sh b/scripts/docs-proposal/publish.sh index 90fa8c4ff..6f05f070f 100755 --- a/scripts/docs-proposal/publish.sh +++ b/scripts/docs-proposal/publish.sh @@ -1,9 +1,11 @@ #!/usr/bin/env bash -# Validates a documentation proposal and opens or updates its pull request. +# Opens or updates the pull request for a validated documentation proposal. # # Usage: scripts/docs-proposal/publish.sh # holds proposal.patch, rationale.md, and evidence.json from # propose.sh. Requires GH_TOKEN with contents and pull-requests write access. +# Run validate.sh first, in a job without write credentials: this script +# re-checks the path allowlist but never builds or executes proposal content. set -euo pipefail sha="$1" @@ -36,14 +38,7 @@ if [ ! -s "$work_dir/proposal.patch" ]; then exit 0 fi -git switch --quiet --detach "$sha" -git apply --index "$work_dir/proposal.patch" -if ! disallowed="$(git diff --cached --name-only --no-renames "$sha" | docs_proposal_disallowed_paths)"; then - printf '::error::Proposal changes paths outside docs/guide/** and docs/index.md:\n%s\n' "$disallowed" - exit 1 -fi - -(cd docs && npm ci && npm run lint && npm run format && npm run build) +docs_proposal_apply "$sha" "$work_dir/proposal.patch" git -c user.name='github-actions[bot]' \ -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \ @@ -52,17 +47,19 @@ git -c user.name='github-actions[bot]' \ -m "Generated by the docs-proposal workflow from $sha." head_sha="$(git rev-parse HEAD)" -# An identical tree means this is a retry of an already-published proposal; -# skip the push and review so reviewers do not see duplicate comments. -remote_tree="" +# An identical tree means a previous run already pushed this proposal. Reuse +# its commit instead of pushing, so a retry resumes any pull request or review +# step that failed after the push. +remote_head="" if git fetch --quiet origin "refs/heads/$branch" 2>/dev/null; then - remote_tree="$(git rev-parse 'FETCH_HEAD^{tree}')" + remote_head="$(git rev-parse FETCH_HEAD)" fi -if [ -n "$existing" ] && [ "$remote_tree" = "$(git rev-parse 'HEAD^{tree}')" ]; then - printf 'Proposal #%s for %s is unchanged.\n' "$pr_number" "$sha" - exit 0 +if [ -n "$remote_head" ] && [ "$(git rev-parse "$remote_head^{tree}")" = "$(git rev-parse 'HEAD^{tree}')" ]; then + head_sha="$remote_head" + printf 'Proposal branch for %s is unchanged; not pushing.\n' "$sha" +else + git push --quiet --force origin "HEAD:refs/heads/$branch" fi -git push --quiet --force origin "HEAD:refs/heads/$branch" subject="$(git log -1 --format=%s "$sha")" origin_pr="$(gh api "repos/{owner}/{repo}/commits/$sha/pulls" --jq '.[0].number // empty' || true)" @@ -75,6 +72,15 @@ else pr_number="${pr_url##*/}" fi +# Skip the review when this commit already has one, so retries never post +# duplicate comments. +posted_review="$(gh api --paginate "repos/{owner}/{repo}/pulls/$pr_number/reviews" \ + --jq ".[] | select(.commit_id == \"$head_sha\" and .user.login == \"github-actions[bot]\") | .id")" +if [ -n "$posted_review" ]; then + printf 'Proposal #%s for %s is already published.\n' "$pr_number" "$sha" + exit 0 +fi + git diff --no-renames "$sha" "$head_sha" -- docs \ | node "$here/hunks.mjs" review "$work_dir/evidence.json" "$head_sha" > "$work_dir/review.json" gh api --method POST "repos/{owner}/{repo}/pulls/$pr_number/reviews" --input "$work_dir/review.json" > /dev/null diff --git a/scripts/docs-proposal/test.sh b/scripts/docs-proposal/test.sh index 6a6eb1181..311f2d678 100755 --- a/scripts/docs-proposal/test.sh +++ b/scripts/docs-proposal/test.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# Tests the documentation proposal helpers and publish flow with stubbed gh -# and npm against a throwaway git repository. +# Tests the documentation proposal helpers, validate flow, and publish flow +# with stubbed gh and npm against a throwaway git repository. set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -70,7 +70,11 @@ case "$1 $2" in "pr list") printf '%s' "${STUB_PR:-}" ;; "pr create") printf 'https://github.com/example/repo/pull/7\n' ;; "api repos/{owner}/{repo}/commits/"*) printf '%s' "${STUB_ORIGIN_PR:-}" ;; + "api --paginate") printf '%s' "${STUB_REVIEWS:-}" ;; "api --method") + if [ -n "${STUB_FAIL_REVIEW:-}" ]; then + exit 1 + fi for arg in "$@"; do last="$arg"; done cp "$last" "$STUB_REVIEW" ;; @@ -118,6 +122,19 @@ run_publish() { fi } +run_validate() { + : > "$STUB_LOG" + if "$here/validate.sh" "$merge_sha" "$tmp/proposal" > "$tmp/out.log" 2>&1; then + printf '0' + else + printf '1' + fi +} + +remote_head() { + git ls-remote origin "refs/heads/$branch" | cut -f1 +} + remote_has_branch() { if git ls-remote --exit-code -q origin "refs/heads/$branch" >/dev/null; then printf 'yes' @@ -126,6 +143,18 @@ remote_has_branch() { fi } +make_patch "" +assert_eq "$(run_validate)" 0 "should validate an empty proposal" +assert_eq "$(grep -c 'npm' "$STUB_LOG" || true)" 0 "should not run the docs gates for an empty proposal" + +make_patch docs/superpowers/notes.md +assert_eq "$(run_validate)" 1 "should reject a disallowed path before the docs gates" +assert_eq "$(grep -c 'npm' "$STUB_LOG" || true)" 0 "should not build a disallowed proposal" + +make_patch docs/guide/cli.md +assert_eq "$(run_validate)" 0 "should validate an allowed proposal" +assert_contains "$(cat "$STUB_LOG")" "npm run build" "should run the docs build" + make_patch "" assert_eq "$(STUB_PR="" run_publish)" 0 "should succeed on an empty proposal" assert_eq "$(grep -c 'pr create\|pr close' "$STUB_LOG" || true)" 0 "should not open or close a PR for an empty diff" @@ -143,16 +172,23 @@ make_patch docs/guide/cli.md assert_eq "$(STUB_PR="" STUB_ORIGIN_PR=42 run_publish)" 0 "should publish a valid proposal" assert_eq "$(remote_has_branch)" yes "should push the proposal branch" assert_contains "$(cat "$STUB_LOG")" "pr create --base main --head $branch" "should open a PR from the proposal branch" -assert_contains "$(cat "$STUB_LOG")" "npm run build" "should run the docs build" +assert_eq "$(grep -c '^npm' "$STUB_LOG" || true)" 0 "should never build proposal content while publishing" assert_contains "$(cat "$STUB_REVIEW" 2>/dev/null)" '"path": "docs/guide/cli.md"' "should post a review for the hunk" -assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should succeed on an unchanged retry" -assert_eq "$(grep -c 'pr edit\|pr create\|reviews' "$STUB_LOG" || true)" 0 "should not re-post an unchanged proposal" +published_head="$(remote_head)" +assert_eq "$(STUB_PR="7 OPEN" STUB_REVIEWS=101 run_publish)" 0 "should succeed on an unchanged retry" +assert_eq "$(remote_head)" "$published_head" "should not push an unchanged proposal" +assert_eq "$(grep -c 'pr create\|--method POST' "$STUB_LOG" || true)" 0 "should not re-post an unchanged proposal" make_patch docs/index.md -assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should update a changed proposal" +assert_eq "$(STUB_PR="7 OPEN" STUB_FAIL_REVIEW=1 run_publish)" 1 "should fail when the review cannot be posted" +pushed_head="$(remote_head)" assert_contains "$(cat "$STUB_LOG")" "pr edit 7" "should edit the existing PR" + +assert_eq "$(STUB_PR="7 OPEN" run_publish)" 0 "should resume a proposal whose review failed" +assert_eq "$(remote_head)" "$pushed_head" "should not push again when resuming" assert_contains "$(cat "$STUB_REVIEW" 2>/dev/null)" '"path": "docs/index.md"' "should review the new hunk" +assert_contains "$(cat "$STUB_REVIEW" 2>/dev/null)" "\"commit_id\": \"$pushed_head\"" "should review the pushed commit" assert_eq "$(STUB_PR="7 CLOSED" run_publish)" 0 "should succeed for a closed proposal" assert_eq "$(grep -c 'pr edit\|pr create' "$STUB_LOG" || true)" 0 "should never reopen or recreate a closed proposal" diff --git a/scripts/docs-proposal/validate.sh b/scripts/docs-proposal/validate.sh new file mode 100755 index 000000000..08066718f --- /dev/null +++ b/scripts/docs-proposal/validate.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Applies a documentation proposal and runs the docs gates against it. +# +# Usage: scripts/docs-proposal/validate.sh +# holds proposal.patch from propose.sh. VitePress executes Vue in +# Markdown during the build, so run this only without write credentials; +# publish.sh never executes proposal content. +set -euo pipefail + +sha="$1" +work_dir="$(cd "$2" && pwd)" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(git rev-parse --show-toplevel)" +# shellcheck source=scripts/docs-proposal/lib.sh +source "$here/lib.sh" + +cd "$root" +docs_proposal_require_sha "$sha" + +if [ ! -s "$work_dir/proposal.patch" ]; then + printf 'No documentation changes proposed for %s.\n' "$sha" + exit 0 +fi + +docs_proposal_apply "$sha" "$work_dir/proposal.patch" +(cd docs && npm ci && npm run lint && npm run format && npm run build) +printf 'Validated proposal for %s.\n' "$sha" From 7871acdd56324d20c182e0c3093e5bd535dd96e9 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 5 Oct 2026 17:08:59 +0530 Subject: [PATCH 126/230] Rewrite publisher origin URLs in proxied response headers Publisher pages had origin URLs rewritten in the body but not in headers, so browsers followed Location and Refresh to the origin, preloaded Link targets from it, and blocked rewritten resources under the forwarded CSP. Map origin URLs in Location, Content-Location, Refresh, and Link (including imagesrcset) to the serving host on every publisher response, and add serving-host sources beside origin sources in Content-Security-Policy and its report-only variant. Rewriting runs before the template-cache gate so cached templates replay the rewritten policy headers; bump the template schema to 6 so entries stored with origin-only policy metadata are not replayed. Leave a Location or Refresh target unchanged when it names the current request URL under a different scheme than the origin, so an origin scheme upgrade cannot loop the browser. Closes #1128 Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/lib.rs | 1 + .../src/platform/template_cache.rs | 7 +- crates/trusted-server-core/src/publisher.rs | 249 ++++- .../src/response_header_rewrite.rs | 851 ++++++++++++++++++ docs/guide/configuration.md | 45 + 5 files changed, 1147 insertions(+), 6 deletions(-) create mode 100644 crates/trusted-server-core/src/response_header_rewrite.rs diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 249f741ab..031c5b652 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -62,6 +62,7 @@ pub mod proxy; pub mod publisher; pub mod redacted; pub mod request_signing; +pub(crate) mod response_header_rewrite; pub mod response_privacy; pub mod rsc_flight; pub(crate) mod s3_sigv4; diff --git a/crates/trusted-server-core/src/platform/template_cache.rs b/crates/trusted-server-core/src/platform/template_cache.rs index 440e5a7b8..b6f9c81ca 100644 --- a/crates/trusted-server-core/src/platform/template_cache.rs +++ b/crates/trusted-server-core/src/platform/template_cache.rs @@ -37,7 +37,8 @@ use crate::creative_opportunities::AssemblyMode; /// | 3 | Marker became ``; canonical collision-safe key, explicit origin freshness, and complete repeated document-policy metadata | /// | 4 | Marker is the shorter, accurate [`AD_ASSEMBLY_SEAM`](crate::publisher::AD_ASSEMBLY_SEAM) | /// | 5 | Key gained `request_path`, so entries from version 4 hash differently and must not be read | -pub const TEMPLATE_SCHEMA_VERSION: u32 = 5; +/// | 6 | Replayed CSP and `Link` policy metadata maps publisher-origin URLs to the serving host | +pub const TEMPLATE_SCHEMA_VERSION: u32 = 6; /// Surrogate key attached to every template so an incident can purge the template cache globally. pub const TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY: &str = "ts-template"; @@ -1049,9 +1050,9 @@ mod tests { let rendered = key().to_cache_key(); assert_eq!( rendered, - "ts-template-cache-v5-499cb43a3160fe173ffa53ea0b999658c8f2c50fbe23818ab451a59f7dc040da" + "ts-template-cache-v6-7c9268c385cc2114543f9cf5410d2c792f9ba4deaefb54a40dc3eea685e71d16" ); - assert!(rendered.starts_with("ts-template-cache-v5-")); + assert!(rendered.starts_with("ts-template-cache-v6-")); assert_eq!(rendered.len(), 85); for sensitive in ["example.com", "/news/article", "rsc", "abc123"] { assert!( diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 61bc7efe5..320637ec4 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -75,6 +75,7 @@ use crate::platform::{ reader_url_surrogate_key, }; use crate::price_bucket::{PriceGranularity, price_bucket}; +use crate::response_header_rewrite::{OriginHeaderRewrite, rewrite_origin_urls_in_headers}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -4519,6 +4520,7 @@ pub async fn handle_publisher_request( .path_and_query() .map(http::uri::PathAndQuery::as_str) .unwrap_or("/"); + let request_path_and_query = origin_path_and_query.to_string(); let target_uri = format!("{origin_scheme}://{origin_host}{origin_path_and_query}") .parse::() .change_context(TrustedServerError::Proxy { @@ -5213,6 +5215,21 @@ pub async fn handle_publisher_request( return Ok(PublisherResponse::Buffered(response)); } + // Map origin URLs in navigation, preload, and CSP headers to the serving host, + // matching the body rewrite, on every route. Runs before the template-cache gate + // so a stored template replays the rewritten policy headers; the cache key is + // already scoped to the request host. + rewrite_origin_urls_in_headers( + response.headers_mut(), + &OriginHeaderRewrite { + origin_host: &origin_host, + request_host, + request_scheme, + origin_scheme: &origin_scheme, + request_path_and_query: &request_path_and_query, + }, + ); + let template_cache_policy = TemplateCachePolicy::from_settings(settings); let gate_content_type = response .headers() @@ -9309,10 +9326,11 @@ mod tests { fn shared_template_ad_seam_is_readable_and_versioned() { assert_eq!( (crate::platform::TEMPLATE_SCHEMA_VERSION, AD_ASSEMBLY_SEAM,), - (5, ""), + (6, ""), "changing the seam must bump the cache schema, or a deploy assembles \ against a marker that moved. The converse does not hold — the schema \ - also moves when the cache key's shape changes, as it did for v5 — so \ + also moves when the cache key's shape or replayed metadata changes, as \ + it did for v5 and v6 — so \ updating this pin with an unchanged seam is legitimate." ); assert_eq!( @@ -11996,7 +12014,7 @@ mod tests { #[test] fn parser_validation_does_not_change_the_cached_schema() { - assert_eq!(crate::platform::TEMPLATE_SCHEMA_VERSION, 5); + assert_eq!(crate::platform::TEMPLATE_SCHEMA_VERSION, 6); assert_eq!(AD_ASSEMBLY_SEAM, ""); assert!(!contains_publisher_esi_directive( AD_ASSEMBLY_SEAM.as_bytes() @@ -14695,6 +14713,62 @@ mod tests { ); } + #[tokio::test] + async fn a_cache_hit_replays_policy_headers_rewritten_to_the_serving_host() { + // The header rewrite runs before the template-cache gate captures policy + // metadata, so a hit replays the serving-host CSP and Link rather than the + // origin-only values that would block or bypass the rewritten page. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ( + "content-security-policy", + "img-src https://origin.test-publisher.com", + ), + ( + "link", + "; rel=preload; as=style", + ), + ], + ); + + let cold = run(&settings, &services, navigation_request()).await; + let warm = run(&settings, &services, navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "should serve the second navigation from the template cache" + ); + let value = |response: &Response, name: header::HeaderName| { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + .expect("should carry the policy header") + }; + for name in [header::CONTENT_SECURITY_POLICY, header::LINK] { + let replayed = value(&warm, name.clone()); + assert!( + replayed.contains("://ts.example.com"), + "should replay `{name}` rewritten to the serving host: {replayed}" + ); + assert_eq!( + replayed, + value(&cold, name.clone()), + "should replay the same `{name}` the miss delivered" + ); + } + } + #[tokio::test] async fn a_cache_hit_preserves_every_repeated_policy_header_in_order() { let stub = Arc::new(StubHttpClient::new()); @@ -16124,6 +16198,175 @@ mod tests { ); } + fn proxy_response_headers(response: &PublisherResponse) -> &header::HeaderMap { + match response { + PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } + | PublisherResponse::Buffered(response) => response.headers(), + } + } + + fn publisher_page_request() -> Request { + HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/redirect") + .header(header::HOST, "publisher.example") + .header("fastly-ssl", "1") + .body(EdgeBody::empty()) + .expect("should build request") + } + + #[tokio::test] + async fn transformed_page_headers_map_origin_urls_to_serving_host() { + let settings = create_test_settings(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 302, + b"go" + .to_vec(), + vec![ + (header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8"), + ( + header::LOCATION.as_str(), + "https://origin.test-publisher.com/landing.html", + ), + ( + header::REFRESH.as_str(), + "0; url=https://origin.test-publisher.com/refreshed.html", + ), + ( + header::LINK.as_str(), + "; rel=preload; as=style", + ), + ( + header::CONTENT_SECURITY_POLICY.as_str(), + "default-src https://origin.test-publisher.com 'unsafe-inline'", + ), + ], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + let response = run_publisher_proxy(&settings, &services, publisher_page_request()).await; + + assert!( + matches!(response, PublisherResponse::Stream { .. }), + "should stream the redirect HTML body through the rewriter" + ); + let headers = proxy_response_headers(&response); + let value = |name: header::HeaderName| { + headers + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + assert_eq!( + value(header::LOCATION).as_deref(), + Some("https://publisher.example/landing.html"), + "should keep the redirect on the serving host" + ); + assert_eq!( + value(header::REFRESH).as_deref(), + Some("0; url=https://publisher.example/refreshed.html"), + "should keep the refresh navigation on the serving host" + ); + assert_eq!( + value(header::LINK).as_deref(), + Some("; rel=preload; as=style"), + "should preload through the serving host" + ); + assert_eq!( + value(header::CONTENT_SECURITY_POLICY).as_deref(), + Some( + "default-src https://origin.test-publisher.com https://publisher.example 'unsafe-inline'" + ), + "should permit the serving host alongside the origin" + ); + } + + #[tokio::test] + async fn bodiless_redirect_location_maps_origin_url_to_serving_host() { + let settings = create_test_settings(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 301, + Vec::new(), + vec![( + header::LOCATION.as_str(), + "http://origin.test-publisher.com/moved?x=1", + )], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + let response = run_publisher_proxy(&settings, &services, publisher_page_request()).await; + + assert!( + matches!(response, PublisherResponse::Buffered(_)), + "should return the bodiless redirect unmodified" + ); + assert_eq!( + proxy_response_headers(&response) + .get(header::LOCATION) + .and_then(|value| value.to_str().ok()), + Some("https://publisher.example/moved?x=1"), + "should rewrite the redirect target even without a transformed body" + ); + } + + async fn redirect_location_for_current_url(origin_url: &str, location: &str) -> String { + let mut settings = create_test_settings(); + settings.publisher.origin_url = origin_url.to_string(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 303, + Vec::new(), + vec![(header::LOCATION.as_str(), location)], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + let response = run_publisher_proxy(&settings, &services, publisher_page_request()).await; + + proxy_response_headers(&response) + .get(header::LOCATION) + .and_then(|value| value.to_str().ok()) + .expect("should keep a Location header") + .to_string() + } + + #[tokio::test] + async fn scheme_upgrade_redirect_to_current_url_is_not_rewritten_into_a_loop() { + let location = redirect_location_for_current_url( + "http://origin.test-publisher.com", + "https://origin.test-publisher.com/redirect", + ) + .await; + + assert_eq!( + location, "https://origin.test-publisher.com/redirect", + "should not redirect the browser back to the URL it requested" + ); + } + + #[tokio::test] + async fn same_scheme_redirect_to_current_url_stays_on_serving_host() { + let location = redirect_location_for_current_url( + "https://origin.test-publisher.com", + "https://origin.test-publisher.com/redirect", + ) + .await; + + assert_eq!( + location, "https://publisher.example/redirect", + "should rewrite a POST-redirect-GET to the same URL" + ); + } + #[tokio::test] async fn publisher_asset_policy_response_with_cookie_is_private_after_finalization() { let settings = Settings::from_toml(&format!( diff --git a/crates/trusted-server-core/src/response_header_rewrite.rs b/crates/trusted-server-core/src/response_header_rewrite.rs new file mode 100644 index 000000000..cc01f1cb4 --- /dev/null +++ b/crates/trusted-server-core/src/response_header_rewrite.rs @@ -0,0 +1,851 @@ +//! Rewriting of publisher-origin URLs in proxied response headers. +//! +//! The publisher-page path rewrites origin URLs in the response body to the +//! serving host. Headers that carry URLs must follow the same mapping, or the +//! browser acts on them against the origin: it follows `Location` and `Refresh` +//! off the appliance, preloads `Link` targets from the origin, and enforces a +//! Content Security Policy that does not permit the rewritten resources. +//! +//! Two rules apply: +//! +//! - **Navigation and preload headers** (`Location`, `Content-Location`, +//! `Refresh`, `Link`): an absolute or protocol-relative URL whose authority is +//! the publisher origin is mapped to the serving host, using the same +//! scheme mapping as the HTML body rewriter. Relative URLs and other hosts +//! are left unchanged. +//! - **Content Security Policy** (`Content-Security-Policy`, +//! `Content-Security-Policy-Report-Only`): every source expression that names +//! the publisher origin gains a sibling expression for the serving host. +//! Nothing is removed, so resources the origin still serves directly remain +//! permitted. Host changes cannot authorize nonce-less inline inserts; that +//! is out of scope here. +//! +//! Each response's own header values are rewritten, so per-page policies are +//! preserved. Operator `[response_headers]` overrides are applied later and +//! still take precedence. + +use edgezero_core::http::{HeaderMap, HeaderName, HeaderValue, header}; + +/// Headers whose value is a Content Security Policy. +const CSP_HEADERS: &[HeaderName] = &[ + header::CONTENT_SECURITY_POLICY, + header::CONTENT_SECURITY_POLICY_REPORT_ONLY, +]; + +/// CSP directives whose values are report endpoints, not source lists. +/// +/// Adding a serving-host endpoint would duplicate every violation report. +const CSP_NON_SOURCE_DIRECTIVES: &[&str] = &["report-uri", "report-to"]; + +/// The origin-to-serving-host mapping applied to response headers. +#[derive(Debug, Clone, Copy)] +pub(crate) struct OriginHeaderRewrite<'a> { + /// Publisher origin authority (`host` or `host:port`). + pub origin_host: &'a str, + /// Serving host authority the browser addressed. + pub request_host: &'a str, + /// Scheme the browser used to reach the serving host. + pub request_scheme: &'a str, + /// Scheme Trusted Server uses to fetch the publisher origin. + pub origin_scheme: &'a str, + /// Path and query of the request being answered (`/page?x=1`). + /// + /// An origin redirect to this URL under a different scheme than + /// [`Self::origin_scheme`] (for example, forcing HTTPS) is left unchanged. + /// Rewriting it would send the browser back to the URL it just requested, + /// Trusted Server would refetch the origin over the same scheme, and the + /// origin would redirect again. + pub request_path_and_query: &'a str, +} + +impl OriginHeaderRewrite<'_> { + fn is_active(&self) -> bool { + !self.origin_host.is_empty() + && !self.request_host.is_empty() + && !self.request_scheme.is_empty() + && !self.origin_host.eq_ignore_ascii_case(self.request_host) + } + + /// Rewrite a URL reference that points at the publisher origin. + /// + /// Returns [`None`] when `url` is relative, names another host, or names the + /// origin host with a different port. + fn rewrite_url(&self, url: &str) -> Option { + let rest = strip_prefix_ignore_ascii_case(url, "https://") + .or_else(|| strip_prefix_ignore_ascii_case(url, "http://")); + if let Some(rest) = rest { + let suffix = self.strip_origin_authority(rest)?; + return Some(format!( + "{}://{}{suffix}", + self.request_scheme, self.request_host + )); + } + + let suffix = self.strip_origin_authority(url.strip_prefix("//")?)?; + Some(format!("//{}{suffix}", self.request_host)) + } + + /// Rewrite a navigation target (`Location`, `Refresh`), unless rewriting + /// it would loop. + /// + /// Only a scheme change can loop. A same-scheme target equal to the current + /// request (POST-redirect-GET, a cookie-setting redirect, a periodic + /// reload) depends on state and is rewritten like any other target. + fn rewrite_navigation_url(&self, url: &str) -> Option { + let rewritten = self.rewrite_url(url)?; + if self.is_scheme_change_to_current_request(url) { + log::debug!("Keeping origin scheme-change redirect to the current request URL"); + return None; + } + Some(rewritten) + } + + fn is_scheme_change_to_current_request(&self, url: &str) -> bool { + let Some((scheme, rest)) = url.split_once("://") else { + return false; + }; + if scheme.eq_ignore_ascii_case(self.origin_scheme) { + return false; + } + let Some(suffix) = self.strip_origin_authority(rest) else { + return false; + }; + let target = suffix.split_once('#').map_or(suffix, |(target, _)| target); + let target = if target.is_empty() { "/" } else { target }; + target == self.request_path_and_query + } + + /// Strip the origin authority from the start of `rest`, returning the + /// path, query, and fragment that follow it. + fn strip_origin_authority<'b>(&self, rest: &'b str) -> Option<&'b str> { + let suffix = strip_prefix_ignore_ascii_case(rest, self.origin_host)?; + let at_boundary = suffix.is_empty() || matches!(suffix.as_bytes()[0], b'/' | b'?' | b'#'); + at_boundary.then_some(suffix) + } + + /// Build the serving-host CSP source for a source expression naming the + /// origin, or [`None`] when the expression names something else. + fn csp_source_for(&self, source: &str) -> Option { + if source.starts_with('\'') { + return None; + } + if source.contains("://") { + return self.rewrite_url(source); + } + let suffix = self.strip_origin_authority(source)?; + Some(format!("{}{suffix}", self.request_host)) + } +} + +/// Rewrite publisher-origin URLs in navigation, preload, and CSP headers. +/// +/// Values that are not valid UTF-8, or whose rewrite would not be a valid +/// header value, are left unchanged. Multiple values of the same header keep +/// their order. +/// +/// # Examples +/// +/// ```ignore +/// let rewrite = OriginHeaderRewrite { +/// origin_host: "origin.example.com", +/// request_host: "www.example.com", +/// request_scheme: "https", +/// origin_scheme: "https", +/// request_path_and_query: "/page", +/// }; +/// rewrite_origin_urls_in_headers(response.headers_mut(), &rewrite); +/// ``` +pub(crate) fn rewrite_origin_urls_in_headers( + headers: &mut HeaderMap, + rewrite: &OriginHeaderRewrite<'_>, +) { + if !rewrite.is_active() { + return; + } + + rewrite_header_values(headers, &header::LOCATION, |value| { + rewrite.rewrite_navigation_url(value.trim()) + }); + rewrite_header_values(headers, &header::CONTENT_LOCATION, |value| { + rewrite.rewrite_url(value.trim()) + }); + rewrite_header_values(headers, &header::REFRESH, |value| { + rewrite_refresh(value, rewrite) + }); + rewrite_header_values(headers, &header::LINK, |value| rewrite_link(value, rewrite)); + for name in CSP_HEADERS { + rewrite_header_values(headers, name, |value| rewrite_csp(value, rewrite)); + } +} + +fn rewrite_header_values( + headers: &mut HeaderMap, + name: &HeaderName, + rewrite_value: impl Fn(&str) -> Option, +) { + let mut changed = false; + let values: Vec = headers + .get_all(name) + .iter() + .map(|value| { + let rewritten = value + .to_str() + .ok() + .and_then(&rewrite_value) + .and_then(|rewritten| HeaderValue::from_str(&rewritten).ok()); + match rewritten { + Some(rewritten) => { + changed = true; + rewritten + } + None => value.clone(), + } + }) + .collect(); + + if !changed { + return; + } + + log::debug!("Rewriting publisher origin URLs in `{name}` response header"); + headers.remove(name); + for value in values { + headers.append(name.clone(), value); + } +} + +/// Rewrite the URL in a `Refresh` value such as `0; url=https://origin/next`. +/// +/// Accepts the forms browsers parse: whitespace or a `;` or `,` separator, an optional +/// case-insensitive `url=` prefix, and an optionally quoted URL. +fn rewrite_refresh(value: &str, rewrite: &OriginHeaderRewrite<'_>) -> Option { + let bytes = value.as_bytes(); + let skip_whitespace = |mut index: usize| { + while index < bytes.len() && bytes[index].is_ascii_whitespace() { + index += 1; + } + index + }; + + let time_start = skip_whitespace(0); + let mut index = time_start; + while index < bytes.len() && (bytes[index].is_ascii_digit() || bytes[index] == b'.') { + index += 1; + } + if index == time_start { + return None; + } + if !matches!(bytes.get(index), Some(b';' | b',')) + && !bytes.get(index).is_some_and(u8::is_ascii_whitespace) + { + return None; + } + index = skip_whitespace(index); + if matches!(bytes.get(index), Some(b';' | b',')) { + index += 1; + } + index = skip_whitespace(index); + + if strip_prefix_ignore_ascii_case(&value[index..], "url").is_some() { + let after_url = skip_whitespace(index + 3); + if bytes.get(after_url) == Some(&b'=') { + index = skip_whitespace(after_url + 1); + } + } + + let (start, end) = match bytes.get(index) { + Some("e @ (b'"' | b'\'')) => { + let start = index + 1; + let end = value[start..] + .find(char::from(quote)) + .map_or(value.len(), |offset| start + offset); + (start, end) + } + Some(_) => (index, value.trim_end().len()), + None => return None, + }; + + let rewritten = rewrite.rewrite_navigation_url(&value[start..end])?; + Some(format!("{}{rewritten}{}", &value[..start], &value[end..])) +} + +/// Rewrite every `` in a `Link` value, and the candidate URLs +/// of quoted `imagesrcset` parameters. +/// +/// `<` inside a quoted parameter value is not treated as a link target. +fn rewrite_link(value: &str, rewrite: &OriginHeaderRewrite<'_>) -> Option { + let bytes = value.as_bytes(); + let mut out = String::with_capacity(value.len()); + let mut copied_until = 0; + let mut index = 0; + let mut in_quotes = false; + let mut changed = false; + + while index < bytes.len() { + match bytes[index] { + b'\\' if in_quotes => index += 1, + b'"' if !in_quotes && is_imagesrcset_value(&value[..index]) => { + let start = index + 1; + let end = quoted_string_end(bytes, start); + if let Some(rewritten) = rewrite_srcset(&value[start..end], rewrite) { + out.push_str(&value[copied_until..start]); + out.push_str(&rewritten); + copied_until = end; + changed = true; + } + index = end; + } + b'"' => in_quotes = !in_quotes, + b'<' if !in_quotes => { + let start = index + 1; + let Some(offset) = value[start..].find('>') else { + break; + }; + let end = start + offset; + if let Some(rewritten) = rewrite.rewrite_url(value[start..end].trim()) { + out.push_str(&value[copied_until..start]); + out.push_str(&rewritten); + copied_until = end; + changed = true; + } + index = end; + } + _ => {} + } + index += 1; + } + + if !changed { + return None; + } + out.push_str(&value[copied_until..]); + Some(out) +} + +/// Whether the text before an opening quote ends with `imagesrcset=`. +fn is_imagesrcset_value(before_quote: &str) -> bool { + before_quote + .trim_end() + .strip_suffix('=') + .map(str::trim_end) + .and_then(|name| { + name.len() + .checked_sub("imagesrcset".len()) + .map(|start| &name[start..]) + }) + .is_some_and(|name| name.eq_ignore_ascii_case("imagesrcset")) +} + +/// Index of the closing quote of a quoted string whose content starts at +/// `start`, or the end of `bytes` when it is unterminated. +fn quoted_string_end(bytes: &[u8], start: usize) -> usize { + let mut index = start; + while index < bytes.len() { + match bytes[index] { + b'\\' => index += 1, + b'"' => return index, + _ => {} + } + index += 1; + } + bytes.len() +} + +/// Rewrite the URL of each comma-separated `srcset` candidate, keeping its +/// descriptor and spacing. +fn rewrite_srcset(srcset: &str, rewrite: &OriginHeaderRewrite<'_>) -> Option { + let mut changed = false; + let candidates: Vec = srcset + .split(',') + .map(|candidate| { + let leading = candidate.len() - candidate.trim_start().len(); + let url_end = candidate[leading..] + .find(|c: char| c.is_ascii_whitespace()) + .map_or(candidate.len(), |offset| leading + offset); + match rewrite.rewrite_url(&candidate[leading..url_end]) { + Some(rewritten) => { + changed = true; + format!( + "{}{rewritten}{}", + &candidate[..leading], + &candidate[url_end..] + ) + } + None => candidate.to_string(), + } + }) + .collect(); + changed.then(|| candidates.join(",")) +} + +/// Add serving-host sources next to every source expression naming the origin. +/// +/// Handles comma-separated policy lists and `;`-separated directives. A +/// serving-host source already present in the directive is not duplicated. +fn rewrite_csp(value: &str, rewrite: &OriginHeaderRewrite<'_>) -> Option { + let mut insertions: Vec<(usize, String)> = Vec::new(); + + let mut directive_start = 0; + for directive in value.split([';', ',']) { + let offset = directive_start; + directive_start += directive.len() + 1; + + let mut tokens = tokens_with_offsets(directive); + let Some((_, name)) = tokens.next() else { + continue; + }; + if CSP_NON_SOURCE_DIRECTIVES + .iter() + .any(|skipped| name.eq_ignore_ascii_case(skipped)) + { + continue; + } + + let sources: Vec<(usize, &str)> = tokens.collect(); + let directive_insertions_start = insertions.len(); + for (token_offset, source) in &sources { + let Some(added) = rewrite.csp_source_for(source) else { + continue; + }; + let already_present = sources + .iter() + .map(|(_, existing)| *existing) + .chain( + insertions[directive_insertions_start..] + .iter() + .map(|(_, queued)| queued.as_str()), + ) + .any(|existing| existing.eq_ignore_ascii_case(&added)); + if !already_present { + insertions.push((offset + token_offset + source.len(), added)); + } + } + } + + if insertions.is_empty() { + return None; + } + + let extra: usize = insertions.iter().map(|(_, added)| added.len() + 1).sum(); + let mut out = String::with_capacity(value.len() + extra); + let mut copied_until = 0; + for (position, added) in insertions { + out.push_str(&value[copied_until..position]); + out.push(' '); + out.push_str(&added); + copied_until = position; + } + out.push_str(&value[copied_until..]); + Some(out) +} + +fn tokens_with_offsets(text: &str) -> impl Iterator { + text.split(|c: char| c.is_ascii_whitespace()) + .filter(|token| !token.is_empty()) + .map(move |token| (token.as_ptr() as usize - text.as_ptr() as usize, token)) +} + +fn strip_prefix_ignore_ascii_case<'a>(text: &'a str, prefix: &str) -> Option<&'a str> { + let head = text.get(..prefix.len())?; + head.eq_ignore_ascii_case(prefix) + .then(|| &text[prefix.len()..]) +} + +#[cfg(test)] +mod tests { + use super::*; + + const REWRITE: OriginHeaderRewrite<'static> = OriginHeaderRewrite { + origin_host: "origin.example.com", + request_host: "www.example.com", + request_scheme: "https", + origin_scheme: "http", + request_path_and_query: "/current?page=1", + }; + + fn rewritten(name: &HeaderName, values: &[&str]) -> Vec { + let mut headers = HeaderMap::new(); + for value in values { + headers.append( + name.clone(), + HeaderValue::from_str(value).expect("should build test header value"), + ); + } + rewrite_origin_urls_in_headers(&mut headers, &REWRITE); + headers + .get_all(name) + .iter() + .map(|value| { + value + .to_str() + .expect("should keep header value ascii") + .to_string() + }) + .collect() + } + + fn single(name: &HeaderName, value: &str) -> String { + rewritten(name, &[value]).remove(0) + } + + #[test] + fn location_rewrites_absolute_origin_urls() { + assert_eq!( + single(&header::LOCATION, "http://origin.example.com/landing.html"), + "https://www.example.com/landing.html", + "should map an http origin redirect to the serving host" + ); + assert_eq!( + single(&header::LOCATION, "HTTPS://Origin.Example.com?next=1#top"), + "https://www.example.com?next=1#top", + "should match scheme and host case-insensitively" + ); + assert_eq!( + single(&header::LOCATION, "https://origin.example.com"), + "https://www.example.com", + "should rewrite a bare origin URL" + ); + assert_eq!( + single(&header::CONTENT_LOCATION, "//origin.example.com/canonical"), + "//www.example.com/canonical", + "should keep protocol-relative URLs protocol-relative" + ); + } + + #[test] + fn location_leaves_other_targets_unchanged() { + for value in [ + "/relative/path", + "landing.html", + "https://other.example.com/landing.html", + "https://cdn.origin.example.com/landing.html", + "https://origin.example.com.example.net/landing.html", + "https://origin.example.com:8443/landing.html", + "https://other.example.com/?return=https://origin.example.com/", + "ftp://origin.example.com/file", + ] { + assert_eq!( + single(&header::LOCATION, value), + value, + "should not rewrite `{value}`" + ); + } + } + + #[test] + fn origin_with_port_requires_matching_port() { + let rewrite = OriginHeaderRewrite { + origin_host: "127.0.0.1:8301", + request_host: "127.0.0.1:3031", + request_scheme: "http", + origin_scheme: "http", + request_path_and_query: "/", + }; + assert_eq!( + rewrite.rewrite_url("http://127.0.0.1:8301/landing.html"), + Some("http://127.0.0.1:3031/landing.html".to_string()), + "should map the origin authority including its port" + ); + assert_eq!( + rewrite.rewrite_url("http://127.0.0.1:83010/landing.html"), + None, + "should not match a longer port" + ); + assert_eq!( + rewrite.rewrite_url("http://127.0.0.1/landing.html"), + None, + "should not match the host without the origin port" + ); + } + + #[test] + fn navigation_headers_keep_scheme_change_to_current_request() { + for value in [ + "https://origin.example.com/current?page=1", + "HTTPS://origin.example.com/current?page=1#top", + ] { + assert_eq!( + single(&header::LOCATION, value), + value, + "should keep scheme-change redirect `{value}` to avoid a loop" + ); + } + assert_eq!( + single( + &header::REFRESH, + "0; url=https://origin.example.com/current?page=1" + ), + "0; url=https://origin.example.com/current?page=1", + "should keep a scheme-change refresh to the same URL" + ); + } + + #[test] + fn navigation_headers_rewrite_same_scheme_targets_to_current_request() { + assert_eq!( + single( + &header::LOCATION, + "http://origin.example.com/current?page=1" + ), + "https://www.example.com/current?page=1", + "should rewrite a POST-redirect-GET to the same URL" + ); + assert_eq!( + single(&header::LOCATION, "//origin.example.com/current?page=1"), + "//www.example.com/current?page=1", + "should rewrite a protocol-relative self target" + ); + assert_eq!( + single( + &header::REFRESH, + "60; url=http://origin.example.com/current?page=1" + ), + "60; url=https://www.example.com/current?page=1", + "should keep a periodic reload on the serving host" + ); + assert_eq!( + single( + &header::LOCATION, + "https://origin.example.com/current?page=2" + ), + "https://www.example.com/current?page=2", + "should still rewrite a redirect to a different query" + ); + assert_eq!( + single( + &header::CONTENT_LOCATION, + "https://origin.example.com/current?page=1" + ), + "https://www.example.com/current?page=1", + "should rewrite non-navigation self references" + ); + } + + #[test] + fn navigation_to_origin_root_is_compared_as_slash() { + let rewrite = OriginHeaderRewrite { + request_path_and_query: "/", + ..REWRITE + }; + assert_eq!( + rewrite.rewrite_navigation_url("https://origin.example.com"), + None, + "should treat an empty path as `/`" + ); + } + + #[test] + fn refresh_rewrites_url_part() { + assert_eq!( + single( + &header::REFRESH, + "0; url=https://origin.example.com/refreshed.html" + ), + "0; url=https://www.example.com/refreshed.html", + "should rewrite the url= target" + ); + assert_eq!( + single( + &header::REFRESH, + "5,URL = 'http://origin.example.com/next' " + ), + "5,URL = 'https://www.example.com/next' ", + "should handle comma separator, spacing, case, and quotes" + ); + assert_eq!( + single(&header::REFRESH, "1.5; \"//origin.example.com/next\""), + "1.5; \"//www.example.com/next\"", + "should handle a quoted URL without url=" + ); + assert_eq!( + single(&header::REFRESH, "0 url=https://origin.example.com/next"), + "0 url=https://www.example.com/next", + "should accept whitespace as the separator" + ); + assert_eq!( + single(&header::REFRESH, "3; https://origin.example.com/next"), + "3; https://www.example.com/next", + "should handle a bare URL without url=" + ); + } + + #[test] + fn refresh_leaves_other_values_unchanged() { + for value in [ + "30", + "0; url=/relative", + "0; url=https://other.example.com/next", + "0x; url=https://origin.example.com/next", + "; url=https://origin.example.com/next", + ] { + assert_eq!( + single(&header::REFRESH, value), + value, + "should not rewrite `{value}`" + ); + } + } + + #[test] + fn link_rewrites_each_origin_target() { + assert_eq!( + single( + &header::LINK, + "; rel=preload; as=style, \ + ; rel=preload; as=script, \ + ; rel=preload; as=font, \ + ; rel=prefetch" + ), + "; rel=preload; as=style, \ + ; rel=preload; as=script, \ + ; rel=preload; as=font, \ + ; rel=prefetch", + "should rewrite origin targets and keep params and other targets" + ); + } + + #[test] + fn link_rewrites_imagesrcset_candidates() { + assert_eq!( + single( + &header::LINK, + "; rel=preload; as=image; \ + imagesrcset=\"https://origin.example.com/a.jpg 1x, /b.jpg 2x, \ + https://cdn.example.net/c.jpg 3x\"; imagesizes=\"100vw\"" + ), + "; rel=preload; as=image; \ + imagesrcset=\"https://www.example.com/a.jpg 1x, /b.jpg 2x, \ + https://cdn.example.net/c.jpg 3x\"; imagesizes=\"100vw\"", + "should rewrite origin srcset candidates and keep the rest" + ); + } + + #[test] + fn link_ignores_angle_brackets_in_quoted_params() { + let value = "; rel=preload; title=\"x \\\" y\""; + assert_eq!( + single(&header::LINK, value), + value, + "should not treat quoted text as a link target" + ); + } + + #[test] + fn link_rewrites_every_value_and_keeps_order() { + assert_eq!( + rewritten( + &header::LINK, + &[ + "; rel=preload; as=script", + "; rel=preload; as=style", + ], + ), + vec![ + "; rel=preload; as=script".to_string(), + "; rel=preload; as=style".to_string(), + ], + "should rewrite per value and preserve ordering" + ); + } + + #[test] + fn csp_adds_serving_host_beside_origin_sources() { + assert_eq!( + single( + &header::CONTENT_SECURITY_POLICY, + "default-src http://origin.example.com 'unsafe-inline'; \ + img-src origin.example.com/images/ data:; script-src 'self'" + ), + "default-src http://origin.example.com https://www.example.com 'unsafe-inline'; \ + img-src origin.example.com/images/ www.example.com/images/ data:; script-src 'self'", + "should add the serving host and keep the origin sources" + ); + } + + #[test] + fn csp_adds_one_serving_host_source_for_equivalent_origin_sources() { + assert_eq!( + single( + &header::CONTENT_SECURITY_POLICY, + "img-src http://origin.example.com https://origin.example.com" + ), + "img-src http://origin.example.com https://www.example.com https://origin.example.com", + "should not add the same serving-host source twice" + ); + } + + #[test] + fn csp_handles_policy_lists_and_report_only() { + assert_eq!( + single( + &header::CONTENT_SECURITY_POLICY_REPORT_ONLY, + "img-src https://origin.example.com, connect-src https://origin.example.com/api" + ), + "img-src https://origin.example.com https://www.example.com, \ + connect-src https://origin.example.com/api https://www.example.com/api", + "should rewrite each policy in a comma-separated list" + ); + } + + #[test] + fn csp_does_not_duplicate_or_touch_unrelated_sources() { + for value in [ + "default-src https://origin.example.com https://www.example.com", + "default-src 'self' https://cdn.example.net *.origin.example.com", + "default-src 'self'; report-uri https://origin.example.com/csp-report", + "img-src https://origin.example.com:8443", + ] { + assert_eq!( + single(&header::CONTENT_SECURITY_POLICY, value), + value, + "should leave `{value}` unchanged" + ); + } + } + + #[test] + fn skips_rewrite_when_hosts_match_or_are_missing() { + let mut headers = HeaderMap::new(); + headers.insert( + header::LOCATION, + HeaderValue::from_static("https://origin.example.com/next"), + ); + for rewrite in [ + OriginHeaderRewrite { + request_host: "ORIGIN.example.com", + ..REWRITE + }, + OriginHeaderRewrite { + request_host: "", + ..REWRITE + }, + OriginHeaderRewrite { + origin_host: "", + ..REWRITE + }, + ] { + rewrite_origin_urls_in_headers(&mut headers, &rewrite); + assert_eq!( + headers.get(header::LOCATION), + Some(&HeaderValue::from_static("https://origin.example.com/next")), + "should leave headers untouched for {rewrite:?}" + ); + } + } + + #[test] + fn leaves_unrelated_headers_untouched() { + let mut headers = HeaderMap::new(); + headers.insert( + header::CONTENT_TYPE, + HeaderValue::from_static("text/html; charset=utf-8"), + ); + headers.insert( + "x-origin", + HeaderValue::from_static("https://origin.example.com/"), + ); + let before = headers.clone(); + rewrite_origin_urls_in_headers(&mut headers, &REWRITE); + assert_eq!(headers, before, "should only rewrite URL-bearing headers"); + } +} diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 09a59a01f..c48a2cb9f 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -775,6 +775,51 @@ Edit TOML, validate, and push again for those changes. Custom headers may be overwritten by application logic. Standard headers (`Content-Type`, `Content-Length`) are controlled by the application. ::: +### Origin URLs in proxied response headers + +Trusted Server rewrites publisher-origin URLs in page bodies to the serving +host. It applies the same mapping to URL-bearing headers on every proxied +publisher response, so the browser stays on the serving host. No configuration +is required. + +| Header | Behavior | +| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `Location`, `Content-Location` | An absolute or protocol-relative URL on the origin is mapped to the serving host. Relative URLs, other hosts, and other ports are unchanged. | +| `Refresh` | The URL after the delay (`0; url=...`) is mapped the same way. | +| `Link` | Each `<...>` target and quoted `imagesrcset` candidate on the origin is mapped; `rel`, `as`, and other parameters are kept. | +| `Content-Security-Policy`,
`Content-Security-Policy-Report-Only` | Every source naming the origin gains a serving-host source beside it. Origin sources are kept, and `report-uri` is unchanged. | + +For example, with the origin `origin.example.com` served at `www.example.com`: + +```http +Location: https://origin.example.com/landing.html +Content-Security-Policy: img-src https://origin.example.com data: +``` + +is delivered as: + +```http +Location: https://www.example.com/landing.html +Content-Security-Policy: img-src https://origin.example.com https://www.example.com data: +``` + +One exception prevents a redirect loop. When the origin redirects to the URL +being requested but under a different scheme from `publisher.origin_url` (for +example, an `http://` origin forcing HTTPS), the `Location` or `Refresh` target +is left unchanged. Rewriting it would send the browser back to the same URL, and +the origin would redirect again. Same-scheme redirects to the current URL, such +as after a form POST, are rewritten as usual. + +Each response keeps its own per-page values. A `[response_headers]` entry for +the same header still replaces the rewritten value. + +::: warning CSP and injected scripts +Host rewriting only adds serving-host sources. It does not authorize inline +content Trusted Server inserts without a nonce, or the `/static/tsjs=` bundle +when the policy allows neither `'self'` nor the serving host. Check +`script-src` (or `default-src`) on pages with a restrictive policy. +::: + ## Request Signing Configuration for Ed25519 request signing and JWKS management. From 7bc0e8781c880552d98ce939b569ada45dcfb755 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Mon, 5 Oct 2026 17:59:24 +0530 Subject: [PATCH 127/230] Stop replaying request bodies on 301/302/303 proxy redirects Followed 301, 302 and 303 redirects are now sent as a bodyless GET (HEAD stays HEAD) and drop body-describing headers such as Content-Type, so a POST payload no longer reaches a redirect target the caller did not address. 307 and 308 keep the original method and body. Per-hop HTTPS and allowlist checks are unchanged. Closes #431 Signed-off-by: dhruv8sh --- crates/trusted-server-core/src/proxy.rs | 204 +++++++++++++++++++++++- 1 file changed, 201 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index 5d659cdbf..98b08b99e 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -72,6 +72,15 @@ const PROXY_FORWARD_HEADERS: [header::HeaderName; 5] = [ HEADER_X_FORWARDED_FOR, ]; +/// Request headers that describe a body and must not ride on a bodyless redirect follow-up. +const REQUEST_BODY_HEADERS: [header::HeaderName; 5] = [ + header::CONTENT_TYPE, + header::CONTENT_LENGTH, + header::CONTENT_ENCODING, + header::CONTENT_LANGUAGE, + header::CONTENT_LOCATION, +]; + /// Curated request headers preserved for asset proxying. /// /// Unlike the HTML publisher fallback, asset requests need cache validation and @@ -760,6 +769,10 @@ struct ProxyRedirectPolicy<'a> { /// the caller's EC ID as a `ts-ec` query parameter to the target URL. /// Optional bodies/headers can be supplied via [`ProxyRequestConfig`]. /// +/// Followed `307`/`308` redirects keep the original method and body. Followed +/// `301`/`302`/`303` redirects are sent as a bodyless `GET` (a `HEAD` stays +/// `HEAD`) without body-describing headers such as `Content-Type`. +/// /// # Errors /// /// - [`TrustedServerError::Proxy`] if the target URL is invalid, uses an unsupported @@ -1265,6 +1278,9 @@ async fn proxy_with_redirects( let mut current_url = target_url_parsed.to_string(); let mut current_method: Method = req.method().clone(); + let mut current_body = body; + // Set once a redirect rewrites the request into a bodyless follow-up. + let mut body_dropped = false; for redirect_attempt in 0..=MAX_REDIRECTS { let parsed_url = url::Url::parse(¤t_url).map_err(|_| { @@ -1346,10 +1362,15 @@ async fn proxy_with_redirects( // insert() replaces any existing value, matching set_header() semantics. outbound_headers.insert(name.clone(), value.clone()); } + if body_dropped { + for header_name in &REQUEST_BODY_HEADERS { + outbound_headers.remove(header_name); + } + } for (name, value) in &outbound_headers { builder = builder.header(name, value); } - let body_bytes = body.map(<[u8]>::to_vec).unwrap_or_default(); + let body_bytes = current_body.map(<[u8]>::to_vec).unwrap_or_default(); let edge_req = builder .body(EdgeBody::from(body_bytes)) @@ -1477,8 +1498,18 @@ async fn proxy_with_redirects( ); current_url = next_url.to_string(); - if status == StatusCode::SEE_OTHER { - current_method = Method::GET; + // 307/308 replay the original method and body by design. 301/302/303 + // become a bodyless GET (HEAD stays HEAD) so those redirects cannot + // carry the payload to a target the caller did not address. + if matches!( + status, + StatusCode::MOVED_PERMANENTLY | StatusCode::FOUND | StatusCode::SEE_OTHER + ) { + if current_method != Method::HEAD { + current_method = Method::GET; + } + current_body = None; + body_dropped = true; } } @@ -3989,6 +4020,173 @@ mod tests { }); } + const REDIRECT_TEST_BODY: &[u8] = b"fictional-payload=example"; + + /// Send `method` with a body through one redirect of `status` and return the stub. + fn proxy_through_redirect(method: Method, status: u16) -> Arc { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + status, + Vec::new(), + vec![("location", "https://redirect.example.com/final")], + ); + stub.push_response(200, b"ok".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = create_test_settings(); + let req = build_http_request(method, "https://edge.example/"); + + futures::executor::block_on(proxy_request( + &settings, + req, + ProxyRequestConfig::new("https://source.example.com/start") + .without_ec_id() + .without_forward_headers() + .with_body(REDIRECT_TEST_BODY.to_vec()) + .with_header( + header::CONTENT_TYPE, + HeaderValue::from_static("application/x-www-form-urlencoded"), + ) + .with_header(header::CONTENT_LANGUAGE, HeaderValue::from_static("en-US")) + .with_header( + header::CONTENT_LENGTH, + HeaderValue::from(REDIRECT_TEST_BODY.len()), + ), + &services, + )) + .expect("should follow redirect"); + + stub + } + + fn recorded_header_names(stub: &StubHttpClient, index: usize) -> Vec { + stub.recorded_request_headers()[index] + .iter() + .map(|(name, _)| name.clone()) + .collect() + } + + #[test] + fn proxy_request_post_redirect_drops_body_for_301_302_303() { + for status in [301, 302, 303] { + let stub = proxy_through_redirect(Method::POST, status); + + assert_eq!( + stub.recorded_request_methods(), + vec!["POST".to_string(), "GET".to_string()], + "should follow a {status} POST redirect with GET" + ); + assert_eq!( + stub.recorded_request_uris()[1], + "https://redirect.example.com/final", + "should send the {status} follow-up to the Location target" + ); + assert_eq!( + stub.recorded_request_bodies(), + vec![REDIRECT_TEST_BODY.to_vec(), Vec::new()], + "should send the body only on the first hop for {status}" + ); + let first_hop_headers = recorded_header_names(&stub, 0); + let follow_up_headers = recorded_header_names(&stub, 1); + for body_header in ["content-type", "content-language", "content-length"] { + assert!( + first_hop_headers.iter().any(|name| name == body_header), + "should send {body_header} on the first {status} hop" + ); + assert!( + !follow_up_headers.iter().any(|name| name == body_header), + "should not forward {body_header} on a bodyless {status} follow-up" + ); + } + } + } + + #[test] + fn proxy_request_post_redirect_retains_method_and_body_for_307_308() { + for status in [307, 308] { + let stub = proxy_through_redirect(Method::POST, status); + + assert_eq!( + stub.recorded_request_methods(), + vec!["POST".to_string(), "POST".to_string()], + "should keep POST across a {status} redirect" + ); + assert_eq!( + stub.recorded_request_bodies(), + vec![REDIRECT_TEST_BODY.to_vec(), REDIRECT_TEST_BODY.to_vec()], + "should replay the exact body across a {status} redirect" + ); + assert!( + recorded_header_names(&stub, 1) + .iter() + .any(|name| name == "content-type"), + "should keep Content-Type across a {status} redirect" + ); + } + } + + #[test] + fn proxy_request_head_redirect_stays_head_for_301_302_303() { + for status in [301, 302, 303] { + let stub = proxy_through_redirect(Method::HEAD, status); + + assert_eq!( + stub.recorded_request_methods(), + vec!["HEAD".to_string(), "HEAD".to_string()], + "should keep HEAD across a {status} redirect" + ); + assert!( + stub.recorded_request_bodies()[1].is_empty(), + "should send no body on the {status} HEAD follow-up" + ); + } + } + + #[test] + fn proxy_request_post_redirect_still_enforces_per_hop_allowlist() { + futures::executor::block_on(async { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 307, + Vec::new(), + vec![("location", "https://blocked.example.com/final")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = create_test_settings(); + let allowed_domains = vec!["source.example.com".to_string()]; + let req = build_http_request(Method::POST, "https://edge.example/"); + + let err = proxy_request( + &settings, + req, + ProxyRequestConfig::new("https://source.example.com/start") + .without_ec_id() + .without_forward_headers() + .with_body(REDIRECT_TEST_BODY.to_vec()) + .with_allowed_domains(&allowed_domains), + &services, + ) + .await + .expect_err("should block a redirect to a host outside the allowlist"); + + assert!( + matches!( + err.current_context(), + TrustedServerError::AllowlistViolation { host } if host == "blocked.example.com" + ), + "should report the blocked redirect host" + ); + assert_eq!( + stub.recorded_request_bodies().len(), + 1, + "should never send the body to the blocked host" + ); + }); + } + #[test] fn proxy_request_forwards_stream_response_flag_to_platform_request() { futures::executor::block_on(async { From af697810ba3e5bd22a0afe7e45fbb1830db3ad89 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 5 Oct 2026 14:20:19 -0500 Subject: [PATCH 128/230] Fail closed on uncaptured Flight source --- .../src/integrations/nextjs/mod.rs | 160 +++++++++++++++++- .../integrations/nextjs/rsc_placeholders.rs | 157 ++++++++++++++--- .../src/integrations/nextjs/rsc_stream.rs | 10 ++ 3 files changed, 296 insertions(+), 31 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index d76b82be1..4b6292fb1 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -255,7 +255,7 @@ mod tests { } #[test] - fn non_code_flight_spellings_preserve_gtm_and_later_flight_rewrites() { + fn non_code_flight_spellings_preserve_source_and_later_flight_rewrites() { let payload = "0:{\"url\":\"https://origin.example.com/page\"}\n"; let flight = flight_html(&[payload]); for prefix in [ @@ -291,8 +291,8 @@ mod tests { ); let output = String::from_utf8(bytes).expect("should emit UTF-8"); assert!( - output.contains(&format!("")), - "should preserve non-code source and rewrite ordinary GTM at split {split}: {output}" + output.contains(&format!("")), + "should conservatively retain GTM beside inert Flight at split {split}: {output}" ); assert!( output.contains(r#"\"url\":\"https://test.example.com/page\""#), @@ -421,6 +421,160 @@ mod tests { } } + #[test] + fn lexer_false_negatives_preserve_raw_flight_and_later_rewrites() { + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js?id=GTM-MIX1", "text": "é😀"}); + let body = data.to_string(); + let payload = format!("1:T{:x},{body}", body.len()); + let push = format!( + "self.__next_f.push([1,{}]);", + serde_json::to_string(&payload).expect("should encode Flight payload") + ); + let mut rewritten = data; + rewritten["url"] = json!("https://test.example.com/page"); + rewritten["tag"] = json!("/integrations/google_tag_manager/gtm.js?id=GTM-MIX1"); + for prefix in [ + "foo: { bar() } /'/.test(s);", + "switch (k) { case 1: {} /'/.test(s) }", + "const f = () => {}\n/'/.test(s);", + "const o = {while: (a) / 2, b: '/'};", + "for (const m of /'/g.exec(s) || []) {}", + ] { + // Exercise both an entirely missed script and a missed call after + // a captured one. Capturing a safe payload must not unprotect raw + // Flight elsewhere in the same parser fragment. + for leading in ["", push.as_str()] { + let source = format!("{leading}{prefix}{push}"); + let assert_output = |output: &str| { + let payloads = flight_payloads(output); + let missed = usize::from(!leading.is_empty()); + assert_eq!(payloads.len(), missed + 2, "should emit each push once"); + assert_eq!( + payloads[missed], payload, + "should retain missed Flight bytes and lengths: {prefix}" + ); + if missed != 0 { + assert_mixed_t_model(&payloads[0], &rewritten); + } + assert_mixed_t_model(&payloads[missed + 1], &rewritten); + assert!(!output.contains("__ts_rsc_"), "should resolve all captures"); + }; + for split in (0..=source.len()).filter(|split| source.is_char_boundary(*split)) { + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let first = format!("", &source[split..]); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept ambiguous JavaScript prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should preserve raw and finish later Flight"), + ); + assert_output(&String::from_utf8(bytes).expect("should emit UTF-8")); + } + let html = format!(""); + for compression in [Compression::None, Compression::Gzip] { + for chunk_size in [32, 1000, 8192] { + assert_output(&mixed_output_with_compression( + &html, + compression, + chunk_size, + )); + } + } + } + } + } + + #[test] + fn raw_flight_protection_survives_lexer_resynchronization_and_capture() { + let missed_body = "`http://www.googletagmanager.com/gtm.js`"; + let missed_payload = format!("1:T{:x},{missed_body}", missed_body.len()); + let missed_push = format!( + "self.__next_f.push([1,{}]);", + serde_json::to_string(&missed_payload).expect("should encode missed Flight") + ); + let data = json!({"url": "https://origin.example.com/page", "tag": "http://www.googletagmanager.com/gtm.js?id=GTM-MIX1"}); + let body = data.to_string(); + let push = format!( + "self.__next_f.push([1,{}]);", + serde_json::to_string(&format!("1:T{:x},{body}", body.len())) + .expect("should encode capturable Flight") + ); + let mut expected = data; + expected["url"] = json!("https://test.example.com/page"); + expected["tag"] = json!("/integrations/google_tag_manager/gtm.js?id=GTM-MIX1"); + let source = format!("foo: {{ bar() }} /'/.test(s);{missed_push} /'/.test(s);{push}"); + let assert_output = |output: &str, boundary: usize| { + let payloads = flight_payloads(output); + assert_eq!( + payloads.len(), + 3, + "should retain missed and captured pushes" + ); + assert_eq!( + payloads[0], missed_payload, + "should preserve raw continuation at boundary {boundary}" + ); + assert_mixed_t_model(&payloads[1], &expected); + assert_mixed_t_model(&payloads[2], &expected); + assert!( + !output.contains("__ts_rsc_"), + "should resolve safe captures" + ); + }; + for split in 0..=source.len() { + let mut processor = create_html_processor(mixed_config(10000, 10000)); + let first = format!("", &source[split..]); + let mut bytes = processor + .process_chunk(first.as_bytes(), false) + .expect("should accept raw Flight prefix"); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should finish raw continuation and safe captures"), + ); + assert_output(&String::from_utf8(bytes).expect("should emit UTF-8"), split); + } + let html = format!(""); + for compression in [Compression::None, Compression::Gzip] { + for chunk_size in [32, 1000, 8192] { + assert_output( + &mixed_output_with_compression(&html, compression, chunk_size), + chunk_size, + ); + } + } + // A qualified head can be observed but not buffered when an earlier + // unrelated property makes the tentative receiver claim fail. + let body_start = missed_push.find('`').expect("should locate raw T body"); + let first = format!( + "", &continuation[split..]); + bytes.extend( + processor + .process_chunk(last.as_bytes(), true) + .expect("should finish later safe captures"), + ); + assert_output(&String::from_utf8(bytes).expect("should emit UTF-8"), split); + } + } + #[test] fn mixed_next_data_flight_mentions_do_not_claim_json_or_hide_gtm() { for text in [ diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs index 8b596cbdd..c6d2cc0be 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_placeholders.rs @@ -76,6 +76,7 @@ impl NextJsRscPlaceholderRewriter { // capture. A malformed or unrecognized remainder stays original/protected. let mut ranges = Vec::new(); let mut cursor = 0; + let mut unclaimed_raw_flight = false; let mut trimmed = std::mem::take(&mut state.rsc_receiver_trimmed); let mut queued_bytes = state.captured_payload_bytes; loop { @@ -92,6 +93,8 @@ impl NextJsRscPlaceholderRewriter { break; }; if let Some(arguments) = INERT_FLIGHT_CONTROL_ARGUMENTS.find(&remaining[head.end()..]) { + unclaimed_raw_flight |= + contains_qualified_flight(&content[cursor..cursor + head.start()]); cursor += head.end() + arguments.end(); trimmed = false; continue; @@ -142,6 +145,8 @@ impl NextJsRscPlaceholderRewriter { }; } queued_bytes = total.expect("should have validated queue size"); + unclaimed_raw_flight |= + contains_qualified_flight(&content[cursor..cursor + head.start()]); ranges.push((start, end)); cursor = end + 1; } @@ -152,6 +157,9 @@ impl NextJsRscPlaceholderRewriter { ScriptRewriteAction::Keep }; } + let raw_flight_remaining = state.raw_flight_unclaimed + || unclaimed_raw_flight + || contains_qualified_flight(&content[cursor..]); let mut rewritten = String::with_capacity(content.len()); cursor = 0; for (start, end) in ranges { @@ -168,7 +176,10 @@ impl NextJsRscPlaceholderRewriter { } state.captured_payload_bytes = queued_bytes; rewritten.push_str(&content[cursor..]); - state.current_fragment_protected = false; + // Captured payloads and recognized bootstrap controls are safe, but + // raw push spellings outside them may be executable lexer misses. + // Preserve those bytes even when another call was captured successfully. + state.current_fragment_protected = raw_flight_remaining; ScriptRewriteAction::replace(rewritten) } @@ -188,10 +199,11 @@ impl NextJsRscPlaceholderRewriter { self.rewrite_complete(&complete, true, state, limit, max_queued_payload_bytes) } FragmentCapture::Suppress => { - // Captured Flight bytes are withheld, so downstream stages see - // only empty source or a released receiver prefix, never a raw - // payload. Let GTM retain and rewrite preceding ordinary source. - state.current_fragment_protected = false; + // Captured bytes are withheld, so ordinary preceding source may + // still receive GTM rewriting. An earlier unclaimed raw head can + // leave its continuation in a released prefix, however, and that + // source must stay protected even during this later capture. + state.current_fragment_protected = state.raw_flight_unclaimed; ScriptRewriteAction::RemoveNode } FragmentCapture::Restore(restored) => { @@ -395,9 +407,10 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction { let shared = document_state(ctx.document_state); - let mut state = shared + let mut guard = shared .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); + let state = &mut *guard; if std::mem::take(&mut state.next_data_fragment) { // Pages data is not executable Flight, even when a JSON string // contains a complete push spelling. Clear the downstream snapshot @@ -405,6 +418,9 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { state.current_fragment_protected = false; state.flight_node_owned = false; state.flight_qualifier_tail.clear(); + state.raw_flight_seen = false; + state.raw_flight_unclaimed = false; + state.raw_qualifier_tail.clear(); state.flight_lexical = ScriptLexer::default(); return ScriptRewriteAction::Keep; } @@ -412,16 +428,40 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { state.current_fragment_protected = false; return ScriptRewriteAction::Keep; } + // A bounded raw observer is a safety guard, not a capture parser. Inert + // spellings may conservatively miss GTM rewriting in this script, but a + // lexical false negative must never let GTM corrupt raw Flight lengths. + let raw_head_end = observe_flight_qualifier( + content, + &mut state.raw_qualifier_tail, + &mut state.raw_flight_seen, + ); + let previously_owned = state.flight_node_owned; let lexical_start = state.flight_lexical.clone(); let code = state.flight_lexical.mask(content); - if state.flight_lexical.is_opaque() { + let code_head_end = if state.flight_lexical.is_opaque() { // At the lexical nesting limit, retain the existing conservative raw // qualification. Ordinary scripts without a push do not bypass Flight. - observe_flight_qualifier(&mut state, content); + let head_end = observe_flight_qualifier( + content, + &mut state.flight_qualifier_tail, + &mut state.flight_node_owned, + ); state.bypass_rsc |= state.flight_node_owned; + head_end } else { - observe_flight_qualifier(&mut state, &code); - } + observe_flight_qualifier( + &code, + &mut state.flight_qualifier_tail, + &mut state.flight_node_owned, + ) + }; + // An unclaimed raw head may precede this fragment's capture buffer. + // Compare first-head offsets: a later lexical claim in the same fragment + // does not prove an earlier raw head safe, nor does a later fragment. + state.raw_flight_unclaimed |= !previously_owned + && raw_head_end + .is_some_and(|raw_end| code_head_end.is_none_or(|code_end| raw_end < code_end)); // A call head can complete after a tentative property probe overflowed. // Once it proves Flight, preserve its entire raw continuation/group. if state.flight_node_owned @@ -432,11 +472,26 @@ impl IntegrationScriptRewriter for NextJsRscPlaceholderRewriter { { state.bypass_rsc = true; } - state.current_fragment_protected = state.flight_node_owned; - let action = self.rewrite_fragment(content, &code, lexical_start, ctx, &mut state); + state.current_fragment_protected = state.flight_node_owned || state.raw_flight_seen; + let action = self.rewrite_fragment(content, &code, lexical_start, ctx, state); + // Ownership alone does not prove buffering: an earlier unrelated + // property can cause the fragment to stream without a capture claim. + // Never let a later capture unprotect that raw source's continuation. + if !ctx.is_last_in_text_node + && state.raw_flight_seen + && !matches!( + state.rsc_script, + super::rsc_stream::FragmentState::Buffering(_) + ) + { + state.raw_flight_unclaimed = true; + } if ctx.is_last_in_text_node { state.flight_node_owned = false; state.flight_qualifier_tail.clear(); + state.raw_flight_seen = false; + state.raw_flight_unclaimed = false; + state.raw_qualifier_tail.clear(); state.rsc_probe.clear(); state.rsc_receiver_context.clear(); state.rsc_receiver_trimmed = false; @@ -480,34 +535,36 @@ fn first_qualified_flight_push(source: &str) -> Option> { } /// Observe a bounded whitespace-normalized call head without buffering script -/// source, including initializer heads split after document bypass. -fn observe_flight_qualifier(state: &mut super::rsc_stream::NextJsDocumentState, content: &str) { - if state.flight_node_owned { - return; +/// source, including initializer heads split after document bypass. Return its +/// original fragment byte-end so raw and masked observations can be compared. +fn observe_flight_qualifier(content: &str, tail: &mut String, seen: &mut bool) -> Option { + if *seen { + return None; } - for character in content.chars() { + for (offset, character) in content.char_indices() { let character = if character.is_whitespace() { ' ' } else { character }; - if character == ' ' && state.flight_qualifier_tail.ends_with(' ') { + if character == ' ' && tail.ends_with(' ') { continue; } - state.flight_qualifier_tail.push(character); - if state.flight_qualifier_tail.len() > FLIGHT_QUALIFIER_TAIL_BYTES { - let start = state.flight_qualifier_tail.len() - FLIGHT_QUALIFIER_TAIL_BYTES; - let start = (start..state.flight_qualifier_tail.len()) - .find(|index| state.flight_qualifier_tail.is_char_boundary(*index)) + tail.push(character); + if tail.len() > FLIGHT_QUALIFIER_TAIL_BYTES { + let start = tail.len() - FLIGHT_QUALIFIER_TAIL_BYTES; + let start = (start..tail.len()) + .find(|index| tail.is_char_boundary(*index)) .expect("should find bounded qualifier character boundary"); - state.flight_qualifier_tail.drain(..start); + tail.drain(..start); } - if character == '(' && contains_qualified_flight(&state.flight_qualifier_tail) { - state.flight_node_owned = true; - state.flight_qualifier_tail.clear(); - break; + if character == '(' && contains_qualified_flight(tail) { + *seen = true; + tail.clear(); + return Some(offset + 1); } } + None } /// Bytes to withhold so a `__next_f` identifier split across text fragments can @@ -573,6 +630,50 @@ mod tests { let shared = document_state(&state); let state = shared.lock().expect("should lock state"); assert!(!state.flight_node_owned, "quoted text must not own Flight"); + assert!( + state.raw_flight_seen, + "raw spelling should trigger only protection" + ); + assert!( + state.current_fragment_protected, + "should protect possible raw Flight" + ); + assert!( + !state.bypass_rsc, + "inert source must not bypass later scripts" + ); + } + + #[test] + fn raw_qualifier_observation_is_bounded_and_survives_every_split() { + for head in [ + "self.__next_f . push (", + "window.__next_f . push (", + "(self.__next_f = self.__next_f || []).push (", + "(window.__next_f = window.__next_f || []).push (", + ] { + let head = head.replace(' ', &" \n".repeat(64)); + for split in 0..=head.len() { + let mut tail = String::new(); + let mut seen = false; + let _ = observe_flight_qualifier(&"é😀".repeat(1000), &mut tail, &mut seen); + assert!( + tail.len() <= FLIGHT_QUALIFIER_TAIL_BYTES, + "should bound raw tail" + ); + assert!(!seen, "ordinary source must not protect Flight"); + let _ = observe_flight_qualifier(&head[..split], &mut tail, &mut seen); + let _ = observe_flight_qualifier(&head[split..], &mut tail, &mut seen); + assert!( + seen, + "should recognize normalized raw head at split {split}" + ); + assert!( + tail.is_empty(), + "should release tail once qualification succeeds" + ); + } + } } #[test] diff --git a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs index 22812c707..28a1cc45d 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/rsc_stream.rs @@ -59,6 +59,13 @@ pub(super) struct NextJsDocumentState { pub(super) current_fragment_protected: bool, /// Bounded qualifier observation remains active even after document bypass. pub(super) flight_qualifier_tail: String, + /// Raw qualification protects possible Flight even when the lexer misses it. + /// It never authorizes capture and lasts only for the current script node. + pub(super) raw_flight_seen: bool, + /// A raw head observed without a lexical claim may have already streamed. + /// Later captures cannot make its remaining source safe for ordinary GTM. + pub(super) raw_flight_unclaimed: bool, + pub(super) raw_qualifier_tail: String, /// Lexical context advances with original parser fragments, never replacements. pub(super) flight_lexical: ScriptLexer, /// Snapshot at the beginning of a claimed buffer for source-offset matching. @@ -82,6 +89,9 @@ impl Default for NextJsDocumentState { flight_node_owned: false, current_fragment_protected: false, flight_qualifier_tail: String::new(), + raw_flight_seen: false, + raw_flight_unclaimed: false, + raw_qualifier_tail: String::new(), flight_lexical: ScriptLexer::default(), rsc_lexical_start: ScriptLexer::default(), } From 514a617a0e9acd24330ad81f478c99ed02dd25ec Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 5 Oct 2026 14:20:19 -0500 Subject: [PATCH 129/230] Accept underscore request hosts safely --- .../trusted-server-adapter-axum/tests/routes.rs | 8 +++++++- crates/trusted-server-core/src/http_util.rs | 17 +++++++++++------ 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index b9c6379d5..f58013d5d 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -994,7 +994,13 @@ async fn nextjs_gtm_rejects_malformed_request_hosts() { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn nextjs_gtm_accepts_local_request_hosts_and_ports() { - for host in ["localhost:3000", "example.com:8443", "[::1]:3000"] { + for host in [ + "localhost:3000", + "example.com:8443", + "[::1]:3000", + "my_service:8080", + "_service_.example.com:8443", + ] { let origin = Arc::new(nextjs_auction::NextJsAuctionOrigin::with_html_response( &nextjs_auction::script_composition_html(), false, diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 7e3997378..7c19e526d 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -153,8 +153,9 @@ impl RequestInfo { /// before routing. On those paths, `Host` and [`ClientInfo`] TLS detection /// are the effective sources. Other adapters retain their existing policy. /// - /// The selected host must be a DNS name, IPv4 address, or bracketed IPv6 - /// address, optionally followed by a numeric port. Forwarded-header + /// The selected host must be an ASCII hostname (including underscores for + /// local service names), IPv4 address, or bracketed IPv6 address, optionally + /// followed by a numeric port. Forwarded-header /// precedence is unchanged. Requests without any host retain an empty host. /// /// # Errors @@ -235,11 +236,11 @@ fn is_valid_request_host(value: &str) -> bool { || host.split('.').any(|label| { label.is_empty() || label.len() > 63 - || !label.starts_with(|c: char| c.is_ascii_alphanumeric()) - || !label.ends_with(|c: char| c.is_ascii_alphanumeric()) + || !label.starts_with(|c: char| c.is_ascii_alphanumeric() || c == '_') + || !label.ends_with(|c: char| c.is_ascii_alphanumeric() || c == '_') || !label .bytes() - .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-') + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_')) }) { return false; @@ -650,6 +651,11 @@ mod tests { "example.com:443", "localhost:3000", "internal-proxy:8080", + "my_service:8080", + "service_.example.com:8443", + "_service.example.com", + "_.example.com", + "bad_label.example.com", "192.0.2.1:8080", "[2001:db8::1]", "[2001:db8::1]:443", @@ -693,7 +699,6 @@ mod tests { "example.com:+443", "example.com:443:80", "bad host.example.com", - "bad_label.example.com", "-bad.example.com", "bad-.example.com", "example..com", From 750ba8dc470aa37f3eb62512cc2b5a025bab8f9d Mon Sep 17 00:00:00 2001 From: prk-Jr Date: Tue, 6 Oct 2026 11:59:17 +0530 Subject: [PATCH 130/230] Add default-off mobile ad-rendering trace workflow --- .github/workflows/integration-tests.yml | 117 +- Cargo.lock | 27 +- Cargo.toml | 12 +- crates/trusted-server-adapter-axum/src/app.rs | 140 +- .../src/app.rs | 157 +- .../trusted-server-adapter-fastly/src/app.rs | 271 +- .../trusted-server-adapter-fastly/src/main.rs | 329 +- crates/trusted-server-adapter-spin/src/app.rs | 188 +- crates/trusted-server-core/Cargo.toml | 2 +- .../benches/html_processor_bench.rs | 1 + .../src/auction/endpoints.rs | 430 +- .../src/auction/formats.rs | 366 +- .../src/auction/orchestrator.rs | 609 +- crates/trusted-server-core/src/auth.rs | 10 +- crates/trusted-server-core/src/config.rs | 100 +- .../trusted-server-core/src/html_processor.rs | 26 + .../src/integrations/gpt_bootstrap.js | 62 +- .../src/integrations/gpt_diagnostics.rs | 110 +- .../src/integrations/js_asset_proxy.rs | 1 + crates/trusted-server-core/src/lib.rs | 1 + crates/trusted-server-core/src/openrtb.rs | 14 + crates/trusted-server-core/src/publisher.rs | 4942 ++++++++++++----- .../trusted-server-core/src/trace/actions.rs | 921 +++ .../trusted-server-core/src/trace/auction.rs | 2038 +++++++ crates/trusted-server-core/src/trace/carry.rs | 889 +++ .../trusted-server-core/src/trace/context.rs | 362 ++ .../trusted-server-core/src/trace/cookies.rs | 596 ++ .../trusted-server-core/src/trace/dispatch.rs | 470 ++ crates/trusted-server-core/src/trace/mod.rs | 38 + .../trusted-server-core/src/trace/routes.rs | 1100 ++++ crates/trusted-server-core/src/trace/shell.rs | 230 + .../src/trace/slot_refs.rs | 169 + crates/trusted-server-core/src/trace/types.rs | 358 ++ .../Cargo.toml | 6 + .../browser/global-setup.ts | 86 +- .../browser/global-teardown.ts | 44 +- .../browser/helpers/state.ts | 39 +- .../browser/helpers/trace-fixture.ts | 59 + .../browser/helpers/trace-gpt-fixture.js | 146 + .../browser/helpers/trace-report-fixture.ts | 63 + .../playwright.trace-runtime.config.ts | 47 + .../tests/nextjs/mobile-trace-live.spec.ts | 745 +++ .../tests/shared/mobile-trace-runtime.spec.ts | 225 + .../browser/tests/shared/mobile-trace.spec.ts | 666 +++ .../browser/trace-fixture.test.cjs | 94 + .../configs/trusted-server.trace-auth.toml | 40 + .../configs/trusted-server.trace.toml | 61 + .../nextjs/app/api/trace-bidder/route.ts | 73 + .../src/bin/generate-viceroy-config.rs | 229 +- .../tests/common/config.rs | 69 + .../tests/common/mod.rs | 1 + .../tests/common/trace_boundary.rs | 1020 ++++ .../tests/environments/axum.rs | 25 +- .../tests/environments/cloudflare.rs | 28 +- .../tests/integration.rs | 50 + .../tests/parity.rs | 519 ++ crates/trusted-server-js/Cargo.toml | 1 + crates/trusted-server-js/build.rs | 129 + crates/trusted-server-js/lib/.prettierignore | 2 +- crates/trusted-server-js/lib/build-all.mjs | 26 +- crates/trusted-server-js/lib/eslint.config.js | 2 +- .../trusted-server-js/lib/src/core/auction.ts | 15 +- .../lib/src/core/global.d.ts | 2 + .../trusted-server-js/lib/src/core/index.ts | 3 + .../trusted-server-js/lib/src/core/types.ts | 18 +- .../lib/src/integrations/gpt/index.ts | 40 +- .../src/integrations/gpt_diagnostics/api.ts | 10 +- .../src/integrations/gpt_diagnostics/index.ts | 29 +- .../integrations/gpt_diagnostics/overlay.ts | 51 + .../src/integrations/gpt_diagnostics/store.ts | 61 +- .../lib/src/integrations/prebid/index.ts | 91 +- .../lib/src/trace/collector.ts | 147 + .../lib/src/trace/context.ts | 164 + .../lib/src/trace/correlation.ts | 121 + .../trusted-server-js/lib/src/trace/export.ts | 87 + crates/trusted-server-js/lib/src/trace/gpt.ts | 138 + .../lib/src/trace/handoff.ts | 144 + .../trusted-server-js/lib/src/trace/json.ts | 92 + .../lib/src/trace/lifecycle.ts | 77 + .../lib/src/trace/omissions.ts | 27 + .../lib/src/trace/pending.ts | 242 + .../lib/src/trace/projection.ts | 258 + .../lib/src/trace/report-types.ts | 189 + .../lib/src/trace/report-validation.ts | 504 ++ .../lib/src/trace/report-view.ts | 614 ++ .../trusted-server-js/lib/src/trace/report.ts | 306 + .../lib/src/trace/runtime.ts | 160 + .../trusted-server-js/lib/src/trace/setup.ts | 143 + .../trusted-server-js/lib/src/trace/shape.ts | 32 + .../lib/src/trace/storage.ts | 96 + .../trusted-server-js/lib/src/trace/types.ts | 151 + .../lib/src/trace/validation.ts | 289 + .../lib/src/trace/viewer.css | 95 + .../trusted-server-js/lib/src/trace/viewer.ts | 8 + .../lib/test/core/index.test.ts | 23 + .../lib/test/integrations/gpt/ad_init.test.ts | 114 + .../gpt/schedule_initial_ad_init.test.ts | 123 + .../test/integrations/gpt/spa_hook.test.ts | 157 + .../gpt_diagnostics/index.test.ts | 51 + .../gpt_diagnostics/overlay.test.ts | 46 + .../gpt_diagnostics/store.test.ts | 166 + .../test/integrations/prebid/index.test.ts | 367 ++ .../test/prebid-artifact-integration.test.mjs | 252 +- .../lib/test/trace-assets.test.mjs | 34 + .../lib/test/trace/collector.test.ts | 172 + .../lib/test/trace/context.test.ts | 205 + .../lib/test/trace/correlation.test.ts | 160 + .../lib/test/trace/direct-api.test.ts | 233 + .../lib/test/trace/evidence.test.ts | 484 ++ .../lib/test/trace/export.test.ts | 181 + .../lib/test/trace/fixtures.ts | 218 + .../lib/test/trace/gpt-fixtures.ts | 61 + .../lib/test/trace/gpt-transport.test.ts | 176 + .../lib/test/trace/handoff.test.ts | 196 + .../lib/test/trace/lifecycle.test.ts | 153 + .../lib/test/trace/pending.test.ts | 257 + .../lib/test/trace/projection.test.ts | 331 ++ .../lib/test/trace/report.test.ts | 332 ++ .../lib/test/trace/runtime.test.ts | 118 + .../lib/test/trace/setup.test.ts | 162 + .../lib/test/trace/storage.test.ts | 146 + .../lib/test/trace/tokens.test.ts | 51 + .../lib/test/trace/types.test.ts | 149 + .../lib/test/trace/validation.test.ts | 596 ++ .../lib/test/trace/viewer.test.ts | 382 ++ .../lib/trace-asset-sources.mjs | 27 + .../lib/trace-assets-manifest.json | 16 + .../trusted-server-js/lib/trace-assets/v1.css | 1 + .../trusted-server-js/lib/trace-assets/v1.js | 1 + crates/trusted-server-js/src/lib.rs | 1 + crates/trusted-server-js/src/trace_assets.rs | 70 + docs/guide/configuration.md | 15 +- docs/guide/integrations/gpt-diagnostics.md | 155 +- ...ile-ad-render-trace-implementation-plan.md | 1072 ++++ ...-mobile-ad-render-trace-endpoint-design.md | 249 +- .../generate-integration-viceroy-configs.sh | 16 +- scripts/integration-tests-browser.sh | 20 +- scripts/integration-tests.sh | 21 +- trusted-server.example.toml | 4 +- 139 files changed, 29952 insertions(+), 1797 deletions(-) create mode 100644 crates/trusted-server-core/src/trace/actions.rs create mode 100644 crates/trusted-server-core/src/trace/auction.rs create mode 100644 crates/trusted-server-core/src/trace/carry.rs create mode 100644 crates/trusted-server-core/src/trace/context.rs create mode 100644 crates/trusted-server-core/src/trace/cookies.rs create mode 100644 crates/trusted-server-core/src/trace/dispatch.rs create mode 100644 crates/trusted-server-core/src/trace/mod.rs create mode 100644 crates/trusted-server-core/src/trace/routes.rs create mode 100644 crates/trusted-server-core/src/trace/shell.rs create mode 100644 crates/trusted-server-core/src/trace/slot_refs.rs create mode 100644 crates/trusted-server-core/src/trace/types.rs create mode 100644 crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts create mode 100644 crates/trusted-server-integration-tests/browser/helpers/trace-gpt-fixture.js create mode 100644 crates/trusted-server-integration-tests/browser/helpers/trace-report-fixture.ts create mode 100644 crates/trusted-server-integration-tests/browser/playwright.trace-runtime.config.ts create mode 100644 crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts create mode 100644 crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace-runtime.spec.ts create mode 100644 crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts create mode 100644 crates/trusted-server-integration-tests/browser/trace-fixture.test.cjs create mode 100644 crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml create mode 100644 crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml create mode 100644 crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/app/api/trace-bidder/route.ts create mode 100644 crates/trusted-server-integration-tests/tests/common/trace_boundary.rs create mode 100644 crates/trusted-server-js/lib/src/trace/collector.ts create mode 100644 crates/trusted-server-js/lib/src/trace/context.ts create mode 100644 crates/trusted-server-js/lib/src/trace/correlation.ts create mode 100644 crates/trusted-server-js/lib/src/trace/export.ts create mode 100644 crates/trusted-server-js/lib/src/trace/gpt.ts create mode 100644 crates/trusted-server-js/lib/src/trace/handoff.ts create mode 100644 crates/trusted-server-js/lib/src/trace/json.ts create mode 100644 crates/trusted-server-js/lib/src/trace/lifecycle.ts create mode 100644 crates/trusted-server-js/lib/src/trace/omissions.ts create mode 100644 crates/trusted-server-js/lib/src/trace/pending.ts create mode 100644 crates/trusted-server-js/lib/src/trace/projection.ts create mode 100644 crates/trusted-server-js/lib/src/trace/report-types.ts create mode 100644 crates/trusted-server-js/lib/src/trace/report-validation.ts create mode 100644 crates/trusted-server-js/lib/src/trace/report-view.ts create mode 100644 crates/trusted-server-js/lib/src/trace/report.ts create mode 100644 crates/trusted-server-js/lib/src/trace/runtime.ts create mode 100644 crates/trusted-server-js/lib/src/trace/setup.ts create mode 100644 crates/trusted-server-js/lib/src/trace/shape.ts create mode 100644 crates/trusted-server-js/lib/src/trace/storage.ts create mode 100644 crates/trusted-server-js/lib/src/trace/types.ts create mode 100644 crates/trusted-server-js/lib/src/trace/validation.ts create mode 100644 crates/trusted-server-js/lib/src/trace/viewer.css create mode 100644 crates/trusted-server-js/lib/src/trace/viewer.ts create mode 100644 crates/trusted-server-js/lib/test/trace-assets.test.mjs create mode 100644 crates/trusted-server-js/lib/test/trace/collector.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/context.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/correlation.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/direct-api.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/evidence.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/export.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/fixtures.ts create mode 100644 crates/trusted-server-js/lib/test/trace/gpt-fixtures.ts create mode 100644 crates/trusted-server-js/lib/test/trace/gpt-transport.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/handoff.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/lifecycle.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/pending.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/projection.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/report.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/runtime.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/setup.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/storage.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/tokens.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/types.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/validation.test.ts create mode 100644 crates/trusted-server-js/lib/test/trace/viewer.test.ts create mode 100644 crates/trusted-server-js/lib/trace-asset-sources.mjs create mode 100644 crates/trusted-server-js/lib/trace-assets-manifest.json create mode 100644 crates/trusted-server-js/lib/trace-assets/v1.css create mode 100644 crates/trusted-server-js/lib/trace-assets/v1.js create mode 100644 crates/trusted-server-js/src/trace_assets.rs create mode 100644 docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index 157a3582d..34878a346 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -1,4 +1,4 @@ -name: "Integration Tests" +name: 'Integration Tests' permissions: contents: read @@ -30,11 +30,14 @@ jobs: uses: ./.github/actions/setup-integration-test-env with: origin-port: ${{ env.ORIGIN_PORT }} - install-viceroy: "false" - build-cloudflare: "true" + install-viceroy: 'false' + build-cloudflare: 'true' - name: Generate integration Viceroy configs - run: ./scripts/generate-integration-viceroy-configs.sh + run: | + ./scripts/generate-integration-viceroy-configs.sh + INTEGRATION_APP_CONFIG_PATH=crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml INTEGRATION_BIDDER_ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT" ARTIFACTS_DIR="$ARTIFACTS_DIR/trace" ./scripts/generate-integration-viceroy-configs.sh + INTEGRATION_APP_CONFIG_PATH=crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml ARTIFACTS_DIR="$ARTIFACTS_DIR/trace-auth" ./scripts/generate-integration-viceroy-configs.sh env: INTEGRATION_ORIGIN_PORT: ${{ env.ORIGIN_PORT }} @@ -68,10 +71,10 @@ jobs: uses: ./.github/actions/setup-integration-test-env with: origin-port: ${{ env.ORIGIN_PORT }} - install-viceroy: "true" - build-wasm: "false" - build-axum: "false" - build-test-images: "false" + install-viceroy: 'true' + build-wasm: 'false' + build-axum: 'false' + build-test-images: 'false' - name: Download integration test artifacts uses: actions/download-artifact@v4 @@ -105,6 +108,7 @@ jobs: --target x86_64-unknown-linux-gnu -- --include-ignored --skip test_wordpress_fastly --skip test_nextjs_fastly + --skip trace_browser_workflow --skip trace_runtime_boundary --test-threads=1 env: WASM_BINARY_PATH: ${{ env.WASM_ARTIFACT_PATH }} @@ -127,9 +131,9 @@ jobs: uses: ./.github/actions/setup-integration-test-env with: origin-port: ${{ env.ORIGIN_PORT }} - install-viceroy: "true" - build-wasm: "false" - build-test-images: "false" + install-viceroy: 'true' + build-wasm: 'false' + build-test-images: 'false' - name: Download integration test artifacts uses: actions/download-artifact@v4 @@ -152,6 +156,83 @@ jobs: VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/configs/viceroy.toml RUST_LOG: info + trace-runtime-tests: + name: trace runtime and browser acceptance + needs: prepare-artifacts + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + + - name: Set up trace test runtime + id: shared-setup + uses: ./.github/actions/setup-integration-test-env + with: + origin-port: ${{ env.ORIGIN_PORT }} + install-viceroy: 'true' + build-wasm: 'false' + build-axum: 'false' + build-test-images: 'false' + + - name: Download integration test artifacts + uses: actions/download-artifact@v4 + with: + name: integration-test-artifacts + path: ${{ env.ARTIFACTS_DIR }} + + - name: Restore runtime binaries and framework images + run: | + chmod +x "$AXUM_ARTIFACT_PATH" + mkdir -p crates/trusted-server-adapter-cloudflare/build + cp -r "$CF_BUILD_ARTIFACT_PATH/." crates/trusted-server-adapter-cloudflare/build/ + docker load --input "$DOCKER_ARTIFACT_PATH" + + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.shared-setup.outputs.node-version }} + cache: npm + cache-dependency-path: | + crates/trusted-server-integration-tests/browser/package-lock.json + crates/trusted-server-js/lib/package-lock.json + + - name: Install browser dependencies + run: | + npm ci --prefix crates/trusted-server-js/lib + npm ci --prefix crates/trusted-server-integration-tests/browser + cd crates/trusted-server-integration-tests/browser + npx playwright install --with-deps chromium + npm install -g wrangler@4.83.0 + + - name: Install the verified Spin runtime + run: | + trace_spin_dir="$RUNNER_TEMP/trace-spin" + mkdir -p "$trace_spin_dir" + curl --fail --location --silent --show-error https://github.com/spinframework/spin/releases/download/v4.0.0/spin-v4.0.0-linux-amd64.tar.gz --output "$trace_spin_dir/spin.tar.gz" + tar -xzf "$trace_spin_dir/spin.tar.gz" -C "$trace_spin_dir" spin + echo "$trace_spin_dir" >> "$GITHUB_PATH" + + - name: Build the production Spin component + run: cargo build --package trusted-server-adapter-spin --target wasm32-wasip1 --features spin --release + + - name: Run trace acceptance against prepared runtimes + run: | + cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --target x86_64-unknown-linux-gnu --test integration trace_runtime_boundary -- --ignored --test-threads=1 --nocapture + cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --target x86_64-unknown-linux-gnu --test integration trace_browser_workflow -- --ignored --test-threads=1 --nocapture + env: + WASM_BINARY_PATH: ${{ env.WASM_ARTIFACT_PATH }} + AXUM_BINARY_PATH: ${{ env.AXUM_ARTIFACT_PATH }} + CLOUDFLARE_WRANGLER_DIR: ${{ github.workspace }}/crates/trusted-server-adapter-cloudflare + INTEGRATION_ORIGIN_PORT: ${{ env.ORIGIN_PORT }} + + - name: Upload trace browser evidence + uses: actions/upload-artifact@v4 + if: always() + with: + name: trace-runtime-browser-evidence + path: crates/trusted-server-integration-tests/browser/test-results/trace-runtime-*/ + retention-days: 7 + browser-tests: name: browser integration tests needs: prepare-artifacts @@ -165,9 +246,9 @@ jobs: uses: ./.github/actions/setup-integration-test-env with: origin-port: ${{ env.ORIGIN_PORT }} - install-viceroy: "true" - build-wasm: "false" - build-test-images: "false" + install-viceroy: 'true' + build-wasm: 'false' + build-test-images: 'false' - name: Download integration test artifacts uses: actions/download-artifact@v4 @@ -204,6 +285,10 @@ jobs: working-directory: crates/trusted-server-integration-tests/browser run: node --test initial-render/pages.test.cjs + - name: Test trace fixture cleanup + working-directory: crates/trusted-server-integration-tests/browser + run: node --test trace-fixture.test.cjs + - name: Test initial render ownership working-directory: crates/trusted-server-integration-tests/browser # Let the runner build its Rubicon + Shared ID artifact; the bidder-only @@ -225,6 +310,8 @@ jobs: WASM_BINARY_PATH: ${{ env.WASM_ARTIFACT_PATH }} INTEGRATION_ORIGIN_PORT: ${{ env.ORIGIN_PORT }} VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/configs/viceroy.toml + TRACE_VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/trace/configs/viceroy.toml + TRACE_AUTH_VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/trace-auth/configs/viceroy.toml TEST_FRAMEWORK: nextjs PLAYWRIGHT_HTML_REPORT: playwright-report-nextjs run: npx playwright test @@ -244,6 +331,8 @@ jobs: WASM_BINARY_PATH: ${{ env.WASM_ARTIFACT_PATH }} INTEGRATION_ORIGIN_PORT: ${{ env.ORIGIN_PORT }} VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/configs/viceroy.toml + TRACE_VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/trace/configs/viceroy.toml + TRACE_AUTH_VICEROY_CONFIG_PATH: ${{ env.ARTIFACTS_DIR }}/trace-auth/configs/viceroy.toml TEST_FRAMEWORK: wordpress PLAYWRIGHT_HTML_REPORT: playwright-report-wordpress run: npx playwright test diff --git a/Cargo.lock b/Cargo.lock index 7a45462b1..10dd15c38 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -788,7 +788,7 @@ version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] @@ -1428,7 +1428,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "toml", ] @@ -1436,7 +1436,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "anyhow", "async-trait", @@ -1464,7 +1464,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "anyhow", "async-trait", @@ -1487,7 +1487,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "anyhow", "async-stream", @@ -1516,7 +1516,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "anyhow", "async-trait", @@ -1543,7 +1543,7 @@ dependencies = [ [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "chrono", "clap", @@ -1568,7 +1568,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "anyhow", "async-compression", @@ -1599,7 +1599,7 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.8#567964158e4f8bd0d52321b9801de44966422e1b" +source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277" dependencies = [ "log", "proc-macro2", @@ -5537,12 +5537,14 @@ version = "0.1.0" dependencies = [ "async-trait", "axum", + "base64", "bytes", "derive_more", "edgezero-adapter-axum", "edgezero-core", "env_logger", "error-stack", + "futures", "http", "http-body-util", "libc", @@ -5550,6 +5552,8 @@ dependencies = [ "reqwest 0.12.28", "scraper", "serde_json", + "temp-env", + "tempfile", "testcontainers", "tokio", "toml", @@ -5558,6 +5562,8 @@ dependencies = [ "trusted-server-adapter-cloudflare", "trusted-server-adapter-spin", "trusted-server-core", + "trusted-server-js", + "url", "urlencoding", ] @@ -5567,6 +5573,7 @@ version = "0.1.0" dependencies = [ "build-print", "hex", + "serde_json", "sha2 0.10.9", "which", ] @@ -6031,7 +6038,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index ac0cac621..d82609a0f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -55,12 +55,12 @@ cssparser = "0.36" derive_more = { version = "2.0", features = ["display", "error"] } directories = "5" ed25519-dalek = { version = "2.2", features = ["rand_core"] } -edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8", default-features = false } -edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8", default-features = false } -edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8", default-features = false } -edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8", default-features = false } -edgezero-cli = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8" } -edgezero-core = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.8", default-features = false } +edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false } +edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false } +edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false } +edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false } +edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277" } +edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false } env_logger = "0.11" error-stack = "0.6" esi = "0.7.2" diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index caba714d6..db6e6d71d 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1,6 +1,7 @@ use core::future::Future; use std::sync::Arc; +use edgezero_adapter_axum::context::AxumRequestContext; use edgezero_core::app::Hooks; use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; @@ -37,7 +38,8 @@ use trusted_server_core::settings_data::{ default_config_key, default_config_store_name, get_settings_from_config_store, }; -use trusted_server_core::platform::RuntimeServices; +use trusted_server_core::platform::{ClientInfo, RuntimeServices}; +use trusted_server_core::trace::{TraceMetadata, TracePreDispatchHook}; use crate::middleware::{AuthMiddleware, FinalizeResponseMiddleware, SanitizeRequestMiddleware}; use crate::platform::{AxumPlatformConfigStore, AxumPlatformSecretStore, build_runtime_services}; @@ -673,8 +675,13 @@ impl TrustedServerApp { fn build_router(state: &Arc) -> RouterService { let fallback = fallback_handler(Arc::clone(state)); + let trace_state = Arc::clone(state); let mut router = RouterService::builder() + .pre_dispatch_hook(Arc::new(TracePreDispatchHook::new( + Arc::clone(&state.settings), + Arc::new(move |request| trace_metadata(&trace_state, request)), + ))) // Outermost middleware: strips the configured trusted-client-IP // headers before anything else sees the request. Must stay first — // any middleware registered ahead of it would observe the @@ -715,3 +722,134 @@ fn build_router(state: &Arc) -> RouterService { router.build() } + +fn trace_metadata(state: &AppState, request: &Request) -> TraceMetadata { + if let Some(services) = &state.services { + let client_info = services.client_info().clone(); + let geo = client_info.client_ip.and_then(|client_ip| { + services.geo().lookup(Some(client_ip)).unwrap_or_else(|_| { + log::warn!("trace_geo_unavailable"); + None + }) + }); + return TraceMetadata { client_info, geo }; + } + TraceMetadata { + client_info: ClientInfo { + client_ip: AxumRequestContext::get(request) + .and_then(|context| context.remote_addr) + .map(|peer| peer.ip()), + ..ClientInfo::default() + }, + geo: None, + } +} +#[cfg(test)] +mod trace_dispatch_tests { + use super::*; + use futures::executor::block_on; + use trusted_server_core::trace::TraceTerminalResponse; + + fn router(enabled: bool) -> RouterService { + let settings = Settings::from_toml(&format!( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "example-user" + password = "example-password" + [publisher] + domain = "publisher.example.com" + cookie_domain = ".publisher.example.com" + origin_url = "https://origin.example.com" + proxy_secret = "fictional-proxy-secret" + [ec] + passphrase = "fictional-passphrase-at-least-32-bytes" + [request_signing] + enabled = false + config_store_id = "fictional-config" + secret_store_id = "fictional-secrets" + [integrations.gpt_diagnostics] + enabled = true + trace_page_enabled = {enabled} + "# + )) + .expect("should parse trace settings"); + TrustedServerApp::routes_with_settings(settings).expect("should build adapter routes") + } + + #[tokio::test] + async fn trace_dispatch_reserves_all_methods_before_ordinary_lifecycle() { + let router = router(true); + for (method, path, status) in [ + (Method::GET, "/_ts/trace/state", StatusCode::OK), + (Method::HEAD, "/_ts/trace", StatusCode::OK), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN), + ( + Method::PATCH, + "/_ts/trace/state", + StatusCode::METHOD_NOT_ALLOWED, + ), + ( + Method::from_bytes(b"EXAMPLE-METHOD").expect("should parse extension method"), + "/_ts/trace", + StatusCode::METHOD_NOT_ALLOWED, + ), + (Method::GET, "/_ts/trace/extra", StatusCode::NOT_FOUND), + (Method::GET, "/%5Fts/trace", StatusCode::BAD_REQUEST), + ] { + let head = method == Method::HEAD; + let request = edgezero_core::http::request_builder() + .method(method) + .uri(format!("https://publisher.example.com{path}")) + .body(edgezero_core::body::Body::empty()) + .expect("should build trace request"); + let response = + block_on(router.oneshot(request)).expect("should return local trace policy"); + assert_eq!(response.status(), status, "should bypass ordinary dispatch"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark terminal trace responses" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should never manufacture a mutation" + ); + if head { + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer HEAD") + .len(), + 0, + "should remove HEAD bodies" + ); + } + } + let response = block_on( + self::router(false).oneshot( + edgezero_core::http::request_builder() + .uri("/_ts/trace/state") + .body(edgezero_core::body::Body::empty()) + .expect("should build disabled request"), + ), + ) + .expect("should reserve disabled namespace"); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should hide disabled feature" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should harden disabled feature" + ); + } +} diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 87b9567e7..834396c61 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -25,9 +25,9 @@ use trusted_server_core::ec::admin::{ use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; -use trusted_server_core::platform::RuntimeServices; #[cfg(target_arch = "wasm32")] use trusted_server_core::platform::StoreName; +use trusted_server_core::platform::{ClientInfo, GeoInfo, RuntimeServices}; use trusted_server_core::proxy::{ handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign, @@ -41,6 +41,7 @@ use trusted_server_core::request_signing::{ handle_trusted_server_discovery, handle_verify_signature, }; use trusted_server_core::settings::Settings; +use trusted_server_core::trace::{TraceMetadata, TracePreDispatchHook}; use crate::middleware::{AuthMiddleware, FinalizeResponseMiddleware, SanitizeRequestMiddleware}; use crate::platform::build_runtime_services; @@ -596,7 +597,12 @@ fn build_router(state: &Arc) -> RouterService { } }; + let trace_state = Arc::clone(&state); let mut router = RouterService::builder() + .pre_dispatch_hook(Arc::new(TracePreDispatchHook::new( + Arc::clone(&state.settings), + Arc::new(move |request| trace_metadata(&trace_state, request)), + ))) // Outermost middleware: strips the configured trusted-client-IP // headers before anything else sees the request. Must stay first — // any middleware registered ahead of it would observe the @@ -773,6 +779,46 @@ fn build_router(state: &Arc) -> RouterService { } } +fn trace_metadata(state: &AppState, request: &Request) -> TraceMetadata { + if let Some(services) = &state.services { + let client_info = services.client_info().clone(); + let geo = client_info.client_ip.and_then(|client_ip| { + services.geo().lookup(Some(client_ip)).unwrap_or_else(|_| { + log::warn!("trace_geo_unavailable"); + None + }) + }); + return TraceMetadata { client_info, geo }; + } + let client_ip = request + .headers() + .get("cf-connecting-ip") + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.parse().ok()); + let geo = request + .headers() + .get("cf-ipcountry") + .and_then(|value| value.to_str().ok()) + .filter(|country| *country != "XX") + .map(|country| GeoInfo { + city: String::new(), + country: country.to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }); + TraceMetadata { + client_info: ClientInfo { + client_ip, + ..ClientInfo::default() + }, + geo, + } +} + #[cfg(test)] mod tests { use super::*; @@ -1017,3 +1063,112 @@ mod tests { ); } } +#[cfg(test)] +mod trace_dispatch_tests { + use super::*; + use futures::executor::block_on; + use trusted_server_core::trace::TraceTerminalResponse; + + fn router(enabled: bool) -> RouterService { + let settings = Settings::from_toml(&format!( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "example-user" + password = "example-password" + [publisher] + domain = "publisher.example.com" + cookie_domain = ".publisher.example.com" + origin_url = "https://origin.example.com" + proxy_secret = "fictional-proxy-secret" + [ec] + passphrase = "fictional-passphrase-at-least-32-bytes" + [request_signing] + enabled = false + config_store_id = "fictional-config" + secret_store_id = "fictional-secrets" + [integrations.gpt_diagnostics] + enabled = true + trace_page_enabled = {enabled} + "# + )) + .expect("should parse trace settings"); + TrustedServerApp::routes_with_settings(settings).expect("should build adapter routes") + } + + #[test] + fn trace_dispatch_reserves_all_methods_before_ordinary_lifecycle() { + let router = router(true); + for (method, path, status) in [ + (Method::GET, "/_ts/trace/state", StatusCode::OK), + (Method::HEAD, "/_ts/trace", StatusCode::OK), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN), + ( + Method::PATCH, + "/_ts/trace/state", + StatusCode::METHOD_NOT_ALLOWED, + ), + ( + Method::from_bytes(b"EXAMPLE-METHOD").expect("should parse extension method"), + "/_ts/trace", + StatusCode::METHOD_NOT_ALLOWED, + ), + (Method::GET, "/_ts/trace/extra", StatusCode::NOT_FOUND), + (Method::GET, "/%5Fts/trace", StatusCode::BAD_REQUEST), + ] { + let head = method == Method::HEAD; + let request = edgezero_core::http::request_builder() + .method(method) + .uri(format!("https://publisher.example.com{path}")) + .body(edgezero_core::body::Body::empty()) + .expect("should build trace request"); + let response = + block_on(router.oneshot(request)).expect("should return local trace policy"); + assert_eq!(response.status(), status, "should bypass ordinary dispatch"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark terminal trace responses" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should never manufacture a mutation" + ); + if head { + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer HEAD") + .len(), + 0, + "should remove HEAD bodies" + ); + } + } + let response = block_on( + self::router(false).oneshot( + edgezero_core::http::request_builder() + .uri("/_ts/trace/state") + .body(edgezero_core::body::Body::empty()) + .expect("should build disabled request"), + ), + ) + .expect("should reserve disabled namespace"); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should hide disabled feature" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should harden disabled feature" + ); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index b19653c9e..1c3fedbc9 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -125,7 +125,7 @@ use trusted_server_core::integrations::{ RequestFilterRegistryOutcome, }; use trusted_server_core::platform::{ - ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices, StoreName, + ClientInfo, GeoInfo, PlatformGeo as _, PlatformKvStore, RuntimeServices, StoreName, }; use trusted_server_core::proxy::{ AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click, @@ -143,6 +143,7 @@ use trusted_server_core::request_signing::{ use trusted_server_core::settings::{ProxyAssetRoute, Settings}; use trusted_server_core::settings_data::{DEFAULT_CONFIG_STORE_ID, get_settings_from_config_store}; use trusted_server_core::tester_cookie::{handle_clear_tester, handle_set_tester}; +use trusted_server_core::trace::{TraceMetadata, TracePreDispatchHook}; use crate::middleware::{AuthMiddleware, FinalizeResponseMiddleware}; use crate::platform::{ @@ -1287,6 +1288,10 @@ impl TrustedServerApp { fn routes_for_state(state: &Arc) -> RouterService { let mut router = RouterService::builder() + .pre_dispatch_hook(Arc::new(TracePreDispatchHook::new( + Arc::clone(&state.settings), + Arc::new(trace_metadata), + ))) .middleware(FinalizeResponseMiddleware::new( Arc::clone(&state.settings), Arc::new(FastlyPlatformGeo), @@ -1326,6 +1331,26 @@ impl TrustedServerApp { } } +fn trace_metadata(request: &Request) -> TraceMetadata { + let client_info = request + .extensions() + .get::() + .cloned() + .unwrap_or_else(|| ClientInfo { + client_ip: FastlyRequestContext::get(request).and_then(|context| context.client_ip), + ..ClientInfo::default() + }); + let geo = client_info.client_ip.and_then(|client_ip| { + FastlyPlatformGeo + .lookup(Some(client_ip)) + .unwrap_or_else(|_| { + log::warn!("trace_geo_unavailable"); + None + }) + }); + TraceMetadata { client_info, geo } +} + impl Hooks for TrustedServerApp { fn name() -> &'static str { "TrustedServer" @@ -1657,6 +1682,139 @@ mod tests { /// bot-protection metadata reaches integration filters like `DataDome`. struct ClientInfoCapturingFilter(Arc>>); + struct TraceCountingKv(Arc); + + impl TraceCountingKv { + fn unavailable(&self) -> edgezero_core::key_value_store::KvError { + self.0.fetch_add(1, Ordering::SeqCst); + edgezero_core::key_value_store::KvError::Unavailable + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformKvStore for TraceCountingKv { + async fn get_bytes( + &self, + _key: &str, + ) -> Result, edgezero_core::key_value_store::KvError> { + Err(self.unavailable()) + } + async fn put_bytes( + &self, + _key: &str, + _value: Bytes, + ) -> Result<(), edgezero_core::key_value_store::KvError> { + Err(self.unavailable()) + } + async fn put_bytes_with_ttl( + &self, + _key: &str, + _value: Bytes, + _ttl: Duration, + ) -> Result<(), edgezero_core::key_value_store::KvError> { + Err(self.unavailable()) + } + async fn delete(&self, _key: &str) -> Result<(), edgezero_core::key_value_store::KvError> { + Err(self.unavailable()) + } + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result + { + Err(self.unavailable()) + } + } + + struct TraceCountingTelemetry(Arc); + + #[async_trait::async_trait(?Send)] + impl trusted_server_core::auction::AuctionTelemetrySink for TraceCountingTelemetry { + fn is_enabled(&self) -> bool { + self.0.fetch_add(1, Ordering::SeqCst); + true + } + async fn emit_auction_events( + &self, + _services: &RuntimeServices, + _batch: trusted_server_core::auction::AuctionEventBatch, + ) -> Result<(), Report> { + self.0.fetch_add(1, Ordering::SeqCst); + Ok(()) + } + } + + #[test] + fn trace_dispatch_bypasses_fastly_identity_filters_and_telemetry() { + let mut settings = test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should configure trace"); + let mut state = build_state_from_settings(settings).expect("should build trace state"); + let kv = Arc::new(AtomicUsize::new(0)); + let telemetry = Arc::new(AtomicUsize::new(0)); + let filter = Arc::new(Mutex::new(None)); + let mutable = Arc::get_mut(&mut state) + .expect("should retain unique test state before router construction"); + mutable.default_kv_store = Arc::new(TraceCountingKv(Arc::clone(&kv))); + mutable.auction_telemetry_sink = Arc::new(TraceCountingTelemetry(Arc::clone(&telemetry))); + mutable.registry = Arc::new(IntegrationRegistry::from_request_filters(vec![Arc::new( + ClientInfoCapturingFilter(Arc::clone(&filter)), + )])); + let router = TrustedServerApp::routes_for_state(&state); + for (method, path, status) in [ + (Method::GET, "/_ts/trace/state", StatusCode::OK), + (Method::GET, "/_ts/trace", StatusCode::OK), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN), + (Method::PATCH, "/_ts/trace", StatusCode::METHOD_NOT_ALLOWED), + ] { + let response = route(&router, empty_request(method, path)); + assert_eq!( + response.status(), + status, + "should serve trace through the terminal dispatcher" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "should not assemble ordinary filter effects" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "should not enter identity finalization" + ); + } + assert_eq!( + kv.load(Ordering::SeqCst), + 0, + "should never touch identity KV for trace" + ); + assert_eq!( + telemetry.load(Ordering::SeqCst), + 0, + "should never inspect or emit auction telemetry for trace" + ); + assert!( + filter + .lock() + .expect("should inspect filter counter") + .is_none(), + "should never run integration request filters for trace" + ); + } + #[async_trait::async_trait(?Send)] impl IntegrationRequestFilter for ClientInfoCapturingFilter { fn integration_id(&self) -> &'static str { @@ -3347,3 +3505,114 @@ mod tests { ); } } +#[cfg(test)] +mod trace_dispatch_tests { + use super::*; + use futures::executor::block_on; + use trusted_server_core::trace::TraceTerminalResponse; + + fn router(enabled: bool) -> RouterService { + let settings = Settings::from_toml(&format!( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "example-user" + password = "example-password" + [publisher] + domain = "publisher.example.com" + cookie_domain = ".publisher.example.com" + origin_url = "https://origin.example.com" + proxy_secret = "fictional-proxy-secret" + [ec] + passphrase = "fictional-passphrase-at-least-32-bytes" + [request_signing] + enabled = false + config_store_id = "fictional-config" + secret_store_id = "fictional-secrets" + [integrations.gpt_diagnostics] + enabled = true + trace_page_enabled = {enabled} + "# + )) + .expect("should parse trace settings"); + TrustedServerApp::routes_for_state( + &build_state_from_settings(settings).expect("should build Fastly state"), + ) + } + + #[test] + fn trace_dispatch_reserves_all_methods_before_ordinary_lifecycle() { + let router = router(true); + for (method, path, status) in [ + (Method::GET, "/_ts/trace/state", StatusCode::OK), + (Method::HEAD, "/_ts/trace", StatusCode::OK), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN), + ( + Method::PATCH, + "/_ts/trace/state", + StatusCode::METHOD_NOT_ALLOWED, + ), + ( + Method::from_bytes(b"EXAMPLE-METHOD").expect("should parse extension method"), + "/_ts/trace", + StatusCode::METHOD_NOT_ALLOWED, + ), + (Method::GET, "/_ts/trace/extra", StatusCode::NOT_FOUND), + (Method::GET, "/%5Fts/trace", StatusCode::BAD_REQUEST), + ] { + let head = method == Method::HEAD; + let request = edgezero_core::http::request_builder() + .method(method) + .uri(format!("https://publisher.example.com{path}")) + .body(edgezero_core::body::Body::empty()) + .expect("should build trace request"); + let response = + block_on(router.oneshot(request)).expect("should return local trace policy"); + assert_eq!(response.status(), status, "should bypass ordinary dispatch"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark terminal trace responses" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should never manufacture a mutation" + ); + if head { + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer HEAD") + .len(), + 0, + "should remove HEAD bodies" + ); + } + } + let response = block_on( + self::router(false).oneshot( + edgezero_core::http::request_builder() + .uri("/_ts/trace/state") + .body(edgezero_core::body::Body::empty()) + .expect("should build disabled request"), + ), + ) + .expect("should reserve disabled namespace"); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should hide disabled feature" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should harden disabled feature" + ); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ce7264e78..c1444ccee 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,7 +1,7 @@ use std::sync::Arc; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; -use edgezero_adapter_fastly::request::into_core_request; +use edgezero_adapter_fastly::request::{capture_request_ingress, into_core_request_with_ingress}; use edgezero_adapter_fastly::runtime_env_config; use edgezero_core::app::Hooks as _; use edgezero_core::body::Body as EdgeBody; @@ -27,6 +27,7 @@ use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; +use trusted_server_core::trace::{TraceTerminalResponse, is_trace_path}; mod app; mod backend; @@ -46,7 +47,7 @@ use crate::app::{ }; use crate::ec_kv::FastlyEcKvStore; use crate::middleware::{HEADER_X_TS_FINALIZED, apply_finalize_headers, resolve_geo_for_response}; -use crate::platform::{FastlyPlatformGeo, client_info_from_request}; +use crate::platform::{FastlyPlatformGeo, client_info_from_request, resolve_client_ip}; use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. @@ -76,26 +77,32 @@ fn health_response(req: &FastlyRequest) -> Option { /// [`fastly::Response::stream_to_client`] explicitly. fn main() { let req = FastlyRequest::from_client(); + let ingress = capture_request_ingress(&req); + let trace_path = is_trace_path(req.get_path()); // Health probe bypasses logging, settings, and app construction as a cheap liveness signal. - if let Some(response) = health_response(&req) { + if !trace_path && let Some(response) = health_response(&req) { response.send_to_client(); return; } logging::init_logger(); - edgezero_main(req); + edgezero_main(req, ingress, trace_path); } /// Handles a request through the `EdgeZero` router path. -fn edgezero_main(mut req: FastlyRequest) { +fn edgezero_main( + mut req: FastlyRequest, + ingress: edgezero_core::request::RequestIngress, + trace_path: bool, +) { let runtime_env = runtime_env_config(TrustedServerApp::stores()); let runtime_stores = RuntimeStoreConfig::from_env(&runtime_env); // Short-circuit the JA4 debug probe before app construction. Must run here // because TLS/JA4 accessors are only available on FastlyRequest before // conversion to edgezero types. - if req.get_method() == FastlyMethod::GET && req.get_path() == "/_ts/debug/ja4" { + if !trace_path && req.get_method() == FastlyMethod::GET && req.get_path() == "/_ts/debug/ja4" { match load_settings_from_config_store(&runtime_stores) { Ok(settings) if settings.debug.ja4_endpoint_enabled => { build_ja4_debug_response(&req).send_to_client(); @@ -134,32 +141,38 @@ fn edgezero_main(mut req: FastlyRequest) { // Resolve the trusted client IP, then strip client-spoofable forwarded // headers before dispatch. One call keeps resolution ahead of the // sanitization that removes the headers it reads. - let resolved_client_ip = compat::resolve_and_sanitize_client_ip(&mut req, trusted_client_ip); - - // Re-inject a trusted TLS scheme signal after sanitization has stripped any - // client-sent fastly-ssl header. Setting it from Fastly's native TLS - // metadata here is authoritative. detect_request_scheme in http_util checks - // this header so scheme-sensitive logic produces https URLs on HTTPS traffic. - if req.get_tls_protocol().ok().flatten().is_some() - || req.get_tls_cipher_openssl_name().ok().flatten().is_some() - { - req.set_header("fastly-ssl", "1"); - } + let resolved_client_ip = if trace_path { + resolve_client_ip(&req, req.get_client_ip_addr(), trusted_client_ip) + } else { + compat::resolve_and_sanitize_client_ip(&mut req, trusted_client_ip) + }; - // Strip any client-supplied x-ts-tls-* headers before injecting the trusted - // values from the Fastly SDK. Must run after sanitize_fastly_forwarded_headers. - req.remove_header("x-ts-tls-protocol"); - req.remove_header("x-ts-tls-cipher"); - if let Some(proto) = req.get_tls_protocol().ok().flatten().map(str::to_owned) { - req.set_header("x-ts-tls-protocol", proto); - } - if let Some(cipher) = req - .get_tls_cipher_openssl_name() - .ok() - .flatten() - .map(str::to_owned) - { - req.set_header("x-ts-tls-cipher", cipher); + if !trace_path { + // Re-inject a trusted TLS scheme signal after sanitization has stripped any + // client-sent fastly-ssl header. Setting it from Fastly's native TLS + // metadata here is authoritative. detect_request_scheme in http_util checks + // this header so scheme-sensitive logic produces https URLs on HTTPS traffic. + if req.get_tls_protocol().ok().flatten().is_some() + || req.get_tls_cipher_openssl_name().ok().flatten().is_some() + { + req.set_header("fastly-ssl", "1"); + } + + // Strip any client-supplied x-ts-tls-* headers before injecting the trusted + // values from the Fastly SDK. Must run after sanitize_fastly_forwarded_headers. + req.remove_header("x-ts-tls-protocol"); + req.remove_header("x-ts-tls-cipher"); + if let Some(proto) = req.get_tls_protocol().ok().flatten().map(str::to_owned) { + req.set_header("x-ts-tls-protocol", proto); + } + if let Some(cipher) = req + .get_tls_cipher_openssl_name() + .ok() + .flatten() + .map(str::to_owned) + { + req.set_header("x-ts-tls-cipher", cipher); + } } // Capture metadata from the original FastlyRequest before conversion. These @@ -167,15 +180,17 @@ fn edgezero_main(mut req: FastlyRequest) { // request extensions for build_per_request_services and EC bot classification. let client_info = client_info_from_request(&req, resolved_client_ip); let client_ip = client_info.client_ip; - let device_signals = derive_device_signals(&req); + let device_signals = (!trace_path).then(|| derive_device_signals(&req)); // Dispatch directly through the EdgeZero router without an intermediate // fastly::Response conversion. That preserves duplicate header values such // as multiple Set-Cookie headers. - let mut response = match into_core_request(req) { + let mut response = match into_core_request_with_ingress(req, ingress) { Ok(mut core_req) => { core_req.extensions_mut().insert(config_store); - core_req.extensions_mut().insert(device_signals); + if let Some(device_signals) = device_signals { + core_req.extensions_mut().insert(device_signals); + } core_req.extensions_mut().insert(client_info); match futures::executor::block_on(app.router().oneshot(core_req)) { Ok(response) => response, @@ -191,6 +206,15 @@ fn edgezero_main(mut req: FastlyRequest) { } }; + if response + .extensions() + .get::() + .is_some() + { + send_response_to_client(response); + return; + } + // Pop response extensions before the Fastly conversion, which drops them. let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); @@ -285,6 +309,13 @@ fn apply_entry_point_finalize_headers( response: &mut HttpResponse, client_ip: Option, ) { + if response + .extensions() + .get::() + .is_some() + { + return; + } let geo_info = resolve_geo_for_response(response, client_ip, |client_ip| { FastlyPlatformGeo.lookup(client_ip).unwrap_or_else(|e| { log::warn!("entry-point geo lookup failed: {e}"); @@ -373,6 +404,10 @@ fn send_edgezero_response( ) { apply_terminal_response_effects(&mut response, request_filter_effects); + send_response_to_client(response); +} + +fn send_response_to_client(response: HttpResponse) { let (parts, body) = response.into_parts(); match body { @@ -405,6 +440,13 @@ fn apply_terminal_response_effects( response: &mut HttpResponse, request_filter_effects: Option<&RequestFilterEffects>, ) { + if response + .extensions() + .get::() + .is_some() + { + return; + } let must_remain_private = response .extensions() .get::() @@ -522,9 +564,14 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { use super::*; + use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; - use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; + use edgezero_core::http::{HeaderValue, Method, StatusCode}; + use edgezero_core::request::{ + CapturedTarget, HeaderFidelity, InboundOrigin, OriginSource, RequestIngress, + TargetUnavailable, + }; use fastly::mime; use trusted_server_core::integrations::HeaderMutation; @@ -554,6 +601,218 @@ mod tests { .expect("should parse test settings") } + #[test] + fn trace_dispatch_terminal_asset_bypasses_hostile_native_finalizers() { + let mut settings = test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should configure trace"); + settings.handlers.insert(0,serde_json::from_value(serde_json::json!({"path":"^/_ts/trace","username":"example-user","password":"example-password"})) + .expect("should configure trace protection")); + let mut request = edgezero_core::http::request_builder() + .uri("https://publisher.example.com/_ts/trace/assets/v1.js") + .header( + "authorization", + format!( + "Basic {}", + base64::engine::general_purpose::STANDARD + .encode("example-user:example-password") + ), + ) + .body(EdgeBody::empty()) + .expect("should build protected asset request"); + let trusted_server_core::trace::TracePreflight::Ready(dispatch) = + trusted_server_core::trace::preflight(&settings, &mut request) + else { + panic!("should accept protected fixed asset"); + }; + let mut response = dispatch.fixed_asset(b"example immutable asset"); + let original = response.headers().clone(); + let effects = RequestFilterEffects { + request_headers: Vec::new(), + response_headers: vec![ + HeaderMutation::set("content-security-policy", "default-src *"), + HeaderMutation::set("content-type", "text/plain"), + HeaderMutation::append("set-cookie", "example=operator"), + HeaderMutation::set("cache-control", "public, max-age=3600"), + ], + }; + apply_entry_point_finalize_headers(&settings, &mut response, None); + apply_terminal_response_effects(&mut response, Some(&effects)); + assert_eq!( + response.headers(), + &original, + "should bypass every ordinary trace finalizer and preserve protected digest" + ); + let native = compat::to_fastly_response(response); + assert_eq!( + native.get_header("etag").map(HeaderValue::as_bytes), + original.get("etag").map(HeaderValue::as_bytes), + "should preserve strong ETag through native conversion" + ); + assert!( + native.get_header("set-cookie").is_none(), + "should never acquire an operator cookie" + ); + } + + #[test] + fn trace_dispatch_terminal_actions_errors_and_head_preserve_native_contract() { + let mut settings = test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should configure trace"); + settings.handlers.insert(0,serde_json::from_value(serde_json::json!({"path":"^/_ts/trace","username":"example-user","password":"example-password"})) + .expect("should configure trace protection")); + let hook = trusted_server_core::trace::TracePreDispatchHook::new( + Arc::new(settings.clone()), + Arc::new(|_| panic!("should never query metadata for actions, errors or HEAD")), + ); + let effects = RequestFilterEffects { + request_headers: Vec::new(), + response_headers: vec![ + HeaderMutation::set("content-security-policy", "default-src *"), + HeaderMutation::set("content-type", "text/plain"), + HeaderMutation::append("set-cookie", "example=operator"), + HeaderMutation::set("cache-control", "public, max-age=3600"), + ], + }; + for (method, path, authenticated, expected, action) in [ + ( + Method::GET, + "/_ts/trace", + false, + StatusCode::UNAUTHORIZED, + None, + ), + ( + Method::HEAD, + "/_ts/trace", + false, + StatusCode::UNAUTHORIZED, + None, + ), + (Method::HEAD, "/_ts/trace", true, StatusCode::OK, None), + ( + Method::PATCH, + "/_ts/trace/state", + true, + StatusCode::METHOD_NOT_ALLOWED, + None, + ), + ( + Method::POST, + "/_ts/trace/enable", + true, + StatusCode::OK, + Some("enable"), + ), + ( + Method::POST, + "/_ts/trace/end", + true, + StatusCode::OK, + Some("end"), + ), + ] { + let head = method == Method::HEAD; + let mut request = edgezero_core::http::request_builder() + .method(method) + .uri(path) + .header("host", "publisher.example.com") + .body(EdgeBody::empty()) + .expect("should build terminal trace request"); + if authenticated { + request.headers_mut().insert( + "authorization", + HeaderValue::from_str(&format!( + "Basic {}", + base64::engine::general_purpose::STANDARD + .encode("example-user:example-password") + )) + .expect("should encode fictional trace credentials"), + ); + } + if let Some(action) = action { + request.headers_mut().insert( + "origin", + HeaderValue::from_static("https://publisher.example.com"), + ); + request + .headers_mut() + .insert("sec-fetch-site", HeaderValue::from_static("same-origin")); + request.headers_mut().insert( + "x-ts-trace-action", + HeaderValue::from_str(action).expect("should encode fixed action"), + ); + request.extensions_mut().insert( + RequestIngress::new( + CapturedTarget::Unavailable(TargetUnavailable::NotExposed), + Some( + InboundOrigin::parse( + "https", + "publisher.example.com", + OriginSource::RuntimeUri, + ) + .expect("should validate fictional runtime origin"), + ), + HeaderFidelity::default(), + vec![], + ) + .expect("should freeze trusted runtime origin"), + ); + } + let mut response = futures::executor::block_on( + edgezero_core::router::PreDispatchHook::handle(&hook, &mut request), + ) + .expect("should build local trace response") + .expect("should intercept reserved request"); + assert_eq!( + response.status(), + expected, + "should preserve action and preflight status" + ); + let original = response.headers().clone(); + apply_entry_point_finalize_headers(&settings, &mut response, None); + apply_terminal_response_effects(&mut response, Some(&effects)); + assert_eq!( + response.headers(), + &original, + "should bypass hostile ordinary finalizers for every terminal response" + ); + let mut native = compat::to_fastly_response(response); + for (name, value) in &original { + assert_eq!( + native.get_header(name.as_str()).map(HeaderValue::as_bytes), + Some(value.as_bytes()), + "should preserve complete trace headers through native conversion" + ); + } + assert_eq!( + native + .get_headers() + .filter(|(name, _)| name.as_str() == "set-cookie") + .count(), + usize::from(action.is_some()), + "should emit only one deliberate action cookie and none on reads or errors" + ); + if head { + assert!( + native.take_body_bytes().is_empty(), + "should preserve bodyless trace HEAD through native conversion" + ); + } + } + } + #[test] fn pull_sync_noop_states_skip_post_send_graph_factory() { let calls = std::cell::Cell::new(0); diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index b8e8a2492..57338e769 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -28,7 +28,7 @@ use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::http_util::sanitize_forwarded_headers; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; -use trusted_server_core::platform::RuntimeServices; +use trusted_server_core::platform::{ClientInfo, RuntimeServices}; #[cfg(all(feature = "spin", target_arch = "wasm32"))] use trusted_server_core::platform::{PlatformConfigStore, StoreName}; use trusted_server_core::proxy::{ @@ -46,6 +46,7 @@ use trusted_server_core::request_signing::{ use trusted_server_core::settings::Settings; #[cfg(all(feature = "spin", target_arch = "wasm32"))] use trusted_server_core::settings_data::{default_config_key, default_secret_store_name}; +use trusted_server_core::trace::{TraceMetadata, TracePreDispatchHook}; use crate::middleware::{ AuthMiddleware, FinalizeResponseMiddleware, NormalizeMiddleware, SanitizeRequestMiddleware, @@ -880,7 +881,12 @@ fn build_router(state: &Arc) -> RouterService { let legacy_admin_deny = |_ctx: RequestContext| async { Ok::(legacy_admin_alias_denied()) }; + let trace_state = Arc::clone(&state); let mut builder = RouterService::builder() + .pre_dispatch_hook(Arc::new(TracePreDispatchHook::new( + Arc::clone(&state.settings), + Arc::new(move |request| trace_metadata(&trace_state, request)), + ))) // Outermost middleware: strips the configured trusted-client-IP // headers before anything else sees the request. Must stay first — // any middleware registered ahead of it would observe the @@ -973,6 +979,33 @@ fn build_router(state: &Arc) -> RouterService { } } +fn trace_metadata(state: &AppState, request: &Request) -> TraceMetadata { + if let Some(services) = &state.services { + let client_info = services.client_info().clone(); + let geo = client_info.client_ip.and_then(|client_ip| { + services.geo().lookup(Some(client_ip)).unwrap_or_else(|_| { + log::warn!("trace_geo_unavailable"); + None + }) + }); + return TraceMetadata { client_info, geo }; + } + let client_ip = request + .headers() + .get_all("spin-client-addr") + .iter() + .next_back() + .and_then(|value| value.to_str().ok()) + .and_then(parse_client_addr); + TraceMetadata { + client_info: ClientInfo { + client_ip, + ..ClientInfo::default() + }, + geo: None, + } +} + #[cfg(test)] mod tests { use super::*; @@ -1401,3 +1434,156 @@ mod tests { ); } } +#[cfg(test)] +mod trace_dispatch_tests { + use super::*; + use futures::executor::block_on; + use trusted_server_core::trace::TraceTerminalResponse; + + fn router(enabled: bool) -> RouterService { + let settings = Settings::from_toml(&format!( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "example-user" + password = "example-password" + [publisher] + domain = "publisher.example.com" + cookie_domain = ".publisher.example.com" + origin_url = "https://origin.example.com" + proxy_secret = "fictional-proxy-secret" + [ec] + passphrase = "fictional-passphrase-at-least-32-bytes" + [request_signing] + enabled = false + config_store_id = "fictional-config" + secret_store_id = "fictional-secrets" + [integrations.gpt_diagnostics] + enabled = true + trace_page_enabled = {enabled} + "# + )) + .expect("should parse trace settings"); + TrustedServerApp::routes_with_settings(settings).expect("should build adapter routes") + } + + #[test] + fn trace_dispatch_reserves_all_methods_before_ordinary_lifecycle() { + let router = router(true); + for (method, path, status) in [ + (Method::GET, "/_ts/trace/state", StatusCode::OK), + (Method::HEAD, "/_ts/trace", StatusCode::OK), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN), + ( + Method::PATCH, + "/_ts/trace/state", + StatusCode::METHOD_NOT_ALLOWED, + ), + ( + Method::from_bytes(b"EXAMPLE-METHOD").expect("should parse extension method"), + "/_ts/trace", + StatusCode::METHOD_NOT_ALLOWED, + ), + (Method::GET, "/_ts/trace/extra", StatusCode::NOT_FOUND), + (Method::GET, "/%5Fts/trace", StatusCode::BAD_REQUEST), + ] { + let head = method == Method::HEAD; + let request = edgezero_core::http::request_builder() + .method(method) + .uri(format!("https://publisher.example.com{path}")) + .body(edgezero_core::body::Body::empty()) + .expect("should build trace request"); + let response = + block_on(router.oneshot(request)).expect("should return local trace policy"); + assert_eq!(response.status(), status, "should bypass ordinary dispatch"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark terminal trace responses" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should never manufacture a mutation" + ); + if head { + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer HEAD") + .len(), + 0, + "should remove HEAD bodies" + ); + } + } + let response = block_on( + self::router(false).oneshot( + edgezero_core::http::request_builder() + .uri("/_ts/trace/state") + .body(edgezero_core::body::Body::empty()) + .expect("should build disabled request"), + ), + ) + .expect("should reserve disabled namespace"); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should hide disabled feature" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should harden disabled feature" + ); + } + + #[test] + fn trace_dispatch_setup_uses_last_runtime_client_addr_without_normalizing_request() { + let router = router(true); + let mut request = edgezero_core::http::request_builder() + .uri("/_ts/trace") + .header("spin-client-addr", "203.0.113.22:1234") + .header("spin-full-url", "https://untrusted.example.com/other") + .body(edgezero_core::body::Body::empty()) + .expect("should build Spin setup"); + request.headers_mut().append( + "spin-client-addr", + HeaderValue::from_static("192.0.2.99:4321"), + ); + SpinRequestContext::insert( + &mut request, + SpinRequestContext { + client_addr: Some( + "203.0.113.22" + .parse() + .expect("should parse spoofed example IP"), + ), + full_url: Some("https://untrusted.example.com/other".to_owned()), + }, + ); + let response = block_on(router.oneshot(request)).expect("should render setup locally"); + let bytes = response + .into_body() + .into_bytes() + .expect("should buffer setup"); + let html = String::from_utf8(bytes.to_vec()).expect("should render UTF8 setup"); + assert!( + html.contains("192.0.2.0/24"), + "should project the trusted last synthetic address" + ); + assert!( + !html.contains("203.0.113."), + "should not use the spoofed first SDK-context address" + ); + assert!( + !html.contains("untrusted.example.com"), + "should never project or trust spin-full-url" + ); + } +} diff --git a/crates/trusted-server-core/Cargo.toml b/crates/trusted-server-core/Cargo.toml index 3dbf41df3..63cd13399 100644 --- a/crates/trusted-server-core/Cargo.toml +++ b/crates/trusted-server-core/Cargo.toml @@ -40,7 +40,7 @@ mime = { workspace = true } rand = { workspace = true } regex = { workspace = true } serde = { workspace = true } -serde_json = { workspace = true } +serde_json = { workspace = true, features = ["raw_value"] } sha2 = { workspace = true } subtle = { workspace = true } toml = { workspace = true } diff --git a/crates/trusted-server-core/benches/html_processor_bench.rs b/crates/trusted-server-core/benches/html_processor_bench.rs index 19aa0b82f..49d2ed83c 100644 --- a/crates/trusted-server-core/benches/html_processor_bench.rs +++ b/crates/trusted-server-core/benches/html_processor_bench.rs @@ -16,6 +16,7 @@ fn make_config() -> HtmlProcessorConfig { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, // The benchmark measures URL rewriting, not ad injection, and // `ad_slots_script` is `None` here — matching the previous behaviour, // which inferred no body-close work from that. diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index ab3585e3d..572c375fc 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -23,17 +23,18 @@ use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; use crate::platform::RuntimeServices; use crate::settings::Settings; +use crate::trace::{ + TraceAuctionCarry, TraceAuctionSource, TraceAuctionTerminalReason, TraceAuctionTerminalStatus, + TraceCaptureGate, TraceClientSlotRefs, +}; use super::AuctionOrchestrator; -use super::formats::{ - convert_to_openrtb_response, convert_to_openrtb_response_with_report, - convert_tsjs_to_auction_request, -}; +use super::formats::{convert_to_openrtb_response_with_trace, convert_tsjs_to_auction_request}; use super::telemetry::{ AuctionObservationContext, AuctionSource, AuctionTerminalOutcome, build_auction_events, emit_auction_events_best_effort_lazy, }; -use super::types::AuctionContext; +use super::types::{AuctionContext, AuctionRequest}; const MAX_CLIENT_EID_SOURCES: usize = 64; const MAX_CLIENT_UIDS_PER_SOURCE: usize = 32; @@ -167,7 +168,16 @@ pub async fn handle_auction( body.ad_units.len() ); - let http_req = Request::from_parts(parts, EdgeBody::empty()); + let mut http_req = Request::from_parts(parts, EdgeBody::empty()); + if http_req + .extensions() + .get::() + .is_some_and(TraceCaptureGate::base_active) + { + http_req + .extensions_mut() + .insert(TraceClientSlotRefs::from_raw(&body_bytes)); + } // Story 5 middleware contract: auction is a read-only EC route. // It must not generate EC IDs; it only consumes pre-routed context. @@ -192,6 +202,13 @@ pub async fn handle_auction( ec_id.as_deref(), None, )?; + let trace = capture_api_trace(&mut http_req, &auction_request); + if let Some(trace) = &trace { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } let observation = AuctionObservationContext::from_auction_request( AuctionSource::AuctionApi, &auction_request, @@ -216,12 +233,14 @@ pub async fn handle_auction( total_time_ms: 0, metadata: HashMap::new(), }; - return convert_to_openrtb_response( + return convert_to_openrtb_response_with_trace( &empty_result, settings, &auction_request, ec_context.ec_allowed(), - ); + trace.as_ref(), + ) + .map(|conversion| retain_private_trace(conversion.response, trace)); } // Server-side auction consent gate. The publisher-navigation and @@ -246,6 +265,13 @@ pub async fn handle_auction( ec_id.as_deref(), None, )?; + let trace = capture_api_trace(&mut http_req, &auction_request); + if let Some(trace) = &trace { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } let observation = AuctionObservationContext::from_auction_request( AuctionSource::AuctionApi, &auction_request, @@ -269,12 +295,14 @@ pub async fn handle_auction( total_time_ms: 0, metadata: HashMap::new(), }; - return convert_to_openrtb_response( + return convert_to_openrtb_response_with_trace( &empty_result, settings, &auction_request, ec_context.ec_allowed(), - ); + trace.as_ref(), + ) + .map(|conversion| retain_private_trace(conversion.response, trace)); } // Parse client-provided EIDs from the current request body. When the @@ -346,6 +374,9 @@ pub async fn handle_auction( log::warn!("Auction EIDs stripped by TCF consent gating"); } + let trace = capture_api_trace(&mut http_req, &auction_request); + let _trace_cancellation = trace.as_ref().map(TraceAuctionCarry::cancellation_guard); + // Create auction context let context = AuctionContext { settings, @@ -385,11 +416,12 @@ pub async fn handle_auction( } }; - let conversion = match convert_to_openrtb_response_with_report( + let conversion = match convert_to_openrtb_response_with_trace( &result, settings, &auction_request, ec_context.ec_allowed(), + trace.as_ref(), ) { Ok(conversion) => conversion, Err(error) => { @@ -430,7 +462,41 @@ pub async fn handle_auction( result.total_time_ms ); - Ok(conversion.response) + Ok(retain_private_trace(conversion.response, trace)) +} + +fn capture_api_trace( + request: &mut Request, + auction: &AuctionRequest, +) -> Option { + let active = request + .extensions() + .get::() + .is_some_and(TraceCaptureGate::base_active); + let carry = TraceAuctionCarry::capture_if_enabled( + active, + TraceAuctionSource::AuctionApi, + &auction.slots, + )?; + if let Some(refs) = request + .extensions() + .get::() + .and_then(TraceClientSlotRefs::accepted_refs) + { + carry.bind_client_refs(&refs); + } + request.extensions_mut().insert(carry.clone()); + Some(carry) +} + +fn retain_private_trace( + mut response: Response, + trace: Option, +) -> Response { + if let Some(trace) = trace { + response.extensions_mut().insert(trace); + } + response } /// Resolves partner EIDs from the KV identity graph for bidstream decoration. @@ -625,15 +691,18 @@ mod tests { use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::ConsentContext; use crate::error::IntoHttpResponse as _; + use crate::integrations::adserver_mock::{AdServerMockConfig, AdServerMockProvider}; use crate::openrtb::Uid; use crate::platform::test_support::{ - NoopBackend, NoopConfigStore, NoopGeo, NoopHttpClient, NoopSecretStore, StubHttpClient, - noop_services, + NoopBackend, NoopConfigStore, NoopGeo, NoopHttpClient, NoopSecretStore, StubBackend, + StubHttpClient, noop_services, }; use crate::platform::{ClientInfo, PlatformHttpClient, PlatformHttpRequest, PlatformResponse}; use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + use crate::trace::TracePreDispatchHook; use base64::Engine as _; use base64::engine::general_purpose::STANDARD as BASE64; + use edgezero_core::router::PreDispatchHook as _; use serde_json::json; use std::sync::{Arc, Mutex}; @@ -1012,6 +1081,339 @@ mod tests { ); } + #[tokio::test] + async fn trace_slot_conversion_api_keeps_refs_out_of_actual_bidder_and_mediator_payloads() { + const SLOT_REF: &str = "ts-slot-00000000-0000-4000-8000-000000000001"; + let settings_toml = format!( + "{}\n[auction]\nenabled = true\n\n[auction.providers.bidder]\nprotocol = \"openrtb-2.6\"\nprofile = \"standard\"\nendpoint = \"https://bidder.example.com/auction\"\nrouting = \"all_eligible\"\n", + crate_test_settings_str() + ); + let mut settings = + Settings::from_toml(&settings_toml).expect("should parse live provider settings"); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable trace"); + let settings = Arc::new(settings); + let plan = Arc::new( + crate::auction::compile_auction_plan(&settings) + .expect("should compile production provider plan"), + ); + let mediator = AdServerMockProvider::new(AdServerMockConfig { + enabled: true, + endpoint: "https://mediator.example.com/auction".to_string(), + timeout_ms: 500, + ..Default::default() + }); + let orchestrator = AuctionOrchestrator::from_plan(plan, Some(Arc::new(mediator))); + let http = Arc::new(StubHttpClient::new()); + http.push_response(204, Vec::new()); + http.push_response(204, Vec::new()); + let services = RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(Arc::clone(&http) as Arc<_>) + .geo(Arc::new(NoopGeo)) + .client_info(ClientInfo::default()) + .build(); + let body = json!({"adUnits":[{"code":"slot","mediaTypes":{"banner":{"sizes":[[300,250]]}},"ext":{"trusted_server":{"trace_slot_ref":SLOT_REF}}}]}); + let mut request = Request::builder() + .method("POST") + .uri("https://publisher.example.com/auction") + .header(header::COOKIE, "__Host-ts-console=1") + .body(EdgeBody::from( + serde_json::to_vec(&body).expect("should encode source token"), + )) + .expect("should build request"); + TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load API setup metadata")), + ) + .handle(&mut request) + .await + .expect("should freeze trace gate"); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, None); + let response = handle_auction( + &settings, + &orchestrator, + None, + None, + &mut ec_context, + &services, + request, + ) + .await + .expect("should execute bidder and mediator"); + let bytes = response + .into_body() + .into_bytes() + .expect("should collect API evidence"); + let value: JsonValue = serde_json::from_slice(&bytes).expect("should decode response"); + let evidence = &value["ext"]["trusted_server"]["trace_auction"]["evidence"]; + let auction_token = evidence["diagnostic_auction_id"] + .as_str() + .expect("should deliver auction token"); + assert_eq!( + evidence["slots"][0]["slot_ref"], SLOT_REF, + "should echo exact accepted ref" + ); + assert_eq!( + evidence["terminal_status"], "completed", + "should preserve completed zero bids" + ); + assert_eq!( + evidence["slots"][0]["candidate"], "no_candidate", + "should observe no delivered candidate" + ); + let bodies = http.recorded_request_bodies(); + assert_eq!( + bodies.len(), + 2, + "should actually serialize bidder and mediator requests" + ); + for bytes in bodies { + let body = String::from_utf8(bytes).expect("should serialize UTF-8 outbound JSON"); + for private in [ + SLOT_REF, + auction_token, + "trace_slot_ref", + "trace_auction", + "diagnostic_auction_id", + ] { + assert!( + !body.contains(private), + "should keep trace association out of outbound JSON: {private}" + ); + } + } + } + + #[tokio::test] + async fn trace_slot_conversion_api_echoes_only_unique_accepted_refs_when_frozen_active() { + const FIRST: &str = "ts-slot-00000000-0000-4000-8000-000000000001"; + const SECOND: &str = "ts-slot-00000000-0000-4000-8000-000000000002"; + for active in [false, true] { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable trace feature"); + let settings = Arc::new(settings); + let orchestrator = AuctionOrchestrator::new(AuctionConfig { + enabled: false, + ..Default::default() + }); + let services = services_with_telemetry(Arc::new(RecordingTelemetrySink::default())); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, None); + let body = json!({"adUnits":[ + {"code":"filtered","mediaTypes":{"video":{}},"ext":{"trusted_server":{"trace_slot_ref":SECOND}}}, + {"code":"duplicate","mediaTypes":{"banner":{"sizes":[[300,250]]}},"ext":{"trusted_server":{"trace_slot_ref":FIRST}}}, + {"code":"duplicate","mediaTypes":{"banner":{"sizes":[[300,250]]}},"ext":{"trusted_server":{"trace_slot_ref":FIRST}}}, + {"code":"unique","mediaTypes":{"banner":{"sizes":[[300,250]]}},"ext":{"trusted_server":{"trace_slot_ref":SECOND}}} + ]}); + let mut request = Request::builder() + .method("POST") + .uri("https://publisher.example.com/auction") + .body(EdgeBody::from( + serde_json::to_vec(&body).expect("should serialize request"), + )) + .expect("should build request"); + if active { + request.headers_mut().insert( + header::COOKIE, + "__Host-ts-console=1".parse().expect("should build cookie"), + ); + } + TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load API setup metadata")), + ) + .handle(&mut request) + .await + .expect("should freeze cookie gate"); + let response = handle_auction( + &settings, + &orchestrator, + None, + None, + &mut ec_context, + &services, + request, + ) + .await + .expect("should preserve accepted ordinary ads request"); + assert_eq!( + response.status(), + StatusCode::OK, + "should preserve no-bid status" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should not refresh console cookie" + ); + assert_eq!( + response + .extensions() + .get::() + .is_some(), + active, + "should gate private evidence response" + ); + let bytes = response + .into_body() + .into_bytes() + .expect("should collect response"); + let value: JsonValue = serde_json::from_slice(&bytes).expect("should decode response"); + let transport = &value["ext"]["trusted_server"]["trace_auction"]; + if !active { + assert!(transport.is_null(), "should omit inactive transport"); + continue; + } + let evidence = &transport["evidence"]; + assert_eq!( + evidence["source"], "auction_api", + "should observe API source" + ); + assert_eq!( + evidence["terminal_reason"], "policy_skipped", + "should retain skipped reason" + ); + let slots = evidence["slots"] + .as_array() + .expect("should publish accepted slots"); + assert_eq!(slots.len(), 3, "should exclude filtered occurrence"); + assert_ne!( + slots[0]["slot_ref"], FIRST, + "should replace repeated accepted token" + ); + assert_ne!( + slots[1]["slot_ref"], FIRST, + "should replace every repeated token occurrence" + ); + assert_ne!( + slots[0]["slot_ref"], slots[1]["slot_ref"], + "should preserve distinct occurrence refs" + ); + assert_eq!( + slots[2]["slot_ref"], SECOND, + "should ignore filtered token repetition" + ); + for (index, slot) in slots.iter().enumerate() { + assert_eq!( + slot["slot_number"], + index + 1, + "should number accepted occurrences" + ); + } + } + } + + #[tokio::test] + async fn trace_auction_api_frozen_disabled_capture_delivers_private_evidence() { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable trace capture"); + let settings = Arc::new(settings); + + let config = AuctionConfig { + enabled: false, + providers: AuctionConfig::legacy_provider_map(&["panic_provider"]), + timeout_ms: 2000, + mediator: None, + ..Default::default() + }; + let mut orchestrator = AuctionOrchestrator::new(config); + orchestrator.register_provider(Arc::new(PanicOnBidProvider)); + let telemetry_sink = Arc::new(RecordingTelemetrySink::default()); + let services = services_with_telemetry(Arc::clone(&telemetry_sink)); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, None); + let body = json!({ + "adUnits": [{ + "code": "div-gpt-ad-1", + "mediaTypes": { "banner": { "sizes": [[300, 250]] } } + }] + }); + let mut request = Request::builder() + .method("POST") + .header("cookie", "__Host-ts-console=1") + .uri("https://test-publisher.example/auction") + .body(EdgeBody::from( + serde_json::to_vec(&body).expect("should serialize disabled-auction body"), + )) + .expect("should build disabled-auction request"); + + let hook = TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load setup metadata on API capture")), + ); + assert!( + hook.handle(&mut request) + .await + .expect("should freeze API cookies") + .is_none(), + "should retain ordinary API routing" + ); + + let response = handle_auction( + &settings, + &orchestrator, + None, + None, + &mut ec_context, + &services, + request, + ) + .await + .expect("disabled auction should return a no-bid response"); + + assert_eq!( + response.status(), + StatusCode::OK, + "disabled auction should return a 200 no-bid response" + ); + let carry = response + .extensions() + .get::() + .expect("should retain private API terminal facts"); + let facts = serde_json::to_value( + carry + .transport() + .expect("should finish the skipped API carry"), + ) + .expect("should serialize public-safe facts"); + assert_eq!( + facts["evidence"]["source"], "auction_api", + "should preserve the real API source" + ); + assert_eq!( + facts["evidence"]["terminal_reason"], "policy_skipped", + "should observe the explicit disabled policy" + ); + let bytes = response + .into_body() + .into_bytes() + .expect("should preserve the ordinary OpenRTB response"); + let response: JsonValue = + serde_json::from_slice(&bytes).expect("should decode response evidence"); + assert_eq!( + response["ext"]["trusted_server"]["trace_auction"], facts, + "should deliver the same checked terminal facts" + ); + } + #[tokio::test] async fn all_planned_launch_failures_return_bad_gateway_and_execution_failed_telemetry() { let settings_toml = format!( diff --git a/crates/trusted-server-core/src/auction/formats.rs b/crates/trusted-server-core/src/auction/formats.rs index a0df63d94..cc918cb09 100644 --- a/crates/trusted-server-core/src/auction/formats.rs +++ b/crates/trusted-server-core/src/auction/formats.rs @@ -22,10 +22,12 @@ use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::openrtb::{ BidExt, BidTrustedServerExt, OpenRtbBid, OpenRtbResponse, ResponseExt, SeatBid, ToExt, - to_openrtb_i32, + TraceResponseExt, to_openrtb_i32, }; use crate::platform::RuntimeServices; +use crate::response_privacy::enforce_terminal_private_cache_privacy; use crate::settings::Settings; +use crate::trace::{TraceAuctionCarry, TraceAuctionTransportV1, TraceClientSlotRefs}; use super::orchestrator::OrchestrationResult; use super::types::{ @@ -177,7 +179,11 @@ pub fn convert_tsjs_to_auction_request( // Convert ad units to slots let mut slots = Vec::new(); - for unit in &body.ad_units { + let trace_refs = req.extensions().get::(); + if let Some(refs) = trace_refs { + refs.begin_conversion(); + } + for (input_index, unit) in body.ad_units.iter().enumerate() { if let Some(media_types) = &unit.media_types && let Some(banner) = &media_types.banner { @@ -212,6 +218,9 @@ pub fn convert_tsjs_to_auction_request( targeting: HashMap::new(), bidders, }); + if let Some(refs) = trace_refs { + refs.record_accepted(input_index); + } } } @@ -348,6 +357,16 @@ pub(crate) fn convert_to_openrtb_response_with_report( settings: &Settings, auction_request: &AuctionRequest, ec_allowed: bool, +) -> Result> { + convert_to_openrtb_response_with_trace(result, settings, auction_request, ec_allowed, None) +} + +pub(crate) fn convert_to_openrtb_response_with_trace( + result: &OrchestrationResult, + settings: &Settings, + auction_request: &AuctionRequest, + ec_allowed: bool, + trace: Option<&TraceAuctionCarry>, ) -> Result> { let mut seatbids = Vec::with_capacity(result.winning_bids.len()); let rewrite_creatives = settings.auction.rewrite_creatives; @@ -495,10 +514,20 @@ pub(crate) fn convert_to_openrtb_response_with_report( .map(ProviderSummary::from) .collect(); + let trusted_server = trace.map(|carry| { + carry.observe_delivery(&result.winning_bids, &delivery.delivered_winner_slots); + TraceResponseExt { + trace_auction: carry + .transport() + .unwrap_or_else(TraceAuctionTransportV1::unavailable), + } + }); + let response_body = OpenRtbResponse { id: Some(auction_request.id.to_string()), seatbid: seatbids, ext: ResponseExt { + trusted_server, orchestrator: OrchestratorExt { strategy: strategy_name.to_string(), providers: result.provider_responses.len(), @@ -548,6 +577,10 @@ pub(crate) fn convert_to_openrtb_response_with_report( } } + if trace.is_some() { + enforce_terminal_private_cache_privacy(&mut response); + } + Ok(OpenRtbResponseConversion { response, delivery }) } @@ -1919,6 +1952,166 @@ mod tests { assert!(bid.get("w").is_none(), "should omit out-of-range width"); assert!(bid.get("h").is_none(), "should omit out-of-range height"); } + mod trace_slot_conversion_api_transport_tests { + use super::*; + use crate::response_privacy::TerminalPrivateResponse; + use crate::trace::{TraceAuctionSource, TraceAuctionTerminalStatus, TraceProviderRole}; + + fn observed(request: &AuctionRequest, result: &OrchestrationResult) -> TraceAuctionCarry { + let carry = TraceAuctionCarry::capture_if_enabled( + true, + TraceAuctionSource::AuctionApi, + &request.slots, + ) + .expect("should capture the active API auction"); + for response in &result.provider_responses { + let call = carry.launch_provider(TraceProviderRole::Bidder); + carry.observe_response(call, response); + } + carry.finish(TraceAuctionTerminalStatus::Completed, None); + carry + } + + #[test] + fn trace_slot_conversion_api_uses_actual_delivery_dispositions_and_preserves_bids() { + for deliverable in [false, true] { + let settings = make_settings(); + let auction = make_auction_request(); + let mut bid = make_bid("div-gpt-top", "private-bidder-sentinel", Some(1.0)); + if !deliverable { + bid.creative = None; + } + let result = make_result(bid); + let carry = observed(&auction, &result); + let token = carry.token(); + let baseline = response_json( + convert_to_openrtb_response(&result, &settings, &auction, false) + .expect("should produce ordinary baseline bids"), + ); + + let conversion = convert_to_openrtb_response_with_trace( + &result, + &settings, + &auction, + false, + Some(&carry), + ) + .expect("should preserve ordinary response conversion"); + + assert_eq!( + conversion + .response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|header| header.to_str().ok()), + Some("no-store, private"), + "should make evidence-bearing API responses terminal-private" + ); + assert!( + conversion + .response + .extensions() + .get::() + .is_some(), + "should retain ordinary ads/identity finalization with terminal privacy" + ); + let actual = response_json(conversion.response); + assert_eq!( + actual["seatbid"], baseline["seatbid"], + "should preserve actual bids and seats" + ); + assert_eq!( + actual["ext"]["orchestrator"], baseline["ext"]["orchestrator"], + "should preserve existing orchestrator extension" + ); + let transport = &actual["ext"]["trusted_server"]["trace_auction"]; + assert_eq!( + transport["evidence"]["diagnostic_auction_id"], + token.as_str(), + "should publish the original pre-dispatch token" + ); + assert_eq!( + transport["evidence"]["slots"][0]["candidate"], + if deliverable { + "selected" + } else { + "selected_unrenderable" + }, + "should use the definitive conversion delivery set" + ); + assert!( + !transport.to_string().contains("private-bidder-sentinel"), + "should exclude ordinary bidder identities from evidence" + ); + } + } + + #[test] + fn trace_slot_conversion_api_projection_failure_preserves_normal_bids() { + let settings = make_settings(); + let mut auction = make_auction_request(); + // The ordinary auction may retain its full work; only public trace + // omission-counter overflow fails the diagnostic projection. + auction.slots[0].formats = vec![ + AdFormat { + width: 300, + height: 250, + media_type: MediaType::Banner + }; + usize::from(u16::MAX) + 17 + ]; + let result = make_result(make_bid("div-gpt-top", "fictional-bidder", Some(1.0))); + let carry = observed(&auction, &result); + let baseline = response_json( + convert_to_openrtb_response(&result, &settings, &auction, false) + .expect("should preserve ordinary bids without trace"), + ); + + let conversion = convert_to_openrtb_response_with_trace( + &result, + &settings, + &auction, + false, + Some(&carry), + ) + .expect("should not fail bids on trace projection failure"); + + let actual = response_json(conversion.response); + assert_eq!( + actual["seatbid"], baseline["seatbid"], + "should preserve ordinary bids after bounded projection fails" + ); + assert_eq!( + actual["ext"]["trusted_server"]["trace_auction"], + json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed"}), + "should publish only the exact exclusive unavailable envelope" + ); + } + + #[test] + fn trace_slot_conversion_api_absent_gate_keeps_ordinary_response_and_headers() { + let settings = make_settings(); + let auction = make_auction_request(); + let result = make_empty_result(); + + let conversion = + convert_to_openrtb_response_with_trace(&result, &settings, &auction, false, None) + .expect("should preserve trace-off conversion"); + + assert!( + !conversion + .response + .headers() + .contains_key(header::CACHE_CONTROL), + "should not change ordinary cache policy when trace is absent" + ); + let actual = response_json(conversion.response); + assert!( + actual["ext"].get("trusted_server").is_none(), + "should allocate no public trace member when the base gate is absent" + ); + } + } } #[cfg(test)] @@ -2101,3 +2294,172 @@ mod convert_tests { ); } } + +#[cfg(test)] +mod trace_slot_conversion_tests { + use super::*; + use crate::platform::test_support::noop_services; + use crate::test_support::tests::create_test_settings; + use crate::trace::{TraceClientSlotRefs, TraceSlotRef}; + use serde_json::json; + + const FIRST: &str = "ts-slot-00000000-0000-4000-8000-000000000001"; + const SECOND: &str = "ts-slot-00000000-0000-4000-8000-000000000002"; + + fn convert(raw: &[u8]) -> (AuctionRequest, Vec>) { + let body: AdRequest = + serde_json::from_slice(raw).expect("should preserve ordinary request acceptance"); + let settings = create_test_settings(); + let services = noop_services(); + let mut request = Request::new(EdgeBody::empty()); + let baseline = convert_tsjs_to_auction_request( + &body, + &settings, + &services, + &request, + ConsentContext::default(), + None, + None, + ) + .expect("should accept the ordinary request"); + let refs = TraceClientSlotRefs::from_raw(raw); + request.extensions_mut().insert(refs.clone()); + + let converted = convert_tsjs_to_auction_request( + &body, + &settings, + &services, + &request, + ConsentContext::default(), + None, + None, + ) + .expect("should not reject optional trace input"); + + assert_eq!( + serde_json::to_value(&converted.slots).expect("should serialize converted slots"), + serde_json::to_value(&baseline.slots).expect("should serialize baseline slots"), + "should keep ordinary slot acceptance, ordering, formats and bidder params" + ); + let accepted = refs + .accepted_refs() + .expect("should retain request-local occurrence observations"); + assert_eq!( + accepted.len(), + converted.slots.len(), + "should bind in the exact accepted slots.push branch" + ); + (converted, accepted) + } + + #[test] + fn trace_slot_conversion_follows_accepted_occurrences_after_filtering_and_grouping() { + let raw = serde_json::to_vec(&json!({"adUnits":[ + {"code":"filtered","mediaTypes":{"video":{}},"ext":{"trusted_server":{"trace_slot_ref":SECOND}}}, + {"code":"duplicate-code","mediaTypes":{"banner":{"sizes":[[300,250]]}},"bids":[{"bidder":"fictional-a","params":{"zone":1}},{"bidder":"fictional-b","params":{"zone":2}}],"ext":{"trusted_server":{"trace_slot_ref":FIRST}}}, + {"code":"filtered-again","ext":{"trusted_server":{"trace_slot_ref":FIRST}}}, + {"code":"duplicate-code","mediaTypes":{"banner":{"sizes":[]}},"ext":{"trusted_server":{"trace_slot_ref":SECOND}}} + ]})).expect("should serialize final grouped units"); + + let (auction, refs) = convert(&raw); + + assert_eq!( + auction.slots.len(), + 2, + "should keep only accepted banner occurrences" + ); + assert_eq!( + auction.slots[0].bidders.len(), + 2, + "should preserve grouping of ordinary bidder params" + ); + assert_eq!( + refs, + vec![ + Some(TraceSlotRef::parse(FIRST).expect("should parse the first client ref")), + Some(TraceSlotRef::parse(SECOND).expect("should parse the second client ref")) + ], + "should use exact accepted source occurrences despite duplicate ordinary keys" + ); + } + + #[test] + fn trace_slot_conversion_malformed_optional_shapes_preserve_ordinary_acceptance() { + for ext in [ + json!(null), + json!(true), + json!(42), + json!("private-string"), + json!([]), + json!({}), + json!({"trusted_server":null}), + json!({"trusted_server":true}), + json!({"trusted_server":[]}), + json!({"trusted_server":"private-string"}), + json!({"trusted_server":{"trace_slot_ref":null}}), + json!({"trusted_server":{"trace_slot_ref":42}}), + json!({"trusted_server":{"trace_slot_ref":true}}), + json!({"trusted_server":{"trace_slot_ref":{}}}), + json!({"trusted_server":{"trace_slot_ref":[]}}), + json!({"trusted_server":{"trace_slot_ref":format!(" {FIRST}")}}), + ] { + let raw = serde_json::to_vec(&json!({"adUnits":[{"code":"example-slot","mediaTypes":{"banner":{"sizes":[[300,250]]}},"ext":ext}]})).expect("should serialize a malformed optional extension"); + let (_, refs) = convert(&raw); + assert_eq!( + refs, + vec![None], + "should decline only optional association for malformed input" + ); + } + } + + #[test] + fn trace_slot_conversion_oversized_optional_number_declines_only_one_occurrence() { + // This numeric JSON literal cannot be represented by json! or Value. + for ext in [ + r#"1e999"#, + r#"{"trusted_server":1e999}"#, + r#"{"trusted_server":{"trace_slot_ref":1e999}}"#, + ] { + let raw = format!( + r#"{{"adUnits":[{{"code":"first","mediaTypes":{{"banner":{{"sizes":[[300,250]]}}}},"ext":{{"trusted_server":{{"trace_slot_ref":"{FIRST}"}}}}}},{{"code":"invalid","mediaTypes":{{"banner":{{"sizes":[[300,250]]}}}},"ext":{ext}}},{{"code":"last","mediaTypes":{{"banner":{{"sizes":[[300,250]]}}}},"ext":{{"trusted_server":{{"trace_slot_ref":"{SECOND}"}}}}}}]}}"# + ); + let (_, refs) = convert(raw.as_bytes()); + assert_eq!( + refs, + vec![ + Some(TraceSlotRef::parse(FIRST).expect("should parse first ref")), + None, + Some(TraceSlotRef::parse(SECOND).expect("should parse last ref")) + ], + "should isolate malformed optional numeric association to its source occurrence" + ); + } + } + + #[test] + fn trace_slot_conversion_duplicate_members_decline_only_the_affected_occurrence() { + // Duplicate object members cannot be expressed by json! or Value. + for ext in [ + r#""ext":{"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"}},"ext":{"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"}}"#, + r#""ext":{"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"},"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"}}"#, + r#""ext":{"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001","trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"}}"#, + r#""ext":{"trusted_server":{"trace_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001","\u0074race_slot_ref":"ts-slot-00000000-0000-4000-8000-000000000001"}}"#, + ] { + let raw = format!( + r#"{{"adUnits":[{{"code":"example-slot","mediaTypes":{{"banner":{{"sizes":[[300,250]]}}}},{ext}}},{{"code":"other-slot","mediaTypes":{{"banner":{{"sizes":[[300,250]]}}}},"ext":{{"trusted_server":{{"trace_slot_ref":"{SECOND}"}}}}}}]}}"# + ); + let (_, refs) = convert(raw.as_bytes()); + assert_eq!( + refs, + vec![ + None, + Some( + TraceSlotRef::parse(SECOND).expect("should retain the unrelated valid ref") + ) + ], + "should detect decoded duplicate members without rejecting or poisoning other units" + ); + } + } +} diff --git a/crates/trusted-server-core/src/auction/orchestrator.rs b/crates/trusted-server-core/src/auction/orchestrator.rs index 204201e60..f29f448f3 100644 --- a/crates/trusted-server-core/src/auction/orchestrator.rs +++ b/crates/trusted-server-core/src/auction/orchestrator.rs @@ -23,6 +23,10 @@ use super::routing::route_auction; use super::telemetry::AbandonedProviderCall; use super::types::{AuctionContext, AuctionRequest, AuctionResponse, Bid, BidStatus}; use crate::request_signing::RequestSigner; +use crate::trace::{ + TraceAuctionCarry, TraceAuctionTerminalReason, TraceAuctionTerminalStatus, + TraceProviderObservation, TraceProviderRole, +}; /// In-flight auction requests dispatched to SSP backends. /// @@ -45,6 +49,7 @@ pub struct DispatchedAuction { planned_unused_bidder_params: HashMap, planned_unroutable_bidder_count: u32, planned_provider_order: HashMap, + trace: Option, } struct ProviderLaunchState { @@ -94,6 +99,12 @@ impl DispatchedAuction { Vec, u64, ) { + if let Some(trace) = &self.trace { + trace.finish( + TraceAuctionTerminalStatus::Abandoned, + Some(TraceAuctionTerminalReason::Unknown), + ); + } let elapsed_ms = self.auction_start.elapsed().as_millis() as u64; let abandoned = self .backend_to_provider @@ -118,6 +129,10 @@ impl DispatchedAuction { elapsed_ms, ) } + + pub(crate) fn trace_carry(&self) -> Option { + self.trace.clone() + } } #[cfg(test)] @@ -136,8 +151,14 @@ impl DispatchedAuction { planned_unused_bidder_params: HashMap::new(), planned_unroutable_bidder_count: 0, planned_provider_order: HashMap::new(), + trace: None, } } + + pub(crate) fn with_trace_for_test(mut self, trace: TraceAuctionCarry) -> Self { + self.trace = Some(trace); + self + } } const PROVIDER_ERROR_MESSAGE_CHARS: usize = 500; @@ -328,6 +349,7 @@ struct PlannedLaunchState { provider: Arc, started_at: Instant, parse_state: Option, + trace_observation: Option, } #[cfg(test)] @@ -498,6 +520,7 @@ impl AuctionOrchestratorHarness { provider, started_at, parse_state, + trace_observation: None, }); pending.push(launched); } @@ -852,7 +875,9 @@ impl AuctionOrchestrator { request: &AuctionRequest, context: &AuctionContext<'_>, ) -> Result> { - match self.dispatch_auction(request, context).await { + let trace = context.request.extensions().get::(); + let _trace_guard = trace.map(TraceAuctionCarry::cancellation_guard); + match self.dispatch_planned_auction(request, context).await { DispatchAuctionOutcome::Dispatched(dispatched) => Ok(self .collect_dispatched_auction(dispatched, context.services, context) .await), @@ -860,6 +885,9 @@ impl AuctionOrchestrator { fatal_admission_error, .. } => { + if let Some(trace) = trace { + trace.finish_dispatch_failed(false); + } if let Some(error) = fatal_admission_error { return Err(error.change_context(TrustedServerError::Auction { message: "Planned auction admission failed".to_string(), @@ -871,8 +899,14 @@ impl AuctionOrchestrator { } DispatchAuctionOutcome::NotStarted => { if self.planned_providers.is_empty() { + if let Some(trace) = trace { + trace.finish(TraceAuctionTerminalStatus::Completed, None); + } Ok(OrchestrationResult::no_bid()) } else { + if let Some(trace) = trace { + trace.finish_dispatch_failed(true); + } Err(Report::new(TrustedServerError::Auction { message: "No planned provider request was started".to_string(), })) @@ -893,6 +927,12 @@ impl AuctionOrchestrator { context: &AuctionContext<'_>, ) -> Result> { if !self.enabled { + if let Some(trace) = context.request.extensions().get::() { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } return Ok(OrchestrationResult::no_bid()); } #[cfg(not(test))] @@ -1589,6 +1629,11 @@ impl AuctionOrchestrator { context: &AuctionContext<'_>, ) -> DispatchAuctionOutcome { let plan = &self.plan; + let trace = context + .request + .extensions() + .get::() + .cloned(); if self.planned_providers.is_empty() { return DispatchAuctionOutcome::NotStarted; } @@ -1724,6 +1769,9 @@ impl AuctionOrchestrator { continue; } let started_at = Instant::now(); + let trace_observation = trace + .as_ref() + .map(|trace| trace.launch_provider(TraceProviderRole::Bidder)); match provider .request_bids_routed( input, @@ -1741,6 +1789,9 @@ impl AuctionOrchestrator { parse_state, }) => { let Some(backend_name) = pending.backend_name().map(str::to_string) else { + if let (Some(trace), Some(observation)) = (&trace, trace_observation) { + trace.observe_failure(observation); + } launch_failure_count += 1; completed_responses.push(provider_launch_failed_response( provider.provider_name(), @@ -1754,10 +1805,14 @@ impl AuctionOrchestrator { provider, started_at, parse_state, + trace_observation, }); pending_requests.push(pending.with_backend_name(backend_name)); } Entry::Occupied(_) => { + if let (Some(trace), Some(observation)) = (&trace, trace_observation) { + trace.observe_failure(observation); + } launch_failure_count += 1; completed_responses.push(provider_launch_failed_response( provider.provider_name(), @@ -1767,10 +1822,16 @@ impl AuctionOrchestrator { } } Ok(ProviderRequestOutcome::Immediate(response)) => { + if let (Some(trace), Some(observation)) = (&trace, trace_observation) { + trace.observe_response(observation, &response); + } immediate_response_count += 1; completed_responses.push(response); } Err(error) => { + if let (Some(trace), Some(observation)) = (&trace, trace_observation) { + trace.observe_failure(observation); + } log::warn!( "Planned provider '{}' failed to dispatch: {error:?}", provider.provider_name() @@ -1830,6 +1891,7 @@ impl AuctionOrchestrator { planned_unused_bidder_params, planned_unroutable_bidder_count, planned_provider_order, + trace, }) } @@ -1851,10 +1913,36 @@ impl AuctionOrchestrator { context: &AuctionContext<'_>, ) -> DispatchAuctionOutcome { if !self.enabled { + if let Some(trace) = context.request.extensions().get::() { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } return DispatchAuctionOutcome::NotStarted; } if !cfg!(test) || self.plan_backed { - return self.dispatch_planned_auction(request, context).await; + let trace = context.request.extensions().get::(); + let mut guard = trace.map(TraceAuctionCarry::cancellation_guard); + let outcome = self.dispatch_planned_auction(request, context).await; + match &outcome { + DispatchAuctionOutcome::Dispatched(_) => { + if let Some(guard) = &mut guard { + guard.disarm(); + } + } + DispatchAuctionOutcome::DispatchFailed { .. } => { + if let Some(trace) = trace { + trace.finish_dispatch_failed(false); + } + } + DispatchAuctionOutcome::NotStarted => { + if let Some(trace) = trace { + trace.finish_dispatch_failed(true); + } + } + } + return outcome; } #[cfg(test)] let provider_names = self @@ -2079,6 +2167,11 @@ impl AuctionOrchestrator { planned_unused_bidder_params: HashMap::new(), planned_unroutable_bidder_count: 0, planned_provider_order: HashMap::new(), + trace: context + .request + .extensions() + .get::() + .cloned(), }) } @@ -2111,7 +2204,9 @@ impl AuctionOrchestrator { planned_unused_bidder_params, planned_unroutable_bidder_count, planned_provider_order, + trace, } = dispatched; + let _trace_guard = trace.as_ref().map(TraceAuctionCarry::cancellation_guard); log::info!( "Collecting {} in-flight SSP responses (timeout: {}ms remaining: {}ms)", @@ -2134,6 +2229,9 @@ impl AuctionOrchestrator { }) { Ok(r) => r, Err(e) => { + if let Some(trace) = &trace { + trace.observe_collection_failure(); + } log::warn!("select() failed during auction collection: {:?}", e); // An outer select failure means the platform could not poll // any outstanding handle. Attribute every tracked launch as @@ -2150,6 +2248,11 @@ impl AuctionOrchestrator { ) }) .chain(planned_backend_to_provider.drain().map(|(_, state)| { + if let (Some(trace), Some(observation)) = + (&trace, state.trace_observation) + { + trace.observe_failure(observation); + } let response_time_ms = state.started_at.elapsed().as_millis() as u64; provider_transport_failed_response( state.provider.provider_name(), @@ -2213,6 +2316,11 @@ impl AuctionOrchestrator { } else if let Some(state) = planned_backend_to_provider.remove(&backend_name) { let response_time_ms = state.started_at.elapsed().as_millis() as u64; if deadline_policy.rejects_late_completion(auction_start, timeout_ms) { + if let (Some(trace), Some(observation)) = + (&trace, state.trace_observation) + { + trace.observe_failure(observation); + } responses.push(provider_timeout_response( state.provider.provider_name(), response_time_ms, @@ -2228,13 +2336,27 @@ impl AuctionOrchestrator { ) .await { - Ok(response) => responses.push(response), - Err(error) => responses.push(provider_error_response( - state.provider.provider_name(), - response_time_ms, - ERROR_TYPE_PARSE_RESPONSE, - &error, - )), + Ok(response) => { + if let (Some(trace), Some(observation)) = + (&trace, state.trace_observation) + { + trace.observe_response(observation, &response); + } + responses.push(response); + } + Err(error) => { + if let (Some(trace), Some(observation)) = + (&trace, state.trace_observation) + { + trace.observe_failure(observation); + } + responses.push(provider_error_response( + state.provider.provider_name(), + response_time_ms, + ERROR_TYPE_PARSE_RESPONSE, + &error, + )); + } } } else { log::warn!( @@ -2261,6 +2383,11 @@ impl AuctionOrchestrator { )); } else if let Some(state) = planned_backend_to_provider.remove(backend_name) { + if let (Some(trace), Some(observation)) = + (&trace, state.trace_observation) + { + trace.observe_failure(observation); + } let response_time_ms = state.started_at.elapsed().as_millis() as u64; log::warn!( "Planned provider '{}' request failed: {:?}", @@ -2309,6 +2436,9 @@ impl AuctionOrchestrator { } backend_to_provider.clear(); for state in planned_backend_to_provider.into_values() { + if let (Some(trace), Some(observation)) = (&trace, state.trace_observation) { + trace.observe_failure(observation); + } responses.push(provider_timeout_response( state.provider.provider_name(), state.started_at.elapsed().as_millis() as u64, @@ -2349,6 +2479,9 @@ impl AuctionOrchestrator { let remaining = remaining_budget_ms(auction_start, timeout_ms); let logical_budget_ms = remaining.min(mediator.timeout_ms()); if logical_budget_ms == 0 { + if let Some(trace) = &trace { + trace.finish(TraceAuctionTerminalStatus::Completed, None); + } log::warn!( "A_deadline exhausted before mediator '{}' — returning {} SSP bids without mediation", mediator.provider_name(), @@ -2367,6 +2500,9 @@ impl AuctionOrchestrator { .backend() .canonicalize_transport_timeout_ms(logical_budget_ms, mediator.timeout_ms()); if transport_timeout_ms == 0 { + if let Some(trace) = &trace { + trace.finish(TraceAuctionTerminalStatus::Completed, None); + } log::warn!( "Mediator '{}' transport budget canonicalized to zero — returning {} SSP bids without mediation", mediator.provider_name(), @@ -2409,11 +2545,20 @@ impl AuctionOrchestrator { provider_responses: Some(&responses), services: context.services, }; + let mut trace_observation = trace + .as_ref() + .map(|trace| trace.launch_provider(TraceProviderRole::Mediator)); let mediator_response = match mediator .request_bids(&request, &mediator_context) .await { - Ok(ProviderRequestOutcome::Immediate(response)) => Some(response), + Ok(ProviderRequestOutcome::Immediate(response)) => { + if let (Some(trace), Some(observation)) = (&trace, trace_observation.take()) + { + trace.observe_response(observation, &response); + } + Some(response) + } Ok(ProviderRequestOutcome::Pending { request: pending, parse_state, @@ -2445,7 +2590,14 @@ impl AuctionOrchestrator { ) .await { - Ok(response) => Some(response), + Ok(response) => { + if let (Some(trace), Some(observation)) = + (&trace, trace_observation.take()) + { + trace.observe_response(observation, &response); + } + Some(response) + } Err(error) => { log::warn!( "Mediator '{}' parse failed: {:?}", @@ -2472,6 +2624,10 @@ impl AuctionOrchestrator { } }; + if let (Some(trace), Some(observation)) = (&trace, trace_observation.take()) { + trace.observe_failure(observation); + } + if let Some(mediator_response) = mediator_response { let winning = mediator_response .bids @@ -2499,6 +2655,9 @@ impl AuctionOrchestrator { (None, self.select_winning_bids(&responses, &floor_prices)) }; + if let Some(trace) = &trace { + trace.finish(TraceAuctionTerminalStatus::Completed, None); + } OrchestrationResult { provider_responses: responses, mediator_response, @@ -2740,6 +2899,434 @@ mod tests { request } + mod trace_auction_live_tests { + use super::*; + use crate::trace::{TraceAuctionCarry, TraceAuctionSource}; + use edgezero_core::body::Body as EdgeBody; + use http::Request; + + fn captured_request(request: &AuctionRequest) -> (Request, TraceAuctionCarry) { + let carry = TraceAuctionCarry::capture_if_enabled( + true, + TraceAuctionSource::AuctionApi, + &request.slots, + ) + .expect("should capture the enabled request"); + let mut inbound = Request::new(EdgeBody::empty()); + inbound.extensions_mut().insert(carry.clone()); + (inbound, carry) + } + + fn evidence(carry: &TraceAuctionCarry) -> serde_json::Value { + serde_json::to_value( + carry + .transport() + .expect("should terminalize the live auction"), + ) + .expect("should serialize redacted facts")["evidence"] + .clone() + } + + #[tokio::test] + async fn trace_auction_live_empty_plan_distinguishes_sync_success_from_split_failure() { + let plan = Arc::new( + AuctionPlan::compile(planned_config(&[], false)) + .expect("should compile an empty plan"), + ); + let orchestrator = AuctionOrchestrator::from_plan(plan, None); + let http = Arc::new(StubHttpClient::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let settings = create_test_settings(); + let request = planned_request(); + for split in [false, true] { + let (inbound, carry) = captured_request(&request); + let token = carry.token(); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + if split { + assert!( + matches!( + orchestrator.dispatch_auction(&request, &context).await, + DispatchAuctionOutcome::NotStarted + ), + "should preserve the ordinary split outcome" + ); + } else { + let result = orchestrator + .run_auction(&request, &context) + .await + .expect("should preserve successful empty-plan execution"); + assert!( + result.winning_bids.is_empty(), + "should preserve an ordinary zero-bid result" + ); + } + let value = evidence(&carry); + assert_eq!( + value["diagnostic_auction_id"], + token.as_str(), + "should preserve the pre-dispatch token" + ); + assert_eq!( + value["terminal_status"], + if split { + "dispatch_failed" + } else { + "completed" + }, + "should map the observed path semantics" + ); + assert_eq!( + value["provider_calls"], + serde_json::json!([]), + "should not fabricate a launched provider" + ); + } + } + + #[tokio::test] + async fn trace_auction_live_launch_failure_survives_the_ordinary_error() { + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[("launch-fail", RoutingMode::AllEligible)], + false, + )) + .expect("should compile the launch-failure plan"), + ); + let orchestrator = AuctionOrchestrator::from_plan(plan, None); + let backend = Arc::new(NamingBackend::new(BackendNamingPolicy::Axum)); + backend.fail_ensure_for("launch-fail"); + let http = Arc::new(StubHttpClient::new()); + let services = build_services_with_backend_and_http_client( + Arc::clone(&backend) as Arc<_>, + Arc::clone(&http) as Arc<_>, + ); + let settings = create_test_settings(); + let request = planned_request(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + let _error = orchestrator + .run_auction(&request, &context) + .await + .expect_err("should preserve the ordinary launch failure"); + let value = evidence(&carry); + assert_eq!( + value["terminal_status"], "dispatch_failed", + "should preserve the actual dispatch failure" + ); + assert_eq!( + value["terminal_reason"], "provider_execution_failed", + "should use the directly observed launch failure" + ); + assert_eq!( + value["provider_calls"][0]["status"], "error", + "should retain the failed actual invocation" + ); + assert!( + !value.to_string().contains("launch-fail"), + "should exclude provider identities from public facts" + ); + assert!( + http.recorded_backend_names().is_empty(), + "should preserve ordinary network behavior" + ); + } + + #[tokio::test] + async fn trace_auction_live_collection_failure_keeps_launch_order_through_local_fallback() { + let http = Arc::new(OuterSelectErrorHttpClient::new()); + http.push_response(204, Vec::new()); + http.push_response(204, Vec::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[ + ("provider-b", RoutingMode::AllEligible), + ("provider-a", RoutingMode::AllEligible), + ], + false, + )) + .expect("should compile the ordered plan"), + ); + let orchestrator = AuctionOrchestrator::from_plan(plan, None); + let request = planned_request(); + let settings = create_test_settings(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + let result = orchestrator + .run_auction(&request, &context) + .await + .expect("should preserve ordinary local fallback"); + assert!( + result.winning_bids.is_empty(), + "should preserve ordinary ranking" + ); + let value = evidence(&carry); + assert_eq!( + value["terminal_status"], "execution_failed", + "should not infer successful collection from an ordinary Ok result" + ); + assert_eq!( + value["terminal_reason"], "collection_failed", + "should record the failure at its source" + ); + assert_eq!( + value["provider_calls"][0]["provider_number"], 1, + "should preserve first launch order" + ); + assert_eq!( + value["provider_calls"][1]["provider_number"], 2, + "should preserve second launch order" + ); + assert_eq!( + value["provider_calls"][0]["status"], "error", + "should retain transport failure facts" + ); + } + + #[tokio::test] + async fn trace_auction_live_mediator_no_bid_is_an_actual_ordered_call() { + let http = Arc::new(StubHttpClient::new()); + http.push_response(204, Vec::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[("bidder", RoutingMode::AllEligible)], + false, + )) + .expect("should compile bidder plan"), + ); + let orchestrator = + AuctionOrchestrator::from_plan(plan, Some(Arc::new(ImmediateNoBidProvider))); + let request = planned_request(); + let settings = create_test_settings(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + orchestrator + .run_auction(&request, &context) + .await + .expect("should retain ordinary successful zero-bid execution"); + let value = evidence(&carry); + assert_eq!( + value["terminal_status"], "completed", + "should not label ordinary zero bids a failure" + ); + assert_eq!( + value["provider_calls"][0]["role"], "bidder", + "should retain the bidder launch" + ); + assert_eq!( + value["provider_calls"][1]["role"], "mediator", + "should retain the real mediator invocation" + ); + assert_eq!( + value["provider_calls"][1]["status"], "no_bid", + "should retain mediator zero-bid status" + ); + } + struct NeverFinishesMediator; + + #[async_trait::async_trait(?Send)] + impl AuctionProvider for NeverFinishesMediator { + fn provider_name(&self) -> &str { + "pending-mediator" + } + async fn request_bids( + &self, + _request: &AuctionRequest, + _context: &AuctionContext<'_>, + ) -> Result> { + std::future::pending().await + } + async fn parse_response( + &self, + _response: PlatformResponse, + _response_time_ms: u64, + ) -> Result> { + panic!("should never parse an uncompleted mediator invocation"); + } + fn timeout_ms(&self) -> u32 { + 777 + } + } + + #[tokio::test] + async fn trace_auction_live_cancellation_preserves_pending_mediator_after_bidder_collection() + { + let http = Arc::new(StubHttpClient::new()); + http.push_response(204, Vec::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[("bidder", RoutingMode::AllEligible)], + false, + )) + .expect("should compile the bidder plan"), + ); + let orchestrator = + AuctionOrchestrator::from_plan(plan, Some(Arc::new(NeverFinishesMediator))); + let request = planned_request(); + let settings = create_test_settings(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + let mut pending = Box::pin(orchestrator.run_auction(&request, &context)); + let mut task = std::task::Context::from_waker(std::task::Waker::noop()); + assert!( + std::future::Future::poll(pending.as_mut(), &mut task).is_pending(), + "should stop at the actual mediator await" + ); + assert!( + carry.transport().is_none(), + "should not invent a terminal result while work is pending" + ); + + drop(pending); + + let facts = evidence(&carry); + assert_eq!( + facts["terminal_status"], "abandoned", + "should finish synchronously when collection is cancelled" + ); + assert_eq!( + facts["provider_calls"][0]["status"], "no_bid", + "should retain the collected bidder result" + ); + assert_eq!( + facts["provider_calls"][1]["role"], "mediator", + "should retain the actual pending mediator launch" + ); + assert_eq!( + facts["provider_calls"][1]["status"], "abandoned", + "should close only still-pending observations" + ); + } + + #[tokio::test] + async fn trace_auction_live_mediator_failure_preserves_successful_local_fallback() { + let http = Arc::new(StubHttpClient::new()); + http.push_response(204, Vec::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[("bidder", RoutingMode::AllEligible)], + false, + )) + .expect("should compile the bidder plan"), + ); + let orchestrator = + AuctionOrchestrator::from_plan(plan, Some(Arc::new(LaunchFailingProvider))); + let request = planned_request(); + let settings = create_test_settings(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 777, + transport_timeout_ms: 777, + provider_responses: None, + services: &services, + }; + + let result = orchestrator + .run_auction(&request, &context) + .await + .expect("should preserve successful ordinary local fallback"); + + assert!( + result.mediator_response.is_none(), + "should preserve the ordinary discarded mediator response" + ); + let facts = evidence(&carry); + assert_eq!( + facts["terminal_status"], "completed", + "should not infer whole-auction failure from provider error" + ); + assert!( + facts.get("terminal_reason").is_none(), + "should omit a reason for completed execution" + ); + assert_eq!( + facts["provider_calls"][1]["status"], "error", + "should preserve the failed mediator invocation before fallback" + ); + } + + #[tokio::test] + async fn trace_auction_live_zero_budget_has_no_synthetic_provider_launches() { + let http = Arc::new(StubHttpClient::new()); + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let plan = Arc::new( + AuctionPlan::compile(planned_config( + &[("bidder", RoutingMode::AllEligible)], + false, + )) + .expect("should compile the bidder plan"), + ); + let orchestrator = + AuctionOrchestrator::from_plan(plan, Some(Arc::new(NeverFinishesMediator))); + let request = planned_request(); + let settings = create_test_settings(); + let (inbound, carry) = captured_request(&request); + let context = AuctionContext { + settings: &settings, + request: &inbound, + timeout_ms: 0, + transport_timeout_ms: 0, + provider_responses: None, + services: &services, + }; + + orchestrator + .run_auction(&request, &context) + .await + .expect("should preserve the ordinary zero-budget result"); + + assert_eq!( + evidence(&carry)["provider_calls"], + serde_json::json!([]), + "should exclude synthetic timeout responses from launched work" + ); + assert!( + http.recorded_backend_names().is_empty(), + "should not add network work for trace" + ); + } + } + #[tokio::test] async fn disabled_from_plan_is_a_no_work_kill_switch_for_sync_and_split_paths() { let plan = Arc::new( diff --git a/crates/trusted-server-core/src/auth.rs b/crates/trusted-server-core/src/auth.rs index 9bbd40b69..e563fd616 100644 --- a/crates/trusted-server-core/src/auth.rs +++ b/crates/trusted-server-core/src/auth.rs @@ -11,6 +11,10 @@ use crate::settings::Settings; const BASIC_AUTH_REALM: &str = r#"Basic realm="Trusted Server""#; +/// Requests a fixed auth-failure category instead of logging a trace path. +#[derive(Clone, Copy, Debug)] +pub(crate) struct TraceAuthLogPolicy; + /// Marks the single `Authorization` value Trusted Server validated. /// /// The shared template cache may exempt this value from its normal authorization @@ -119,7 +123,11 @@ pub fn enforce_basic_auth( .insert(EdgeTerminatedAuthorization(authorization_digest)); Ok(None) } else { - log::warn!("Basic auth failed for path: {}", req.uri().path()); + if req.extensions().get::().is_some() { + log::warn!("trace_auth_failed"); + } else { + log::warn!("Basic auth failed for path: {}", req.uri().path()); + } Ok(Some(unauthorized_response())) } } diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 79f100e2f..3db5ea93a 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -22,7 +22,7 @@ use crate::integrations::{ didomi::DidomiIntegrationConfig, google_tag_manager::GoogleTagManagerConfig, gpt::GptConfig, - gpt_diagnostics::GptDiagnosticsConfig, + gpt_diagnostics::{GPT_DIAGNOSTICS_INTEGRATION_ID, GptDiagnosticsConfig}, js_asset_proxy::{JS_ASSET_PROXY_INTEGRATION_ID, JsAssetProxyConfig}, lockr::LockrConfig, nextjs::NextJsIntegrationConfig, @@ -248,6 +248,7 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report Result<(), Report> { settings.reject_placeholder_secrets()?; validate_js_asset_proxy_config(settings)?; + validate_gpt_diagnostics_config(settings)?; settings.validate_admin_handler_passwords()?; let plan = crate::auction::compile_auction_plan(settings)?; validate_enabled_integrations(settings, &plan, true)?; @@ -277,6 +279,28 @@ pub fn validate_settings_for_runtime( Ok(()) } +fn validate_gpt_diagnostics_config(settings: &Settings) -> Result<(), Report> { + let Some(raw_config) = settings.integrations.get(GPT_DIAGNOSTICS_INTEGRATION_ID) else { + return Ok(()); + }; + // Validate this dependency even when the normal integration lookup skips + // disabled integrations, including the default when enabled is omitted. + let config: GptDiagnosticsConfig = serde_json::from_value(raw_config.clone()).map_err(|error| { + Report::new(TrustedServerError::Configuration { + message: format!( + "integration startup failed for `{GPT_DIAGNOSTICS_INTEGRATION_ID}`: configuration could not be parsed: {error}" + ), + }) + })?; + config.validate().map_err(|error| { + Report::new(TrustedServerError::Configuration { + message: format!( + "integration startup failed for `{GPT_DIAGNOSTICS_INTEGRATION_ID}`: {error}" + ), + }) + }) +} + fn validate_js_asset_proxy_config(settings: &Settings) -> Result<(), Report> { let Some(raw_config) = settings.integrations.get(JS_ASSET_PROXY_INTEGRATION_ID) else { return Ok(()); @@ -1094,6 +1118,80 @@ gam_network_id = "99999" ); } + #[test] + fn trace_config_requires_enabled_on_both_validation_paths() { + for raw in [ + serde_json::json!({"enabled": false, "trace_page_enabled": true}), + serde_json::json!({"trace_page_enabled": true}), + ] { + let mut settings = valid_settings(); + settings + .integrations + .insert_config("gpt_diagnostics", &raw) + .expect("should insert trace configuration"); + + for validation in [validate_settings_for_deploy, validate_settings_for_runtime] { + let error = validation(&settings) + .expect_err("should reject trace without enabled diagnostics"); + assert!( + format!("{error:?}").contains("trace_page_enabled requires enabled = true"), + "should identify the trace configuration dependency: {error:?}" + ); + } + } + } + + #[test] + fn trace_config_accepts_valid_flags_and_rejects_disabled_unknown_fields() { + for raw in [ + serde_json::json!({}), + serde_json::json!({"enabled": true, "trace_page_enabled": true}), + serde_json::json!({"enabled": true, "trace_page_enabled": false}), + serde_json::json!({"enabled": false, "trace_page_enabled": false}), + serde_json::json!({"trace_page_enabled": false}), + serde_json::json!({"enabled": true}), + ] { + let mut settings = valid_settings(); + settings + .integrations + .insert_config("gpt_diagnostics", &raw) + .expect("should insert valid trace configuration"); + validate_settings_for_deploy(&settings) + .expect("should accept valid trace configuration at deployment"); + validate_settings_for_runtime(&settings) + .expect("should accept valid trace configuration at runtime"); + } + + for (raw, expected) in [ + ( + serde_json::json!({"enabled": false, "trace_page_enabeld": false}), + "unknown field", + ), + ( + serde_json::json!({"trace_page_enabeld": false}), + "unknown field", + ), + ( + serde_json::json!({"enabled": false, "trace_page_enabled": "false"}), + "boolean", + ), + ] { + let mut settings = valid_settings(); + settings + .integrations + .insert_config("gpt_diagnostics", &raw) + .expect("should insert invalid trace configuration"); + for validation in [validate_settings_for_deploy, validate_settings_for_runtime] { + let error = validation(&settings) + .expect_err("should reject invalid disabled trace configuration"); + assert!( + format!("{error:?}").contains(expected), + "should identify the configuration schema error" + ); + } + } + } + #[test] fn runtime_validation_rejects_placeholders() { let settings = Settings::from_toml( diff --git a/crates/trusted-server-core/src/html_processor.rs b/crates/trusted-server-core/src/html_processor.rs index 3889d7d59..4c148a970 100644 --- a/crates/trusted-server-core/src/html_processor.rs +++ b/crates/trusted-server-core/src/html_processor.rs @@ -91,6 +91,10 @@ pub struct HtmlProcessorConfig { pub max_buffered_body_bytes: usize, /// Request-scoped conditional diagnostics delivery decision. pub gpt_diagnostics: Option, + /// Request-only trace bootstrap, excluded from shared templates and ESI. + /// + /// The publisher supplies script-safe bytes only for an eligible document. + pub trace_bootstrap: Option, /// What the `` seam injects. Decided by the caller rather than inferred /// from [`Self::ad_slots_script`]. pub body_close: BodyCloseInjection, @@ -122,6 +126,7 @@ impl HtmlProcessorConfig { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: settings.publisher.max_buffered_body_bytes, gpt_diagnostics: None, + trace_bootstrap: None, body_close: BodyCloseInjection::None, suppress_datadome_client_side_tag: false, csp_nonce_observed: None, @@ -164,6 +169,13 @@ impl HtmlProcessorConfig { self } + /// Attach the already gated, request-only trace bootstrap. + #[must_use] + pub(crate) fn with_trace_bootstrap(mut self, bootstrap: Option) -> Self { + self.trace_bootstrap = bootstrap; + self + } + /// Watch the document for a response-bound CSP nonce delivered in its own markup. /// /// Pass `Some` only when the completed transform may be stored as a shared template; @@ -271,6 +283,7 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso let body_close = config.body_close.clone(); let ad_bids_state = config.ad_bids_state.clone(); let gpt_diagnostics = config.gpt_diagnostics.clone(); + let trace_bootstrap = config.trace_bootstrap.clone(); // No source-comment neutralization here: rewriting a publisher comment that happens // to match the reserved marker would change publisher content bytes. Collisions are @@ -301,6 +314,7 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso let document_state = document_state.clone(); let ad_slots_script = ad_slots_script.clone(); let gpt_diagnostics = gpt_diagnostics.clone(); + let trace_bootstrap = trace_bootstrap.clone(); move |el| { if !injected_tsjs.get() { let mut snippet = String::new(); @@ -325,6 +339,9 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso { snippet.push_str(&bootstrap); } + if let Some(bootstrap) = &trace_bootstrap { + snippet.push_str(bootstrap); + } // Main bundle: core + non-deferred integrations (synchronous). let immediate_ids = integrations.js_module_ids_immediate(); let script_attributes = integrations.tsjs_script_tag_attributes(); @@ -748,6 +765,7 @@ mod tests { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -1553,6 +1571,7 @@ mod tests { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1630,6 +1649,7 @@ mod tests { ad_bids_state: state, max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1669,6 +1689,7 @@ mod tests { ad_bids_state: state, max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1707,6 +1728,7 @@ mod tests { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1763,6 +1785,7 @@ mod tests { ad_bids_state: state, max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1793,6 +1816,7 @@ mod tests { ad_bids_state: state, max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut processor = create_html_processor(config); @@ -1818,6 +1842,7 @@ mod tests { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -2079,6 +2104,7 @@ mod tests { ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let source = diff --git a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js index afae77535..ef4893a41 100644 --- a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js +++ b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js @@ -326,7 +326,11 @@ // and deliberately identical to the bundle scheduler — the impression is // spent on a viewed tab, and the post-hydration guarantee holds whenever // the request is actually issued. - ts.scheduleInitialAdInit = function (initialBids, initialSlots) { + ts.scheduleInitialAdInit = function ( + initialBids, + initialSlots, + traceAuctionTransport, + ) { // The bundle may replace this scheduler after the fallback claims the initial // pass. Keep the latch on the shared document API so replacement cannot reset it. if ((ts.navGeneration || 0) !== 0 || ts.initialAdInitScheduled) return; @@ -338,6 +342,17 @@ if (initialBids !== undefined) ts.bids = initialBids; var fire = function () { if ((ts.navGeneration || 0) !== 0) return; + try { + if (window.__tsjs_trace_active === true && ts.traceGpt) { + ts.traceGpt.observeTransport( + initialSlots === undefined ? ts.adSlots : initialSlots, + traceAuctionTransport, + "initial_navigation_ssat", + ); + } + } catch (_) { + // Trace failure cannot interrupt the initial advertising pass. + } if (typeof ts.adInit === "function") ts.adInit(); }; var afterFrames = function () { @@ -349,6 +364,49 @@ else window.addEventListener("load", afterFrames, { once: true }); }; + function recordTraceOpportunity(gptSlot, slot, bid, fallback) { + if (window.__tsjs_trace_active !== true) return; + try { + var bridge = ts.traceGpt; + if (!bridge || !bridge.identity(slot) || !ts.gptDiagnosticsRecorder) return; + var trace = bridge.opportunity(slot, bid.hb_auction_id); + var handoff = + ts.gptSlotHandoffs && ts.gptSlotHandoffs[gptSlot.getSlotElementId()]; + var requestedSizes = fallback ? slot.formats : handoff && handoff.formats; + var nonempty = function (value) { + return typeof value === "string" && value.length > 0; + }; + // Mirror the bundle's opportunity classification at the same concrete + // GPT binding. Token validation remains in the single core bridge. + var hasTargeting = [ + "hb_pb", + "hb_bidder", + "hb_adid", + "hb_cache_host", + "hb_cache_path", + ].some(function (key) { + return nonempty(bid[key]); + }); + var opportunity = !hasTargeting + ? "no_candidate" + : nonempty(bid.hb_adid) && + (nonempty(bid.adm) || + (nonempty(bid.hb_cache_host) && nonempty(bid.hb_cache_path))) + ? "renderable_candidate" + : "unrenderable_candidate"; + ts.gptDiagnosticsRecorder.recordTrustedServerOpportunity( + gptSlot, + slot.id, + opportunity, + trace.auctionId, + requestedSizes, + trace.identity, + ); + } catch (_) { + // Diagnostics never change targeting, display, or refresh. + } + } + function findSlotByElementId(pubads, elementId) { var slots = pubads.getSlots ? pubads.getSlots() : []; return ( @@ -721,6 +779,7 @@ ts.prevGptSlots = ts.prevGptSlots || []; ts.prevGptSlots.push(gptSlot); } + recordTraceOpportunity(gptSlot, slot, bid, true); if (!ts.servicesEnabled) { pubads.enableSingleRequest(); window.googletag.enableServices(); @@ -866,6 +925,7 @@ // by the bundle's render bridge (index.ts) once it loads. divToSlotId[actualDivId] = slot.id; var slotElementId = s.getSlotElementId(); + recordTraceOpportunity(s, slot, b, false); var targetingKeys = Object.keys(slot.targeting || {}); nextSlotTargetingKeys[actualDivId] = targetingKeys; if (slotElementId && slotElementId !== actualDivId) { diff --git a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs index 1447a8358..963d183c5 100644 --- a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs +++ b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs @@ -1,4 +1,4 @@ -//! Query-activated, browser-session GPT runtime diagnostics integration. +//! Query-activated GPT runtime diagnostics with a bounded session lifetime. //! //! Deployment configuration makes the standalone browser module available. //! Exact `ts_console` directives establish or clear a host-only session cookie; @@ -8,7 +8,8 @@ use error_stack::{Report, ResultExt}; use http::{HeaderValue, Method, Request, Response, Uri, header, uri::PathAndQuery}; use serde::Deserialize; -use validator::Validate; +use std::borrow::Cow; +use validator::{Validate, ValidationError}; use edgezero_core::body::Body as EdgeBody; @@ -24,20 +25,36 @@ use super::IntegrationRegistration; pub const GPT_DIAGNOSTICS_INTEGRATION_ID: &str = "gpt_diagnostics"; /// Reserved activation query parameter. pub const GPT_DIAGNOSTICS_QUERY: &str = "ts_console"; -/// Host-only browser-session activation cookie. +/// Host-only activation cookie with a 30-minute explicit-activation lifetime. pub const GPT_DIAGNOSTICS_COOKIE: &str = "__Host-ts-console"; -const SET_CONSOLE_COOKIE: &str = "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax"; +const SET_CONSOLE_COOKIE: &str = + "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800"; const CLEAR_CONSOLE_COOKIE: &str = "__Host-ts-console=; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=0"; /// Configuration for the GPT runtime diagnostics integration. #[derive(Debug, Clone, Deserialize, Validate)] #[serde(deny_unknown_fields)] +#[validate(schema(function = "validate_trace_dependency"))] pub struct GptDiagnosticsConfig { /// Whether the GPT diagnostics browser module is available. #[serde(default)] pub enabled: bool, + /// Whether the mobile trace journey is available under operator auth rules. + /// + /// Requires [`Self::enabled`]. Defaults to false. + #[serde(default)] + pub trace_page_enabled: bool, +} + +fn validate_trace_dependency(config: &GptDiagnosticsConfig) -> Result<(), ValidationError> { + if config.trace_page_enabled && !config.enabled { + let mut error = ValidationError::new("trace_requires_enabled"); + error.message = Some(Cow::Borrowed("trace_page_enabled requires enabled = true")); + return Err(error); + } + Ok(()) } impl IntegrationConfig for GptDiagnosticsConfig { @@ -52,12 +69,23 @@ pub enum GptDiagnosticsCookieAction { /// Do not mutate the activation cookie. #[default] None, - /// Establish a host-only browser-session activation cookie. + /// Establish or renew the host-only 30-minute activation cookie. SetSession, /// Clear the activation cookie. ClearSession, } +impl GptDiagnosticsCookieAction { + /// Return the shared host-only cookie policy for an explicit action. + pub(crate) fn set_cookie_header(self) -> Option { + match self { + Self::None => None, + Self::SetSession => Some(HeaderValue::from_static(SET_CONSOLE_COOKIE)), + Self::ClearSession => Some(HeaderValue::from_static(CLEAR_CONSOLE_COOKIE)), + } + } +} + /// Immutable request-scoped diagnostics decision. #[derive(Clone, Debug, Default, PartialEq, Eq)] pub struct GptDiagnosticsRequestDecision { @@ -320,16 +348,7 @@ pub fn finalize_response( decision: &GptDiagnosticsRequestDecision, response: &mut Response, ) { - let cookie = match decision.cookie_action { - GptDiagnosticsCookieAction::None => None, - GptDiagnosticsCookieAction::SetSession => { - Some(HeaderValue::from_static(SET_CONSOLE_COOKIE)) - } - GptDiagnosticsCookieAction::ClearSession => { - Some(HeaderValue::from_static(CLEAR_CONSOLE_COOKIE)) - } - }; - if let Some(cookie) = cookie { + if let Some(cookie) = decision.cookie_action.set_cookie_header() { response.headers_mut().append(header::SET_COOKIE, cookie); } @@ -675,6 +694,67 @@ mod tests { ); } + #[test] + fn explicit_activation_has_bounded_lifetime_even_when_trace_is_disabled() { + for trace_enabled in [false, true] { + let mut settings = settings(true); + if trace_enabled { + settings + .integrations + .insert_config( + GPT_DIAGNOSTICS_INTEGRATION_ID, + &json!({"enabled": true, "trace_page_enabled": true}), + ) + .expect("should insert diagnostics trace flag"); + } + + for query in ["1", "true", "1"] { + let mut request = navigation( + &format!("https://publisher.example/?ts_console={query}"), + Some("__Host-ts-console=1"), + ); + let decision = prepare_request(&settings, &mut request) + .expect("should prepare deliberate activation"); + let mut response = Response::new(EdgeBody::empty()); + finalize_response(&decision, &mut response); + assert_eq!( + response.headers()[header::SET_COOKIE], + "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800", + "should renew bounded lifetime only on explicit activation" + ); + } + + let mut request = navigation( + "https://publisher.example/article", + Some("__Host-ts-console=1"), + ); + let decision = prepare_request(&settings, &mut request) + .expect("should prepare an established session"); + let mut response = Response::new(EdgeBody::empty()); + finalize_response(&decision, &mut response); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should not extend lifetime on ordinary requests" + ); + + for query in ["0", "false"] { + let mut request = navigation( + &format!("https://publisher.example/?ts_console={query}"), + Some("__Host-ts-console=1"), + ); + let decision = prepare_request(&settings, &mut request) + .expect("should prepare deliberate deactivation"); + let mut response = Response::new(EdgeBody::empty()); + finalize_response(&decision, &mut response); + assert_eq!( + response.headers()[header::SET_COOKIE], + "__Host-ts-console=; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=0", + "should clear using the same host-only cookie scope" + ); + } + } + } + #[test] fn config_rejects_unknown_fields() { let mut settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 6b301bf93..bb1a875e8 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -639,6 +639,7 @@ mod tests { ad_bids_state: Arc::new(std::sync::Mutex::new(None)), max_buffered_body_bytes: 16 * 1024 * 1024, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }); let pipeline_config = PipelineConfig { diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 249f741ab..5028d0658 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -72,6 +72,7 @@ pub mod streaming_processor; pub mod streaming_replacer; pub mod test_support; pub mod tester_cookie; +pub mod trace; pub mod tsjs; #[cfg(test)] diff --git a/crates/trusted-server-core/src/openrtb.rs b/crates/trusted-server-core/src/openrtb.rs index 4aded7488..67cd7f695 100644 --- a/crates/trusted-server-core/src/openrtb.rs +++ b/crates/trusted-server-core/src/openrtb.rs @@ -186,10 +186,23 @@ pub struct BidTrustedServerExt<'a> { #[derive(Debug, Serialize)] pub struct ResponseExt { pub orchestrator: OrchestratorExt, + /// Optional request-scoped trace transport under its own namespace. + #[serde(skip_serializing_if = "Option::is_none")] + pub trusted_server: Option, } impl ToExt for ResponseExt {} +/// Request-scoped trace data beside ordinary orchestration metadata. +/// +/// The closed [`crate::trace::TraceAuctionTransportV1`] contains bounded evidence +/// or its exclusive unavailable marker; provider payloads never use this type. +#[derive(Debug, Serialize)] +pub struct TraceResponseExt { + /// Checked evidence or its fixed projection-failure transport envelope. + pub trace_auction: crate::trace::TraceAuctionTransportV1, +} + #[cfg(test)] mod tests { use super::*; @@ -216,6 +229,7 @@ mod tests { }; let ext = ResponseExt { + trusted_server: None, orchestrator: OrchestratorExt { strategy: "parallel_only".to_owned(), providers: 2, diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 61bc7efe5..a546b4b03 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -27,6 +27,7 @@ use std::time::{Duration, Instant, SystemTime}; use brotli::Decompressor; use brotli::enc::BrotliEncoderParams; use brotli::enc::writer::CompressorWriter; +use chrono::{DateTime, Utc}; use cookie::CookieJar; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt}; @@ -70,7 +71,7 @@ use crate::html_processor::BodyCloseInjection; use crate::http_util::{RequestInfo, is_navigation_request, serve_static_with_etag}; use crate::integrations::IntegrationRegistry; use crate::platform::{ - GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, + ClientInfo, GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY, VarySpec, contains_publisher_esi_directive, reader_url_surrogate_key, }; @@ -89,6 +90,10 @@ use crate::streaming_processor::{ STREAM_CHUNK_SIZE, StreamProcessor, StreamingPipeline, }; use crate::streaming_replacer::create_url_replacer; +use crate::trace::{ + TraceAuctionCarry, TraceAuctionSource, TraceAuctionTerminalReason, TraceAuctionTerminalStatus, + TraceCaptureGate, project_request_context, +}; const SUPPORTED_ENCODING_VALUES: [&str; 3] = ["gzip", "deflate", "br"]; const DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); @@ -633,6 +638,7 @@ struct ProcessResponseParams<'a> { suppress_datadome_client_side_tag: bool, gpt_diagnostics: Option<&'a crate::integrations::gpt_diagnostics::GptDiagnosticsRequestDecision>, + trace_bootstrap: Option<&'a str>, /// See [`HtmlStreamProcessorParams::shared_template_authorized`]. shared_template_authorized: bool, /// See [`HtmlStreamProcessorParams::csp_nonce_observed`]. @@ -675,6 +681,7 @@ impl PublisherBodyProcessor { ad_bids_state: Arc::clone(params.ad_bids_state.script_cell()), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.clone(), + trace_bootstrap: params.trace_bootstrap.clone(), shared_template_authorized: params.template_cache_key.is_some(), csp_nonce_observed: params.csp_nonce_observed.clone(), deferred_inline_marker: inline_seam_token @@ -759,6 +766,7 @@ fn process_response_streaming( ad_bids_state: params.ad_bids_state.clone(), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.cloned(), + trace_bootstrap: params.trace_bootstrap.map(str::to_owned), shared_template_authorized: params.shared_template_authorized, csp_nonce_observed: params.csp_nonce_observed.cloned(), deferred_inline_marker: None, @@ -958,6 +966,7 @@ fn passthrough_finish_segments( /// error paths that can still await (see [`abandon_hold_auction`]). struct DispatchedAuctionGuard { dispatched: Option, + trace: Option, /// Stays `true` from dispatch until collection (or telemetry-emitting /// abandonment) reaches a terminal result. [`Self::take`] removes the /// dispatched auction to hand it to the async collector but deliberately @@ -969,8 +978,10 @@ struct DispatchedAuctionGuard { impl DispatchedAuctionGuard { fn new(dispatched: DispatchedAuction) -> Self { + let trace = dispatched.trace_carry(); Self { dispatched: Some(dispatched), + trace, armed: true, } } @@ -992,6 +1003,12 @@ impl DispatchedAuctionGuard { impl Drop for DispatchedAuctionGuard { fn drop(&mut self) { if self.armed { + if let Some(trace) = &self.trace { + trace.finish( + crate::trace::TraceAuctionTerminalStatus::Abandoned, + Some(crate::trace::TraceAuctionTerminalReason::Unknown), + ); + } log::warn!( "Dispatched server-side auction dropped without collection; SSP bid responses discarded (publisher body stream aborted or never polled)" ); @@ -1316,6 +1333,7 @@ struct HtmlStreamProcessorParams<'a> { ad_bids_state: Arc>>, suppress_datadome_client_side_tag: bool, gpt_diagnostics: Option, + trace_bootstrap: Option, /// Whether a shared template was authorized for this response. /// /// Carried rather than re-derived so both seams see the same answer. See @@ -1503,11 +1521,15 @@ fn create_html_stream_processor( ); let assembly_mode = effective_assembly_mode(params.settings, params.shared_template_authorized); + // A deliverable skipped trace auction needs the request-only body seam even + // when ordinary ad-stack policy withheld every slot definition. + let request_body_seam = params.ad_slots_script.is_some() + || (matches!(assembly_mode, AssemblyMode::Inline) && params.trace_bootstrap.is_some()); let body_close = match (assembly_mode, params.deferred_inline_marker) { (AssemblyMode::Inline, Some(marker)) if params.ad_slots_script.is_some() => { BodyCloseInjection::DeferredInlineMarker(marker) } - _ => body_close_injection(assembly_mode, params.ad_slots_script.is_some()), + _ => body_close_injection(assembly_mode, request_body_seam), }; let gpt_diagnostics = template_gpt_diagnostics(assembly_mode, params.gpt_diagnostics); @@ -1522,6 +1544,10 @@ fn create_html_stream_processor( let config = config .with_ad_state(params.ad_slots_script, params.ad_bids_state) .with_gpt_diagnostics(gpt_diagnostics) + .with_trace_bootstrap(match assembly_mode { + AssemblyMode::Inline => params.trace_bootstrap, + AssemblyMode::Esi => None, + }) .with_body_close(body_close) .with_csp_nonce_observer(csp_nonce_observed) .with_datadome_client_tag_suppression(params.suppress_datadome_client_side_tag); @@ -1720,6 +1746,8 @@ pub struct OwnedProcessResponseParams { /// Request-scoped conditional diagnostics delivery decision. pub(crate) gpt_diagnostics: Option, + /// Request-only trace bootstrap, excluded from shared templates and ESI. + pub(crate) trace_bootstrap: Option, /// Set by the transform when the document carries a response-bound CSP nonce. /// /// `None` wherever no transform runs. Recorded by the HTML parser rather than @@ -2151,6 +2179,7 @@ fn build_template_assembly_params( dispatched_auction: None, price_granularity, gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -2908,6 +2937,7 @@ pub fn stream_publisher_body( ad_bids_state: params.ad_bids_state.script_cell(), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.as_ref(), + trace_bootstrap: params.trace_bootstrap.as_deref(), shared_template_authorized: params.template_cache_key.is_some(), csp_nonce_observed: params.csp_nonce_observed.as_ref(), }; @@ -3013,6 +3043,7 @@ pub async fn stream_publisher_body_async( ad_bids_state: Arc::clone(params.ad_bids_state.script_cell()), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.clone(), + trace_bootstrap: params.trace_bootstrap.clone(), shared_template_authorized: params.template_cache_key.is_some(), csp_nonce_observed: params.csp_nonce_observed.clone(), deferred_inline_marker: inline_seam_token @@ -3250,6 +3281,8 @@ pub(crate) struct AdBidsState { bids: Arc>>, /// Optional per-request diagnostics emitted before either bids-script shape. debug_prefix: Arc>, + /// Optional trace carry belongs to this request, outside the ordinary bid map. + trace: Option, } #[cfg(test)] @@ -3272,7 +3305,7 @@ impl AdBidsState { /// Record one auction result, rendering the script from the same map that is /// stored, so the two representations cannot drift. fn set(&self, bid_map: serde_json::Map) { - let bids_script = build_bids_script(&bid_map); + let bids_script = build_bids_script_with_trace(&bid_map, self.trace()); *self.script.lock().expect("should lock bid script") = Some(bids_script); *self.bids.lock().expect("should lock bid map") = bid_map; } @@ -3285,9 +3318,21 @@ impl AdBidsState { self.bids.lock().expect("should lock bid map").clone() } + fn trace(&self) -> Option<&TraceAuctionCarry> { + self.trace.as_ref() + } + + fn attach_trace(&mut self, trace: TraceAuctionCarry) { + self.trace = Some(trace); + self.set(self.bids()); + } + /// Build the shared-template seam, retaining the same debug prefix as inline. fn build_seam_script(&self, slots_json: &str) -> String { - let seam = build_seam_script(slots_json, &self.bids()); + let seam = match self.trace() { + Some(trace) => build_seam_script_with_trace(slots_json, &self.bids(), Some(trace)), + None => build_seam_script(slots_json, &self.bids()), + }; let prefix = self .debug_prefix .lock() @@ -3354,6 +3399,9 @@ pub(crate) fn write_bids_to_state( auction_id, ); let delivered_winner_slots = bid_map.keys().cloned().collect(); + if let Some(trace) = ad_bids_state.trace() { + trace.observe_delivery(winning_bids, &delivered_winner_slots); + } ad_bids_state.set(bid_map); delivered_winner_slots } @@ -4161,6 +4209,12 @@ async fn emit_abandoned_auction( dispatched: DispatchedAuction, reason: &'static str, ) { + if let Some(trace) = dispatched.trace_carry() { + trace.finish( + crate::trace::TraceAuctionTerminalStatus::Abandoned, + Some(crate::trace::TraceAuctionTerminalReason::Unknown), + ); + } let Some(observation) = observation else { return; }; @@ -4191,10 +4245,16 @@ async fn collect_non_html_auction( services: &RuntimeServices, settings: &Settings, ) { - let auction_id = telemetry - .auction_request - .as_ref() - .and_then(|_| diagnostics_auction_id(settings)); + let auction_id = params + .ad_bids_state + .trace() + .map(|trace| trace.token().to_string()) + .or_else(|| { + telemetry + .auction_request + .as_ref() + .and_then(|_| diagnostics_auction_id(settings)) + }); let placeholder = mediator_placeholder_request(); let result = orchestrator .collect_dispatched_auction( @@ -4245,10 +4305,15 @@ async fn collect_stream_auction( settings, request_origin, } = deps; - let auction_id = telemetry - .auction_request - .as_ref() - .and_then(|_| diagnostics_auction_id(settings)); + let auction_id = ad_bids_state + .trace() + .map(|trace| trace.token().to_string()) + .or_else(|| { + telemetry + .auction_request + .as_ref() + .and_then(|_| diagnostics_auction_id(settings)) + }); log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); @@ -4435,6 +4500,7 @@ pub async fn handle_publisher_request( // idempotent call as a direct-handler safety net and for focused tests. let gpt_diagnostics = crate::integrations::gpt_diagnostics::prepare_request(settings, &mut req)?; + let trace_gate = req.extensions().get::().copied(); // Prebid.js requests are not intercepted here anymore. The HTML processor removes // publisher-supplied Prebid scripts; the unified TSJS bundle includes Prebid.js when enabled. @@ -4589,7 +4655,7 @@ pub async fn handle_publisher_request( .and_then(|co| co.auction_timeout_ms) .unwrap_or(settings.auction.timeout_ms); - let ad_bids_state = AdBidsState::default(); + let mut ad_bids_state = AdBidsState::default(); let price_granularity = settings .creative_opportunities @@ -4601,7 +4667,7 @@ pub async fn handle_publisher_request( // keys on it, and that gate now runs while the request is still in hand. let assembly_mode = configured_assembly_mode(settings); - let auction_client_request = request_head_snapshot(&req); + let mut auction_client_request = request_head_snapshot(&req); // Everything request-derived is computed here, in one place, because this is // the last point where the request is still in hand: the origin send below @@ -4657,6 +4723,8 @@ pub async fn handle_publisher_request( let datadome_suppression_requires_origin = suppress_datadome_client_side_tag; let datadome_suppression_requires_full_body = suppress_datadome_client_side_tag && is_html_document_request(&req); + let diagnostics_requires_full_body = + gpt_diagnostics.requires_private_no_store() && is_html_document_request(&req); // The reader's own request semantics, read before any stripping. A reader who asked // for a range or a conditional response must not be handed a full document // synthesized from a template shared with other readers, whatever the origin is then @@ -4671,7 +4739,10 @@ pub async fn handle_publisher_request( // a panic-prone invariant in the public request handler. let reader_compression = reader_compression.unwrap_or(Compression::None); - if should_run_ad_stack || datadome_suppression_requires_full_body { + if should_run_ad_stack + || datadome_suppression_requires_full_body + || diagnostics_requires_full_body + { // HTML document contexts whose output may be synthesized must not // receive a cached 304 or partial 206. Non-document subresources contain // no executable injected tag, so retain their validators and ranges. @@ -4883,6 +4954,40 @@ pub async fn handle_publisher_request( // can be mutated and sent to origin immediately after. let mut auction_observation: Option = None; + let mut trace_request = trace_gate + .filter(|gate| gate.document_eligible(&gpt_diagnostics)) + .map(|_| { + build_auction_request( + &MatchedSlotsContext { + matched_slots: &matched_slots, + request_path: &request_path, + }, + ec_id, + &consent_context, + &request_info, + &settings.publisher.domain, + auction_client_request + .headers() + .get("user-agent") + .and_then(|value| value.to_str().ok()), + ) + }); + let trace_auction = trace_request.as_ref().and_then(|request| { + TraceAuctionCarry::capture_if_enabled( + true, + TraceAuctionSource::InitialNavigationSsat, + &request.slots, + ) + }); + if let Some(trace) = &trace_auction { + auction_client_request + .extensions_mut() + .insert(trace.clone()); + } + let mut trace_cancellation = trace_auction + .as_ref() + .map(TraceAuctionCarry::cancellation_guard); + let mut auction_request_for_telemetry: Option = None; let mut dispatched_auction = if matched_slots.is_empty() { None @@ -4913,14 +5018,16 @@ pub async fn handle_publisher_request( matched_slots: &matched_slots, request_path: &request_path, }; - let mut auction_request = build_auction_request( - &slots_ctx, - ec_id, - &consent_context, - &request_info, - &settings.publisher.domain, - user_agent, - ); + let mut auction_request = trace_request.take().unwrap_or_else(|| { + build_auction_request( + &slots_ctx, + ec_id, + &consent_context, + &request_info, + &settings.publisher.domain, + user_agent, + ) + }); apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { @@ -4999,6 +5106,12 @@ pub async fn handle_publisher_request( } } } else { + if let Some(trace) = &trace_auction { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } let skip_reason = if ad_templates_disabled { "ad_templates_disabled" } else if !auction.orchestrator.is_enabled() { @@ -5026,6 +5139,9 @@ pub async fn handle_publisher_request( None } }; + if let Some(trace) = &trace_auction { + ad_bids_state.attach_trace(trace.clone()); + } log::info!( "dispatch_auction: {}", if dispatched_auction.is_some() { @@ -5183,13 +5299,19 @@ pub async fn handle_publisher_request( } }; + if let Some(trace) = &trace_auction { + response.extensions_mut().insert(trace.clone()); + } + log::debug!( "Publisher origin response received: status={}, header_count={}", response.status(), response.headers().len() ); - if should_run_ad_stack && response.status() == StatusCode::NOT_MODIFIED { + if (should_run_ad_stack || diagnostics_requires_full_body) + && response.status() == StatusCode::NOT_MODIFIED + { if let Some(dispatched) = dispatched_auction.take() { emit_abandoned_auction( services, @@ -5279,13 +5401,27 @@ pub async fn handle_publisher_request( // a marker, the head seam emitted no `adSlots`, and nothing assembled either. let assembly_mode = effective_assembly_mode(settings, template_cache_key.is_some()); - let ad_slots_script = template_ad_slots_script( - assembly_mode, - should_run_ad_stack, - settings, - &matched_slots, - &request_path, - ); + let ad_slots_script = if matches!(assembly_mode, AssemblyMode::Inline) + && should_run_ad_stack + && trace_auction.is_some() + { + settings.creative_opportunities.as_ref().map(|co_config| { + build_ad_slots_script_with_trace( + &matched_slots, + co_config, + &request_path, + trace_auction.as_ref(), + ) + }) + } else { + template_ad_slots_script( + assembly_mode, + should_run_ad_stack, + settings, + &matched_slots, + &request_path, + ) + }; // §4.7: HTML with synthesized per-navigation auction state must not be // stored or validated as an origin representation. Strip both browser and @@ -5458,6 +5594,22 @@ pub async fn handle_publisher_request( let body = std::mem::replace(response.body_mut(), EdgeBody::empty()); response.headers_mut().remove(header::CONTENT_LENGTH); + let trace_bootstrap = trace_gate + .filter(|gate| { + is_html_content_type(&content_type) && gate.document_eligible(&gpt_diagnostics) + }) + .map(|gate| { + trace_document_bootstrap( + &gate, + services.client_info(), + ec_context.geo_info(), + Utc::now(), + ) + }); + + if let Some(guard) = &mut trace_cancellation { + guard.disarm(); + } Ok(PublisherResponse::Stream { response, @@ -5484,6 +5636,7 @@ pub async fn handle_publisher_request( dispatched_auction, price_granularity, gpt_diagnostics: Some(gpt_diagnostics), + trace_bootstrap, }), }) } @@ -5660,6 +5813,39 @@ fn html_escape_for_script(s: &str) -> String { out } +fn trace_document_bootstrap( + gate: &TraceCaptureGate, + client: &ClientInfo, + geo: Option<&GeoInfo>, + captured_at: DateTime, +) -> String { + let mut bootstrap = String::from(""); + bootstrap +} + +fn trace_document_context_script(context: &impl serde::Serialize) -> Option { + let json = match serde_json::to_string(context) { + Ok(json) => json, + Err(_) => { + log::warn!("trace_document_context_unavailable"); + return None; + } + }; + Some(format!( + "(function(){{var c=JSON.parse(\"{}\");Object.freeze(c.network);Object.freeze(c.cookies.ts_ec);Object.freeze(c.cookies.ts_eids);Object.freeze(c.cookies.ts_tester);Object.freeze(c.cookies.diagnostics_session);Object.freeze(c.cookies);window.__tsjs_trace_request_context=Object.freeze(c);}})();", + html_escape_for_script(&json), + )) +} + /// Maximum length Google Ad Manager accepts for a key-value targeting value. /// Longer values are rejected by GAM, so the key never reaches the creative and /// the `hb_adid` render handshake cannot complete. @@ -5926,9 +6112,24 @@ pub(crate) fn build_bid_map_with_auction_id( /// The JSON is embedded via `JSON.parse(…)` so the browser parser never sees /// raw `` sequences inside the string. pub(crate) fn build_bids_script(bid_map: &serde_json::Map) -> String { + build_bids_script_with_trace(bid_map, None) +} + +fn build_bids_script_with_trace( + bid_map: &serde_json::Map, + trace: Option<&TraceAuctionCarry>, +) -> String { let json = serde_json::to_string(bid_map) .expect("serde_json::to_string of Map should be infallible"); let escaped = html_escape_for_script(&json); + let transport = trace_transport_script(trace); + let (transport_script, scheduler_arguments) = + transport.as_ref().map_or((String::new(), "b"), |json| { + ( + format!("var x=JSON.parse(\"{}\");", html_escape_for_script(json)), + "b,undefined,x", + ) + }); // adInit() defines GPT slots on the publisher's `-container` wrappers, which // mutates those ad-slot subtrees. Calling it synchronously here (this script // runs at body-parse time) lands those mutations inside React's hydration @@ -5964,11 +6165,12 @@ pub(crate) fn build_bids_script(bid_map: &serde_json::Map(function(){{\ var t=window.tsjs=window.tsjs||{{}};\ var b=JSON.parse(\"{}\");\ +{}\ var s=t.scheduleInitialAdInit;\ -if(typeof s===\"function\")s(b);\ +if(typeof s===\"function\")s({});\ else t.bids=b;\ }})();", - escaped + escaped, transport_script, scheduler_arguments ) } @@ -5994,23 +6196,60 @@ else t.bids=b;\ pub(crate) fn build_seam_script( slots_json: &str, bid_map: &serde_json::Map, +) -> String { + build_seam_script_with_trace(slots_json, bid_map, None) +} + +fn trace_transport_script(trace: Option<&TraceAuctionCarry>) -> Option { + trace + .and_then(TraceAuctionCarry::transport) + .map(|transport| { + serde_json::to_string(&transport) + .unwrap_or_else(|_| trace_serialization_unavailable().to_string()) + }) +} + +fn trace_transport_value(transport: &crate::trace::TraceAuctionTransportV1) -> serde_json::Value { + serde_json::to_value(transport).unwrap_or_else(|_| trace_serialization_unavailable()) +} + +fn trace_serialization_unavailable() -> serde_json::Value { + log::warn!("trace evidence serialization fails"); + serde_json::json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed"}) +} + +fn build_seam_script_with_trace( + slots_json: &str, + bid_map: &serde_json::Map, + trace: Option<&TraceAuctionCarry>, ) -> String { // The local test script probes the minified `var a=JSON.parse`, // `var b=JSON.parse`, and `s(b,a)` literals below. Update the harness with any // semantically equivalent rewrite so its black-box checks keep matching output. let bids = serde_json::to_string(bid_map) .expect("serde_json::to_string of Map should be infallible"); + let transport = trace_transport_script(trace); + let (transport_script, scheduler_arguments) = + transport.as_ref().map_or((String::new(), "b,a"), |json| { + ( + format!("var x=JSON.parse(\"{}\");", html_escape_for_script(json)), + "b,a,x", + ) + }); format!( "", html_escape_for_script(slots_json), - html_escape_for_script(&bids) + html_escape_for_script(&bids), + transport_script, + scheduler_arguments ) } @@ -6705,14 +6944,39 @@ pub(crate) fn build_ad_slots_script( matched_slots: &[crate::creative_opportunities::CreativeOpportunitySlot], co_config: &crate::creative_opportunities::CreativeOpportunitiesConfig, request_path: &str, +) -> String { + build_ad_slots_script_with_trace(matched_slots, co_config, request_path, None) +} + +fn request_slot_jsons( + matched_slots: &[crate::creative_opportunities::CreativeOpportunitySlot], + co_config: &crate::creative_opportunities::CreativeOpportunitiesConfig, + section: &str, + trace: Option<&TraceAuctionCarry>, +) -> Vec { + matched_slots + .iter() + .enumerate() + .filter_map(|(index, slot)| { + let mut value = build_slot_json(slot, co_config, section)?; + if let Some(slot_ref) = trace.and_then(|trace| trace.slot_ref(index)) { + value["ext"] = serde_json::json!({"trusted_server":{"trace_slot_ref":slot_ref}}); + } + Some(value) + }) + .collect() +} + +fn build_ad_slots_script_with_trace( + matched_slots: &[crate::creative_opportunities::CreativeOpportunitySlot], + co_config: &crate::creative_opportunities::CreativeOpportunitiesConfig, + request_path: &str, + trace: Option<&TraceAuctionCarry>, ) -> String { // `{section}` derives from the same raw path `page_patterns` matched // against; derive it once for every slot on this request. let section = co_config.section_for_path(request_path); - let slots: Vec = matched_slots - .iter() - .filter_map(|slot| build_slot_json(slot, co_config, §ion)) - .collect(); + let slots = request_slot_jsons(matched_slots, co_config, §ion, trace); let json = serde_json::to_string(&slots) .expect("serde_json::to_string of Vec should be infallible"); let escaped = html_escape_for_script(&json); @@ -6881,7 +7145,7 @@ pub async fn handle_page_bids( kv: Option<&KvIdentityGraph>, auction: AuctionDispatch<'_>, ec_context: &mut EcContext, - req: Request, + mut req: Request, ) -> Result, Report> { // CSRF-style gate: refuse cross-site invocations before any other work — // including the not-configured 404 below, which would otherwise tell a @@ -7031,6 +7295,35 @@ pub async fn handle_page_bids( // unchanged) but skip the live auction, matching the existing behavior. let ad_stack_enabled = ad_templates_enabled && auction_enabled && consent_allows_auction; + let mut trace_request = req + .extensions() + .get::() + .filter(|gate| gate.base_active()) + .map(|_| { + build_auction_request( + &MatchedSlotsContext { + matched_slots: &matched_slots, + request_path: &path_param, + }, + ec_id.as_deref(), + &consent_context, + &request_info, + &settings.publisher.domain, + req.headers() + .get("user-agent") + .and_then(|value| value.to_str().ok()), + ) + }); + let trace_auction = trace_request.as_ref().and_then(|request| { + TraceAuctionCarry::capture_if_enabled(true, TraceAuctionSource::SpaPageBids, &request.slots) + }); + if let Some(trace) = &trace_auction { + req.extensions_mut().insert(trace.clone()); + } + let _trace_cancellation = trace_auction + .as_ref() + .map(TraceAuctionCarry::cancellation_guard); + let (winning_bids, prebuilt_bid_map) = if matched_slots.is_empty() { (std::collections::HashMap::new(), None) } else { @@ -7069,14 +7362,16 @@ pub async fn handle_page_bids( if !matches!(page_bids_kv_snapshot, crate::ec::EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(page_bids_kv_snapshot.clone()); } - let mut auction_request = build_auction_request( - &slots_ctx, - ec_id.as_deref(), - &consent_context, - &request_info, - &settings.publisher.domain, - user_agent, - ); + let mut auction_request = trace_request.take().unwrap_or_else(|| { + build_auction_request( + &slots_ctx, + ec_id.as_deref(), + &consent_context, + &request_info, + &settings.publisher.domain, + user_agent, + ) + }); apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { @@ -7108,7 +7403,10 @@ pub async fn handle_page_bids( { Ok(result) => { let winning_bids = result.winning_bids.clone(); - let auction_id = diagnostics_auction_id(settings); + let auction_id = trace_auction + .as_ref() + .map(|trace| trace.token().to_string()) + .or_else(|| diagnostics_auction_id(settings)); let bid_map = build_bid_map_with_auction_id( &winning_bids, co_config.price_granularity, @@ -7118,6 +7416,9 @@ pub async fn handle_page_bids( auction_id.as_deref(), ); let delivered_winner_slots = bid_map.keys().cloned().collect(); + if let Some(trace) = &trace_auction { + trace.observe_delivery(&winning_bids, &delivered_winner_slots); + } emit_auction_events_best_effort_lazy(services, || { build_auction_events( observation, @@ -7150,6 +7451,12 @@ pub async fn handle_page_bids( } } } else { + if let Some(trace) = &trace_auction { + trace.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ); + } let skip_reason = if !ad_templates_enabled { "ad_templates_disabled" } else if !auction_enabled { @@ -7188,23 +7495,30 @@ pub async fn handle_page_bids( None, ) }); + if let Some(trace) = &trace_auction { + trace.observe_delivery(&winning_bids, &bid_map.keys().cloned().collect()); + } // Gate slots on the ad-stack kill switch / consent: when disabled, return no // slots so the SPA hook does not call `adInit()` / create GPT slots. let slots_json: Vec = if ad_stack_enabled { let section = co_config.section_for_path(&path_param); - matched_slots - .iter() - .filter_map(|slot| build_slot_json(slot, co_config, §ion)) - .collect() + request_slot_jsons(&matched_slots, co_config, §ion, trace_auction.as_ref()) } else { Vec::new() }; - let body = serde_json::json!({ + let mut body = serde_json::json!({ "slots": slots_json, "bids": bid_map, }); + if let Some(trace) = &trace_auction { + body["trace_auction"] = trace_transport_value( + &trace + .transport() + .unwrap_or_else(crate::trace::TraceAuctionTransportV1::unavailable), + ); + } let body = serde_json::to_string(&body).change_context(TrustedServerError::Proxy { message: "Failed to serialize page-bids response".to_string(), })?; @@ -7216,6 +7530,9 @@ pub async fn handle_page_bids( ); enforce_terminal_private_cache_privacy(&mut response); mark_deprecated_alias(&mut response, is_legacy_alias); + if let Some(trace) = trace_auction { + response.extensions_mut().insert(trace); + } Ok(response) } @@ -8988,6 +9305,7 @@ mod tests { dispatched_auction: None, price_granularity: Default::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -9184,1077 +9502,1683 @@ mod tests { ); } - /// Drive `handle_publisher_request` with no creative opportunities — a plain - /// proxy with no server-side auction. Hides the auction/EC wiring so callers - /// read like a simple `(settings, services, req)` proxy. - async fn run_publisher_proxy( - settings: &Settings, - services: &RuntimeServices, - req: Request, - ) -> PublisherResponse { - let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); - let mut ec_context = - EcContext::read_from_request(settings, &req, services).expect("should read EC context"); - handle_publisher_request( - settings, - services, - None, - &mut ec_context, - AuctionDispatch { - orchestrator: &orchestrator, - slots: &[], - registry: None, - }, - req, - EdgeCacheHeader::SurrogateControl, - ) - .await - .expect("should proxy publisher request") - } + mod trace_document_tests { + use super::*; + use std::net::{IpAddr, Ipv4Addr}; - mod rendered_template_identity_tests { - //! The gate the plan's Task 3 Step 2 actually asks for. - //! - //! Every other test in this area exercises the decision functions with - //! hand-built inputs. That is how three HIGH review findings sat in covered, - //! passing code: the decisions were right and nothing checked what the - //! composition of them *renders*. - //! - //! These tests render whole documents through `create_html_processor`, - //! composing the same three decisions `create_html_stream_processor` uses, - //! and compare bytes. A future request-dependent injection added at either - //! seam fails here even if every decision function is left untouched. + use chrono::TimeZone as _; + use edgezero_core::router::PreDispatchHook as _; + use futures::executor::block_on; - use super::template_neutrality_tests::{settings_with_slots, slot}; - use super::*; - use crate::creative_opportunities::AssemblyMode; - use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; - use crate::integrations::IntegrationRegistry; - use crate::integrations::gpt_diagnostics::GptDiagnosticsRequestDecision; + use crate::platform::test_support::{ + StubHttpClient, build_services_with_http_client_and_client_ip, + }; + use crate::trace::{TracePreDispatchHook, TraceTerminalResponse}; - const DOCUMENT: &[u8] = - b"t

content

"; + fn gate_and_decision( + settings: &Settings, + ) -> ( + TraceCaptureGate, + crate::integrations::gpt_diagnostics::GptDiagnosticsRequestDecision, + ) { + let mut request = request("", Some("__Host-ts-console=1"), "document"); + let hook = TracePreDispatchHook::new( + Arc::new(settings.clone()), + Arc::new(|_| panic!("should not query setup metadata for a publisher document")), + ); + assert!( + block_on(hook.handle(&mut request)) + .expect("should freeze document gate") + .is_none(), + "should continue publisher routing" + ); + let decision = + crate::integrations::gpt_diagnostics::prepare_request(settings, &mut request) + .expect("should prepare active navigation decision"); + let gate = *request + .extensions() + .get::() + .expect("should retain frozen incoming gate after cookie sanitation"); + (gate, decision) + } + + fn fixed_clock() -> DateTime { + Utc.with_ymd_and_hms(2026, 10, 6, 1, 2, 3) + .single() + .expect("should construct a strict UTC example clock") + } + + fn context_from_script(script: &str) -> serde_json::Value { + let literal = script + .split_once("var c=JSON.parse(") + .expect("should parse context through a quoted JSON string") + .1 + .split_once(");Object.freeze(c.network)") + .expect("should freeze the newly parsed context before publishing it") + .0; + let json: String = serde_json::from_str(literal) + .expect("should recover the original JSON from the script-safe string literal"); + serde_json::from_str(&json).expect("should decode the exact bounded request context") + } + + struct UnserializableContext; + + impl serde::Serialize for UnserializableContext { + fn serialize(&self, _serializer: S) -> Result { + Err(::custom( + "fictional context serialization failure", + )) + } + } - /// One request's worth of variation. Everything here is request-scoped and - /// must not reach a shared template. - #[derive(Debug, Clone, Copy)] - struct RequestShape { - /// Folds in consent, bot classification, prefetch and the kill switch. - ad_stack_ran: bool, - /// Cookie- or query-activated. - diagnostics_active: bool, - /// A resolved auction, present only when one was dispatched. - bids_available: bool, + fn settings(trace_enabled: bool) -> Settings { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":trace_enabled}), + ) + .expect("should configure document trace gate"); + settings } - /// Build the config exactly as `create_html_stream_processor` does, so a - /// drift between a decision and its use is caught rather than hidden. - fn render(mode: AssemblyMode, shape: RequestShape) -> String { - let settings = settings_with_slots(); - let slots = [slot()]; + fn request(query: &str, cookie: Option<&str>, destination: &str) -> Request { + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri(format!("https://publisher.example/article{query}")) + .header(header::HOST, "publisher.example") + .header("sec-fetch-dest", destination) + .header(header::USER_AGENT, "example-browser") + .body(EdgeBody::empty()) + .expect("should build publisher document request"); + if let Some(cookie) = cookie { + request.headers_mut().insert( + header::COOKIE, + HeaderValue::from_str(cookie).expect("should encode fictional cookie input"), + ); + } + request + } - let ad_slots_script = - template_ad_slots_script(mode, shape.ad_stack_ran, &settings, &slots, "/"); - let body_close = body_close_injection(mode, ad_slots_script.is_some()); - let gpt_diagnostics = template_gpt_diagnostics( - mode, - shape - .diagnostics_active - .then(GptDiagnosticsRequestDecision::active_for_tests), + fn render( + settings: &Settings, + mut request: Request, + content_type: &str, + ) -> (Response, String) { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers(200, b"Examplepublisher ads remain".to_vec(), + vec![("content-type",content_type),("cache-control","public, max-age=300"),("etag","\"origin-example\"")]); + let services = build_services_with_http_client_and_client_ip( + stub, + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), + ); + let hook = TracePreDispatchHook::new( + Arc::new(settings.clone()), + Arc::new(|_| { + panic!("should not request setup metadata on ordinary publisher traffic") + }), ); - - let ad_bids_state = - std::sync::Arc::new(std::sync::Mutex::new(shape.bids_available.then(|| { - r#""#.to_string() - }))); - - let config = HtmlProcessorConfig { - csp_nonce_observed: None, - origin_host: "origin.example.com".to_string(), - request_host: "example.com".to_string(), - request_scheme: "https".to_string(), - integrations: IntegrationRegistry::empty_for_tests(), - ad_slots_script, - ad_bids_state, - max_buffered_body_bytes: 16 * 1024 * 1024, - gpt_diagnostics, - body_close, - suppress_datadome_client_side_tag: false, - }; - - let mut processor = create_html_processor(config); - let out = processor - .process_chunk(DOCUMENT, true) - .expect("should process the document"); - String::from_utf8(out).expect("output should be utf8") - } - - fn every_shape() -> Vec { - let mut shapes = Vec::new(); - for ad_stack_ran in [false, true] { - for diagnostics_active in [false, true] { - for bids_available in [false, true] { - shapes.push(RequestShape { - ad_stack_ran, - diagnostics_active, - bids_available, - }); - } - } - } - shapes - } - - #[test] - fn shared_template_ad_seam_is_readable_and_versioned() { - assert_eq!( - (crate::platform::TEMPLATE_SCHEMA_VERSION, AD_ASSEMBLY_SEAM,), - (5, ""), - "changing the seam must bump the cache schema, or a deploy assembles \ - against a marker that moved. The converse does not hold — the schema \ - also moves when the cache key's shape changes, as it did for v5 — so \ - updating this pin with an unchanged seam is legitimate." - ); - assert_eq!( - body_close_injection(AssemblyMode::Esi, false), - BodyCloseInjection::Marker(TEMPLATE_SEAM_PLACEHOLDER.to_string()), - "the seam's position must come from the parser, never from a byte search" - ); - assert_ne!( - TEMPLATE_SEAM_PLACEHOLDER, AD_ASSEMBLY_SEAM, - "a publisher document carrying the seam bytes must still receive \ - correctly positioned bids, which a shared marker would make impossible" + assert!( + block_on(hook.handle(&mut request)) + .expect("should freeze ordinary request gate") + .is_none(), + "should continue ordinary publisher routing" ); + crate::integrations::gpt_diagnostics::prepare_request(settings, &mut request) + .expect("should prepare effective diagnostics decision"); + assert!( + !request + .headers() + .get(header::COOKIE) + .is_some_and(|value| value + .as_bytes() + .windows(17) + .any(|part| part == b"__Host-ts-console")), + "should exercise sanitized cookies rather than re-reading the original session" + ); + let publisher = block_on(run_publisher_proxy(settings, &services, request)); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let registry = + IntegrationRegistry::new(settings).expect("should build publisher registry"); + let response = block_on(buffer_publisher_response_async( + publisher, + &Method::GET, + settings, + ®istry, + &orchestrator, + &services, + )) + .expect("should finalize publisher response"); + let (parts, body) = response.into_parts(); + let text = String::from_utf8( + body.into_bytes() + .expect("should buffer transformed test body") + .to_vec(), + ) + .expect("should produce UTF8 publisher document"); + (Response::from_parts(parts, EdgeBody::empty()), text) } #[test] - fn shared_modes_render_byte_identical_documents_for_every_request_shape() { - let mode = AssemblyMode::Esi; - let shapes = every_shape(); - let baseline = render(mode, shapes[0]); - - for shape in &shapes[1..] { - let rendered = render(mode, *shape); + fn trace_document_gate_uses_frozen_cookie_and_effective_navigation_decision() { + for (trace_enabled, cookie, query, destination, prefetch, bot, active) in [ + ( + true, + Some("__Host-ts-console=1"), + "", + "document", + false, + false, + true, + ), + (true, None, "?ts_console=1", "document", false, false, false), + ( + true, + Some("__Host-ts-console=1"), + "?ts_console=0", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1"), + "?ts_console=invalid", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1"), + "?ts_console=1&ts_console=1", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1; __Host-ts-console=1"), + "", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=invalid"), + "", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1; unrelated=example,value"), + "", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1; unrelated=\u{fffd}"), + "", + "document", + false, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1"), + "", + "document", + true, + false, + false, + ), + ( + true, + Some("__Host-ts-console=1"), + "", + "document", + false, + true, + false, + ), + ( + true, + Some("__Host-ts-console=1"), + "", + "empty", + false, + false, + false, + ), + ( + false, + Some("__Host-ts-console=1"), + "", + "document", + false, + false, + false, + ), + (true, None, "", "document", false, false, false), + ] { + let settings = settings(trace_enabled); + let mut request = request(query, cookie, destination); + if prefetch { + request + .headers_mut() + .insert("sec-purpose", HeaderValue::from_static("prefetch")); + } + if bot { + request.headers_mut().insert( + header::USER_AGENT, + HeaderValue::from_static(BOT_USER_AGENT_FRAGMENTS[0]), + ); + assert!( + is_bot_user_agent(&request), + "should exercise the existing crawler opt-out policy" + ); + } + let (response, html) = render(&settings, request, "text/html; charset=utf-8"); assert_eq!( - rendered, baseline, - "{mode:?}: rendered template differs for {shape:?}. A shared \ - template that varies by request freezes the first-filling \ - request's decision for every later reader." + html.contains("window.__tsjs_trace_active=true"), + active, + "should evaluate frozen session plus effective decision for query={query} destination={destination} trace={trace_enabled} prefetch={prefetch} bot={bot} cookie={cookie:?}" + ); + assert_eq!( + html.contains("window.__tsjs_trace_request_context"), + active, + "should expose context only for an eligible document" + ); + assert!( + html.contains("publisher ads remain"), + "should preserve publisher advertising content" + ); + assert!( + !html.contains("/_ts/trace/assets/"), + "should not inject setup-page assets into publisher traffic" ); + assert!( + response + .extensions() + .get::() + .is_none(), + "should retain ordinary publisher finalization rather than local trace terminal policy" + ); + if active { + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should retain terminal diagnostics privacy" + ); + assert!( + !response.headers().contains_key(header::ETAG), + "should strip origin validators from private document" + ); + assert!( + html.contains("192.0.2.0/24") && !html.contains("192.0.2.99"), + "should project only masked trusted client IP" + ); + } } } #[test] - fn shared_mode_templates_contain_no_request_scoped_markers() { - // Byte-identity alone would be satisfied by rendering the same wrong - // thing every time, so also assert the specific things that must be - // absent. - let mode = AssemblyMode::Esi; - let rendered = render( - mode, - RequestShape { - ad_stack_ran: true, - diagnostics_active: true, - bids_available: true, - }, - ); - for forbidden in [ - ".adSlots", - ".bids=", - "__tsjs_gpt_diagnostics_active", - "history.replaceState", + fn trace_document_non_html_responses_never_publish_bootstrap() { + for content_type in [ + "application/json", + "text/x-component", + "application/octet-stream", ] { + let (_, html) = render( + &settings(true), + request("", Some("__Host-ts-console=1"), "document"), + content_type, + ); assert!( - !rendered.contains(forbidden), - "{mode:?}: template contains request-scoped `{forbidden}`:\n{rendered}" + !html.contains("__tsjs_trace_"), + "should leave API, SPA Flight and binary responses without document globals" ); } } #[test] - fn inline_still_varies_by_request_as_it_must() { - // The shared-mode assertions would also pass if rendering were broken - // everywhere. Inline responses are per-navigation and never shared, so - // they *should* differ — this proves the test can tell the difference. - let with_ads = render( - AssemblyMode::Inline, - RequestShape { - ad_stack_ran: true, - diagnostics_active: false, - bids_available: true, - }, + fn trace_document_context_is_script_safe_and_emitted_once_before_main_bundle() { + let settings = settings(true); + let (gate, decision) = gate_and_decision(&settings); + let sentinel = "", + "should omit failed context and retain the independent evaluated gate" ); assert!( - script[fallback..].contains("t.bids=b"), - "the fallback should still assign bids: {script}" + trace_document_context_script(&UnserializableContext).is_none(), + "should omit a serialization failure without publishing error text" + ); + let mut response = Response::new(EdgeBody::empty()); + crate::integrations::gpt_diagnostics::finalize_response(&decision, &mut response); + crate::response_privacy::apply_response_headers_with_cache_privacy( + &settings, + &mut response, ); - } - } - - mod page_bids_format_tests { - //! Page-bids is a JSON API. The old executable fragment was part of the removed - //! parser-based path and must not remain as an accidental public surface. - - use super::*; - - #[test] - fn an_unknown_format_response_is_not_storable() { - // Every response from this endpoint is per-user. An error is no exception: - // a cached 400 would be replayed to clients asking correctly. - let response = page_bids_unknown_format(); - assert_eq!(response.status(), StatusCode::BAD_REQUEST); assert_eq!( - response - .headers() - .get(header::CACHE_CONTROL) - .and_then(|v| v.to_str().ok()), - Some("no-store, private") + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should retain diagnostics privacy after failed context and hostile operator cache policy" + ); + assert!( + !response.headers().contains_key("surrogate-control"), + "should prevent edge cache overrides after a failed context" ); assert!( response .extensions() .get::() .is_some(), - "page-bids errors should remain terminal-private after late response effects" + "should preserve ordinary diagnostics terminal privacy" ); - } - - #[test] - fn a_preflight_denial_is_terminal_private() { - let response = page_bids_preflight_denied(); - assert!( response .extensions() - .get::() - .is_some(), - "page-bids preflight denial should remain private after late response effects" + .get::() + .is_none(), + "should never substitute local trace terminal policy for a publisher document" + ); + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html".to_owned(); + params.gpt_diagnostics = Some(decision); + params.trace_bootstrap = Some(bootstrap); + let mut output = Vec::new(); + stream_publisher_body( + EdgeBody::from("publisher ads remain"), + &mut output, + ¶ms, + &settings, + &IntegrationRegistry::new(&settings).expect("should build registry"), + ) + .expect("should preserve advertising after context failure"); + let html = String::from_utf8(output).expect("should preserve UTF8 document"); + assert!( + html.contains("publisher ads remain") + && html.contains("window.__tsjs_trace_active=true") + && !html.contains("window.__tsjs_trace_request_context"), + "should retain advertising and the flag while omitting unavailable context" ); } - } - - mod template_cache_store_authorization_tests { - //! The store is authorized by the key's presence and nothing else. These cover - //! the two ways that could silently break: storing without authorization, and - //! storing twice for one request. - - use super::*; - /// Records what was stored, so the assertions are about behaviour rather than - /// about a call not returning an error. - #[derive(Default)] - struct RecordingCache { - stored: Arc>>, + #[test] + fn trace_document_shared_esi_processor_drops_request_bootstrap_explicitly() { + let mut settings = settings(true); + settings.creative_opportunities = Some( + serde_json::from_value( + serde_json::json!({"gam_network_id":"12345","assembly_mode":"esi"}), + ) + .expect("should configure fictional shared assembly"), + ); + let (gate, decision) = gate_and_decision(&settings); + let registry = IntegrationRegistry::new(&settings).expect("should build registry"); + let mut processor = create_html_stream_processor(HtmlStreamProcessorParams { + origin_host: "origin.example.com", + request_host: "publisher.example.com", + request_scheme: "https", + settings: &settings, + integration_registry: ®istry, + ad_slots_script: None, + ad_bids_state: Arc::new(Mutex::new(None)), + suppress_datadome_client_side_tag: false, + gpt_diagnostics: Some(decision), + trace_bootstrap: Some(trace_document_bootstrap( + &gate, + &ClientInfo::default(), + None, + fixed_clock(), + )), + shared_template_authorized: true, + csp_nonce_observed: None, + deferred_inline_marker: None, + }) + .expect("should create shared processor"); + let bytes = processor + .process_chunk( + b"shared example", + true, + ) + .expect("should transform shared example"); + let html = String::from_utf8(bytes).expect("should produce UTF8 shared template"); + assert!( + !html.contains("__tsjs_trace_") && !html.contains("__tsjs_gpt_diagnostics_active"), + "should explicitly exclude both request bootstraps from shared ESI bytes" + ); } - struct RecordingReservation { - stored: Arc>>, - url: String, + #[cfg(not(target_arch = "wasm32"))] + #[test] + #[ignore = "requires the declared Node toolchain for emitted-script execution"] + fn trace_document_emitted_script_deep_freezes_owned_context_in_node() { + let settings = settings(true); + let (gate, _) = gate_and_decision(&settings); + let sentinel = "")) + .expect("should contain exactly one inline script wrapper"); + let probe = format!( + "'use strict';globalThis.window={{}};{script};const c=window.__tsjs_trace_request_context;if(window.__tsjs_trace_active!==true)throw Error('missing literal gate');if(c.network.edge_hostname!==process.argv[1])throw Error('metadata did not roundtrip');if(Object.hasOwn(c.network,'tls_protocol'))throw Error('unsupported metadata published');for(const v of [c,c.network,c.cookies,c.cookies.ts_ec,c.cookies.ts_eids,c.cookies.ts_tester,c.cookies.diagnostics_session])if(!Object.isFrozen(v))throw Error('mutable context');let rejected=false;try{{c.cookies.diagnostics_session.state='absent'}}catch{{rejected=true}}if(!rejected)throw Error('context mutation accepted');" + ); + let output = std::process::Command::new("node") + .arg("-e") + .arg(probe) + .arg(sentinel) + .output() + .expect("should execute emitted script with the declared Node toolchain"); + assert!( + output.status.success(), + "should deeply freeze freshly parsed context: {}", + String::from_utf8_lossy(&output.stderr) + ); } + } - impl crate::platform::PlatformTemplateCacheReservation for RecordingReservation { - fn insert( - self: Box, - _metadata: &crate::platform::TemplateMetadata, - body: Vec, - _max_age: Duration, - ) -> Result<(), crate::platform::TemplateCacheError> { - self.stored - .lock() - .expect("should lock recorded stores") - .push((self.url, body.len())); - Ok(()) - } - - fn cancel(self: Box) -> Result<(), crate::platform::TemplateCacheError> { - Ok(()) - } - } - - #[async_trait::async_trait(?Send)] - impl crate::platform::PlatformTemplateCache for RecordingCache { - async fn get( - &self, - _key: &crate::platform::TemplateCacheKey, - ) -> Result - { - Err(crate::platform::TemplateCacheMiss::NotFound) - } + /// Drive `handle_publisher_request` with no creative opportunities — a plain + /// proxy with no server-side auction. Hides the auction/EC wiring so callers + /// read like a simple `(settings, services, req)` proxy. + async fn run_publisher_proxy( + settings: &Settings, + services: &RuntimeServices, + req: Request, + ) -> PublisherResponse { + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let mut ec_context = + EcContext::read_from_request(settings, &req, services).expect("should read EC context"); + handle_publisher_request( + settings, + services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + req, + EdgeCacheHeader::SurrogateControl, + ) + .await + .expect("should proxy publisher request") + } - async fn put( - &self, - key: &crate::platform::TemplateCacheKey, - _metadata: &crate::platform::TemplateMetadata, - body: Vec, - _max_age: Duration, - ) -> Result<(), crate::platform::TemplateCacheError> { - self.stored - .lock() - .expect("should lock recorded stores") - .push((key.url.clone(), body.len())); - Ok(()) - } + mod rendered_template_identity_tests { + //! The gate the plan's Task 3 Step 2 actually asks for. + //! + //! Every other test in this area exercises the decision functions with + //! hand-built inputs. That is how three HIGH review findings sat in covered, + //! passing code: the decisions were right and nothing checked what the + //! composition of them *renders*. + //! + //! These tests render whole documents through `create_html_processor`, + //! composing the same three decisions `create_html_stream_processor` uses, + //! and compare bytes. A future request-dependent injection added at either + //! seam fails here even if every decision function is left untouched. - async fn purge_url( - &self, - _key: &crate::platform::TemplateCacheKey, - ) -> Result<(), crate::platform::TemplateCacheError> { - Ok(()) - } + use super::template_neutrality_tests::{settings_with_slots, slot}; + use super::*; + use crate::creative_opportunities::AssemblyMode; + use crate::html_processor::{HtmlProcessorConfig, create_html_processor}; + use crate::integrations::IntegrationRegistry; + use crate::integrations::gpt_diagnostics::GptDiagnosticsRequestDecision; - async fn purge_url_surrogate_key( - &self, - _key: &str, - ) -> Result<(), crate::platform::TemplateCacheError> { - Ok(()) - } + const DOCUMENT: &[u8] = + b"t

content

"; - async fn purge_all(&self) -> Result<(), crate::platform::TemplateCacheError> { - Ok(()) - } + /// One request's worth of variation. Everything here is request-scoped and + /// must not reach a shared template. + #[derive(Debug, Clone, Copy)] + struct RequestShape { + /// Folds in consent, bot classification, prefetch and the kill switch. + ad_stack_ran: bool, + /// Cookie- or query-activated. + diagnostics_active: bool, + /// A resolved auction, present only when one was dispatched. + bids_available: bool, } - impl RecordingCache { - fn recorded(&self) -> Vec<(String, usize)> { - self.stored - .lock() - .expect("should lock recorded stores") - .clone() - } - } + /// Build the config exactly as `create_html_stream_processor` does, so a + /// drift between a decision and its use is caught rather than hidden. + fn render(mode: AssemblyMode, shape: RequestShape) -> String { + let settings = settings_with_slots(); + let slots = [slot()]; - fn key() -> crate::platform::TemplateCacheKey { - crate::platform::TemplateCacheKey { - url: "https://example.com/page".to_string(), + let ad_slots_script = + template_ad_slots_script(mode, shape.ad_stack_ran, &settings, &slots, "/"); + let body_close = body_close_injection(mode, ad_slots_script.is_some()); + let gpt_diagnostics = template_gpt_diagnostics( + mode, + shape + .diagnostics_active + .then(GptDiagnosticsRequestDecision::active_for_tests), + ); + + let ad_bids_state = + std::sync::Arc::new(std::sync::Mutex::new(shape.bids_available.then(|| { + r#""#.to_string() + }))); + + let config = HtmlProcessorConfig { + csp_nonce_observed: None, + origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), - request_path: "/page".to_string(), - origin_identity: "https://origin.example.com\0origin.example.com".to_string(), - assembly_mode: AssemblyMode::Esi, - vary_values: vec![], - cookie_values: Vec::new(), - template_fingerprint: "fp".to_string(), - schema_version: crate::platform::TEMPLATE_SCHEMA_VERSION, - } + integrations: IntegrationRegistry::empty_for_tests(), + ad_slots_script, + ad_bids_state, + max_buffered_body_bytes: 16 * 1024 * 1024, + gpt_diagnostics, + trace_bootstrap: None, + body_close, + suppress_datadome_client_side_tag: false, + }; + + let mut processor = create_html_processor(config); + let out = processor + .process_chunk(DOCUMENT, true) + .expect("should process the document"); + String::from_utf8(out).expect("output should be utf8") } - fn authorization(cache: &RecordingCache) -> AuthorizedTemplateStore { - AuthorizedTemplateStore { - reservation: crate::platform::TemplateCacheReservation::new(Box::new( - RecordingReservation { - stored: Arc::clone(&cache.stored), - url: key().url, - }, - )), - expires_at: Instant::now() + Duration::from_secs(30), + fn every_shape() -> Vec { + let mut shapes = Vec::new(); + for ad_stack_ran in [false, true] { + for diagnostics_active in [false, true] { + for bids_available in [false, true] { + shapes.push(RequestShape { + ad_stack_ran, + diagnostics_active, + bids_available, + }); + } + } } + shapes } - #[tokio::test] - async fn an_unauthorized_response_stores_nothing() { - // `None` is what the gate leaves behind on every bypass, and it is also the - // default for Inline. If this ever stored, every bypass reason would be - // decorative. - let cache = Arc::new(RecordingCache::default()); - let settings = create_test_settings(); - let mut params = make_stream_params(&settings, "identity"); - params.template_cache_key = None; - - let _ = store_template_if_authorized(&mut params, b"body").await; - - assert!( - cache.recorded().is_empty(), - "a response the gate rejected must not reach the cache" + #[test] + fn shared_template_ad_seam_is_readable_and_versioned() { + assert_eq!( + (crate::platform::TEMPLATE_SCHEMA_VERSION, AD_ASSEMBLY_SEAM,), + (5, ""), + "changing the seam must bump the cache schema, or a deploy assembles \ + against a marker that moved. The converse does not hold — the schema \ + also moves when the cache key's shape changes, as it did for v5 — so \ + updating this pin with an unchanged seam is legitimate." ); - } - - #[tokio::test] - async fn an_authorized_response_stores_the_transformed_bytes() { - let cache = Arc::new(RecordingCache::default()); - let settings = create_test_settings(); - let mut params = make_stream_params(&settings, "identity"); - params.template_cache_key = Some(authorization(&cache)); - - let _ = store_template_if_authorized(&mut params, b"transformed").await; - assert_eq!( - cache.recorded(), - vec![("https://example.com/page".to_string(), 24)], - "the authorized response should store its transformed bytes" + body_close_injection(AssemblyMode::Esi, false), + BodyCloseInjection::Marker(TEMPLATE_SEAM_PLACEHOLDER.to_string()), + "the seam's position must come from the parser, never from a byte search" + ); + assert_ne!( + TEMPLATE_SEAM_PLACEHOLDER, AD_ASSEMBLY_SEAM, + "a publisher document carrying the seam bytes must still receive \ + correctly positioned bids, which a shared marker would make impossible" ); } - #[tokio::test] - async fn authorization_is_consumed_so_one_request_stores_once() { - // The finalizers are layered, and a future change could plausibly call this - // from both. Taking the key makes a double store impossible rather than - // merely unlikely. - let cache = Arc::new(RecordingCache::default()); - let settings = create_test_settings(); - let mut params = make_stream_params(&settings, "identity"); - params.template_cache_key = Some(authorization(&cache)); - let _ = store_template_if_authorized(&mut params, b"first").await; - let _ = store_template_if_authorized(&mut params, b"second").await; + #[test] + fn shared_modes_render_byte_identical_documents_for_every_request_shape() { + let mode = AssemblyMode::Esi; + let shapes = every_shape(); + let baseline = render(mode, shapes[0]); - assert_eq!( - cache.recorded().len(), - 1, - "authorization must be single-use" - ); + for shape in &shapes[1..] { + let rendered = render(mode, *shape); + assert_eq!( + rendered, baseline, + "{mode:?}: rendered template differs for {shape:?}. A shared \ + template that varies by request freezes the first-filling \ + request's decision for every later reader." + ); + } } - #[tokio::test] - async fn origin_freshness_keeps_ticking_while_the_template_is_built() { - let cache = Arc::new(RecordingCache::default()); - let settings = create_test_settings(); - let mut params = make_stream_params(&settings, "identity"); - let mut expired = authorization(&cache); - expired.expires_at = Instant::now(); - params.template_cache_key = Some(expired); - - let outcome = store_template_if_authorized(&mut params, b"body").await; + #[test] + fn shared_mode_templates_contain_no_request_scoped_markers() { + // Byte-identity alone would be satisfied by rendering the same wrong + // thing every time, so also assert the specific things that must be + // absent. + let mode = AssemblyMode::Esi; + let rendered = render( + mode, + RequestShape { + ad_stack_ran: true, + diagnostics_active: true, + bids_available: true, + }, + ); + for forbidden in [ + ".adSlots", + ".bids=", + "__tsjs_gpt_diagnostics_active", + "history.replaceState", + ] { + assert!( + !rendered.contains(forbidden), + "{mode:?}: template contains request-scoped `{forbidden}`:\n{rendered}" + ); + } + } - assert_eq!(outcome, Some(TemplateStoreOutcome::Expired)); + #[test] + fn inline_still_varies_by_request_as_it_must() { + // The shared-mode assertions would also pass if rendering were broken + // everywhere. Inline responses are per-navigation and never shared, so + // they *should* differ — this proves the test can tell the difference. + let with_ads = render( + AssemblyMode::Inline, + RequestShape { + ad_stack_ran: true, + diagnostics_active: false, + bids_available: true, + }, + ); + let without = render( + AssemblyMode::Inline, + RequestShape { + ad_stack_ran: false, + diagnostics_active: false, + bids_available: false, + }, + ); + assert_ne!( + with_ads, without, + "inline must still vary by request; if it does not, this harness is \ + not rendering what it claims to" + ); assert!( - cache.recorded().is_empty(), - "template cache must not invent a new TTL after origin freshness elapsed" + with_ads.contains(".adSlots"), + "inline with a matched slot should carry adSlots" ); } } - mod template_cache_end_to_end_tests { - //! The chain, end to end: a second request for the same URL must be served from - //! the cache without touching the origin. Everything else in this file tests a - //! link; this tests that they connect. - + mod template_fingerprint_tests { use super::*; - use crate::platform::ClientInfo; - use crate::platform::test_support::{ - NoopConfigStore, NoopGeo, NoopSecretStore, StubBackend, StubHttpClient, - }; - use crate::test_support::tests::crate_test_settings_str; - use std::collections::HashMap; - use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; - /// A working cache, unlike the recorder above — this one has to actually return - /// what it stored, or a hit proves nothing. - #[derive(Default)] - struct MemoryTemplateCache { - entries: Arc>>, - /// Set only after a reservation has committed its reader-neutral bytes. - insert_completed: Arc, - /// Every key `get` was called with, so a test can assert what was *asked - /// for* rather than only what came back. A lookup that names a schema - /// version this binary cannot assemble is the bug; whether the entry - /// happened to survive the later marker check is not the same question. - lookups: Arc>>, - /// Every key `put` was called with, so a test can re-key an entry exactly - /// instead of reconstructing what the request derived. - stored_keys: Arc>>, - /// Per-entry freshness handed from publisher policy to the platform cache. - stored_max_ages: Arc>>, - /// Force the lookup transaction to fail, for the fail-open + telemetry - /// contract. A backend outage must never become a publisher outage. - fail_lookup: AtomicBool, - /// Surrogate keys a purge asked for. This double stores by cache key, so it - /// cannot resolve a surrogate key to entries the way the platform does — - /// recording the request is what a test can assert on. - purged_surrogate_keys: Arc>>, + /// Base settings with one integration's config replaced. + /// + /// Edits the parsed `[integrations]` map rather than appending TOML, so the two + /// fixtures differ in exactly the field under test — the base settings already + /// declare `[integrations.prebid]`, and a second table would not parse. + fn settings_with_prebid(enabled: bool, timeout_ms: u32) -> Settings { + let mut settings = create_test_settings(); + settings.integrations.insert( + "prebid".to_string(), + serde_json::json!({ + "enabled": enabled, + "external_bundle_url": "https://assets.example.com/prebid/bundle.js", + "timeout_ms": timeout_ms, + }), + ); + settings } - struct MemoryTemplateReservation { - entries: Arc>>, - insert_completed: Arc, - stored_keys: Arc>>, - stored_max_ages: Arc>>, - key: crate::platform::TemplateCacheKey, + #[test] + fn disabling_an_integration_changes_the_fingerprint() { + // The fingerprint was `concatenated_hash(all_module_ids())` — every module + // compiled into the binary, so a constant for that binary. Turning an + // integration off changed the injected `\"\\\u{2028}\u{2029}"}}) + .as_object() + .expect("should construct bids map") + .clone(); + let slots = serde_json::json!([{"id":"slot","targeting":{"example":"\"\\\u{2028}\u{2029}"}}]); + let mut state = AdBidsState::default(); + state.set(bids.clone()); + state.attach_trace(carry); + for script in [ + inline_bids_script(&state), + state.build_seam_script(&slots.to_string()), + ] { + assert_eq!( + decode_script_json(&script, "var x=JSON.parse("), + expected, + "should preserve every checked terminal envelope" + ); + assert_eq!( + decode_script_json(&script, "var b=JSON.parse("), + serde_json::Value::Object(bids.clone()), + "should preserve ordinary bids" + ); + assert_eq!( + script.matches("").count(), + 1, + "should escape embedded closing tags" + ); + assert!( + !script.contains('\u{2028}') && !script.contains('\u{2029}'), + "should escape JavaScript separators" + ); + } + assert_eq!( + decode_script_json( + &state.build_seam_script(&slots.to_string()), + "var a=JSON.parse(" + ), + slots, + "should preserve ordinary slot definitions" + ); } - let response = run_bidding_at_price( - &settings, - &services, - cookie_policy_request(&[field]), - if index == 0 { 3.5 } else { 7.5 }, + } + + #[test] + fn trace_transport_projection_failure_keeps_the_ordinary_bids_and_slot_helper_clean() { + let settings = settings_with_mode("inline"); + let co = settings + .creative_opportunities + .as_ref() + .expect("should configure slots"); + let slot = article_slot(); + let mut auction_slot = slot.to_ad_slot(); + auction_slot.formats = + vec![auction_slot.formats[0].clone(); usize::from(u16::MAX) + 17]; + let carry = TraceAuctionCarry::capture_if_enabled( + true, + TraceAuctionSource::InitialNavigationSsat, + &[auction_slot], ) - .await; + .expect("should capture source failure privately"); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + let mut state = AdBidsState::default(); + let bids = serde_json::json!({"slot":{"hb_pb":"1.50"}}) + .as_object() + .expect("should construct bid map") + .clone(); + state.set(bids.clone()); + state.attach_trace(carry); + let script = inline_bids_script(&state); assert_eq!( - response.headers()[HEADER_X_TS_TEMPLATE_CACHE], - if index == 0 { "miss-stored" } else { "hit" }, - "should serve the second reader from the shared template" + decode_script_json(&script, "var x=JSON.parse("), + serde_json::json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed"}), + "should deliver only the fixed unavailable envelope" + ); + assert_eq!( + decode_script_json(&script, "var b=JSON.parse("), + serde_json::Value::Object(bids), + "should preserve normal bid data" ); assert!( - response.headers()[header::CACHE_CONTROL] - .to_str() - .expect("should read cache policy") - .contains("private"), - "should keep reader output private" + build_slot_json(&slot, co, "article") + .expect("should render generic slot") + .get("ext") + .is_none(), + "should leave shared generic slot helper token-free" ); - let document = - String::from_utf8(body_of(response).await).expect("should decode document"); - assert_eq!( - seam_bids(&document) - .get("test-slot") - .and_then(|bid| bid.get("hb_pb")) - .and_then(serde_json::Value::as_str), - Some(if index == 0 { "3.50" } else { "7.50" }), - "should assemble this request's winning bid" + assert!( + !build_ad_slots_script(&[slot], co, "/article").contains("trace_slot_ref"), + "should leave template producer token-free" ); } - let requests = stub.recorded_request_uris(); - assert_eq!( - requests - .iter() - .filter(|uri| uri.contains("/article")) - .count(), - 1, - "should fetch one shared origin template" - ); - assert_eq!( - requests.len(), - 3, - "should run a fresh auction on each reader request" - ); - let entries = cache.entries.lock().expect("should lock templates"); - assert_eq!(entries.len(), 1, "should share within a variant"); - let stored = String::from_utf8_lossy( - &entries - .values() - .next() - .expect("should store a template") - .body, - ); - assert!( - stored.contains(AD_ASSEMBLY_SEAM), - "should keep the unresolved reader assembly marker" - ); - for reader_bytes in ["reader1", "reader2", "window.tsjs", "hb_pb"] { + + #[tokio::test] + async fn trace_transport_live_document_binds_slots_and_reuses_token_for_actual_delivery() + { + for finalizer in [Finalizer::Buffered, Finalizer::Streaming] { + for winning in [false, true] { + let config = format!( + "{}\n[auction]\nenabled=true\n[auction.providers.bidder]\nprotocol=\"openrtb-2.6\"\nprofile=\"standard\"\nendpoint=\"https://bidder.example.com/auction\"\nrouting=\"all_eligible\"\n[creative_opportunities]\ngam_network_id=\"12345\"\n", + crate_test_settings_str() + ); + let mut settings = + Settings::from_toml(&config).expect("should configure actual bidder"); + settings.publisher.domain = "publisher.example.com".to_owned(); + settings.publisher.origin_url = "https://origin.example.com".to_owned(); + settings.proxy.allowed_domains = + vec!["*.example.com".to_owned(), "*.example".to_owned()]; + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable trace"); + let settings = Arc::new(settings); + let orchestrator = Arc::new(AuctionOrchestrator::from_plan( + Arc::new( + crate::auction::compile_auction_plan(&settings) + .expect("should compile production bidder plan"), + ), + None, + )); + let stub = Arc::new(StubHttpClient::new()); + if winning { + stub.push_response(200, serde_json::to_vec(&serde_json::json!({"seatbid":[{"seat":"bidder","bid":[{"id":"bid","impid":"test-slot","price":1.5,"w":728,"h":90,"adm":"
Example creative
"}]}]})).expect("should encode winning provider response")); + } else { + stub.push_response(204, Vec::new()); + } + queue_shareable_html(&stub); + let services = services_for_ip( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), + ); + let mut request = navigation_request_with_cookie("__Host-ts-console=1"); + TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load setup metadata")), + ) + .handle(&mut request) + .await + .expect("should freeze document gate"); + crate::integrations::gpt_diagnostics::prepare_request( + &settings, + &mut request, + ) + .expect("should freeze diagnostics decision"); + let mut ec_context = EcContext::new_for_test(None, scheduling_consent()); + let slots = [article_slot()]; + let publisher = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &slots, + registry: None, + }, + request, + EdgeCacheHeader::SMaxageFallback, + ) + .await + .expect("should dispatch real publisher auction"); + let registry = IntegrationRegistry::new(&settings) + .expect("should build integration registry"); + let response = finalize_test_publisher_response( + publisher, + &settings, + &services, + ®istry, + orchestrator, + finalizer, + ) + .await; + assert!( + response + .extensions() + .get::() + .is_some(), + "should preserve ordinary terminal-private response" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "should keep publisher finalization active" + ); + let carry = response + .extensions() + .get::() + .expect("should retain request carry") + .clone(); + let token = carry.token().to_string(); + let document = String::from_utf8(body_of(response).await) + .expect("should deliver HTML"); + let transport = decode_script_json(&document, "var x=JSON.parse("); + let evidence = &transport["evidence"]; + assert_eq!( + evidence["source"], "initial_navigation_ssat", + "should retain source" + ); + assert_eq!( + evidence["terminal_status"], "completed", + "should finish actual provider work" + ); + assert_eq!( + evidence["diagnostic_auction_id"], token, + "should reuse pre-dispatch identity" + ); + let emitted = decode_script_json(&document, ".adSlots=JSON.parse("); + assert_eq!( + emitted[0]["ext"]["trusted_server"]["trace_slot_ref"], + evidence["slots"][0]["slot_ref"], + "should bind exact emitted slot occurrence" + ); + assert_eq!( + evidence["slots"][0]["candidate"], + if winning { "selected" } else { "no_candidate" }, + "should use actual bid-map delivery" + ); + let bids = decode_script_json(&document, "var b=JSON.parse("); + if winning { + assert_eq!( + bids["test-slot"]["hb_auction_id"], token, + "should share the opportunity token" + ); + } + assert_eq!( + document.matches("var x=JSON.parse(").count(), + 1, + "should deliver one request transport" + ); + assert_eq!( + stub.recorded_request_bodies().len(), + 2, + "should execute bidder and origin HTTP calls" + ); + } + } + } + + fn services_for_ip( + stub: Arc, + cache: Arc, + ip: IpAddr, + ) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(stub) + .geo(Arc::new(NoopGeo)) + .client_info(ClientInfo { + client_ip: Some(ip), + ..ClientInfo::default() + }) + .template_cache(cache) + .build() + } + + async fn run_frozen( + settings: &Arc, + services: &RuntimeServices, + mut request: Request, + ) -> Response { + let hook = TracePreDispatchHook::new( + Arc::clone(settings), + Arc::new(|_| panic!("should never query setup metadata for publisher traffic")), + ); assert!( - !stored.contains(reader_bytes), - "should exclude per-reader state from stored bytes" + hook.handle(&mut request) + .await + .expect("should freeze incoming publisher cookie health") + .is_none(), + "should continue publisher routing" ); + crate::integrations::gpt_diagnostics::prepare_request(settings, &mut request) + .expect( + "should freeze effective document diagnostics before publisher processing", + ); + run(settings, services, request).await + } + + fn settings_without_ad_stack() -> Arc { + let mut settings = settings_with_mode("inline"); + settings.auction.enabled = false; + settings.creative_opportunities = None; + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should configure document tracing without the ad stack"); + Arc::new(settings) + } + + fn conditional_trace_request(cookie: bool) -> Request { + let mut request = if cookie { + navigation_request_with_cookie("__Host-ts-console=1") + } else { + navigation_request() + }; + for (name, value) in [ + (header::IF_NONE_MATCH, "\"cached-document\""), + (header::IF_MODIFIED_SINCE, "Wed, 21 Oct 2015 07:28:00 GMT"), + (header::RANGE, "bytes=0-99"), + (header::IF_RANGE, "\"cached-document\""), + ] { + request + .headers_mut() + .insert(name, HeaderValue::from_static(value)); + } + request + } + + #[tokio::test] + async fn trace_document_reload_gets_full_origin_body_without_the_ad_stack() { + for (cookie, query, destination, strip, active) in [ + (true, None, "document", true, true), + (false, Some("ts_console=1"), "document", true, false), + (true, Some("ts_console=0"), "document", true, false), + (false, None, "document", false, false), + (true, None, "script", false, false), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_for_ip( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), + ); + let settings = settings_without_ad_stack(); + let mut request = conditional_trace_request(cookie); + if let Some(query) = query { + *request.uri_mut() = format!("https://ts.example.com/article?{query}") + .parse() + .expect("should construct an explicit diagnostics directive"); + } + request + .headers_mut() + .insert("sec-fetch-dest", HeaderValue::from_static(destination)); + queue_shareable_html(&stub); + + let response = run_frozen(&settings, &services, request).await; + + assert_eq!( + response.status(), + StatusCode::OK, + "should return origin HTML" + ); + let recorded = stub.recorded_request_headers(); + let outbound = recorded.first().expect("should record the origin request"); + for name in [ + header::IF_NONE_MATCH, + header::IF_MODIFIED_SINCE, + header::RANGE, + header::IF_RANGE, + ] { + assert_eq!( + outbound + .iter() + .any(|(key, _)| key.eq_ignore_ascii_case(name.as_str())), + !strip, + "should strip {name} only for request-scoped document changes" + ); + } + let document = String::from_utf8(body_of(response).await) + .expect("should render the complete publisher document"); + assert_eq!( + document.contains("window.__tsjs_trace_active=true"), + active, + "should bootstrap only a frozen active incoming document session" + ); + } } - } - #[tokio::test] - async fn template_cookie_publisher_warm_variant_finalizes_ec_withdrawal() { - for finalizer in [Finalizer::Streaming, Finalizer::Buffered] { - let mut settings = cookie_policy_settings(Some(&["ab_bucket"]), None, true); - Arc::make_mut(&mut settings).auction.providers = - crate::auction_config_types::AuctionConfig::legacy_provider_map(&[ - SCHEDULING_PROVIDER, - ]); - let stub = Arc::new(StubHttpClient::new()); - let cache = Arc::new(MemoryTemplateCache::default()); - let services = services(Arc::clone(&stub), Arc::clone(&cache)); - let graph = KvIdentityGraph::in_memory("cookie-withdrawal-store"); - let identities = [ - format!("{}.Read01", "a".repeat(64)), - format!("{}.Read02", "b".repeat(64)), - ]; - for (index, identity) in identities.iter().enumerate() { - assert!( - crate::ec::generation::is_valid_ec_id(identity), - "should use valid EC identities in the fixture" + #[tokio::test] + async fn trace_transport_document_gate_delivers_skip_facts_without_telemetry() { + for (cookie, query, active) in [ + (false, None, false), + (false, Some("ts_console=1"), false), + (true, Some("ts_console=0"), false), + (true, None, true), + ] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_for_ip( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), ); - graph - .create( - identity, - &crate::ec::kv_types::KvEntry::minimal( - "example.com", - &format!("partner-reader-{index}"), - crate::ec::current_timestamp(), - ), - ) - .expect("should seed a live reader identity"); - } - let registry = IntegrationRegistry::new(&settings) - .expect("should create integration registry"); - let partner = serde_json::from_value(serde_json::json!({ - "name": "Example partner", - "source_domain": "example.com", - "bidstream_enabled": true - })) - .expect("should deserialize fixture partner"); - let partners = PartnerRegistry::from_config(&[partner]) - .expect("should create partner registry"); - // Only the cold request has an origin response available. - queue_shareable_html(&stub); + let settings = settings_without_ad_stack(); + let mut request = conditional_trace_request(cookie); + if let Some(query) = query { + *request.uri_mut() = format!("https://ts.example.com/article?{query}") + .parse() + .expect("should build a deliberate diagnostics directive"); + } + queue_shareable_html(&stub); - for (index, (identity, withdrawn)) in [ - (&identities[0], false), - (&identities[1], false), - (&identities[1], true), - ] - .into_iter() - .enumerate() - { - let consent = if withdrawn { - ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::UsState( - "CA".to_owned(), - ), - gpc: true, - ..Default::default() - } + let response = run_frozen(&settings, &services, request).await; + + assert_eq!( + response.status(), + StatusCode::OK, + "should preserve ordinary publisher delivery" + ); + let carry = response + .extensions() + .get::(); + assert_eq!( + carry.is_some(), + active, + "should allocate only for the frozen eligible document gate" + ); + let expected_token = if let Some(carry) = carry { + let value = serde_json::to_value( + carry + .transport() + .expect("should retain directly observed no-slot terminal facts"), + ) + .expect("should serialize the private transport"); + assert_eq!( + value["evidence"]["source"], "initial_navigation_ssat", + "should retain the observed source" + ); + assert_eq!( + value["evidence"]["terminal_status"], "skipped", + "should not require a telemetry observation for a skip" + ); + assert_eq!( + value["evidence"]["terminal_reason"], "no_eligible_slots", + "should preserve the definitive empty slot list" + ); + Some( + value["evidence"]["diagnostic_auction_id"] + .as_str() + .expect("should preserve auction token") + .to_owned(), + ) } else { - scheduling_consent() + None }; - let mut ec_context = EcContext::new_for_test(Some(identity.clone()), consent); - assert_eq!( - ec_context.ec_allowed(), - !withdrawn, - "should apply reader consent" + let document = String::from_utf8(body_of(response).await) + .expect("should preserve the publisher HTML"); + if let Some(token) = expected_token { + assert!( + document.contains(&token), + "should deliver the directly observed skipped auction even with zero bids" + ); + assert!( + document.contains("s(b,undefined,x)"), + "should supply optional transport to the initial scheduler" + ); + } else { + assert!( + !document.contains("initial_navigation_ssat"), + "should omit inactive transport" + ); + } + } + } + + #[tokio::test] + async fn trace_document_reload_rejects_unexpected_origin_304_without_the_ad_stack() { + for content_type in [None, Some("text/html")] { + let stub = Arc::new(StubHttpClient::new()); + let services = services_for_ip( + Arc::clone(&stub), + Arc::new(MemoryTemplateCache::default()), + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), ); - let captured = Arc::new(Mutex::new(None)); - let mut orchestrator = AuctionOrchestrator::new(settings.auction.clone()); - orchestrator.register_provider(Arc::new(SchedulingCaptureProvider { - captured: Arc::clone(&captured), - http: Arc::clone(&stub), - lookups: Arc::new(AtomicUsize::new(0)), - })); - let orchestrator = Arc::new(orchestrator); - let cookies = format!("ab_bucket=A; ts-ec={identity}"); - let response = handle_publisher_request( - &settings, + let mut headers = vec![("etag", "\"cached-document\"")]; + if let Some(content_type) = content_type { + headers.push(("content-type", content_type)); + } + stub.push_response_with_headers(304, Vec::new(), headers); + + let response = run_frozen( + &settings_without_ad_stack(), &services, - Some(&graph), - &mut ec_context, - AuctionDispatch { - orchestrator: &orchestrator, - slots: &[article_slot()], - registry: Some(&partners), - }, - cookie_policy_request(&[cookies.as_bytes()]), - EdgeCacheHeader::SMaxageFallback, + conditional_trace_request(true), ) - .await - .expect("should serve a reader with an existing EC"); + .await; + + assert_eq!( + response.status(), + StatusCode::BAD_GATEWAY, + "should reject an origin that cannot deliver a fresh diagnostics document" + ); + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should keep the terminal error private" + ); assert!( - ec_context.kv_snapshot().entry_for(identity).is_some(), - "should preload this reader's identity even during withdrawal on a hit" + !response.headers().contains_key(header::ETAG), + "should not preserve stale document validators" ); - let mut response = finalize_test_publisher_response( - response, + assert!( + !String::from_utf8(body_of(response).await) + .expect("should return a fixed UTF-8 error") + .contains("__tsjs_trace_"), + "should not fabricate a publisher capture from an empty response" + ); + } + } + + #[tokio::test] + async fn trace_document_active_readers_bypass_warm_template_and_keep_distinct_contexts() + { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut settings = settings_with_mode("esi"); + settings + .creative_opportunities + .as_mut() + .expect("should configure shared assembly") + .origin_is_cookie_independent = Some(true); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should configure document trace"); + let settings = Arc::new(settings); + let first = services_for_ip( + Arc::clone(&stub), + Arc::clone(&cache), + IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)), + ); + let second = services_for_ip( + Arc::clone(&stub), + Arc::clone(&cache), + IpAddr::V4(Ipv4Addr::new(198, 51, 100, 129)), + ); + for _ in 0..3 { + queue_shareable_html(&stub); + } + let warm = String::from_utf8( + body_of(run_frozen(&settings, &first, navigation_request()).await).await, + ) + .expect("should warm a neutral shared document"); + assert!( + !warm.contains("__tsjs_trace_"), + "should leave inactive shared documents without trace globals" + ); + for (services, mask, other_mask, raw_ip) in [ + (&first, "192.0.2.0/24", "198.51.100.0/24", "192.0.2.99"), + (&second, "198.51.100.0/24", "192.0.2.0/24", "198.51.100.129"), + ] { + let response = run_frozen( &settings, - &services, - ®istry, - orchestrator, - finalizer, + services, + navigation_request_with_cookie("__Host-ts-console=1"), ) .await; - crate::ec::finalize::ec_finalize_response( - &settings, - &mut ec_context, - Some(&graph), - &partners, - None, - None, - &mut response, - ); assert_eq!( - response.headers()[HEADER_X_TS_TEMPLATE_CACHE], - if index == 0 { "miss-stored" } else { "hit" }, - "should share one variant across identities and withdrawal" + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should preserve per-reader terminal privacy" ); - let cache_control = response.headers()[header::CACHE_CONTROL] - .to_str() - .expect("should decode cache policy"); - for directive in ["private", "no-store"] { - assert!( - cache_control - .split(',') - .any(|value| value.trim() == directive), - "should keep finalized reader responses private and uncacheable" - ); - } - assert_eq!( + assert!( response - .headers() - .get_all(header::SET_COOKIE) - .iter() - .any(|value| { - let value = value.to_str().expect("should decode response cookie"); - value.starts_with("ts-ec=") && value.contains("Max-Age=0") - }), - withdrawn, - "should expire the EC cookie only for the withdrawing reader" + .extensions() + .get::() + .is_some(), + "should retain existing publisher terminal guard" ); - let body = body_of(response).await; - assert!(!body.is_empty(), "should render a complete reader response"); - let captured = captured.lock().expect("should lock captured auction"); - let auction = captured.as_ref().expect("should dispatch a reader auction"); + let document = String::from_utf8(body_of(response).await) + .expect("should produce active reader document"); assert_eq!( - auction.request.user.id.as_deref(), - if withdrawn { - None - } else { - Some(identity.as_str()) - }, - "should use this reader's identity and suppress it after withdrawal" + document.matches("window.__tsjs_trace_active=true").count(), + 1, + "should inject once per active origin response" + ); + assert!( + document.contains(mask) + && !document.contains(other_mask) + && !document.contains(raw_ip), + "should project only this reader's masked trusted facts" ); - if withdrawn { - assert!( - auction.request.user.eids.is_none(), - "should suppress withdrawn EIDs" - ); - } else { - let eids = auction - .request - .user - .eids - .as_ref() - .expect("should include consenting reader EIDs"); - assert_eq!(eids.len(), 1, "should expose only the configured partner"); - assert_eq!( - eids[0].source, "example.com", - "should use the registered source" - ); - assert_eq!( - eids[0].uids[0].id, - format!("partner-reader-{index}"), - "should use this reader's partner identity on cold and warm requests" - ); - } } + let inactive = String::from_utf8( + body_of(run_frozen(&settings, &second, navigation_request()).await).await, + ) + .expect("should reuse neutral cached document"); + assert!( + !inactive.contains("__tsjs_trace_") && !inactive.contains("198.51.100.0/24"), + "should never replay an active reader's context from shared storage" + ); assert_eq!( stub.recorded_request_uris().len(), - 1, - "should fetch origin only once" + 3, + "should fetch origin for both active readers and use cached bytes only for inactive requests" ); assert_eq!( - looked_up_cache_keys(&cache).len(), - 3, - "should look up every reader" + cache + .lookups + .lock() + .expect("should lock cache lookup record") + .len(), + 2, + "should bypass shared cache lookup entirely for active trace documents" ); + let entries = cache + .entries + .lock() + .expect("should lock stored shared entries"); assert_eq!( - stored_cache_keys(&cache).len(), + entries.len(), 1, - "should store only the cold template" + "should retain only the original neutral shared template" ); - for (index, identity) in identities.iter().enumerate() { - let (entry, _) = graph - .get(identity) - .expect("should read reader identity") - .expect("should retain the live row or its tombstone"); - assert_eq!( - entry.consent.ok, - index == 0, - "should revoke only the second reader" - ); - assert_eq!( - entry.ids.is_empty(), - index == 1, - "should clear only revoked partner IDs" - ); - } - let entries = cache.entries.lock().expect("should lock cached templates"); - let stored = &entries - .values() - .next() - .expect("should retain shared template") - .body; - let stored = String::from_utf8_lossy(stored); - for identity in &identities { + for entry in entries.values() { + let shared = String::from_utf8_lossy(&entry.body); assert!( - !stored.contains(identity), - "should keep reader identities out of cached bytes" + !shared.contains("__tsjs_trace_") + && !shared.contains("192.0.2.0/24") + && !shared.contains("198.51.100.0/24"), + "should never store trace globals or either reader's context in shared template bytes" ); } } } - #[tokio::test] - async fn by_default_a_cookie_bearing_request_uses_no_shared_cache() { - // The shipped default, and the reason the cache is nearly inert on real - // traffic: TS sets its own identity cookie, so essentially every repeat - // visitor arrives carrying one and is excluded in both directions. - let stub = Arc::new(StubHttpClient::new()); - let cache = Arc::new(MemoryTemplateCache::default()); - let settings = Arc::new(settings_with_mode("esi")); - let services = services(Arc::clone(&stub), Arc::clone(&cache)); - queue_shareable_html(&stub); - queue_shareable_html(&stub); - - let _ = run(&settings, &services, cookie_navigation_request()).await; - let _ = run(&settings, &services, cookie_navigation_request()).await; - - assert_eq!( - stub.recorded_request_uris().len(), - 2, - "both requests must reach the origin" - ); - assert!( - cache - .entries - .lock() - .expect("should lock entries") - .is_empty(), - "and neither may store a template" - ); - } - - #[tokio::test] - async fn a_declared_cookie_independent_origin_lets_repeat_visitors_share() { - // The opt-in. Without it the cache can only ever serve first-ever page - // views, which is not the population the issue cares about. - let stub = Arc::new(StubHttpClient::new()); - let cache = Arc::new(MemoryTemplateCache::default()); - let mut raw = settings_with_mode("esi"); - raw.creative_opportunities - .as_mut() - .expect("fixture configures creative opportunities") - .origin_is_cookie_independent = Some(true); - let settings = Arc::new(raw); - let services = services(Arc::clone(&stub), Arc::clone(&cache)); - queue_shareable_html(&stub); - queue_shareable_html(&stub); - - let _ = run(&settings, &services, cookie_navigation_request()).await; - let _ = run(&settings, &services, cookie_navigation_request()).await; - - assert_eq!( - stub.recorded_request_uris().len(), - 1, - "the second cookie-bearing request should be served from the cache" - ); - } - - #[tokio::test] - async fn an_active_diagnostics_request_never_stores_a_template() { - // An independent review reintroduced a diagnostics leak scoped to - // A request-private diagnostics mutation could otherwise leak through a - // shared template. `requires_private_no_store()` is a - // strict superset of the condition under which diagnostics markup is - // emitted, and that stamp lands *before* the template cache gate reads response headers, - // so such a request never stores a template at all. - // - // That is a coincidence between two independent conditions, and the whole - // protection rests on it. This pins the consequence directly, so the - // relationship is checked rather than merely reasoned about. - let stub = Arc::new(StubHttpClient::new()); - let cache = Arc::new(MemoryTemplateCache::default()); - let settings = Arc::new(settings_with_mode("esi")); - let services = services(Arc::clone(&stub), Arc::clone(&cache)); - queue_shareable_html(&stub); - - let _ = run(&settings, &services, diagnostics_navigation_request()).await; - - assert!( - cache - .entries - .lock() - .expect("should lock entries") - .is_empty(), - "a reader running diagnostics must not contribute to a shared cache" - ); - } - - #[tokio::test] - async fn active_diagnostics_bypass_an_already_warm_template() { - let stub = Arc::new(StubHttpClient::new()); - let cache = Arc::new(MemoryTemplateCache::default()); - let settings = Arc::new(settings_with_mode("esi")); - let services = services(Arc::clone(&stub), Arc::clone(&cache)); - queue_shareable_html(&stub); - queue_shareable_html(&stub); - - let _ = body_of(run(&settings, &services, navigation_request()).await).await; - let response = run(&settings, &services, diagnostics_navigation_request()).await; - let document = String::from_utf8(body_of(response).await) - .expect("diagnostics document should be UTF-8"); - - assert_eq!( - stub.recorded_request_uris().len(), - 2, - "request-private diagnostics must reach the origin even when an ordinary \ - shared template is warm" - ); - assert_eq!( - cache.lookups.lock().expect("should lock lookups").len(), - 1, - "the diagnostics request must not consult template cache at all" - ); - assert!( - document.contains("__tsjs_gpt_diagnostics_active"), - "origin fallback must retain the request-private diagnostics bootstrap" - ); - } - #[tokio::test] async fn datadome_suppressed_request_bypasses_a_warm_shared_template() { let stub = Arc::new(StubHttpClient::new()); @@ -18168,6 +19769,81 @@ mod tests { ); } + mod trace_auction_publisher_terminal_tests { + use super::*; + use crate::trace::{ + TraceAuctionCarry, TraceAuctionSource, TraceAuctionTerminalStatus, TraceProviderRole, + }; + + fn pending() -> (DispatchedAuction, TraceAuctionCarry) { + let request = test_auction_request(); + let trace = TraceAuctionCarry::capture_if_enabled( + true, + TraceAuctionSource::InitialNavigationSsat, + &request.slots, + ) + .expect("should capture the enabled publisher auction"); + let _observation = trace.launch_provider(TraceProviderRole::Bidder); + ( + DispatchedAuction::empty_for_test(request, 10).with_trace_for_test(trace.clone()), + trace, + ) + } + + fn status(trace: &TraceAuctionCarry) -> serde_json::Value { + serde_json::to_value( + trace + .transport() + .expect("should terminalize before telemetry"), + ) + .expect("should serialize the bounded transport")["evidence"]["terminal_status"] + .clone() + } + + #[tokio::test] + async fn trace_auction_abandons_without_a_telemetry_observation() { + let (dispatched, trace) = pending(); + emit_abandoned_auction( + &crate::platform::test_support::noop_services(), + None, + dispatched, + "example_disconnection", + ) + .await; + assert_eq!( + status(&trace), + "abandoned", + "should terminalize independently of telemetry construction" + ); + } + + #[test] + fn trace_auction_publisher_guard_preserves_a_clone_after_take() { + let (dispatched, trace) = pending(); + let mut guard = DispatchedAuctionGuard::new(dispatched); + let _taken = guard.take().expect("should begin collection"); + drop(guard); + assert_eq!( + status(&trace), + "abandoned", + "should synchronously terminalize cancellation during collection" + ); + } + + #[test] + fn trace_auction_publisher_drop_cannot_overwrite_completed_facts() { + let (dispatched, trace) = pending(); + let guard = DispatchedAuctionGuard::new(dispatched); + trace.finish(TraceAuctionTerminalStatus::Completed, None); + drop(guard); + assert_eq!( + status(&trace), + "completed", + "should preserve a completed terminal observation" + ); + } + } + fn response_body_string(response: http::Response) -> String { String::from_utf8( response @@ -19463,6 +21139,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; @@ -19522,6 +21199,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; @@ -19570,6 +21248,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let body = EdgeBody::from_stream(futures::stream::iter(vec![Ok::<_, io::Error>( @@ -19696,6 +21375,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let body = EdgeBody::stream(futures::stream::iter(vec![ @@ -19760,6 +21440,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let compressed = @@ -19827,6 +21508,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let compressed = @@ -19894,6 +21576,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let compressed = @@ -19961,6 +21644,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let compressed = @@ -20010,6 +21694,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -20233,6 +21918,7 @@ mod tests { )), price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let body = EdgeBody::stream(futures::stream::iter(vec![ @@ -20308,6 +21994,7 @@ mod tests { )), price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; // The `` that triggers bid injection lives in the SECOND gzip @@ -20382,6 +22069,7 @@ mod tests { )), price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let body = EdgeBody::stream(futures::stream::iter(vec![bytes::Bytes::from_static( @@ -20450,6 +22138,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let publisher_response = PublisherResponse::Stream { @@ -20606,6 +22295,7 @@ mod tests { dispatched_auction, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } } @@ -21294,6 +22984,7 @@ mod tests { )), price_granularity: PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, } }; @@ -21485,6 +23176,7 @@ mod tests { )), price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let publisher_response = PublisherResponse::Stream { @@ -21563,6 +23255,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut output = Vec::new(); @@ -21624,6 +23317,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; @@ -21742,6 +23436,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; let mut output = Vec::new(); @@ -21809,6 +23504,7 @@ mod tests { dispatched_auction: None, price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, + trace_bootstrap: None, suppress_datadome_client_side_tag: false, }; @@ -23671,6 +25367,198 @@ mod tests { } } + #[tokio::test] + async fn trace_transport_spa_aliases_bind_exact_slots_and_reuse_pre_dispatch_token() { + for endpoint in [PAGE_BIDS_PATH, PAGE_BIDS_LEGACY_PATH] { + for winning_bid in [false, true] { + let config = format!( + "{}\n[auction]\nenabled=true\n[auction.providers.bidder]\nprotocol=\"openrtb-2.6\"\nprofile=\"standard\"\nendpoint=\"https://bidder.example.com/auction\"\nrouting=\"all_eligible\"\n[creative_opportunities]\ngam_network_id=\"12345\"\n", + crate_test_settings_str() + ); + let mut settings = + Settings::from_toml(&config).expect("should configure production bidder"); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable active trace"); + let settings = Arc::new(settings); + let plan = Arc::new( + crate::auction::compile_auction_plan(&settings) + .expect("should compile actual provider plan"), + ); + let orchestrator = AuctionOrchestrator::from_plan(plan, None); + let http = Arc::new(StubHttpClient::new()); + if winning_bid { + http.push_response(200, serde_json::to_vec(&serde_json::json!({"seatbid":[{"seat":"bidder","bid":[{"id":"bid","impid":"atf","price":1.5,"w":300,"h":250,"adm":"
Example creative
"}]}]})).expect("should encode actual winning response")); + } else { + http.push_response(204, Vec::new()); + } + let services = build_services_with_http_client(Arc::clone(&http) as Arc<_>); + let mut request = make_page_bids_request_on(endpoint, "/2024/article"); + set_test_header(&mut request, "cookie", "__Host-ts-console=1"); + let hook = crate::trace::TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load SPA setup metadata")), + ); + edgezero_core::router::PreDispatchHook::handle(&hook, &mut request) + .await + .expect("should freeze SPA gate"); + let mut ec_context = consent_allowing_ec_context(); + let slots = article_slot(); + let response = handle_page_bids( + &settings, + &services, + None, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &slots, + registry: None, + }, + &mut ec_context, + request, + ) + .await + .expect("should execute SPA auction"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should keep both aliases private" + ); + let carry = response + .extensions() + .get::() + .expect("should retain request carry"); + let token = carry.token().to_string(); + let value: serde_json::Value = serde_json::from_slice( + &response + .into_body() + .into_bytes() + .expect("should collect SPA JSON"), + ) + .expect("should decode JSON"); + let evidence = &value["trace_auction"]["evidence"]; + assert_eq!( + evidence["diagnostic_auction_id"], token, + "should reuse original token" + ); + assert_eq!( + evidence["source"], "spa_page_bids", + "should preserve actual source" + ); + assert_eq!( + evidence["terminal_status"], "completed", + "should preserve completed zero bids and winners" + ); + assert_eq!( + evidence["slots"][0]["slot_ref"], + value["slots"][0]["ext"]["trusted_server"]["trace_slot_ref"], + "should bind exact emitted slot occurrence" + ); + assert_eq!( + evidence["slots"][0]["candidate"], + if winning_bid { + "selected" + } else { + "no_candidate" + }, + "should project actual delivered map" + ); + if winning_bid { + assert_eq!( + value["bids"]["atf"]["hb_auction_id"], token, + "should share opportunity and evidence token" + ); + } + assert_eq!( + http.recorded_request_bodies().len(), + 1, + "should execute one real provider launch" + ); + let source = String::from_utf8(http.recorded_request_bodies()[0].clone()) + .expect("should serialize ordinary provider JSON"); + assert!( + !source.contains("trace_slot_ref") && !source.contains(&token), + "should keep trace carry out of provider request" + ); + } + } + } + + #[tokio::test] + async fn trace_transport_spa_frozen_gate_delivers_empty_slots_without_telemetry() { + for active in [false, true] { + let mut settings = settings_with_co(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .expect("should enable request-local trace evidence"); + let settings = Arc::new(settings); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let mut request = make_page_bids_request("/2024/article"); + if active { + set_test_header(&mut request, "cookie", "__Host-ts-console=1"); + } + let hook = crate::trace::TracePreDispatchHook::new( + Arc::clone(&settings), + Arc::new(|_| panic!("should not load setup metadata on SPA capture")), + ); + assert!( + edgezero_core::router::PreDispatchHook::handle(&hook, &mut request) + .await + .expect("should freeze SPA cookies") + .is_none(), + "should keep ordinary SPA routing" + ); + + let response = run_page_bids_response(&settings, &orchestrator, &[], request).await; + + let carry = response + .extensions() + .get::(); + assert_eq!( + carry.is_some(), + active, + "should capture only the frozen valid session" + ); + let expected = if let Some(carry) = carry { + let facts = serde_json::to_value( + carry.transport().expect("should finish without dispatch"), + ) + .expect("should serialize bounded facts"); + assert_eq!( + facts["evidence"]["source"], "spa_page_bids", + "should preserve the SPA source" + ); + assert_eq!( + facts["evidence"]["terminal_reason"], "no_eligible_slots", + "should preserve the definitive empty slot list" + ); + Some(facts) + } else { + None + }; + let bytes = response + .into_body() + .into_bytes() + .expect("should keep ordinary JSON"); + let value: serde_json::Value = + serde_json::from_slice(&bytes).expect("should decode SPA envelope"); + assert_eq!( + value.get("trace_auction"), + expected.as_ref(), + "should deliver checked transport only for the frozen base gate" + ); + } + } + #[tokio::test] async fn page_bids_format_rejects_removed_unknown_and_empty_values() { let settings = settings_with_co(); diff --git a/crates/trusted-server-core/src/trace/actions.rs b/crates/trusted-server-core/src/trace/actions.rs new file mode 100644 index 000000000..e25b3533e --- /dev/null +++ b/crates/trusted-server-core/src/trace/actions.rs @@ -0,0 +1,921 @@ +//! Deliberate, authenticated trace cookie mutations. + +use edgezero_core::body::Body as EdgeBody; +use edgezero_core::request::RequestIngress; +use futures::StreamExt as _; +use http::{HeaderMap, HeaderName, HeaderValue, Request, Response, StatusCode, header}; +use url::Url; + +use crate::integrations::gpt_diagnostics::GptDiagnosticsCookieAction; + +use super::routes::TraceRoute; + +const ACTION_HEADER: HeaderName = HeaderName::from_static("x-ts-trace-action"); +const FETCH_SITE_HEADER: HeaderName = HeaderName::from_static("sec-fetch-site"); + +/// Validate one deliberate mutation after authenticated route preflight. +/// +/// The caller must apply [`super::TraceDispatch::respond`] to every result. +/// Cookie health never prevents a valid enable or end request. +/// +/// # Performance +/// +/// Rejected controls and framing never poll the body. Streaming validation +/// skips empty chunks and stops on the first nonempty chunk or read error; +/// it does not allocate a body buffer or introduce a transport deadline. +pub(crate) async fn action_response( + request: &mut Request, + route: TraceRoute, +) -> Response { + let (action, cookie_action) = match route { + TraceRoute::Enable => (b"enable".as_slice(), GptDiagnosticsCookieAction::SetSession), + TraceRoute::End => (b"end".as_slice(), GptDiagnosticsCookieAction::ClearSession), + _ => return failure(StatusCode::INTERNAL_SERVER_ERROR), + }; + if request.uri().query().is_some() + || single_header(request.headers(), ACTION_HEADER).map(HeaderValue::as_bytes) + != Some(action) + || single_header(request.headers(), FETCH_SITE_HEADER).map(HeaderValue::as_bytes) + != Some(b"same-origin".as_slice()) + || !matches_trusted_origin(request) + { + return failure(StatusCode::FORBIDDEN); + } + if !accepts_empty_framing(request.headers()) { + return failure(StatusCode::PAYLOAD_TOO_LARGE); + } + let body = std::mem::replace(request.body_mut(), EdgeBody::empty()); + if let Some(status) = empty_body_rejection(body).await { + return failure(status); + } + let Some(cookie) = cookie_action.set_cookie_header() else { + return failure(StatusCode::INTERNAL_SERVER_ERROR); + }; + let mut response = Response::new(EdgeBody::from(br#"{"mutation_requested":true}"#.as_slice())); + response.headers_mut().insert(header::SET_COOKIE, cookie); + response +} + +fn failure(status: StatusCode) -> Response { + let body: &'static [u8] = match status { + StatusCode::FORBIDDEN => br#"{"error":"trace action rejected"}"#, + StatusCode::PAYLOAD_TOO_LARGE => br#"{"error":"trace body must be empty"}"#, + StatusCode::BAD_REQUEST => br#"{"error":"trace body unavailable"}"#, + _ => br#"{"error":"trace action unavailable"}"#, + }; + let mut response = Response::new(EdgeBody::from(body)); + *response.status_mut() = status; + response +} + +fn single_header(headers: &HeaderMap, name: HeaderName) -> Option<&HeaderValue> { + let mut fields = headers.get_all(name).iter(); + let value = fields.next()?; + fields.next().is_none().then_some(value) +} + +fn matches_trusted_origin(request: &Request) -> bool { + let Some(trusted) = request + .extensions() + .get::() + .and_then(RequestIngress::origin) + else { + return false; + }; + let Some(expected) = canonical_authority_origin(trusted.scheme(), trusted.authority()) else { + return false; + }; + let Some(origin) = single_header(request.headers(), header::ORIGIN) + .and_then(|origin| core::str::from_utf8(origin.as_bytes()).ok()) + .and_then(canonical_origin) + else { + return false; + }; + if origin != expected + || request + .uri() + .scheme_str() + .is_some_and(|scheme| !scheme.eq_ignore_ascii_case(trusted.scheme())) + { + return false; + } + let authority = request.uri().authority(); + if authority.is_some_and(|authority| { + canonical_authority_origin(trusted.scheme(), authority.as_str()).as_ref() != Some(&expected) + }) { + return false; + } + match single_header(request.headers(), header::HOST) { + Some(host) => { + core::str::from_utf8(host.as_bytes()) + .ok() + .and_then(|host| canonical_authority_origin(trusted.scheme(), host)) + .as_ref() + == Some(&expected) + } + None if request.headers().contains_key(header::HOST) => false, + None => request.uri().scheme().is_some() && authority.is_some(), + } +} + +fn canonical_origin(value: &str) -> Option { + let (scheme, authority) = value.split_once("://")?; + canonical_authority_origin(scheme, authority) +} + +fn canonical_authority_origin(scheme: &str, authority: &str) -> Option { + if !(scheme.eq_ignore_ascii_case("https") || scheme.eq_ignore_ascii_case("http")) + || authority.is_empty() + || authority.chars().any(|character| { + character.is_whitespace() + || character.is_control() + || matches!(character, '/' | '?' | '#' | '\\' | '@' | ',') + }) + || !valid_port(authority) + { + return None; + } + let url = Url::parse(&format!("{scheme}://{authority}")).ok()?; + if url.host_str().is_none_or(str::is_empty) { + return None; + } + Some(url.origin().ascii_serialization()) +} + +fn valid_port(authority: &str) -> bool { + let port = if authority.starts_with('[') { + let Some(end) = authority.find(']') else { + return false; + }; + let suffix = &authority[end + 1..]; + if suffix.is_empty() { + return true; + } + let Some(port) = suffix.strip_prefix(':') else { + return false; + }; + Some(port) + } else { + authority.split_once(':').map(|(_, port)| port) + }; + port.is_none_or(|port| { + !port.is_empty() + && port.bytes().all(|byte| byte.is_ascii_digit()) + && port.parse::().is_ok() + }) +} + +fn accepts_empty_framing(headers: &HeaderMap) -> bool { + if headers.contains_key(header::TRANSFER_ENCODING) { + return false; + } + let mut lengths = headers.get_all(header::CONTENT_LENGTH).iter(); + let Some(length) = lengths.next() else { + return true; + }; + lengths.next().is_none() + && !length.as_bytes().is_empty() + && length.as_bytes().iter().all(|byte| *byte == b'0') +} + +async fn empty_body_rejection(body: EdgeBody) -> Option { + match body { + EdgeBody::Once(bytes) if bytes.is_empty() => None, + EdgeBody::Once(_) => Some(StatusCode::PAYLOAD_TOO_LARGE), + EdgeBody::Stream(mut stream) => { + while let Some(chunk) = stream.next().await { + match chunk { + Ok(bytes) if bytes.is_empty() => {} + Ok(_) => return Some(StatusCode::PAYLOAD_TOO_LARGE), + Err(_) => return Some(StatusCode::BAD_REQUEST), + } + } + None + } + } +} + +#[cfg(test)] +mod tests { + use std::cell::Cell; + use std::io; + use std::rc::Rc; + use std::task::Poll; + + use base64::{Engine as _, engine::general_purpose::STANDARD}; + use bytes::Bytes; + use edgezero_core::body::Body as EdgeBody; + use edgezero_core::request::{ + CapturedTarget, HeaderFidelity, InboundOrigin, OriginSource, RequestIngress, + TargetUnavailable, + }; + use futures::{executor::block_on, stream}; + use http::{HeaderName, HeaderValue, Method, Request, Response, StatusCode, header}; + use serde_json::{Value, json}; + + use crate::settings::{Handler, Settings}; + use crate::test_support::tests::create_test_settings; + use crate::trace::{TracePreflight, TraceTerminalResponse, inspect_cookies, preflight}; + + fn settings(enabled: bool, auth: bool) -> Settings { + let mut settings = create_test_settings(); + settings.integrations.insert( + "gpt_diagnostics".to_owned(), + json!({"enabled":true, "trace_page_enabled":enabled}), + ); + if auth { + let handler: Handler = serde_json::from_value( + json!({"path":"^/", "username":"example-user", "password":"example-password"}), + ) + .expect("should create an example authentication rule"); + settings.handlers.insert(0, handler); + } + settings + } + + fn metadata(scheme: &str, authority: &str) -> RequestIngress { + RequestIngress::new( + CapturedTarget::Unavailable(TargetUnavailable::NotExposed), + Some( + InboundOrigin::parse(scheme, authority, OriginSource::RuntimeUri) + .expect("should validate the trusted fictional origin"), + ), + HeaderFidelity::default(), + vec![], + ) + .expect("should construct origin-only ingress metadata") + } + + fn request(action: &str) -> Request { + let mut request = Request::builder() + .method(Method::POST) + .uri(format!("/_ts/trace/{action}")) + .header(header::HOST, "publisher.example") + .header(header::ORIGIN, "https://publisher.example") + .header("sec-fetch-site", "same-origin") + .header("x-ts-trace-action", action) + .body(EdgeBody::empty()) + .expect("should create a deliberate action request"); + request + .extensions_mut() + .insert(metadata("https", "publisher.example")); + request + } + + fn respond(settings: &Settings, mut request: Request) -> Response { + match preflight(settings, &mut request) { + TracePreflight::Response(response) => response, + TracePreflight::Ready(context) => { + let response = block_on(super::super::action_response(&mut request, context.route)); + context.respond(response) + } + TracePreflight::NotTrace => panic!("should preflight every exact action route locally"), + } + } + + fn private_error(response: Response, expected: StatusCode) { + assert_eq!( + response.status(), + expected, + "should return the exact bounded action failure status" + ); + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should harden action errors locally" + ); + assert_eq!( + response.headers()[header::CONTENT_TYPE], + "application/json; charset=utf-8", + "should use bounded JSON errors" + ); + assert_eq!( + response.headers()[header::X_CONTENT_TYPE_OPTIONS], + "nosniff", + "should preserve content type hardening" + ); + assert_eq!( + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .count(), + 0, + "should never mutate cookies on rejection" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should bypass ordinary lifecycle finalization" + ); + let body = response + .into_body() + .into_bytes() + .expect("should return a bounded buffered error"); + assert!( + body.len() < 128 && !String::from_utf8_lossy(&body).contains("fictional-secret"), + "should expose no controls, stream errors or parser text" + ); + } + + fn unpolled_body(polls: Rc>) -> EdgeBody { + EdgeBody::stream(stream::poll_fn(move |_| -> Poll> { + polls.set(polls.get() + 1); + panic!("should reject unsafe headers before polling a body"); + })) + } + + fn assert_mutation(response: Response, action: &str) { + assert_eq!( + response.status(), + StatusCode::OK, + "should accept only the complete deliberate action" + ); + let values: Vec<_> = response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .collect(); + assert_eq!( + values.len(), + 1, + "should request exactly one host-only cookie mutation" + ); + let expected = if action == "enable" { + "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800" + } else { + "__Host-ts-console=; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=0" + }; + assert_eq!( + values[0], expected, + "should reuse the exact shared cookie lifetime and scope" + ); + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should keep mutation responses private" + ); + let value: Value = response + .into_body() + .to_json() + .expect("should parse a bounded mutation result"); + assert_eq!( + value, + json!({"mutation_requested":true}), + "should claim only the requested mutation" + ); + } + + #[test] + fn trace_actions_accept_enable_and_end_independently_of_cookie_health() { + for action in ["enable", "end"] { + for field in [ + None, + Some("__Host-ts-console=invalid-example"), + Some("__Host-ts-console=1; __Host-ts-console=1"), + Some("__Host-ts-console=1; unrelated=a,b"), + Some("__Host-ts-console=1; unrelated=�"), + ] { + let mut request = request(action); + if let Some(field) = field { + request.headers_mut().insert( + header::COOKIE, + HeaderValue::from_bytes(field.as_bytes()) + .expect("should construct runtime-visible Cookie text"), + ); + } + let frozen = inspect_cookies( + request.headers(), + request.extensions().get::(), + ); + assert!( + !frozen.observed_active(), + "should keep invalid or ambiguous follow-up observation inactive" + ); + assert_mutation(respond(&settings(true, false), request), action); + for _ in 0..2 { + let state = super::super::state_response(&frozen); + assert_eq!( + state + .into_body() + .to_json::() + .expect("should decode the separate observation"), + json!({"observed_active":false}), + "should not infer browser activation from a successful mutation request" + ); + } + } + } + } + + #[test] + fn trace_actions_origin_canonicalization_accepts_case_default_ports_and_ipv6() { + for (scheme, authority, origin, host) in [ + ( + "https", + "publisher.example", + "HTTPS://PUBLISHER.EXAMPLE:443", + "PUBLISHER.EXAMPLE:443", + ), + ( + "http", + "publisher.example", + "http://PUBLISHER.EXAMPLE:80", + "publisher.example:80", + ), + ( + "https", + "publisher.example:8443", + "https://PUBLISHER.EXAMPLE:8443", + "publisher.example:8443", + ), + ( + "https", + "[2001:db8::1]", + "https://[2001:0DB8:0:0:0:0:0:1]:443", + "[2001:DB8::1]:443", + ), + ] { + let mut request = request("enable"); + request.extensions_mut().insert(metadata(scheme, authority)); + request.headers_mut().insert( + header::ORIGIN, + HeaderValue::from_str(origin) + .expect("should construct a canonical-equivalent origin"), + ); + request.headers_mut().insert( + header::HOST, + HeaderValue::from_str(host).expect("should construct a canonical-equivalent host"), + ); + assert_mutation(respond(&settings(true, false), request), "enable"); + } + } + + #[test] + fn trace_actions_origin_syntax_is_rejected_before_url_normalization_and_body_polling() { + for origin in [ + "https://publisher.example/", + "https://publisher.example/path", + "https://publisher.example?", + "https://publisher.example?fictional-secret=1", + "https://publisher.example#", + "https://publisher.example#fictional-secret", + "https://@publisher.example", + "https://example-user@publisher.example", + "https://publisher.example\\", + "https://publisher.example,https://publisher.example", + " https://publisher.example", + "https://publisher.example ", + "https://publisher.example\t", + "https://publisher.example:", + "https://publisher.example:+443", + "https://publisher.example:65536", + "null", + "file://publisher.example", + "https://", + "https:///publisher.example", + "https:publisher.example", + "http://publisher.example", + "https://other.example", + "https://publisher.example:8443", + ] { + let polls = Rc::new(Cell::new(0)); + let mut request = request("enable"); + request.headers_mut().insert( + header::ORIGIN, + HeaderValue::from_str(origin) + .expect("should construct the visible malformed origin"), + ); + *request.body_mut() = unpolled_body(Rc::clone(&polls)); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + assert_eq!( + polls.get(), + 0, + "should reject the entire Origin value before body inspection" + ); + } + } + + #[test] + fn trace_actions_controls_require_exact_single_values_and_no_query() { + for name in ["origin", "sec-fetch-site", "x-ts-trace-action"] { + let mut request = request("enable"); + request.headers_mut().remove(name); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + for second in [ + if name == "origin" { + "https://publisher.example" + } else if name == "sec-fetch-site" { + "same-origin" + } else { + "enable" + }, + "fictional-secret-conflict", + ] { + let mut request = self::request("enable"); + request.headers_mut().append( + HeaderName::from_bytes(name.as_bytes()) + .expect("should construct the control name"), + HeaderValue::from_str(second).expect("should construct a repeated control"), + ); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + } + for (name, value) in [ + ("sec-fetch-site", "same-site"), + ("sec-fetch-site", "Same-Origin"), + ("sec-fetch-site", "same-origin, same-origin"), + ("sec-fetch-site", " same-origin"), + ("x-ts-trace-action", "end"), + ("x-ts-trace-action", "Enable"), + ("x-ts-trace-action", "enable, enable"), + ("x-ts-trace-action", "enable "), + ] { + let mut request = request("enable"); + request.headers_mut().insert( + HeaderName::from_bytes(name.as_bytes()).expect("should construct the control name"), + HeaderValue::from_str(value).expect("should construct a visible control value"), + ); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + for target in [ + "/_ts/trace/enable?", + "/_ts/trace/enable?ts_console=1", + "/_ts/trace/enable?fictional-secret=1", + ] { + let mut request = request("enable"); + *request.uri_mut() = target.parse().expect("should retain the visible query"); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + } + + #[test] + fn trace_actions_host_is_single_unfolded_valid_and_matches_trusted_authority() { + for value in [ + "other.example", + "publisher.example:8443", + "publisher.example:", + "publisher.example:+443", + "publisher.example:65536", + "publisher.example, publisher.example", + "publisher.example/", + "publisher.example?", + "publisher.example#", + "@publisher.example", + "publisher.example\\", + " publisher.example", + "publisher.example ", + "", + ] { + let mut request = request("enable"); + request.headers_mut().insert( + header::HOST, + HeaderValue::from_str(value).expect("should construct the visible malformed Host"), + ); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + for value in ["publisher.example", "other.example"] { + let mut request = request("enable"); + request.headers_mut().append( + header::HOST, + HeaderValue::from_str(value).expect("should construct the second Host"), + ); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + let mut request = request("enable"); + request.headers_mut().remove(header::HOST); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + + #[test] + fn trace_actions_absolute_uri_requires_all_present_facts_to_agree() { + for host in [None, Some("PUBLISHER.EXAMPLE:443")] { + let mut request = request("enable"); + *request.uri_mut() = "https://PUBLISHER.EXAMPLE:443/_ts/trace/enable" + .parse() + .expect("should build a matching absolute URI"); + if let Some(host) = host { + request.headers_mut().insert( + header::HOST, + HeaderValue::from_str(host).expect("should construct the matching Host"), + ); + } else { + request.headers_mut().remove(header::HOST); + } + assert_mutation(respond(&settings(true, false), request), "enable"); + } + for target in [ + "http://publisher.example/_ts/trace/enable", + "https://other.example/_ts/trace/enable", + "https://publisher.example:8443/_ts/trace/enable", + "ftp://publisher.example/_ts/trace/enable", + "https://@publisher.example/_ts/trace/enable", + ] { + let mut request = request("enable"); + *request.uri_mut() = target + .parse() + .expect("should construct conflicting visible absolute URI facts"); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + let mut request = request("enable"); + *request.uri_mut() = "https://publisher.example/_ts/trace/enable" + .parse() + .expect("should construct an absolute request"); + request + .headers_mut() + .insert(header::HOST, HeaderValue::from_static("other.example")); + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + + #[test] + fn trace_actions_forwarded_headers_and_runtime_compatibility_cannot_supply_trust() { + for missing_origin in [false, true] { + let mut request = request("enable"); + request.extensions_mut().remove::(); + if !missing_origin { + request.extensions_mut().insert( + RequestIngress::new( + CapturedTarget::Unavailable(TargetUnavailable::NotExposed), + None, + HeaderFidelity::default(), + vec![], + ) + .expect("should construct metadata without origin trust"), + ); + } + for (name, value) in [ + ("forwarded", "proto=https;host=publisher.example"), + ("x-forwarded-proto", "https"), + ("x-forwarded-host", "publisher.example"), + ("x-ts-original-scheme", "https"), + ] { + request.headers_mut().insert( + HeaderName::from_bytes(name.as_bytes()) + .expect("should construct an untrusted compatibility header name"), + HeaderValue::from_static(value), + ); + } + private_error( + respond(&settings(true, false), request), + StatusCode::FORBIDDEN, + ); + } + } + + #[test] + fn trace_actions_header_failures_never_poll_a_hostile_stream() { + for (name, value, status) in [ + ("origin", "https://other.example", StatusCode::FORBIDDEN), + ( + "x-ts-trace-action", + "invalid-example", + StatusCode::FORBIDDEN, + ), + ("sec-fetch-site", "cross-site", StatusCode::FORBIDDEN), + ("host", "other.example", StatusCode::FORBIDDEN), + ("transfer-encoding", "", StatusCode::PAYLOAD_TOO_LARGE), + ( + "transfer-encoding", + "chunked", + StatusCode::PAYLOAD_TOO_LARGE, + ), + ("content-length", "1", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "01", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "+0", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "-0", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "0,0", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", " 0", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "0 ", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "", StatusCode::PAYLOAD_TOO_LARGE), + ("content-length", "zero", StatusCode::PAYLOAD_TOO_LARGE), + ] { + let polls = Rc::new(Cell::new(0)); + let mut request = request("enable"); + request.headers_mut().insert( + HeaderName::from_bytes(name.as_bytes()) + .expect("should construct the visible header name"), + HeaderValue::from_str(value).expect("should construct the malformed visible field"), + ); + *request.body_mut() = unpolled_body(Rc::clone(&polls)); + private_error(respond(&settings(true, false), request), status); + assert_eq!( + polls.get(), + 0, + "should reject headers without polling the stream" + ); + } + for second in ["0", "1"] { + let polls = Rc::new(Cell::new(0)); + let mut request = request("enable"); + request + .headers_mut() + .append(header::CONTENT_LENGTH, HeaderValue::from_static("0")); + request + .headers_mut() + .append(header::CONTENT_LENGTH, HeaderValue::from_static(second)); + *request.body_mut() = unpolled_body(Rc::clone(&polls)); + private_error( + respond(&settings(true, false), request), + StatusCode::PAYLOAD_TOO_LARGE, + ); + assert_eq!( + polls.get(), + 0, + "should reject visible repeated lengths without body polls" + ); + } + } + + #[test] + fn trace_actions_empty_buffered_body_accepts_absent_and_all_zero_lengths() { + for length in [None, Some("0"), Some("00"), Some("000")] { + let mut request = request("end"); + if let Some(length) = length { + request + .headers_mut() + .insert(header::CONTENT_LENGTH, HeaderValue::from_static(length)); + } + assert_mutation(respond(&settings(true, false), request), "end"); + } + } + + #[test] + fn trace_actions_nonempty_buffered_body_rejects_even_with_zero_framing() { + for length in [None, Some("0"), Some("000")] { + let mut request = request("end"); + *request.body_mut() = EdgeBody::from("fictional-secret-body"); + if let Some(length) = length { + request + .headers_mut() + .insert(header::CONTENT_LENGTH, HeaderValue::from_static(length)); + } + private_error( + respond(&settings(true, false), request), + StatusCode::PAYLOAD_TOO_LARGE, + ); + } + } + + #[test] + fn trace_actions_stream_requires_clean_eof_after_empty_chunks() { + let polls = Rc::new(Cell::new(0)); + let captured = Rc::clone(&polls); + let mut request = request("enable"); + *request.body_mut() = EdgeBody::stream(stream::poll_fn(move |_| { + captured.set(captured.get() + 1); + Poll::Ready(if captured.get() <= 2 { + Some(Bytes::new()) + } else { + None + }) + })); + assert_mutation(respond(&settings(true, false), request), "enable"); + assert_eq!( + polls.get(), + 3, + "should skip empty chunks and prove clean EOF" + ); + } + + #[test] + fn trace_actions_stream_stops_on_first_nonempty_chunk_without_later_polling() { + let polls = Rc::new(Cell::new(0)); + let captured = Rc::clone(&polls); + let mut request = request("enable"); + *request.body_mut() = EdgeBody::stream(stream::poll_fn(move |_| { + captured.set(captured.get() + 1); + Poll::Ready(Some(match captured.get() { + 1 => Bytes::new(), + 2 => Bytes::from_static(b"x"), + _ => panic!("should not poll after the first nonempty body chunk"), + })) + })); + private_error( + respond(&settings(true, false), request), + StatusCode::PAYLOAD_TOO_LARGE, + ); + assert_eq!(polls.get(), 2, "should stop at the first real byte"); + } + + #[test] + fn trace_actions_stream_error_is_bounded_bad_request_without_later_polling() { + let polls = Rc::new(Cell::new(0)); + let captured = Rc::clone(&polls); + let mut request = request("end"); + *request.body_mut() = EdgeBody::from_stream(stream::poll_fn( + move |_| -> Poll>> { + captured.set(captured.get() + 1); + Poll::Ready(Some(match captured.get() { + 1 => Ok(Bytes::new()), + 2 => Err(io::Error::other("fictional-secret-stream-error")), + _ => panic!("should not poll after the stream error"), + })) + }, + )); + private_error( + respond(&settings(true, false), request), + StatusCode::BAD_REQUEST, + ); + assert_eq!( + polls.get(), + 2, + "should require successful EOF and stop after error" + ); + } + + #[test] + fn trace_actions_preflight_authentication_precedes_controls_flag_and_body() { + for enabled in [false, true] { + for credentials in [ + None, + Some("example-user:incorrect-example"), + Some("example-user:example-password"), + ] { + let polls = Rc::new(Cell::new(0)); + let mut request = request("enable"); + request.headers_mut().remove(header::ORIGIN); + *request.body_mut() = unpolled_body(Rc::clone(&polls)); + if let Some(credentials) = credentials { + request.headers_mut().insert( + header::AUTHORIZATION, + HeaderValue::from_str(&format!("Basic {}", STANDARD.encode(credentials))) + .expect("should create fictional credentials"), + ); + } + let response = respond(&settings(enabled, true), request); + let expected = if credentials != Some("example-user:example-password") { + StatusCode::UNAUTHORIZED + } else if !enabled { + StatusCode::NOT_FOUND + } else { + StatusCode::FORBIDDEN + }; + assert_eq!( + response.headers().contains_key(header::WWW_AUTHENTICATE), + expected == StatusCode::UNAUTHORIZED, + "should preserve challenges only when auth rejects" + ); + private_error(response, expected); + assert_eq!( + polls.get(), + 0, + "should not read bodies before successful auth and header validation" + ); + } + } + let mut request = request("enable"); + *request.method_mut() = Method::HEAD; + request.headers_mut().remove(header::ORIGIN); + let response = respond(&settings(true, false), request); + assert_eq!( + response.status(), + StatusCode::METHOD_NOT_ALLOWED, + "should reject unsupported methods before action controls" + ); + assert_eq!( + response.headers()[header::ALLOW], + "POST", + "should retain path-specific Allow" + ); + assert!( + response + .into_body() + .into_bytes() + .expect("should return a local HEAD error") + .is_empty(), + "should keep HEAD action errors bodyless" + ); + } +} diff --git a/crates/trusted-server-core/src/trace/auction.rs b/crates/trusted-server-core/src/trace/auction.rs new file mode 100644 index 000000000..221324c82 --- /dev/null +++ b/crates/trusted-server-core/src/trace/auction.rs @@ -0,0 +1,2038 @@ +//! Closed public auction evidence and exact opaque correlation tokens. + +use error_stack::Report; +use serde::{ + Deserialize, Deserializer, Serialize, + de::{Error as _, MapAccess, SeqAccess, Visitor}, +}; +use std::{fmt, marker::PhantomData}; +use uuid::Uuid; + +const PROVIDER_LIMIT: usize = 16; +const SLOT_LIMIT: usize = 64; +const REQUESTED_SIZE_LIMIT: usize = 16; +const MAXIMUM_DIMENSION: u32 = 100_000; + +/// A bounded failure to validate an exact opaque trace token. +#[derive(Debug, derive_more::Display)] +#[display("invalid trace token")] +pub struct TraceTokenError; + +impl core::error::Error for TraceTokenError {} + +/// An exact lowercase UUID-v4 auction token, independent of internal IDs. +#[derive(Clone, Debug, Eq, Hash, PartialEq, Serialize, Deserialize, derive_more::Display)] +#[serde(try_from = "String")] +pub struct DiagnosticAuctionId(String); + +impl DiagnosticAuctionId { + /// Parse an exact auction token without normalization. + /// + /// # Errors + /// + /// Returns [`TraceTokenError`] when the supplied bytes are not the public + /// lowercase, unhyphenated UUID-v4 auction-token shape. + /// + /// # Examples + /// + /// ``` + /// use trusted_server_core::trace::DiagnosticAuctionId; + /// let token = DiagnosticAuctionId::parse("ts-auc-00000000000040008000000000000000")?; + /// assert_eq!(token.as_str(), "ts-auc-00000000000040008000000000000000"); + /// # Ok::<(), error_stack::Report>(()) + /// ``` + /// + /// # Performance + /// + /// Validation checks a fixed byte shape and allocates only for accepted input. + pub fn parse(value: &str) -> Result> { + if !valid_token(value, "ts-auc-", false) { + return Err(Report::new(TraceTokenError)); + } + Ok(Self(value.to_owned())) + } + + /// Generate a fresh public auction token using a UUID v4. + /// + /// # Panics + /// + /// Panics if the UUID random source cannot provide randomness. + /// + /// # Examples + /// + /// ``` + /// let token = trusted_server_core::trace::DiagnosticAuctionId::generate(); + /// assert!(token.as_str().starts_with("ts-auc-")); + /// ``` + #[must_use] + pub fn generate() -> Self { + Self(format!("ts-auc-{}", Uuid::new_v4().simple())) + } + + /// Borrow the exact stored correlation bytes. + /// + /// # Examples + /// + /// ``` + /// let token = trusted_server_core::trace::DiagnosticAuctionId::generate(); + /// assert_eq!(token.as_str().len(), 39); + /// ``` + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl TryFrom for DiagnosticAuctionId { + type Error = Report; + + fn try_from(value: String) -> Result { + if !valid_token(&value, "ts-auc-", false) { + return Err(Report::new(TraceTokenError)); + } + Ok(Self(value)) + } +} + +impl From for String { + fn from(value: DiagnosticAuctionId) -> Self { + value.0 + } +} + +/// An exact lowercase hyphenated UUID-v4 reference for one accepted slot. +#[derive(Clone, Debug, Eq, Hash, PartialEq, Serialize, Deserialize, derive_more::Display)] +#[serde(try_from = "String")] +pub struct TraceSlotRef(String); + +impl TraceSlotRef { + /// Parse an exact slot reference without normalization. + /// + /// # Errors + /// + /// Returns [`TraceTokenError`] for any other token shape, UUID version or + /// variant, including a normalized alternative. + /// + /// # Examples + /// + /// ``` + /// use trusted_server_core::trace::TraceSlotRef; + /// let token = TraceSlotRef::parse("ts-slot-00000000-0000-4000-8000-000000000000")?; + /// assert_eq!(token.as_str().len(), 44); + /// # Ok::<(), error_stack::Report>(()) + /// ``` + /// + /// # Performance + /// + /// Validation checks a fixed byte shape and allocates only for accepted input. + pub fn parse(value: &str) -> Result> { + if !valid_token(value, "ts-slot-", true) { + return Err(Report::new(TraceTokenError)); + } + Ok(Self(value.to_owned())) + } + + /// Generate a fresh canonical slot reference using a UUID v4. + /// + /// # Panics + /// + /// Panics if the UUID random source cannot provide randomness. + /// + /// # Examples + /// + /// ``` + /// let token = trusted_server_core::trace::TraceSlotRef::generate(); + /// assert!(token.as_str().starts_with("ts-slot-")); + /// ``` + #[must_use] + pub fn generate() -> Self { + Self(format!("ts-slot-{}", Uuid::new_v4().hyphenated())) + } + + /// Borrow the exact stored reference bytes. + /// + /// # Examples + /// + /// ``` + /// let token = trusted_server_core::trace::TraceSlotRef::generate(); + /// assert_eq!(token.as_str().len(), 44); + /// ``` + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl TryFrom for TraceSlotRef { + type Error = Report; + + fn try_from(value: String) -> Result { + if !valid_token(&value, "ts-slot-", true) { + return Err(Report::new(TraceTokenError)); + } + Ok(Self(value)) + } +} + +impl From for String { + fn from(value: TraceSlotRef) -> Self { + value.0 + } +} + +fn valid_token(value: &str, prefix: &str, hyphenated: bool) -> bool { + let Some(uuid) = value.strip_prefix(prefix).map(str::as_bytes) else { + return false; + }; + let (length, version, variant) = if hyphenated { + (36, 14, 19) + } else { + (32, 12, 16) + }; + uuid.len() == length + && uuid[version] == b'4' + && matches!(uuid[variant], b'8' | b'9' | b'a' | b'b') + && uuid.iter().enumerate().all(|(index, byte)| { + if hyphenated && matches!(index, 8 | 13 | 18 | 23) { + *byte == b'-' + } else { + matches!(byte, b'0'..=b'9' | b'a'..=b'f') + } + }) +} + +/// The server call site that produced the auction observation. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceAuctionSource { + /// The initial publisher document's server-side auction. + InitialNavigationSsat, + /// The server-side auction for a SPA page-bids request. + SpaPageBids, + /// The programmatic auction API. + AuctionApi, +} + +/// The directly observed auction terminal outcome. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceAuctionTerminalStatus { + /// Execution completed, including a zero-bid result. + Completed, + /// Auction execution or collection failed. + ExecutionFailed, + /// The split dispatch path could not start the auction. + DispatchFailed, + /// Dispatched work could not be collected or delivered. + Abandoned, + /// Consent, policy, or the definitive empty slot list skipped execution. + Skipped, +} + +/// A bounded terminal category without provider text or errors. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceAuctionTerminalReason { + /// An explicit policy or consent decision skipped execution. + PolicySkipped, + /// The definitive accepted slot list is empty. + NoEligibleSlots, + /// The unsuccessful split dispatch path launched no provider. + NoProviderLaunched, + /// A provider execution failure is directly known. + ProviderExecutionFailed, + /// Collecting dispatched work failed. + CollectionFailed, + /// No more specific allowlisted reason is observed. + Unknown, +} + +/// The role of one auction-wide provider call without its identity. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceProviderRole { + /// A bidder call. + Bidder, + /// A mediation call. + Mediator, + /// The role is not known. + Unknown, +} + +/// The directly observed status of one auction-wide provider call. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceProviderStatus { + /// The call returned a successful response with bids. + Success, + /// The call returned no bids. + NoBid, + /// The call failed. + Error, + /// The call remains in flight at the observed point. + Pending, + /// The in-flight call was abandoned. + Abandoned, + /// The status cannot be determined. + Unknown, +} + +/// The observed selection and final delivery disposition for an accepted slot. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TraceSlotCandidate { + /// The selected candidate entered the final response. + Selected, + /// No candidate was selected. + NoCandidate, + /// A selected winner could not safely enter the final response. + SelectedUnrenderable, + /// Existing observations cannot distinguish the accepted slot instance. + Unknown, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct TraceProviderCall { + #[serde(deserialize_with = "deserialize_ordinal")] + provider_number: u16, + #[serde(deserialize_with = "deserialize_string_enum")] + role: TraceProviderRole, + #[serde(deserialize_with = "deserialize_string_enum")] + status: TraceProviderStatus, + #[serde( + default, + deserialize_with = "deserialize_optional_duration", + skip_serializing_if = "Option::is_none" + )] + response_time_ms: Option, + #[serde(deserialize_with = "deserialize_count")] + returned_bid_count: u16, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +struct TraceSize([u32; 2]); + +impl<'de> Deserialize<'de> for TraceSize { + fn deserialize>(deserializer: D) -> Result { + let dimensions = <[TraceDimension; 2]>::deserialize(deserializer)?; + Ok(Self(dimensions.map(|dimension| dimension.0))) + } +} + +struct TraceDimension(u32); + +impl<'de> Deserialize<'de> for TraceDimension { + fn deserialize>(deserializer: D) -> Result { + let dimension = deserializer.deserialize_any(UnsignedVisitor::)?; + if dimension == 0 { + return Err(D::Error::custom("invalid trace dimension")); + } + u32::try_from(dimension) + .map(Self) + .map_err(|_| D::Error::custom("invalid trace dimension")) + } +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct TraceAuctionSlot { + #[serde(deserialize_with = "deserialize_ordinal")] + slot_number: u16, + slot_ref: TraceSlotRef, + #[serde(deserialize_with = "deserialize_sizes")] + requested_sizes: Vec, + #[serde(deserialize_with = "deserialize_count")] + returned_bid_count: u16, + #[serde(deserialize_with = "deserialize_string_enum")] + candidate: TraceSlotCandidate, + #[serde( + default, + deserialize_with = "deserialize_present", + skip_serializing_if = "Option::is_none" + )] + selected_creative_size: Option, +} + +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct TraceAuctionTruncation { + #[serde(deserialize_with = "deserialize_count")] + omitted_provider_calls: u16, + #[serde(deserialize_with = "deserialize_count")] + omitted_slots: u16, + #[serde(deserialize_with = "deserialize_count")] + omitted_nested_values: u16, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum ProviderToSlotNoBid { + Unavailable, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct TraceAuctionCoverage { + #[serde(deserialize_with = "deserialize_string_enum")] + provider_to_slot_no_bid: ProviderToSlotNoBid, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct TraceAuctionEvidenceFields { + #[serde(deserialize_with = "deserialize_version")] + schema_version: u8, + diagnostic_auction_id: DiagnosticAuctionId, + #[serde(deserialize_with = "deserialize_string_enum")] + source: TraceAuctionSource, + #[serde(deserialize_with = "deserialize_string_enum")] + terminal_status: TraceAuctionTerminalStatus, + #[serde( + default, + deserialize_with = "deserialize_optional_enum", + skip_serializing_if = "Option::is_none" + )] + terminal_reason: Option, + #[serde( + default, + deserialize_with = "deserialize_optional_duration", + skip_serializing_if = "Option::is_none" + )] + total_time_ms: Option, + #[serde(deserialize_with = "deserialize_providers")] + provider_calls: Vec, + #[serde(deserialize_with = "deserialize_slots")] + slots: Vec, + #[serde(deserialize_with = "deserialize_object")] + truncation: TraceAuctionTruncation, + #[serde(deserialize_with = "deserialize_object")] + coverage: TraceAuctionCoverage, +} + +/// Closed version-one server facts, separate from bid payloads and telemetry. +/// +/// Fields remain private so object-only deserialization and checked core +/// projection are the only constructors. Serialization includes only opaque +/// tokens, bounded counts, dimensions and durations, and allowlisted categories. +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(transparent)] +pub struct TraceAuctionEvidenceV1( + #[serde(deserialize_with = "deserialize_object")] TraceAuctionEvidenceFields, +); + +impl TraceAuctionEvidenceV1 { + /// Borrow this observation's exact public auction token. + /// + /// # Examples + /// + /// ``` + /// # use trusted_server_core::trace::TraceAuctionEvidenceV1; + /// fn token(evidence: &TraceAuctionEvidenceV1) -> &str { + /// evidence.diagnostic_auction_id().as_str() + /// } + /// ``` + #[must_use] + pub fn diagnostic_auction_id(&self) -> &DiagnosticAuctionId { + &self.0.diagnostic_auction_id + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum TraceUnavailableReason { + EvidenceProjectionFailed, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(untagged, deny_unknown_fields)] +enum TraceTransportOutcome { + Evidence { + #[serde(deserialize_with = "deserialize_version")] + schema_version: u8, + evidence: TraceAuctionEvidenceV1, + }, + Unavailable { + #[serde(deserialize_with = "deserialize_version")] + schema_version: u8, + #[serde(deserialize_with = "deserialize_string_enum")] + unavailable_reason: TraceUnavailableReason, + }, +} + +/// An exclusive evidence or fixed projection-failure transport envelope. +/// +/// The unavailable branch never includes raw input, errors, or partial evidence. +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(transparent)] +pub struct TraceAuctionTransportV1( + #[serde(deserialize_with = "deserialize_object")] TraceTransportOutcome, +); + +impl TraceAuctionTransportV1 { + /// Construct the fixed bounded projection-failure marker. + /// + /// # Examples + /// + /// ``` + /// let transport = trusted_server_core::trace::TraceAuctionTransportV1::unavailable(); + /// assert!(transport.evidence().is_none()); + /// ``` + #[must_use] + pub fn unavailable() -> Self { + Self(TraceTransportOutcome::Unavailable { + schema_version: 1, + unavailable_reason: TraceUnavailableReason::EvidenceProjectionFailed, + }) + } + + /// Borrow complete evidence, absent when projection was unavailable. + /// + /// # Examples + /// + /// ``` + /// let transport = trusted_server_core::trace::TraceAuctionTransportV1::unavailable(); + /// assert!(transport.evidence().is_none()); + /// ``` + #[must_use] + pub fn evidence(&self) -> Option<&TraceAuctionEvidenceV1> { + match &self.0 { + TraceTransportOutcome::Evidence { evidence, .. } => Some(evidence), + TraceTransportOutcome::Unavailable { .. } => None, + } + } +} + +impl From for TraceAuctionTransportV1 { + fn from(evidence: TraceAuctionEvidenceV1) -> Self { + Self(TraceTransportOutcome::Evidence { + schema_version: 1, + evidence, + }) + } +} + +struct UnsignedVisitor; + +impl<'de, const MAXIMUM: u32> Visitor<'de> for UnsignedVisitor { + type Value = u64; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a bounded nonnegative integer") + } + + fn visit_u64(self, value: u64) -> Result { + if value <= u64::from(MAXIMUM) { + Ok(value) + } else { + Err(E::custom("invalid trace integer")) + } + } + + fn visit_i64(self, value: i64) -> Result { + let value = u64::try_from(value).map_err(|_| E::custom("invalid trace integer"))?; + self.visit_u64(value) + } + + fn visit_f64(self, value: f64) -> Result { + if !value.is_finite() || value < 0.0 || value > f64::from(MAXIMUM) || value.fract() != 0.0 { + return Err(E::custom("invalid trace integer")); + } + // The accepted value is an exact integer within the u32 range. + Ok(value as u64) + } +} + +fn deserialize_count<'de, D: Deserializer<'de>>(deserializer: D) -> Result { + let value = deserializer.deserialize_any(UnsignedVisitor::<{ u16::MAX as u32 }>)?; + u16::try_from(value).map_err(|_| D::Error::custom("invalid trace count")) +} + +fn deserialize_ordinal<'de, D: Deserializer<'de>>(deserializer: D) -> Result { + let ordinal = deserialize_count(deserializer)?; + if ordinal == 0 { + return Err(D::Error::custom("invalid trace ordinal")); + } + Ok(ordinal) +} + +fn deserialize_optional_duration<'de, D: Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + let value = deserializer.deserialize_any(UnsignedVisitor::<{ u32::MAX }>)?; + u32::try_from(value) + .map(Some) + .map_err(|_| D::Error::custom("invalid trace duration")) +} + +fn deserialize_version<'de, D: Deserializer<'de>>(deserializer: D) -> Result { + let version = deserializer.deserialize_any(UnsignedVisitor::<1>)?; + if version != 1 { + return Err(D::Error::custom("unsupported trace schema")); + } + Ok(1) +} + +fn deserialize_present<'de, D: Deserializer<'de>, T: Deserialize<'de>>( + deserializer: D, +) -> Result, D::Error> { + T::deserialize(deserializer).map(Some) +} + +fn deserialize_string_enum<'de, D: Deserializer<'de>, T: Deserialize<'de>>( + deserializer: D, +) -> Result { + let value = String::deserialize(deserializer)?; + T::deserialize(serde::de::value::StringDeserializer::::new(value)) +} + +fn deserialize_optional_enum<'de, D: Deserializer<'de>, T: Deserialize<'de>>( + deserializer: D, +) -> Result, D::Error> { + deserialize_string_enum(deserializer).map(Some) +} + +struct ObjectValue(T); + +impl<'de, T: Deserialize<'de>> Deserialize<'de> for ObjectValue { + fn deserialize>(deserializer: D) -> Result { + deserialize_object(deserializer).map(Self) + } +} + +struct ObjectVisitor(PhantomData); + +impl<'de, T: Deserialize<'de>> Visitor<'de> for ObjectVisitor { + type Value = T; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a trace object") + } + + fn visit_map>(self, object: A) -> Result { + T::deserialize(serde::de::value::MapAccessDeserializer::new(object)) + } +} + +fn deserialize_object<'de, D: Deserializer<'de>, T: Deserialize<'de>>( + deserializer: D, +) -> Result { + deserializer.deserialize_map(ObjectVisitor(PhantomData)) +} + +struct BoundedSequence { + object_members: bool, + marker: PhantomData, +} + +impl<'de, T: Deserialize<'de>, const LIMIT: usize> Visitor<'de> for BoundedSequence { + type Value = Vec; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a bounded trace array") + } + + fn visit_seq>(self, mut sequence: A) -> Result { + let mut values = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(LIMIT)); + while let Some(value) = if self.object_members { + sequence + .next_element::>()? + .map(|value| value.0) + } else { + sequence.next_element::()? + } { + if values.len() == LIMIT { + return Err(A::Error::custom("oversized trace array")); + } + values.push(value); + } + Ok(values) + } +} + +fn deserialize_providers<'de, D: Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + deserializer.deserialize_seq(BoundedSequence:: { + object_members: true, + marker: PhantomData, + }) +} + +fn deserialize_slots<'de, D: Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + deserializer.deserialize_seq(BoundedSequence:: { + object_members: true, + marker: PhantomData, + }) +} + +fn deserialize_sizes<'de, D: Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + deserializer.deserialize_seq(BoundedSequence:: { + object_members: false, + marker: PhantomData, + }) +} + +pub(crate) mod projection { + use std::time::Duration; + + use error_stack::Report; + + use super::{ + DiagnosticAuctionId, MAXIMUM_DIMENSION, PROVIDER_LIMIT, ProviderToSlotNoBid, + REQUESTED_SIZE_LIMIT, SLOT_LIMIT, TraceAuctionCoverage, TraceAuctionEvidenceFields, + TraceAuctionEvidenceV1, TraceAuctionSlot, TraceAuctionSource, TraceAuctionTerminalReason, + TraceAuctionTerminalStatus, TraceAuctionTransportV1, TraceAuctionTruncation, + TraceProviderCall, TraceProviderRole, TraceProviderStatus, TraceSize, TraceSlotCandidate, + TraceSlotRef, + }; + + #[derive(Clone, Debug, Eq, PartialEq)] + pub(crate) struct ObservedTraceProviderCall { + pub(crate) provider_number: usize, + pub(crate) role: TraceProviderRole, + pub(crate) status: TraceProviderStatus, + pub(crate) response_time: Option, + pub(crate) returned_bid_count: usize, + } + + #[derive(Clone, Debug, Eq, PartialEq)] + pub(crate) struct ObservedTraceSlot { + pub(crate) slot_number: usize, + pub(crate) slot_ref: TraceSlotRef, + pub(crate) requested_sizes: Vec<[u32; 2]>, + pub(crate) returned_bid_count: usize, + pub(crate) candidate: TraceSlotCandidate, + pub(crate) selected_creative_size: Option<[u32; 2]>, + } + + #[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] + pub(crate) struct ObservedTraceTruncation { + pub(crate) omitted_provider_calls: usize, + pub(crate) omitted_slots: usize, + pub(crate) omitted_nested_values: usize, + } + + #[derive(Clone, Debug, Eq, PartialEq)] + pub(crate) struct ObservedTraceAuction { + pub(crate) diagnostic_auction_id: DiagnosticAuctionId, + pub(crate) source: TraceAuctionSource, + pub(crate) terminal_status: TraceAuctionTerminalStatus, + pub(crate) terminal_reason: Option, + pub(crate) total_time: Option, + pub(crate) provider_calls: Vec, + pub(crate) slots: Vec, + pub(crate) truncation: ObservedTraceTruncation, + } + + #[derive(Debug, derive_more::Display)] + #[display("trace evidence projection failed")] + struct TraceEvidenceProjectionError; + + impl core::error::Error for TraceEvidenceProjectionError {} + + pub(crate) fn project_auction_transport( + observed: &ObservedTraceAuction, + ) -> TraceAuctionTransportV1 { + match project_auction_evidence(observed) { + Ok(evidence) => TraceAuctionTransportV1::from(evidence), + Err(_) => TraceAuctionTransportV1::unavailable(), + } + } + + fn project_auction_evidence( + observed: &ObservedTraceAuction, + ) -> Result> { + // Required invalid source facts reject evidence even beyond its limits. + for provider in &observed.provider_calls { + checked_ordinal(provider.provider_number)?; + checked_count(provider.returned_bid_count)?; + } + for slot in &observed.slots { + checked_ordinal(slot.slot_number)?; + checked_count(slot.returned_bid_count)?; + } + let mut truncation = TraceAuctionTruncation { + omitted_provider_calls: checked_count(observed.truncation.omitted_provider_calls)?, + omitted_slots: checked_count(observed.truncation.omitted_slots)?, + omitted_nested_values: checked_count(observed.truncation.omitted_nested_values)?, + }; + let retained_providers = observed.provider_calls.len().min(PROVIDER_LIMIT); + add_omissions( + &mut truncation.omitted_provider_calls, + observed.provider_calls.len() - retained_providers, + )?; + let retained_slots = observed.slots.len().min(SLOT_LIMIT); + add_omissions( + &mut truncation.omitted_slots, + observed.slots.len() - retained_slots, + )?; + let total_time_ms = + project_duration(observed.total_time, &mut truncation.omitted_nested_values)?; + + let mut provider_calls = Vec::with_capacity(retained_providers); + for provider in observed.provider_calls.iter().take(PROVIDER_LIMIT) { + provider_calls.push(TraceProviderCall { + provider_number: checked_ordinal(provider.provider_number)?, + role: provider.role, + status: provider.status, + response_time_ms: project_duration( + provider.response_time, + &mut truncation.omitted_nested_values, + )?, + returned_bid_count: checked_count(provider.returned_bid_count)?, + }); + } + + let mut slots = Vec::with_capacity(retained_slots); + for slot in observed.slots.iter().take(SLOT_LIMIT) { + let retained_sizes = slot.requested_sizes.len().min(REQUESTED_SIZE_LIMIT); + add_omissions( + &mut truncation.omitted_nested_values, + slot.requested_sizes.len() - retained_sizes, + )?; + let mut requested_sizes = Vec::with_capacity(retained_sizes); + for size in slot.requested_sizes.iter().take(REQUESTED_SIZE_LIMIT) { + if let Some(size) = + project_size(Some(*size), &mut truncation.omitted_nested_values)? + { + requested_sizes.push(size); + } + } + slots.push(TraceAuctionSlot { + slot_number: checked_ordinal(slot.slot_number)?, + slot_ref: slot.slot_ref.clone(), + requested_sizes, + returned_bid_count: checked_count(slot.returned_bid_count)?, + candidate: slot.candidate, + selected_creative_size: project_size( + slot.selected_creative_size, + &mut truncation.omitted_nested_values, + )?, + }); + } + + Ok(TraceAuctionEvidenceV1(TraceAuctionEvidenceFields { + schema_version: 1, + diagnostic_auction_id: observed.diagnostic_auction_id.clone(), + source: observed.source, + terminal_status: observed.terminal_status, + terminal_reason: observed.terminal_reason, + total_time_ms, + provider_calls, + slots, + truncation, + coverage: TraceAuctionCoverage { + provider_to_slot_no_bid: ProviderToSlotNoBid::Unavailable, + }, + })) + } + + fn checked_count(count: usize) -> Result> { + u16::try_from(count).map_err(|_| Report::new(TraceEvidenceProjectionError)) + } + + fn checked_ordinal(ordinal: usize) -> Result> { + let ordinal = checked_count(ordinal)?; + if ordinal == 0 { + return Err(Report::new(TraceEvidenceProjectionError)); + } + Ok(ordinal) + } + + fn add_omissions( + counter: &mut u16, + additional: usize, + ) -> Result<(), Report> { + let additional = checked_count(additional)?; + *counter = counter + .checked_add(additional) + .ok_or_else(|| Report::new(TraceEvidenceProjectionError))?; + Ok(()) + } + + fn project_duration( + duration: Option, + omissions: &mut u16, + ) -> Result, Report> { + let Some(duration) = duration else { + return Ok(None); + }; + match u32::try_from(duration.as_millis()) { + Ok(milliseconds) => Ok(Some(milliseconds)), + Err(_) => { + add_omissions(omissions, 1)?; + Ok(None) + } + } + } + + fn project_size( + size: Option<[u32; 2]>, + omissions: &mut u16, + ) -> Result, Report> { + let Some(size) = size else { + return Ok(None); + }; + if size + .iter() + .all(|dimension| (1..=MAXIMUM_DIMENSION).contains(dimension)) + { + Ok(Some(TraceSize(size))) + } else { + add_omissions(omissions, 1)?; + Ok(None) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + use serde_json::json; + use std::time::Duration; + + use super::projection::{ + ObservedTraceAuction, ObservedTraceProviderCall, ObservedTraceSlot, + ObservedTraceTruncation, project_auction_transport, + }; + + const AUCTION_TOKEN: &str = "ts-auc-00000000000040008000000000000000"; + const SLOT_TOKEN: &str = "ts-slot-00000000-0000-4000-8000-000000000000"; + + #[test] + fn trace_auction_tokens_accept_exact_v4_rfc_variants() { + for variant in ['8', '9', 'a', 'b'] { + let auction = format!("ts-auc-0123456789ab4def{variant}123456789abcdef"); + let slot = format!("ts-slot-01234567-89ab-4def-{variant}123-456789abcdef"); + assert_eq!( + DiagnosticAuctionId::parse(&auction) + .expect("should accept exact auction token") + .as_str(), + auction, + "should retain exact auction bytes" + ); + assert_eq!( + TraceSlotRef::parse(&slot) + .expect("should accept exact slot token") + .as_str(), + slot, + "should retain exact slot bytes" + ); + } + } + + #[test] + fn trace_auction_tokens_reject_normalized_and_internal_alternatives() { + for invalid in [ + "", + "00000000-0000-4000-8000-000000000000", + "ts-auc-00000000-0000-4000-8000-000000000000", + "ts-auc-00000000000010008000000000000000", + "ts-auc-00000000000040007000000000000000", + "ts-auc-0000000000004000c000000000000000", + "ts-auc-0000000000004000A000000000000000", + "ts-auc-0000000000004000800000000000000g", + " ts-auc-00000000000040008000000000000000", + "ts-auc-00000000000040008000000000000000\n", + "TS-AUC-00000000000040008000000000000000", + "ts-auc-00000000000040008000000000000000\u{202e}", + ] { + assert!( + DiagnosticAuctionId::parse(invalid).is_err(), + "should reject invalid auction token" + ); + } + for invalid in [ + "", + AUCTION_TOKEN, + "ts-slot-00000000000040008000000000000000", + "ts-slot-00000000-0000-1000-8000-000000000000", + "ts-slot-00000000-0000-4000-7000-000000000000", + "ts-slot-00000000-0000-4000-c000-000000000000", + "ts-slot-00000000-0000-4000-A000-000000000000", + "ts-slot-00000000-0000-4000-8000-00000000000g", + " ts-slot-00000000-0000-4000-8000-000000000000", + "ts-slot-00000000-0000-4000-8000-000000000000\n", + "TS-SLOT-00000000-0000-4000-8000-000000000000", + "ts-slot-00000000-0000-4000-8000-000000000000\u{202e}", + ] { + assert!( + TraceSlotRef::parse(invalid).is_err(), + "should reject invalid slot reference" + ); + } + } + + #[test] + fn trace_auction_tokens_generate_distinct_exact_owned_values() { + let auctions = [ + DiagnosticAuctionId::generate(), + DiagnosticAuctionId::generate(), + ]; + let slots = [TraceSlotRef::generate(), TraceSlotRef::generate()]; + assert_ne!( + auctions[0], auctions[1], + "should generate fresh auction IDs" + ); + assert_ne!(slots[0], slots[1], "should generate fresh slot references"); + for auction in auctions { + assert_eq!( + auction.as_str().len(), + 39, + "should retain simple UUID shape" + ); + assert_eq!( + DiagnosticAuctionId::parse(auction.as_str()) + .expect("should validate generated auction token"), + auction, + "should round-trip exact auction bytes" + ); + } + for slot in slots { + assert_eq!( + slot.as_str().len(), + 44, + "should retain hyphenated UUID shape" + ); + assert_eq!( + TraceSlotRef::parse(slot.as_str()).expect("should validate generated slot ref"), + slot, + "should round-trip exact slot bytes" + ); + } + } + + #[test] + fn trace_auction_tokens_use_strict_string_serde() { + let auction = + DiagnosticAuctionId::parse(AUCTION_TOKEN).expect("should accept exact auction token"); + let slot = TraceSlotRef::parse(SLOT_TOKEN).expect("should accept exact slot token"); + assert_eq!( + json!(auction), + json!(AUCTION_TOKEN), + "should serialize only opaque bytes" + ); + assert_eq!( + json!(slot), + json!(SLOT_TOKEN), + "should serialize only opaque bytes" + ); + for value in [ + json!(null), + json!(1), + json!({}), + json!([]), + json!("internal-example-id"), + ] { + assert!( + serde_json::from_value::(value.clone()).is_err(), + "should reject invalid auction JSON" + ); + assert!( + serde_json::from_value::(value).is_err(), + "should reject invalid slot JSON" + ); + } + } + + fn evidence_json() -> serde_json::Value { + json!({ + "schema_version": 1, + "diagnostic_auction_id": AUCTION_TOKEN, + "source": "initial_navigation_ssat", + "terminal_status": "completed", + "terminal_reason": "unknown", + "total_time_ms": 10, + "provider_calls": [{ + "provider_number": 1, + "role": "bidder", + "status": "success", + "response_time_ms": 5, + "returned_bid_count": 2 + }], + "slots": [{ + "slot_number": 1, + "slot_ref": SLOT_TOKEN, + "requested_sizes": [[300, 250]], + "returned_bid_count": 2, + "candidate": "selected", + "selected_creative_size": [300, 250] + }], + "truncation": { + "omitted_provider_calls": 0, + "omitted_slots": 0, + "omitted_nested_values": 0 + }, + "coverage": {"provider_to_slot_no_bid": "unavailable"} + }) + } + + fn assert_invalid_evidence(value: serde_json::Value) { + assert!( + serde_json::from_value::(value).is_err(), + "should reject invalid public auction evidence" + ); + } + + #[test] + fn trace_auction_evidence_round_trips_the_exact_owned_schema() { + let input = evidence_json(); + let original = input.clone(); + let evidence = serde_json::from_value::(input) + .expect("should accept complete evidence"); + assert_eq!( + json!(evidence), + original, + "should preserve only exact schema fields" + ); + assert_eq!( + evidence.diagnostic_auction_id().as_str(), + AUCTION_TOKEN, + "should expose the checked correlation token" + ); + let transport = TraceAuctionTransportV1::from(evidence); + assert_eq!( + json!(transport), + json!({"schema_version":1,"evidence":original}), + "should use exclusive evidence envelope" + ); + assert!( + transport.evidence().is_some(), + "should expose available evidence" + ); + } + + #[test] + fn trace_auction_evidence_rejects_unknown_keys_at_every_boundary() { + for pointer in [ + "", + "/provider_calls/0", + "/slots/0", + "/truncation", + "/coverage", + ] { + let mut input = evidence_json(); + input + .pointer_mut(pointer) + .expect("should find the schema object") + .as_object_mut() + .expect("should expose the schema object") + .insert("future_example_field".to_owned(), json!("example")); + assert_invalid_evidence(input); + } + } + + #[test] + fn trace_auction_evidence_accepts_only_documented_enum_members() { + for (pointer, members) in [ + ( + "/source", + &["initial_navigation_ssat", "spa_page_bids", "auction_api"][..], + ), + ( + "/terminal_status", + &[ + "completed", + "execution_failed", + "dispatch_failed", + "abandoned", + "skipped", + ][..], + ), + ( + "/terminal_reason", + &[ + "policy_skipped", + "no_eligible_slots", + "no_provider_launched", + "provider_execution_failed", + "collection_failed", + "unknown", + ][..], + ), + ( + "/provider_calls/0/role", + &["bidder", "mediator", "unknown"][..], + ), + ( + "/provider_calls/0/status", + &[ + "success", + "no_bid", + "error", + "pending", + "abandoned", + "unknown", + ][..], + ), + ( + "/slots/0/candidate", + &[ + "selected", + "no_candidate", + "selected_unrenderable", + "unknown", + ][..], + ), + ] { + for member in members { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find enum member") = json!(member); + let accepted = serde_json::from_value::(input.clone()) + .expect("should accept documented enum member"); + assert_eq!(json!(accepted), input, "should preserve exact enum bytes"); + } + for invalid in [ + json!("future_example_member"), + json!("UNKNOWN"), + json!("unknown "), + json!("\u{202e}unknown"), + json!(null), + json!(0), + json!({}), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find enum member") = invalid; + assert_invalid_evidence(input); + } + } + } + + #[test] + fn trace_auction_evidence_requires_primitive_strings_for_enum_fields() { + for (pointer, member) in [ + ("/source", "initial_navigation_ssat"), + ("/terminal_status", "completed"), + ("/terminal_reason", "unknown"), + ("/provider_calls/0/role", "bidder"), + ("/provider_calls/0/status", "success"), + ("/slots/0/candidate", "selected"), + ("/coverage/provider_to_slot_no_bid", "unavailable"), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find enum field") = json!({(member): null}); + let encoded = + serde_json::to_string(&input).expect("should serialize object-form enum fixture"); + assert!( + serde_json::from_str::(&encoded).is_err(), + "should reject object-form enum from JSON bytes" + ); + assert_invalid_evidence(input); + } + } + + #[test] + fn trace_auction_evidence_requires_objects_at_every_object_boundary() { + let original = evidence_json(); + for (pointer, alternative) in [ + ( + "", + json!([ + 1, + AUCTION_TOKEN, + "initial_navigation_ssat", + "completed", + "unknown", + 10, + original["provider_calls"], + original["slots"], + original["truncation"], + original["coverage"] + ]), + ), + ("/provider_calls/0", json!([1, "bidder", "success", 5, 2])), + ( + "/slots/0", + json!([1, SLOT_TOKEN, [[300, 250]], 2, "selected", [300, 250]]), + ), + ("/truncation", json!([0, 0, 0])), + ("/coverage", json!(["unavailable"])), + ] { + let mut input = original.clone(); + *input + .pointer_mut(pointer) + .expect("should find object boundary") = alternative; + let encoded = + serde_json::to_string(&input).expect("should encode positional-array fixture"); + assert!( + serde_json::from_str::(&encoded).is_err(), + "should reject positional arrays from JSON bytes" + ); + assert_invalid_evidence(input); + } + } + + #[test] + fn trace_auction_evidence_rejects_null_optional_and_missing_required_fields() { + for pointer in [ + "/terminal_reason", + "/total_time_ms", + "/provider_calls/0/response_time_ms", + "/slots/0/selected_creative_size", + ] { + let mut input = evidence_json(); + *input + .pointer_mut(pointer) + .expect("should find optional field") = json!(null); + assert_invalid_evidence(input); + } + for pointer in [ + "", + "/provider_calls/0", + "/slots/0", + "/truncation", + "/coverage", + ] { + let source = evidence_json(); + let names: Vec<_> = source + .pointer(pointer) + .expect("should find required object") + .as_object() + .expect("should expose required object") + .keys() + .filter(|name| { + !matches!( + name.as_str(), + "terminal_reason" + | "total_time_ms" + | "response_time_ms" + | "selected_creative_size" + ) + }) + .cloned() + .collect(); + for name in names { + let mut input = source.clone(); + input + .pointer_mut(pointer) + .expect("should find required object") + .as_object_mut() + .expect("should expose required object") + .remove(&name); + assert_invalid_evidence(input); + } + } + let mut input = evidence_json(); + input + .as_object_mut() + .expect("should expose evidence") + .remove("terminal_reason"); + input + .as_object_mut() + .expect("should expose evidence") + .remove("total_time_ms"); + input["provider_calls"][0] + .as_object_mut() + .expect("should expose provider") + .remove("response_time_ms"); + input["slots"][0] + .as_object_mut() + .expect("should expose slot") + .remove("selected_creative_size"); + let accepted = serde_json::from_value::(input.clone()) + .expect("should accept absent optional facts"); + assert_eq!( + json!(accepted), + input, + "should leave unsupported optional facts absent" + ); + } + + #[test] + fn trace_auction_evidence_enforces_exact_cardinality_limits() { + for (pointer, limit) in [ + ("/provider_calls", 16), + ("/slots", 64), + ("/slots/0/requested_sizes", 16), + ] { + let mut input = evidence_json(); + let member = input.pointer(pointer).expect("should find bounded array")[0].clone(); + *input + .pointer_mut(pointer) + .expect("should find bounded array") = json!(vec![member.clone(); limit]); + assert!( + serde_json::from_value::(input.clone()).is_ok(), + "should accept exact array limit" + ); + input + .pointer_mut(pointer) + .expect("should find bounded array") + .as_array_mut() + .expect("should expose array") + .push(member); + assert_invalid_evidence(input); + } + let mut empty = evidence_json(); + empty["provider_calls"] = json!([]); + empty["slots"] = json!([]); + assert!( + serde_json::from_value::(empty).is_ok(), + "should accept completed zero-bid evidence" + ); + } + + #[test] + fn trace_auction_evidence_requires_positive_u16_ordinals_and_u16_counts() { + for pointer in ["/provider_calls/0/provider_number", "/slots/0/slot_number"] { + for valid in [1_u32, u32::from(u16::MAX)] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find ordinal") = json!(valid); + assert!( + serde_json::from_value::(input).is_ok(), + "should accept positive u16 ordinal" + ); + } + for invalid in [ + json!(0), + json!(-1), + json!(65536), + json!(1.5), + json!("1"), + json!(null), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find ordinal") = invalid; + assert_invalid_evidence(input); + } + } + for pointer in [ + "/provider_calls/0/returned_bid_count", + "/slots/0/returned_bid_count", + "/truncation/omitted_provider_calls", + "/truncation/omitted_slots", + "/truncation/omitted_nested_values", + ] { + for valid in [0_u32, u32::from(u16::MAX)] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find count") = json!(valid); + assert!( + serde_json::from_value::(input).is_ok(), + "should accept u16 count" + ); + } + for invalid in [ + json!(-1), + json!(65536), + json!(1.5), + json!("0"), + json!(null), + json!(9007199254740992_u64), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find count") = invalid; + assert_invalid_evidence(input); + } + } + } + + #[test] + fn trace_auction_evidence_requires_bounded_durations_dimensions_version_and_coverage() { + for pointer in ["/total_time_ms", "/provider_calls/0/response_time_ms"] { + for valid in [0_u64, u64::from(u32::MAX)] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find duration") = json!(valid); + assert!( + serde_json::from_value::(input).is_ok(), + "should accept u32 duration" + ); + } + for invalid in [ + json!(-1), + json!(4294967296_u64), + json!(0.5), + json!("0"), + json!(null), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find duration") = invalid; + assert_invalid_evidence(input); + } + } + for pointer in [ + "/slots/0/requested_sizes/0", + "/slots/0/selected_creative_size", + ] { + for valid in [[1, 1], [100000, 100000]] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find dimensions") = json!(valid); + assert!( + serde_json::from_value::(input).is_ok(), + "should accept dimension bounds" + ); + } + for invalid in [ + json!([0, 1]), + json!([1, 0]), + json!([100001, 1]), + json!([1, 100001]), + json!([1]), + json!([1, 1, 1]), + json!([-1, 1]), + json!([1.5, 1]), + json!(["1", 1]), + json!(null), + ] { + let mut input = evidence_json(); + *input.pointer_mut(pointer).expect("should find dimensions") = invalid; + assert_invalid_evidence(input); + } + } + for invalid in [json!(0), json!(2), json!("1"), json!(null), json!(1.5)] { + let mut input = evidence_json(); + input["schema_version"] = invalid; + assert_invalid_evidence(input); + } + for invalid in [ + json!("available"), + json!("Unavailable"), + json!(null), + json!(0), + ] { + let mut input = evidence_json(); + input["coverage"]["provider_to_slot_no_bid"] = invalid; + assert_invalid_evidence(input); + } + } + + #[test] + fn trace_auction_evidence_accepts_integral_json_numbers_without_syntax_coercion() { + let mut input = evidence_json(); + for pointer in [ + "/schema_version", + "/total_time_ms", + "/provider_calls/0/provider_number", + "/provider_calls/0/returned_bid_count", + "/provider_calls/0/response_time_ms", + "/slots/0/slot_number", + "/slots/0/returned_bid_count", + "/truncation/omitted_slots", + ] { + *input + .pointer_mut(pointer) + .expect("should find bounded integer") = json!(1.0); + } + input["slots"][0]["requested_sizes"] = json!([[1.0, 100000.0]]); + input["slots"][0]["selected_creative_size"] = json!([1.0, 100000.0]); + assert!( + serde_json::from_value::(input.clone()).is_ok(), + "should accept finite integral JSON values like the browser" + ); + let scientific = serde_json::to_string(&input) + .expect("should serialize numeric fixture") + .replace("1.0", "1e0"); + assert!( + serde_json::from_str::(&scientific).is_ok(), + "should accept integral scientific notation" + ); + } + + #[test] + fn trace_auction_transport_is_exclusive_and_rejects_extra_keys() { + let marker = TraceAuctionTransportV1::unavailable(); + assert_eq!( + json!(marker), + json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed"}), + "should expose only fixed bounded projection failure" + ); + assert!( + marker.evidence().is_none(), + "should distinguish failed projection" + ); + for input in [ + json!({"schema_version":1}), + json!({"schema_version":1,"evidence":evidence_json(),"unavailable_reason":"evidence_projection_failed"}), + json!({"schema_version":1,"evidence":null}), + json!({"schema_version":2,"evidence":evidence_json()}), + json!({"schema_version":1,"unavailable_reason":"example raw provider failure"}), + json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed","future_example_field":0}), + json!({"schema_version":1,"evidence":evidence_json(),"future_example_field":0}), + ] { + assert!( + serde_json::from_value::(input).is_err(), + "should reject malformed whole envelope" + ); + } + for input in [ + json!(marker), + json!({"schema_version":1,"evidence":evidence_json()}), + ] { + let accepted = serde_json::from_value::(input.clone()) + .expect("should accept exactly one envelope branch"); + assert_eq!(json!(accepted), input, "should retain exact branch schema"); + } + } + + #[test] + fn trace_auction_transport_marker_requires_primitive_string_reason() { + let input = json!({ + "schema_version":1, + "unavailable_reason":{"evidence_projection_failed":null} + }); + let encoded = + serde_json::to_string(&input).expect("should serialize object-form marker fixture"); + assert!( + serde_json::from_str::(&encoded).is_err(), + "should reject object-form marker from JSON bytes" + ); + assert!( + serde_json::from_value::(input).is_err(), + "should reject object-form marker from a value" + ); + } + + #[test] + fn trace_auction_transport_requires_an_object_envelope() { + for input in [ + json!([1, evidence_json()]), + json!([1, "evidence_projection_failed"]), + ] { + let encoded = + serde_json::to_string(&input).expect("should encode positional transport fixture"); + assert!( + serde_json::from_str::(&encoded).is_err(), + "should reject positional transport from JSON bytes" + ); + assert!( + serde_json::from_value::(input).is_err(), + "should reject positional transport from a value" + ); + } + } + + #[test] + fn trace_auction_evidence_rejects_duplicate_members_and_invalid_unicode() { + let evidence = serde_json::to_string(&evidence_json()).expect("should serialize fixture"); + let duplicate = evidence.replacen( + "\"schema_version\":1", + "\"schema_version\":1,\"schema_version\":1", + 1, + ); + assert!( + serde_json::from_str::(&duplicate).is_err(), + "should reject duplicate schema members" + ); + let duplicate = evidence.replacen( + "\"provider_number\":1", + "\"provider_number\":1,\"provider_number\":1", + 1, + ); + assert!( + serde_json::from_str::(&duplicate).is_err(), + "should reject duplicate provider members" + ); + let invalid_unicode = evidence.replace(AUCTION_TOKEN, "\\ud800"); + assert!( + serde_json::from_str::(&invalid_unicode).is_err(), + "should reject invalid Unicode strings" + ); + for invalid in [ + "example".repeat(30), + format!("{AUCTION_TOKEN}\u{0085}"), + format!("{SLOT_TOKEN}\u{2066}"), + ] { + let mut input = evidence_json(); + input["diagnostic_auction_id"] = json!(invalid); + assert_invalid_evidence(input); + } + } + + fn observed_auction() -> ObservedTraceAuction { + ObservedTraceAuction { + diagnostic_auction_id: DiagnosticAuctionId::parse(AUCTION_TOKEN) + .expect("should parse observation token"), + source: TraceAuctionSource::InitialNavigationSsat, + terminal_status: TraceAuctionTerminalStatus::Completed, + terminal_reason: Some(TraceAuctionTerminalReason::Unknown), + total_time: Some(Duration::from_millis(10)), + provider_calls: vec![ObservedTraceProviderCall { + provider_number: 1, + role: TraceProviderRole::Bidder, + status: TraceProviderStatus::Success, + response_time: Some(Duration::from_millis(5)), + returned_bid_count: 2, + }], + slots: vec![ObservedTraceSlot { + slot_number: 1, + slot_ref: TraceSlotRef::parse(SLOT_TOKEN) + .expect("should parse observation slot ref"), + requested_sizes: vec![[300, 250]], + returned_bid_count: 2, + candidate: TraceSlotCandidate::Selected, + selected_creative_size: Some([300, 250]), + }], + truncation: ObservedTraceTruncation::default(), + } + } + + #[test] + fn trace_auction_projection_copies_exact_public_facts_without_mutation() { + let observed = observed_auction(); + let original = observed.clone(); + let projected = project_auction_transport(&observed); + assert_eq!( + json!(projected), + json!({"schema_version":1,"evidence":evidence_json()}), + "should project the closed public model" + ); + assert_eq!( + observed, original, + "should leave caller-owned facts unchanged" + ); + let validated = serde_json::from_value::(json!(projected)) + .expect("should validate core-produced transport"); + assert_eq!( + validated, projected, + "should produce the same strict contract accepted at ingress" + ); + } + + #[test] + fn trace_auction_projection_retains_order_and_counts_all_bounded_discards() { + let mut observed = observed_auction(); + let provider = observed.provider_calls[0].clone(); + observed.provider_calls = (1..=18) + .map(|provider_number| ObservedTraceProviderCall { + provider_number, + ..provider.clone() + }) + .collect(); + let mut slot = observed.slots[0].clone(); + slot.requested_sizes = (1..=20).map(|dimension| [dimension, 250]).collect(); + observed.slots = (1..=66) + .map(|slot_number| ObservedTraceSlot { + slot_number, + slot_ref: TraceSlotRef::generate(), + ..slot.clone() + }) + .collect(); + observed.truncation = ObservedTraceTruncation { + omitted_provider_calls: 3, + omitted_slots: 4, + omitted_nested_values: 5, + }; + let original = observed.clone(); + let projected = project_auction_transport(&observed); + let evidence = json!(projected.evidence().expect("should project bounded facts")); + assert_eq!( + evidence["provider_calls"] + .as_array() + .expect("should expose providers") + .len(), + 16, + "should retain first provider calls" + ); + assert_eq!( + evidence["slots"] + .as_array() + .expect("should expose slots") + .len(), + 64, + "should retain first definitive slots" + ); + for (index, provider) in evidence["provider_calls"] + .as_array() + .expect("should expose providers") + .iter() + .enumerate() + { + assert_eq!( + provider["provider_number"], + json!(index + 1), + "should preserve launch order" + ); + } + for (index, slot) in evidence["slots"] + .as_array() + .expect("should expose slots") + .iter() + .enumerate() + { + assert_eq!( + slot["slot_number"], + json!(index + 1), + "should preserve definitive slot order" + ); + assert_eq!( + slot["slot_ref"], + json!(observed.slots[index].slot_ref), + "should preserve the exact accepted token association" + ); + assert_eq!( + slot["requested_sizes"], + json!( + (1..=16) + .map(|dimension| [dimension, 250]) + .collect::>() + ), + "should retain first requested sizes" + ); + } + assert_eq!( + evidence["truncation"], + json!({"omitted_provider_calls":5,"omitted_slots":6,"omitted_nested_values":261}), + "should add every omitted retained-slot size with checked counters" + ); + assert_eq!( + observed, original, + "should preserve all ordinary caller work and facts" + ); + } + + #[test] + fn trace_auction_projection_omits_optional_overflow_and_invalid_sizes_with_counts() { + let mut observed = observed_auction(); + observed.total_time = Some(Duration::MAX); + observed.provider_calls[0].response_time = + Some(Duration::from_millis(u64::from(u32::MAX) + 1)); + observed.slots[0].requested_sizes = vec![ + [0, 250], + [300, 0], + [100001, 250], + [300, 100001], + [1, 1], + [100000, 100000], + ]; + observed.slots[0].selected_creative_size = Some([0, 250]); + let projected = project_auction_transport(&observed); + let evidence = json!( + projected + .evidence() + .expect("should preserve required facts") + ); + assert!( + evidence.get("total_time_ms").is_none(), + "should omit unrepresentable monotonic duration" + ); + assert!( + evidence["provider_calls"][0] + .get("response_time_ms") + .is_none(), + "should omit provider duration overflow" + ); + assert!( + evidence["slots"][0].get("selected_creative_size").is_none(), + "should omit invalid optional creative dimensions" + ); + assert_eq!( + evidence["slots"][0]["requested_sizes"], + json!([[1, 1], [100000, 100000]]), + "should omit invalid optional requested sizes without normalizing" + ); + assert_eq!( + evidence["truncation"]["omitted_nested_values"], + json!(7), + "should count every unsupported optional fact" + ); + assert_eq!( + evidence["slots"][0]["candidate"], + json!("selected"), + "should preserve observed candidate disposition" + ); + } + + #[test] + fn trace_auction_projection_keeps_exact_numeric_boundaries_and_zero_durations() { + let mut observed = observed_auction(); + observed.total_time = Some(Duration::ZERO); + observed.provider_calls[0].response_time = Some(Duration::from_millis(u64::from(u32::MAX))); + observed.provider_calls[0].provider_number = usize::from(u16::MAX); + observed.provider_calls[0].returned_bid_count = usize::from(u16::MAX); + observed.slots[0].slot_number = usize::from(u16::MAX); + observed.slots[0].returned_bid_count = usize::from(u16::MAX); + observed.slots[0].requested_sizes = vec![[1, 1], [100000, 100000]]; + observed.slots[0].selected_creative_size = Some([100000, 100000]); + let projected = project_auction_transport(&observed); + let evidence = json!(projected.evidence().expect("should preserve exact bounds")); + assert_eq!( + evidence["total_time_ms"], + json!(0), + "should preserve zero monotonic duration" + ); + assert_eq!( + evidence["provider_calls"][0]["response_time_ms"], + json!(u32::MAX), + "should preserve maximum duration without clamping" + ); + assert_eq!( + evidence["provider_calls"][0]["provider_number"], + json!(u16::MAX), + "should preserve maximum ordinal" + ); + assert_eq!( + evidence["slots"][0]["returned_bid_count"], + json!(u16::MAX), + "should preserve maximum bid count" + ); + assert_eq!( + evidence["truncation"], + json!({"omitted_provider_calls":0,"omitted_slots":0,"omitted_nested_values":0}), + "should not count representable values as omissions" + ); + } + + #[test] + fn trace_auction_projection_required_conversion_failure_emits_only_marker() { + for field in [ + "provider_number", + "provider_count", + "slot_number", + "slot_count", + ] { + for invalid in [usize::from(u16::MAX) + 1, usize::MAX] { + let mut observed = observed_auction(); + match field { + "provider_number" => observed.provider_calls[0].provider_number = invalid, + "provider_count" => observed.provider_calls[0].returned_bid_count = invalid, + "slot_number" => observed.slots[0].slot_number = invalid, + "slot_count" => observed.slots[0].returned_bid_count = invalid, + _ => panic!("should use the documented test field"), + } + assert_eq!( + json!(project_auction_transport(&observed)), + json!(TraceAuctionTransportV1::unavailable()), + "should reject required conversion without exposing partial facts" + ); + } + } + for provider_ordinal in [true, false] { + let mut observed = observed_auction(); + if provider_ordinal { + observed.provider_calls[0].provider_number = 0; + } else { + observed.slots[0].slot_number = 0; + } + assert!( + project_auction_transport(&observed).evidence().is_none(), + "should reject zero ordinal" + ); + } + } + + #[test] + fn trace_auction_projection_rejects_required_invalid_tail_facts() { + for field in [ + "provider_number", + "provider_count", + "slot_number", + "slot_count", + ] { + let mut observed = observed_auction(); + observed.provider_calls = vec![observed.provider_calls[0].clone(); 17]; + observed.slots = vec![observed.slots[0].clone(); 65]; + match field { + "provider_number" => observed.provider_calls[16].provider_number = 0, + "provider_count" => { + observed.provider_calls[16].returned_bid_count = usize::from(u16::MAX) + 1 + } + "slot_number" => observed.slots[64].slot_number = 0, + "slot_count" => observed.slots[64].returned_bid_count = usize::from(u16::MAX) + 1, + _ => panic!("should use the documented tail field"), + } + assert_eq!( + json!(project_auction_transport(&observed)), + json!(TraceAuctionTransportV1::unavailable()), + "should reject required invalid facts before tail truncation" + ); + } + let mut observed = observed_auction(); + observed.slots = vec![observed.slots[0].clone(); 65]; + observed.slots[64].requested_sizes = vec![[0, 0]]; + observed.slots[64].selected_creative_size = Some([0, 0]); + let projected = project_auction_transport(&observed); + let evidence = json!( + projected + .evidence() + .expect("should truncate optional tail facts") + ); + assert_eq!( + evidence["truncation"], + json!({"omitted_provider_calls":0,"omitted_slots":1,"omitted_nested_values":0}), + "should keep optional nested accounting within retained slots" + ); + } + + #[test] + fn trace_auction_projection_omission_overflow_never_wraps_or_saturates() { + for field in ["providers", "slots", "nested"] { + let mut observed = observed_auction(); + match field { + "providers" => observed.truncation.omitted_provider_calls = usize::from(u16::MAX), + "slots" => observed.truncation.omitted_slots = usize::from(u16::MAX), + "nested" => observed.truncation.omitted_nested_values = usize::from(u16::MAX), + _ => panic!("should use the documented omission field"), + } + assert!( + project_auction_transport(&observed).evidence().is_some(), + "should accept exact omission counter maximum" + ); + match field { + "providers" => { + observed.provider_calls = vec![observed.provider_calls[0].clone(); 17] + } + "slots" => observed.slots = vec![observed.slots[0].clone(); 65], + "nested" => observed.total_time = Some(Duration::MAX), + _ => panic!("should use the documented omission field"), + } + assert_eq!( + json!(project_auction_transport(&observed)), + json!(TraceAuctionTransportV1::unavailable()), + "should fail checked omission addition instead of saturation" + ); + } + for truncation in [ + ObservedTraceTruncation { + omitted_provider_calls: usize::from(u16::MAX) + 1, + ..ObservedTraceTruncation::default() + }, + ObservedTraceTruncation { + omitted_slots: usize::MAX, + ..ObservedTraceTruncation::default() + }, + ObservedTraceTruncation { + omitted_nested_values: usize::MAX, + ..ObservedTraceTruncation::default() + }, + ] { + let mut observed = observed_auction(); + observed.truncation = truncation; + assert!( + project_auction_transport(&observed).evidence().is_none(), + "should reject unrepresentable carried omissions" + ); + } + } + + #[test] + fn trace_auction_projection_counts_sizes_in_the_first_prefix_without_backfilling() { + let mut observed = observed_auction(); + observed.slots[0].requested_sizes = vec![[0, 0]; 16]; + observed.slots[0].requested_sizes.push([300, 250]); + let projected = project_auction_transport(&observed); + let evidence = json!( + projected + .evidence() + .expect("should omit malformed optional sizes") + ); + assert_eq!( + evidence["slots"][0]["requested_sizes"], + json!([]), + "should never backfill from beyond the requested-size prefix" + ); + assert_eq!( + evidence["truncation"]["omitted_nested_values"], + json!(17), + "should count invalid prefix and discarded tail separately" + ); + observed.slots[0].requested_sizes = vec![[300, 250]; usize::from(u16::MAX) + 17]; + assert!( + project_auction_transport(&observed).evidence().is_none(), + "should reject more than u16 omitted sizes" + ); + } +} diff --git a/crates/trusted-server-core/src/trace/carry.rs b/crates/trusted-server-core/src/trace/carry.rs new file mode 100644 index 000000000..87033a9be --- /dev/null +++ b/crates/trusted-server-core/src/trace/carry.rs @@ -0,0 +1,889 @@ +//! Private, request-local observations at the live auction boundary. + +use std::collections::{HashMap, HashSet}; +use std::sync::{Arc, Mutex}; + +use web_time::Instant; + +use crate::auction::types::{AdSlot, AuctionResponse, Bid, BidStatus}; + +use super::auction::projection::{ + ObservedTraceAuction, ObservedTraceProviderCall, ObservedTraceSlot, ObservedTraceTruncation, + project_auction_transport, +}; + +use super::auction::{ + DiagnosticAuctionId, TraceAuctionSource, TraceAuctionTerminalReason, + TraceAuctionTerminalStatus, TraceAuctionTransportV1, TraceProviderRole, TraceProviderStatus, + TraceSlotCandidate, TraceSlotRef, +}; + +const PROVIDER_LIMIT: usize = 16; +const SLOT_LIMIT: usize = 64; +const SIZE_LIMIT: usize = 16; + +/// Ephemeral facts; provider identities, bid bodies, and creatives are never retained. +#[derive(Clone)] +pub(crate) struct TraceAuctionCarry { + token: DiagnosticAuctionId, + state: Arc>, +} + +struct TraceAuctionState { + observed: ObservedTraceAuction, + started_at: Instant, + provider_starts: Vec, + provider_count: usize, + slot_keys: Vec, + slot_refs: Vec, + buckets: HashMap, + terminal: bool, + invalid: bool, + collection_failed: bool, + provider_failed: bool, + transport: Option, +} + +#[derive(Default)] +struct SlotBucket { + accepted_instances: usize, + returned_bid_count: u16, +} + +impl TraceAuctionState { + fn refresh_projection(&mut self) { + if self.terminal { + self.transport = Some(if self.invalid { + TraceAuctionTransportV1::unavailable() + } else { + project_auction_transport(&self.observed) + }); + } + } +} + +/// One observation is consumed at its actual response or failure boundary. +pub(crate) struct TraceProviderObservation { + number: Option, + started_at: Instant, +} + +/// Cancellation terminalizes synchronously without constructing telemetry. +pub(crate) struct TraceAuctionCancellationGuard { + carry: TraceAuctionCarry, + armed: bool, +} + +impl TraceAuctionCancellationGuard { + pub(crate) fn disarm(&mut self) { + self.armed = false; + } +} + +impl Drop for TraceAuctionCancellationGuard { + fn drop(&mut self) { + if self.armed { + self.carry.finish( + TraceAuctionTerminalStatus::Abandoned, + Some(TraceAuctionTerminalReason::Unknown), + ); + } + } +} + +impl TraceAuctionCarry { + pub(crate) fn capture_if_enabled( + active: bool, + source: TraceAuctionSource, + slots: &[AdSlot], + ) -> Option { + if !active { + return None; + } + let token = DiagnosticAuctionId::generate(); + let mut state = TraceAuctionState { + observed: ObservedTraceAuction { + diagnostic_auction_id: token.clone(), + source, + terminal_status: TraceAuctionTerminalStatus::Completed, + terminal_reason: None, + total_time: None, + provider_calls: Vec::new(), + slots: Vec::new(), + truncation: ObservedTraceTruncation::default(), + }, + started_at: Instant::now(), + provider_starts: Vec::new(), + provider_count: 0, + slot_keys: Vec::new(), + slot_refs: Vec::new(), + buckets: HashMap::new(), + terminal: false, + invalid: slots.len() > usize::from(u16::MAX), + collection_failed: false, + provider_failed: false, + transport: None, + }; + if !state.invalid { + for (index, slot) in slots.iter().enumerate() { + state + .buckets + .entry(slot.id.clone()) + .or_default() + .accepted_instances += 1; + let slot_ref = TraceSlotRef::generate(); + state.slot_refs.push(slot_ref.clone()); + if index >= SLOT_LIMIT { + state.observed.truncation.omitted_slots += 1; + continue; + } + let retained_sizes = slot.formats.len().min(SIZE_LIMIT); + let omitted_sizes = slot.formats.len() - retained_sizes; + if let Some(omitted) = state + .observed + .truncation + .omitted_nested_values + .checked_add(omitted_sizes) + { + state.observed.truncation.omitted_nested_values = omitted; + state.invalid |= omitted > usize::from(u16::MAX); + } else { + state.invalid = true; + } + state.slot_keys.push(slot.id.clone()); + state.observed.slots.push(ObservedTraceSlot { + slot_number: index + 1, + slot_ref, + requested_sizes: slot + .formats + .iter() + .take(SIZE_LIMIT) + .map(|format| [format.width, format.height]) + .collect(), + returned_bid_count: 0, + candidate: TraceSlotCandidate::NoCandidate, + selected_creative_size: None, + }); + } + } + let carry = Self { + token, + state: Arc::new(Mutex::new(state)), + }; + if slots.is_empty() { + carry.finish( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::NoEligibleSlots), + ); + } + Some(carry) + } + + pub(crate) fn token(&self) -> DiagnosticAuctionId { + self.token.clone() + } + + pub(crate) fn slot_ref(&self, index: usize) -> Option { + self.state.lock().ok()?.slot_refs.get(index).cloned() + } + + pub(crate) fn cancellation_guard(&self) -> TraceAuctionCancellationGuard { + TraceAuctionCancellationGuard { + carry: self.clone(), + armed: true, + } + } + + pub(crate) fn bind_client_refs(&self, refs: &[Option]) { + let Ok(mut state) = self.state.lock() else { + return; + }; + if refs.len() != state.slot_refs.len() { + return; + } + let mut seen = HashSet::new(); + let mut duplicates = HashSet::new(); + for token in refs.iter().flatten() { + if !seen.insert(token.clone()) { + duplicates.insert(token.clone()); + } + } + // Reserve even duplicated client refs so a fallback can never masquerade + // as a client association that was deliberately declined. + let mut used = seen; + for (index, client) in refs.iter().enumerate() { + let token = + if let Some(token) = client.as_ref().filter(|token| !duplicates.contains(*token)) { + token.clone() + } else { + let mut token = state.slot_refs[index].clone(); + while !used.insert(token.clone()) { + token = TraceSlotRef::generate(); + } + token + }; + state.slot_refs[index] = token.clone(); + if let Some(slot) = state.observed.slots.get_mut(index) { + slot.slot_ref = token; + } + } + state.refresh_projection(); + } + + pub(crate) fn observe_delivery( + &self, + winners: &HashMap, + delivered: &HashSet, + ) { + let Ok(mut state) = self.state.lock() else { + return; + }; + for index in 0..state.observed.slots.len() { + let key = &state.slot_keys[index]; + let ambiguous = state + .buckets + .get(key) + .is_none_or(|bucket| bucket.accepted_instances > 1); + let winner = (!ambiguous).then(|| winners.get(key)).flatten(); + let candidate = if ambiguous { + TraceSlotCandidate::Unknown + } else if winner.is_some() { + if delivered.contains(key) { + TraceSlotCandidate::Selected + } else { + TraceSlotCandidate::SelectedUnrenderable + } + } else { + TraceSlotCandidate::NoCandidate + }; + let selected_size = winner.map(|bid| [bid.width, bid.height]); + let slot = &mut state.observed.slots[index]; + slot.candidate = candidate; + slot.selected_creative_size = selected_size; + } + // Delivery is a later observation. It cannot remint identity or rewrite + // terminal status/time; pure projection also avoids accumulating omissions. + state.refresh_projection(); + } + + pub(crate) fn launch_provider(&self, role: TraceProviderRole) -> TraceProviderObservation { + let started_at = Instant::now(); + let mut provider = TraceProviderObservation { + number: None, + started_at, + }; + let Ok(mut state) = self.state.lock() else { + return provider; + }; + if state.terminal { + return provider; + } + let Some(number) = state.provider_count.checked_add(1) else { + state.invalid = true; + return provider; + }; + state.provider_count = number; + state.invalid |= number > usize::from(u16::MAX); + provider.number = Some(number); + if number <= PROVIDER_LIMIT { + state.provider_starts.push(started_at); + state + .observed + .provider_calls + .push(ObservedTraceProviderCall { + provider_number: number, + role, + status: TraceProviderStatus::Pending, + response_time: None, + returned_bid_count: 0, + }); + } else { + state.observed.truncation.omitted_provider_calls = number - PROVIDER_LIMIT; + } + provider + } + + pub(crate) fn observe_response( + &self, + provider: TraceProviderObservation, + response: &AuctionResponse, + ) { + let status = match response.status { + BidStatus::Success if response.bids.is_empty() => TraceProviderStatus::NoBid, + BidStatus::Success => TraceProviderStatus::Success, + BidStatus::NoBid => TraceProviderStatus::NoBid, + BidStatus::Error => TraceProviderStatus::Error, + BidStatus::Pending => TraceProviderStatus::Pending, + }; + self.record_provider_outcome( + provider, + status, + response.bids.len(), + response.bids.iter().map(|bid| bid.slot_id.as_str()), + ); + } + + pub(crate) fn observe_failure(&self, provider: TraceProviderObservation) { + self.record_provider_outcome(provider, TraceProviderStatus::Error, 0, std::iter::empty()); + } + + pub(crate) fn observe_collection_failure(&self) { + if let Ok(mut state) = self.state.lock() + && !state.terminal + { + state.collection_failed = true; + } + } + + // Consume the private observation so a completion cannot reuse its launch. + #[allow(clippy::needless_pass_by_value)] + fn record_provider_outcome<'a>( + &self, + provider: TraceProviderObservation, + status: TraceProviderStatus, + returned_bid_count: usize, + slot_ids: impl Iterator, + ) { + let Some(number) = provider.number else { + return; + }; + let Ok(mut state) = self.state.lock() else { + return; + }; + if state.terminal { + return; + } + state.invalid |= returned_bid_count > usize::from(u16::MAX); + state.provider_failed |= status == TraceProviderStatus::Error; + if let Some(observed) = state.observed.provider_calls.get_mut(number - 1) { + observed.status = status; + observed.response_time = Some(provider.started_at.elapsed()); + observed.returned_bid_count = returned_bid_count; + } + // Only accepted routing keys enter this bounded private table. Tail + // counts remain required source facts even when details are omitted. + for key in slot_ids { + if let Some(bucket) = state.buckets.get_mut(key) { + if let Some(count) = bucket.returned_bid_count.checked_add(1) { + bucket.returned_bid_count = count; + } else { + state.invalid = true; + } + } + } + } + + pub(crate) fn finish( + &self, + status: TraceAuctionTerminalStatus, + reason: Option, + ) { + let Ok(mut state) = self.state.lock() else { + return; + }; + if state.terminal { + return; + } + state.terminal = true; + state.observed.diagnostic_auction_id = self.token.clone(); + state.observed.terminal_status = + if status == TraceAuctionTerminalStatus::Completed && state.collection_failed { + TraceAuctionTerminalStatus::ExecutionFailed + } else { + status + }; + state.observed.terminal_reason = + if status == TraceAuctionTerminalStatus::Completed && state.collection_failed { + Some(TraceAuctionTerminalReason::CollectionFailed) + } else { + reason + }; + state.observed.total_time = Some(state.started_at.elapsed()); + for index in 0..state.observed.slots.len() { + let Some(bucket) = state.buckets.get(&state.slot_keys[index]) else { + continue; + }; + let count = usize::from(bucket.returned_bid_count); + let ambiguous = bucket.accepted_instances > 1; + let slot = &mut state.observed.slots[index]; + slot.returned_bid_count = count; + slot.candidate = if ambiguous || count > 0 { + TraceSlotCandidate::Unknown + } else { + TraceSlotCandidate::NoCandidate + }; + } + if status == TraceAuctionTerminalStatus::Abandoned { + for index in 0..state.observed.provider_calls.len() { + let duration = state.provider_starts[index].elapsed(); + let provider = &mut state.observed.provider_calls[index]; + if provider.status == TraceProviderStatus::Pending { + provider.status = TraceProviderStatus::Abandoned; + provider.response_time = Some(duration); + } + } + } + state.refresh_projection(); + } + + pub(crate) fn finish_dispatch_failed(&self, no_provider_launched: bool) { + let reason = if no_provider_launched { + TraceAuctionTerminalReason::NoProviderLaunched + } else if self.state.lock().is_ok_and(|state| state.provider_failed) { + TraceAuctionTerminalReason::ProviderExecutionFailed + } else { + TraceAuctionTerminalReason::Unknown + }; + self.finish(TraceAuctionTerminalStatus::DispatchFailed, Some(reason)); + } + + pub(crate) fn transport(&self) -> Option { + match self.state.lock() { + Ok(state) => state.transport.clone(), + Err(_) => Some(TraceAuctionTransportV1::unavailable()), + } + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use serde_json::{Value, json}; + + use crate::auction::types::{AdFormat, AdSlot, MediaType}; + + use super::*; + + fn slot(id: &str) -> AdSlot { + AdSlot { + id: id.to_string(), + formats: vec![AdFormat { + media_type: MediaType::Banner, + width: 300, + height: 250, + }], + floor_price: None, + targeting: HashMap::new(), + bidders: HashMap::new(), + } + } + + fn capture(slots: &[AdSlot]) -> TraceAuctionCarry { + TraceAuctionCarry::capture_if_enabled(true, TraceAuctionSource::AuctionApi, slots) + .expect("should capture a gated auction") + } + + fn transport(carry: &TraceAuctionCarry) -> Value { + serde_json::to_value( + carry + .transport() + .expect("should project a terminal auction"), + ) + .expect("should serialize the bounded transport") + } + + #[test] + fn trace_auction_identity_is_stable_across_terminal_outcomes() { + for (status, reason) in [ + (TraceAuctionTerminalStatus::Completed, None), + ( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::PolicySkipped), + ), + ( + TraceAuctionTerminalStatus::Skipped, + Some(TraceAuctionTerminalReason::NoEligibleSlots), + ), + ( + TraceAuctionTerminalStatus::DispatchFailed, + Some(TraceAuctionTerminalReason::NoProviderLaunched), + ), + ( + TraceAuctionTerminalStatus::ExecutionFailed, + Some(TraceAuctionTerminalReason::ProviderExecutionFailed), + ), + ( + TraceAuctionTerminalStatus::Abandoned, + Some(TraceAuctionTerminalReason::Unknown), + ), + ] { + let carry = capture(&[slot("example-slot")]); + assert!( + carry.slot_ref(0).is_some(), + "should retain the definitive accepted occurrence reference" + ); + let token = carry.token(); + let outer = carry.clone(); + assert!( + carry.transport().is_none(), + "should not invent a pending terminal result" + ); + + carry.finish(status, reason); + outer.finish( + TraceAuctionTerminalStatus::Abandoned, + Some(TraceAuctionTerminalReason::Unknown), + ); + + assert_eq!(outer.token(), token, "should retain the pre-dispatch token"); + let value = transport(&outer); + assert_eq!( + value["evidence"]["diagnostic_auction_id"], + token.as_str(), + "should use the same public token" + ); + assert_eq!( + value["evidence"]["terminal_status"], + json!(status), + "should terminalize idempotently" + ); + assert_eq!( + value["evidence"]["terminal_reason"], + json!(reason), + "should preserve the directly observed reason" + ); + } + } + + #[test] + fn trace_auction_disabled_allocates_no_carry() { + assert!( + TraceAuctionCarry::capture_if_enabled( + false, + TraceAuctionSource::AuctionApi, + &[slot("example-slot")] + ) + .is_none(), + "should not allocate trace state when the frozen gate is false" + ); + } + + #[test] + fn trace_auction_empty_definitive_slots_terminalize_without_dispatch() { + let carry = capture(&[]); + assert_eq!( + transport(&carry)["evidence"]["terminal_reason"], + "no_eligible_slots", + "should retain no-slot facts without fabricating telemetry or dispatch" + ); + } + + #[test] + fn trace_auction_provider_numbers_follow_launch_order() { + let carry = capture(&[slot("example-slot")]); + let first = carry.launch_provider(TraceProviderRole::Bidder); + let second = carry.launch_provider(TraceProviderRole::Bidder); + let third = carry.launch_provider(TraceProviderRole::Mediator); + + carry.record_provider_outcome(second, TraceProviderStatus::NoBid, 0, std::iter::empty()); + carry.record_provider_outcome( + third, + TraceProviderStatus::Success, + 2, + std::iter::repeat_n("example-slot", 2), + ); + carry.record_provider_outcome(first, TraceProviderStatus::Error, 0, std::iter::empty()); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + + let value = transport(&carry); + let providers = value["evidence"]["provider_calls"] + .as_array() + .expect("should retain provider facts"); + assert_eq!( + providers + .iter() + .map(|provider| provider["provider_number"].clone()) + .collect::>(), + vec![json!(1), json!(2), json!(3)], + "should number actual launches rather than completions" + ); + assert_eq!( + providers[0]["status"], "error", + "should retain the first call's disposition" + ); + assert_eq!( + providers[1]["status"], "no_bid", + "should retain a zero-bid response" + ); + assert_eq!( + providers[2]["role"], "mediator", + "should distinguish mediation without an identity" + ); + assert_eq!( + value["evidence"]["slots"][0]["returned_bid_count"], 2, + "should count actual returned records" + ); + } + + #[test] + fn trace_auction_checks_tail_bucket_counts_before_prefix_projection() { + let slots = (0..65) + .map(|index| slot(&format!("example-slot-{index}"))) + .collect::>(); + let carry = capture(&slots); + for _ in 0..4 { + let provider = carry.launch_provider(TraceProviderRole::Bidder); + carry.record_provider_outcome( + provider, + TraceProviderStatus::Success, + 20_000, + std::iter::repeat_n("example-slot-64", 20_000), + ); + } + carry.finish(TraceAuctionTerminalStatus::Completed, None); + assert_eq!( + transport(&carry), + json!({"schema_version":1,"unavailable_reason":"evidence_projection_failed"}), + "should reject a required count overflow outside the retained prefix" + ); + } + + #[test] + fn trace_auction_retains_valid_disjoint_tail_counts_without_global_saturation() { + let slots = (0..66) + .map(|index| slot(&format!("example-slot-{index}"))) + .collect::>(); + let carry = capture(&slots); + for id in ["example-slot-64", "example-slot-65"] { + let provider = carry.launch_provider(TraceProviderRole::Bidder); + carry.record_provider_outcome( + provider, + TraceProviderStatus::Success, + usize::from(u16::MAX), + std::iter::repeat_n(id, usize::from(u16::MAX)), + ); + } + carry.finish(TraceAuctionTerminalStatus::Completed, None); + let value = transport(&carry); + assert_eq!( + value["evidence"]["slots"] + .as_array() + .expect("should retain the prefix") + .len(), + 64, + "should preserve the public slot bound" + ); + assert_eq!( + value["evidence"]["truncation"]["omitted_slots"], 2, + "should count omitted definitive slots exactly" + ); + assert_eq!( + value["evidence"]["provider_calls"][0]["returned_bid_count"], + u16::MAX, + "should retain valid per-provider counts without global clamping" + ); + } + + #[test] + fn trace_auction_duplicate_keys_share_non_disjoint_counts_and_unknown_candidates() { + let carry = capture(&[slot("example-duplicate"), slot("example-duplicate")]); + let provider = carry.launch_provider(TraceProviderRole::Bidder); + carry.record_provider_outcome( + provider, + TraceProviderStatus::Success, + 1, + std::iter::once("example-duplicate"), + ); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + let value = transport(&carry); + let slots = value["evidence"]["slots"] + .as_array() + .expect("should retain both definitive slots"); + assert_ne!( + slots[0]["slot_ref"], slots[1]["slot_ref"], + "should retain distinct auction-local opaque refs" + ); + for slot in slots { + assert_eq!( + slot["returned_bid_count"], 1, + "should expose the shared bucket without splitting it" + ); + assert_eq!( + slot["candidate"], "unknown", + "should not infer a duplicate-key winner" + ); + } + } + + #[test] + fn trace_auction_abandonment_retains_launch_order_and_is_terminal() { + let carry = capture(&[slot("example-slot")]); + let _first = carry.launch_provider(TraceProviderRole::Bidder); + let _second = carry.launch_provider(TraceProviderRole::Mediator); + carry.finish( + TraceAuctionTerminalStatus::Abandoned, + Some(TraceAuctionTerminalReason::Unknown), + ); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + let value = transport(&carry); + assert_eq!( + value["evidence"]["terminal_status"], "abandoned", + "should not overwrite abandoned facts" + ); + assert_eq!( + value["evidence"]["provider_calls"][0]["status"], "abandoned", + "should synchronously terminalize outstanding calls" + ); + assert_eq!( + value["evidence"]["provider_calls"][1]["provider_number"], 2, + "should retain mediator launch order" + ); + } + fn selected_bid(id: &str) -> Bid { + serde_json::from_value(json!({"slot_id":id,"price":1.0,"currency":"USD","creative":"
Example creative
","bidder":"fictional-bidder","width":300,"height":250,"metadata":{}})).expect("should build a fictional selected bid") + } + + #[test] + fn trace_slot_conversion_client_refs_echo_only_unique_accepted_tokens() { + let first = TraceSlotRef::parse("ts-slot-00000000-0000-4000-8000-000000000001") + .expect("should parse the first client ref"); + let repeated = TraceSlotRef::parse("ts-slot-00000000-0000-4000-8000-000000000002") + .expect("should parse the repeated client ref"); + let carry = capture(&[ + slot("first"), + slot("duplicate"), + slot("duplicate"), + slot("missing"), + ]); + + carry.bind_client_refs(&[ + Some(first.clone()), + Some(repeated.clone()), + Some(repeated.clone()), + None, + ]); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + + let value = transport(&carry); + let slots = value["evidence"]["slots"] + .as_array() + .expect("should retain accepted slots"); + assert_eq!( + slots[0]["slot_ref"], + first.as_str(), + "should echo the exact unique accepted client ref" + ); + for slot in &slots[1..] { + assert_ne!( + slot["slot_ref"], + repeated.as_str(), + "should replace every repeated accepted token with a fresh server ref" + ); + } + assert_eq!( + slots + .iter() + .map(|slot| slot["slot_ref"] + .as_str() + .expect("should retain opaque string refs")) + .collect::>() + .len(), + 4, + "should keep all accepted occurrence refs distinct" + ); + } + + #[test] + fn trace_slot_conversion_delivery_reprojects_without_rewriting_terminal_facts() { + let carry = capture(&[ + slot("selected"), + slot("dropped"), + slot("lost"), + slot("duplicate"), + slot("duplicate"), + slot("invalid-size"), + ]); + let mut invalid_size = selected_bid("invalid-size"); + invalid_size.width = 0; + let bids = vec![ + selected_bid("selected"), + selected_bid("dropped"), + selected_bid("lost"), + selected_bid("duplicate"), + invalid_size, + ]; + let provider = carry.launch_provider(TraceProviderRole::Bidder); + carry.observe_response( + provider, + &AuctionResponse::success("fictional-provider", bids.clone(), 0), + ); + carry.finish(TraceAuctionTerminalStatus::Completed, None); + let before = transport(&carry); + let winners = bids + .into_iter() + .filter(|bid| bid.slot_id != "lost") + .map(|bid| (bid.slot_id.clone(), bid)) + .collect(); + + carry.observe_delivery( + &winners, + &HashSet::from([ + "selected".to_string(), + "duplicate".to_string(), + "invalid-size".to_string(), + ]), + ); + + let after = transport(&carry); + assert_eq!( + after["evidence"]["total_time_ms"], before["evidence"]["total_time_ms"], + "should preserve immutable terminal timing" + ); + assert_eq!( + after["evidence"]["terminal_status"], "completed", + "should preserve the observed terminal status" + ); + let slots = after["evidence"]["slots"] + .as_array() + .expect("should retain definitive slots"); + assert_eq!( + slots[0]["candidate"], "selected", + "should require actual delivery inclusion for selected" + ); + assert_eq!( + slots[0]["selected_creative_size"], + json!([300, 250]), + "should copy only the selected dimensions" + ); + assert_eq!( + slots[1]["candidate"], "selected_unrenderable", + "should preserve a winner omitted by response conversion" + ); + assert_eq!( + slots[2]["candidate"], "no_candidate", + "should preserve actual no-winner selection despite a returned bid" + ); + for slot in &slots[3..5] { + assert_eq!( + slot["candidate"], "unknown", + "should not infer an instance disposition from a duplicate routing key" + ); + assert!( + slot.get("selected_creative_size").is_none(), + "should omit ambiguous instance dimensions" + ); + } + assert!( + slots[5].get("selected_creative_size").is_none(), + "should omit malformed optional dimensions" + ); + assert_eq!( + after["evidence"]["truncation"]["omitted_nested_values"], 1, + "should count the malformed optional size exactly once" + ); + carry.observe_delivery( + &winners, + &HashSet::from([ + "selected".to_string(), + "duplicate".to_string(), + "invalid-size".to_string(), + ]), + ); + assert_eq!( + transport(&carry), + after, + "should reproject without accumulating omission counts or rewriting terminal timing" + ); + } +} diff --git a/crates/trusted-server-core/src/trace/context.rs b/crates/trusted-server-core/src/trace/context.rs new file mode 100644 index 000000000..c2c7648dc --- /dev/null +++ b/crates/trusted-server-core/src/trace/context.rs @@ -0,0 +1,362 @@ +//! Allowlisted, bounded request context projection. + +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; + +use chrono::{DateTime, Datelike as _, SecondsFormat, Timelike as _, Utc}; +use error_stack::Report; + +use crate::platform::{ClientInfo, GeoInfo}; + +use super::types::{TraceCookies, TraceNetwork, TraceRequestContextV1}; + +/// Bounded failure to represent the supplied capture clock in version one. +#[derive(Debug, derive_more::Display)] +#[display("invalid trace capture clock")] +pub struct ContextProjectionError; + +impl core::error::Error for ContextProjectionError {} + +/// Project frozen cookie health and trusted read-only network facts. +/// +/// Full IPs, fingerprints, city, coordinates, and identity services never enter +/// the returned context. Unsupported optional facts remain absent. Invalid +/// platform strings are omitted with fixed field categories and no values. +/// +/// # Errors +/// +/// Returns a bounded error for a capture clock outside four-digit UTC years or +/// with leap-second serialization, which the version-one schema does not accept. +/// +/// # Examples +/// +/// ``` +/// use chrono::Utc; +/// use http::HeaderMap; +/// use trusted_server_core::platform::ClientInfo; +/// use trusted_server_core::trace::{inspect_cookies, project_request_context}; +/// let cookies = inspect_cookies(&HeaderMap::new(), None); +/// let context = project_request_context(&ClientInfo::default(), None, &cookies, Utc::now())?; +/// assert!(!context.cookies().observed_active()); +/// # Ok::<(), error_stack::Report>(()) +/// ``` +/// +/// # Performance +/// +/// Projection copies only bounded optional strings and the fixed cookie health. +pub fn project_request_context( + client: &ClientInfo, + geo: Option<&GeoInfo>, + cookies: &TraceCookies, + captured_at: DateTime, +) -> Result> { + if !(0..=9999).contains(&captured_at.year()) || captured_at.nanosecond() >= 1_000_000_000 { + return Err(Report::new(ContextProjectionError)); + } + Ok(TraceRequestContextV1 { + schema_version: 1, + captured_at: captured_at.to_rfc3339_opts(SecondsFormat::Millis, true), + network: TraceNetwork { + masked_client_ip: client.client_ip.map(mask_ip), + country: bounded_string( + geo.map(|geo| geo.country.as_str()), + 2, + true, + "trace_network_country_omitted", + ), + region: bounded_string( + geo.and_then(|geo| geo.region.as_deref()), + 32, + false, + "trace_network_region_omitted", + ), + asn: geo.and_then(|geo| geo.asn), + tls_protocol: bounded_string( + client.tls_protocol.as_deref(), + 32, + false, + "trace_network_tls_protocol_omitted", + ), + tls_cipher: bounded_string( + client.tls_cipher.as_deref(), + 32, + false, + "trace_network_tls_cipher_omitted", + ), + edge_hostname: bounded_string( + client.server_hostname.as_deref(), + 128, + false, + "trace_network_edge_hostname_omitted", + ), + edge_region: bounded_string( + client.server_region.as_deref(), + 128, + false, + "trace_network_edge_region_omitted", + ), + }, + cookies: *cookies, + }) +} + +fn mask_ip(ip: IpAddr) -> String { + match ip { + IpAddr::V4(ip) => { + let [first, second, third, _] = ip.octets(); + format!("{}/24", Ipv4Addr::new(first, second, third, 0)) + } + IpAddr::V6(ip) => { + let [first, second, third, ..] = ip.segments(); + format!("{}/48", Ipv6Addr::new(first, second, third, 0, 0, 0, 0, 0)) + } + } +} + +fn bounded_string( + value: Option<&str>, + limit: usize, + ascii: bool, + category: &'static str, +) -> Option { + let value = value?; + if value.len() > limit || (ascii && !value.is_ascii()) || value.chars().any(|character| { + character.is_control() + || matches!(character, '\u{061c}' | '\u{200e}' | '\u{200f}' | '\u{202a}'..='\u{202e}' | '\u{2066}'..='\u{2069}') + }) { + log::warn!("{category}"); + None + } else { + Some(value.to_owned()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; + + use chrono::{TimeZone as _, Utc}; + use http::{HeaderMap, HeaderValue, header}; + use serde_json::json; + + use crate::platform::{ClientInfo, GeoInfo}; + use crate::trace::cookies::inspect_cookies; + + fn captured_at() -> chrono::DateTime { + Utc.with_ymd_and_hms(2026, 10, 5, 12, 34, 56) + .single() + .expect("should construct a valid fixed UTC capture clock") + } + + fn geo() -> GeoInfo { + GeoInfo { + city: "fictional-private-city".to_owned(), + country: "US".to_owned(), + continent: "fictional-private-continent".to_owned(), + latitude: 12.345, + longitude: 56.789, + metro_code: 123, + region: Some("CA".to_owned()), + asn: Some(64512), + } + } + + #[test] + fn trace_context_masks_and_omits_forbidden_fields() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + for (ip, mask) in [ + (IpAddr::V4(Ipv4Addr::new(192, 0, 2, 129)), "192.0.2.0/24"), + ( + IpAddr::V6( + "2001:db8:1234:5678::1" + .parse::() + .expect("should parse the documentation IPv6 address"), + ), + "2001:db8:1234::/48", + ), + (IpAddr::V6(Ipv6Addr::LOCALHOST), "::/48"), + ] { + let client = ClientInfo { + client_ip: Some(ip), + tls_protocol: Some("TLSv1.3".to_owned()), + tls_cipher: Some("TLS_AES_128_GCM_SHA256".to_owned()), + tls_ja4: Some("fictional-private-ja4".to_owned()), + h2_fingerprint: Some("fictional-private-h2".to_owned()), + server_hostname: Some("edge.example".to_owned()), + server_region: Some("example-region".to_owned()), + }; + let context = project_request_context(&client, Some(&geo()), &cookies, captured_at()) + .expect("should project valid read-only request facts"); + let actual = serde_json::to_value(context).expect("should serialize the context"); + assert_eq!( + actual, + json!({ + "schema_version": 1, + "captured_at": "2026-10-05T12:34:56.000Z", + "network": { + "masked_client_ip": mask, + "country": "US", + "region": "CA", + "asn": 64512, + "tls_protocol": "TLSv1.3", + "tls_cipher": "TLS_AES_128_GCM_SHA256", + "edge_hostname": "edge.example", + "edge_region": "example-region" + }, + "cookies": cookies + }), + "should emit only the exact network and context allowlist" + ); + let serialized = actual.to_string(); + assert!( + !serialized.contains("fictional-private") + && !serialized.contains("12.345") + && !serialized.contains("56.789") + && !serialized.contains(&ip.to_string()), + "should omit full IP, fingerprints, city and coordinates" + ); + } + } + + #[test] + fn missing_optional_facts_stay_absent_without_fallback_metadata() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + let actual = serde_json::to_value( + project_request_context(&ClientInfo::default(), None, &cookies, captured_at()) + .expect("should project unavailable optional facts"), + ) + .expect("should serialize the context"); + assert_eq!( + actual["network"], + json!({}), + "should not invent platform HTTP, POP, geo or IP fallback facts" + ); + } + + #[test] + fn invalid_optional_strings_are_omitted_without_shortening() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + for invalid in [ + "x".repeat(129), + "fictional\nprivate".to_owned(), + "fictional\u{0085}private".to_owned(), + "fictional\u{061c}private".to_owned(), + "fictional\u{202e}private".to_owned(), + "fictional\u{2069}private".to_owned(), + ] { + let client = ClientInfo { + tls_protocol: Some(invalid.clone()), + tls_cipher: Some(invalid.clone()), + server_hostname: Some(invalid.clone()), + server_region: Some(invalid.clone()), + ..ClientInfo::default() + }; + let mut geo = geo(); + geo.country = invalid.clone(); + geo.region = Some(invalid); + let actual = serde_json::to_value( + project_request_context(&client, Some(&geo), &cookies, captured_at()) + .expect("should omit invalid optional fields"), + ) + .expect("should serialize the context"); + assert_eq!( + actual["network"], + json!({"asn": 64512}), + "should omit failing fields instead of shortening or exposing values" + ); + } + } + + #[test] + fn optional_string_bounds_measure_utf8_bytes() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + for oversized in [false, true] { + let suffix = if oversized { "x" } else { "" }; + let short = format!("{}{suffix}", "é".repeat(16)); + let long = format!("{}{suffix}", "é".repeat(64)); + let client = ClientInfo { + tls_protocol: Some(short.clone()), + tls_cipher: Some(short.clone()), + server_hostname: Some(long.clone()), + server_region: Some(long.clone()), + ..ClientInfo::default() + }; + let mut geo = geo(); + geo.country = if oversized { + "é".to_owned() + } else { + "US".to_owned() + }; + geo.region = Some(short.clone()); + let actual = serde_json::to_value( + project_request_context(&client, Some(&geo), &cookies, captured_at()) + .expect("should apply per-field UTF-8 bounds"), + ) + .expect("should serialize the context"); + if oversized { + assert_eq!( + actual["network"], + json!({"asn": 64512}), + "should omit invalid ASCII or oversized UTF-8 fields" + ); + } else { + assert_eq!( + actual["network"], + json!({"asn": 64512, "country": "US", "region": short, "tls_protocol": short, "tls_cipher": short, "edge_hostname": long, "edge_region": long}), + "should retain values exactly at their byte limit" + ); + } + } + } + + #[test] + fn context_uses_frozen_cookie_inspection_after_header_sanitization() { + let mut headers = HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_static("__Host-ts-console=1; __Host-ts-console=invalid-example"), + ); + let cookies = inspect_cookies(&headers, None); + headers.remove(header::COOKIE); + let actual = serde_json::to_value( + project_request_context(&ClientInfo::default(), None, &cookies, captured_at()) + .expect("should project the frozen health"), + ) + .expect("should serialize the context"); + assert_eq!( + actual["cookies"]["diagnostics_session"], + json!({"source":"request", "state":"duplicate", "detail":"multiple_values"}), + "should never reread sanitized Cookie fields" + ); + } + + #[test] + fn invalid_capture_clock_representation_is_rejected() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + for year in [-1, 10000] { + let clock = Utc + .with_ymd_and_hms(year, 1, 1, 0, 0, 0) + .single() + .expect("should construct a supported chrono extended year"); + assert!( + project_request_context(&ClientInfo::default(), None, &cookies, clock).is_err(), + "should reject years outside the exact four-digit UTC schema" + ); + } + } + + #[test] + fn leap_second_clock_is_rejected_before_schema_serialization() { + let cookies = inspect_cookies(&HeaderMap::new(), None); + let clock = captured_at() + .with_second(59) + .expect("should set the last ordinary second") + .with_nanosecond(1_000_000_000) + .expect("should construct chrono's leap-second representation"); + assert!( + project_request_context(&ClientInfo::default(), None, &cookies, clock).is_err(), + "should reject second60 rather than serialize an invalid UTC schema timestamp" + ); + } +} diff --git a/crates/trusted-server-core/src/trace/cookies.rs b/crates/trusted-server-core/src/trace/cookies.rs new file mode 100644 index 000000000..2070d0504 --- /dev/null +++ b/crates/trusted-server-core/src/trace/cookies.rs @@ -0,0 +1,596 @@ +//! Read-only health inspection of runtime-visible Cookie fields. + +use edgezero_core::request::{Preservation, RequestIngress}; +use http::{HeaderMap, header}; + +use crate::ec::generation::is_valid_ec_id; +use crate::ec::prebid_eids::parse_prebid_eids_cookie; + +use super::types::{ + CookieHealth, InvalidCookieDetail, TraceCookies, UnavailableCookieDetail, ValidCookieDetail, +}; + +const MAX_COOKIE_HEADER_BYTES: usize = 16 * 1024; +const COOKIE_NAMES: [&str; 4] = ["ts-ec", "ts-eids", "ts-tester", "__Host-ts-console"]; +const VALUE_LIMITS: [usize; 4] = [512, 8 * 1024, 16, 16]; + +/// Freeze redacted health from complete runtime-visible Cookie fields. +/// +/// Aggregate size, actual UTF-8, then preservation ambiguity take precedence +/// over cookie-specific results. Missing fidelity does not reject readable +/// marker-free fields. This never mutates headers or accesses runtime services. +/// +/// # Examples +/// +/// ``` +/// use http::{HeaderMap, HeaderValue, header}; +/// let mut headers = HeaderMap::new(); +/// headers.insert(header::COOKIE, HeaderValue::from_static("__Host-ts-console=1")); +/// let cookies = trusted_server_core::trace::inspect_cookies(&headers, None); +/// assert!(cookies.observed_active()); +/// ``` +/// +/// # Performance +/// +/// Inspection is linear in visible bytes, bounded to 16 KiB before parsing. +/// Only the existing bounded EID parser allocates while validating values. +#[must_use] +pub fn inspect_cookies(headers: &HeaderMap, ingress: Option<&RequestIngress>) -> TraceCookies { + let fields = headers.get_all(header::COOKIE); + let total = fields.iter().fold(0usize, |total, value| { + total.saturating_add(value.as_bytes().len()) + }); + if total > MAX_COOKIE_HEADER_BYTES { + return unavailable(UnavailableCookieDetail::HeaderTooLarge); + } + if fields + .iter() + .any(|value| core::str::from_utf8(value.as_bytes()).is_err()) + { + return unavailable(UnavailableCookieDetail::HeaderNotUtf8); + } + let fidelity = ingress.map(|ingress| ingress.header_fidelity(&header::COOKIE)); + let octets_preserved = + fidelity.is_some_and(|fidelity| fidelity.octets() == Preservation::Preserved); + let multiplicity_preserved = + fidelity.is_some_and(|fidelity| fidelity.field_multiplicity() == Preservation::Preserved); + let text = || { + fields + .iter() + .filter_map(|value| core::str::from_utf8(value.as_bytes()).ok()) + }; + if text().any(|value| { + (!octets_preserved && value.contains('\u{fffd}')) + || (!multiplicity_preserved && value.contains(',')) + }) { + return unavailable(UnavailableCookieDetail::RuntimeHeaderAmbiguous); + } + let mut counts = [0usize; 4]; + let mut first_values = [None; 4]; + for field in text() { + for segment in field.split(';') { + let segment = segment.trim_matches(|character: char| character.is_ascii_whitespace()); + let (name, value) = match segment.split_once('=') { + Some((name, value)) => (name, Some(value)), + None => ( + segment.split_ascii_whitespace().next().unwrap_or_default(), + None, + ), + }; + let Some(index) = COOKIE_NAMES.iter().position(|reserved| *reserved == name) else { + continue; + }; + if counts[index] == 0 { + first_values[index] = value; + } + counts[index] += 1; + } + } + TraceCookies::new(std::array::from_fn(|index| match counts[index] { + 0 => CookieHealth::absent(), + 1 => validate_value(index, first_values[index]), + _ => CookieHealth::duplicate(), + })) +} + +fn unavailable(detail: UnavailableCookieDetail) -> TraceCookies { + TraceCookies::new([CookieHealth::unavailable(detail); 4]) +} + +fn validate_value(index: usize, value: Option<&str>) -> CookieHealth { + let Some(value) = value else { + return CookieHealth::invalid(InvalidCookieDetail::Malformed); + }; + if value.len() > VALUE_LIMITS[index] { + return CookieHealth::invalid(InvalidCookieDetail::Oversized); + } + let (valid, detail, invalid) = match index { + 0 => ( + is_valid_ec_id(value), + ValidCookieDetail::EcFormat, + InvalidCookieDetail::Malformed, + ), + 1 => ( + parse_prebid_eids_cookie(value).is_ok(), + ValidCookieDetail::EidsFormat, + InvalidCookieDetail::Malformed, + ), + 2 => ( + value == "true", + ValidCookieDetail::TesterValue, + InvalidCookieDetail::UnsupportedValue, + ), + _ => ( + value == "1", + ValidCookieDetail::DiagnosticsValue, + InvalidCookieDetail::UnsupportedValue, + ), + }; + if valid { + CookieHealth::valid(detail) + } else { + CookieHealth::invalid(invalid) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + use base64::{Engine as _, engine::general_purpose::STANDARD}; + use edgezero_core::request::{ + CapturedTarget, HeaderFidelity, Preservation, RequestIngress, TargetUnavailable, + }; + use http::{HeaderMap, HeaderValue, header}; + use serde_json::{Value, json}; + + const RESERVED: [(&str, &str, &str); 4] = [ + ("ts-ec", "ts_ec", "valid_ec_format"), + ("ts-eids", "ts_eids", "valid_eids_format"), + ("ts-tester", "ts_tester", "valid_tester_value"), + ( + "__Host-ts-console", + "diagnostics_session", + "valid_diagnostics_value", + ), + ]; + + fn headers(values: &[&[u8]]) -> HeaderMap { + let mut headers = HeaderMap::new(); + for value in values { + headers.append( + header::COOKIE, + HeaderValue::from_bytes(value).expect("should construct visible Cookie bytes"), + ); + } + headers + } + + fn ingress( + common: HeaderFidelity, + overrides: Vec<(http::HeaderName, HeaderFidelity)>, + ) -> RequestIngress { + RequestIngress::new( + CapturedTarget::Unavailable(TargetUnavailable::NotExposed), + None, + common, + overrides, + ) + .expect("should construct distinct field fidelity overrides") + } + + fn fidelity(octets: Preservation, multiplicity: Preservation) -> HeaderFidelity { + HeaderFidelity::new( + octets, + multiplicity, + Preservation::Unavailable, + Preservation::Transformed, + ) + } + + fn inspect(headers: &HeaderMap, ingress: Option<&RequestIngress>) -> Value { + serde_json::to_value(inspect_cookies(headers, ingress)) + .expect("should serialize redacted cookie health") + } + + fn health(state: &str, detail: Option<&str>) -> Value { + let mut health = json!({"state": state, "source": "request"}); + if let Some(detail) = detail { + health["detail"] = json!(detail); + } + health + } + + fn assert_all(actual: &Value, state: &str, detail: Option<&str>) { + for (_, key, _) in RESERVED { + assert_eq!( + actual[key], + health(state, detail), + "should give {key} the aggregate or absent health result" + ); + } + assert_eq!( + actual + .as_object() + .expect("should serialize an object") + .len(), + 4, + "should serialize only the four reserved cookie names" + ); + } + + fn valid_values() -> [String; 4] { + [ + format!("{}.aB1234", "a".repeat(64)), + STANDARD.encode( + serde_json::to_vec(&json!([ + {"source": "identity.example", "uids": [{"id": "fictional-private-eid"}]} + ])) + .expect("should encode the fictional EID payload"), + ), + "true".to_owned(), + "1".to_owned(), + ] + } + + #[test] + fn valid_reserved_values_use_only_their_named_detail() { + for ((name, key, detail), value) in RESERVED.into_iter().zip(valid_values()) { + let value = format!("{name}={value}"); + let headers = headers(&[value.as_bytes()]); + let before = headers.clone(); + let actual = inspect(&headers, None); + assert_eq!( + actual[key], + health("present_valid", Some(detail)), + "should use the canonical validator for {name}" + ); + for (_, other, _) in RESERVED { + if other != key { + assert_eq!( + actual[other], + health("absent", None), + "should not infer another cookie" + ); + } + } + assert_eq!( + headers, before, + "should not sanitize or mutate incoming fields" + ); + let serialized = actual.to_string(); + assert!( + !serialized.contains("fictional-private-eid") + && !serialized.contains(&"a".repeat(64)), + "should serialize shape without values or IDs" + ); + } + } + + #[test] + fn observed_active_requires_exactly_one_valid_frozen_session() { + for (field, expected) in [ + ("__Host-ts-console=1", true), + ("__Host-ts-console=invalid-example", false), + ("__Host-ts-console", false), + ("__Host-ts-console=1; __Host-ts-console=1", false), + ("__Host-ts-console=1; unrelated=a,b", false), + ("ts-tester=true", false), + ("", false), + ] { + let cookies = inspect_cookies(&headers(&[field.as_bytes()]), None); + assert_eq!( + cookies.observed_active(), + expected, + "should derive activity only from frozen session health" + ); + } + } + + #[test] + fn duplicates_win_over_malformed_valid_and_oversized_values() { + for ((name, key, _), valid) in RESERVED.into_iter().zip(valid_values()) { + for invalid in [ + "".to_owned(), + "invalid-example".to_owned(), + "x".repeat(9000), + ] { + let first = format!("{name}={valid}"); + let second = format!("{name}={invalid}"); + for fields in [ + vec![first.clone(), second.clone()], + vec![format!("{first}; {second}")], + vec![format!("{name}; {first}")], + vec![second.clone(), first.clone()], + ] { + let fields: Vec<&[u8]> = fields.iter().map(String::as_bytes).collect(); + let actual = inspect(&headers(&fields), None); + assert_eq!( + actual[key], + health("duplicate", Some("multiple_values")), + "should count every exact occurrence before validating {name}" + ); + } + } + } + } + + #[test] + fn grammar_counts_malformed_reserved_tokens_and_keeps_values_complete() { + for (name, key, _) in RESERVED { + for segment in [name.to_owned(), format!("{name} invalid-example")] { + let actual = inspect(&headers(&[segment.as_bytes()]), None); + assert_eq!( + actual[key], + health("present_invalid", Some("malformed")), + "should count no-equals reserved tokens exactly" + ); + } + for segment in [ + format!("{name}-extra"), + format!("{name} =true"), + name.to_uppercase(), + ] { + let actual = inspect(&headers(&[segment.as_bytes()]), None); + assert_all(&actual, "absent", None); + } + } + let actual = inspect(&headers(&[b" bad pair; =invalid; unrelated==value; ts-tester=true=extra; __Host-ts-console=1=extra"]), None); + assert_eq!( + actual["ts_tester"], + health("present_invalid", Some("unsupported_value")), + "should retain additional equals in the tester value" + ); + assert_eq!( + actual["diagnostics_session"], + health("present_invalid", Some("unsupported_value")), + "should retain additional equals in the session value" + ); + let actual = inspect( + &headers(&[b"\t ts-tester=true ; __Host-ts-console= 1"]), + None, + ); + assert_eq!( + actual["ts_tester"], + health("present_valid", Some("valid_tester_value")), + "should trim only whole-segment ASCII whitespace" + ); + assert_eq!( + actual["diagnostics_session"], + health("present_invalid", Some("unsupported_value")), + "should not normalize value whitespace after equals" + ); + } + + #[test] + fn canonical_rejections_and_visible_per_value_caps_remain_independent() { + for ((name, key, _), limit) in RESERVED.into_iter().zip([512, 8192, 16, 16]) { + for (value, detail) in [ + ( + "x".repeat(limit), + if key == "ts_tester" || key == "diagnostics_session" { + "unsupported_value" + } else { + "malformed" + }, + ), + ("x".repeat(limit + 1), "oversized"), + ("é".repeat(limit / 2 + 1), "oversized"), + ] { + let field = format!("{name}={value}"); + let actual = inspect(&headers(&[field.as_bytes()]), None); + assert_eq!( + actual[key], + health("present_invalid", Some(detail)), + "should enforce the UTF-8 value byte limit for {name}" + ); + for (_, other, _) in RESERVED { + if other != key { + assert_eq!( + actual[other], + health("absent", None), + "should keep unrelated health independent" + ); + } + } + } + } + let uppercase = format!("ts-ec={}.aB1234", "A".repeat(64)); + let actual = inspect(&headers(&[uppercase.as_bytes()]), None); + assert_eq!( + actual["ts_ec"], + health("present_invalid", Some("malformed")), + "should reuse canonical lowercase EC validation" + ); + } + + #[test] + fn visible_aggregate_cap_precedes_actual_utf8_and_ambiguity() { + for length in [16384, 16385] { + let field = format!("unrelated={}", "x".repeat(length - "unrelated=".len())); + let actual = inspect(&headers(&[field.as_bytes()]), None); + if length == 16384 { + assert_all(&actual, "absent", None); + } else { + assert_all(&actual, "unavailable", Some("header_too_large")); + } + } + let first = "x".repeat(8192); + let second = "x".repeat(8192); + assert_all( + &inspect(&headers(&[first.as_bytes(), second.as_bytes()]), None), + "absent", + None, + ); + assert_all( + &inspect(&headers(&[first.as_bytes(), second.as_bytes(), b"x"]), None), + "unavailable", + Some("header_too_large"), + ); + let oversized = "x".repeat(16385); + assert_all( + &inspect(&headers(&[b"\xff", b",", oversized.as_bytes()]), None), + "unavailable", + Some("header_too_large"), + ); + assert_all( + &inspect(&headers(&[b"\xff", b",", b"__Host-ts-console=1"]), None), + "unavailable", + Some("header_not_utf8"), + ); + } + + #[test] + fn all_non_preserved_statuses_and_missing_metadata_allow_marker_free_fields() { + for status in [ + Preservation::Unknown, + Preservation::Transformed, + Preservation::Unavailable, + ] { + let metadata = ingress(fidelity(status, status), vec![]); + for metadata in [None, Some(&metadata)] { + assert_all(&inspect(&HeaderMap::new(), metadata), "absent", None); + let actual = inspect( + &headers(&["__Host-ts-console=1; unrelated=readable-é".as_bytes()]), + metadata, + ); + assert_eq!( + actual["diagnostics_session"], + health("present_valid", Some("valid_diagnostics_value")), + "should preserve ordinary activation without ambiguity markers" + ); + } + } + } + + #[test] + fn ambiguity_markers_anywhere_suppress_all_cookie_results_without_comma_splitting() { + for status in [ + Preservation::Unknown, + Preservation::Transformed, + Preservation::Unavailable, + ] { + let metadata = ingress(fidelity(status, status), vec![]); + for field in [ + "__Host-ts-console=1; unrelated=�", + "__Host-ts-console=1; unrelated=\"�\"", + "__Host-ts-console=1, __Host-ts-console=1", + "__Host-ts-console=1; unrelated=\"a,b\"", + "__Host-ts-console=1; malformed-unrelated,", + ] { + for metadata in [None, Some(&metadata)] { + assert_all( + &inspect(&headers(&[field.as_bytes()]), metadata), + "unavailable", + Some("runtime_header_ambiguous"), + ); + } + } + } + } + + #[test] + fn cookie_specific_axes_are_independent_and_order_is_not_a_prerequisite() { + for (octets, multiplicity, field, expected) in [ + ( + Preservation::Preserved, + Preservation::Unknown, + "__Host-ts-console=1; unrelated=�", + "present_valid", + ), + ( + Preservation::Unknown, + Preservation::Preserved, + "__Host-ts-console=1; unrelated=a,b", + "present_valid", + ), + ( + Preservation::Unknown, + Preservation::Preserved, + "__Host-ts-console=1; unrelated=�", + "unavailable", + ), + ( + Preservation::Preserved, + Preservation::Unknown, + "__Host-ts-console=1; unrelated=a,b", + "unavailable", + ), + ] { + let metadata = ingress( + HeaderFidelity::default(), + vec![(header::COOKIE, fidelity(octets, multiplicity))], + ); + let actual = inspect(&headers(&[field.as_bytes()]), Some(&metadata)); + if expected == "unavailable" { + assert_all(&actual, "unavailable", Some("runtime_header_ambiguous")); + } else { + assert_eq!( + actual["diagnostics_session"], + health("present_valid", Some("valid_diagnostics_value")), + "should use only the relevant Cookie preservation axis" + ); + } + } + let preserved = fidelity(Preservation::Preserved, Preservation::Preserved); + let metadata = ingress( + preserved, + vec![(header::CONTENT_TYPE, HeaderFidelity::default())], + ); + let actual = inspect( + &headers(&["unrelated=é,�; __Host-ts-console=1".as_bytes()]), + Some(&metadata), + ); + assert_eq!( + actual["diagnostics_session"], + health("present_valid", Some("valid_diagnostics_value")), + "should permit preserved unrelated markers and valid other Unicode" + ); + let metadata = ingress( + HeaderFidelity::default(), + vec![(header::CONTENT_TYPE, preserved)], + ); + assert_all( + &inspect( + &headers(&["unrelated=�; __Host-ts-console=1".as_bytes()]), + Some(&metadata), + ), + "unavailable", + Some("runtime_header_ambiguous"), + ); + } + + #[test] + fn preserved_markers_follow_full_value_validation_and_exact_duplicate_counts() { + let metadata = ingress( + fidelity(Preservation::Preserved, Preservation::Preserved), + vec![], + ); + for (field, detail) in [ + ("__Host-ts-console=1, __Host-ts-console=1", "oversized"), + ("__Host-ts-console=1,x", "unsupported_value"), + ("__Host-ts-console=�", "unsupported_value"), + ] { + let actual = inspect(&headers(&[field.as_bytes()]), Some(&metadata)); + assert_eq!( + actual["diagnostics_session"], + health("present_invalid", Some(detail)), + "should never comma-split or select a partial session value" + ); + } + let actual = inspect( + &headers(&[b"__Host-ts-console=1", b"__Host-ts-console=1"]), + Some(&metadata), + ); + assert_eq!( + actual["diagnostics_session"], + health("duplicate", Some("multiple_values")), + "should count exact preserved fields independently of order fidelity" + ); + assert_all( + &inspect(&headers(&[b"unrelated=\xff"]), Some(&metadata)), + "unavailable", + Some("header_not_utf8"), + ); + } +} diff --git a/crates/trusted-server-core/src/trace/dispatch.rs b/crates/trusted-server-core/src/trace/dispatch.rs new file mode 100644 index 000000000..907a6fac5 --- /dev/null +++ b/crates/trusted-server-core/src/trace/dispatch.rs @@ -0,0 +1,470 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use chrono::Utc; +use edgezero_core::body::Body; +use edgezero_core::error::EdgeError; +use edgezero_core::request::RequestIngress; +use edgezero_core::router::PreDispatchHook; +use http::{Method, Request, Response, StatusCode}; +use trusted_server_js::trace_assets::trace_asset; + +use super::actions::action_response; +use super::routes::{TraceRoute, state_response}; +use super::shell::render_setup_shell; +use super::{TraceCookies, TracePreflight, inspect_cookies, preflight, project_request_context}; +use crate::integrations::gpt_diagnostics::{ + GPT_DIAGNOSTICS_INTEGRATION_ID, GptDiagnosticsConfig, GptDiagnosticsRequestDecision, +}; +use crate::platform::{ClientInfo, GeoInfo}; +use crate::settings::Settings; + +/// Immutable incoming-cookie gate captured before ordinary request preparation. +#[derive(Clone, Copy, Debug)] +pub struct TraceCaptureGate { + cookies: TraceCookies, + base_active: bool, +} + +impl TraceCaptureGate { + /// Return cookie health frozen at the runtime-visible request boundary. + /// + /// # Examples + /// + /// ```ignore + /// let observed = gate.cookies().observed_active(); + /// ``` + #[must_use] + pub fn cookies(&self) -> &TraceCookies { + &self.cookies + } + + /// Whether the incoming session permits request-scoped trace evidence. + /// + /// # Examples + /// + /// ```ignore + /// if gate.base_active() { capture_optional_evidence(); } + /// ``` + #[must_use] + pub fn base_active(&self) -> bool { + self.base_active + } + + /// Combine frozen session validity with the existing navigation decision. + /// + /// # Examples + /// + /// ```ignore + /// let eligible = gate.document_eligible(&diagnostics_decision); + /// ``` + #[must_use] + pub fn document_eligible(&self, decision: &GptDiagnosticsRequestDecision) -> bool { + self.base_active && decision.active() + } +} + +/// Read-only platform facts supplied only for an authenticated setup GET. +pub struct TraceMetadata { + /// Trusted client connection metadata. + pub client_info: ClientInfo, + /// Optional read-only geolocation result. + pub geo: Option, +} + +type MetadataSupplier = dyn Fn(&Request) -> TraceMetadata + Send + Sync; + +/// Intercept the trace namespace before ordinary router lookup and middleware. +pub struct TracePreDispatchHook { + settings: Arc, + metadata: Arc, +} + +impl TracePreDispatchHook { + /// Capture application settings and a lazy read-only metadata supplier. + /// + /// # Examples + /// + /// ```ignore + /// let hook = TracePreDispatchHook::new(settings, Arc::new(read_only_metadata)); + /// ``` + /// + /// # Performance + /// + /// Metadata is requested only for authenticated setup-page GET requests. + #[must_use] + pub fn new(settings: Arc, metadata: Arc) -> Self { + Self { settings, metadata } + } +} + +#[async_trait(?Send)] +impl PreDispatchHook for TracePreDispatchHook { + async fn handle( + &self, + request: &mut Request, + ) -> Result>, EdgeError> { + let dispatch = match preflight(&self.settings, request) { + TracePreflight::NotTrace => { + let enabled = self + .settings + .integration_config::(GPT_DIAGNOSTICS_INTEGRATION_ID) + .ok() + .flatten() + .is_some_and(|config| config.enabled && config.trace_page_enabled); + if enabled { + let cookies = inspect_cookies( + request.headers(), + request.extensions().get::(), + ); + request.extensions_mut().insert(TraceCaptureGate { + base_active: cookies.observed_active(), + cookies, + }); + } + return Ok(None); + } + TracePreflight::Response(response) => return Ok(Some(response)), + TracePreflight::Ready(dispatch) => dispatch, + }; + let response = match dispatch.route { + TraceRoute::Javascript | TraceRoute::Stylesheet => { + return Ok(Some(match trace_asset(request.uri().path()) { + Some(asset) => dispatch.fixed_asset(asset.bytes), + None => dispatch.respond(unavailable_response()), + })); + } + TraceRoute::Enable | TraceRoute::End => action_response(request, dispatch.route).await, + TraceRoute::State => { + let cookies = inspect_cookies( + request.headers(), + request.extensions().get::(), + ); + state_response(&cookies) + } + TraceRoute::Shell if request.method() == Method::HEAD => Response::new(Body::empty()), + TraceRoute::Shell => { + let cookies = inspect_cookies( + request.headers(), + request.extensions().get::(), + ); + let metadata = (self.metadata)(request); + match project_request_context( + &metadata.client_info, + metadata.geo.as_ref(), + &cookies, + Utc::now(), + ) { + Ok(context) => match render_setup_shell(&context) { + Ok(shell) => Response::new(Body::from(shell)), + Err(_) => unavailable_response(), + }, + Err(_) => unavailable_response(), + } + } + }; + Ok(Some(dispatch.respond(response))) + } +} + +fn unavailable_response() -> Response { + let mut response = Response::new(Body::from(br#"{"error":"trace unavailable"}"#.as_slice())); + *response.status_mut() = StatusCode::INTERNAL_SERVER_ERROR; + response +} + +#[cfg(test)] +mod tests { + use std::sync::atomic::{AtomicUsize, Ordering}; + + use edgezero_core::context::RequestContext; + use edgezero_core::middleware::{Middleware, Next}; + use edgezero_core::router::RouterService; + use futures::executor::block_on; + use http::{Method, StatusCode, header}; + use serde_json::json; + + use crate::test_support::tests::create_test_settings; + use crate::trace::TraceTerminalResponse; + + use super::*; + + fn hook(enabled: bool, calls: Arc) -> TracePreDispatchHook { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &json!({"enabled": true,"trace_page_enabled": enabled}), + ) + .expect("should insert trace settings"); + TracePreDispatchHook::new( + Arc::new(settings), + Arc::new(move |_| { + calls.fetch_add(1, Ordering::SeqCst); + TraceMetadata { + client_info: ClientInfo::default(), + geo: None, + } + }), + ) + } + + #[test] + fn trace_dispatch_terminal_routes_only_project_get_shell_metadata() { + let calls = Arc::new(AtomicUsize::new(0)); + let hook = hook(true, Arc::clone(&calls)); + for (method, path, status, projected) in [ + (Method::GET, "/_ts/trace", StatusCode::OK, 1), + (Method::HEAD, "/_ts/trace", StatusCode::OK, 0), + (Method::GET, "/_ts/trace/state", StatusCode::OK, 0), + (Method::GET, "/_ts/trace/assets/v1.js", StatusCode::OK, 0), + (Method::GET, "/_ts/trace/assets/v1.css", StatusCode::OK, 0), + (Method::POST, "/_ts/trace/enable", StatusCode::FORBIDDEN, 0), + (Method::POST, "/_ts/trace/end", StatusCode::FORBIDDEN, 0), + ( + Method::PATCH, + "/_ts/trace", + StatusCode::METHOD_NOT_ALLOWED, + 0, + ), + (Method::GET, "/_ts/trace/extra", StatusCode::NOT_FOUND, 0), + (Method::GET, "/%5Fts/trace", StatusCode::BAD_REQUEST, 0), + ] { + calls.store(0, Ordering::SeqCst); + let is_head = method == Method::HEAD; + let mut request = Request::builder() + .method(method) + .uri(path) + .body(Body::empty()) + .expect("should build trace request"); + let response = block_on(hook.handle(&mut request)) + .expect("should return local policy response") + .expect("should intercept namespace before lookup"); + assert_eq!( + response.status(), + status, + "should dispatch the exact trace route" + ); + assert_eq!( + calls.load(Ordering::SeqCst), + projected, + "should lazily project metadata only for GET shell" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should bypass ordinary finalization" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should not mutate cookies on read or rejected action" + ); + if is_head { + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer shell HEAD") + .len(), + 0, + "should remove the HEAD body" + ); + } + } + } + + #[test] + fn trace_dispatch_disabled_namespace_avoids_metadata() { + let calls = Arc::new(AtomicUsize::new(0)); + let hook = hook(false, Arc::clone(&calls)); + let mut request = Request::builder() + .uri("/_ts/trace") + .body(Body::empty()) + .expect("should build disabled request"); + let response = block_on(hook.handle(&mut request)) + .expect("should produce local response") + .expect("should reserve disabled namespace"); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should hide disabled trace routes" + ); + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "should never fetch disabled metadata" + ); + } + + #[test] + fn trace_dispatch_freezes_ordinary_cookie_gate_before_preparation() { + for (enabled, cookie, expected) in [ + (true, "__Host-ts-console=1", Some(true)), + (true, "__Host-ts-console=1, unrelated=value", Some(false)), + ( + true, + "__Host-ts-console=1; __Host-ts-console=1", + Some(false), + ), + (true, "__Host-ts-console=invalid", Some(false)), + (false, "__Host-ts-console=1", None), + ] { + let calls = Arc::new(AtomicUsize::new(0)); + let hook = hook(enabled, Arc::clone(&calls)); + let mut request = Request::builder() + .uri("/article") + .header(header::COOKIE, cookie) + .body(Body::empty()) + .expect("should build ordinary request"); + assert!( + block_on(hook.handle(&mut request)) + .expect("should continue ordinary dispatch") + .is_none(), + "should continue ordinary requests" + ); + let gate = request.extensions().get::().copied(); + assert_eq!( + gate.map(|gate| gate.base_active()), + expected, + "should freeze conservative incoming validity only when configured" + ); + request.headers_mut().remove(header::COOKIE); + assert_eq!( + gate.map(|gate| gate.cookies().observed_active()), + expected, + "should retain no dependency on sanitized headers" + ); + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "should never project ordinary metadata before auth" + ); + } + } + + struct OrdinaryLifecycleCounter(Arc); + + #[async_trait(?Send)] + impl Middleware for OrdinaryLifecycleCounter { + async fn handle( + &self, + context: RequestContext, + next: Next<'_>, + ) -> Result, EdgeError> { + self.0.fetch_add(1, Ordering::SeqCst); + next.run(context).await + } + } + + #[test] + fn trace_dispatch_router_hook_bypasses_all_ordinary_middleware_and_handlers() { + let ordinary = Arc::new(AtomicUsize::new(0)); + let metadata = Arc::new(AtomicUsize::new(0)); + let handler_calls = Arc::clone(&ordinary); + let router = RouterService::builder() + .pre_dispatch_hook(Arc::new(hook(true, Arc::clone(&metadata)))) + .middleware(OrdinaryLifecycleCounter(Arc::clone(&ordinary))) + .route("/{*rest}", Method::GET, move |_context: RequestContext| { + let handler_calls = Arc::clone(&handler_calls); + async move { + handler_calls.fetch_add(1, Ordering::SeqCst); + Ok::<_, EdgeError>(Response::new(Body::empty())) + } + }) + .build(); + for path in [ + "/_ts/trace", + "/_ts/trace/state", + "/_ts/trace/assets/v1.js", + "/_ts/trace/extra", + ] { + let request = Request::builder() + .uri(path) + .body(Body::empty()) + .expect("should build trace route"); + let response = + block_on(router.oneshot(request)).expect("should serve local trace response"); + assert!( + response + .extensions() + .get::() + .is_some(), + "should terminate before ordinary lifecycle" + ); + } + assert_eq!( + ordinary.load(Ordering::SeqCst), + 0, + "should bypass every ordinary middleware and publisher handler" + ); + assert_eq!( + metadata.load(Ordering::SeqCst), + 1, + "should project only the authenticated setup GET" + ); + let request = Request::builder() + .uri("/article") + .body(Body::empty()) + .expect("should build ordinary route"); + let _ = block_on(router.oneshot(request)).expect("should preserve ordinary lifecycle"); + assert_eq!( + ordinary.load(Ordering::SeqCst), + 2, + "should run middleware and handler for ordinary requests" + ); + } + + #[test] + fn trace_dispatch_document_gate_combines_incoming_session_and_effective_decision() { + for (cookie, query, expected_base, expected_document) in [ + (None, "?ts_console=1", false, false), + (Some("__Host-ts-console=1"), "", true, true), + (Some("__Host-ts-console=1"), "?ts_console=0", true, false), + ( + Some("__Host-ts-console=1, unrelated=value"), + "?ts_console=1", + false, + false, + ), + ] { + let hook = hook(true, Arc::new(AtomicUsize::new(0))); + let mut request = Request::builder() + .uri(format!("https://publisher.example.com/article{query}")) + .header("accept", "text/html") + .body(Body::empty()) + .expect("should build navigation"); + if let Some(cookie) = cookie { + request.headers_mut().insert( + header::COOKIE, + http::HeaderValue::from_str(cookie).expect("should parse example cookie"), + ); + } + let _ = + block_on(hook.handle(&mut request)).expect("should continue ordinary navigation"); + let gate = *request + .extensions() + .get::() + .expect("should freeze incoming gate"); + let _ = + crate::integrations::gpt_diagnostics::prepare_request(&hook.settings, &mut request) + .expect("should preserve console preparation"); + let decision = request + .extensions() + .get::() + .expect("should store existing decision"); + assert_eq!( + gate.base_active(), + expected_base, + "should evaluate incoming session only once" + ); + assert_eq!( + gate.document_eligible(decision), + expected_document, + "should combine frozen validity with effective navigation eligibility" + ); + } + } +} diff --git a/crates/trusted-server-core/src/trace/mod.rs b/crates/trusted-server-core/src/trace/mod.rs new file mode 100644 index 000000000..ac3ca7590 --- /dev/null +++ b/crates/trusted-server-core/src/trace/mod.rs @@ -0,0 +1,38 @@ +//! Application-visible mobile trace routing and response policy. + +mod actions; +mod auction; +mod carry; +mod slot_refs; +pub(crate) use carry::{TraceAuctionCarry, TraceProviderObservation}; +pub(crate) use slot_refs::TraceClientSlotRefs; +mod context; +mod cookies; +mod dispatch; +mod routes; +mod shell; +mod types; + +#[cfg(test)] +pub(crate) use actions::action_response; +pub use auction::{ + DiagnosticAuctionId, TraceAuctionEvidenceV1, TraceAuctionSource, TraceAuctionTerminalReason, + TraceAuctionTerminalStatus, TraceAuctionTransportV1, TraceProviderRole, TraceProviderStatus, + TraceSlotCandidate, TraceSlotRef, TraceTokenError, +}; +pub use context::{ContextProjectionError, project_request_context}; +pub use cookies::inspect_cookies; +pub use dispatch::{TraceCaptureGate, TraceMetadata, TracePreDispatchHook}; +#[cfg(test)] +pub(crate) use routes::state_response; +pub use routes::{TraceDispatch, TracePreflight, is_trace_path, preflight}; +pub use types::{ + CookieHealth, CookieHealthDetail, CookieHealthState, TraceCookies, TraceNetwork, + TraceRequestContextV1, +}; + +/// Marks a local trace response that bypasses ordinary adapter finalization. +/// +/// Adapters must preserve the trace response contract through conversion. +#[derive(Clone, Copy, Debug)] +pub struct TraceTerminalResponse; diff --git a/crates/trusted-server-core/src/trace/routes.rs b/crates/trusted-server-core/src/trace/routes.rs new file mode 100644 index 000000000..01a9a96cb --- /dev/null +++ b/crates/trusted-server-core/src/trace/routes.rs @@ -0,0 +1,1100 @@ +use std::borrow::Cow; + +use edgezero_core::body::Body as EdgeBody; +use http::{HeaderValue, Method, Request, Response, StatusCode, header}; +use sha2::{Digest as _, Sha256}; + +use crate::auth::{TraceAuthLogPolicy, enforce_basic_auth}; +use crate::integrations::gpt_diagnostics::{GPT_DIAGNOSTICS_INTEGRATION_ID, GptDiagnosticsConfig}; +use crate::response_privacy::enforce_terminal_private_cache_privacy; +use crate::settings::Settings; + +use super::TraceTerminalResponse; + +const NAMESPACE: &str = "/_ts/trace"; +const MAX_DECODE_ROUNDS: usize = 4; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum TraceRoute { + Shell, + State, + Enable, + End, + Javascript, + Stylesheet, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum Classification { + NotTrace, + Route(TraceRoute), + Rejected(StatusCode), +} + +/// Result of authenticated trace-route preflight. +pub enum TracePreflight { + /// Continue ordinary dispatch without inspecting diagnostics configuration. + NotTrace, + /// Terminate with a hardened local challenge or policy error. + Response(Response), + /// Invoke the route handler and finalize its response with this context. + Ready(TraceDispatch), +} + +/// Authenticated route context for a later trace handler. +/// +/// This is a response-policy seam, not a successful action or cookie observation. +pub struct TraceDispatch { + pub(crate) route: TraceRoute, + head: bool, + protected: bool, +} + +impl TraceDispatch { + /// Apply dynamic trace response policy after a handler produces its result. + /// + /// Successful assets require [`Self::fixed_asset`] instead. Errors are always + /// JSON and private; GET and HEAD responses cannot set cookies. + /// + /// # Examples + /// + /// ```ignore + /// let response = context.respond(handler_response); + /// ``` + #[must_use] + pub fn respond(self, mut response: Response) -> Response { + if self.route.is_asset() && response.status().is_success() { + response = error_response(StatusCode::INTERNAL_SERVER_ERROR); + } + let mime = if self.route == TraceRoute::Shell && response.status().is_success() { + "text/html; charset=utf-8" + } else { + "application/json; charset=utf-8" + }; + if !matches!(self.route, TraceRoute::Enable | TraceRoute::End) + || !response.status().is_success() + { + response.headers_mut().remove(header::SET_COOKIE); + } + harden_response(&mut response, mime, self.head); + response + } + + /// Build an asset response from immutable build bytes. + /// + /// This method does not locate an asset or manufacture a missing build. + /// An asset lookup must supply its verified bytes. Non-asset contexts fail + /// with a fixed private error. + /// + /// # Examples + /// + /// ```ignore + /// let response = context.fixed_asset(verified_asset_bytes); + /// ``` + /// + /// # Panics + /// + /// Panics if a quoted ASCII SHA-256 digest cannot be represented as a header. + /// The digest representation always satisfies that invariant. + #[must_use] + pub fn fixed_asset(self, bytes: &'static [u8]) -> Response { + let mime = match self.route { + TraceRoute::Javascript => "application/javascript; charset=utf-8", + TraceRoute::Stylesheet => "text/css; charset=utf-8", + _ => return self.respond(error_response(StatusCode::INTERNAL_SERVER_ERROR)), + }; + let mut response = Response::new(EdgeBody::from(bytes)); + harden_response(&mut response, mime, self.head); + if !self.protected { + response.headers_mut().insert( + header::CACHE_CONTROL, + HeaderValue::from_static("public, max-age=31536000, immutable"), + ); + response + .extensions_mut() + .remove::(); + } + // Private assets also retain their byte-derived validator. The shared + // privacy helper removes validators from dynamic responses first. + let etag = format!("\"{:x}\"", Sha256::digest(bytes)); + response.headers_mut().insert( + header::ETAG, + HeaderValue::from_str(&etag).expect("should encode an ASCII SHA-256 ETag"), + ); + response + } +} + +impl TraceRoute { + fn is_asset(self) -> bool { + matches!(self, Self::Javascript | Self::Stylesheet) + } + + fn allows(self, method: &Method) -> bool { + if matches!(self, Self::Enable | Self::End) { + method == Method::POST + } else { + method == Method::GET || method == Method::HEAD + } + } + + fn allow(self) -> &'static str { + if matches!(self, Self::Enable | Self::End) { + "POST" + } else { + "GET, HEAD" + } + } +} + +fn error_response(status: StatusCode) -> Response { + let body: &'static [u8] = match status { + StatusCode::BAD_REQUEST => br#"{"error":"invalid trace path"}"#, + StatusCode::UNAUTHORIZED => br#"{"error":"unauthorized"}"#, + StatusCode::NOT_FOUND => br#"{"error":"trace route not found"}"#, + StatusCode::METHOD_NOT_ALLOWED => br#"{"error":"method not allowed"}"#, + _ => br#"{"error":"trace unavailable"}"#, + }; + let mut response = Response::new(EdgeBody::from(body)); + *response.status_mut() = status; + response +} + +/// Build the exact observed state from frozen cookie health after preflight. +/// +/// Apply [`TraceDispatch::respond`] for dynamic privacy and HEAD body removal. +/// A false result reports no valid observed session, never cookie absence. +pub(crate) fn state_response(cookies: &super::TraceCookies) -> Response { + let body: &'static [u8] = if cookies.observed_active() { + br#"{"observed_active":true}"# + } else { + br#"{"observed_active":false}"# + }; + Response::new(EdgeBody::from(body)) +} + +fn harden_response(response: &mut Response, mime: &'static str, head: bool) { + enforce_terminal_private_cache_privacy(response); + let headers = response.headers_mut(); + headers.insert(header::CONTENT_TYPE, HeaderValue::from_static(mime)); + headers.insert( + header::X_CONTENT_TYPE_OPTIONS, + HeaderValue::from_static("nosniff"), + ); + headers.insert( + header::REFERRER_POLICY, + HeaderValue::from_static("no-referrer"), + ); + headers.insert(header::CONTENT_SECURITY_POLICY, HeaderValue::from_static("default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self'; img-src data:")); + headers.insert( + "permissions-policy", + HeaderValue::from_static("camera=(), microphone=(), geolocation=(), payment=(), usb=()"), + ); + response.extensions_mut().insert(TraceTerminalResponse); + if head { + *response.body_mut() = EdgeBody::empty(); + } +} + +fn reject(mut response: Response, head: bool) -> TracePreflight { + response.headers_mut().remove(header::SET_COOKIE); + harden_response(&mut response, "application/json; charset=utf-8", head); + TracePreflight::Response(response) +} + +/// Return whether the visible pathname is reserved for local trace handling. +/// +/// Encoded aliases are reserved for rejection and never become supported routes. +/// No original-target metadata participates in this decision. +/// +/// # Examples +/// +/// ``` +/// assert!(trusted_server_core::trace::is_trace_path("/_ts/trace")); +/// assert!(!trusted_server_core::trace::is_trace_path("/article")); +/// ``` +/// +/// # Performance +/// +/// Ordinary paths without encoded or ambiguous segments avoid allocations. +#[must_use] +pub fn is_trace_path(path: &str) -> bool { + classify_path(path) != Classification::NotTrace +} + +/// Authenticate and validate a visible trace route before handler dispatch. +/// +/// A ready result does not observe cookies, mutate a session, or supply assets. +/// Route handlers implement those contracts in subsequent layers. +/// +/// # Examples +/// +/// ```ignore +/// match trusted_server_core::trace::preflight(&settings, &mut request) { +/// trusted_server_core::trace::TracePreflight::Response(response) => return response, +/// trusted_server_core::trace::TracePreflight::NotTrace => continue_dispatch(request), +/// trusted_server_core::trace::TracePreflight::Ready(context) => handle_trace(context, request), +/// } +/// ``` +#[must_use] +pub fn preflight(settings: &Settings, request: &mut Request) -> TracePreflight { + let classification = classify_path(request.uri().path()); + if classification == Classification::NotTrace { + return TracePreflight::NotTrace; + } + let head = request.method() == Method::HEAD; + request.extensions_mut().insert(TraceAuthLogPolicy); + match enforce_basic_auth(settings, request) { + Ok(Some(mut response)) => { + *response.body_mut() = error_response(StatusCode::UNAUTHORIZED).into_body(); + return reject(response, head); + } + Err(_) => return reject(error_response(StatusCode::INTERNAL_SERVER_ERROR), head), + Ok(None) => {} + } + let enabled = + match settings.integration_config::(GPT_DIAGNOSTICS_INTEGRATION_ID) { + Ok(Some(config)) => config.trace_page_enabled, + Ok(None) => false, + Err(_) => return reject(error_response(StatusCode::INTERNAL_SERVER_ERROR), head), + }; + if !enabled { + return reject(error_response(StatusCode::NOT_FOUND), head); + } + let route = match classification { + Classification::Route(route) => route, + Classification::Rejected(status) => return reject(error_response(status), head), + Classification::NotTrace => return TracePreflight::NotTrace, + }; + if !route.allows(request.method()) { + let mut response = error_response(StatusCode::METHOD_NOT_ALLOWED); + response + .headers_mut() + .insert(header::ALLOW, HeaderValue::from_static(route.allow())); + return reject(response, head); + } + let protected = match settings.handler_for_path(request.uri().path()) { + Ok(handler) => handler.is_some(), + Err(_) => return reject(error_response(StatusCode::INTERNAL_SERVER_ERROR), head), + }; + TracePreflight::Ready(TraceDispatch { + route, + head, + protected, + }) +} + +fn exact_route(path: &str) -> Option { + match path { + "/_ts/trace" => Some(TraceRoute::Shell), + "/_ts/trace/state" => Some(TraceRoute::State), + "/_ts/trace/enable" => Some(TraceRoute::Enable), + "/_ts/trace/end" => Some(TraceRoute::End), + "/_ts/trace/assets/v1.js" => Some(TraceRoute::Javascript), + "/_ts/trace/assets/v1.css" => Some(TraceRoute::Stylesheet), + _ => None, + } +} + +fn has_dot_segments(path: &str) -> bool { + path.split('/').any(|segment| matches!(segment, "." | "..")) +} + +fn reserves_namespace(path: &str) -> bool { + if path.starts_with(NAMESPACE) { + return true; + } + if !path.contains("//") && !has_dot_segments(path) && !path.contains('\\') { + return false; + } + // Alternative spellings are considered only to reserve and reject them. + // The resulting path is never used for routing or authentication. + let mut segments = Vec::new(); + for segment in path.split(['/', '\\']) { + match segment { + "" | "." => {} + ".." => { + segments.pop(); + } + _ => segments.push(segment), + } + // Once a visible alternative spelling reaches the namespace, later + // dot segments cannot erase its reservation at application dispatch. + if segments.first() == Some(&"_ts") + && segments + .get(1) + .is_some_and(|segment| segment.starts_with("trace")) + { + return true; + } + } + false +} + +fn encoded_separator(path: &str) -> bool { + path.as_bytes().windows(3).any(|bytes| { + bytes[0] == b'%' + && (bytes[1] == b'2' && bytes[2].eq_ignore_ascii_case(&b'f') + || bytes[1] == b'5' && bytes[2].eq_ignore_ascii_case(&b'c')) + }) +} + +fn valid_escapes(path: &str) -> bool { + let mut bytes = path.bytes(); + while let Some(byte) = bytes.next() { + if byte == b'%' + && (!bytes.next().is_some_and(|value| value.is_ascii_hexdigit()) + || !bytes.next().is_some_and(|value| value.is_ascii_hexdigit())) + { + return false; + } + } + true +} + +fn classify_path(path: &str) -> Classification { + if let Some(route) = exact_route(path) { + return Classification::Route(route); + } + let mut reserved = reserves_namespace(path); + if reserved && (has_dot_segments(path) || path.contains('\\')) { + return Classification::Rejected(StatusCode::BAD_REQUEST); + } + let mut current = Cow::Borrowed(path); + let mut invalid_encoding = false; + for _ in 0..MAX_DECODE_ROUNDS { + if !current.contains('%') { + break; + } + invalid_encoding |= !valid_escapes(¤t); + // Decode bytes so an unreadable suffix cannot hide an ASCII namespace + // prefix. Lossy text is used exclusively for reservation, never routing. + let decoded_bytes = urlencoding::decode_binary(current.as_bytes()); + invalid_encoding |= core::str::from_utf8(&decoded_bytes).is_err(); + let decoded = String::from_utf8_lossy(&decoded_bytes); + let decoded_reserved = reserves_namespace(&decoded); + if (reserved || decoded_reserved) + && (invalid_encoding + || encoded_separator(¤t) + || has_dot_segments(&decoded) + || !reserved + || exact_route(&decoded).is_some()) + { + return Classification::Rejected(StatusCode::BAD_REQUEST); + } + reserved |= decoded_reserved; + if decoded == current { + break; + } + current = Cow::Owned(decoded.into_owned()); + } + if !reserved { + Classification::NotTrace + } else if current.contains('%') { + Classification::Rejected(StatusCode::BAD_REQUEST) + } else { + Classification::Rejected(StatusCode::NOT_FOUND) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + use std::sync::{Mutex, Once}; + + use base64::{Engine as _, engine::general_purpose::STANDARD}; + use http::{HeaderValue, header}; + use log::{LevelFilter, Log, Metadata, Record}; + use serde_json::json; + + use crate::auth::EdgeTerminatedAuthorization; + use crate::cache_policy::EDGE_CACHE_HEADER_NAMES; + use crate::settings::Handler; + use crate::test_support::tests::create_test_settings; + use crate::trace::TraceTerminalResponse; + + fn handler(pattern: &str, username: &str, password: &str) -> Handler { + serde_json::from_value(json!({"path": pattern, "username": username, "password": password})) + .expect("should deserialize the example authentication handler") + } + + fn settings(enabled: bool, pattern: Option<&str>) -> Settings { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &json!({"enabled": true, "trace_page_enabled": enabled}), + ) + .expect("should insert trace configuration"); + if let Some(pattern) = pattern { + settings + .handlers + .insert(0, handler(pattern, "example-user", "example-password")); + } + settings + } + + fn request(method: Method, path: &str, credentials: bool) -> Request { + let mut builder = Request::builder() + .method(method) + .uri(format!("https://publisher.example.com{path}")); + if credentials { + builder = builder.header( + header::AUTHORIZATION, + format!("Basic {}", STANDARD.encode("example-user:example-password")), + ); + } + builder + .body(EdgeBody::empty()) + .expect("should build trace request") + } + + fn rejected(settings: &Settings, request: &mut Request) -> Response { + match preflight(settings, request) { + TracePreflight::Response(response) => response, + _ => panic!("should terminate the rejected trace request"), + } + } + + fn ready(settings: &Settings, request: &mut Request) -> TraceDispatch { + match preflight(settings, request) { + TracePreflight::Ready(dispatch) => dispatch, + _ => panic!("should dispatch the accepted trace request"), + } + } + + fn assert_private(response: &Response) { + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should enforce private non-storable responses" + ); + for name in EDGE_CACHE_HEADER_NAMES { + assert!( + !response.headers().contains_key(*name), + "should remove shared cache directive {name}" + ); + } + assert_eq!( + response.headers()[header::X_CONTENT_TYPE_OPTIONS], + "nosniff", + "should disable MIME sniffing" + ); + assert_eq!( + response.headers()[header::REFERRER_POLICY], + "no-referrer", + "should suppress referrers" + ); + assert_eq!( + response.headers()[header::CONTENT_SECURITY_POLICY], + "default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self'; img-src data:", + "should retain the exact trace CSP" + ); + assert_eq!( + response.headers()["permissions-policy"], + "camera=(), microphone=(), geolocation=(), payment=(), usb=()", + "should disable unrelated browser capabilities" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should bypass ordinary finalizers" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should not mutate cookies during preflight" + ); + } + + #[test] + fn authentication_precedes_disabled_path_and_method_policy() { + for enabled in [false, true] { + let settings = settings(enabled, Some("^/")); + for path in [ + "/_ts/trace", + "/_ts/trace/state", + "/_ts/trace/enable", + "/_ts/trace/end", + "/_ts/trace/assets/v1.js", + "/_ts/trace/assets/v1.css", + "/%5Fts/trace", + "/_ts/trace/../secret-example", + ] { + let mut request = request( + Method::from_bytes(b"TRACE-EXAMPLE").expect("should parse extension method"), + path, + false, + ); + let response = rejected(&settings, &mut request); + assert_eq!( + response.status(), + StatusCode::UNAUTHORIZED, + "should authenticate before trace policies" + ); + assert_eq!( + response.headers()[header::WWW_AUTHENTICATE], + "Basic realm=\"Trusted Server\"", + "should preserve the configured challenge" + ); + assert_eq!( + response.headers()[header::CONTENT_TYPE], + "application/json; charset=utf-8", + "should harden challenge MIME" + ); + assert_private(&response); + } + } + } + + #[test] + fn literal_path_authentication_and_first_match_are_preserved() { + let mut request = request(Method::GET, "/%5Fts/trace?secret=example-sentinel", false); + let response = rejected(&settings(true, Some("^/_ts")), &mut request); + assert_eq!( + response.status(), + StatusCode::BAD_REQUEST, + "should reject the alias without decoding the auth path" + ); + assert!( + !response.headers().contains_key(header::WWW_AUTHENTICATE), + "should not manufacture an auth rule" + ); + assert_private(&response); + let body = response + .into_body() + .into_bytes() + .expect("should return a fixed error body"); + assert!( + !String::from_utf8_lossy(&body).contains("example-sentinel"), + "should not reflect query or path data" + ); + + let mut request = self::request(Method::GET, "/%5Fts/trace", false); + let response = rejected(&settings(false, Some("^/_ts")), &mut request); + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should apply disabled feature before path errors" + ); + + let mut settings = settings(true, Some("^/")); + settings + .handlers + .insert(0, handler("^/_ts/trace$", "narrow-user", "narrow-password")); + let mut request = self::request(Method::GET, "/_ts/trace", true); + let response = rejected(&settings, &mut request); + assert_eq!( + response.status(), + StatusCode::UNAUTHORIZED, + "should enforce the first matching handler" + ); + } + + #[test] + fn method_matrix_and_head_errors_are_local_and_bodyless() { + let settings = settings(true, None); + for (path, allow, supported) in [ + ("/_ts/trace", "GET, HEAD", Method::GET), + ("/_ts/trace/state", "GET, HEAD", Method::GET), + ("/_ts/trace/enable", "POST", Method::POST), + ("/_ts/trace/end", "POST", Method::POST), + ("/_ts/trace/assets/v1.js", "GET, HEAD", Method::GET), + ("/_ts/trace/assets/v1.css", "GET, HEAD", Method::GET), + ] { + let mut request = request(supported, path, false); + ready(&settings, &mut request); + for method in [ + Method::PUT, + Method::from_bytes(b"TRACE-EXAMPLE").expect("should parse extension method"), + ] { + let mut request = self::request(method, path, false); + let response = rejected(&settings, &mut request); + assert_eq!( + response.status(), + StatusCode::METHOD_NOT_ALLOWED, + "should reject unsupported visible methods locally" + ); + assert_eq!( + response.headers()[header::ALLOW], + allow, + "should retain path-specific allowed methods" + ); + assert_private(&response); + } + } + for (path, settings) in [ + ("/_ts/trace/enable", self::settings(true, None)), + ("/_ts/trace/end", self::settings(true, None)), + ("/_ts/trace/extra", self::settings(true, None)), + ("/_ts/trace", self::settings(false, None)), + ("/_ts/trace", self::settings(true, Some("^/"))), + ] { + let mut request = request(Method::HEAD, path, false); + let response = rejected(&settings, &mut request); + assert_private(&response); + assert!( + response + .into_body() + .into_bytes() + .expect("should return a fixed body") + .is_empty(), + "should suppress every HEAD error body" + ); + } + } + + #[test] + fn dynamic_finalization_preserves_response_contract_and_head_semantics() { + let settings = settings(true, Some("^/")); + for (path, mime) in [ + ("/_ts/trace", "text/html; charset=utf-8"), + ("/_ts/trace/state", "application/json; charset=utf-8"), + ] { + for method in [Method::GET, Method::HEAD] { + let mut request = request(method.clone(), path, true); + let dispatch = ready(&settings, &mut request); + assert!( + request + .extensions() + .get::() + .is_some(), + "should preserve successful authorization marker" + ); + let response = Response::builder() + .header(header::CACHE_CONTROL, "public, max-age=60") + .header("surrogate-control", "max-age=60") + .header(header::ETAG, "\"example\"") + .header(header::LAST_MODIFIED, "Wed, 12 Aug 2026 00:00:00 GMT") + .header(header::SET_COOKIE, "unrelated=example") + .body(EdgeBody::from("example content")) + .expect("should build handler response"); + let response = dispatch.respond(response); + assert_private(&response); + assert_eq!( + response.headers()[header::CONTENT_TYPE], + mime, + "should use route-appropriate MIME" + ); + assert!( + !response.headers().contains_key(header::ETAG), + "should remove dynamic validators" + ); + assert!( + !response.headers().contains_key(header::LAST_MODIFIED), + "should remove dynamic validators" + ); + if method == Method::HEAD { + assert!( + response + .into_body() + .into_bytes() + .expect("should return a fixed body") + .is_empty(), + "should suppress successful HEAD body" + ); + } + } + } + } + + #[test] + fn fixed_asset_policy_depends_on_auth_coverage_and_build_bytes() { + for protected in [false, true] { + let settings = settings(true, protected.then_some("^/_ts")); + for (path, mime) in [ + ( + "/_ts/trace/assets/v1.js", + "application/javascript; charset=utf-8", + ), + ("/_ts/trace/assets/v1.css", "text/css; charset=utf-8"), + ] { + let mut request = request(Method::GET, path, protected); + let response = ready(&settings, &mut request).fixed_asset(b"example build bytes"); + assert_eq!( + response.headers()[header::CONTENT_TYPE], + mime, + "should serve fixed assets with exact MIME" + ); + assert_eq!( + response.headers()[header::X_CONTENT_TYPE_OPTIONS], + "nosniff", + "should retain asset MIME protection" + ); + let etag = response.headers()[header::ETAG] + .to_str() + .expect("should return an ASCII ETag"); + assert!( + etag.starts_with('"') && etag.ends_with('"') && !etag.starts_with("W/"), + "should derive a strong quoted ETag" + ); + if protected { + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "should keep protected assets private" + ); + } else { + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "public, max-age=31536000, immutable", + "should cache only unprotected fixed assets" + ); + } + let mut other = self::request(Method::HEAD, path, protected); + let head = ready(&settings, &mut other).fixed_asset(b"example build bytes"); + assert_eq!( + head.headers(), + response.headers(), + "should preserve GET headers on HEAD" + ); + assert!( + head.into_body() + .into_bytes() + .expect("should return a fixed body") + .is_empty(), + "should suppress asset HEAD bodies" + ); + let mut other = self::request(Method::GET, path, protected); + let changed = + ready(&settings, &mut other).fixed_asset(b"changed example build bytes"); + assert_ne!( + changed.headers()[header::ETAG], + response.headers()[header::ETAG], + "should derive ETag from actual build bytes" + ); + } + } + } + + struct CapturedLogs(Mutex>); + + impl Log for CapturedLogs { + fn enabled(&self, _metadata: &Metadata<'_>) -> bool { + true + } + fn log(&self, record: &Record<'_>) { + let message = record.args().to_string(); + if message == "trace_auth_failed" + || message.contains("fictional-secret-sentinel") + || message == "Basic auth failed for path: /ordinary-example-sentinel" + { + let mut logs = self.0.lock().expect("should capture log messages"); + if logs.len() < 3 && !logs.contains(&message) { + logs.push(message); + } + } + } + fn flush(&self) {} + } + + static LOGS: CapturedLogs = CapturedLogs(Mutex::new(Vec::new())); + static LOG_INIT: Once = Once::new(); + + #[test] + fn trace_auth_failure_logs_a_fixed_category_without_the_path() { + LOG_INIT.call_once(|| { + log::set_logger(&LOGS).expect("should install the test logger"); + log::set_max_level(LevelFilter::Warn); + }); + let mut request = request(Method::GET, "/_ts/trace/fictional-secret-sentinel", false); + request.headers_mut().insert( + header::AUTHORIZATION, + HeaderValue::from_str(&format!( + "Basic {}", + STANDARD.encode("example-user:incorrect-example") + )) + .expect("should build incorrect credentials"), + ); + let response = rejected(&settings(true, Some("^/")), &mut request); + assert_eq!( + response.status(), + StatusCode::UNAUTHORIZED, + "should authenticate before reporting the reserved path error" + ); + let logs = LOGS.0.lock().expect("should inspect captured messages"); + assert!( + logs.iter().any(|message| message == "trace_auth_failed"), + "should emit only a fixed diagnostic category" + ); + assert!( + logs.iter() + .all(|message| !message.contains("fictional-secret-sentinel")), + "should not log the confidential trace path" + ); + drop(logs); + let mut ordinary = self::request(Method::GET, "/ordinary-example-sentinel", false); + ordinary.headers_mut().insert( + header::AUTHORIZATION, + request.headers()[header::AUTHORIZATION].clone(), + ); + assert!( + enforce_basic_auth(&settings(true, Some("^/")), &mut ordinary) + .expect("should evaluate ordinary auth") + .is_some(), + "should retain ordinary credential rejection" + ); + assert!( + LOGS.0 + .lock() + .expect("should inspect ordinary auth messages") + .iter() + .any(|message| message == "Basic auth failed for path: /ordinary-example-sentinel"), + "should retain ordinary path logging outside trace" + ); + let body = response + .into_body() + .into_bytes() + .expect("should return a fixed challenge"); + assert!( + !String::from_utf8_lossy(&body).contains("fictional-secret-sentinel"), + "should not reflect the trace path in a challenge" + ); + } + + #[test] + fn visible_trace_prefix_cannot_be_erased_by_dot_segments() { + for path in [ + "/_ts//trace/../ordinary", + "//_ts/trace/../../health", + "/%5Fts//trace/../ordinary", + "/_ts%2Ftrace/../ordinary", + "/ordinary/../_ts//trace/../../health", + ] { + assert_eq!( + classify_path(path), + Classification::Rejected(StatusCode::BAD_REQUEST), + "should retain the visible reserved prefix before later dot segments {path}" + ); + assert!( + is_trace_path(path), + "should suppress native shortcuts for visible reserved ambiguity {path}" + ); + let mut request = request(Method::GET, path, false); + let response = rejected(&settings(true, None), &mut request); + assert_eq!( + response.status(), + StatusCode::BAD_REQUEST, + "should terminate the visible ambiguity locally" + ); + assert_private(&response); + } + } + + #[test] + fn encoded_namespace_with_unreadable_suffix_never_falls_through() { + for path in [ + "/%5Fts/trace/%GG", + "/%255Fts/trace/%GG", + "/%5Fts/trace/%FF", + "/%255Fts/trace/%FF", + ] { + assert_eq!( + classify_path(path), + Classification::Rejected(StatusCode::BAD_REQUEST), + "should reserve the encoded namespace despite an unreadable suffix {path}" + ); + } + } + + #[test] + fn unrelated_preflight_leaves_the_request_untouched() { + let mut settings = settings(true, None); + settings.integrations.insert( + "gpt_diagnostics".to_string(), + json!({"enabled": "invalid-example"}), + ); + let mut request = request(Method::GET, "/article?ts_console=1", true); + request.headers_mut().insert( + header::COOKIE, + HeaderValue::from_static("__Host-ts-console=1; ts-ec=example"), + ); + let before_uri = request.uri().clone(); + let before_headers = request.headers().clone(); + assert!( + matches!(preflight(&settings, &mut request), TracePreflight::NotTrace), + "should continue ordinary dispatch without parsing trace configuration" + ); + assert_eq!(request.uri(), &before_uri, "should retain the ordinary URI"); + assert_eq!( + request.headers(), + &before_headers, + "should retain ordinary headers" + ); + assert!( + request.extensions().get::().is_none(), + "should retain ordinary auth logging" + ); + assert!( + request + .extensions() + .get::() + .is_none(), + "should not authenticate ordinary requests early" + ); + } + + #[test] + fn authentication_failure_does_not_read_the_body() { + let mut request = request(Method::POST, "/_ts/trace/enable", false); + *request.body_mut() = EdgeBody::stream(futures::stream::poll_fn( + |_| -> std::task::Poll> { + panic!("should never inspect a body before authentication"); + }, + )); + let response = rejected(&settings(true, Some("^/")), &mut request); + assert_eq!( + response.status(), + StatusCode::UNAUTHORIZED, + "should authenticate without consuming the stream" + ); + } + + #[test] + fn trace_actions_state_response_uses_only_frozen_session_health() { + for (field, active) in [ + ("__Host-ts-console=1", true), + ("__Host-ts-console=invalid-example", false), + ("__Host-ts-console", false), + ("__Host-ts-console=1; __Host-ts-console=1", false), + ("__Host-ts-console=1; unrelated=a,b", false), + ("__Host-ts-console=1; unrelated=�", false), + ("", false), + ] { + for method in [Method::GET, Method::HEAD] { + let mut request = request(method.clone(), "/_ts/trace/state?ts_console=1", false); + request.headers_mut().insert( + header::COOKIE, + HeaderValue::from_bytes(field.as_bytes()) + .expect("should construct frozen runtime-visible Cookie text"), + ); + let frozen = super::super::inspect_cookies(request.headers(), None); + request.headers_mut().remove(header::COOKIE); + let context = ready(&settings(true, None), &mut request); + let response = context.respond(state_response(&frozen)); + assert_eq!( + response.status(), + StatusCode::OK, + "should observe the frozen state locally" + ); + assert_private(&response); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should never refresh or clear a cookie on state requests" + ); + let body = response.into_body(); + if method == Method::HEAD { + assert!( + body.into_bytes() + .expect("should return a bounded HEAD response") + .is_empty(), + "should remove every state HEAD body" + ); + } else { + assert_eq!( + body.to_json::() + .expect("should parse exact observed state JSON"), + json!({"observed_active":active}), + "should emit only the frozen observed_active boolean" + ); + } + } + } + } + + #[test] + fn fixed_asset_queries_do_not_affect_bytes_or_headers() { + let settings = settings(true, None); + let mut plain = request(Method::GET, "/_ts/trace/assets/v1.js", false); + let mut queried = request( + Method::GET, + "/_ts/trace/assets/v1.js?secret=fictional-query-sentinel", + false, + ); + let plain = ready(&settings, &mut plain).fixed_asset(b"example build bytes"); + let queried = ready(&settings, &mut queried).fixed_asset(b"example build bytes"); + assert_eq!( + queried.headers(), + plain.headers(), + "should exclude query data from static headers" + ); + assert_eq!( + queried.into_body().into_bytes(), + plain.into_body().into_bytes(), + "should exclude query data from static bytes" + ); + } + + #[test] + fn exact_routes_and_reserved_shapes_are_classified_without_normalization() { + for (path, route) in [ + ("/_ts/trace", TraceRoute::Shell), + ("/_ts/trace/state", TraceRoute::State), + ("/_ts/trace/enable", TraceRoute::Enable), + ("/_ts/trace/end", TraceRoute::End), + ("/_ts/trace/assets/v1.js", TraceRoute::Javascript), + ("/_ts/trace/assets/v1.css", TraceRoute::Stylesheet), + ] { + assert_eq!( + classify_path(path), + Classification::Route(route), + "should classify the exact route {path}" + ); + } + for path in [ + "/_ts/trace/", + "/_ts/trace/state/extra", + "/_ts/trace-extra", + "/_ts/tracer", + "/_ts/trace/assets/v2.js", + "//_ts/trace", + "/_ts//trace", + "/_ts/trace//state", + ] { + assert_eq!( + classify_path(path), + Classification::Rejected(StatusCode::NOT_FOUND), + "should reserve the unsupported spelling {path}" + ); + } + for path in [ + "/%5Fts/trace", + "/_ts/%74race", + "/_ts/trace%2Fend", + "/_ts/trace%252Fend", + "/_ts/trace/../ordinary", + "/_ts/./trace", + "/ordinary/../_ts/trace", + "/_ts/trace/%2e%2e/ordinary", + "/_ts/trace%GG", + "/_ts/trace%FF", + "/_ts/trace%252525252Fend", + ] { + assert_eq!( + classify_path(path), + Classification::Rejected(StatusCode::BAD_REQUEST), + "should reject the ambiguous spelling {path}" + ); + } + for path in [ + "/", + "/article", + "/_ts/debug/ja4", + "/health", + "/article%2Fother", + "/article%GG", + "/%FF", + "/_ts/not-trace", + "/_ts/../ordinary", + ] { + assert_eq!( + classify_path(path), + Classification::NotTrace, + "should preserve unrelated traffic {path}" + ); + } + } +} diff --git a/crates/trusted-server-core/src/trace/shell.rs b/crates/trusted-server-core/src/trace/shell.rs new file mode 100644 index 000000000..a45270db0 --- /dev/null +++ b/crates/trusted-server-core/src/trace/shell.rs @@ -0,0 +1,230 @@ +//! Escaped setup-page shell using only frozen, redacted request facts. + +use error_stack::{Report, ResultExt as _}; + +use super::types::TraceRequestContextV1; + +/// Bounded failure category for setup-shell projection. +#[derive(Debug, derive_more::Display)] +pub(crate) enum ShellError { + /// The redacted request context could not be serialized. + #[display("Trace setup facts are unavailable")] + Serialization, +} + +impl core::error::Error for ShellError {} + +/// Render setup controls and frozen redacted facts using fixed external assets. +/// +/// # Errors +/// +/// Returns [`ShellError::Serialization`] if the context cannot be serialized. +pub(crate) fn render_setup_shell( + context: &TraceRequestContextV1, +) -> Result> { + let context_json = serde_json::to_string(context).change_context(ShellError::Serialization)?; + let context_text = escape_text(&context_json); + let active = context.cookies().observed_active(); + let observed_state = if active { + "Tracing is on — cookie observed by server" + } else { + "Tracing is off — no valid diagnostics session observed" + }; + Ok(format!( + r#" + + + + +Trusted Server ad diagnostics + + + + + +
+

Trusted Server ad diagnostics

+

No previous ad failure can be recovered. Enable tracing, then reproduce the problem on a freshly reloaded page.

+
+

Tracing controls

+

{observed_state}

+
+ + + +
+

+

Return to the affected page, reload once, reproduce the problem, then select View trace results.

+

If history is not useful, reopen the affected article on the exact same hostname and in this same tab, then reload once.

+ +
+
+

Setup request

+

These facts describe this setup request. They do not describe a previously affected publisher page.

+

Approximate network facts

+

Masked network identifiers are approximate and may still identify a network.

+
+

Owned cookie health

+

Only the shape of Trusted Server cookies visible in this request is inspected. Cookie values and browser cookie attributes are excluded.

+ + +
+
+ +"# + )) +} + +fn escape_text(value: &str) -> String { + let mut escaped = String::with_capacity(value.len()); + for character in value.chars() { + match character { + '&' => escaped.push_str("&"), + '<' => escaped.push_str("<"), + '>' => escaped.push_str(">"), + '"' => escaped.push_str("""), + '\'' => escaped.push_str("'"), + _ => escaped.push(character), + } + } + escaped +} + +#[cfg(test)] +mod tests { + use chrono::{TimeZone as _, Utc}; + use http::{HeaderMap, HeaderValue, header}; + + use crate::platform::ClientInfo; + use crate::trace::context::project_request_context; + use crate::trace::cookies::inspect_cookies; + + use super::*; + + fn context(cookie: &str) -> crate::trace::types::TraceRequestContextV1 { + let mut headers = HeaderMap::new(); + headers.insert( + header::COOKIE, + HeaderValue::from_str(cookie).expect("should encode the fictional cookie fixture"), + ); + let cookies = inspect_cookies(&headers, None); + project_request_context( + &ClientInfo { + tls_cipher: Some("".to_owned()), + server_hostname: Some("\" onload='x' <&>".to_owned()), + ..ClientInfo::default() + }, + None, + &cookies, + Utc.with_ymd_and_hms(2026, 10, 5, 12, 0, 0) + .single() + .expect("should create the fixed UTC capture clock"), + ) + .expect("should project only bounded request facts") + } + + #[test] + fn trace_shell_uses_fixed_assets_and_escapes_context_as_text() { + let html = render_setup_shell(&context( + "unrelated=fictional-cookie-secret; __Host-ts-console=1", + )) + .expect("should serialize the redacted setup shell"); + assert!( + html.contains("Trusted Server ad diagnostics"), + "should use the approved page title" + ); + assert!( + html.contains("Setup request"), + "should distinguish setup facts from a traced publisher document" + ); + assert!( + html.contains("data-observed-active=\"true\""), + "should derive state from frozen cookie health" + ); + assert!( + html.contains("Tracing is on — cookie observed by server"), + "should display the setup request observation" + ); + assert!( + html.contains("/_ts/trace/assets/v1.js"), + "should reference the exact immutable script route" + ); + assert!( + html.contains("/_ts/trace/assets/v1.css"), + "should reference the exact immutable stylesheet route" + ); + assert_eq!( + html.matches(""), + "should never execute reflected platform facts" + ); + assert!( + !html.contains("fictional-cookie-secret"), + "should never retain unrelated cookie values" + ); + assert!( + !html.contains("window."), + "should not inline executable request data" + ); + assert!( + !html.contains("style="), + "should respect the external-only stylesheet CSP" + ); + } + + #[test] + fn trace_shell_off_state_preserves_end_and_recovery_accessibility() { + let html = render_setup_shell(&context("unrelated=comma,value; __Host-ts-console=1")) + .expect("should retain setup controls when cookie inspection is ambiguous"); + assert!( + html.contains("data-observed-active=\"false\""), + "should never activate an ambiguous session" + ); + assert!( + html.contains("no valid diagnostics session observed"), + "should describe observation without claiming cookie absence" + ); + for id in ["trace-enable", "trace-end", "trace-back", "trace-status"] { + assert!( + html.contains(&format!("id=\"{id}\"")), + "should expose all deliberate controls and status" + ); + } + assert!( + html.contains("aria-live=\"polite\""), + "should announce action results accessibly" + ); + assert!( + html.contains("exact same hostname and in this same tab"), + "should retain history-independent recovery guidance" + ); + assert!( + html.contains("reload once"), + "should require a fresh cookie-bearing reproduction document" + ); + assert!( + html.contains("data:image/png;base64,"), + "should avoid an automatic publisher favicon request" + ); + assert!( + html.contains("width=device-width, initial-scale=1"), + "should preserve mobile zoom" + ); + assert!( + !html.contains("maximum-scale"), + "should not prevent browser zoom" + ); + } +} diff --git a/crates/trusted-server-core/src/trace/slot_refs.rs b/crates/trusted-server-core/src/trace/slot_refs.rs new file mode 100644 index 000000000..b110bcd1b --- /dev/null +++ b/crates/trusted-server-core/src/trace/slot_refs.rs @@ -0,0 +1,169 @@ +//! Optional client tokens follow the exact accepted conversion occurrence. + +use std::fmt; +use std::sync::{Arc, Mutex}; + +use serde::Deserialize; +use serde::de::{DeserializeSeed, Deserializer, IgnoredAny, MapAccess, SeqAccess, Visitor}; +use serde_json::value::RawValue; + +use super::TraceSlotRef; + +/// Raw input occurrences and definitive accepted occurrences are separate. +/// Only the normal converter decides which source occurrence becomes a slot. +#[derive(Clone, Default)] +pub(crate) struct TraceClientSlotRefs { + source: Arc<[Option]>, + accepted: Arc>>>, +} + +impl TraceClientSlotRefs { + pub(crate) fn from_raw(body: &[u8]) -> Self { + let source = serde_json::from_slice::(body) + .map(|request| request.ad_units.0) + .unwrap_or_default(); + Self { + source: Arc::from(source), + accepted: Arc::new(Mutex::new(Vec::new())), + } + } + + pub(crate) fn begin_conversion(&self) { + if let Ok(mut accepted) = self.accepted.lock() { + accepted.clear(); + } + } + + pub(crate) fn record_accepted(&self, input_index: usize) { + if let Ok(mut accepted) = self.accepted.lock() { + accepted.push(self.source.get(input_index).cloned().flatten()); + } + } + + pub(crate) fn accepted_refs(&self) -> Option>> { + self.accepted.lock().ok().map(|accepted| accepted.clone()) + } +} + +#[derive(Deserialize)] +struct RawTraceRequest { + #[serde(rename = "adUnits")] + ad_units: RawTraceUnits, +} + +struct RawTraceUnits(Vec>); + +impl<'de> Deserialize<'de> for RawTraceUnits { + fn deserialize>(deserializer: D) -> Result { + deserializer.deserialize_seq(TraceUnitsVisitor) + } +} + +struct TraceUnitsVisitor; + +impl<'de> Visitor<'de> for TraceUnitsVisitor { + type Value = RawTraceUnits; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("the already-accepted adUnits array") + } + + fn visit_seq>(self, mut sequence: A) -> Result { + let mut units = Vec::new(); + while let Some(unit) = sequence.next_element::<&RawValue>()? { + // Borrow only during extraction. Optional numeric range failures + // cannot discard another unit's valid association. + let mut parser = serde_json::Deserializer::from_str(unit.get()); + units.push( + TraceMemberStage::Unit + .deserialize(&mut parser) + .unwrap_or(None), + ); + } + Ok(RawTraceUnits(units)) + } +} + +#[derive(Clone, Copy)] +enum TraceMemberStage { + Unit, + Extension, + Namespace, + Token, +} + +impl TraceMemberStage { + fn member(self) -> Option<(&'static str, Self)> { + match self { + Self::Unit => Some(("ext", Self::Extension)), + Self::Extension => Some(("trusted_server", Self::Namespace)), + Self::Namespace => Some(("trace_slot_ref", Self::Token)), + Self::Token => None, + } + } +} + +impl<'de> DeserializeSeed<'de> for TraceMemberStage { + type Value = Option; + + fn deserialize>(self, deserializer: D) -> Result { + deserializer.deserialize_any(TraceMemberVisitor(self)) + } +} + +struct TraceMemberVisitor(TraceMemberStage); + +impl<'de> Visitor<'de> for TraceMemberVisitor { + type Value = Option; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("an optional trace namespace member") + } + + fn visit_map>(self, mut map: A) -> Result { + let mut seen = false; + let mut duplicate = false; + let mut token = None; + while let Some(key) = map.next_key::()? { + if let Some((member, stage)) = self.0.member() + && key == member + { + duplicate |= seen; + seen = true; + token = map.next_value_seed(stage)?; + } else { + map.next_value::()?; + } + } + Ok(if duplicate { None } else { token }) + } + + fn visit_seq>(self, mut sequence: A) -> Result { + while sequence.next_element::()?.is_some() {} + Ok(None) + } + + fn visit_str(self, value: &str) -> Result { + Ok(if matches!(self.0, TraceMemberStage::Token) { + TraceSlotRef::parse(value).ok() + } else { + None + }) + } + + fn visit_bool(self, _value: bool) -> Result { + Ok(None) + } + fn visit_i64(self, _value: i64) -> Result { + Ok(None) + } + fn visit_u64(self, _value: u64) -> Result { + Ok(None) + } + fn visit_f64(self, _value: f64) -> Result { + Ok(None) + } + fn visit_unit(self) -> Result { + Ok(None) + } +} diff --git a/crates/trusted-server-core/src/trace/types.rs b/crates/trusted-server-core/src/trace/types.rs new file mode 100644 index 000000000..5f880471e --- /dev/null +++ b/crates/trusted-server-core/src/trace/types.rs @@ -0,0 +1,358 @@ +//! Closed public request and cookie projection types. + +use serde::Serialize; + +/// Shape of one incoming reserved cookie without its value. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum CookieHealthState { + /// No exact occurrence is visible. + Absent, + /// One complete value passes its canonical validator. + PresentValid, + /// One occurrence is malformed, oversized or unsupported. + PresentInvalid, + /// More than one exact occurrence is visible. + Duplicate, + /// Aggregate inspection cannot be trusted. + Unavailable, +} + +/// Bounded explanation of cookie shape without values or parser messages. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum CookieHealthDetail { + /// The complete EC value passes the canonical validator. + ValidEcFormat, + /// The complete EID value passes the bounded existing parser. + ValidEidsFormat, + /// The tester value is exactly `true`. + ValidTesterValue, + /// The diagnostics value is exactly `1`. + ValidDiagnosticsValue, + /// The occurrence does not match the expected syntax. + Malformed, + /// The visible value exceeds its individual bound. + Oversized, + /// The complete literal value is unsupported. + UnsupportedValue, + /// More than one exact reserved occurrence is visible. + MultipleValues, + /// Visible Cookie fields exceed the aggregate byte bound. + HeaderTooLarge, + /// A visible Cookie field contains actual invalid UTF-8 bytes. + HeaderNotUtf8, + /// Markers make unproved runtime preservation ambiguous. + RuntimeHeaderAmbiguous, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +enum CookieSource { + Request, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +pub(super) enum ValidCookieDetail { + #[serde(rename = "valid_ec_format")] + EcFormat, + #[serde(rename = "valid_eids_format")] + EidsFormat, + #[serde(rename = "valid_tester_value")] + TesterValue, + #[serde(rename = "valid_diagnostics_value")] + DiagnosticsValue, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum InvalidCookieDetail { + Malformed, + Oversized, + UnsupportedValue, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum UnavailableCookieDetail { + HeaderTooLarge, + HeaderNotUtf8, + RuntimeHeaderAmbiguous, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +enum DuplicateCookieDetail { + MultipleValues, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(tag = "state", rename_all = "snake_case")] +enum CookieOutcome { + Absent, + PresentValid { detail: ValidCookieDetail }, + PresentInvalid { detail: InvalidCookieDetail }, + Duplicate { detail: DuplicateCookieDetail }, + Unavailable { detail: UnavailableCookieDetail }, +} + +/// Immutable health with only legal state and detail combinations. +/// +/// Construction stays inside the scanner, which also binds valid details to +/// their reserved cookie names. Serialization always uses `source: request`. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +pub struct CookieHealth { + source: CookieSource, + #[serde(flatten)] + outcome: CookieOutcome, +} + +impl CookieHealth { + /// Return the observed shape of this cookie. + /// + /// # Examples + /// + /// ``` + /// use http::HeaderMap; + /// use trusted_server_core::trace::{CookieHealthState, inspect_cookies}; + /// let cookies = inspect_cookies(&HeaderMap::new(), None); + /// assert_eq!(cookies.ts_ec().state(), CookieHealthState::Absent); + /// ``` + #[must_use] + pub fn state(&self) -> CookieHealthState { + match self.outcome { + CookieOutcome::Absent => CookieHealthState::Absent, + CookieOutcome::PresentValid { .. } => CookieHealthState::PresentValid, + CookieOutcome::PresentInvalid { .. } => CookieHealthState::PresentInvalid, + CookieOutcome::Duplicate { .. } => CookieHealthState::Duplicate, + CookieOutcome::Unavailable { .. } => CookieHealthState::Unavailable, + } + } + + /// Return the bounded explanation, absent when no occurrence is visible. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert_eq!(cookies.ts_ec().detail(), None); + /// ``` + #[must_use] + pub fn detail(&self) -> Option { + Some(match self.outcome { + CookieOutcome::Absent => return None, + CookieOutcome::PresentValid { detail } => match detail { + ValidCookieDetail::EcFormat => CookieHealthDetail::ValidEcFormat, + ValidCookieDetail::EidsFormat => CookieHealthDetail::ValidEidsFormat, + ValidCookieDetail::TesterValue => CookieHealthDetail::ValidTesterValue, + ValidCookieDetail::DiagnosticsValue => CookieHealthDetail::ValidDiagnosticsValue, + }, + CookieOutcome::PresentInvalid { detail } => match detail { + InvalidCookieDetail::Malformed => CookieHealthDetail::Malformed, + InvalidCookieDetail::Oversized => CookieHealthDetail::Oversized, + InvalidCookieDetail::UnsupportedValue => CookieHealthDetail::UnsupportedValue, + }, + CookieOutcome::Duplicate { .. } => CookieHealthDetail::MultipleValues, + CookieOutcome::Unavailable { detail } => match detail { + UnavailableCookieDetail::HeaderTooLarge => CookieHealthDetail::HeaderTooLarge, + UnavailableCookieDetail::HeaderNotUtf8 => CookieHealthDetail::HeaderNotUtf8, + UnavailableCookieDetail::RuntimeHeaderAmbiguous => { + CookieHealthDetail::RuntimeHeaderAmbiguous + } + }, + }) + } + + pub(super) const fn absent() -> Self { + Self::new(CookieOutcome::Absent) + } + + pub(super) const fn valid(detail: ValidCookieDetail) -> Self { + Self::new(CookieOutcome::PresentValid { detail }) + } + + pub(super) const fn invalid(detail: InvalidCookieDetail) -> Self { + Self::new(CookieOutcome::PresentInvalid { detail }) + } + + pub(super) const fn duplicate() -> Self { + Self::new(CookieOutcome::Duplicate { + detail: DuplicateCookieDetail::MultipleValues, + }) + } + + pub(super) const fn unavailable(detail: UnavailableCookieDetail) -> Self { + Self::new(CookieOutcome::Unavailable { detail }) + } + + const fn new(outcome: CookieOutcome) -> Self { + Self { + source: CookieSource::Request, + outcome, + } + } +} + +/// Frozen redacted observations of the four reserved cookies. +/// +/// This owns no incoming headers, values, fidelity metadata or identity state. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +pub struct TraceCookies { + ts_ec: CookieHealth, + ts_eids: CookieHealth, + ts_tester: CookieHealth, + diagnostics_session: CookieHealth, +} + +impl TraceCookies { + /// Report whether exactly one valid diagnostics session was inspected. + /// + /// A false result means no valid session was observed, not cookie absence. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert!(!cookies.observed_active()); + /// ``` + #[must_use] + pub const fn observed_active(&self) -> bool { + matches!( + self.diagnostics_session.outcome, + CookieOutcome::PresentValid { + detail: ValidCookieDetail::DiagnosticsValue + } + ) + } + + pub(super) const fn new(health: [CookieHealth; 4]) -> Self { + Self { + ts_ec: health[0], + ts_eids: health[1], + ts_tester: health[2], + diagnostics_session: health[3], + } + } + + /// Borrow the frozen EC shape. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert_eq!(cookies.ts_ec().detail(), None); + /// ``` + #[must_use] + pub const fn ts_ec(&self) -> &CookieHealth { + &self.ts_ec + } + + /// Borrow the frozen EID shape. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert_eq!(cookies.ts_eids().detail(), None); + /// ``` + #[must_use] + pub const fn ts_eids(&self) -> &CookieHealth { + &self.ts_eids + } + + /// Borrow the frozen tester shape. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert_eq!(cookies.ts_tester().detail(), None); + /// ``` + #[must_use] + pub const fn ts_tester(&self) -> &CookieHealth { + &self.ts_tester + } + + /// Borrow the frozen diagnostics-session shape. + /// + /// # Examples + /// + /// ``` + /// let cookies = trusted_server_core::trace::inspect_cookies(&http::HeaderMap::new(), None); + /// assert_eq!(cookies.diagnostics_session().detail(), None); + /// ``` + #[must_use] + pub const fn diagnostics_session(&self) -> &CookieHealth { + &self.diagnostics_session + } +} + +/// Optional allowlisted network facts with bounded strings and a masked IP. +/// +/// Unsupported HTTP-version and POP enrichment remains absent in this phase. +#[derive(Clone, Debug, Serialize)] +pub struct TraceNetwork { + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) masked_client_ip: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) country: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) region: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) asn: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) tls_protocol: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) tls_cipher: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) edge_hostname: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(super) edge_region: Option, +} + +/// Immutable version-one request context without page or identity properties. +#[derive(Clone, Debug, Serialize)] +pub struct TraceRequestContextV1 { + pub(super) schema_version: u8, + pub(super) captured_at: String, + pub(super) network: TraceNetwork, + pub(super) cookies: TraceCookies, +} + +impl TraceRequestContextV1 { + /// Borrow the frozen cookie observations used by state and capture gates. + /// + /// # Examples + /// + /// ```ignore + /// let observed_active = context.cookies().observed_active(); + /// ``` + #[must_use] + pub const fn cookies(&self) -> &TraceCookies { + &self.cookies + } + + /// Borrow the exact UTC capture timestamp. + /// + /// # Examples + /// + /// ```ignore + /// assert!(context.captured_at().ends_with('Z')); + /// ``` + #[must_use] + pub fn captured_at(&self) -> &str { + &self.captured_at + } + + /// Borrow the allowlisted network projection. + /// + /// # Examples + /// + /// ```ignore + /// let network = serde_json::to_value(context.network())?; + /// ``` + #[must_use] + pub const fn network(&self) -> &TraceNetwork { + &self.network + } +} diff --git a/crates/trusted-server-integration-tests/Cargo.toml b/crates/trusted-server-integration-tests/Cargo.toml index 39f985860..a1a632b58 100644 --- a/crates/trusted-server-integration-tests/Cargo.toml +++ b/crates/trusted-server-integration-tests/Cargo.toml @@ -25,14 +25,18 @@ edgezero-core = { workspace = true } serde_json = { workspace = true } toml = { workspace = true } trusted-server-core = { workspace = true, features = ["test-utils"] } +url = { workspace = true } [dev-dependencies] async-trait = { workspace = true } +base64 = { workspace = true } axum = { workspace = true } bytes = { workspace = true } derive_more = { workspace = true } edgezero-adapter-axum = { workspace = true, features = ["axum"] } env_logger = { workspace = true } +futures = { workspace = true } +trusted-server-js = { workspace = true } error-stack = { workspace = true } http = { workspace = true } http-body-util = { workspace = true } @@ -41,6 +45,8 @@ log = { workspace = true } reqwest = { workspace = true, features = ["blocking", "cookies"] } scraper = { workspace = true } testcontainers = { workspace = true } +temp-env = { workspace = true } +tempfile = { workspace = true } tokio = { workspace = true, features = ["rt-multi-thread"] } tower = { workspace = true, features = ["util"] } trusted-server-adapter-axum = { path = "../trusted-server-adapter-axum" } diff --git a/crates/trusted-server-integration-tests/browser/global-setup.ts b/crates/trusted-server-integration-tests/browser/global-setup.ts index f54d92dbe..efded44d8 100644 --- a/crates/trusted-server-integration-tests/browser/global-setup.ts +++ b/crates/trusted-server-integration-tests/browser/global-setup.ts @@ -1,80 +1,98 @@ -import { writeFileSync } from "node:fs"; -import { resolve } from "node:path"; +import { writeFileSync } from 'node:fs' +import { resolve } from 'node:path' +import type { TestState } from './helpers/state.js' import { startContainer, startViceroy, stopContainer, stopViceroy, -} from "./helpers/infra.js"; +} from './helpers/infra.js' -const STATE_FILE = resolve(__dirname, ".browser-test-state.json"); +const STATE_FILE = resolve(__dirname, '.browser-test-state.json') const WASM_PATH = process.env.WASM_BINARY_PATH || resolve( __dirname, - "../../../target/wasm32-wasip1/release/trusted-server-adapter-fastly.wasm", - ); + '../../../target/wasm32-wasip1/release/trusted-server-adapter-fastly.wasm' + ) const VICEROY_CONFIG = process.env.VICEROY_CONFIG_PATH || resolve( __dirname, - "../../../target/integration-test-artifacts/configs/viceroy.toml", - ); + '../../../target/integration-test-artifacts/configs/viceroy.toml' + ) /** Persist current state so global-teardown can always clean up. */ -function writeState(state: { - baseUrl?: string; - containerId?: string; - viceroyPid?: number; - framework: string; -}): void { - writeFileSync(STATE_FILE, JSON.stringify(state, null, 2)); +function writeState(state: Partial & { framework: string }): void { + writeFileSync(STATE_FILE, JSON.stringify(state, null, 2)) } async function globalSetup(): Promise { - const framework = process.env.TEST_FRAMEWORK || "nextjs"; - let containerId: string | undefined; - let viceroyPid: number | undefined; + const framework = process.env.TEST_FRAMEWORK || 'nextjs' + let containerId: string | undefined + let viceroyPid: number | undefined + const extraViceroyPids: number[] = [] try { - console.log(`[global-setup] Starting ${framework} container...`); - containerId = await startContainer(framework); + console.log(`[global-setup] Starting ${framework} container...`) + containerId = await startContainer(framework) // Write partial state immediately so teardown can stop the container // even if Viceroy startup fails below. - writeState({ containerId, framework }); + writeState({ containerId, framework }) - console.log(`[global-setup] Starting Viceroy (WASM: ${WASM_PATH})...`); - const viceroy = await startViceroy(WASM_PATH, VICEROY_CONFIG); - viceroyPid = viceroy.process.pid; + console.log(`[global-setup] Starting Viceroy (WASM: ${WASM_PATH})...`) + const viceroy = await startViceroy(WASM_PATH, VICEROY_CONFIG) + viceroyPid = viceroy.process.pid - console.log(`[global-setup] Viceroy ready at ${viceroy.baseUrl}`); + console.log(`[global-setup] Viceroy ready at ${viceroy.baseUrl}`) // Write complete state for tests and teardown - writeState({ + const state: Partial & { framework: string } = { baseUrl: viceroy.baseUrl, containerId, viceroyPid, framework, - }); + extraViceroyPids, + } + writeState(state) + const traceConfigs = [ + [process.env.TRACE_VICEROY_CONFIG_PATH, 'traceBaseUrl'], + [process.env.TRACE_AUTH_VICEROY_CONFIG_PATH, 'traceAuthBaseUrl'], + ] as const + for (const [config, key] of traceConfigs) { + if (!config) continue + const fixture = await startViceroy(WASM_PATH, config) + if (fixture.process.pid !== undefined) + extraViceroyPids.push(fixture.process.pid) + state[key] = fixture.baseUrl + writeState(state) + } } catch (err) { // Clean up any resources that were started before re-throwing - console.error("[global-setup] Setup failed, cleaning up..."); - if (viceroyPid) await stopViceroy(viceroyPid); - if (containerId) stopContainer(containerId); + console.error('[global-setup] Setup failed, cleaning up...') + for (const pid of [viceroyPid, ...extraViceroyPids]) { + if (pid === undefined) continue + try { + await stopViceroy(pid) + } catch { + console.warn(`[global-setup] Could not stop owned runtime ${pid}`) + } + } + if (containerId) stopContainer(containerId) // Remove partial state file since we cleaned up manually try { - const { unlinkSync } = await import("node:fs"); - unlinkSync(STATE_FILE); + const { unlinkSync } = await import('node:fs') + unlinkSync(STATE_FILE) } catch { // State file may not exist } - throw err; + throw err } } -export default globalSetup; +export default globalSetup diff --git a/crates/trusted-server-integration-tests/browser/global-teardown.ts b/crates/trusted-server-integration-tests/browser/global-teardown.ts index 97bdf2828..dbc895199 100644 --- a/crates/trusted-server-integration-tests/browser/global-teardown.ts +++ b/crates/trusted-server-integration-tests/browser/global-teardown.ts @@ -1,33 +1,47 @@ -import { readFileSync, unlinkSync } from "node:fs"; -import { resolve } from "node:path"; -import { stopContainer, stopViceroy } from "./helpers/infra.js"; +import { readFileSync, unlinkSync } from 'node:fs' +import { resolve } from 'node:path' +import { stopContainer, stopViceroy } from './helpers/infra.js' -const STATE_FILE = resolve(__dirname, ".browser-test-state.json"); +const STATE_FILE = resolve(__dirname, '.browser-test-state.json') async function globalTeardown(): Promise { - let state: { containerId?: string; viceroyPid?: number }; + let state: { + containerId?: string + viceroyPid?: number + extraViceroyPids?: number[] + } try { - state = JSON.parse(readFileSync(STATE_FILE, "utf-8")); + state = JSON.parse(readFileSync(STATE_FILE, 'utf-8')) } catch { - console.warn("[global-teardown] No state file found, nothing to clean up"); - return; + console.warn('[global-teardown] No state file found, nothing to clean up') + return } - if (state.viceroyPid) { - console.log(`[global-teardown] Stopping Viceroy (pid: ${state.viceroyPid})`); - await stopViceroy(state.viceroyPid); + let cleanupError: unknown + const pids = new Set([state.viceroyPid, ...(state.extraViceroyPids ?? [])]) + for (const pid of pids) { + if (pid === undefined) continue + console.log(`[global-teardown] Stopping Viceroy (pid: ${pid})`) + try { + await stopViceroy(pid) + } catch (error) { + cleanupError ??= error + } } if (state.containerId) { - console.log(`[global-teardown] Stopping container ${state.containerId.slice(0, 12)}...`); - stopContainer(state.containerId); + console.log( + `[global-teardown] Stopping container ${state.containerId.slice(0, 12)}...` + ) + stopContainer(state.containerId) } try { - unlinkSync(STATE_FILE); + unlinkSync(STATE_FILE) } catch { // Already removed } + if (cleanupError) throw cleanupError } -export default globalTeardown; +export default globalTeardown diff --git a/crates/trusted-server-integration-tests/browser/helpers/state.ts b/crates/trusted-server-integration-tests/browser/helpers/state.ts index b8f5d4b66..1ca34367c 100644 --- a/crates/trusted-server-integration-tests/browser/helpers/state.ts +++ b/crates/trusted-server-integration-tests/browser/helpers/state.ts @@ -1,35 +1,42 @@ -import { readFileSync } from "node:fs"; -import { resolve } from "node:path"; +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' export interface TestState { - baseUrl: string; - containerId: string; - viceroyPid: number; - framework: string; + baseUrl: string + containerId: string + viceroyPid: number + framework: string + traceBaseUrl?: string + traceAuthBaseUrl?: string + extraViceroyPids?: number[] } -const KNOWN_FRAMEWORKS = ["nextjs", "wordpress"] as const; +const KNOWN_FRAMEWORKS = ['nextjs', 'wordpress'] as const -const STATE_FILE = resolve(__dirname, "../.browser-test-state.json"); -let cachedState: TestState | undefined; +const STATE_FILE = resolve(__dirname, '../.browser-test-state.json') +let cachedState: TestState | undefined /** Read the state written by global-setup.ts. */ export function readState(): TestState { - const state: TestState = JSON.parse(readFileSync(STATE_FILE, "utf-8")); - if (!KNOWN_FRAMEWORKS.includes(state.framework as (typeof KNOWN_FRAMEWORKS)[number])) { + const state: TestState = JSON.parse(readFileSync(STATE_FILE, 'utf-8')) + if ( + !KNOWN_FRAMEWORKS.includes( + state.framework as (typeof KNOWN_FRAMEWORKS)[number] + ) + ) { throw new Error( - `Unknown framework "${state.framework}" in state file. Expected one of: ${KNOWN_FRAMEWORKS.join(", ")}`, - ); + `Unknown framework "${state.framework}" in state file. Expected one of: ${KNOWN_FRAMEWORKS.join(', ')}` + ) } - return state; + return state } /** Read the state once and reuse it for the rest of the test process. */ function getCachedState(): TestState { - return (cachedState ??= readState()); + return (cachedState ??= readState()) } /** Resolve an absolute runtime URL from the current browser test state. */ export function runtimeUrl(path: string): string { - return new URL(path, getCachedState().baseUrl).toString(); + return new URL(path, getCachedState().baseUrl).toString() } diff --git a/crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts b/crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts new file mode 100644 index 000000000..eb02de88c --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts @@ -0,0 +1,59 @@ +import { expect, type Page } from '@playwright/test' +import { readState } from './state.js' + +/** Clicks the real handoff button inside the existing closed shadow root. */ +export async function clickTraceHandoff(page: Page): Promise { + const session = await page.context().newCDPSession(page) + try { + let backendNodeId: number | undefined + await expect + .poll(async () => { + const tree = await session.send('Accessibility.getFullAXTree') + const nodes = tree.nodes as Array<{ + ignored: boolean + backendDOMNodeId?: number + role?: { value?: string } + name?: { value?: string } + }> + const buttons = nodes.filter( + (node) => + !node.ignored && + node.role?.value === 'button' && + node.name?.value === 'View trace results' + ) + backendNodeId = + buttons.length === 1 ? buttons[0].backendDOMNodeId : undefined + return backendNodeId + }) + .toBeDefined() + const { model } = await session.send('DOM.getBoxModel', { backendNodeId }) + const points = model.border as number[] + // A real pointer click keeps browser user activation and the actual + // production button handler. No private callback or report is injected. + await page.mouse.click( + (points[0] + points[2] + points[4] + points[6]) / 4, + (points[1] + points[3] + points[5] + points[7]) / 4 + ) + } finally { + await session.detach() + } +} + +/** Resolves a browser-suitable localhost URL for the dedicated trace fixture. */ +export function traceRuntimeUrl(path: string, authenticated = false): string { + const state = readState() + const baseUrl = authenticated ? state.traceAuthBaseUrl : state.traceBaseUrl + if (!baseUrl) + throw new Error( + 'Run scripts/integration-tests-browser.sh to create the dedicated trace fixtures' + ) + const url = new URL(path, baseUrl) + url.hostname = 'localhost' + return url.toString() +} + +/** Fictional credentials already present in the integration secret-store fixture. */ +export const TRACE_FIXTURE_CREDENTIALS = { + username: 'admin', + password: 'integration-admin-password-32-bytes-ok', +} diff --git a/crates/trusted-server-integration-tests/browser/helpers/trace-gpt-fixture.js b/crates/trusted-server-integration-tests/browser/helpers/trace-gpt-fixture.js new file mode 100644 index 000000000..37644e6b7 --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/helpers/trace-gpt-fixture.js @@ -0,0 +1,146 @@ +;(() => { + const slots = new Map() + const listeners = new Map() + const requests = [] + let initialLoadDisabled = false + + function emit(name, slot, facts = {}) { + for (const listener of listeners.get(name) || []) + listener({ slot, ...facts }) + } + + const service = { + getSlots: () => [...slots.values()], + getTargeting: () => [], + getTargetingKeys: () => [], + setTargeting() { + return service + }, + clearTargeting() { + return service + }, + enableSingleRequest() {}, + disableInitialLoad() { + initialLoadDisabled = true + }, + isInitialLoadDisabled: () => initialLoadDisabled, + addEventListener(name, callback) { + const callbacks = listeners.get(name) || [] + callbacks.push(callback) + listeners.set(name, callbacks) + return service + }, + removeEventListener(name, callback) { + listeners.set( + name, + (listeners.get(name) || []).filter((entry) => entry !== callback) + ) + return service + }, + refresh(selected = [...slots.values()]) { + for (const slot of selected) { + if (slots.get(slot.getSlotElementId()) !== slot) continue + requests.push({ + slot, + targeting: Object.fromEntries( + slot.getTargetingKeys().map((key) => [key, slot.getTargeting(key)]) + ), + }) + emit('slotRequested', slot) + } + }, + } + + window.googletag = { + apiReady: true, + pubadsReady: true, + cmd: { + push(...callbacks) { + callbacks.forEach((callback) => callback()) + return callbacks.length + }, + }, + pubads: () => service, + defineSlot(path, sizes, id) { + const targeting = new Map() + const dimensions = typeof sizes[0] === 'number' ? [sizes] : sizes + const slot = { + getSlotElementId: () => id, + getAdUnitPath: () => path, + getSizes: () => + dimensions.map(([width, height]) => ({ + getWidth: () => width, + getHeight: () => height, + })), + getTargeting: (key) => targeting.get(key) || [], + getTargetingKeys: () => [...targeting.keys()], + setTargeting(key, value) { + targeting.set( + key, + (Array.isArray(value) ? value : [value]).map(String) + ) + return slot + }, + clearTargeting(key) { + if (key === undefined) targeting.clear() + else targeting.delete(key) + return slot + }, + updateTargetingFromMap(values) { + for (const [key, value] of Object.entries(values)) { + if (value === null) targeting.delete(key) + else slot.setTargeting(key, value) + } + return slot + }, + addService: () => slot, + setConfig: () => slot, + } + slots.set(id, slot) + return slot + }, + destroySlots(selected = [...slots.values()]) { + for (const slot of selected) slots.delete(slot.getSlotElementId()) + return true + }, + enableServices() {}, + display(target) { + const id = + typeof target === 'string' + ? target + : target.getSlotElementId?.() || target.id + const slot = slots.get(id) + if (slot && !initialLoadDisabled) service.refresh([slot]) + }, + getConfig: () => ({ disableInitialLoad: initialLoadDisabled }), + setConfig(config) { + if (typeof config.disableInitialLoad === 'boolean') + initialLoadDisabled = config.disableInitialLoad + }, + } + + // This fixture supplies GPT's documented callbacks. Activation and auction + // evidence come exclusively from the real publisher response and built TSJS. + window.__traceGptFixture = { + requests, + complete(index) { + const request = requests[index] + if (!request) throw new Error('should complete an observed GPT request') + const adId = request.targeting.hb_adid?.[0] + if (adId) { + const frame = document.createElement('iframe') + frame.width = '300' + frame.height = '250' + frame.src = `https://creative.example.com/fixture-puc?adId=${encodeURIComponent(adId)}` + document + .getElementById(request.slot.getSlotElementId()) + .replaceChildren(frame) + } + emit('slotResponseReceived', request.slot) + emit('slotRenderEnded', request.slot, { + isEmpty: !adId, + ...(adId ? { size: [300, 250] } : {}), + }) + }, + } +})() diff --git a/crates/trusted-server-integration-tests/browser/helpers/trace-report-fixture.ts b/crates/trusted-server-integration-tests/browser/helpers/trace-report-fixture.ts new file mode 100644 index 000000000..0452e5a71 --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/helpers/trace-report-fixture.ts @@ -0,0 +1,63 @@ +/** An explicitly browser-carried fixture; this does not prove live auction capture. */ +export function storedTraceReportFixture(origin: string, now = Date.now()) { + const capturedAt = new Date(now).toISOString() + const absent = { source: 'request', state: 'absent' } + const counts = { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 } + return { + stored_at_ms: now, + report: { + schema_version: 1, + captured_at: capturedAt, + request_context: { + schema_version: 1, + captured_at: capturedAt, + network: { + tls_cipher: '', + edge_hostname: `${'x'.repeat(116)}.example.com`, + }, + cookies: { + ts_ec: absent, + ts_eids: absent, + ts_tester: absent, + diagnostics_session: { + source: 'request', + state: 'unavailable', + detail: 'runtime_header_ambiguous', + }, + }, + }, + server_auctions: [], + slot_correlations: [], + gpt_diagnostics: { + schema_version: 1, + source_schema_version: 1, + capturedAt, + page: { origin, pathname: '/[redacted]' }, + slots: [], + callbackIssues: [], + coverage: { + slotRequested: counts, + slotResponseReceived: counts, + slotRenderEnded: counts, + slotOnload: counts, + impressionViewable: counts, + slotVisibilityChanged: counts, + }, + metadata: { + droppedCallbacks: 0, + evictedSlots: 0, + evictedRequestCycles: 0, + }, + }, + auction_coverage: { capture_status: 'not_observed', issues: [] }, + truncation: { + omitted_server_auctions: 0, + omitted_slot_correlations: 0, + omitted_request_cycles: 0, + omitted_callback_issues: 0, + omitted_attribution_issues: 0, + omitted_nested_values: 0, + }, + }, + } +} diff --git a/crates/trusted-server-integration-tests/browser/playwright.trace-runtime.config.ts b/crates/trusted-server-integration-tests/browser/playwright.trace-runtime.config.ts new file mode 100644 index 000000000..0f7a139c1 --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/playwright.trace-runtime.config.ts @@ -0,0 +1,47 @@ +import { defineConfig } from '@playwright/test' + +const origin = process.env.TRACE_BROWSER_ORIGIN +const runtime = process.env.TRACE_BROWSER_RUNTIME +if ( + !origin || + !runtime || + !['fastly', 'axum', 'cloudflare', 'spin'].includes(runtime) +) + throw new Error( + 'The Rust trace browser harness must supply its runtime and origin' + ) +const url = new URL(origin) +if ( + url.protocol !== 'http:' || + url.hostname !== 'localhost' || + !url.port || + url.username || + url.password || + url.pathname !== '/' || + url.search || + url.hash +) + throw new Error( + 'Trace browser tests require an explicit localhost HTTP origin' + ) + +// Rust owns the runtime and origin container. This config never starts or stops +// another fixture, and Playwright creates a fresh browser context for each test. +export default defineConfig({ + testDir: './tests', + testMatch: 'shared/mobile-trace-runtime.spec.ts', + timeout: 60_000, + retries: 0, + workers: 1, + use: { + baseURL: url.origin, + headless: true, + viewport: { width: 390, height: 844 }, + acceptDownloads: true, + screenshot: 'only-on-failure', + trace: 'retain-on-failure', + }, + projects: [{ name: runtime, use: { browserName: 'chromium' } }], + reporter: [['list']], + outputDir: `./test-results/trace-runtime-${runtime}`, +}) diff --git a/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts b/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts new file mode 100644 index 000000000..7e5ac4223 --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts @@ -0,0 +1,745 @@ +import { expect, test, type Page } from '@playwright/test' +import { createHash } from 'node:crypto' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { readState } from '../../helpers/state.js' +import { + clickTraceHandoff, + traceRuntimeUrl, +} from '../../helpers/trace-fixture.js' + +function bidderEndpoint(): string { + return `http://127.0.0.1:${process.env.INTEGRATION_ORIGIN_PORT || '8888'}/api/trace-bidder` +} + +const pucBanner = readFileSync( + resolve( + __dirname, + '../../node_modules/prebid-universal-creative/dist/banner.js' + ), + 'utf8' +) + +async function activatePublisher(page: Page, path: string): Promise { + await page.addInitScript({ + path: resolve(__dirname, '../../helpers/trace-gpt-fixture.js'), + }) + const publisher = traceRuntimeUrl(path) + await page.goto(publisher) + expect( + await page.evaluate(() => Reflect.get(window, '__tsjs_trace_active')) + ).toBeUndefined() + await page.goto(traceRuntimeUrl('/_ts/trace')) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toContainText( + 'Tracing is on' + ) + await page + .getByRole('button', { name: 'Return to previous page', exact: true }) + .click() + await page.waitForURL(publisher) + const response = await page.reload() + expect(response?.headers()['cache-control']).toContain('no-store') + await page.waitForFunction(() => { + const fixture = Reflect.get(window, '__traceGptFixture') + return ( + fixture?.requests.length > 0 && + Boolean(Reflect.get(window, 'tsjs')?.gptDiagnostics) + ) + }) + return publisher +} + +test.beforeEach(async ({ request }, testInfo) => { + if (readState().framework !== 'nextjs') testInfo.skip() + expect( + (await request.put(bidderEndpoint(), { data: { mode: 'empty' } })).status() + ).toBe(200) +}) + +test('captures a real zero-bid SSAT auction and carries its GPT cycle through the same-tab viewer', async ({ + page, + request, +}) => { + const observedRequests = (await (await request.get(bidderEndpoint())).json()) + .requests + const publisher = await activatePublisher( + page, + '/gpt-diagnostics?trace_fixture=empty' + ) + await page.evaluate(() => + Reflect.get(window, '__traceGptFixture').complete(0) + ) + await expect + .poll( + async () => (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBeGreaterThanOrEqual(observedRequests + 2) + const evidence = await page.evaluate(() => + Reflect.get(window, 'tsjs').traceEvidence.snapshot() + ) + expect(evidence.ok).toBe(true) + expect(evidence.value.serverAuctions).toHaveLength(1) + expect(evidence.value.serverAuctions[0]).toMatchObject({ + source: 'initial_navigation_ssat', + terminal_status: 'completed', + provider_calls: [ + { + provider_number: 1, + role: 'bidder', + status: 'no_bid', + returned_bid_count: 0, + }, + ], + slots: [ + { slot_number: 1, candidate: 'no_candidate', returned_bid_count: 0 }, + ], + }) + expect(evidence.value.slotCorrelations).toHaveLength(1) + expect(evidence.value.slotCorrelations[0].diagnostic_auction_id).toBe( + evidence.value.serverAuctions[0].diagnostic_auction_id + ) + expect(evidence.value.slotCorrelations[0].slot_ref).toBe( + evidence.value.serverAuctions[0].slots[0].slot_ref + ) + await clickTraceHandoff(page) + await page.waitForURL(traceRuntimeUrl('/_ts/trace')) + await expect( + page.getByRole('heading', { + name: 'Trusted Server trace results', + exact: true, + }) + ).toBeVisible() + await expect( + page.getByText('Browser-carried, unverified diagnostic data', { + exact: true, + }) + ).toBeVisible() + const report = await page.evaluate( + () => + JSON.parse(sessionStorage.getItem('trusted-server.trace.report.v1')!) + .report + ) + expect(report.server_auctions).toEqual(evidence.value.serverAuctions) + expect(report.slot_correlations).toEqual(evidence.value.slotCorrelations) + expect(report.gpt_diagnostics.page.origin).toBe(new URL(publisher).origin) + expect(report.gpt_diagnostics.page.pathname).toBe('/[redacted]') + expect(report.gpt_diagnostics.slots[0].requests[0].isEmpty).toBe(true) + await expect( + page.getByText('Auction 1: Initial-page server auction (SSAT)', { + exact: true, + }) + ).toBeVisible() + await expect( + page.getByText('Unavailable in v1', { + exact: true, + }) + ).toBeVisible() +}) + +test('retains the real failed-provider SSAT observation without claiming a delivered creative', async ({ + page, + request, +}) => { + expect( + (await request.put(bidderEndpoint(), { data: { mode: 'error' } })).status() + ).toBe(200) + const before = (await (await request.get(bidderEndpoint())).json()).requests + await activatePublisher(page, '/gpt-diagnostics') + await page.evaluate(() => + Reflect.get(window, '__traceGptFixture').complete(0) + ) + await expect + .poll( + async () => (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBeGreaterThanOrEqual(before + 2) + const evidence = await page.evaluate(() => + Reflect.get(window, 'tsjs').traceEvidence.snapshot() + ) + expect(evidence.ok).toBe(true) + expect(evidence.value.serverAuctions).toHaveLength(1) + expect(evidence.value.serverAuctions[0]).toMatchObject({ + source: 'initial_navigation_ssat', + terminal_status: 'completed', + provider_calls: [{ status: 'error', returned_bid_count: 0 }], + slots: [{ candidate: 'no_candidate', returned_bid_count: 0 }], + }) + await expect(page.locator('iframe')).toHaveCount(0) + await clickTraceHandoff(page) + await page.waitForURL(traceRuntimeUrl('/_ts/trace')) + await expect(page.locator('#trace-report')).toBeVisible() + await expect( + page.getByText('Trusted Server creative rendered', { exact: true }) + ).toHaveCount(0) + const report = await page.evaluate( + () => + JSON.parse(sessionStorage.getItem('trusted-server.trace.report.v1')!) + .report + ) + expect(report.server_auctions).toEqual(evidence.value.serverAuctions) + expect(JSON.stringify(report)).not.toContain('controlled fixture failure') +}) + +test('proves a selected SSAT creative through the real PUC message bridge before reporting participation', async ({ + page, + request, +}) => { + expect( + ( + await request.put(bidderEndpoint(), { data: { mode: 'selected' } }) + ).status() + ).toBe(200) + const before = (await (await request.get(bidderEndpoint())).json()).requests + const publisherOrigin = new URL(traceRuntimeUrl('/')).origin + await page.route( + 'https://creative.example.com/fixture-puc*', + async (route) => { + const url = new URL(route.request().url()) + if (url.pathname === '/fixture-puc.js') { + await route.fulfill({ contentType: 'text/javascript', body: pucBanner }) + return + } + await route.fulfill({ + contentType: 'text/html', + body: ``, + }) + } + ) + await activatePublisher(page, '/gpt-diagnostics') + await page.evaluate(() => + Reflect.get(window, '__traceGptFixture').complete(0) + ) + await expect + .poll(async () => { + const texts = await Promise.all( + page.frames().map((frame) => frame.locator('.marker').allTextContents()) + ) + return texts.flat() + }) + .toContain('Trace fixture creative') + await expect + .poll( + async () => (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBeGreaterThanOrEqual(before + 2) + await expect + .poll(() => + page.evaluate( + () => + Reflect.get(window, 'tsjs').gptDiagnostics.snapshot().slots[0] + ?.requests[0]?.delivery + ) + ) + .toBe('trusted_server_response_sent') + const evidence = await page.evaluate(() => + Reflect.get(window, 'tsjs').traceEvidence.snapshot() + ) + expect(evidence.ok).toBe(true) + expect(evidence.value.serverAuctions).toHaveLength(1) + expect(evidence.value.serverAuctions[0]).toMatchObject({ + source: 'initial_navigation_ssat', + terminal_status: 'completed', + slots: [{ candidate: 'selected', selected_creative_size: [300, 250] }], + }) + expect(evidence.value.slotCorrelations).toHaveLength(1) + expect(evidence.value.slotCorrelations[0]).toMatchObject({ + diagnostic_auction_id: + evidence.value.serverAuctions[0].diagnostic_auction_id, + slot_ref: evidence.value.serverAuctions[0].slots[0].slot_ref, + request_number: 1, + }) + await clickTraceHandoff(page) + await page.waitForURL(traceRuntimeUrl('/_ts/trace')) + await expect( + page + .locator('section') + .filter({ + has: page.getByRole('heading', { + name: 'Server auctions', + exact: true, + }), + }) + .getByText('Trusted Server creative rendered', { exact: true }) + ).toBeVisible() + const report = await page.evaluate( + () => + JSON.parse(sessionStorage.getItem('trusted-server.trace.report.v1')!) + .report + ) + expect(report.server_auctions).toEqual(evidence.value.serverAuctions) + expect(report.slot_correlations).toEqual(evidence.value.slotCorrelations) + expect(report.gpt_diagnostics.slots[0].requests[0]).toMatchObject({ + isEmpty: false, + delivery: 'trusted_server_response_sent', + trustedServerCreativeResponseAtMs: expect.any(Number), + renderAtMs: expect.any(Number), + }) + expect(JSON.stringify(report)).not.toContain('Trace fixture creative') + expect(JSON.stringify(report)).not.toContain('example-creative') +}) + +for (const mode of ['selected', 'empty'] as const) { + test(`captures the core /auction caller's real ${mode} response independently of GPT`, async ({ + page, + request, + }) => { + await activatePublisher(page, '/gpt-diagnostics') + expect( + (await request.put(bidderEndpoint(), { data: { mode } })).status() + ).toBe(200) + const before = (await (await request.get(bidderEndpoint())).json()).requests + const responsePromise = page.waitForResponse( + (response) => + new URL(response.url()).pathname === '/auction' && + response.request().method() === 'POST' + ) + await page.evaluate(() => { + const container = document.createElement('div') + container.id = 'example-core-api-slot' + document.body.append(container) + const api = Reflect.get(window, 'tsjs') + api.addAdUnits({ + code: container.id, + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bids: [{ bidder: 'example', params: {} }], + }) + api.requestAds() + }) + const response = await responsePromise + expect(response.status()).toBe(200) + expect(response.headers()['cache-control']).toContain('no-store') + const outgoing = response.request().postDataJSON() + const slotRef = outgoing.adUnits[0].ext.trusted_server.trace_slot_ref + expect(slotRef).toMatch( + /^ts-slot-[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/ + ) + const body = await response.json() + const evidence = body.ext?.trusted_server?.trace_auction?.evidence + expect(evidence).toMatchObject({ + source: 'auction_api', + terminal_status: 'completed', + provider_calls: [ + { + provider_number: 1, + role: 'bidder', + status: mode === 'selected' ? 'success' : 'no_bid', + returned_bid_count: mode === 'selected' ? 1 : 0, + }, + ], + slots: [ + { + slot_number: 1, + slot_ref: slotRef, + candidate: mode === 'selected' ? 'selected' : 'no_candidate', + returned_bid_count: mode === 'selected' ? 1 : 0, + }, + ], + }) + await expect + .poll( + async () => + (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBe(before + 1) + await expect + .poll(() => + page.evaluate(() => + Reflect.get(window, 'tsjs') + .traceEvidence.snapshot() + .value.serverAuctions.filter( + (auction: { source: string }) => auction.source === 'auction_api' + ) + ) + ) + .toEqual([evidence]) + const sidecars = await page.evaluate( + () => + Reflect.get(window, 'tsjs').traceEvidence.snapshot().value + .slotCorrelations + ) + expect( + sidecars.filter( + (sidecar: { diagnostic_auction_id: string }) => + sidecar.diagnostic_auction_id === evidence.diagnostic_auction_id + ) + ).toEqual([]) + const publicEvidence = JSON.stringify(evidence) + for (const forbidden of [ + 'example-core-api-slot', + 'example-bidder', + 'example-creative', + 'Trace fixture creative', + 'bidder.example.com', + ]) + expect(publicEvidence).not.toContain(forbidden) + if (mode === 'selected') { + await expect( + page.frameLocator('#example-core-api-slot iframe').locator('.marker') + ).toHaveText('Trace fixture creative') + } else { + await expect(page.locator('#example-core-api-slot iframe')).toHaveCount(0) + } + }) +} + +for (const transport of ['absent', 'malformed'] as const) { + test(`preserves the real core API creative when optional trace evidence is ${transport}`, async ({ + page, + request, + }) => { + await activatePublisher(page, '/gpt-diagnostics') + expect( + ( + await request.put(bidderEndpoint(), { data: { mode: 'selected' } }) + ).status() + ).toBe(200) + const before = (await (await request.get(bidderEndpoint())).json()).requests + await page.route('**/auction', async (route) => { + const response = await route.fetch() + const body = await response.json() + // Preserve the actual bidder response and ads. Alter only the optional + // diagnostic member to exercise missing and invalid transport behavior. + expect(body.ext?.trusted_server?.trace_auction?.evidence.source).toBe( + 'auction_api' + ) + if (transport === 'absent') delete body.ext.trusted_server.trace_auction + else body.ext.trusted_server.trace_auction = { schema_version: 999 } + await route.fulfill({ response, json: body }) + }) + await page.evaluate(() => { + const container = document.createElement('div') + container.id = 'example-core-api-slot' + document.body.append(container) + const api = Reflect.get(window, 'tsjs') + api.addAdUnits({ + code: container.id, + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bids: [{ bidder: 'example', params: {} }], + }) + api.requestAds() + }) + await expect( + page.frameLocator('#example-core-api-slot iframe').locator('.marker') + ).toHaveText('Trace fixture creative') + await expect + .poll( + async () => + (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBe(before + 1) + const captured = await page.evaluate(() => + Reflect.get(window, 'tsjs').traceEvidence.snapshot() + ) + expect(captured.ok).toBe(true) + expect( + captured.value.serverAuctions.filter( + (auction: { source: string }) => auction.source === 'auction_api' + ) + ).toEqual([]) + expect(captured.value.issues).toEqual( + transport === 'malformed' ? ['evidence_validation_failed'] : [] + ) + }) +} + +test('captures the pinned real Prebid adapter /auction response without a GPT join', async ({ + page, + request, +}) => { + await activatePublisher(page, '/gpt-diagnostics') + const dist = resolve(__dirname, '../../../../trusted-server-js/dist') + const manifest = JSON.parse( + readFileSync(resolve(dist, 'prebid/manifest.json'), 'utf8') + ) + const external = readFileSync(resolve(dist, 'prebid', manifest.filename)) + expect(createHash('sha256').update(external).digest('hex')).toBe( + manifest.sha256 + ) + await page.addScriptTag({ content: external.toString('utf8') }) + await page.addScriptTag({ path: resolve(dist, 'tsjs-prebid.js') }) + expect( + await page.evaluate(() => typeof Reflect.get(window, 'pbjs').onEvent) + ).toBe('function') + const before = (await (await request.get(bidderEndpoint())).json()).requests + const responsePromise = page.waitForResponse( + (response) => + new URL(response.url()).pathname === '/auction' && + response.request().method() === 'POST' + ) + await page.evaluate( + () => + new Promise((done) => { + Reflect.get(window, 'pbjs').requestBids({ + adUnits: [ + { + code: 'example-prebid-api-slot', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bids: [], + }, + ], + timeout: 3000, + bidsBackHandler: done, + }) + }) + ) + const response = await responsePromise + expect(response.status()).toBe(200) + expect(response.headers()['cache-control']).toContain('no-store') + const outgoing = response.request().postDataJSON() + const slotRef = outgoing.adUnits[0].ext.trusted_server.trace_slot_ref + expect(slotRef).toMatch( + /^ts-slot-[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/ + ) + const body = await response.json() + const auction = body.ext?.trusted_server?.trace_auction?.evidence + expect(auction).toMatchObject({ + source: 'auction_api', + terminal_status: 'completed', + provider_calls: [{ status: 'no_bid', returned_bid_count: 0 }], + slots: [ + { slot_ref: slotRef, candidate: 'no_candidate', returned_bid_count: 0 }, + ], + }) + await expect + .poll( + async () => (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBe(before + 1) + await expect + .poll(() => + page.evaluate(() => + Reflect.get(window, 'tsjs') + .traceEvidence.snapshot() + .value.serverAuctions.filter( + (record: { source: string }) => record.source === 'auction_api' + ) + ) + ) + .toEqual([auction]) + expect( + await page.evaluate( + (id) => + Reflect.get(window, 'tsjs') + .traceEvidence.snapshot() + .value.slotCorrelations.filter( + (sidecar: { diagnostic_auction_id: string }) => + sidecar.diagnostic_auction_id === id + ), + auction.diagnostic_auction_id + ) + ).toEqual([]) +}) + +for (const legacy of [false, true]) { + test(`carries real ${legacy ? 'legacy fallback' : 'canonical'} SPA page-bids into its exact GPT cycle`, async ({ + page, + request, + }) => { + await activatePublisher(page, '/gpt-diagnostics') + await page.evaluate(() => + Reflect.get(window, '__traceGptFixture').complete(0) + ) + const before = (await (await request.get(bidderEndpoint())).json()).requests + if (legacy) { + // Simulate an unavailable canonical route only. The legacy response and + // every auction/evidence payload still come from the real Rust server. + await page.route('**/_ts/page-bids?*', (route) => + route.fulfill({ status: 404, body: '' }) + ) + } + const endpoint = legacy ? '/__ts/page-bids' : '/_ts/page-bids' + const homeResponse = page.waitForResponse((response) => { + const url = new URL(response.url()) + return url.pathname === endpoint && url.searchParams.get('path') === '/' + }) + await page + .getByRole('navigation', { name: 'Fixture navigation', exact: true }) + .getByRole('link', { name: 'Home', exact: true }) + .click() + await page.waitForURL(traceRuntimeUrl('/')) + const home = await homeResponse + expect(home.status()).toBe(200) + const homeAuction = (await home.json()).trace_auction?.evidence + expect(homeAuction).toMatchObject({ + source: 'spa_page_bids', + terminal_status: 'skipped', + terminal_reason: 'no_eligible_slots', + slots: [], + }) + await expect + .poll(() => + page.evaluate( + (id) => + Reflect.get(window, 'tsjs') + .traceEvidence.snapshot() + .value.serverAuctions.some( + (auction: { diagnostic_auction_id: string }) => + auction.diagnostic_auction_id === id + ), + homeAuction.diagnostic_auction_id + ) + ) + .toBe(true) + const pageBidsResponse = page.waitForResponse((response) => { + const url = new URL(response.url()) + return ( + url.pathname === endpoint && + url.searchParams.get('path') === '/gpt-diagnostics' + ) + }) + await page.goBack() + await page.waitForURL(traceRuntimeUrl('/gpt-diagnostics')) + const response = await pageBidsResponse + expect(response.status()).toBe(200) + expect(response.headers()['cache-control']).toContain('no-store') + expect(response.request().headers()['x-tsjs-page-bids']).toBe( + legacy ? 'fallback' : '1' + ) + const body = await response.json() + const auction = body.trace_auction?.evidence + expect(auction).toMatchObject({ + source: 'spa_page_bids', + terminal_status: 'completed', + provider_calls: [{ status: 'no_bid', returned_bid_count: 0 }], + slots: [{ candidate: 'no_candidate', returned_bid_count: 0 }], + }) + expect(body.slots[0].ext.trusted_server.trace_slot_ref).toBe( + auction.slots[0].slot_ref + ) + await expect + .poll( + async () => + (await (await request.get(bidderEndpoint())).json()).requests + ) + .toBe(before + 1) + await page.waitForFunction( + () => Reflect.get(window, '__traceGptFixture').requests.length >= 2 + ) + await page.evaluate(() => { + const fixture = Reflect.get(window, '__traceGptFixture') + fixture.complete(fixture.requests.length - 1) + }) + await expect + .poll(() => + page.evaluate( + (id) => + Reflect.get(window, 'tsjs') + .traceEvidence.snapshot() + .value.slotCorrelations.filter( + (sidecar: { diagnostic_auction_id: string }) => + sidecar.diagnostic_auction_id === id + ), + auction.diagnostic_auction_id + ) + ) + .toEqual([ + { + schema_version: 1, + diagnostic_auction_id: auction.diagnostic_auction_id, + slot_ref: auction.slots[0].slot_ref, + runtime_slot_number: expect.any(Number), + request_number: expect.any(Number), + }, + ]) + await clickTraceHandoff(page) + await page.waitForURL(traceRuntimeUrl('/_ts/trace')) + const report = await page.evaluate( + () => + JSON.parse(sessionStorage.getItem('trusted-server.trace.report.v1')!) + .report + ) + expect(report.server_auctions).toContainEqual(auction) + expect( + report.server_auctions.some( + (record: { source: string }) => + record.source === 'initial_navigation_ssat' + ) + ).toBe(true) + await expect( + page + .locator('section') + .filter({ + has: page.getByRole('heading', { + name: 'Server auctions', + exact: true, + }), + }) + .locator(':scope > details > summary') + ).toHaveText([ + 'Auction 1: Initial-page server auction (SSAT)', + 'Auction 2: Trusted Server page-refresh auction', + 'Auction 3: Trusted Server page-refresh auction', + ]) + }) +} + +test('controlled bidder serves real OpenRTB selected, empty and error responses', async ({ + request, +}) => { + const endpoint = bidderEndpoint() + const before = await request.get(endpoint) + expect(before.status()).toBe(200) + const observedRequests = (await before.json()).requests + const auction = (mode: string) => ({ + id: 'example-request', + site: { + page: `https://publisher.example.com/gpt-diagnostics?trace_fixture=${mode}`, + }, + imp: [{ id: 'example-slot', banner: { format: [{ w: 300, h: 250 }] } }], + }) + const selected = await request.post(endpoint, { data: auction('selected') }) + expect(selected.status()).toBe(200) + expect(await selected.json()).toMatchObject({ + id: 'example-request', + cur: 'USD', + seatbid: [ + { + seat: 'example-bidder', + bid: [{ impid: 'example-slot', price: 1, w: 300, h: 250 }], + }, + ], + }) + const empty = await request.post(endpoint, { data: auction('empty') }) + expect(empty.status()).toBe(200) + expect(await empty.json()).toEqual({ + id: 'example-request', + cur: 'USD', + seatbid: [], + }) + const failed = await request.post(endpoint, { data: auction('error') }) + expect(failed.status()).toBe(503) + expect(await failed.json()).toEqual({ error: 'controlled fixture failure' }) + const after = await request.get(endpoint) + expect((await after.json()).requests).toBe(observedRequests + 3) + const control = await request.put(endpoint, { data: { mode: 'selected' } }) + expect(control.status()).toBe(200) + expect(await control.json()).toEqual({ mode: 'selected' }) + const controlled = await request.post(endpoint, { + data: { + ...auction('empty'), + site: { page: 'https://publisher.example.com/gpt-diagnostics' }, + }, + }) + expect((await controlled.json()).seatbid).toHaveLength(1) + const invalidControl = await request.put(endpoint, { + data: { mode: 'unexpected' }, + }) + expect(invalidControl.status()).toBe(400) + for (const invalidBody of ['null', '{']) { + const invalid = await request.put(endpoint, { + data: invalidBody, + headers: { 'Content-Type': 'application/json' }, + }) + expect(invalid.status()).toBe(400) + expect(await invalid.json()).toEqual({ error: 'invalid fixture mode' }) + } + await request.put(endpoint, { data: { mode: 'empty' } }) +}) diff --git a/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace-runtime.spec.ts b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace-runtime.spec.ts new file mode 100644 index 000000000..691224c5b --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace-runtime.spec.ts @@ -0,0 +1,225 @@ +import { expect, test } from '@playwright/test' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { clickTraceHandoff } from '../../helpers/trace-fixture.js' + +const SESSION = '__Host-ts-console' +const REPORT_KEY = 'trusted-server.trace.report.v1' +const PUBLISHER_PATH = '/gpt-diagnostics' + +test.skip( + !process.env.TRACE_BROWSER_ORIGIN, + 'Run the Rust-owned runtime workflow with playwright.trace-runtime.config.ts' +) + +test('stores a real browser session and captures, views, exports and clears its publisher report', async ({ + page, + baseURL, +}) => { + if (!baseURL) + throw new Error('The Rust harness must supply the runtime origin') + const publisher = new URL(PUBLISHER_PATH, baseURL).toString() + const setup = new URL('/_ts/trace', baseURL).toString() + const controls: string[] = [] + page.on('request', (request) => { + const path = new URL(request.url()).pathname + if ( + path.startsWith('/_ts/trace/') && + !path.startsWith('/_ts/trace/assets/') + ) + controls.push(`${request.method()} ${path}`) + }) + // Only GPT's callbacks are mocked. The adapter serves all activation, + // request-context, TSJS, capture, handoff and viewer bytes itself. + await page.addInitScript({ + path: resolve(__dirname, '../../helpers/trace-gpt-fixture.js'), + }) + await page.goto(publisher) + expect( + await page.evaluate(() => Reflect.get(window, '__tsjs_trace_active')) + ).toBeUndefined() + expect( + (await page.context().cookies()).find((cookie) => cookie.name === SESSION) + ).toBeUndefined() + + const shell = await page.goto(setup) + expect(shell?.status()).toBe(200) + expect(shell?.headers()['cache-control']).toBe('no-store, private') + expect(shell?.headers()['set-cookie']).toBeUndefined() + await expect( + page.getByRole('heading', { name: 'Setup request', exact: true }) + ).toBeVisible() + expect(controls).toEqual([]) + expect( + (await page.context().cookies()).find((cookie) => cookie.name === SESSION) + ).toBeUndefined() + const historyLength = await page.evaluate(() => history.length) + const observedState = page.waitForResponse( + (response) => + new URL(response.url()).pathname === '/_ts/trace/state' && + response.request().method() === 'GET' + ) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + expect(await (await observedState).json()).toEqual({ observed_active: true }) + await expect(page.locator('#trace-session-state')).toHaveText( + 'Tracing is on — cookie observed by server' + ) + expect(controls).toEqual(['POST /_ts/trace/enable', 'GET /_ts/trace/state']) + expect(await page.evaluate(() => history.length)).toBe(historyLength) + const cookie = (await page.context().cookies()).find( + (item) => item.name === SESSION + ) + expect(cookie).toMatchObject({ + value: '1', + domain: 'localhost', + path: '/', + secure: true, + httpOnly: true, + sameSite: 'Lax', + }) + expect( + Math.abs((cookie?.expires ?? 0) - (Date.now() / 1000 + 1800)) + ).toBeLessThan(10) + expect(await page.evaluate(() => document.cookie)).not.toContain(SESSION) + + await page + .getByRole('button', { name: 'Return to previous page', exact: true }) + .click() + await page.waitForURL(publisher) + const documentResponse = await page.reload() + expect(documentResponse?.status()).toBe(200) + expect(documentResponse?.headers()['cache-control']).toContain('no-store') + expect(documentResponse?.headers()['cache-control']).toContain('private') + await page.waitForFunction(() => { + const api = Reflect.get(window, 'tsjs') + return ( + Reflect.get(window, '__tsjs_trace_active') === true && + Boolean(api?.gptDiagnostics && api?.traceEvidence) + ) + }) + const context = await page.evaluate(() => { + const value = Reflect.get(window, '__tsjs_trace_request_context') + if ( + ![ + value, + value.network, + value.cookies, + ...Object.values(value.cookies), + ].every(Object.isFrozen) + ) + throw new Error('The publisher must emit frozen request context') + return value + }) + expect(context.cookies.diagnostics_session).toMatchObject({ + source: 'request', + state: 'present_valid', + detail: 'valid_diagnostics_value', + }) + + // No server opportunity or correlation tokens are invented for this disabled + // auction fixture. A client GPT cycle exercises the real observer and export. + await page.evaluate(() => { + const gpt = Reflect.get(window, 'googletag') + gpt + .defineSlot( + '/123456789/example-runtime', + [300, 250], + 'gpt-diagnostics-slot-primary' + ) + .addService(gpt.pubads()) + gpt.display('gpt-diagnostics-slot-primary') + Reflect.get(window, '__traceGptFixture').complete(0) + }) + await page.waitForFunction( + () => + Reflect.get(window, 'tsjs').gptDiagnostics.snapshot().slots[0] + ?.requests[0]?.isEmpty === true + ) + const evidence = await page.evaluate(() => + Reflect.get(window, 'tsjs').traceEvidence.snapshot() + ) + expect(evidence.ok).toBe(true) + expect(evidence.value.slotCorrelations).toEqual([]) + await page.evaluate(() => + sessionStorage.setItem('unrelated-fixture-key', 'retained') + ) + await clickTraceHandoff(page) + await page.waitForURL(setup) + const article = page.locator('#trace-report') + await expect(article).toContainText( + 'Browser-carried, unverified diagnostic data' + ) + const stored = await page.evaluate( + (key) => JSON.parse(sessionStorage.getItem(key)!), + REPORT_KEY + ) + expect(stored.report.request_context).toEqual(context) + expect(stored.report.server_auctions).toEqual(evidence.value.serverAuctions) + expect(stored.report.slot_correlations).toEqual([]) + expect(stored.report.auction_coverage.capture_status).toBe( + evidence.value.serverAuctions.length ? 'complete' : 'not_observed' + ) + expect(stored.report.gpt_diagnostics.page).toEqual({ + origin: baseURL, + pathname: '/[redacted]', + }) + expect(stored.report.gpt_diagnostics.slots[0].requests[0].isEmpty).toBe(true) + expect( + stored.report.gpt_diagnostics.slots[0].requests[0].trustedServerAuctionId + ).toBeUndefined() + + await page.context().grantPermissions(['clipboard-read', 'clipboard-write'], { + origin: baseURL, + }) + await page.getByRole('button', { name: 'Copy', exact: true }).focus() + await page.keyboard.press('Enter') + await expect(page.locator('#trace-export-status')).toHaveText('Copied JSON.') + const copied = await page.evaluate(() => navigator.clipboard.readText()) + const downloading = page.waitForEvent('download') + await page.getByRole('button', { name: 'Download', exact: true }).click() + const downloaded = await downloading + expect(downloaded.suggestedFilename()).toBe('trusted-server-trace-v1.json') + const destination = await downloaded.path() + expect(destination).not.toBeNull() + const bytes = readFileSync(destination!, 'utf8') + expect(bytes).toBe(copied) + expect(JSON.parse(bytes)).toEqual(stored.report) + expect(bytes).not.toContain('stored_at_ms') + expect(bytes).not.toContain('gpt-diagnostics-slot-primary') + expect(bytes).not.toContain('/123456789/example-runtime') + + controls.length = 0 + const endedState = page.waitForResponse( + (response) => + new URL(response.url()).pathname === '/_ts/trace/state' && + response.request().method() === 'GET' + ) + page.once('dialog', (dialog) => dialog.accept()) + await page + .getByRole('button', { name: 'Clear report and end tracing', exact: true }) + .click() + expect(await (await endedState).json()).toEqual({ observed_active: false }) + await expect(article).toHaveCount(0) + await expect(page.locator('#trace-cleanup-local-status')).toHaveText( + 'Local report deleted from this tab.' + ) + await expect(page.locator('#trace-cleanup-server-status')).toHaveText( + 'Tracing is off — no valid diagnostics session observed.' + ) + expect(controls).toEqual(['POST /_ts/trace/end', 'GET /_ts/trace/state']) + expect( + await page.evaluate((key) => sessionStorage.getItem(key), REPORT_KEY) + ).toBeNull() + expect( + await page.evaluate(() => sessionStorage.getItem('unrelated-fixture-key')) + ).toBe('retained') + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + await page.goto(publisher) + expect( + await page.evaluate(() => Reflect.get(window, '__tsjs_trace_active')) + ).toBeUndefined() +}) diff --git a/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts new file mode 100644 index 000000000..fa44e9fad --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts @@ -0,0 +1,666 @@ +import { expect, test } from '@playwright/test' +import { createHash } from 'node:crypto' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { runtimeUrl, readState } from '../../helpers/state.js' +import { + traceRuntimeUrl, + TRACE_FIXTURE_CREDENTIALS, +} from '../../helpers/trace-fixture.js' +import { storedTraceReportFixture } from '../../helpers/trace-report-fixture.js' + +const SESSION = '__Host-ts-console' +const CSP = + "default-src 'none'; script-src 'self'; style-src 'self'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'none'; connect-src 'self'; img-src data:" + +test.describe('mobile trace foundation', () => { + test('serves a read-only hardened shell and HEAD without a session mutation', async ({ + page, + request, + }) => { + const paths: string[] = [] + page.on('request', (request) => paths.push(new URL(request.url()).pathname)) + const response = await page.goto(traceRuntimeUrl('/_ts/trace')) + expect(response?.status()).toBe(200) + const headers = response?.headers() ?? {} + expect(headers['content-security-policy']).toBe(CSP) + expect(headers['cache-control']).toBe('no-store, private') + expect(headers['x-content-type-options']).toBe('nosniff') + expect(headers['referrer-policy']).toBe('no-referrer') + expect(headers['permissions-policy']).toBe( + 'camera=(), microphone=(), geolocation=(), payment=(), usb=()' + ) + expect(headers['set-cookie']).toBeUndefined() + await expect( + page.getByRole('heading', { name: 'Setup request', exact: true }) + ).toBeVisible() + expect(await page.context().cookies()).toEqual([]) + expect(await page.locator('link[rel="icon"]').getAttribute('href')).toMatch( + /^data:/ + ) + expect( + await page.locator('script:not([src]), style, [style]').count() + ).toBe(0) + expect(paths.sort()).toEqual([ + '/_ts/trace', + '/_ts/trace/assets/v1.css', + '/_ts/trace/assets/v1.js', + ]) + const head = await request.head(traceRuntimeUrl('/_ts/trace')) + expect(head.status()).toBe(200) + expect(await head.body()).toHaveLength(0) + expect(head.headers()['set-cookie']).toBeUndefined() + const manifest = JSON.parse( + readFileSync( + resolve( + __dirname, + '../../../../trusted-server-js/lib/trace-assets-manifest.json' + ), + 'utf8' + ) + ) as { assets: { path: string; sha256: string }[] } + for (const asset of manifest.assets) { + const delivered = await request.get(traceRuntimeUrl(asset.path)) + expect(delivered.status()).toBe(200) + expect( + createHash('sha256') + .update(await delivered.body()) + .digest('hex') + ).toBe(asset.sha256) + expect(delivered.headers()['etag']).toBe(`"${asset.sha256}"`) + expect(delivered.headers()['cache-control']).toBe( + 'public, max-age=31536000, immutable' + ) + expect(delivered.headers()['set-cookie']).toBeUndefined() + const assetHead = await request.head(traceRuntimeUrl(asset.path)) + expect(assetHead.status()).toBe(200) + expect(await assetHead.body()).toHaveLength(0) + expect(assetHead.headers()['etag']).toBe(delivered.headers()['etag']) + } + }) + + test('accepts the unchanged browser cookie only after an explicit tap and confirms through a separate request', async ({ + page, + }) => { + const requests: string[] = [] + page.on('request', (request) => { + if (new URL(request.url()).pathname.startsWith('/_ts/trace/')) + requests.push(`${request.method()} ${new URL(request.url()).pathname}`) + }) + await page.goto(traceRuntimeUrl('/_ts/trace')) + const history = await page.evaluate(() => window.history.length) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toHaveText( + 'Tracing is on — cookie observed by server' + ) + expect(requests.filter((entry) => !entry.includes('/assets/'))).toEqual([ + 'POST /_ts/trace/enable', + 'GET /_ts/trace/state', + ]) + expect(await page.evaluate(() => window.history.length)).toBe(history) + const cookie = (await page.context().cookies()).find( + (item) => item.name === SESSION + ) + expect(cookie).toMatchObject({ + value: '1', + domain: 'localhost', + path: '/', + secure: true, + httpOnly: true, + sameSite: 'Lax', + }) + expect( + Math.abs((cookie?.expires ?? 0) - (Date.now() / 1000 + 1800)) + ).toBeLessThan(10) + expect(await page.evaluate(() => document.cookie)).not.toContain(SESSION) + await expect(page.locator('#trace-status')).toContainText('reload once') + await page.getByRole('button', { name: 'End tracing', exact: true }).click() + await expect(page.locator('#trace-session-state')).toHaveText( + 'Tracing is off — no valid diagnostics session observed' + ) + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + }) + + test('GET control routes cannot activate or end a browser session', async ({ + request, + }) => { + for (const path of ['enable', 'end']) { + const response = await request.get(traceRuntimeUrl(`/_ts/trace/${path}`)) + expect(response.status()).toBe(405) + expect(response.headers()['set-cookie']).toBeUndefined() + expect(response.headers()['cache-control']).toBe('no-store, private') + } + const state = await request.get(traceRuntimeUrl('/_ts/trace/state')) + expect(await state.json()).toEqual({ observed_active: false }) + }) + + test('returns through history and activates only a freshly reloaded eligible publisher document', async ({ + page, + }) => { + await page.goto(traceRuntimeUrl('/')) + expect( + await page.evaluate(() => Reflect.get(window, '__tsjs_trace_active')) + ).toBeUndefined() + await page.goto(traceRuntimeUrl('/_ts/trace')) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toContainText( + 'Tracing is on' + ) + await page + .getByRole('button', { name: 'Return to previous page', exact: true }) + .click() + await page.waitForURL(traceRuntimeUrl('/')) + await page.reload() + await page.waitForFunction( + () => Reflect.get(window, '__tsjs_trace_active') === true + ) + expect( + await page.evaluate(() => { + const context = Reflect.get(window, '__tsjs_trace_request_context') + return [ + context, + context.network, + context.cookies, + ...Object.values(context.cookies), + ].every(Object.isFrozen) + }) + ).toBe(true) + await page.goto(traceRuntimeUrl('/?ts_console=0')) + expect( + await page.evaluate(() => Reflect.get(window, '__tsjs_trace_active')) + ).toBeUndefined() + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + }) + + test('protects shell and assets before feature or method handling when configured', async ({ + request, + }) => { + for (const path of [ + '/_ts/trace', + '/_ts/trace/assets/v1.js', + '/_ts/trace/not-a-route', + ]) { + const response = await request.get(traceRuntimeUrl(path, true)) + expect(response.status()).toBe(401) + expect(response.headers()['www-authenticate']).toBeDefined() + expect(response.headers()['cache-control']).toBe('no-store, private') + expect(response.headers()['set-cookie']).toBeUndefined() + } + const credentials = Buffer.from( + `${TRACE_FIXTURE_CREDENTIALS.username}:${TRACE_FIXTURE_CREDENTIALS.password}` + ).toString('base64') + const shell = await request.get(traceRuntimeUrl('/_ts/trace', true), { + headers: { Authorization: `Basic ${credentials}` }, + }) + expect(shell.status()).toBe(200) + const asset = await request.get( + traceRuntimeUrl('/_ts/trace/assets/v1.js', true), + { headers: { Authorization: `Basic ${credentials}` } } + ) + expect(asset.status()).toBe(200) + expect(asset.headers()['cache-control']).toBe('no-store, private') + }) + + test('leaves the disabled runtime inert even with a browser session cookie', async ({ + page, + }) => { + await page.goto(traceRuntimeUrl('/_ts/trace')) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toContainText( + 'Tracing is on' + ) + const disabled = new URL(runtimeUrl('/')) + disabled.hostname = 'localhost' + await page.goto(disabled.toString()) + expect( + await page.evaluate(() => ({ + active: Reflect.get(window, '__tsjs_trace_active'), + context: Reflect.get(window, '__tsjs_trace_request_context'), + evidence: Reflect.get(window, 'tsjs')?.traceEvidence, + })) + ).toEqual({ active: undefined, context: undefined, evidence: undefined }) + const response = await page + .context() + .request.get(new URL('/_ts/trace', disabled).toString()) + expect(response.status()).toBe(404) + }) + + test('rejects genuine cross-site form and fetch mutations without storing a cookie', async ({ + page, + }) => { + const origin = `http://127.0.0.1:${process.env.INTEGRATION_ORIGIN_PORT ?? '8888'}` + await page.goto(origin) + const destination = traceRuntimeUrl('/_ts/trace/enable') + const result = page.waitForResponse( + (response) => response.url() === destination + ) + await Promise.all([ + page.waitForURL(destination, { waitUntil: 'domcontentloaded' }), + page.evaluate((url) => { + const form = document.createElement('form') + form.method = 'POST' + form.action = url + document.body.append(form) + form.submit() + }, destination), + ]) + expect((await result).status()).toBe(403) + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + await page.goto(origin) + const fetchResponse = page.waitForResponse( + (response) => + response.url() === destination && response.request().method() === 'POST' + ) + await page.evaluate(async (url) => { + try { + await fetch(url, { + method: 'POST', + mode: 'no-cors', + credentials: 'include', + }) + } catch { + /* The browser may also block access to the response. */ + } + }, destination) + expect((await fetchResponse).status()).toBe(403) + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + expect(readState().framework).toMatch(/^(nextjs|wordpress)$/) + }) +}) + +test.describe('browser-carried trace report viewer', () => { + test('blocks injected scripts, styles, frames and off-origin connections under the delivered CSP', async ({ + page, + }) => { + await page.goto(traceRuntimeUrl('/_ts/trace')) + const requests: string[] = [] + page.on('request', (request) => requests.push(request.url())) + await page.evaluate(() => { + const violations: string[] = [] + Reflect.set(window, '__traceFixtureCspViolations', violations) + document.addEventListener('securitypolicyviolation', (event) => + violations.push(event.effectiveDirective) + ) + const script = document.createElement('script') + script.textContent = 'window.__traceFixtureInjectedScript = true' + document.body.append(script) + const style = document.createElement('style') + style.textContent = 'body { display: none }' + document.body.append(style) + const frame = document.createElement('iframe') + frame.src = 'https://blocked.example.com/trace-fixture' + document.body.append(frame) + void fetch('https://blocked.example.com/trace-fixture').catch(() => {}) + }) + await expect + .poll(() => + page.evaluate(() => Reflect.get(window, '__traceFixtureCspViolations')) + ) + .toEqual( + expect.arrayContaining([ + 'script-src-elem', + 'style-src-elem', + 'frame-src', + 'connect-src', + ]) + ) + expect( + await page.evaluate(() => + Reflect.get(window, '__traceFixtureInjectedScript') + ) + ).toBeUndefined() + await expect(page.locator('body')).toBeVisible() + expect(requests).toEqual([]) + }) + + test('treats an opener-cloned report as separate unverified tab data and expires it on a later load', async ({ + page, + }) => { + const url = traceRuntimeUrl('/_ts/trace') + await page.goto(url) + const captured = Date.now() + const stored = storedTraceReportFixture(new URL(url).origin, captured) + await page.evaluate((value) => { + sessionStorage.setItem( + 'trusted-server.trace.report.v1', + JSON.stringify(value) + ) + }, stored) + await page.reload() + const opening = page.waitForEvent('popup') + await page.evaluate((destination) => { + window.open(destination, '_blank') + }, url) + const clone = await opening + try { + await expect(clone.locator('#trace-report')).toContainText( + 'Browser-carried, unverified diagnostic data' + ) + await page + .getByRole('button', { name: 'Delete local report', exact: true }) + .click() + await expect(page.locator('#trace-report')).toHaveCount(0) + await expect(clone.locator('#trace-report')).toBeVisible() + expect( + await clone.evaluate( + () => + JSON.parse( + sessionStorage.getItem('trusted-server.trace.report.v1')! + ).report + ) + ).toEqual(stored.report) + // This is a browser clock-restoration simulation, not a browser restart. + await clone.clock.install({ time: captured + 16 * 60 * 1000 }) + await clone.reload() + await expect(clone.locator('#trace-report')).toHaveCount(0) + await expect(clone.locator('#trace-report-notice')).toContainText( + 'unavailable, expired, or unsupported' + ) + expect( + await clone.evaluate(() => + sessionStorage.getItem('trusted-server.trace.report.v1') + ) + ).toBeNull() + } finally { + await clone.close() + } + }) + + for (const rejection of [ + 'expired', + 'future clock', + 'different origin', + 'unsupported version', + 'forbidden member', + ] as const) { + test(`rejects and removes saved data for ${rejection} without rendering or exporting it`, async ({ + page, + }) => { + const url = traceRuntimeUrl('/_ts/trace') + await page.goto(url) + const now = Date.now() + const stored = storedTraceReportFixture( + new URL(url).origin, + rejection === 'expired' + ? now - 15 * 60 * 1000 - 1000 + : rejection === 'future clock' + ? now + 2 * 60 * 1000 + : now + ) + if (rejection === 'different origin') + stored.report.gpt_diagnostics.page.origin = 'https://other.example.com' + if (rejection === 'unsupported version') stored.report.schema_version = 2 + if (rejection === 'forbidden member') + Reflect.set(stored.report, 'raw_cookie', 'forbidden-fixture-value') + await page.evaluate((value) => { + sessionStorage.setItem( + 'trusted-server.trace.report.v1', + JSON.stringify(value) + ) + sessionStorage.setItem('unrelated-fixture-key', 'retained') + }, stored) + await page.reload() + await expect(page.locator('#trace-report')).toHaveCount(0) + await expect(page.locator('#trace-report-notice')).toContainText( + 'unavailable, expired, or unsupported' + ) + await expect( + page.getByRole('button', { name: 'Download', exact: true }) + ).toHaveCount(0) + expect( + await page.evaluate(() => ({ + report: sessionStorage.getItem('trusted-server.trace.report.v1'), + unrelated: sessionStorage.getItem('unrelated-fixture-key'), + })) + ).toEqual({ report: null, unrelated: 'retained' }) + await expect(page.locator('body')).not.toContainText( + 'forbidden-fixture-value' + ) + }) + } + + test('keeps a report exportable when local deletion fails while independently ending the real server session', async ({ + page, + }) => { + const url = traceRuntimeUrl('/_ts/trace') + await page.goto(url) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toContainText( + 'Tracing is on' + ) + const stored = storedTraceReportFixture(new URL(url).origin) + await page.evaluate((value) => { + sessionStorage.setItem( + 'trusted-server.trace.report.v1', + JSON.stringify(value) + ) + }, stored) + await page.addInitScript(() => { + const remove = Storage.prototype.removeItem + Reflect.set(window, '__traceFixtureDeletionBlocked', true) + Storage.prototype.removeItem = function (key: string) { + if ( + key === 'trusted-server.trace.report.v1' && + Reflect.get(window, '__traceFixtureDeletionBlocked') + ) + throw new DOMException('Fixture deletion blocked', 'SecurityError') + return remove.call(this, key) + } + }) + await page.reload() + page.once('dialog', (dialog) => dialog.accept()) + await page + .getByRole('button', { + name: 'Clear report and end tracing', + exact: true, + }) + .click() + await expect(page.locator('#trace-cleanup-local-status')).toContainText( + 'Local report deletion failed' + ) + await expect(page.locator('#trace-cleanup-server-status')).toContainText( + 'Tracing is off' + ) + await expect(page.locator('#trace-report')).toBeVisible() + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + const download = page.waitForEvent('download') + await page.getByRole('button', { name: 'Download', exact: true }).click() + const file = await (await download).path() + expect(JSON.parse(readFileSync(file!, 'utf8'))).toEqual(stored.report) + await page.evaluate(() => + Reflect.set(window, '__traceFixtureDeletionBlocked', false) + ) + await page + .getByRole('button', { name: 'Delete local report', exact: true }) + .click() + await expect(page.locator('#trace-report')).toHaveCount(0) + await expect(page.locator('#trace-cleanup-local-status')).toHaveText( + 'Local report deleted from this tab.' + ) + }) + + test('deletes the report offline and retries end with a separate server observation after reconnecting', async ({ + page, + }) => { + const url = traceRuntimeUrl('/_ts/trace') + await page.goto(url) + await page + .getByRole('button', { name: 'Enable tracing', exact: true }) + .click() + await expect(page.locator('#trace-session-state')).toContainText( + 'Tracing is on' + ) + await page.evaluate( + (value) => { + sessionStorage.setItem( + 'trusted-server.trace.report.v1', + JSON.stringify(value) + ) + }, + storedTraceReportFixture(new URL(url).origin) + ) + await page.reload() + await page.context().setOffline(true) + page.once('dialog', (dialog) => dialog.accept()) + await page + .getByRole('button', { + name: 'Clear report and end tracing', + exact: true, + }) + .click() + await expect(page.locator('#trace-report')).toHaveCount(0) + await expect(page.locator('#trace-cleanup-local-status')).toHaveText( + 'Local report deleted from this tab.' + ) + await expect(page.locator('#trace-cleanup-server-status')).toContainText( + 'End tracing unconfirmed' + ) + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeDefined() + const requests: string[] = [] + page.on('request', (request) => { + const path = new URL(request.url()).pathname + if (path === '/_ts/trace/end' || path === '/_ts/trace/state') + requests.push(`${request.method()} ${path}`) + }) + await page.context().setOffline(false) + await page + .getByRole('button', { name: 'Retry end tracing', exact: true }) + .click() + await expect(page.locator('#trace-cleanup-server-status')).toContainText( + 'Tracing is off' + ) + expect(requests).toEqual(['POST /_ts/trace/end', 'GET /_ts/trace/state']) + expect( + (await page.context().cookies()).find((item) => item.name === SESSION) + ).toBeUndefined() + }) + + test('renders safely at 320 px and exports the same report through real clipboard and Blob download', async ({ + page, + }) => { + await page.setViewportSize({ width: 320, height: 640 }) + const origin = new URL(traceRuntimeUrl('/_ts/trace')).origin + await page + .context() + .grantPermissions(['clipboard-read', 'clipboard-write'], { origin }) + await page.goto(traceRuntimeUrl('/_ts/trace')) + const stored = storedTraceReportFixture(origin) + await page.evaluate((value) => { + sessionStorage.setItem( + 'trusted-server.trace.report.v1', + JSON.stringify(value) + ) + sessionStorage.setItem('unrelated-fixture-key', 'retained') + }, stored) + const paths: string[] = [] + page.on('request', (request) => paths.push(new URL(request.url()).pathname)) + await page.reload() + const article = page.locator('#trace-report') + await expect(article).toContainText( + 'Browser-carried, unverified diagnostic data' + ) + await expect(article).toContainText( + stored.report.request_context.network.tls_cipher + ) + expect( + await article.locator('img, script, iframe, style, [style]').count() + ).toBe(0) + expect( + await page.locator('#trace-viewer-setup').getAttribute('open') + ).toBeNull() + expect( + await page.evaluate( + () => document.documentElement.scrollWidth <= innerWidth + ) + ).toBe(true) + for (const label of [ + 'Copy', + 'Download', + 'Share', + 'Clear report and end tracing', + 'Delete local report', + ]) { + const button = page.getByRole('button', { name: label, exact: true }) + const box = await button.boundingBox() + expect(box?.width).toBeGreaterThanOrEqual(44) + expect(box?.height).toBeGreaterThanOrEqual(44) + } + expect(paths.sort()).toEqual([ + '/_ts/trace', + '/_ts/trace/assets/v1.css', + '/_ts/trace/assets/v1.js', + ]) + const copy = page.getByRole('button', { name: 'Copy', exact: true }) + await copy.focus() + await page.keyboard.press('Enter') + await expect(page.locator('#trace-export-status')).toHaveText( + 'Copied JSON.' + ) + const copied = await page.evaluate(() => navigator.clipboard.readText()) + const downloading = page.waitForEvent('download') + await page.getByRole('button', { name: 'Download', exact: true }).click() + const downloaded = await downloading + expect(downloaded.suggestedFilename()).toBe('trusted-server-trace-v1.json') + const destination = await downloaded.path() + expect(destination).not.toBeNull() + const file = readFileSync(destination!, 'utf8') + expect(file).toBe(copied) + expect(JSON.parse(file)).toEqual(stored.report) + expect(file).not.toContain('stored_at_ms') + await expect(page.locator('#trace-export-status')).toHaveAttribute( + 'aria-live', + 'polite' + ) + const mutations: string[] = [] + page.on('request', (request) => { + if ( + new URL(request.url()).pathname.startsWith('/_ts/trace/') && + !request.url().includes('/assets/') + ) + mutations.push(`${request.method()} ${new URL(request.url()).pathname}`) + }) + page.once('dialog', (dialog) => dialog.accept()) + await page + .getByRole('button', { + name: 'Clear report and end tracing', + exact: true, + }) + .click() + await expect(article).toHaveCount(0) + await expect(page.locator('#trace-cleanup-local-status')).toHaveText( + 'Local report deleted from this tab.' + ) + await expect(page.locator('#trace-cleanup-server-status')).toHaveText( + 'Tracing is off — no valid diagnostics session observed.' + ) + expect( + await page.evaluate(() => + sessionStorage.getItem('trusted-server.trace.report.v1') + ) + ).toBeNull() + expect( + await page.evaluate(() => sessionStorage.getItem('unrelated-fixture-key')) + ).toBe('retained') + expect(mutations).toEqual(['POST /_ts/trace/end', 'GET /_ts/trace/state']) + }) +}) diff --git a/crates/trusted-server-integration-tests/browser/trace-fixture.test.cjs b/crates/trusted-server-integration-tests/browser/trace-fixture.test.cjs new file mode 100644 index 000000000..2e7f9d90a --- /dev/null +++ b/crates/trusted-server-integration-tests/browser/trace-fixture.test.cjs @@ -0,0 +1,94 @@ +const assert = require('node:assert/strict') +const { readFileSync } = require('node:fs') +const path = require('node:path') +const { test } = require('node:test') +const vm = require('node:vm') +const typescript = require('../../trusted-server-js/lib/node_modules/typescript') + +function load(filename, infra, state = {}) { + const saved = [] + const removed = [] + const fs = { + writeFileSync: (_filename, value) => saved.push(JSON.parse(value)), + readFileSync: () => JSON.stringify(state), + unlinkSync: (filename) => removed.push(filename), + } + const source = typescript.transpileModule( + readFileSync(path.join(__dirname, filename), 'utf8'), + { + compilerOptions: { + module: typescript.ModuleKind.CommonJS, + target: typescript.ScriptTarget.ES2022, + }, + } + ).outputText + const module = { exports: {} } + vm.runInNewContext(source, { + module, + exports: module.exports, + __dirname, + process: { + env: { + TEST_FRAMEWORK: 'nextjs', + TRACE_VICEROY_CONFIG_PATH: 'trace-public-fixture.toml', + TRACE_AUTH_VICEROY_CONFIG_PATH: 'trace-auth-fixture.toml', + }, + }, + console: { log() {}, warn() {}, error() {} }, + require(name) { + if (name === 'node:fs') return fs + if (name === 'node:path') return path + if (name === './helpers/infra.js') return infra + throw new Error(`should explicitly mock dependency ${name}`) + }, + }) + return { run: module.exports.default, saved, removed } +} + +test('failed trace runtime setup stops every already allocated runtime and the container even if one stop fails', async () => { + const stopped = [] + let allocations = 0 + const fixture = load('global-setup.ts', { + startContainer: async () => 'owned-container', + startViceroy: async () => { + allocations += 1 + if (allocations === 3) throw new Error('fixture-auth-runtime-failed') + return { + process: { pid: allocations * 101 }, + baseUrl: `http://127.0.0.1:${8000 + allocations}`, + } + }, + stopViceroy: async (pid) => { + stopped.push(pid) + if (pid === 101) throw new Error('fixture-first-stop-failed') + }, + stopContainer: (id) => stopped.push(id), + }) + await assert.rejects(fixture.run(), /fixture-auth-runtime-failed/) + assert.deepEqual(stopped, [101, 202, 'owned-container']) + assert.equal(fixture.removed.length, 1) + assert.equal(fixture.saved.at(-1).traceBaseUrl, 'http://127.0.0.1:8002') + assert.deepEqual(fixture.saved.at(-1).extraViceroyPids, [202]) +}) + +test('normal teardown attempts all owned resources and removes state after a runtime stop failure', async () => { + const stopped = [] + const fixture = load( + 'global-teardown.ts', + { + stopViceroy: async (pid) => { + stopped.push(pid) + if (pid === 101) throw new Error('fixture-first-stop-failed') + }, + stopContainer: (id) => stopped.push(id), + }, + { + viceroyPid: 101, + extraViceroyPids: [202, 303], + containerId: 'owned-container', + } + ) + await assert.rejects(fixture.run(), /fixture-first-stop-failed/) + assert.deepEqual(stopped, [101, 202, 303, 'owned-container']) + assert.equal(fixture.removed.length, 1) +}) diff --git a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml new file mode 100644 index 000000000..a94f7c515 --- /dev/null +++ b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml @@ -0,0 +1,40 @@ +[[handlers]] +path = "^/_ts/trace" +username = "admin" +password = "integration_admin_password" + +[[handlers]] +path = "^/_ts/admin" +username = "admin" +password = "integration_admin_password" + +[publisher] +domain = "localhost" +cookie_domain = "localhost" +origin_url = "http://127.0.0.1:8888" +proxy_secret = "integration_proxy_secret" + +[ec] +passphrase = "integration_ec_passphrase" +ec_store = "ec_identity_store" + +[[ec.partners]] +name = "Example partner" +source_domain = "partner.example.com" +bidstream_enabled = true +api_token = "integration_partner_token_alpha" + +[request_signing] +enabled = false +config_store_id = "app_config" +secret_store_id = "secrets" + +[integrations.gpt_diagnostics] +enabled = true +trace_page_enabled = true + +[proxy] +certificate_check = false + +[auction] +enabled = false diff --git a/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml new file mode 100644 index 000000000..dcc6b9b1c --- /dev/null +++ b/crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml @@ -0,0 +1,61 @@ +[[handlers]] +path = "^/_ts/admin" +username = "admin" +password = "integration_admin_password" + +[publisher] +domain = "localhost" +cookie_domain = "localhost" +origin_url = "http://127.0.0.1:8888" +proxy_secret = "integration_proxy_secret" + +[ec] +passphrase = "integration_ec_passphrase" +ec_store = "ec_identity_store" + +[[ec.partners]] +name = "Example partner" +source_domain = "partner.example.com" +bidstream_enabled = true +api_token = "integration_partner_token_alpha" + +[request_signing] +enabled = false +config_store_id = "app_config" +secret_store_id = "secrets" + +[integrations.gpt_diagnostics] +enabled = true +trace_page_enabled = true + +[integrations.gpt] +enabled = true +gam_attribution_enabled = false +script_url = "https://ads.example.com/gpt.js" +rewrite_script = false + +[proxy] +certificate_check = false + +[auction] +enabled = true +timeout_ms = 1000 +rewrite_creatives = false + +[auction.providers.example] +protocol = "openrtb-2.6" +profile = "standard" +endpoint = "https://bidder.example.com/api/trace-bidder" +routing = "all_eligible" + +[creative_opportunities] +enabled = true +gam_network_id = "123456789" +auction_timeout_ms = 1000 + +[[creative_opportunities.slot]] +id = "example-primary" +div_id = "gpt-diagnostics-slot-primary" +gam_unit_path = "/123456789/example-primary" +page_patterns = ["/gpt-diagnostics"] +formats = [{ width = 300, height = 250 }] diff --git a/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/app/api/trace-bidder/route.ts b/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/app/api/trace-bidder/route.ts new file mode 100644 index 000000000..f10cc315e --- /dev/null +++ b/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/app/api/trace-bidder/route.ts @@ -0,0 +1,73 @@ +import { NextResponse } from 'next/server' + +export const dynamic = 'force-dynamic' + +let requestCount = 0 +let controlledMode = 'empty' + +/** Counts actual HTTP invocations without retaining bidder request bodies. */ +export async function GET() { + return NextResponse.json({ requests: requestCount }) +} + +/** Selects one bounded scenario for the next real server-side auctions. */ +export async function PUT(request: Request) { + let control: unknown + try { + control = await request.json() + } catch { + return NextResponse.json({ error: 'invalid fixture mode' }, { status: 400 }) + } + if ( + typeof control !== 'object' || + control === null || + !('mode' in control) || + typeof control.mode !== 'string' || + !['selected', 'empty', 'error'].includes(control.mode) + ) { + return NextResponse.json({ error: 'invalid fixture mode' }, { status: 400 }) + } + controlledMode = control.mode + return NextResponse.json({ mode: controlledMode }) +} + +/** Controlled fictional OpenRTB bidder reached by the real Rust HTTP client. */ +export async function POST(request: Request) { + const auction = await request.json() + requestCount += 1 + const mode = + new URL( + auction.site?.page || 'https://publisher.example.com/' + ).searchParams.get('trace_fixture') || controlledMode + if (mode === 'error') { + return NextResponse.json( + { error: 'controlled fixture failure' }, + { status: 503 } + ) + } + const bids = + mode === 'selected' && Array.isArray(auction.imp) + ? auction.imp.map( + ( + impression: { + id: string + banner?: { format?: { w: number; h: number }[] } + }, + index: number + ) => ({ + id: `example-bid-${index}`, + impid: impression.id, + price: 1, + w: impression.banner?.format?.[0]?.w || 300, + h: impression.banner?.format?.[0]?.h || 250, + crid: 'example-creative', + adm: '
Trace fixture creative
', + }) + ) + : [] + return NextResponse.json({ + id: auction.id, + cur: 'USD', + seatbid: bids.length ? [{ seat: 'example-bidder', bid: bids }] : [], + }) +} diff --git a/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs b/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs index aada5dcf5..741c4237c 100644 --- a/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs +++ b/crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs @@ -1,11 +1,15 @@ +use std::collections::BTreeSet; use std::env; use std::error::Error; use std::fs; use std::path::PathBuf; +use std::time::Duration; use edgezero_core::blob_envelope::BlobEnvelope; use trusted_server_core::config::TrustedServerAppConfig; use trusted_server_core::config_payload::{CONFIG_BLOB_KEY, DEFAULT_CONFIG_STORE_ID}; +use trusted_server_core::platform::{BackendNamingPolicy, PlatformBackendSpec}; +use url::{Host, Url}; const GENERATED_AT: &str = "2026-06-23T00:00:00Z"; const GENERATED_STORES_MARKER: &str = " # GENERATED_TRUSTED_SERVER_CONFIG_STORES"; @@ -18,6 +22,7 @@ struct Args { app_config: PathBuf, output: PathBuf, origin_url: Option, + bidder_origin_url: Option, } fn main() -> Result<(), DynError> { @@ -39,7 +44,11 @@ fn run(args: &Args) -> Result<(), DynError> { })?; let envelope_json = build_app_config_envelope(&app_config, args.origin_url.as_deref())?; - let generated_config = inject_generated_config_stores(&template, &envelope_json)?; + let mut generated_config = inject_generated_config_stores(&template, &envelope_json)?; + if let Some(origin) = &args.bidder_origin_url { + generated_config = + inject_controlled_bidder_backends(&generated_config, &app_config, origin)?; + } if let Some(parent) = args.output.parent() { fs::create_dir_all(parent).map_err(|error| { @@ -64,6 +73,7 @@ fn parse_args(args: impl IntoIterator) -> Result let mut app_config = None; let mut output = None; let mut origin_url = None; + let mut bidder_origin_url = None; let mut iter = args.into_iter(); while let Some(arg) = iter.next() { @@ -72,6 +82,9 @@ fn parse_args(args: impl IntoIterator) -> Result "--app-config" => app_config = Some(next_path_arg(&mut iter, "--app-config")?), "--output" => output = Some(next_path_arg(&mut iter, "--output")?), "--origin-url" => origin_url = Some(next_string_arg(&mut iter, "--origin-url")?), + "--bidder-origin-url" => { + bidder_origin_url = Some(next_string_arg(&mut iter, "--bidder-origin-url")?); + } "--help" | "-h" => return Err(error_box(usage())), other => { return Err(error_box(format!( @@ -89,9 +102,99 @@ fn parse_args(args: impl IntoIterator) -> Result .ok_or_else(|| error_box(format!("missing --app-config\n\n{}", usage())))?, output: output.ok_or_else(|| error_box(format!("missing --output\n\n{}", usage())))?, origin_url, + bidder_origin_url, }) } +fn inject_controlled_bidder_backends( + runtime_config: &str, + app_config: &str, + origin: &str, +) -> Result { + let origin = Url::parse(origin)?; + let local = match origin.host() { + Some(Host::Domain("localhost")) => true, + Some(Host::Ipv4(address)) => address.is_loopback(), + Some(Host::Ipv6(address)) => address.is_loopback(), + _ => false, + }; + if !local + || origin.scheme() != "http" + || !origin.username().is_empty() + || origin.password().is_some() + || origin.path() != "/" + || origin.query().is_some() + || origin.fragment().is_some() + { + return Err(error_box( + "bidder fixture requires a plain loopback HTTP origin", + )); + } + let settings = toml::from_str::(app_config)?.into_settings(); + let logical_budget = settings.auction.timeout_ms.max( + settings + .creative_opportunities + .as_ref() + .and_then(|config| config.auction_timeout_ms) + .unwrap_or(settings.auction.timeout_ms), + ); + if logical_budget > 60_000 { + return Err(error_box( + "controlled bidder fixture budget exceeds 60 seconds", + )); + } + let plan = trusted_server_core::auction::compile_auction_plan(&settings) + .map_err(|_| error_box("invalid controlled bidder auction plan"))?; + let mut runtime: toml::Value = toml::from_str(runtime_config)?; + let backends = runtime + .get_mut("local_server") + .and_then(|server| server.get_mut("backends")) + .and_then(toml::Value::as_table_mut) + .ok_or_else(|| error_box("runtime template requires a backend table"))?; + for provider in plan.providers() { + let endpoint = Url::parse(provider.endpoint.as_str())?; + // Reuse production policies for every reachable timer/name, rather + // than duplicating Fastly's quantization or backend-name algorithm. + let timers = (1..=provider.timeout_ms.min(logical_budget)) + .map(|remaining| { + BackendNamingPolicy::Fastly + .canonicalize_transport_timeout_ms(remaining, provider.timeout_ms) + }) + .filter(|timer| *timer != 0) + .collect::>(); + for timer in timers { + let duration = Duration::from_millis(u64::from(timer)); + let spec = PlatformBackendSpec { + scheme: endpoint.scheme().to_owned(), + host: endpoint + .host_str() + .ok_or_else(|| error_box("missing bidder host"))? + .to_owned(), + port: endpoint.port(), + host_header_override: None, + certificate_check: true, + first_byte_timeout: duration, + between_bytes_timeout: duration, + discriminator: Some(provider.id.as_str().to_owned()), + }; + let name = BackendNamingPolicy::Fastly + .predict(&spec) + .map_err(|_| error_box("invalid controlled bidder backend name"))? + .name; + let alias = toml::Value::Table(toml::Table::from_iter([( + "url".to_owned(), + toml::Value::String(origin.as_str().to_owned()), + )])); + if backends.insert(name, alias).is_some() { + return Err(error_box( + "controlled bidder alias collides with template backend", + )); + } + } + } + Ok(toml::to_string(&runtime)?) +} + fn next_path_arg( iter: &mut impl Iterator, flag: &'static str, @@ -108,7 +211,7 @@ fn next_string_arg( } fn usage() -> String { - "usage: generate-viceroy-config --template --app-config --output [--origin-url ]".to_string() + "usage: generate-viceroy-config --template --app-config --output [--origin-url ] [--bidder-origin-url ]".to_string() } fn build_app_config_envelope( @@ -165,6 +268,7 @@ mod tests { use super::*; use error_stack::Report; use std::collections::HashMap; + use tempfile::tempdir; use trusted_server_core::config_payload::settings_from_config_blob; use trusted_server_core::platform::{PlatformError, PlatformSecretStore, StoreId, StoreName}; @@ -245,6 +349,124 @@ mod tests { ); } + #[test] + fn parse_args_accepts_explicit_controlled_bidder_origin() { + assert!( + parse_args([ + "--template".to_string(), + "template.toml".to_string(), + "--app-config".to_string(), + "trusted-server.toml".to_string(), + "--output".to_string(), + "generated.toml".to_string(), + "--bidder-origin-url".to_string(), + "http://127.0.0.1:8888".to_string(), + ]) + .is_ok(), + "should accept an explicit local bidder fixture without changing defaults" + ); + } + + fn run_bidder_fixture(origin: &str) -> Result { + let directory = tempdir().expect("should create isolated generator fixture"); + let template = directory.path().join("template.toml"); + let app_config = directory.path().join("app.toml"); + let output = directory.path().join("output.toml"); + fs::write(&template, TEMPLATE).expect("should write runtime template"); + fs::write( + &app_config, + include_str!("../../fixtures/configs/trusted-server.trace.toml"), + ) + .expect("should write controlled trace app config"); + run(&Args { + template, + app_config, + output: output.clone(), + origin_url: None, + bidder_origin_url: Some(origin.to_string()), + })?; + Ok(toml::from_str(&fs::read_to_string(output)?)?) + } + + #[test] + fn controlled_bidder_uses_backend_aliases_and_preserves_https_app_config() { + let generated = run_bidder_fixture("http://127.0.0.1:8888") + .expect("should generate explicit local bidder aliases"); + let aliases = generated["local_server"]["backends"] + .as_table() + .expect("should retain backend table"); + assert!( + !aliases.is_empty(), + "should route the real Rust bidder HTTP client to the controlled local origin" + ); + for alias in aliases.values() { + assert_eq!( + alias["url"].as_str(), + Some("http://127.0.0.1:8888/"), + "should keep every runtime-only alias local" + ); + } + let envelope: serde_json::Value = serde_json::from_str( + generated["local_server"]["config_stores"][DEFAULT_CONFIG_STORE_ID]["contents"] + [CONFIG_BLOB_KEY] + .as_str() + .expect("should retain validated app envelope"), + ) + .expect("should read unchanged app config envelope"); + assert_eq!( + envelope["data"]["auction"]["providers"]["example"]["endpoint"], + "https://bidder.example.com/api/trace-bidder", + "should preserve production HTTPS and certificate admission" + ); + } + + #[test] + fn controlled_bidder_rejects_non_loopback_or_ambiguous_origins() { + for origin in [ + "http://bidder.example.com", + "http://127.0.0.1:8888/path", + "http://127.0.0.1:8888/?query=value", + "http://user:password@127.0.0.1:8888", + ] { + assert!( + run_bidder_fixture(origin).is_err(), + "should reject a non-local or non-origin bidder fixture URL" + ); + } + } + + #[test] + fn controlled_bidder_omits_unreachable_large_provider_timer_aliases() { + let app_config = include_str!("../../fixtures/configs/trusted-server.trace.toml").replace( + "[auction.providers.example]", + "[auction.providers.example]\ntimeout_ms = 100000", + ); + let generated = + inject_controlled_bidder_backends(TEMPLATE, &app_config, "http://127.0.0.1:8888") + .expect("should bound the configured provider by reachable logical budgets"); + let generated: toml::Value = + toml::from_str(&generated).expect("should generate valid bounded runtime aliases"); + assert_eq!( + generated["local_server"]["backends"] + .as_table() + .expect("should retain aliases") + .len(), + 8, + "should generate only eight reachable timers for the 1000 ms fixture budget" + ); + } + + #[test] + fn controlled_bidder_rejects_excessive_fixture_logical_budget() { + let app_config = include_str!("../../fixtures/configs/trusted-server.trace.toml") + .replace("auction_timeout_ms = 1000", "auction_timeout_ms = 100000"); + assert!( + inject_controlled_bidder_backends(TEMPLATE, &app_config, "http://127.0.0.1:8888") + .is_err(), + "should reject fixture budgets beyond the bounded 60 second timer enumeration" + ); + } + #[test] fn parse_args_accepts_required_flags_and_origin_override() { let args = parse_args([ @@ -265,7 +487,8 @@ mod tests { template: PathBuf::from("template.toml"), app_config: PathBuf::from("trusted-server.toml"), output: PathBuf::from("generated.toml"), - origin_url: Some("http://127.0.0.1:9999".to_string()) + origin_url: Some("http://127.0.0.1:9999".to_string()), + bidder_origin_url: None, }, "should parse expected args" ); diff --git a/crates/trusted-server-integration-tests/tests/common/config.rs b/crates/trusted-server-integration-tests/tests/common/config.rs index 3bcbfe53e..90f3b81a2 100644 --- a/crates/trusted-server-integration-tests/tests/common/config.rs +++ b/crates/trusted-server-integration-tests/tests/common/config.rs @@ -9,6 +9,18 @@ const GENERATED_AT: &str = "2026-06-23T00:00:00Z"; const APP_CONFIG: &str = include_str!("../../fixtures/configs/trusted-server.integration.toml"); pub fn integration_app_config_envelope(origin_port: u16) -> TestResult { + integration_app_config_envelope_with_trace(origin_port, false) +} + +/// Generate an isolated runtime fixture with an explicit trace feature setting. +/// +/// # Errors +/// +/// Returns a bounded fixture-generation error when configuration validation fails. +pub fn integration_app_config_envelope_with_trace( + origin_port: u16, + trace: bool, +) -> TestResult { let origin_url = format!("http://127.0.0.1:{origin_port}"); let app_config: TrustedServerAppConfig = toml::from_str(APP_CONFIG).map_err(|error| { Report::new(TestError::ConfigGeneration).attach(format!( @@ -17,6 +29,18 @@ pub fn integration_app_config_envelope(origin_port: u16) -> TestResult { })?; let mut settings = app_config.into_settings(); settings.publisher.origin_url = origin_url; + if trace { + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":true}), + ) + .map_err(|report| { + Report::new(TestError::ConfigGeneration) + .attach(format!("invalid trace fixture config: {report:?}")) + })?; + } let app_config = TrustedServerAppConfig::new(settings).map_err(|report| { Report::new(TestError::ConfigGeneration) .attach(format!("invalid generated integration config: {report:?}")) @@ -45,6 +69,51 @@ pub fn cloudflare_config_json(origin_port: u16) -> TestResult { #[cfg(test)] mod tests { + use super::*; + + #[test] + fn trace_browser_fixture_keeps_auctions_and_shared_assembly_disabled() { + let envelope: serde_json::Value = serde_json::from_str( + &integration_app_config_envelope_with_trace(8888, true) + .expect("should generate normal-browser trace fixture"), + ) + .expect("should parse trace browser envelope"); + assert_eq!( + envelope["data"]["auction"]["enabled"], false, + "should keep the normal browser workflow independent of bidder transport" + ); + assert!( + envelope["data"] + .get("creative_opportunities") + .is_none_or(serde_json::Value::is_null), + "should keep shared assembly and configured server slots out of this fixture" + ); + assert_eq!( + envelope["data"]["integrations"]["gpt_diagnostics"]["trace_page_enabled"], true, + "should activate trace only through the explicit isolated configuration" + ); + } + + #[test] + fn trace_runtime_boundary_config_enables_only_explicit_fixture() { + let baseline: serde_json::Value = serde_json::from_str( + &integration_app_config_envelope(8888).expect("should generate baseline envelope"), + ) + .expect("should parse baseline envelope"); + assert_ne!( + baseline["data"]["integrations"]["gpt_diagnostics"]["trace_page_enabled"], true, + "should keep the trace feature disabled in the ordinary baseline" + ); + let enabled: serde_json::Value = serde_json::from_str( + &integration_app_config_envelope_with_trace(8888, true) + .expect("should generate trace fixture envelope"), + ) + .expect("should parse trace envelope"); + assert_eq!( + enabled["data"]["integrations"]["gpt_diagnostics"]["trace_page_enabled"], true, + "should explicitly enable isolated trace fixture" + ); + } const FASTLY_CONFIG: &str = include_str!("../../../../fastly.toml"); const VICEROY_TEMPLATE: &str = include_str!("../../fixtures/configs/viceroy-template.toml"); const VICEROY_SECRET_STORE_MAPPING_KEY: &str = diff --git a/crates/trusted-server-integration-tests/tests/common/mod.rs b/crates/trusted-server-integration-tests/tests/common/mod.rs index f5a4b578e..76a8c5f73 100644 --- a/crates/trusted-server-integration-tests/tests/common/mod.rs +++ b/crates/trusted-server-integration-tests/tests/common/mod.rs @@ -2,3 +2,4 @@ pub mod assertions; pub mod config; pub mod ec; pub mod runtime; +pub(crate) mod trace_boundary; diff --git a/crates/trusted-server-integration-tests/tests/common/trace_boundary.rs b/crates/trusted-server-integration-tests/tests/common/trace_boundary.rs new file mode 100644 index 000000000..8891015ed --- /dev/null +++ b/crates/trusted-server-integration-tests/tests/common/trace_boundary.rs @@ -0,0 +1,1020 @@ +//! Isolated actual-runtime probes at the SDK-visible trace boundary. + +use std::fs; +use std::io::{Read as _, Write as _}; +use std::net::TcpStream; +#[cfg(unix)] +use std::os::unix::process::CommandExt as _; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64::{Engine as _, engine::general_purpose::STANDARD}; +use edgezero_core::blob_envelope::BlobEnvelope; +use error_stack::Report; +use http::{HeaderMap, HeaderValue}; +use testcontainers::runners::SyncRunner as _; +use trusted_server_core::config_payload::{CONFIG_BLOB_KEY, DEFAULT_CONFIG_STORE_ID}; + +use crate::common::config::integration_app_config_envelope_with_trace; +use crate::common::runtime::TestError; +use crate::common::runtime::{ + RuntimeEnvironment as _, RuntimeProcess, RuntimeProcessHandle, origin_port, wasm_binary_path, +}; +use crate::environments::{ + axum::AxumDevServer, cloudflare::CloudflareWorkers, fastly::FastlyViceroy, +}; +use crate::frameworks::{FrontendFramework as _, nextjs::NextJs}; + +/// Actual runtime selected by an ignored boundary test. +#[derive(Clone, Copy)] +pub(crate) enum TraceRuntime { + /// Local Cloudflare Workers via Wrangler/workerd. + Cloudflare, + /// Local Fastly Compute via Viceroy. + Fastly, + /// Local Spin HTTP component. + Spin, +} + +/// Real runtime selected by the normal browser workflow. +#[derive(Clone, Copy)] +pub(crate) enum TraceBrowserRuntime { + /// Native Axum dev-server binary. + Axum, + /// Local Cloudflare Workers via Wrangler/workerd. + Cloudflare, + /// Local Fastly Compute via Viceroy. + Fastly, + /// Local Spin HTTP component. + Spin, +} + +impl TraceBrowserRuntime { + fn id(self) -> &'static str { + match self { + Self::Axum => "axum", + Self::Cloudflare => "cloudflare", + Self::Fastly => "fastly", + Self::Spin => "spin", + } + } +} + +struct IsolatedFixture(PathBuf); + +impl IsolatedFixture { + fn new() -> Self { + let stamp = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should read fixture clock") + .as_nanos(); + let path = + std::env::temp_dir().join(format!("ts-trace-boundary-{}-{stamp}", std::process::id())); + fs::create_dir_all(&path).expect("should create isolated runtime fixture"); + Self(path) + } +} + +impl Drop for IsolatedFixture { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.0); + } +} + +struct ChildHandle { + child: Child, + #[cfg(unix)] + process_group: bool, +} +impl RuntimeProcessHandle for ChildHandle {} +impl Drop for ChildHandle { + fn drop(&mut self) { + #[cfg(unix)] + if self.process_group { + // Give Playwright's signal handlers time to close its browsers + // before escalating the owned Node and worker process group. + unsafe { + libc::killpg(self.child.id() as libc::pid_t, libc::SIGTERM); + } + let deadline = Instant::now() + Duration::from_secs(3); + while Instant::now() < deadline { + let group_exists = unsafe { libc::killpg(self.child.id() as libc::pid_t, 0) == 0 }; + if !group_exists { + break; + } + std::thread::sleep(Duration::from_millis(100)); + } + unsafe { + libc::killpg(self.child.id() as libc::pid_t, libc::SIGKILL); + } + } + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + +/// Run raw and normal requests against a separately configured runtime instance. +/// +/// These probes distinguish successful SDK conversion from runtime rejection. +/// +/// # Examples +/// +/// ```ignore +/// exercise(TraceRuntime::Fastly); +/// ``` +pub(crate) fn exercise(runtime: TraceRuntime) { + exercise_case(runtime, true, None, false); + for enabled in [false, true] { + for pattern in [None, Some("^/"), Some("^/_ts")] { + exercise_case(runtime, enabled, pattern, true); + } + } +} + +fn exercise_case(runtime: TraceRuntime, enabled: bool, auth_pattern: Option<&str>, matrix: bool) { + let envelope = integration_app_config_envelope_with_trace(8888, enabled) + .expect("should generate isolated trace config"); + let envelope = if let Some(pattern) = auth_pattern { + let mut envelope: BlobEnvelope = + serde_json::from_str(&envelope).expect("should parse fixture envelope"); + envelope.data["handlers"].as_array_mut().expect("should retain handler array").insert(0,serde_json::json!({"path":pattern,"username":"example-user","password":"integration_admin_password"})); + let envelope = BlobEnvelope::new(envelope.data, envelope.generated_at); + envelope + .verify() + .expect("should rebuild valid fixture envelope integrity"); + serde_json::to_string(&envelope).expect("should serialize trace auth fixture") + } else { + envelope + }; + let check = |base_url: &str| { + if matrix { + check_policy_matrix(base_url, &runtime, enabled, auth_pattern); + } else { + check_boundary(base_url, &runtime); + } + }; + with_trace_runtime(runtime, &envelope, None, check); +} + +fn with_trace_runtime( + runtime: TraceRuntime, + envelope: &str, + publisher_origin: Option<&str>, + check: impl FnOnce(&str), +) { + let fixture = IsolatedFixture::new(); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + match runtime { + TraceRuntime::Cloudflare => { + let original = root.join("crates/trusted-server-adapter-cloudflare"); + copy_directory(&original.join("build"), &fixture.0.join("build")); + let template = fs::read_to_string(original.join("wrangler.ci.toml")) + .expect("should read baseline Wrangler template"); + let binding = serde_json::to_string(&serde_json::json!({CONFIG_BLOB_KEY:envelope})) + .expect("should serialize trace binding"); + let config = template.replace( + "TRUSTED_SERVER_CONFIG = \"{}\"", + &format!("TRUSTED_SERVER_CONFIG = '''{binding}'''"), + ); + fs::write(fixture.0.join("wrangler.toml"), config) + .expect("should write isolated Wrangler config"); + temp_env::with_vars( + [ + ("CLOUDFLARE_WRANGLER_DIR", Some(fixture.0.as_os_str())), + ("CI", None), + ], + || { + let process = CloudflareWorkers + .spawn_with_readiness(wait_for_trace_fixture_ready) + .expect("should spawn isolated Cloudflare trace runtime"); + check(&process.base_url); + }, + ); + } + TraceRuntime::Fastly => { + let template = include_str!("../../fixtures/configs/viceroy-template.toml"); + let config=template.replace(" # GENERATED_TRUSTED_SERVER_CONFIG_STORES",&format!(" [local_server.config_stores.{DEFAULT_CONFIG_STORE_ID}]\n format = \"inline-toml\"\n [local_server.config_stores.{DEFAULT_CONFIG_STORE_ID}.contents]\n {CONFIG_BLOB_KEY} = '''{envelope}'''")); + let path = fixture.0.join("viceroy.toml"); + fs::write(&path, config).expect("should write isolated Viceroy config"); + temp_env::with_var("VICEROY_CONFIG_PATH", Some(path.as_os_str()), || { + let process = FastlyViceroy + .spawn(&wasm_binary_path()) + .expect("should spawn isolated Fastly trace runtime"); + check(&process.base_url); + }); + } + TraceRuntime::Spin => { + let artifact = + root.join("target/wasm32-wasip1/release/trusted_server_adapter_spin.wasm"); + assert!( + artifact.is_file(), + "should build production Spin component before boundary probe" + ); + let mut config = format!( + "spin_manifest_version = 2\n[application]\nname = \"trace-boundary-example\"\nversion = \"0.1.0\"\n[[trigger.http]]\nroute = \"/...\"\ncomponent = \"trace\"\n[component.trace]\nsource = {:?}\nkey_value_stores = [\"default\"]\n", + artifact.to_str().expect("should represent component path") + ); + if let Some(origin) = publisher_origin { + config.push_str(&format!("allowed_outbound_hosts = [{origin:?}]\n")); + } + let secrets = [ + ( + "integration_admin_password", + "integration-admin-password-32-bytes-ok", + ), + ( + "integration_proxy_secret", + "integration-test-proxy-secret-32-bytes-ok", + ), + ( + "integration_ec_passphrase", + "integration-test-ec-secret-padded-32", + ), + ( + "integration_partner_token_alpha", + "integration-test-token-alpha-32-bytes-ok", + ), + ( + "integration_partner_token_bravo", + "integration-test-token-bravo-32-bytes-ok", + ), + ]; + config.push_str("[variables]\n"); + for (name, value) in secrets { + config.push_str(&format!( + "{} = {{ default = {value:?} }}\n", + spin_secret_variable(name) + )); + } + config.push_str("[component.trace.variables]\n"); + for (name, _) in secrets { + let variable = spin_secret_variable(name); + config.push_str(&format!("{variable} = \"{{{{ {variable} }}}}\"\n")); + } + let manifest = fixture.0.join("spin.toml"); + fs::write(&manifest, config).expect("should write isolated Spin manifest"); + let port = crate::environments::find_available_port() + .expect("should reserve Spin runtime port"); + let child = Command::new("spin") + .arg("up") + .arg("--from") + .arg(&manifest) + .arg("--listen") + .arg(format!("127.0.0.1:{port}")) + .arg("--state-dir") + .arg(fixture.0.join("state")) + .arg("--key-value") + .arg(format!("{CONFIG_BLOB_KEY}={envelope}")) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .expect("should spawn isolated Spin runtime"); + let process = RuntimeProcess { + inner: Box::new(ChildHandle { + child, + #[cfg(unix)] + process_group: false, + }), + base_url: format!("http://127.0.0.1:{port}"), + }; + wait_for_trace_fixture_ready(&process.base_url) + .expect("should load Spin trace configuration"); + check(&process.base_url); + } + } +} + +/// Exercise a real host-only cookie and publisher-to-viewer browser workflow. +/// +/// Auction and shared assembly stay disabled. Only GPT callbacks are mocked; +/// activation, context, built bundles, handoff, exports and cleanup are real. +/// Run ignored browser tests sequentially because they share the origin port. +/// +/// # Panics +/// +/// Panics when a required artifact, runtime, container or browser check fails. +pub(crate) fn exercise_browser(runtime: TraceBrowserRuntime) { + let port = origin_port(); + let _container = NextJs + .build_container(port) + .expect("should configure the existing Next.js origin fixture") + .start() + .expect("should start the existing Next.js origin fixture"); + crate::environments::wait_for_ready(&format!("http://127.0.0.1:{port}"), "/", false) + .expect("should make the ordinary publisher origin available"); + let envelope = integration_app_config_envelope_with_trace(port, true) + .expect("should generate isolated normal-browser trace configuration"); + let publisher_origin = format!("http://127.0.0.1:{port}"); + let check = |base_url: &str| run_browser_workflow(runtime, base_url); + match runtime { + TraceBrowserRuntime::Axum => { + let process = AxumDevServer + .spawn_with_app_config(&envelope) + .expect("should spawn the real native trace runtime"); + wait_for_trace_fixture_ready(&process.base_url) + .expect("should load native trace configuration"); + check(&process.base_url); + } + TraceBrowserRuntime::Cloudflare => { + with_trace_runtime(TraceRuntime::Cloudflare, &envelope, None, check); + } + TraceBrowserRuntime::Fastly => { + with_trace_runtime(TraceRuntime::Fastly, &envelope, None, check); + } + TraceBrowserRuntime::Spin => { + with_trace_runtime( + TraceRuntime::Spin, + &envelope, + Some(&publisher_origin), + check, + ); + } + } +} + +fn run_browser_workflow(runtime: TraceBrowserRuntime, base_url: &str) { + let browser_root = Path::new(env!("CARGO_MANIFEST_DIR")).join("browser"); + let mut origin = url::Url::parse(base_url).expect("should parse the owned runtime origin"); + origin + .set_host(Some("localhost")) + .expect("should use browser-suitable localhost without changing the runtime port"); + let mut command = Command::new("node"); + command + .arg(browser_root.join("node_modules/@playwright/test/cli.js")) + .args(["test", "--config", "playwright.trace-runtime.config.ts"]) + .env( + "TRACE_BROWSER_ORIGIN", + origin.origin().ascii_serialization(), + ) + .env("TRACE_BROWSER_RUNTIME", runtime.id()) + .current_dir(&browser_root) + .stdin(Stdio::null()) + .stdout(Stdio::inherit()) + .stderr(Stdio::inherit()); + #[cfg(unix)] + command.process_group(0); + let mut browser = ChildHandle { + child: command.spawn().expect( + "should launch installed Playwright without installing or building dependencies", + ), + #[cfg(unix)] + process_group: true, + }; + let started = Instant::now(); + loop { + if let Some(status) = browser + .child + .try_wait() + .expect("should poll the owned browser workflow") + { + assert!( + status.success(), + "should pass the real {} browser workflow: {status}", + runtime.id() + ); + break; + } + assert!( + started.elapsed() < Duration::from_secs(120), + "should finish the {} browser workflow within its bounded deadline", + runtime.id() + ); + std::thread::sleep(Duration::from_millis(100)); + } +} + +fn wait_for_trace_fixture_ready(base_url: &str) -> crate::common::runtime::TestResult<()> { + let client = reqwest::blocking::Client::new(); + for _ in 0..60 { + if client + .get(format!("{base_url}/_ts/trace/state")) + .timeout(Duration::from_millis(500)) + .send() + .is_ok_and(|response| matches!(response.status().as_u16(), 200 | 401 | 404)) + { + return Ok(()); + } + std::thread::sleep(Duration::from_millis(250)); + } + Err(Report::new(TestError::RuntimeNotReady)) +} + +fn check_policy_matrix( + base_url: &str, + runtime: &TraceRuntime, + enabled: bool, + pattern: Option<&str>, +) { + let url = reqwest::Url::parse(base_url).expect("should parse matrix runtime origin"); + let authority = format!( + "{}:{}", + url.host_str().expect("should expose runtime host"), + url.port().expect("should expose runtime port") + ); + let authenticated = format!( + "Authorization: Basic {}\r\n", + STANDARD.encode("example-user:integration-admin-password-32-bytes-ok") + ); + for (method, path, route_status) in [ + (b"GET".as_slice(), "/_ts/trace/state", 200), + (b"HEAD".as_slice(), "/_ts/trace", 200), + (b"GET".as_slice(), "/_ts/trace/assets/v1.js", 200), + (b"GET".as_slice(), "/_ts/trace/assets/v1.css", 200), + (b"POST".as_slice(), "/_ts/trace/enable", 403), + (b"POST".as_slice(), "/_ts/trace/end", 403), + (b"PATCH".as_slice(), "/_ts/trace/state", 405), + (b"GET".as_slice(), "/_ts/trace/extra", 404), + ( + b"GET".as_slice(), + "/_ts/trace/../trace/state", + if matches!(runtime, TraceRuntime::Spin) { + 400 + } else { + 200 + }, + ), + (b"GET".as_slice(), "/%5Fts/trace", 400), + (b"GET".as_slice(), "/_ts//trace/state", 404), + ] { + for (credentials, headers) in [ + (false, b"".as_slice()), + ( + false, + b"Authorization: Basic ZXhhbXBsZS11c2VyOndyb25n\r\n".as_slice(), + ), + (true, authenticated.as_bytes()), + ] { + let protected = + pattern.is_some_and(|pattern| pattern == "^/" || path.starts_with("/_ts")); + let expected = if protected && !credentials { + 401 + } else if !enabled { + 404 + } else { + route_status + }; + let response = raw_request(&authority, method, path, headers); + assert_eq!( + response.status, expected, + "should apply actual-runtime auth before flag, reserved-path and method policy for {path}" + ); + if !path.contains("/assets/") || expected != 200 { + assert_private(&response); + } else { + assert_eq!( + response.headers["cache-control"], + if protected { + "no-store, private" + } else { + "public, max-age=31536000, immutable" + }, + "should preserve auth-dependent asset privacy" + ); + assert!( + response.headers.contains_key("etag"), + "should preserve protected strong ETag through actual runtime" + ); + } + if expected == 401 { + assert!( + response.headers.contains_key("www-authenticate"), + "should retain local auth challenge" + ); + } + if expected == 405 { + assert_eq!( + response.headers["allow"], "GET, HEAD", + "should retain local method Allow" + ); + } + if method == b"HEAD" { + assert!( + response.body.is_empty(), + "should strip every trace HEAD success/challenge/error body" + ); + } + } + } + if matches!(runtime, TraceRuntime::Fastly) { + let response = raw_request(&authority, b"GET", "/_ts/trace/../../_ts/debug/ja4", b""); + assert_eq!( + response.status, 404, + "should preserve normalized-out disabled native JA4 behavior before ordinary auth" + ); + assert!( + !response.headers.contains_key("content-security-policy"), + "should not relabel normalized-out native JA4 as a trace response" + ); + assert!( + !response.headers.contains_key("set-cookie"), + "should not create diagnostics cookies for ordinary native shortcut" + ); + } +} + +fn spin_secret_variable(key: &str) -> String { + let mut output = String::from("v_trusted_x5fserver_x5fsecrets_v_"); + for byte in key.bytes() { + if byte.is_ascii_lowercase() || byte.is_ascii_digit() { + output.push(char::from(byte)); + } else { + output.push_str(&format!("_x{byte:02x}")); + } + } + output +} + +fn copy_directory(source: &Path, destination: &Path) { + fs::create_dir_all(destination).expect("should create copied runtime build directory"); + for entry in fs::read_dir(source).expect("should locate prebuilt Cloudflare bundle") { + let entry = entry.expect("should read build entry"); + if entry + .file_type() + .expect("should inspect build entry") + .is_dir() + { + copy_directory(&entry.path(), &destination.join(entry.file_name())); + } else { + fs::copy(entry.path(), destination.join(entry.file_name())) + .expect("should copy isolated runtime build asset"); + } + } +} + +fn check_boundary(base_url: &str, runtime: &TraceRuntime) { + let url = reqwest::Url::parse(base_url).expect("should parse actual runtime origin"); + let authority = format!( + "{}:{}", + url.host_str().expect("should expose local runtime host"), + url.port().expect("should expose ephemeral runtime port") + ); + let client = reqwest::blocking::Client::new(); + let state = client + .get(format!("{base_url}/_ts/trace/state")) + .header("cookie", "__Host-ts-console=1") + .send() + .expect("should request ordinary visible session"); + assert_eq!( + state.status(), + 200, + "should activate marker-free Unknown-fidelity runtime cookies" + ); + assert_eq!( + state.text().expect("should read state"), + r#"{"observed_active":true}"#, + "should observe valid incoming session" + ); + for action in ["enable", "end"] { + let response = client + .post(format!("{base_url}/_ts/trace/{action}")) + .header("origin", base_url) + .header("sec-fetch-site", "same-origin") + .header("x-ts-trace-action", action) + .header("cookie", "__Host-ts-console=1, unrelated=value") + .send() + .expect("should perform actual-origin empty action"); + assert_eq!( + response.status(), + 200, + "should accept actual runtime origin" + ); + let expected = if action == "enable" { + "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800" + } else { + "__Host-ts-console=; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=0" + }; + let cookies: Vec<_> = response.headers().get_all("set-cookie").iter().collect(); + assert_eq!( + cookies.len(), + 1, + "should retain exactly one explicit action cookie through native finalization" + ); + assert_eq!( + cookies[0], expected, + "should preserve shared cookie attributes exactly" + ); + assert_eq!( + response.text().expect("should read requested mutation"), + r#"{"mutation_requested":true}"#, + "should report mutation without claiming browser acceptance" + ); + let state = client + .get(format!("{base_url}/_ts/trace/state")) + .header("cookie", "__Host-ts-console=1, unrelated=value") + .send() + .expect("should observe a separate ambiguous-cookie state"); + assert_eq!( + state.text().expect("should read independent state"), + r#"{"observed_active":false}"#, + "should not infer browser cookie acceptance from a requested mutation" + ); + } + let head = raw_request(&authority, b"HEAD", "/_ts/trace", b""); + assert_eq!(head.status, 200, "should serve bodyless setup HEAD"); + assert!(head.body.is_empty(), "should strip setup HEAD body"); + assert_private(&head); + let literal = raw_request( + &authority, + b"GET", + "/_ts/trace", + b"Cookie: __Host-ts-console=1; unrelated=\xef\xbf\xbd\r\n", + ); + assert_eq!( + literal.status, 200, + "should project runtime-visible literal U+FFFD safely" + ); + assert!( + String::from_utf8_lossy(&literal.body).contains("runtime_header_ambiguous"), + "should conservatively reject U+FFFD with unknown original octets" + ); + assert_private(&literal); + let repeated = raw_request( + &authority, + b"GET", + "/_ts/trace", + b"Cookie: __Host-ts-console=1\r\nCookie: __Host-ts-console=1\r\n", + ); + assert_eq!( + repeated.status, 200, + "should retain local cookie-health observation for repeated fields" + ); + let expected = if matches!(runtime, TraceRuntime::Cloudflare) { + "runtime_header_ambiguous" + } else { + "duplicate" + }; + assert!( + String::from_utf8_lossy(&repeated.body).contains(expected), + "should pin the runtime's visible repeated-cookie boundary" + ); + assert_private(&repeated); + let invalid = raw_request( + &authority, + b"GET", + "/_ts/trace", + b"Cookie: __Host-ts-console=1; unrelated=\xff\r\n", + ); + match runtime { + TraceRuntime::Cloudflare | TraceRuntime::Fastly => { + assert_eq!( + invalid.status, 200, + "should inspect bytes accepted at the application boundary" + ); + let reason = if matches!(runtime, TraceRuntime::Cloudflare) { + "runtime_header_ambiguous" + } else { + "header_not_utf8" + }; + assert!( + String::from_utf8_lossy(&invalid.body).contains(reason), + "should distinguish replacement ambiguity from actual visible non-UTF8" + ); + assert_private(&invalid); + } + TraceRuntime::Spin => { + assert_eq!( + invalid.status, 500, + "should pin the pre-component invalid-byte conversion failure separately" + ); + assert!( + !invalid.headers.contains_key("content-security-policy"), + "should not claim local trace hardening before successful SDK conversion" + ); + assert!( + !invalid.headers.contains_key("set-cookie"), + "should not mutate cookies on pre-component conversion failure" + ); + assert!( + !String::from_utf8_lossy(&invalid.body).contains("trace-request-context"), + "should not capture trace context on conversion failure" + ); + } + } + let extension = raw_request(&authority, b"EXAMPLE-METHOD", "/_ts/trace/state", b""); + if matches!(runtime, TraceRuntime::Cloudflare) { + assert_eq!( + extension.status, 501, + "should pin workerd's pre-application method rejection" + ); + assert!( + !extension.headers.contains_key("content-security-policy"), + "should keep wire rejection separate from application hardened 405" + ); + } else { + assert_eq!( + extension.status, 405, + "should reject converted extension method locally" + ); + assert_eq!( + extension.headers["allow"], "GET, HEAD", + "should retain application Allow policy" + ); + assert_private(&extension); + } + let asset = client + .get(format!("{base_url}/_ts/trace/assets/v1.js")) + .send() + .expect("should request fixed asset"); + assert_eq!(asset.status(), 200, "should serve verified fixed bytes"); + assert_eq!( + asset.headers()["cache-control"], + "public, max-age=31536000, immutable", + "should retain immutable asset cache policy" + ); + assert!( + asset.headers().contains_key("etag"), + "should retain strong asset validator through native conversion" + ); + assert!( + !asset.headers().contains_key("set-cookie"), + "should never add identity cookies to fixed assets" + ); + let normalized_in = raw_request(&authority, b"GET", "/_ts/trace/../trace/state", b""); + let expected = if matches!(runtime, TraceRuntime::Spin) { + 400 + } else { + 200 + }; + assert_eq!( + normalized_in.status, expected, + "should classify the actual runtime-visible path, not recover a discarded wire target" + ); + assert_private(&normalized_in); + let normalized_out = raw_request(&authority, b"GET", "/_ts/trace/../../health", b""); + if matches!(runtime, TraceRuntime::Spin) { + assert_eq!( + normalized_out.status, 400, + "should reject dot ambiguity still visible to Spin" + ); + assert_private(&normalized_out); + } else if matches!(runtime, TraceRuntime::Fastly) { + assert_eq!( + normalized_out.status, 200, + "should preserve ordinary health when runtime normalization removes the namespace" + ); + assert_eq!( + normalized_out.body, b"ok", + "should keep normalized-out requests on the ordinary health surface" + ); + } else { + let ordinary = raw_request(&authority, b"GET", "/health", b""); + assert_eq!( + normalized_out.status, ordinary.status, + "should preserve Cloudflare's ordinary publisher fallback for normalized-out health" + ); + assert!( + !String::from_utf8_lossy(&normalized_out.body).contains("trace-request-context"), + "should not capture context after the runtime removes the namespace" + ); + } +} + +struct RawResponse { + status: u16, + headers: HeaderMap, + body: Vec, +} + +fn assert_private(response: &RawResponse) { + assert_eq!( + response.headers["cache-control"], "no-store, private", + "should preserve dynamic/error cache policy through actual runtime" + ); + assert!( + response.headers.contains_key("content-security-policy"), + "should retain fixed trace CSP" + ); + assert!( + !response.headers.contains_key("set-cookie"), + "should not mutate read cookies" + ); +} + +fn raw_request(authority: &str, method: &[u8], path: &str, headers: &[u8]) -> RawResponse { + let mut connection = TcpStream::connect(authority).expect("should connect raw runtime client"); + connection + .set_read_timeout(Some(Duration::from_secs(10))) + .expect("should bound raw response wait"); + connection + .write_all(method) + .expect("should write exact method"); + connection + .write_all( + format!(" {path} HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n").as_bytes(), + ) + .expect("should write runtime authority"); + connection + .write_all(headers) + .expect("should preserve original header bytes and repeated fields"); + connection + .write_all(b"\r\n") + .expect("should finish raw request"); + let mut bytes = Vec::new(); + let mut chunk = [0; 8192]; + while bytes.len() < 2 * 1024 * 1024 { + let available = chunk.len().min(2 * 1024 * 1024 - bytes.len()); + let read = connection + .read(&mut chunk[..available]) + .expect("should read bounded raw response"); + if read == 0 { + break; + } + bytes.extend_from_slice(&chunk[..read]); + if response_complete(&bytes, method == b"HEAD") { + break; + } + } + let end = bytes + .windows(4) + .position(|part| part == b"\r\n\r\n") + .expect("should receive HTTP response headers"); + let text = std::str::from_utf8(&bytes[..end]).expect("should receive UTF8 response headers"); + let mut lines = text.split("\r\n"); + let status = lines + .next() + .expect("should receive status line") + .split_ascii_whitespace() + .nth(1) + .expect("should receive numeric status") + .parse() + .expect("should parse status"); + let mut response_headers = HeaderMap::new(); + for line in lines { + let (name, value) = line.split_once(':').expect("should parse header field"); + response_headers.append( + http::header::HeaderName::from_bytes(name.as_bytes()) + .expect("should parse header name"), + HeaderValue::from_str(value.trim()).expect("should parse response header"), + ); + } + let body = if method != b"HEAD" + && response_headers + .get("transfer-encoding") + .is_some_and(|value| value == "chunked") + { + decode_chunks(&bytes[end + 4..]) + } else { + bytes[end + 4..].to_vec() + }; + RawResponse { + status, + headers: response_headers, + body, + } +} + +fn decode_chunks(mut bytes: &[u8]) -> Vec { + let mut body = Vec::new(); + loop { + let end = bytes + .windows(2) + .position(|part| part == b"\r\n") + .expect("should read chunk length"); + let size = usize::from_str_radix( + std::str::from_utf8(&bytes[..end]) + .expect("should read ASCII chunk length") + .split(';') + .next() + .expect("should read chunk size"), + 16, + ) + .expect("should parse chunk length"); + if size == 0 { + return body; + } + bytes = &bytes[end + 2..]; + body.extend_from_slice( + bytes + .get(..size) + .expect("should receive complete bounded chunk"), + ); + bytes = bytes + .get(size + 2..) + .expect("should receive chunk terminator"); + } +} + +fn response_complete(bytes: &[u8], head: bool) -> bool { + let Some(end) = bytes.windows(4).position(|part| part == b"\r\n\r\n") else { + return false; + }; + if head { + return true; + } + let Ok(headers) = std::str::from_utf8(&bytes[..end]) else { + return false; + }; + let body = &bytes[end + 4..]; + let mut length = None; + let mut chunked = false; + for line in headers.split("\r\n").skip(1) { + let Some((name, value)) = line.split_once(':') else { + return false; + }; + if name.eq_ignore_ascii_case("content-length") { + let Ok(parsed) = value.trim().parse::() else { + return false; + }; + if length.replace(parsed).is_some() { + return false; + } + } + if name.eq_ignore_ascii_case("transfer-encoding") + && value.trim().eq_ignore_ascii_case("chunked") + { + chunked = true; + } + } + if chunked { + return chunks_complete(body); + } + length.is_some_and(|length| body.len() >= length) +} + +fn chunks_complete(mut bytes: &[u8]) -> bool { + loop { + let Some(end) = bytes.windows(2).position(|part| part == b"\r\n") else { + return false; + }; + let Ok(line) = std::str::from_utf8(&bytes[..end]) else { + return false; + }; + let Some(size) = line + .split(';') + .next() + .and_then(|size| usize::from_str_radix(size, 16).ok()) + else { + return false; + }; + bytes = &bytes[end + 2..]; + if size == 0 { + loop { + let Some(end) = bytes.windows(2).position(|part| part == b"\r\n") else { + return false; + }; + if end == 0 { + return true; + } + bytes = &bytes[end + 2..]; + } + } + let Some(next) = size.checked_add(2) else { + return false; + }; + if bytes.get(size..next) != Some(b"\r\n".as_slice()) { + return false; + } + bytes = &bytes[next..]; + } +} + +#[test] +fn trace_raw_response_completion_respects_framing() { + assert!( + response_complete(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n", false), + "should finish an empty framed response without waiting for connection closure" + ); + assert!( + !response_complete(b"HTTP/1.1 200 OK\r\nContent-Length: 3\r\n\r\nab", false), + "should wait for the full declared body" + ); + assert!( + response_complete(b"HTTP/1.1 200 OK\r\nContent-Length: 3\r\n\r\nabc", false), + "should finish at the full declared body" + ); + assert!( + response_complete(b"HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n", true), + "should finish HEAD at the header boundary" + ); + assert!( + !response_complete(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n", true), + "should require the complete header terminator" + ); + assert!( + !response_complete( + b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nabcde\r\n0\r\n", + false + ), + "should wait for the complete final chunk terminator" + ); + assert!( + response_complete( + b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nabcde\r\n0\r\n\r\n", + false + ), + "should finish a complete chunked response" + ); + assert!( + !response_complete( + b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n8\r\n\r\n0\r\n\r\n", + false + ), + "should not mistake a final-chunk marker inside chunk data for completion" + ); + assert!(response_complete(b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n1;example=yes\r\na\r\n0\r\nExample: value\r\n\r\n", false), "should finish chunk extensions and bounded trailers"); + assert!( + !response_complete(b"HTTP/1.1 200 OK\r\n\r\nabc", false), + "should retain EOF framing when no length or transfer framing is present" + ); +} diff --git a/crates/trusted-server-integration-tests/tests/environments/axum.rs b/crates/trusted-server-integration-tests/tests/environments/axum.rs index 6e1388cea..fa9e389c4 100644 --- a/crates/trusted-server-integration-tests/tests/environments/axum.rs +++ b/crates/trusted-server-integration-tests/tests/environments/axum.rs @@ -55,10 +55,27 @@ impl RuntimeEnvironment for AxumDevServer { } fn spawn(&self, _wasm_path: &Path) -> TestResult { + let app_config = integration_app_config_envelope(origin_port())?; + self.spawn_with_app_config(&app_config) + } + + fn health_check_path(&self) -> &str { + "/health" + } +} + +impl AxumDevServer { + /// Spawn the real native binary with an isolated app-config envelope. + /// + /// Normal [`RuntimeEnvironment::spawn`] retains its disabled baseline config. + /// + /// # Errors + /// + /// Returns a spawn or readiness error if the native process is unavailable. + pub(crate) fn spawn_with_app_config(&self, app_config: &str) -> TestResult { let binary = self.binary_path(); let port = super::find_available_port().unwrap_or(AXUM_DEFAULT_PORT); - let app_config = integration_app_config_envelope(origin_port())?; let store_name = default_config_store_name(); let config_key = default_config_key(); let config_variable = config_env_var(store_name.as_ref(), &config_key); @@ -100,12 +117,6 @@ impl RuntimeEnvironment for AxumDevServer { }) } - fn health_check_path(&self) -> &str { - "/health" - } -} - -impl AxumDevServer { /// Resolve the path to the compiled `trusted-server-axum` binary. /// /// Respects the `AXUM_BINARY_PATH` environment variable for CI overrides. diff --git a/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs b/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs index 10d16c0a7..e70388e5e 100644 --- a/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs +++ b/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs @@ -67,6 +67,28 @@ impl RuntimeEnvironment for CloudflareWorkers { } fn spawn(&self, _wasm_path: &Path) -> TestResult { + self.spawn_with_readiness(|base_url| { + super::wait_for_ready(base_url, self.health_check_path(), true) + }) + } + + fn health_check_path(&self) -> &str { + "/.well-known/trusted-server.json" + } +} + +impl CloudflareWorkers { + /// Start an isolated fixture with an explicit authenticated readiness probe. + /// + /// The standard [`RuntimeEnvironment::spawn`] retains its original readiness check. + /// + /// # Errors + /// + /// Returns the existing spawn failure or the supplied readiness-probe error. + pub(crate) fn spawn_with_readiness( + &self, + ready: impl FnOnce(&str) -> TestResult<()>, + ) -> TestResult { let wrangler_dir = self.wrangler_dir(); let config = if std::env::var("CI").is_ok() { write_generated_ci_config(&wrangler_dir)? @@ -141,17 +163,13 @@ impl RuntimeEnvironment for CloudflareWorkers { let handle = CloudflareHandle { child }; let base_url = format!("http://127.0.0.1:{port}"); - super::wait_for_ready(&base_url, self.health_check_path(), true)?; + ready(&base_url)?; Ok(RuntimeProcess { inner: Box::new(handle), base_url, }) } - - fn health_check_path(&self) -> &str { - "/.well-known/trusted-server.json" - } } impl CloudflareWorkers { diff --git a/crates/trusted-server-integration-tests/tests/integration.rs b/crates/trusted-server-integration-tests/tests/integration.rs index ebd026410..0209b964b 100644 --- a/crates/trusted-server-integration-tests/tests/integration.rs +++ b/crates/trusted-server-integration-tests/tests/integration.rs @@ -276,3 +276,53 @@ fn test_ec_lifecycle_fastly() { ); } } +#[test] +#[ignore = "requires Wrangler in PATH and the current Cloudflare build.sh output"] +fn trace_runtime_boundary_cloudflare() { + init_logger(); + common::trace_boundary::exercise(common::trace_boundary::TraceRuntime::Cloudflare); +} + +#[test] +#[ignore = "requires Viceroy and the current production Fastly WASM artifact"] +fn trace_runtime_boundary_fastly() { + init_logger(); + common::trace_boundary::exercise(common::trace_boundary::TraceRuntime::Fastly); +} + +#[test] +#[ignore = "requires Spin and the current production Spin WASM artifact"] +fn trace_runtime_boundary_spin() { + init_logger(); + common::trace_boundary::exercise(common::trace_boundary::TraceRuntime::Spin); +} + +#[test] +#[ignore = "requires Docker, Chromium and the current native Axum binary"] +fn trace_browser_workflow_axum() { + init_logger(); + common::trace_boundary::exercise_browser(common::trace_boundary::TraceBrowserRuntime::Axum); +} + +#[test] +#[ignore = "requires Docker, Chromium, Viceroy and the current release Fastly WASM"] +fn trace_browser_workflow_fastly() { + init_logger(); + common::trace_boundary::exercise_browser(common::trace_boundary::TraceBrowserRuntime::Fastly); +} + +#[test] +#[ignore = "requires Docker, Chromium, Wrangler and the current Cloudflare build output"] +fn trace_browser_workflow_cloudflare() { + init_logger(); + common::trace_boundary::exercise_browser( + common::trace_boundary::TraceBrowserRuntime::Cloudflare, + ); +} + +#[test] +#[ignore = "requires Docker, Chromium, Spin and the current release Spin component"] +fn trace_browser_workflow_spin() { + init_logger(); + common::trace_boundary::exercise_browser(common::trace_boundary::TraceBrowserRuntime::Spin); +} diff --git a/crates/trusted-server-integration-tests/tests/parity.rs b/crates/trusted-server-integration-tests/tests/parity.rs index 2428d7799..0785562df 100644 --- a/crates/trusted-server-integration-tests/tests/parity.rs +++ b/crates/trusted-server-integration-tests/tests/parity.rs @@ -1119,3 +1119,522 @@ async fn adapter_buffers_nextjs_auction_output() { } } } +#[allow(clippy::panic)] +#[cfg(test)] +mod trace_parity { + use std::net::IpAddr; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + + use async_trait::async_trait; + use bytes::Bytes; + use edgezero_core::key_value_store::{KvError, KvPage}; + use edgezero_core::request::{ + CapturedTarget, HeaderFidelity, InboundOrigin, OriginSource, RequestIngress, + TargetUnavailable, + }; + use error_stack::Report; + use http::{Method, StatusCode, header}; + use trusted_server_core::auction::telemetry::{AuctionEventBatch, AuctionTelemetrySink}; + use trusted_server_core::error::TrustedServerError; + use trusted_server_core::platform::{ + BackendNamingPolicy, ClientInfo, GeoInfo, PlatformBackend, PlatformBackendSpec, + PlatformConfigStore, PlatformError, PlatformGeo, PlatformHttpClient, PlatformHttpRequest, + PlatformKvStore, PlatformPendingRequest, PlatformResponse, PlatformSecretStore, + PlatformSelectResult, RuntimeServices, StoreId, StoreName, + }; + use trusted_server_core::trace::TraceTerminalResponse; + + use super::*; + + struct ForbiddenLifecycle; + + impl PlatformConfigStore for ForbiddenLifecycle { + fn get(&self, _store: &StoreName, _key: &str) -> Result> { + panic!("should not read trace config per request"); + } + fn put( + &self, + _store: &StoreId, + _key: &str, + _value: &str, + ) -> Result<(), Report> { + panic!("should not write trace config"); + } + fn delete(&self, _store: &StoreId, _key: &str) -> Result<(), Report> { + panic!("should not delete trace config"); + } + } + impl PlatformSecretStore for ForbiddenLifecycle { + fn get_bytes( + &self, + _store: &StoreName, + _key: &str, + ) -> Result, Report> { + panic!("should not read trace secrets"); + } + fn create( + &self, + _store: &StoreId, + _key: &str, + _value: &str, + ) -> Result<(), Report> { + panic!("should not write trace secrets"); + } + fn delete(&self, _store: &StoreId, _key: &str) -> Result<(), Report> { + panic!("should not delete trace secrets"); + } + } + impl PlatformBackend for ForbiddenLifecycle { + fn naming_policy(&self) -> BackendNamingPolicy { + panic!("should not resolve trace backends"); + } + fn predict_name( + &self, + _spec: &PlatformBackendSpec, + ) -> Result> { + panic!("should not predict trace backends"); + } + fn ensure(&self, _spec: &PlatformBackendSpec) -> Result> { + panic!("should not register trace backends"); + } + } + #[async_trait(?Send)] + impl PlatformKvStore for ForbiddenLifecycle { + async fn get_bytes(&self, _key: &str) -> Result, KvError> { + panic!("should not read trace identity"); + } + async fn put_bytes(&self, _key: &str, _value: Bytes) -> Result<(), KvError> { + panic!("should not write trace identity"); + } + async fn put_bytes_with_ttl( + &self, + _key: &str, + _value: Bytes, + _ttl: Duration, + ) -> Result<(), KvError> { + panic!("should not refresh trace identity"); + } + async fn delete(&self, _key: &str) -> Result<(), KvError> { + panic!("should not delete trace identity"); + } + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + panic!("should not enumerate trace identity"); + } + } + #[async_trait(?Send)] + impl PlatformHttpClient for ForbiddenLifecycle { + async fn send( + &self, + _request: PlatformHttpRequest, + ) -> Result> { + panic!("should not contact trace origin"); + } + async fn send_async( + &self, + _request: PlatformHttpRequest, + ) -> Result> { + panic!("should not launch trace auctions"); + } + async fn select( + &self, + _requests: Vec, + ) -> Result> { + panic!("should not collect trace auctions"); + } + } + #[async_trait(?Send)] + impl AuctionTelemetrySink for ForbiddenLifecycle { + fn is_enabled(&self) -> bool { + panic!("should not inspect trace auction telemetry"); + } + async fn emit_auction_events( + &self, + _services: &RuntimeServices, + _batch: AuctionEventBatch, + ) -> Result<(), Report> { + panic!("should not emit trace auction telemetry"); + } + } + struct CountingGeo(Arc); + impl PlatformGeo for CountingGeo { + fn lookup(&self, ip: Option) -> Result, Report> { + assert_eq!( + ip, + Some( + "192.0.2.99" + .parse() + .expect("should parse injected example IP") + ), + "should use injected trusted client metadata" + ); + self.0.fetch_add(1, Ordering::SeqCst); + Ok(Some(GeoInfo { + city: String::new(), + country: "GB".to_owned(), + continent: String::new(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: Some("EX".to_owned()), + asn: Some(64512), + })) + } + } + + fn routers( + enabled: bool, + auth_pattern: Option<&str>, + calls: Arc, + ) -> Vec<(&'static str, RouterService)> { + routers_with_client( + enabled, + auth_pattern, + calls, + Some( + "192.0.2.99" + .parse() + .expect("should parse injected client IP"), + ), + ) + } + + fn routers_with_client( + enabled: bool, + auth_pattern: Option<&str>, + calls: Arc, + client_ip: Option, + ) -> Vec<(&'static str, RouterService)> { + let mut settings = test_settings(); + settings + .integrations + .insert_config( + "gpt_diagnostics", + &serde_json::json!({"enabled":true,"trace_page_enabled":enabled}), + ) + .expect("should insert trace settings"); + if let Some(pattern) = auth_pattern { + settings.handlers.insert(0,serde_json::from_value(serde_json::json!({"path":pattern,"username":"example-user","password":"example-password"})).expect("should insert trace auth rule")); + } + let forbidden = Arc::new(ForbiddenLifecycle); + let services = RuntimeServices::builder() + .config_store(forbidden.clone()) + .secret_store(forbidden.clone()) + .kv_store(forbidden.clone()) + .backend(forbidden.clone()) + .http_client(forbidden.clone()) + .auction_telemetry_sink(forbidden) + .geo(Arc::new(CountingGeo(calls))) + .client_info(ClientInfo { + client_ip, + ..ClientInfo::default() + }) + .build(); + vec![ + ( + "Axum", + AxumApp::routes_with_settings_and_services(settings.clone(), services.clone()) + .expect("should build Axum trace routes"), + ), + ( + "Cloudflare", + CloudflareApp::routes_with_settings_and_services( + settings.clone(), + services.clone(), + ) + .expect("should build Cloudflare trace routes"), + ), + ( + "Spin", + SpinApp::routes_with_settings_and_services(settings, services) + .expect("should build Spin trace routes"), + ), + ] + } + + #[tokio::test] + async fn trace_dispatch_parity_missing_client_ip_skips_geo() { + let calls = Arc::new(AtomicUsize::new(0)); + for (adapter, router) in routers_with_client(true, None, Arc::clone(&calls), None) { + let response = RouterService::oneshot( + &router, + request_builder() + .uri("/_ts/trace") + .body(edgezero_core::body::Body::empty()) + .expect("should build missing-IP setup"), + ) + .await + .expect("should render setup without optional facts"); + assert_eq!( + response.status(), + StatusCode::OK, + "{adapter} should render missing optional facts" + ); + } + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "should not synthesize a geo lookup without trusted client IP" + ); + } + + #[tokio::test] + async fn trace_dispatch_parity_auth_precedes_flags_aliases_and_methods_without_services() { + let calls = Arc::new(AtomicUsize::new(0)); + for enabled in [false, true] { + for (pattern, path, expected) in [ + (Some("^/"), "/%5Fts/trace", StatusCode::UNAUTHORIZED), + ( + Some("^/_ts"), + "/%5Fts/trace", + if enabled { + StatusCode::BAD_REQUEST + } else { + StatusCode::NOT_FOUND + }, + ), + (Some("^/_ts"), "/_ts/trace/extra", StatusCode::UNAUTHORIZED), + ( + None, + "/_ts/trace", + if enabled { + StatusCode::METHOD_NOT_ALLOWED + } else { + StatusCode::NOT_FOUND + }, + ), + ] { + for (adapter, router) in routers(enabled, pattern, Arc::clone(&calls)) { + let request = request_builder() + .method("EXAMPLE-METHOD") + .uri(path) + .body(edgezero_core::body::Body::empty()) + .expect("should build extension method"); + let response = router + .oneshot(request) + .await + .expect("should reject locally"); + assert_eq!( + response.status(), + expected, + "{adapter} should apply auth then flag/path/method" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "{adapter} should retain terminal marker" + ); + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "no-store, private", + "{adapter} should keep trace errors private" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "{adapter} should not write rejected cookies" + ); + } + } + } + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "should never lookup denied metadata" + ); + } + + #[tokio::test] + async fn trace_dispatch_parity_read_only_setup_and_verified_assets() { + let calls = Arc::new(AtomicUsize::new(0)); + for (adapter, router) in routers(true, None, Arc::clone(&calls)) { + for (method, path, expected_calls) in [ + (Method::HEAD, "/_ts/trace", 0), + (Method::GET, "/_ts/trace/state", 0), + (Method::GET, "/_ts/trace/assets/v1.js", 0), + (Method::GET, "/_ts/trace/assets/v1.css", 0), + (Method::GET, "/_ts/trace", 1), + ] { + calls.store(0, Ordering::SeqCst); + let head = method == Method::HEAD; + let request = request_builder() + .method(method) + .uri(path) + .header("cookie", "__Host-ts-console=1") + .header("cf-connecting-ip", "203.0.113.22") + .header("spin-client-addr", "203.0.113.33:1234") + .body(edgezero_core::body::Body::empty()) + .expect("should build local read"); + let response = RouterService::oneshot(&router, request) + .await + .expect("should serve local read"); + assert_eq!( + response.status(), + StatusCode::OK, + "{adapter} should serve local route" + ); + assert_eq!( + calls.load(Ordering::SeqCst), + expected_calls, + "{adapter} should fetch metadata only for GET setup" + ); + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "{adapter} should not refresh cookies" + ); + let headers = response.headers().clone(); + let bytes = response + .into_body() + .into_bytes() + .expect("should buffer trace response"); + if head { + assert!(bytes.is_empty(), "{adapter} should remove HEAD body"); + } else if path == "/_ts/trace" { + let html = String::from_utf8(bytes.to_vec()).expect("should render UTF8 setup"); + assert!( + html.contains("192.0.2.0/24"), + "{adapter} should prefer injected client metadata" + ); + assert!( + !html.contains("192.0.2.99"), + "{adapter} should redact full client IP" + ); + assert!( + html.contains("64512"), + "{adapter} should project populated ASN" + ); + } else if path.contains("/assets/") { + assert_eq!( + headers[header::CACHE_CONTROL], + "public, max-age=31536000, immutable", + "{adapter} should cache unprotected fixed bytes" + ); + assert_eq!( + bytes.as_ref(), + trusted_server_js_asset(path), + "{adapter} should serve actual verified build bytes" + ); + } else { + assert_eq!( + bytes.as_ref(), + br#"{"observed_active":true}"#, + "{adapter} should return boolean state only" + ); + } + } + } + } + + fn trusted_server_js_asset(path: &str) -> &'static [u8] { + trusted_server_js::trace_assets::trace_asset(path) + .expect("should locate verified fixed asset") + .bytes + } + + #[tokio::test] + async fn trace_dispatch_parity_stream_actions_preserve_cookie_policy_and_separate_state() { + let calls = Arc::new(AtomicUsize::new(0)); + for (adapter, router) in routers(true, None, Arc::clone(&calls)) { + for action in ["enable", "end"] { + let mut request = request_builder() + .method("POST") + .uri(format!("https://publisher.example.com/_ts/trace/{action}")) + .header("host", "publisher.example.com") + .header("origin", "https://publisher.example.com") + .header("sec-fetch-site", "same-origin") + .header("x-ts-trace-action", action) + .header("cookie", "__Host-ts-console=1, unrelated=value") + .body(edgezero_core::body::Body::from_stream( + futures_trace_empty_stream(), + )) + .expect("should build no-content-type streamed action"); + request.extensions_mut().insert( + RequestIngress::new( + CapturedTarget::Unavailable(TargetUnavailable::NotExposed), + Some( + InboundOrigin::parse( + "https", + "publisher.example.com", + OriginSource::RuntimeUri, + ) + .expect("should freeze trusted origin"), + ), + HeaderFidelity::default(), + Vec::new(), + ) + .expect("should create ingress snapshot"), + ); + let response = RouterService::oneshot(&router, request) + .await + .expect("should request local mutation"); + assert_eq!( + response.status(), + StatusCode::OK, + "{adapter} should accept explicit empty streamed action independently of cookie ambiguity" + ); + let cookies: Vec<_> = response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .collect(); + assert_eq!( + cookies.len(), + 1, + "{adapter} should emit exactly one action cookie" + ); + let expected = if action == "enable" { + "__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800" + } else { + "__Host-ts-console=; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=0" + }; + assert_eq!( + cookies[0], expected, + "{adapter} should reuse the shared cookie policy" + ); + assert_eq!( + response + .into_body() + .into_bytes() + .expect("should buffer mutation response") + .as_ref(), + br#"{"mutation_requested":true}"#, + "{adapter} should report requested mutation only" + ); + let state = RouterService::oneshot( + &router, + request_builder() + .uri("/_ts/trace/state") + .header("cookie", "__Host-ts-console=1, unrelated=value") + .body(edgezero_core::body::Body::empty()) + .expect("should build separate state observation"), + ) + .await + .expect("should observe separate state"); + assert_eq!( + state + .into_body() + .into_bytes() + .expect("should buffer state") + .as_ref(), + br#"{"observed_active":false}"#, + "{adapter} should not infer accepted browser cookie mutation" + ); + } + } + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "should not lookup metadata for actions or state" + ); + } + + fn futures_trace_empty_stream() -> impl futures::Stream> { + futures::stream::iter([Ok(Bytes::new()), Ok(Bytes::new())]) + } +} diff --git a/crates/trusted-server-js/Cargo.toml b/crates/trusted-server-js/Cargo.toml index 67a4ac698..5a382a53a 100644 --- a/crates/trusted-server-js/Cargo.toml +++ b/crates/trusted-server-js/Cargo.toml @@ -20,6 +20,7 @@ path = "src/lib.rs" build-print = { workspace = true } hex = { workspace = true } sha2 = { workspace = true } +serde_json = { workspace = true } which = { workspace = true } [dependencies] diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index 8fbd34c9e..c00497351 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -84,6 +84,8 @@ fn main() { ); } + embed_trace_assets(&ts_dir, &dist_dir, &out_dir, !skip && npm.is_some()); + // Discover all tsjs-*.js files in dist/ let mut modules: Vec<(String, String)> = Vec::new(); // (id, filename) if let Ok(entries) = fs::read_dir(&dist_dir) { @@ -169,6 +171,133 @@ fn main() { }); } +fn embed_trace_assets(ts_dir: &Path, dist_dir: &Path, out_dir: &Path, rebuilt: bool) { + let manifest: serde_json::Value = serde_json::from_slice( + &fs::read(ts_dir.join("trace-assets-manifest.json")) + .expect("should read the committed trace asset manifest"), + ) + .expect("should parse the trace asset manifest"); + let object = manifest + .as_object() + .expect("should use a trace manifest object"); + assert!( + object.len() == 3 + && object.contains_key("schema_version") + && object.contains_key("assets") + && object.contains_key("source_sha256"), + "tsjs: trace manifest must contain exactly the documented fields" + ); + assert_eq!( + manifest["schema_version"].as_u64(), + Some(1), + "tsjs: trace manifest schema version must be one" + ); + assert_eq!( + manifest["source_sha256"].as_str(), + Some(trace_source_digest(ts_dir).as_str()), + "tsjs: trace build inputs changed; rebuild and review the unpublished versioned assets" + ); + let assets = manifest["assets"] + .as_array() + .expect("should list the versioned trace assets"); + assert_eq!( + assets.len(), + 2, + "tsjs: version one requires its JS and CSS assets" + ); + let mut generated = String::from("const TRACE_ASSETS: [TraceAsset; 2] = [\n"); + for (asset, file) in assets.iter().zip(["v1.js", "v1.css"]) { + let fields = asset + .as_object() + .expect("should represent trace asset metadata as an object"); + assert!( + fields.len() == 3 + && fields.contains_key("path") + && fields.contains_key("file") + && fields.contains_key("sha256"), + "tsjs: trace asset metadata must contain exactly path, file and sha256" + ); + let path = format!("/_ts/trace/assets/{file}"); + assert_eq!( + asset["path"].as_str(), + Some(path.as_str()), + "tsjs: trace asset path must be exact" + ); + assert_eq!( + asset["file"].as_str(), + Some(file), + "tsjs: trace asset filename must be exact" + ); + let content = fs::read(ts_dir.join("trace-assets").join(file)) + .expect("should read the committed versioned trace asset bytes"); + let digest = hex::encode(Sha256::digest(&content)); + assert_eq!( + asset["sha256"].as_str(), + Some(digest.as_str()), + "tsjs: committed trace asset bytes do not match their manifest digest" + ); + if rebuilt { + assert_eq!( + fs::read(dist_dir.join("trace").join(file)) + .expect("should rebuild both versioned trace assets"), + content, + "tsjs: rebuilt trace bytes differ from committed assets; review the unpublished draft or add a version" + ); + } + fs::write(out_dir.join(format!("trace-{file}")), &content) + .expect("should copy verified trace bytes into the build output"); + writeln!(generated, + "TraceAsset {{ path: \"{path}\", bytes: include_bytes!(concat!(env!(\"OUT_DIR\"), \"/trace-{file}\")), sha256: \"{digest}\" }},") + .expect("should generate the verified trace asset entry"); + } + generated.push_str("];\n"); + fs::write(out_dir.join("trace_assets.rs"), generated) + .expect("should generate the trace asset lookup table"); +} + +fn trace_source_digest(ts_dir: &Path) -> String { + let mut sources = vec![ + "build-all.mjs".to_owned(), + "trace-asset-sources.mjs".to_owned(), + "package-lock.json".to_owned(), + ]; + let mut directories = vec![PathBuf::from("src/trace")]; + while let Some(directory) = directories.pop() { + for entry in + fs::read_dir(ts_dir.join(&directory)).expect("should enumerate trace build inputs") + { + let entry = entry.expect("should read a trace build input entry"); + let relative = directory.join(entry.file_name()); + let kind = entry + .file_type() + .expect("should inspect the trace build input type"); + if kind.is_dir() { + directories.push(relative); + } else if kind.is_file() + && relative + .extension() + .is_some_and(|extension| extension == "ts" || extension == "css") + { + sources.push( + relative + .to_str() + .expect("should use UTF-8 trace build input filenames") + .replace('\\', "/"), + ); + } + } + } + sources.sort(); + let mut hash = Sha256::new(); + for source in sources { + hash.update(source.as_bytes()); + hash.update([0]); + hash.update(fs::read(ts_dir.join(&source)).expect("should read a trace build input")); + hash.update([0]); + } + hex::encode(hash.finalize()) +} + fn bundle_sha256(path: &Path) -> String { let content = fs::read(path).unwrap_or_else(|err| { panic!( diff --git a/crates/trusted-server-js/lib/.prettierignore b/crates/trusted-server-js/lib/.prettierignore index 72274829b..88957b43a 100644 --- a/crates/trusted-server-js/lib/.prettierignore +++ b/crates/trusted-server-js/lib/.prettierignore @@ -1,4 +1,4 @@ node_modules dist coverage - +trace-assets/** diff --git a/crates/trusted-server-js/lib/build-all.mjs b/crates/trusted-server-js/lib/build-all.mjs index 2bfee01b1..340c64330 100644 --- a/crates/trusted-server-js/lib/build-all.mjs +++ b/crates/trusted-server-js/lib/build-all.mjs @@ -39,7 +39,7 @@ const integrationModules = fs.existsSync(integrationsDir) ); }) .sort() - : []; + : []; console.log('[build-all] Discovered integrations:', integrationModules); @@ -83,6 +83,30 @@ await Promise.all( integrationModules.map((name) => buildModule(name, path.join(integrationsDir, name, 'index.ts'))) ); +// The full-document trace viewer is independent of publisher integration bundles. +await build({ + configFile: false, + root: __dirname, + build: { + emptyOutDir: false, + outDir: distDir, + sourcemap: false, + minify: 'esbuild', + cssCodeSplit: false, + rollupOptions: { + input: path.join(srcDir, 'trace', 'viewer.ts'), + output: { + format: 'iife', + inlineDynamicImports: true, + entryFileNames: 'trace/v1.js', + assetFileNames: 'trace/v1.css', + name: 'tsTraceViewer', + }, + }, + }, + logLevel: 'warn', +}); + // List all built files const builtFiles = fs .readdirSync(distDir) diff --git a/crates/trusted-server-js/lib/eslint.config.js b/crates/trusted-server-js/lib/eslint.config.js index 2720ba3a0..2d6b83f03 100644 --- a/crates/trusted-server-js/lib/eslint.config.js +++ b/crates/trusted-server-js/lib/eslint.config.js @@ -9,7 +9,7 @@ import unicorn from 'eslint-plugin-unicorn'; export default [ // Files/folders to ignore { - ignores: ['node_modules', 'dist', 'coverage'], + ignores: ['node_modules', 'dist', 'coverage', 'trace-assets/**'], }, // Base JS recommended js.configs.recommended, diff --git a/crates/trusted-server-js/lib/src/core/auction.ts b/crates/trusted-server-js/lib/src/core/auction.ts index a02684362..4d2e0fb3a 100644 --- a/crates/trusted-server-js/lib/src/core/auction.ts +++ b/crates/trusted-server-js/lib/src/core/auction.ts @@ -3,6 +3,11 @@ // and the Prebid.js trustedServer adapter. import { parseApsRendererDescriptor } from '../integrations/aps/render'; +import { + prepareTraceAuctionRequest, + observeTraceApiResponse, + observeTraceApiFailure, +} from '../trace/runtime'; import { log } from './log'; import type { ApsRendererV1 } from './types'; @@ -13,6 +18,8 @@ import type { ApsRendererV1 } from './types'; /** A single ad unit in the AdRequest payload sent to POST /auction. */ export interface AdRequestUnit { + /** Optional namespaced request-scoped diagnostic slot reference. */ + ext?: Record; code: string; mediaTypes: { banner?: { sizes: number[][] }; @@ -173,7 +180,10 @@ export function parseAuctionResponse(body: any): AuctionBid[] { * Returns an empty array on network or parse errors (non-throwing). */ export async function sendAuction(endpoint: string, request: AdRequest): Promise { + const trace = prepareTraceAuctionRequest(request); + const outgoing = trace?.request ?? request; if (typeof fetch !== 'function') { + observeTraceApiFailure(trace); log.warn('auction: fetch not available'); return []; } @@ -185,18 +195,20 @@ export async function sendAuction(endpoint: string, request: AdRequest): Promise method: 'POST', headers: { 'content-type': 'application/json' }, credentials: 'same-origin', - body: JSON.stringify(request), + body: JSON.stringify(outgoing), keepalive: true, }); const contentType = response.headers.get('content-type') || ''; if (response.ok && contentType.includes('application/json')) { const data: unknown = await response.json(); + observeTraceApiResponse(trace, data); const bids = parseAuctionResponse(data); log.info('auction: received bids', { count: bids.length }); return bids; } + observeTraceApiFailure(trace); log.warn('auction: unexpected response', { ok: response.ok, status: response.status, @@ -204,6 +216,7 @@ export async function sendAuction(endpoint: string, request: AdRequest): Promise }); return []; } catch (error) { + observeTraceApiFailure(trace); log.warn('auction: request failed', error); return []; } diff --git a/crates/trusted-server-js/lib/src/core/global.d.ts b/crates/trusted-server-js/lib/src/core/global.d.ts index c7c8b08fb..bc5d2f247 100644 --- a/crates/trusted-server-js/lib/src/core/global.d.ts +++ b/crates/trusted-server-js/lib/src/core/global.d.ts @@ -2,6 +2,8 @@ import type { TsjsApi } from './types'; declare global { interface Window { + __tsjs_trace_active?: unknown; + __tsjs_trace_request_context?: unknown; tsjs?: TsjsApi; pbjs?: TsjsApi; } diff --git a/crates/trusted-server-js/lib/src/core/index.ts b/crates/trusted-server-js/lib/src/core/index.ts index b2c4e41e1..eccfb9d1b 100644 --- a/crates/trusted-server-js/lib/src/core/index.ts +++ b/crates/trusted-server-js/lib/src/core/index.ts @@ -6,6 +6,8 @@ export type { GptDiagnosticsRequestCycle, TsjsApi, } from './types'; +import { installTraceRuntime } from '../trace/runtime'; + import type { TsjsApi } from './types'; import { addAdUnits } from './registry'; import { renderAdUnit, renderAllAdUnits } from './render'; @@ -44,6 +46,7 @@ api.adSlots ??= []; api.bids ??= {}; // Point global tsjs w.tsjs = api; +installTraceRuntime(api, w); // Single shared queue installQueue(api, w); diff --git a/crates/trusted-server-js/lib/src/core/types.ts b/crates/trusted-server-js/lib/src/core/types.ts index 3407d487a..81a6a1f4a 100644 --- a/crates/trusted-server-js/lib/src/core/types.ts +++ b/crates/trusted-server-js/lib/src/core/types.ts @@ -1,3 +1,7 @@ +import type { TraceCollector } from '../trace/collector'; +import type { TraceGptIdentity } from '../trace/types'; +import type { TraceGptBridge } from '../trace/gpt'; + // Shared TypeScript types for the tsjs core API and extensions. export type Size = readonly [number, number]; @@ -27,6 +31,10 @@ export interface AuctionSlot { div_id: string; formats: Array<[number, number]>; targeting?: Record; + ext?: { + trusted_server?: { trace_slot_ref?: unknown; [key: string]: unknown }; + [key: string]: unknown; + }; } /** Debug-only copy of server-side bid fields exposed for pipeline inspection. */ @@ -332,7 +340,8 @@ export interface GptDiagnosticsRecorder { auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, trustedServerAuctionId?: string, - requestedSlotSizes?: ReadonlyArray + requestedSlotSizes?: ReadonlyArray, + traceIdentity?: TraceGptIdentity ): void; /** Mark slots whose next observed GPT request follows the Prebid refresh path. */ recordPrebidRefresh(slots: GptDiagnosticsSlotHandle[]): void; @@ -404,6 +413,10 @@ export interface FirstImpressionState { } export interface TsjsApi { + /** Internal trace observations, installed only for a literal activated document. */ + traceEvidence?: TraceCollector; + /** Internal active-only bridge shared with the early GPT bootstrap. */ + traceGpt?: TraceGptBridge; version: string; que: Array<() => void>; addAdUnits(units: AdUnit | AdUnit[]): void; @@ -515,7 +528,8 @@ export interface TsjsApi { */ scheduleInitialAdInit?: ( initialBids?: Record, - initialSlots?: AuctionSlot[] + initialSlots?: AuctionSlot[], + traceAuctionTransport?: unknown ) => void; /** Read-only GPT lifecycle diagnostics API, present only in an activated tab. */ gptDiagnostics?: GptDiagnosticsApi; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts index fc18beb5e..9ae87dbbd 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts @@ -8,6 +8,8 @@ import { } from '../../core/first_impression'; import { log } from '../../core/log'; import { resolveSlotElementByDivId } from '../../core/slot_element'; +import { getActiveTraceGptBridge } from '../../trace/runtime'; +import type { TraceGptOpportunity } from '../../trace/gpt'; import type { AuctionSlot, AuctionBidData, @@ -799,10 +801,19 @@ function installInitialLoadDetector(ts: TsjsApi): void { * riding rAF keeps a single code path whose post-hydration-commit guarantee * holds whenever the request is actually issued. */ +function traceOpportunity(slot: AuctionSlot, auctionId: string | undefined): TraceGptOpportunity { + try { + return getActiveTraceGptBridge()?.opportunity(slot, auctionId) ?? { auctionId }; + } catch { + return { auctionId }; + } +} + function installScheduleInitialAdInit(ts: TsjsApi): void { ts.scheduleInitialAdInit = function ( initialBids?: Record, - initialSlots?: AuctionSlot[] + initialSlots?: AuctionSlot[], + traceAuctionTransport?: unknown ) { if ((ts.navGeneration ?? 0) !== 0 || ts.initialAdInitScheduled) return; ts.initialAdInitScheduled = true; @@ -810,6 +821,15 @@ function installScheduleInitialAdInit(ts: TsjsApi): void { if (initialBids !== undefined) ts.bids = initialBids; const runUnlessNavigated = (): void => { if ((ts.navGeneration ?? 0) !== 0) return; + try { + getActiveTraceGptBridge()?.observeTransport( + initialSlots ?? ts.adSlots, + traceAuctionTransport, + 'initial_navigation_ssat' + ); + } catch { + // A diagnostic bridge cannot block the ordinary initial ad pass. + } ts.adInit?.(); }; const afterHydrationFrames = (): void => { @@ -1187,12 +1207,14 @@ function schedulePublisherFirstImpressionFallback( if (tsOwned) (ts.prevGptSlots ??= []).push(gptSlot); try { + const trace = traceOpportunity(slot, bid.hb_auction_id); ts.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( gptSlot, slot.id, trustedServerOpportunity(bid), - bid.hb_auction_id, - slot.formats + trace.auctionId, + slot.formats, + ...(trace.identity ? ([trace.identity] as const) : []) ); } catch { // Diagnostics must not alter fallback delivery. @@ -1382,12 +1404,14 @@ export function installTsAdInit(): void { try { const requestedSlotSizes = ts.gptSlotHandoffs?.[slotDivId2]?.formats; const opportunity = trustedServerOpportunity(bid); + const trace = traceOpportunity(slot, bid.hb_auction_id); ts.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( gptSlot, slot.id, opportunity, - bid.hb_auction_id, - requestedSlotSizes + trace.auctionId, + requestedSlotSizes, + ...(trace.identity ? ([trace.identity] as const) : []) ); } catch { // Diagnostics must not alter ad delivery. @@ -1490,6 +1514,7 @@ export function installTsAdInit(): void { interface PageBidsResponse { slots: AuctionSlot[]; bids: Record; + trace_auction?: unknown; } /** Canonical SPA re-auction endpoint. Mirrors `PAGE_BIDS_PATH` in Rust. */ @@ -1724,6 +1749,11 @@ export function installSpaAuctionHook(): void { if (inflight !== controller) return; ts.adSlots = data.slots; ts.bids = data.bids; + try { + getActiveTraceGptBridge()?.observePageBids(data, data.slots); + } catch { + // A diagnostic bridge cannot block the accepted navigation. + } // This route is now the committed, loaded state — a later failed // navigation rolls back here, and a return trip no-ops correctly. lastAppliedPath = path; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts index 475bc7f93..1388b8f6b 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts @@ -6,6 +6,7 @@ import type { GptDiagnosticsSlotHandle, GptDiagnosticsTrustedServerOpportunity, } from '../../core/types'; +import type { TraceGptIdentity } from '../../trace/types'; import type { GptDiagnosticsBindingManager } from './binding'; import type { GptDiagnosticsStoreSnapshot } from './store'; @@ -18,7 +19,8 @@ interface ApiStore { auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, trustedServerAuctionId?: string, - requestedSlotSizes?: ReadonlyArray + requestedSlotSizes?: ReadonlyArray, + traceIdentity?: TraceGptIdentity ): void; recordPrebidRefresh(slots: GptDiagnosticsSlotHandle[]): void; recordTrustedServerCreativeRequest(auctionSlotId: string): number | undefined; @@ -157,7 +159,8 @@ export class GptDiagnosticsApiController { auctionSlotId, opportunity, trustedServerAuctionId, - requestedSlotSizes + requestedSlotSizes, + traceIdentity ) => safelyRecord(() => { this.store.recordTrustedServerOpportunity( @@ -165,7 +168,8 @@ export class GptDiagnosticsApiController { auctionSlotId, opportunity, trustedServerAuctionId, - requestedSlotSizes + requestedSlotSizes, + ...(traceIdentity ? ([traceIdentity] as const) : []) ); }), recordPrebidRefresh: (slots) => safelyRecord(() => this.store.recordPrebidRefresh(slots)), diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts index d7271710c..ac8a0de4c 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts @@ -1,5 +1,8 @@ import { log } from '../../core/log'; import type { GptDiagnosticsApi, TsjsApi } from '../../core/types'; +import { createTraceHandoff } from '../../trace/handoff'; +import type { TraceHandoff } from '../../trace/handoff'; +import { getActiveTraceCollector } from '../../trace/runtime'; import { GptDiagnosticsApiController } from './api'; import { GptDiagnosticsBadgeManager } from './badges'; @@ -47,11 +50,19 @@ export function installGptDiagnosticsRuntime( let overlay: GptDiagnosticsOverlay | undefined; let slotSizeObserver: GptDiagnosticsSlotSizeObserver | undefined; let apiController: GptDiagnosticsApiController | undefined; + let traceHandoff: TraceHandoff | undefined; try { if (!target.tsjs) throw new Error('TSJS core API unavailable'); - const store = new GptDiagnosticsStore(); + const store = new GptDiagnosticsStore( + target.__tsjs_trace_active === true + ? { + onTraceCorrelation: (value) => + getActiveTraceCollector(target)?.recordCorrelation(value), + } + : {} + ); const observer = new GptDiagnosticsObserver(store, { window: target }); bindings = new GptDiagnosticsBindingManager(store, { window: target, @@ -66,6 +77,12 @@ export function installGptDiagnosticsRuntime( window: target, document: target.document, onExport: () => apiController?.api.export(), + ...(target.__tsjs_trace_active === true + ? { + onViewTrace: () => traceHandoff?.view(), + onDownloadTrace: () => traceHandoff?.download(), + } + : {}), onBadgeLayerChange: (layer) => badges?.setLayer(layer), }); apiController = new GptDiagnosticsApiController(store, bindings, overlay, { @@ -79,6 +96,8 @@ export function installGptDiagnosticsRuntime( const runtime: GptDiagnosticsRuntime = { api, destroy: () => { + traceHandoff?.destroy(); + traceHandoff = undefined; if (target.tsjs?.gptDiagnostics === api) delete target.tsjs.gptDiagnostics; if (target.tsjs?.gptDiagnosticsRecorder === recorder) { delete target.tsjs.gptDiagnosticsRecorder; @@ -93,9 +112,17 @@ export function installGptDiagnosticsRuntime( }; target.tsjs.gptDiagnostics = api; target.tsjs.gptDiagnosticsRecorder = recorder; + if (target.__tsjs_trace_active === true) { + traceHandoff = createTraceHandoff({ + target, + onChange: (state) => overlay?.setTraceState(state), + }); + } target.__tsjs_gpt_diagnostics_runtime = runtime; return api; } catch (error) { + traceHandoff?.destroy(); + traceHandoff = undefined; apiController?.destroy(); overlay?.destroy(); badges?.destroy(); diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index 63c0b6fb3..f419b7aee 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -1,5 +1,6 @@ import { log } from '../../core/log'; import type { GptDiagnosticsRequestCycle } from '../../core/types'; +import type { TraceHandoffState } from '../../trace/handoff'; import type { GptDiagnosticsBindingManager } from './binding'; import { unhandledCase } from './exhaustive'; @@ -29,6 +30,8 @@ interface OverlayOptions { document?: Document; scheduleFrame?: (callback: () => void) => void; onExport?: () => void; + onViewTrace?: () => void; + onDownloadTrace?: () => void; onShadowRoot?: (root: ShadowRoot) => void; onBadgeLayerChange?: (layer: HTMLElement | undefined) => void; } @@ -71,6 +74,9 @@ const PANEL_STYLES = ` button { cursor: pointer; } button:focus-visible, select:focus-visible, summary:focus-visible { outline: 2px solid #60a5fa; outline-offset: 2px; } .tsgd-toolbar { display: flex; gap: 8px; align-items: center; border-bottom: 1px solid #334155; } + .tsgd-toolbar { flex-wrap: wrap; } + .tsgd-trace-action { min-height: 44px; background: #1d4ed8; font-weight: 700; } + .tsgd-trace-status { margin: 0; padding: 0 12px 10px; color: #cbd5e1; } .tsgd-toolbar label { color: #cbd5e1; } .tsgd-summary { color: #cbd5e1; border-bottom: 1px solid #334155; } .tsgd-coverage { margin: 6px 0 0; padding: 0; list-style: none; font-size: 12px; } @@ -335,6 +341,9 @@ export class GptDiagnosticsOverlay { private readonly document: Document; private readonly scheduleFrame: (callback: () => void) => void; private readonly onExport: () => void; + private readonly onViewTrace?: () => void; + private readonly onDownloadTrace?: () => void; + private traceState: TraceHandoffState = { kind: 'ready', downloadAvailable: false }; private readonly onShadowRoot?: (root: ShadowRoot) => void; private readonly onBadgeLayerChange?: (layer: HTMLElement | undefined) => void; private readonly unsubscribeStore: () => void; @@ -360,6 +369,8 @@ export class GptDiagnosticsOverlay { this.scheduleFrame = options.scheduleFrame ?? ((callback) => scheduleFrame(this.window, callback)); this.onExport = options.onExport ?? (() => undefined); + this.onViewTrace = options.onViewTrace; + this.onDownloadTrace = options.onDownloadTrace; this.onShadowRoot = options.onShadowRoot; this.onBadgeLayerChange = options.onBadgeLayerChange; this.unsubscribeStore = this.store.subscribe(() => this.scheduleRender()); @@ -381,6 +392,13 @@ export class GptDiagnosticsOverlay { this.removeHost(); } + /** Updates the optional trace action without changing the GPT-only snapshot. */ + setTraceState(state: TraceHandoffState): void { + if (this.destroyed || !this.onViewTrace) return; + this.traceState = state; + this.scheduleRender(); + } + destroy(): void { if (this.destroyed) return; this.destroyed = true; @@ -556,7 +574,40 @@ export class GptDiagnosticsOverlay { const exportButton = this.button('Export JSON', () => this.onExport()); filterLabel.append(select); toolbar.append(filterLabel, exportButton); + if (this.onViewTrace) { + const viewTrace = this.button('View trace results', this.onViewTrace); + viewTrace.className = 'tsgd-trace-action'; + viewTrace.disabled = + this.traceState.kind === 'capturing' || this.traceState.kind === 'navigating'; + toolbar.prepend(viewTrace); + if (this.traceState.downloadAvailable && this.onDownloadTrace) { + const download = this.button('Download trace report', this.onDownloadTrace); + download.className = 'tsgd-trace-action'; + toolbar.append(download); + } + } panel.append(toolbar); + if (this.onViewTrace) { + const messages: Record = { + ready: 'Capture the current page, then view its trace in this tab.', + capturing: 'Capturing trace results…', + capture_failed: 'Trace results could not be captured. Stay on this page and retry.', + storage_unavailable: + 'Trace results could not be saved in this tab. Download this report or retry.', + navigation_unavailable: + 'The report was saved, but the viewer could not be opened. Download it or retry.', + navigating: 'Opening trace results in this tab…', + download_failed: + 'The download could not be started. Your report remains available for retry.', + downloaded: 'The download was started. Your report remains available.', + }; + const traceStatus = this.document.createElement('p'); + traceStatus.className = 'tsgd-trace-status'; + traceStatus.setAttribute('role', 'status'); + traceStatus.setAttribute('aria-live', 'polite'); + traceStatus.textContent = messages[this.traceState.kind]; + panel.append(traceStatus); + } const summary = this.document.createElement('div'); summary.className = 'tsgd-summary'; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index ca3348ae9..2f2704b54 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -16,6 +16,30 @@ import type { GptDiagnosticsTrustedServerOpportunity, Size, } from '../../core/types'; +import type { TraceGptIdentity, TraceSlotCorrelationV1 } from '../../trace/types'; +import { validDiagnosticAuctionId, validTraceSlotRef } from '../../trace/validation'; + +function normalizedTraceIdentity( + value: TraceGptIdentity | undefined +): TraceGptIdentity | undefined { + try { + if (!value || Reflect.ownKeys(value).length !== 2) return undefined; + const auction = Object.getOwnPropertyDescriptor(value, 'diagnostic_auction_id'); + const slot = Object.getOwnPropertyDescriptor(value, 'slot_ref'); + if ( + !auction || + !slot || + !Object.prototype.hasOwnProperty.call(auction, 'value') || + !Object.prototype.hasOwnProperty.call(slot, 'value') || + !validDiagnosticAuctionId(auction.value) || + !validTraceSlotRef(slot.value) + ) + return undefined; + return Object.freeze({ diagnostic_auction_id: auction.value, slot_ref: slot.value }); + } catch { + return undefined; + } +} export const MAX_DIAGNOSTIC_SLOTS = 64; export const MAX_REQUEST_CYCLES_PER_SLOT = 10; @@ -99,6 +123,7 @@ interface StoreOptions { schedule?: (callback: () => void) => void; /** Deferred marker cleanup and diagnostic-window re-notification. */ defer?: (callback: () => void, delayMs: number) => void; + onTraceCorrelation?: (value: TraceSlotCorrelationV1) => void; } type RequestIntentSource = 'trusted_server_direct' | 'prebid_refresh' | 'publisher_refresh'; @@ -108,6 +133,7 @@ interface PendingSourceEvidence { trustedServerOpportunity?: GptDiagnosticsTrustedServerOpportunity; trustedServerAuctionId?: string; requestedSlotSizes?: ReadonlyArray; + traceIdentity?: TraceGptIdentity; } interface PendingRequestIntent { @@ -298,6 +324,7 @@ export class GptDiagnosticsStore { private readonly now: () => number; private readonly schedule: (callback: () => void) => void; private readonly defer: (callback: () => void, delayMs: number) => void; + private readonly onTraceCorrelation?: (value: TraceSlotCorrelationV1) => void; /** Auction slot ID → GPT slot, established by the Trusted Server integration. */ private readonly trustedServerSlots = new Map(); private readonly pendingRequestIntents = new WeakMap(); @@ -329,6 +356,7 @@ export class GptDiagnosticsStore { this.now = options.now ?? (() => performance.now()); this.schedule = options.schedule ?? ((callback) => queueMicrotask(callback)); this.defer = options.defer ?? ((callback, delayMs) => setTimeout(callback, delayMs)); + this.onTraceCorrelation = options.onTraceCorrelation; } /** Record Trusted Server's opportunity evidence for a GPT slot's next request. */ @@ -337,7 +365,8 @@ export class GptDiagnosticsStore { auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, trustedServerAuctionId?: string, - requestedSlotSizes?: ReadonlyArray + requestedSlotSizes?: ReadonlyArray, + traceIdentity?: TraceGptIdentity ): void { if ( !isSlotObject(slot) || @@ -356,10 +385,19 @@ export class GptDiagnosticsStore { this.trustedServerSlots.delete(oldest); } + const auctionId = normalizedAuctionId(trustedServerAuctionId); + const identity = this.onTraceCorrelation ? normalizedTraceIdentity(traceIdentity) : undefined; + const correlatedIdentity = + identity && + validDiagnosticAuctionId(auctionId) && + auctionId === identity.diagnostic_auction_id + ? identity + : undefined; this.recordRequestIntentSource(slot, 'trusted_server_direct', { trustedServerOpportunity: opportunity, - trustedServerAuctionId: normalizedAuctionId(trustedServerAuctionId), + trustedServerAuctionId: auctionId, requestedSlotSizes: normalizedRequestedSlotSizes(requestedSlotSizes), + ...(correlatedIdentity ? { traceIdentity: correlatedIdentity } : {}), }); } @@ -616,6 +654,23 @@ export class GptDiagnosticsStore { } : {}), }); + if ( + trustedServerEvidence?.traceIdentity && + this.onTraceCorrelation && + Number.isSafeInteger(record.runtimeSlotNumber) && + Number.isSafeInteger(requestNumber) + ) { + try { + this.onTraceCorrelation({ + schema_version: 1, + ...trustedServerEvidence.traceIdentity, + runtime_slot_number: record.runtimeSlotNumber, + request_number: requestNumber, + }); + } catch { + // A failed trace callback must not interrupt the actual GPT cycle. + } + } this.incrementDisposition('slotRequested', 'matched'); this.notify(); } @@ -936,7 +991,7 @@ export class GptDiagnosticsStore { source: RequestIntentSource, facts: Pick< PendingSourceEvidence, - 'trustedServerOpportunity' | 'trustedServerAuctionId' | 'requestedSlotSizes' + 'trustedServerOpportunity' | 'trustedServerAuctionId' | 'requestedSlotSizes' | 'traceIdentity' > = {} ): void { const observedAtMs = this.now(); diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 2ca2fe6b1..dc969429d 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -27,6 +27,8 @@ import { buildAdRequest, parseAuctionResponse } from '../../core/auction'; import { registerApsPrebidRenderer, validateApsRenderer } from '../aps/render'; import type { AuctionBid, AuctionEid } from '../../core/auction'; import type { AuctionSlot, TsjsApi } from '../../core/types'; +import { getActiveTraceCollector, prepareTraceAuctionRequest } from '../../trace/runtime'; +import { createTracePending, type TracePendingHandle } from '../../trace/pending'; import { PREBID_USER_ID_MODULE_REGISTRY, @@ -999,7 +1001,11 @@ type TrustedServerBidRequest = { adUnitCode?: string; code?: string; bidId?: string; + bidderRequestId?: string; }; +function traceHookBindings(bids: TrustedServerBidRequest[]) { + return bids.map((bid) => ({ bidId: bid.bidId ?? '', bidderRequestId: bid.bidderRequestId })); +} type TrustedServerRequest = { method: 'POST'; url: string; @@ -1007,6 +1013,7 @@ type TrustedServerRequest = { options: { contentType: 'application/json' }; bidRequests: TrustedServerBidRequest[]; tsjsBidRequests: TrustedServerBidRequest[]; + tracePending?: TracePendingHandle; }; type PrebidUserIdEid = { @@ -2344,11 +2351,60 @@ export function installPrebidNpm(config?: Partial): typeof pbjs log.warn('[tsjs-prebid] Prebid bundle lacks markWinningBidAsUsed; APS renderer bids disabled'); } + const traceCollector = getActiveTraceCollector(); + let tracePending = traceCollector + ? createTracePending({ + timeout: () => pbjs.getConfig('bidderTimeout'), + }) + : undefined; + if (tracePending) { + const pending = tracePending; + const retireTracePending = (event: PageTransitionEvent): void => { + if (event.persisted) pending.clear(); + else { + pending.destroy(); + try { + window.removeEventListener('pagehide', retireTracePending); + } catch { + /* Already retired diagnostic state cannot affect page cleanup. */ + } + } + }; + try { + window.addEventListener('pagehide', retireTracePending); + } catch { + pending.destroy(); + tracePending = undefined; + } + } + + const activeTracePending = tracePending; + // Register the trustedServer adapter using pbjs.registerBidAdapter(null, code, spec) // eslint-disable-next-line @typescript-eslint/no-explicit-any (pbjs as any).registerBidAdapter(undefined, ADAPTER_CODE, { code: ADAPTER_CODE, supportedMediaTypes: ['banner'], + ...(activeTracePending + ? { + onTimeout(timedOutBids: TrustedServerBidRequest[]) { + try { + if (timedOutBids.length <= 2048) + activeTracePending.failure(traceHookBindings(timedOutBids)); + } catch { + /* Diagnostic hook input cannot interrupt Prebid callbacks. */ + } + }, + onBidderError(value: { bidderRequest?: { bids?: TrustedServerBidRequest[] } }) { + try { + const bids = value.bidderRequest?.bids ?? []; + if (bids.length <= 2048) activeTracePending.failure(traceHookBindings(bids)); + } catch { + /* Error objects and response bodies are never inspected. */ + } + }, + } + : {}), isBidRequestValid(): boolean { return true; // All requests are valid — orchestrator handles filtering @@ -2363,15 +2419,43 @@ export function installPrebidNpm(config?: Partial): typeof pbjs clearPrebidEidsCookie(); } const payload = buildAdRequest(validBidRequests, { eids: auctionEids }); + const traceCarry = activeTracePending ? prepareTraceAuctionRequest(payload) : undefined; + let traceHandle: TracePendingHandle | undefined; + if (activeTracePending && traceCarry) { + try { + const refsByCode = new Map( + traceCarry.request.adUnits + .slice(0, 64) + .map((unit, index) => [unit.code, traceCarry.slotRefs[index]]) + ); + // Oversized ID collections keep exact readable-response capture, while + // declining unsupported hook association rather than guessing a prefix. + const bindings = + validBidRequests.length <= 2048 + ? validBidRequests.map((bid) => ({ + bidId: bid.bidId ?? '', + bidderRequestId: bid.bidderRequestId, + slotRef: refsByCode.get(bid.adUnitCode ?? bid.code ?? ''), + })) + : undefined; + traceHandle = activeTracePending.add(bindings, { + collector: traceCarry.collector, + slotRefs: traceCarry.slotRefs, + }); + } catch { + /* Retaining diagnostic state cannot change the ordinary request. */ + } + } return { method: 'POST', url: auctionEndpoint, - data: JSON.stringify(payload), + data: JSON.stringify(traceCarry?.request ?? payload), options: { contentType: 'application/json' }, // Keep bid requests on the request object so interpretResponse can // map bids without relying on shared mutable adapter state. bidRequests: requestScopedBidRequests, tsjsBidRequests: requestScopedBidRequests, + ...(traceHandle ? { tracePending: traceHandle } : {}), }; }, @@ -2381,6 +2465,11 @@ export function installPrebidNpm(config?: Partial): typeof pbjs request?: Partial ) { const body = serverResponse?.body; + try { + activeTracePending?.response(request?.tracePending, body); + } catch { + /* Optional diagnostic handles cannot interrupt ordinary bid parsing. */ + } log.debug('[tsjs-prebid] interpretResponse', { hasSeatbid: !!body?.seatbid }); const auctionBids = parseAuctionResponse(body); const bidRequests = request?.tsjsBidRequests ?? request?.bidRequests ?? []; diff --git a/crates/trusted-server-js/lib/src/trace/collector.ts b/crates/trusted-server-js/lib/src/trace/collector.ts new file mode 100644 index 000000000..74dfaade1 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/collector.ts @@ -0,0 +1,147 @@ +import { addOmissions } from './omissions'; +import type { TraceCollectorSnapshot } from './report'; +import { TRACE_COVERAGE_ISSUES, type TraceCoverageIssue } from './report-types'; +import type { TraceAuctionEvidenceV1, TraceSlotCorrelationV1 } from './types'; +import { parseTraceAuctionTransport, parseTraceSlotCorrelation, traceEnum } from './validation'; + +export type TraceCollectorRead = + | { readonly ok: true; readonly value: TraceCollectorSnapshot; readonly reason?: never } + | { readonly ok: false; readonly reason: 'omission_counter_overflow'; readonly value?: never }; +/** One shared page-local collector; caller integration supplies the strict activation gate. */ +export interface TraceCollector { + recordTransport(value: unknown): void; + recordTransportFailure(): void; + recordCorrelation(value: unknown): void; + recordInterpretationIssue( + issue: 'correlation_unavailable' | 'external_client_side_unobservable' + ): void; + snapshot(): TraceCollectorRead; + captureStatus(): 'complete' | 'partial' | 'unavailable' | 'not_observed'; + destroy(): void; +} + +/** Retains bounded owned observations in memory without storage or network effects. */ +export function createTraceCollector(): TraceCollector { + let records: TraceAuctionEvidenceV1[] = []; + let sidecars: TraceSlotCorrelationV1[] = []; + const issues = new Set(); + let omittedAuctions = 0; + let omittedSidecars = 0; + let overflow = false; + let destroyed = false; + const active = () => !destroyed && !overflow; + const add = (current: number, added: number): number => { + try { + return addOmissions(current, added); + } catch { + overflow = true; + return current; + } + }; + const prune = (id: string): void => { + const retained = sidecars.filter((sidecar) => sidecar.diagnostic_auction_id !== id); + const removed = sidecars.length - retained.length; + if (!removed) return; + omittedSidecars = add(omittedSidecars, removed); + sidecars = retained; + issues.add('correlation_unavailable'); + }; + const api: TraceCollector = { + recordTransport(value) { + if (!active() || value === undefined) return; + const transport = parseTraceAuctionTransport(value); + if (!transport) { + issues.add('evidence_validation_failed'); + return; + } + if (!transport.evidence) { + issues.add('evidence_projection_failed'); + return; + } + const record = transport.evidence; + if (record.source === 'auction_api') { + issues.add('correlation_unavailable'); + prune(record.diagnostic_auction_id); + } + records.push(record); + if (records.length <= 16) return; + const evicted = records.shift()!; + omittedAuctions = add(omittedAuctions, 1); + issues.add('record_evicted'); + if ( + !records.some( + (retained) => retained.diagnostic_auction_id === evicted.diagnostic_auction_id + ) + ) + prune(evicted.diagnostic_auction_id); + }, + recordTransportFailure() { + if (active()) issues.add('evidence_transport_failed'); + }, + recordCorrelation(value) { + if (!active()) return; + const sidecar = parseTraceSlotCorrelation(value); + if ( + !sidecar || + records.some( + (record) => + record.source === 'auction_api' && + record.diagnostic_auction_id === sidecar.diagnostic_auction_id + ) + ) { + issues.add('correlation_unavailable'); + return; + } + sidecars.push(sidecar); + if (sidecars.length <= 128) return; + sidecars.shift(); + omittedSidecars = add(omittedSidecars, 1); + issues.add('correlation_unavailable'); + }, + recordInterpretationIssue(issue) { + if ( + active() && + traceEnum(issue, ['correlation_unavailable', 'external_client_side_unobservable']) + ) + issues.add(issue); + }, + snapshot() { + if (overflow) return Object.freeze({ ok: false, reason: 'omission_counter_overflow' }); + return Object.freeze({ + ok: true, + value: Object.freeze({ + serverAuctions: Object.freeze([...records]), + slotCorrelations: Object.freeze([...sidecars]), + issues: Object.freeze(TRACE_COVERAGE_ISSUES.filter((issue) => issues.has(issue))), + omittedServerAuctions: omittedAuctions, + omittedSlotCorrelations: omittedSidecars, + }), + }); + }, + captureStatus() { + const failed = [ + 'evidence_projection_failed', + 'evidence_transport_failed', + 'evidence_validation_failed', + 'record_evicted', + ].some((issue) => issues.has(issue as TraceCoverageIssue)); + return records.length + ? failed + ? 'partial' + : 'complete' + : failed + ? 'unavailable' + : 'not_observed'; + }, + destroy() { + destroyed = true; + records = []; + sidecars = []; + issues.clear(); + omittedAuctions = 0; + omittedSidecars = 0; + overflow = false; + }, + }; + return Object.freeze(api); +} diff --git a/crates/trusted-server-js/lib/src/trace/context.ts b/crates/trusted-server-js/lib/src/trace/context.ts new file mode 100644 index 000000000..3263fedb9 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/context.ts @@ -0,0 +1,164 @@ +import type { TraceRequestContextV1 } from './types'; + +/** Tests an own property without relying on newer browser object helpers. */ +export function traceOwn(value: object, key: PropertyKey): boolean { + return Object.prototype.hasOwnProperty.call(value, key); +} + +/** Accepts only plain JSON-shaped objects with their own data properties. */ +export function traceObject(value: unknown): value is Record { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + if (prototype !== Object.prototype && prototype !== null) return false; + return Reflect.ownKeys(value).every((key) => { + if (typeof key !== 'string') return false; + const property = Object.getOwnPropertyDescriptor(value, key); + return property?.enumerable === true && traceOwn(property, 'value'); + }); +} + +/** Enforces an explicit object allowlist, including its required keys. */ +export function traceKeys( + value: Record, + required: readonly string[], + optional: readonly string[] = [] +): boolean { + return ( + required.every((key) => traceOwn(value, key)) && + Object.keys(value).every((key) => required.includes(key) || optional.includes(key)) + ); +} + +/** Validates Unicode and printable text before measuring its UTF-8 byte bound. */ +export function traceText(value: unknown, limit = 128): value is string { + if (typeof value !== 'string') return false; + for (const character of value) { + const code = character.codePointAt(0)!; + if ( + code <= 0x1f || + (code >= 0x7f && code <= 0x9f) || + (code >= 0xd800 && code <= 0xdfff) || + code === 0x061c || + code === 0x200e || + code === 0x200f || + (code >= 0x202a && code <= 0x202e) || + (code >= 0x2066 && code <= 0x2069) + ) + return false; + } + return new TextEncoder().encode(value).length <= limit; +} + +/** Requires an actual calendar date and UTC RFC 3339 serialization. */ +export function traceTimestamp(value: unknown): value is string { + if (typeof value !== 'string') return false; + const match = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,9})?Z$/.exec(value); + if (!match) return false; + const [, yearText, monthText, dayText, hourText, minuteText, secondText] = match; + const year = Number(yearText); + const month = Number(monthText); + const day = Number(dayText); + const leap = year % 4 === 0 && (year % 100 !== 0 || year % 400 === 0); + const days = [31, leap ? 29 : 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]; + return ( + month >= 1 && + month <= 12 && + day >= 1 && + day <= days[month - 1] && + Number(hourText) <= 23 && + Number(minuteText) <= 59 && + Number(secondText) <= 59 + ); +} + +/** Validates a finite nonnegative counter within its explicit upper bound. */ +export function traceInteger(value: unknown, maximum = Number.MAX_SAFE_INTEGER): value is number { + return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 && value <= maximum; +} + +function maskedAddress(value: unknown): boolean { + if (!traceText(value)) return false; + const ipv4 = /^((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.0\/24$/.exec( + value + ); + if (ipv4) return ipv4.slice(1).every((octet) => Number(octet) <= 255); + if (!value.endsWith('::/48')) return false; + const address = value.slice(0, -3); + const prefix = address.slice(0, -2); + if (prefix !== '' && !/^[\da-f]{1,4}(?::[\da-f]{1,4}){0,2}$/.test(prefix)) return false; + try { + return new URL(`http://[${address}]/`).hostname === `[${address}]`; + } catch { + return false; + } +} + +function cookieHealth(value: unknown, validDetail: string): boolean { + if (!traceObject(value) || value.source !== 'request') return false; + if (value.state === 'absent') return traceKeys(value, ['source', 'state']); + if (!traceKeys(value, ['source', 'state', 'detail']) || typeof value.detail !== 'string') + return false; + switch (value.state) { + case 'present_valid': + return value.detail === validDetail; + case 'present_invalid': + return ['malformed', 'oversized', 'unsupported_value'].includes(value.detail); + case 'duplicate': + return value.detail === 'multiple_values'; + case 'unavailable': + return ['header_too_large', 'header_not_utf8', 'runtime_header_ambiguous'].includes( + value.detail + ); + default: + return false; + } +} + +/** Validates the exact request-context schema, including unavailable cookie facts. */ +export function validateTraceRequestContext(value: unknown): value is TraceRequestContextV1 { + try { + return requestContext(value); + } catch { + return false; + } +} + +function requestContext(value: unknown): value is TraceRequestContextV1 { + if ( + !traceObject(value) || + !traceKeys(value, ['schema_version', 'captured_at', 'network', 'cookies']) || + value.schema_version !== 1 || + !traceTimestamp(value.captured_at) || + !traceObject(value.network) || + !traceObject(value.cookies) + ) + return false; + const network = value.network; + const bounds = { + country: 2, + region: 32, + http_version: 32, + tls_protocol: 32, + tls_cipher: 32, + edge_hostname: 128, + edge_region: 128, + edge_pop: 32, + } as const; + if (!traceKeys(network, [], ['masked_client_ip', 'asn', ...Object.keys(bounds)])) return false; + if (traceOwn(network, 'masked_client_ip') && !maskedAddress(network.masked_client_ip)) + return false; + if (traceOwn(network, 'asn') && !traceInteger(network.asn, 0xffff_ffff)) return false; + for (const [key, bound] of Object.entries(bounds)) { + if (traceOwn(network, key) && !traceText(network[key], bound)) return false; + } + if (typeof network.country === 'string' && !/^[\x20-\x7e]{0,2}$/.test(network.country)) + return false; + const cookies = value.cookies; + return ( + traceKeys(cookies, ['ts_ec', 'ts_eids', 'ts_tester', 'diagnostics_session']) && + cookieHealth(cookies.ts_ec, 'valid_ec_format') && + cookieHealth(cookies.ts_eids, 'valid_eids_format') && + cookieHealth(cookies.ts_tester, 'valid_tester_value') && + cookieHealth(cookies.diagnostics_session, 'valid_diagnostics_value') + ); +} diff --git a/crates/trusted-server-js/lib/src/trace/correlation.ts b/crates/trusted-server-js/lib/src/trace/correlation.ts new file mode 100644 index 000000000..c8ecec206 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/correlation.ts @@ -0,0 +1,121 @@ +import type { TraceGptRequestCycle } from './report-types'; +import { parseTraceReport } from './report-validation'; +import type { TraceAuctionEvidenceV1, TraceAuctionSlot } from './types'; + +/** Server candidate facts remain independent when the exact browser join is unknown. */ +export interface TraceSlotEvidenceView { + readonly serverSlot: TraceAuctionSlot; + readonly correlation: 'matched' | 'unknown'; + readonly runtimeSlotNumber?: number; + readonly requestNumber?: number; + readonly cycle?: TraceGptRequestCycle; + readonly pathLabel?: string; + readonly creativeLabel?: string; +} +export interface TraceAuctionEvidenceView { + readonly evidence: TraceAuctionEvidenceV1; + readonly sourceLabel: string; + readonly relativeMilestonesLabel: 'Unavailable in v1'; + readonly providerScopeLabel: 'Auction-wide provider status; per-slot no-bid reason unavailable'; + readonly slots: readonly TraceSlotEvidenceView[]; +} +export interface TraceEvidenceView { + readonly auctions: readonly TraceAuctionEvidenceView[]; +} +function pathLabel(cycle: TraceGptRequestCycle): string { + switch (cycle.requestPath) { + case 'prebid_refresh': + case 'publisher_refresh': + return 'Browser refresh observed; winner not determined'; + case 'competing': + case 'unattributed': + return 'Multiple or unknown delivery paths'; + case 'trusted_server_direct': + return 'Trusted Server request path observed'; + default: + return 'Request path unavailable'; + } +} +const SOURCES = { + initial_navigation_ssat: 'Initial-page server auction (SSAT)', + spa_page_bids: 'Trusted Server page-refresh auction', + auction_api: 'Trusted Server auction API', +} as const; + +/** Joins only unique token pairs and exact exported GPT request-cycle identities. */ +export function joinTraceEvidence( + value: unknown, + origin: string, + capturedAtMs: number +): TraceEvidenceView | undefined { + const report = parseTraceReport(value, origin, capturedAtMs); + if (!report) return undefined; + const cycles = report.gpt_diagnostics.slots.flatMap((slot) => + slot.requests.map((cycle) => ({ runtimeSlotNumber: slot.runtimeSlotNumber, cycle })) + ); + const serverSlots = report.server_auctions.flatMap((auction) => + auction.slots.map((slot) => ({ id: auction.diagnostic_auction_id, slot })) + ); + const auctions = report.server_auctions.map((auction) => { + const id = auction.diagnostic_auction_id; + const uniqueAuction = + report.server_auctions.filter((record) => record.diagnostic_auction_id === id).length === 1; + const slots = auction.slots.map((slot): TraceSlotEvidenceView => { + const unknown: TraceSlotEvidenceView = Object.freeze({ + serverSlot: slot, + correlation: 'unknown', + }); + if (!uniqueAuction || auction.source === 'auction_api') return unknown; + if ( + serverSlots.filter((record) => record.id === id && record.slot.slot_ref === slot.slot_ref) + .length !== 1 + ) + return unknown; + const sidecars = report.slot_correlations.filter( + (sidecar) => sidecar.diagnostic_auction_id === id && sidecar.slot_ref === slot.slot_ref + ); + if (sidecars.length !== 1) return unknown; + const sidecar = sidecars[0]; + if ( + report.slot_correlations.filter( + (record) => + record.runtime_slot_number === sidecar.runtime_slot_number && + record.request_number === sidecar.request_number + ).length !== 1 + ) + return unknown; + const matches = cycles.filter( + (record) => + record.runtimeSlotNumber === sidecar.runtime_slot_number && + record.cycle.requestNumber === sidecar.request_number + ); + if (matches.length !== 1 || matches[0].cycle.trustedServerAuctionId !== id) return unknown; + const cycle = matches[0].cycle; + const participated = + cycle.isEmpty === false && + cycle.renderAtMs !== undefined && + cycle.trustedServerCreativeResponseAtMs !== undefined && + cycle.delivery === 'trusted_server_response_sent'; + return Object.freeze({ + serverSlot: slot, + correlation: 'matched', + runtimeSlotNumber: sidecar.runtime_slot_number, + requestNumber: sidecar.request_number, + cycle, + pathLabel: pathLabel(cycle), + creativeLabel: participated + ? 'Trusted Server creative rendered' + : 'Participation unconfirmed', + }); + }); + return Object.freeze({ + evidence: auction, + sourceLabel: SOURCES[auction.source], + relativeMilestonesLabel: 'Unavailable in v1' as const, + providerScopeLabel: + 'Auction-wide provider status; per-slot no-bid reason unavailable' as const, + slots: Object.freeze(slots), + }); + }); + return Object.freeze({ auctions: Object.freeze(auctions) }); +} diff --git a/crates/trusted-server-js/lib/src/trace/export.ts b/crates/trusted-server-js/lib/src/trace/export.ts new file mode 100644 index 000000000..d46972517 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/export.ts @@ -0,0 +1,87 @@ +import { parseTraceReport } from './report-validation'; + +/** One deterministic file name for every trace export surface. */ +export const TRACE_REPORT_FILENAME = 'trusted-server-trace-v1.json'; +type TraceExportFailure = { readonly status: 'failed' | 'unsupported' | 'invalid_report' }; + +/** Formats the validated public report identically for Copy, Share, and Download. */ +export function formatTraceReport( + value: unknown, + origin: string, + capturedAtMs: number +): string | undefined { + const report = parseTraceReport(value, origin, capturedAtMs); + return report ? JSON.stringify(report, null, 2) : undefined; +} + +/** Starts one explicit download and independently defers its object URL cleanup. */ +export function downloadTraceReport( + value: unknown, + origin: string, + capturedAtMs: number +): { readonly status: 'downloaded' } | TraceExportFailure { + const json = formatTraceReport(value, origin, capturedAtMs); + if (json === undefined) return { status: 'invalid_report' }; + let url: string; + try { + url = URL.createObjectURL(new Blob([json], { type: 'application/json' })); + } catch { + return { status: 'failed' }; + } + let anchor: HTMLAnchorElement | undefined; + try { + anchor = document.createElement('a'); + anchor.href = url; + anchor.download = TRACE_REPORT_FILENAME; + document.body.append(anchor); + anchor.click(); + return { status: 'downloaded' }; + } catch { + return { status: 'failed' }; + } finally { + try { + anchor?.remove(); + } catch { + /* URL cleanup still runs when DOM removal is unavailable. */ + } + window.setTimeout(() => URL.revokeObjectURL(url), 1000); + } +} + +/** Copies validated report JSON only when the user invokes this action. */ +export async function copyTraceReport( + value: unknown, + origin: string, + capturedAtMs: number +): Promise<{ readonly status: 'copied' } | TraceExportFailure> { + const json = formatTraceReport(value, origin, capturedAtMs); + if (json === undefined) return { status: 'invalid_report' }; + try { + if (typeof navigator.clipboard?.writeText !== 'function') return { status: 'unsupported' }; + await navigator.clipboard.writeText(json); + return { status: 'copied' }; + } catch { + return { status: 'failed' }; + } +} + +/** Shares the same validated JSON file when file sharing is supported. */ +export async function shareTraceReport( + value: unknown, + origin: string, + capturedAtMs: number +): Promise<{ readonly status: 'shared' } | TraceExportFailure> { + const json = formatTraceReport(value, origin, capturedAtMs); + if (json === undefined) return { status: 'invalid_report' }; + try { + if (typeof navigator.canShare !== 'function' || typeof navigator.share !== 'function') + return { status: 'unsupported' }; + const file = new File([json], TRACE_REPORT_FILENAME, { type: 'application/json' }); + const data: ShareData = { files: [file] }; + if (!navigator.canShare(data)) return { status: 'unsupported' }; + await navigator.share(data); + return { status: 'shared' }; + } catch { + return { status: 'failed' }; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/gpt.ts b/crates/trusted-server-js/lib/src/trace/gpt.ts new file mode 100644 index 000000000..a2e15be8b --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/gpt.ts @@ -0,0 +1,138 @@ +import type { AuctionSlot } from '../core/types'; + +import type { TraceCollector } from './collector'; +import type { TraceGptIdentity } from './types'; +import { traceItems } from './shape'; +import { + parseTraceAuctionTransport, + validDiagnosticAuctionId, + validTraceSlotRef, +} from './validation'; + +/** Ordinary marker plus the optional exact identity for one concrete GPT opportunity. */ +export interface TraceGptOpportunity { + readonly auctionId: string | undefined; + readonly identity?: TraceGptIdentity; +} + +/** Narrow internal facade for the shared validated SSAT/SPA slot bindings. */ +export interface TraceGptBridge { + observeTransport( + slots: readonly AuctionSlot[] | undefined, + value: unknown, + source: 'initial_navigation_ssat' | 'spa_page_bids' + ): void; + observePageBids(response: unknown, slots: readonly AuctionSlot[]): void; + identity(slot: AuctionSlot): TraceGptIdentity | undefined; + opportunity(slot: AuctionSlot, auctionId: string | undefined): TraceGptOpportunity; +} + +function ownData(value: unknown, key: string): unknown { + if (value === null || typeof value !== 'object') return undefined; + const property = Object.getOwnPropertyDescriptor(value, key); + return property && Object.prototype.hasOwnProperty.call(property, 'value') + ? property.value + : undefined; +} + +/** Creates the active core bridge shared by bootstrap and independent GPT bundles. */ +export function createTraceGptBridge(collector: TraceCollector): TraceGptBridge { + let identities: WeakMap | undefined; + function observeTraceGptTransport( + slots: readonly AuctionSlot[] | undefined, + value: unknown, + source: 'initial_navigation_ssat' | 'spa_page_bids' + ): void { + try { + // A new accepted document/navigation batch replaces slot bindings only. + // Previously consumed store intents and accumulated capture stay intact. + identities = undefined; + if (value === undefined) return; + const transport = parseTraceAuctionTransport(value); + if (!transport || (transport.evidence && transport.evidence.source !== source)) { + collector.recordTransport(null); + return; + } + collector.recordTransport(transport); + if (!transport.evidence) return; + // Correlation failures cannot discard already validated server evidence. + // Read only own array entries and the sole allowed extension member. + let delivered: unknown[] | undefined; + try { + delivered = slots === undefined ? [] : traceItems(slots, 64); + } catch { + delivered = undefined; + } + if (!delivered) { + collector.recordInterpretationIssue('correlation_unavailable'); + return; + } + const evidenceCounts = new Map(); + for (const slot of transport.evidence.slots) + evidenceCounts.set(slot.slot_ref, (evidenceCounts.get(slot.slot_ref) ?? 0) + 1); + const deliveredCounts = new Map(); + const refs = delivered.map((slot) => { + try { + const ref = ownData(ownData(ownData(slot, 'ext'), 'trusted_server'), 'trace_slot_ref'); + if (!validTraceSlotRef(ref)) return undefined; + deliveredCounts.set(ref, (deliveredCounts.get(ref) ?? 0) + 1); + return ref; + } catch { + return undefined; + } + }); + for (const [index, slot] of delivered.entries()) { + const ref = refs[index]; + if (!ref || deliveredCounts.get(ref) !== 1 || evidenceCounts.get(ref) !== 1) { + collector.recordInterpretationIssue('correlation_unavailable'); + continue; + } + identities ??= new WeakMap(); + identities.set( + slot as object, + Object.freeze({ + diagnostic_auction_id: transport.evidence.diagnostic_auction_id, + slot_ref: ref, + }) + ); + } + } catch { + // Diagnostic parsing never changes ordinary slot or bid delivery. + } + } + + /** Reads the sole optional SPA member after the navigation's existing guards. */ + function observeTraceGptPageBids(response: unknown, slots: readonly AuctionSlot[]): void { + if (response === null || typeof response !== 'object') return; + let value: unknown; + try { + const property = Object.getOwnPropertyDescriptor(response, 'trace_auction'); + value = + property && !Object.prototype.hasOwnProperty.call(property, 'value') + ? null + : property?.value; + } catch { + value = null; + } + observeTraceGptTransport(slots, value, 'spa_page_bids'); + } + + /** Returns only the validated identity bound to this exact delivered slot object. */ + return Object.freeze({ + observeTransport: observeTraceGptTransport, + observePageBids: observeTraceGptPageBids, + identity: (slot: AuctionSlot) => identities?.get(slot), + opportunity: (slot: AuctionSlot, auctionId: string | undefined): TraceGptOpportunity => { + const identity = identities?.get(slot); + if (!identity) return { auctionId }; + // Only an absent marker can inherit the validated pre-dispatch token. + // Provided markers keep the store's existing normalization and semantics. + if (auctionId === undefined) return { auctionId: identity.diagnostic_auction_id, identity }; + const normalized = typeof auctionId === 'string' ? auctionId.trim() : undefined; + if (validDiagnosticAuctionId(normalized) && normalized === identity.diagnostic_auction_id) + return { auctionId, identity }; + collector.recordInterpretationIssue('correlation_unavailable'); + return { auctionId }; + }, + }); +} diff --git a/crates/trusted-server-js/lib/src/trace/handoff.ts b/crates/trusted-server-js/lib/src/trace/handoff.ts new file mode 100644 index 000000000..15fe60f65 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/handoff.ts @@ -0,0 +1,144 @@ +import type { TsjsApi } from '../core/types'; + +import { downloadTraceReport } from './export'; +import { buildTraceReport } from './report'; +import type { TraceStoredReportV1 } from './report-types'; +import { storeTraceReport } from './storage'; + +export interface TraceHandoffState { + readonly kind: + | 'ready' + | 'capturing' + | 'capture_failed' + | 'storage_unavailable' + | 'navigation_unavailable' + | 'navigating' + | 'download_failed' + | 'downloaded'; + readonly downloadAvailable: boolean; +} +export interface TraceHandoffOptions { + readonly target: { + __tsjs_trace_active?: unknown; + __tsjs_trace_request_context?: unknown; + tsjs?: TsjsApi; + location: Pick; + sessionStorage: Pick; + }; + readonly now?: () => number; + readonly onChange?: (state: TraceHandoffState) => void; + readonly download?: typeof downloadTraceReport; +} +export interface TraceHandoff { + view(): void; + download(): void; + destroy(): void; +} + +/** Captures one explicit combined report and navigates only after its validated save. */ +export function createTraceHandoff(options: TraceHandoffOptions): TraceHandoff | undefined { + try { + if (options.target.__tsjs_trace_active !== true) return undefined; + } catch { + return undefined; + } + let destroyed = false; + let busy = false; + let recovery: TraceStoredReportV1 | undefined; + let recoveryOrigin: string | undefined; + const notify = (kind: TraceHandoffState['kind']): void => { + if (destroyed) return; + try { + options.onChange?.(Object.freeze({ kind, downloadAvailable: recovery !== undefined })); + } catch { + /* UI callbacks cannot interrupt the capture boundary. */ + } + }; + const action: TraceHandoff = { + view() { + if (destroyed || busy) return; + busy = true; + recovery = undefined; + recoveryOrigin = undefined; + notify('capturing'); + try { + if (destroyed) return; + if (options.target.__tsjs_trace_active !== true) { + notify('capture_failed'); + return; + } + const clock = (options.now ?? Date.now)(); + const origin = options.target.location.origin; + const api = options.target.tsjs; + if (typeof api?.gptDiagnostics?.snapshot !== 'function' || !api.traceEvidence) { + notify('capture_failed'); + return; + } + const collector = api.traceEvidence.snapshot(); + if (destroyed) return; + if (!collector.ok) { + notify('capture_failed'); + return; + } + const result = buildTraceReport({ + capturedAtMs: clock, + origin, + requestContext: options.target.__tsjs_trace_request_context, + gptSource: api.gptDiagnostics.snapshot(), + collector: collector.value, + }); + if (destroyed) return; + if (!result.ok) { + notify('capture_failed'); + return; + } + recovery = result.value; + recoveryOrigin = origin; + let stored = false; + try { + stored = + storeTraceReport(recovery, origin, clock, options.target.sessionStorage).status === + 'stored'; + } catch { + /* Unavailable storage preserves this valid combined recovery report. */ + } + if (destroyed) return; + if (!stored) { + notify('storage_unavailable'); + return; + } + try { + options.target.location.assign('/_ts/trace'); + notify('navigating'); + } catch { + notify('navigation_unavailable'); + } + } catch { + recovery = undefined; + recoveryOrigin = undefined; + notify('capture_failed'); + } finally { + busy = false; + } + }, + download() { + if (destroyed || busy || !recovery || !recoveryOrigin) return; + try { + const result = (options.download ?? downloadTraceReport)( + recovery.report, + recoveryOrigin, + recovery.stored_at_ms + ); + notify(result.status === 'downloaded' ? 'downloaded' : 'download_failed'); + } catch { + notify('download_failed'); + } + }, + destroy() { + destroyed = true; + recovery = undefined; + recoveryOrigin = undefined; + }, + }; + return Object.freeze(action); +} diff --git a/crates/trusted-server-js/lib/src/trace/json.ts b/crates/trusted-server-js/lib/src/trace/json.ts new file mode 100644 index 000000000..6b876def2 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/json.ts @@ -0,0 +1,92 @@ +import { traceObject, traceOwn, traceText } from './context'; +import { traceItems } from './shape'; +const MAX_STORED_BYTES = 512 * 1024; + +/** Measures compact JSON using own data descriptors, never caller serialization. */ +export function boundedJsonShape( + value: unknown, + maximumDepth = 10, + maximumBytes = MAX_STORED_BYTES +): number | undefined { + return traceJsonSnapshot(value, maximumDepth, maximumBytes)?.bytes; +} + +/** Copies and measures the same bounded own JSON data, without caller get traps. */ +export function traceJsonSnapshot( + value: unknown, + maximumDepth = 10, + maximumBytes = MAX_STORED_BYTES +): { value: unknown; bytes: number } | undefined { + const encoder = new TextEncoder(); + let bytes = 0; + const ancestors = new Set(); + const add = (text: string): boolean => { + bytes += encoder.encode(text).length; + return bytes <= maximumBytes; + }; + const walk = (item: unknown, depth: number): { value: unknown } | undefined => { + if (item === null || typeof item === 'boolean' || typeof item === 'number') + return (typeof item !== 'number' || Number.isFinite(item)) && add(JSON.stringify(item)) + ? { value: item } + : undefined; + if (typeof item === 'string') + return traceText(item, maximumBytes) && add(JSON.stringify(item)) + ? { value: item } + : undefined; + if (depth > maximumDepth || typeof item !== 'object' || ancestors.has(item)) return undefined; + ancestors.add(item); + let okay = true; + let owned: unknown; + if (Array.isArray(item)) { + const array: unknown[] = []; + owned = array; + const items = traceItems(item, maximumBytes); + if (!items || !add('[')) okay = false; + else { + for (let index = 0; index < items.length && okay; index += 1) { + if (index !== 0 && !add(',')) { + okay = false; + break; + } + const child = walk(items[index], depth + 1); + if (!child) { + okay = false; + break; + } + array.push(child.value); + } + } + okay = okay && add(']'); + } else if (traceObject(item)) { + const object: Record = Object.create(null); + owned = object; + if (!add('{')) okay = false; + const keys = Object.keys(item); + for (let index = 0; index < keys.length && okay; index += 1) { + const key = keys[index]; + const property = Object.getOwnPropertyDescriptor(item, key); + okay = + traceText(key, 128) && + (index === 0 || add(',')) && + add(JSON.stringify(key)) && + add(':') && + property !== undefined && + traceOwn(property, 'value'); + if (okay && property) { + const child = walk(property.value, depth + 1); + if (!child) okay = false; + else object[key] = child.value; + } + } + okay = okay && add('}'); + } else okay = false; + ancestors.delete(item); + return okay ? { value: owned } : undefined; + }; + try { + const owned = walk(value, 1); + return owned ? { value: owned.value, bytes } : undefined; + } catch { + return undefined; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/lifecycle.ts b/crates/trusted-server-js/lib/src/trace/lifecycle.ts new file mode 100644 index 000000000..e56c4bdac --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/lifecycle.ts @@ -0,0 +1,77 @@ +/** Deliberate server action requested by the trace setup and cleanup controls. */ +export type TraceSessionAction = 'enable' | 'end'; + +/** Separates requested mutation from what the follow-up request observed. */ +export type TraceSessionChangeResult = Readonly<{ + mutation: 'requested' | 'failed'; + observation: 'active' | 'inactive' | 'failed' | 'not_attempted'; + confirmed: boolean; +}>; + +/** Requests an explicit cookie change and verifies the resulting request state. */ +export async function changeTraceSession( + action: TraceSessionAction +): Promise { + return requestAndObserve(action, false); +} + +/** Attempts end and a separate state observation even when the end request fails. */ +export async function endTraceSessionAndObserve(): Promise { + return requestAndObserve('end', true); +} + +async function requestAndObserve( + action: TraceSessionAction, + observeFailedMutation: boolean +): Promise { + const failed: TraceSessionChangeResult = { + mutation: 'failed', + observation: 'not_attempted', + confirmed: false, + }; + if (action !== 'enable' && action !== 'end') return failed; + + let mutation: TraceSessionChangeResult['mutation'] = 'failed'; + try { + const response = await fetch(`/_ts/trace/${action}`, { + method: 'POST', + credentials: 'same-origin', + cache: 'no-store', + headers: { 'X-TS-Trace-Action': action }, + }); + if (response.ok) mutation = 'requested'; + } catch { + /* The cleanup caller still attempts its independent state observation. */ + } + if (mutation === 'failed' && !observeFailedMutation) return failed; + + const unconfirmed: TraceSessionChangeResult = { + mutation, + observation: 'failed', + confirmed: false, + }; + try { + const response = await fetch('/_ts/trace/state', { + method: 'GET', + credentials: 'same-origin', + cache: 'no-store', + }); + if (!response.ok) return unconfirmed; + const state: unknown = await response.json(); + if (state === null || typeof state !== 'object' || Array.isArray(state)) { + return unconfirmed; + } + const keys = Object.keys(state); + if (keys.length !== 1 || keys[0] !== 'observed_active') return unconfirmed; + const observed = Object.getOwnPropertyDescriptor(state, 'observed_active'); + if (!observed || typeof observed.value !== 'boolean') return unconfirmed; + const active: boolean = observed.value; + return { + mutation, + observation: active ? 'active' : 'inactive', + confirmed: mutation === 'requested' && active === (action === 'enable'), + }; + } catch { + return unconfirmed; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/omissions.ts b/crates/trusted-server-js/lib/src/trace/omissions.ts new file mode 100644 index 000000000..d42c297e1 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/omissions.ts @@ -0,0 +1,27 @@ +const omissionFailures = new WeakSet(); + +export function omissionOverflow(): never { + const error = new Error('omission_counter_overflow'); + omissionFailures.add(error); + throw error; +} + +/** Adds exact omissions without wrapping or saturating the public u16 counter. */ +export function addOmissions(current: number, added: number): number { + if ( + !Number.isInteger(current) || + !Number.isInteger(added) || + current < 0 || + added < 0 || + current > 65535 || + added > 65535 - current + ) { + omissionOverflow(); + } + return current + added; +} + +/** Recognizes only locally created overflow failures without inspecting caller errors. */ +export function isOmissionOverflow(value: unknown): boolean { + return typeof value === 'object' && value !== null && omissionFailures.has(value); +} diff --git a/crates/trusted-server-js/lib/src/trace/pending.ts b/crates/trusted-server-js/lib/src/trace/pending.ts new file mode 100644 index 000000000..a346cada5 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/pending.ts @@ -0,0 +1,242 @@ +import type { TraceCollector } from './collector'; +import { observeTraceApiResponse, observeTraceApiFailure } from './runtime'; + +export interface TracePendingCarry { + readonly collector: TraceCollector; + readonly slotRefs: readonly string[]; +} +export interface TraceBidBinding { + readonly bidId: string; + readonly bidderRequestId?: string; + readonly slotRef?: string; +} +export interface TracePendingHandle { + readonly key: symbol; +} +export interface TracePendingOptions { + now?: () => number; + timeout?: () => unknown; + schedule?: (callback: () => void, delay: number) => ReturnType; + cancel?: (timer: ReturnType) => void; +} +export interface TracePending { + /** Undefined bindings retain exact responses while declining uninspectable ID hooks. */ + add( + bindings: readonly TraceBidBinding[] | undefined, + carry: TracePendingCarry + ): TracePendingHandle | undefined; + response(handle: TracePendingHandle | undefined, body: unknown): void; + failure(bids: readonly Pick[]): void; + clear(): void; + destroy(): void; +} +const MAX_TIMEOUT = 2 ** 31 - 1 - 5000; +const MAX_PENDING = 128; +const MAX_BINDINGS = 2048; + +function hookKey(bid: Pick): string | undefined { + const requestId = bid.bidderRequestId; + const id = bid.bidId; + if ( + typeof requestId !== 'string' || + !requestId.length || + requestId.length > 128 || + typeof id !== 'string' || + !id.length || + id.length > 128 + ) + return undefined; + // Length framing keeps the two SDK identities distinct without delimiter assumptions. + return `${requestId.length}:${requestId}${id}`; +} + +/** Checks the captured browser timeout and wall-clock expiry before retaining state. */ +export function pendingExpiry(createdAtMs: number, configuredTimeout: unknown): number | undefined { + if (!Number.isSafeInteger(createdAtMs) || createdAtMs < 0) return undefined; + const timeout = + typeof configuredTimeout === 'number' && + Number.isInteger(configuredTimeout) && + configuredTimeout >= 0 && + configuredTimeout <= MAX_TIMEOUT + ? configuredTimeout + : 3000; + const expiresAtMs = createdAtMs + timeout + 5000; + return Number.isSafeInteger(expiresAtMs) ? expiresAtMs : undefined; +} +interface PendingRecord { + readonly bindings: readonly TraceBidBinding[]; + readonly carry: TracePendingCarry; + readonly expiresAtMs: number; + readonly timer: ReturnType; + readonly unknownIds: boolean; + hookEligible: boolean; +} + +/** Owns bounded request-local tokens and retires markers before diagnostic callbacks. */ +export function createTracePending(options: TracePendingOptions = {}): TracePending { + const now = options.now ?? (() => Date.now()); + const schedule = options.schedule ?? ((callback, delay) => setTimeout(callback, delay)); + const cancel = options.cancel ?? ((timer) => clearTimeout(timer)); + const records = new Map(); + const originalIds = new Map>(); + let destroyed = false; + function remove(handle: TracePendingHandle | undefined): PendingRecord | undefined { + if (!handle) return undefined; + const record = records.get(handle); + if (!record) return undefined; + records.delete(handle); + for (const binding of record.bindings) { + const key = hookKey(binding)!; + const owners = originalIds.get(key); + owners?.delete(handle); + if (!owners?.size) originalIds.delete(key); + } + try { + cancel(record.timer); + } catch { + /* Marker retirement precedes timer cleanup. */ + } + return record; + } + function take(handle: TracePendingHandle | undefined): TracePendingCarry | undefined { + const record = remove(handle); + if (!record) return undefined; + try { + const current = now(); + return Number.isSafeInteger(current) && current >= 0 && current < record.expiresAtMs + ? record.carry + : undefined; + } catch { + return undefined; + } + } + function clear(): void { + for (const handle of records.keys()) remove(handle); + } + function noteUnavailable(record: PendingRecord): void { + try { + record.carry.collector.recordInterpretationIssue('correlation_unavailable'); + } catch { + /* A diagnostic callback cannot alter request delivery. */ + } + } + return Object.freeze({ + add(bindings: readonly TraceBidBinding[] | undefined, carry: TracePendingCarry) { + if (destroyed) return undefined; + try { + const createdAtMs = now(); + let configuredTimeout: unknown; + try { + configuredTimeout = options.timeout?.(); + } catch { + /* Use the pinned default. */ + } + const expiresAtMs = pendingExpiry(createdAtMs, configuredTimeout); + if (expiresAtMs === undefined) return undefined; + // The ID index is bounded independently of exact response ownership. + // Unknown IDs disable hook attribution while their response handle lives. + let unknownIds = bindings === undefined || bindings.length > MAX_BINDINGS; + let repeatedId = false; + const ids = new Set(); + const ownedBindings: TraceBidBinding[] = []; + const collisions = new Set(); + const refs: string[] = []; + for (let index = 0; index < Math.min(64, carry.slotRefs.length); index += 1) + refs.push(carry.slotRefs[index]!); + const acceptedRefs = new Set(refs); + if (!unknownIds && bindings) { + for (const binding of bindings) { + const id = hookKey(binding); + if (id === undefined) { + unknownIds = true; + continue; + } + for (const owner of originalIds.get(id) ?? []) collisions.add(owner); + if (ids.has(id)) { + repeatedId = true; + continue; + } + ids.add(id); + ownedBindings.push( + Object.freeze({ + bidId: binding.bidId, + bidderRequestId: binding.bidderRequestId, + ...(binding.slotRef !== undefined && acceptedRefs.has(binding.slotRef) + ? { slotRef: binding.slotRef } + : {}), + }) + ); + } + } + const affected = new Set(); + let liveUnknown = false; + for (const [handle, record] of records) { + liveUnknown ||= record.unknownIds; + if (unknownIds || collisions.has(handle)) { + record.hookEligible = false; + affected.add(record); + } + } + const handle = Object.freeze({ key: Symbol() }); + const ownedCarry = Object.freeze({ + collector: carry.collector, + slotRefs: Object.freeze(refs), + }); + const timer = schedule(() => { + remove(handle); + }, expiresAtMs - createdAtMs); + const record: PendingRecord = { + bindings: Object.freeze(ownedBindings), + carry: ownedCarry, + expiresAtMs, + timer, + unknownIds, + hookEligible: + !!ownedBindings.length && + !unknownIds && + !liveUnknown && + !repeatedId && + !collisions.size, + }; + records.set(handle, record); + for (const id of ids) { + const owners = originalIds.get(id) ?? new Set(); + owners.add(handle); + originalIds.set(id, owners); + } + if (records.size > MAX_PENDING) remove(records.keys().next().value); + if (!record.hookEligible) affected.add(record); + for (const unavailable of affected) noteUnavailable(unavailable); + return handle; + } catch { + return undefined; + } + }, + response(handle: TracePendingHandle | undefined, body: unknown) { + observeTraceApiResponse(take(handle), body); + }, + failure(bids: readonly Pick[]) { + try { + if (bids.length > MAX_BINDINGS) return; + const handles = new Set(); + for (const bid of bids) { + const id = hookKey(bid); + if (id === undefined) continue; + const owners = originalIds.get(id); + if (owners?.size === 1) { + const handle = owners.values().next().value; + if (handle && records.get(handle)?.hookEligible) handles.add(handle); + } + } + for (const handle of handles) observeTraceApiFailure(take(handle)); + } catch { + /* No raw callback input is retained or reported. */ + } + }, + clear, + destroy() { + destroyed = true; + clear(); + }, + }); +} diff --git a/crates/trusted-server-js/lib/src/trace/projection.ts b/crates/trusted-server-js/lib/src/trace/projection.ts new file mode 100644 index 000000000..862973d35 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/projection.ts @@ -0,0 +1,258 @@ +import { addOmissions, omissionOverflow, isOmissionOverflow } from './omissions'; +export { addOmissions } from './omissions'; +import { traceInteger, traceKeys, traceObject } from './context'; +import { + TRACE_CALLBACK_KINDS, + TRACE_CREATIVE_FAILURES, + type TraceGptDiagnosticsV1, +} from './report-types'; +import { TRACE_CYCLE_KEYS, traceOrigin, validateTraceGptDiagnostics } from './report-validation'; +import { traceEnum, traceItems, traceSize } from './validation'; + +export type TraceProjectionResult = + | { + readonly ok: true; + readonly value: TraceGptDiagnosticsV1; + readonly omittedNestedValues: number; + } + | { + readonly ok: false; + readonly reason: + | 'invalid_source' + | 'unsupported_source_version' + | 'omission_counter_overflow'; + }; + +function sourceObject( + value: unknown, + required: readonly string[], + optional: readonly string[] = [] +): Record { + if (!traceObject(value) || !traceKeys(value, required, optional)) + throw new Error('invalid_source'); + return value; +} +function data(source: Record, key: string): unknown { + return Object.getOwnPropertyDescriptor(source, key)?.value; +} +function copyOptional( + source: Record, + output: Record, + key: string +): void { + // The public API includes own optional undefined values while cycles are pending. + // Omit only those values, matching their absence in its JSON export. + const value = data(source, key); + if (value !== undefined) output[key] = value; +} +function sourceItems(value: unknown, limit: number): unknown[] { + const items = traceItems(value, limit); + if (!items) throw new Error('invalid_source'); + return items; +} +function nestedItems(value: unknown): unknown[] { + if (!Array.isArray(value)) throw new Error('invalid_source'); + const length = Object.getOwnPropertyDescriptor(value, 'length')?.value; + if (!traceInteger(length)) throw new Error('invalid_source'); + if (length > 65535 + 16) omissionOverflow(); + return sourceItems(value, 65535 + 16); +} +function copySize(value: unknown, zero = false): readonly [number, number] { + const items = sourceItems(value, 2); + if (!traceSize(items, zero)) throw new Error('invalid_source'); + return [items[0], items[1]]; +} +function durationProjection(value: unknown): Record { + const source = sourceObject( + value, + [], + [ + 'requestToResponseMs', + 'responseToRenderMs', + 'requestToRenderMs', + 'renderToLoadMs', + 'renderToViewableMs', + ] + ); + const output: Record = {}; + copyOptional(source, output, 'requestToResponseMs'); + copyOptional(source, output, 'responseToRenderMs'); + copyOptional(source, output, 'requestToRenderMs'); + copyOptional(source, output, 'renderToLoadMs'); + copyOptional(source, output, 'renderToViewableMs'); + return output; +} +function cycleProjection(value: unknown, omissions: { value: number }): Record { + const source = sourceObject( + value, + ['requestNumber', 'durations', 'incompleteSequence'], + [...TRACE_CYCLE_KEYS, 'adManager', 'previousCreativeId'] + ); + const output: Record = { + requestNumber: data(source, 'requestNumber'), + durations: durationProjection(data(source, 'durations')), + incompleteSequence: data(source, 'incompleteSequence'), + }; + copyOptional(source, output, 'requestedAtMs'); + copyOptional(source, output, 'responseAtMs'); + copyOptional(source, output, 'renderAtMs'); + copyOptional(source, output, 'loadAtMs'); + copyOptional(source, output, 'viewableAtMs'); + copyOptional(source, output, 'isEmpty'); + copyOptional(source, output, 'isBackfill'); + copyOptional(source, output, 'slotContentChanged'); + copyOptional(source, output, 'responseClass'); + copyOptional(source, output, 'requestPath'); + copyOptional(source, output, 'requestIntentId'); + copyOptional(source, output, 'trustedServerAuctionId'); + copyOptional(source, output, 'opportunityToRequestMs'); + copyOptional(source, output, 'replacedRequestNumber'); + copyOptional(source, output, 'previousRenderToRequestMs'); + copyOptional(source, output, 'creativeChanged'); + copyOptional(source, output, 'loadObservedBeforeRender'); + copyOptional(source, output, 'trustedServerOpportunity'); + copyOptional(source, output, 'trustedServerCreativeRequestAtMs'); + copyOptional(source, output, 'trustedServerCreativeResponseAtMs'); + copyOptional(source, output, 'delivery'); + if (data(source, 'size') !== undefined) output.size = copySize(data(source, 'size')); + if (data(source, 'observedSlotSize') !== undefined) + output.observedSlotSize = copySize(data(source, 'observedSlotSize'), true); + if (data(source, 'requestedSlotSizes') !== undefined) { + const values = nestedItems(data(source, 'requestedSlotSizes')); + if (!values.every((item) => traceSize(item))) throw new Error('invalid_source'); + output.requestedSlotSizes = values.slice(0, 16).map((item) => copySize(item)); + omissions.value = addOmissions(omissions.value, Math.max(0, values.length - 16)); + } + if (data(source, 'trustedServerCreativeFailures') !== undefined) { + const values = nestedItems(data(source, 'trustedServerCreativeFailures')); + if (!values.every((item) => traceEnum(item, TRACE_CREATIVE_FAILURES))) + throw new Error('invalid_source'); + output.trustedServerCreativeFailures = values.slice(0, 16); + omissions.value = addOmissions(omissions.value, Math.max(0, values.length - 16)); + } + return output; +} +function slotProjection(value: unknown, omissions: { value: number }): Record { + const source = sourceObject( + value, + ['runtimeSlotNumber', 'binding', 'requests'], + ['slotElementId', 'adUnitPath', 'currentVisibilityPercentage', 'maximumVisibilityPercentage'] + ); + const binding = sourceObject(data(source, 'binding'), ['status'], ['reason']); + const ownedBinding: Record = { status: data(binding, 'status') }; + copyOptional(binding, ownedBinding, 'reason'); + const output: Record = { + runtimeSlotNumber: data(source, 'runtimeSlotNumber'), + binding: ownedBinding, + requests: sourceItems(data(source, 'requests'), 10).map((item) => + cycleProjection(item, omissions) + ), + }; + copyOptional(source, output, 'currentVisibilityPercentage'); + copyOptional(source, output, 'maximumVisibilityPercentage'); + return output; +} +function callbackProjection(value: unknown): Record { + const source = sourceObject( + value, + ['kind', 'runtimeSlotNumber', 'timestampMs', 'disposition', 'reason'], + ['slotElementId'] + ); + return { + kind: data(source, 'kind'), + runtimeSlotNumber: data(source, 'runtimeSlotNumber'), + timestampMs: data(source, 'timestampMs'), + disposition: data(source, 'disposition'), + reason: data(source, 'reason'), + }; +} +function attributionProjection(value: unknown): Record { + const source = sourceObject( + value, + ['reason', 'timestampMs'], + ['runtimeSlotNumber', 'slotElementId'] + ); + const output: Record = { + reason: data(source, 'reason'), + timestampMs: data(source, 'timestampMs'), + }; + copyOptional(source, output, 'runtimeSlotNumber'); + return output; +} +function coverageProjection(value: unknown): Record { + const source = sourceObject(value, TRACE_CALLBACK_KINDS); + const output: Record = {}; + for (const kind of TRACE_CALLBACK_KINDS) { + const counters = sourceObject(data(source, kind), [ + 'observed', + 'matched', + 'unmatched', + 'ambiguous', + ]); + output[kind] = { + observed: data(counters, 'observed'), + matched: data(counters, 'matched'), + unmatched: data(counters, 'unmatched'), + ambiguous: data(counters, 'ambiguous'), + }; + } + return output; +} +function metadataProjection(value: unknown): Record { + const source = sourceObject( + value, + ['droppedCallbacks', 'evictedSlots', 'evictedRequestCycles'], + ['droppedAttributionIssues'] + ); + const output: Record = { + droppedCallbacks: data(source, 'droppedCallbacks'), + evictedSlots: data(source, 'evictedSlots'), + evictedRequestCycles: data(source, 'evictedRequestCycles'), + }; + copyOptional(source, output, 'droppedAttributionIssues'); + return output; +} +function freezeProjection(value: unknown): void { + if (typeof value !== 'object' || value === null) return; + Object.values(value).forEach(freezeProjection); + Object.freeze(value); +} +/** Copies only explicitly allowlisted TS Console facts into the public GPT model. */ +export function projectTraceGptDiagnostics(value: unknown, origin: string): TraceProjectionResult { + try { + const source = sourceObject( + value, + ['version', 'capturedAt', 'page', 'slots', 'callbackIssues', 'coverage', 'metadata'], + ['attributionIssues'] + ); + if (data(source, 'version') !== 1) return { ok: false, reason: 'unsupported_source_version' }; + const page = sourceObject(data(source, 'page'), ['origin', 'pathname']); + if (typeof data(page, 'pathname') !== 'string') throw new Error('invalid_source'); + const canonicalOrigin = traceOrigin(data(page, 'origin')); + if (canonicalOrigin !== origin || traceOrigin(origin) !== origin) + throw new Error('invalid_source'); + const omissions = { value: 0 }; + const projection: Record = { + schema_version: 1, + source_schema_version: 1, + capturedAt: data(source, 'capturedAt'), + page: { origin: canonicalOrigin, pathname: '/[redacted]' }, + slots: sourceItems(data(source, 'slots'), 64).map((item) => slotProjection(item, omissions)), + callbackIssues: sourceItems(data(source, 'callbackIssues'), 128).map(callbackProjection), + coverage: coverageProjection(data(source, 'coverage')), + metadata: metadataProjection(data(source, 'metadata')), + }; + if (data(source, 'attributionIssues') !== undefined) + projection.attributionIssues = sourceItems(data(source, 'attributionIssues'), 128).map( + attributionProjection + ); + if (!validateTraceGptDiagnostics(projection, origin)) throw new Error('invalid_source'); + freezeProjection(projection); + return { ok: true, value: projection, omittedNestedValues: omissions.value }; + } catch (error) { + return { + ok: false, + reason: isOmissionOverflow(error) ? 'omission_counter_overflow' : 'invalid_source', + }; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/report-types.ts b/crates/trusted-server-js/lib/src/trace/report-types.ts new file mode 100644 index 000000000..1032f2f51 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/report-types.ts @@ -0,0 +1,189 @@ +import type { + TraceAuctionEvidenceV1, + TraceRequestContextV1, + TraceSlotCorrelationV1, +} from './types'; + +export const TRACE_CALLBACK_KINDS = [ + 'slotRequested', + 'slotResponseReceived', + 'slotRenderEnded', + 'slotOnload', + 'impressionViewable', + 'slotVisibilityChanged', +] as const; +export const TRACE_BINDING_REASONS = [ + 'missing_slot_element_id', + 'missing_element', + 'duplicate_dom_id', + 'dom_uniqueness_unverifiable', + 'duplicate_gpt_slot_id', +] as const; +export const TRACE_CALLBACK_REASONS = [ + 'invalid_event_order', + 'missing_response_before_render', + 'invalid_visibility_percentage', + 'evicted_slot', + 'no_compatible_request_cycle', + 'overlapping_request_cycles', +] as const; +export const TRACE_ATTRIBUTION_REASONS = [ + 'creative_request_without_slot', + 'creative_request_without_cycle', + 'creative_request_ambiguous_cycle', + 'creative_request_on_empty_cycle', + 'creative_attempt_capacity', + 'creative_attempt_unknown', + 'creative_attempt_expired', + 'creative_attempt_evicted', +] as const; +export const TRACE_RESPONSE_CLASSES = [ + 'empty', + 'backfill', + 'reservation', + 'unclassified_non_empty', +] as const; +export const TRACE_REQUEST_PATHS = [ + 'trusted_server_direct', + 'prebid_refresh', + 'publisher_refresh', + 'competing', + 'unattributed', +] as const; +export const TRACE_OPPORTUNITIES = [ + 'renderable_candidate', + 'unrenderable_candidate', + 'no_candidate', +] as const; +export const TRACE_CREATIVE_FAILURES = [ + 'missing_render_source', + 'cache_fetch_failed', + 'invalid_cache_payload', + 'response_post_failed', +] as const; +export const TRACE_DELIVERIES = [ + 'trusted_server_response_sent', + 'trusted_server_selected', + 'candidate_unconfirmed', + 'no_candidate', + 'unknown', + 'pending', + 'not_applicable', +] as const; +export const TRACE_COVERAGE_ISSUES = [ + 'evidence_projection_failed', + 'evidence_transport_failed', + 'evidence_validation_failed', + 'record_evicted', + 'correlation_unavailable', + 'external_client_side_unobservable', +] as const; +export type TraceCoverageIssue = (typeof TRACE_COVERAGE_ISSUES)[number]; + +/** Public request cycle facts, with delivery identifiers deliberately excluded. */ +export interface TraceGptRequestCycle { + readonly requestNumber: number; + readonly requestedAtMs?: number; + readonly responseAtMs?: number; + readonly renderAtMs?: number; + readonly loadAtMs?: number; + readonly viewableAtMs?: number; + readonly durations: { + readonly requestToResponseMs?: number; + readonly responseToRenderMs?: number; + readonly requestToRenderMs?: number; + readonly renderToLoadMs?: number; + readonly renderToViewableMs?: number; + }; + readonly isEmpty?: boolean; + readonly requestedSlotSizes?: readonly (readonly [number, number])[]; + readonly size?: readonly [number, number]; + readonly observedSlotSize?: readonly [number, number]; + readonly isBackfill?: boolean; + readonly slotContentChanged?: boolean; + readonly incompleteSequence: boolean; + readonly responseClass?: (typeof TRACE_RESPONSE_CLASSES)[number]; + readonly requestPath?: (typeof TRACE_REQUEST_PATHS)[number]; + readonly requestIntentId?: number; + readonly trustedServerAuctionId?: string; + readonly opportunityToRequestMs?: number; + readonly replacedRequestNumber?: number; + readonly previousRenderToRequestMs?: number; + readonly creativeChanged?: boolean; + readonly loadObservedBeforeRender?: boolean; + readonly trustedServerOpportunity?: (typeof TRACE_OPPORTUNITIES)[number]; + readonly trustedServerCreativeRequestAtMs?: number; + readonly trustedServerCreativeResponseAtMs?: number; + readonly trustedServerCreativeFailures?: readonly (typeof TRACE_CREATIVE_FAILURES)[number][]; + readonly delivery?: (typeof TRACE_DELIVERIES)[number]; +} + +/** Trace-owned GPT snapshot, sourced from the explicitly supported TS Console version. */ +export interface TraceGptDiagnosticsV1 { + readonly schema_version: 1; + readonly source_schema_version: 1; + readonly capturedAt: string; + readonly page: { readonly origin: string; readonly pathname: '/[redacted]' }; + readonly slots: readonly { + readonly runtimeSlotNumber: number; + readonly binding: { + readonly status: 'bound' | 'unbound' | 'ambiguous'; + readonly reason?: (typeof TRACE_BINDING_REASONS)[number]; + }; + readonly currentVisibilityPercentage?: number; + readonly maximumVisibilityPercentage?: number; + readonly requests: readonly TraceGptRequestCycle[]; + }[]; + readonly callbackIssues: readonly { + readonly kind: (typeof TRACE_CALLBACK_KINDS)[number]; + readonly runtimeSlotNumber: number; + readonly timestampMs: number; + readonly disposition: 'matched' | 'unmatched' | 'ambiguous'; + readonly reason: (typeof TRACE_CALLBACK_REASONS)[number]; + }[]; + readonly attributionIssues?: readonly { + readonly reason: (typeof TRACE_ATTRIBUTION_REASONS)[number]; + readonly timestampMs: number; + readonly runtimeSlotNumber?: number; + }[]; + readonly coverage: Readonly< + Record< + (typeof TRACE_CALLBACK_KINDS)[number], + Readonly<{ observed: number; matched: number; unmatched: number; ambiguous: number }> + > + >; + readonly metadata: { + readonly droppedCallbacks: number; + readonly droppedAttributionIssues?: number; + readonly evictedSlots: number; + readonly evictedRequestCycles: number; + }; +} + +/** Combined immutable public report used by the viewer and every export. */ +export interface TraceReportV1 { + readonly schema_version: 1; + readonly captured_at: string; + readonly request_context: TraceRequestContextV1; + readonly server_auctions: readonly TraceAuctionEvidenceV1[]; + readonly slot_correlations: readonly TraceSlotCorrelationV1[]; + readonly gpt_diagnostics: TraceGptDiagnosticsV1; + readonly auction_coverage: { + readonly capture_status: 'complete' | 'partial' | 'unavailable' | 'not_observed'; + readonly issues: readonly TraceCoverageIssue[]; + }; + readonly truncation: { + readonly omitted_server_auctions: number; + readonly omitted_slot_correlations: number; + readonly omitted_request_cycles: number; + readonly omitted_callback_issues: number; + readonly omitted_attribution_issues: number; + readonly omitted_nested_values: number; + }; +} + +/** Exact origin-local storage envelope; its timestamp is independent of request time. */ +export interface TraceStoredReportV1 { + readonly stored_at_ms: number; + readonly report: TraceReportV1; +} diff --git a/crates/trusted-server-js/lib/src/trace/report-validation.ts b/crates/trusted-server-js/lib/src/trace/report-validation.ts new file mode 100644 index 000000000..f93ea9481 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/report-validation.ts @@ -0,0 +1,504 @@ +import { boundedJsonShape, traceJsonSnapshot } from './json'; +export { boundedJsonShape, traceJsonSnapshot } from './json'; +import { + traceInteger, + traceKeys, + traceObject, + traceOwn, + traceText, + traceTimestamp, + validateTraceRequestContext, +} from './context'; +import { + TRACE_ATTRIBUTION_REASONS, + TRACE_BINDING_REASONS, + TRACE_CALLBACK_KINDS, + TRACE_CALLBACK_REASONS, + TRACE_COVERAGE_ISSUES, + TRACE_CREATIVE_FAILURES, + TRACE_DELIVERIES, + TRACE_OPPORTUNITIES, + TRACE_REQUEST_PATHS, + TRACE_RESPONSE_CLASSES, + type TraceGptDiagnosticsV1, + type TraceReportV1, + type TraceStoredReportV1, +} from './report-types'; +import { + traceEnum, + traceItems, + traceSize, + validDiagnosticAuctionId, + validateTraceAuctionEvidence, + validateTraceSlotCorrelation, +} from './validation'; + +export const TRACE_CYCLE_KEYS = [ + 'requestNumber', + 'requestedAtMs', + 'responseAtMs', + 'renderAtMs', + 'loadAtMs', + 'viewableAtMs', + 'durations', + 'isEmpty', + 'requestedSlotSizes', + 'size', + 'observedSlotSize', + 'isBackfill', + 'slotContentChanged', + 'incompleteSequence', + 'responseClass', + 'requestPath', + 'requestIntentId', + 'trustedServerAuctionId', + 'opportunityToRequestMs', + 'replacedRequestNumber', + 'previousRenderToRequestMs', + 'creativeChanged', + 'loadObservedBeforeRender', + 'trustedServerOpportunity', + 'trustedServerCreativeRequestAtMs', + 'trustedServerCreativeResponseAtMs', + 'trustedServerCreativeFailures', + 'delivery', +] as const; +const DURATION_KEYS = [ + 'requestToResponseMs', + 'responseToRenderMs', + 'requestToRenderMs', + 'renderToLoadMs', + 'renderToViewableMs', +] as const; +const TIME_KEYS = [ + 'requestedAtMs', + 'responseAtMs', + 'renderAtMs', + 'loadAtMs', + 'viewableAtMs', + 'opportunityToRequestMs', + 'previousRenderToRequestMs', + 'trustedServerCreativeRequestAtMs', + 'trustedServerCreativeResponseAtMs', +] as const; +const BOOLEAN_KEYS = [ + 'isEmpty', + 'isBackfill', + 'slotContentChanged', + 'creativeChanged', + 'loadObservedBeforeRender', +] as const; +const TRUNCATION_KEYS = [ + 'omitted_server_auctions', + 'omitted_slot_correlations', + 'omitted_request_cycles', + 'omitted_callback_issues', + 'omitted_attribution_issues', + 'omitted_nested_values', +] as const; + +/** Ingests a report into an independently owned immutable public model. */ +export function parseTraceReport( + value: unknown, + origin: string, + storedAtMs: number +): TraceReportV1 | undefined { + const snapshot = traceJsonSnapshot(value); + if ( + !snapshot || + traceOrigin(origin) !== origin || + !traceNumber(storedAtMs) || + !report(snapshot.value, origin, storedAtMs) + ) + return undefined; + freezeOwned(snapshot.value); + return snapshot.value; +} + +/** Ingests a supported, fresh storage wrapper without retaining caller objects. */ +export function parseTraceStoredReport( + value: unknown, + origin: string, + nowMs: number +): TraceStoredReportV1 | undefined { + const snapshot = traceJsonSnapshot(value, 11); + if (!snapshot || !validateTraceStoredReport(snapshot.value, origin, nowMs)) return undefined; + freezeOwned(snapshot.value); + return snapshot.value; +} + +function freezeOwned(value: unknown): void { + if (typeof value !== 'object' || value === null) return; + Object.values(value).forEach(freezeOwned); + Object.freeze(value); +} + +/** Safe categories for incompatible source/public schema versions. */ +export type TraceReportRejection = + | 'unsupported_report_version' + | 'unsupported_auction_version' + | 'unsupported_correlation_version' + | 'unsupported_gpt_version' + | 'unsupported_gpt_source_version' + | 'invalid_report'; + +/** Classifies invalid reports without returning input values or parser errors. */ +export function traceReportRejection( + value: unknown, + origin: string, + storedAtMs: number +): TraceReportRejection | undefined { + try { + const snapshot = traceJsonSnapshot(value); + if (!snapshot) return 'invalid_report'; + value = snapshot.value; + if (validateTraceReport(value, origin, storedAtMs)) return undefined; + if (!traceObject(value)) return 'invalid_report'; + if (traceOwn(value, 'schema_version') && value.schema_version !== 1) + return 'unsupported_report_version'; + const gpt = value.gpt_diagnostics; + if (traceObject(gpt)) { + if (traceOwn(gpt, 'schema_version') && gpt.schema_version !== 1) + return 'unsupported_gpt_version'; + if (traceOwn(gpt, 'source_schema_version') && gpt.source_schema_version !== 1) + return 'unsupported_gpt_source_version'; + } + const auctions = traceItems(value.server_auctions, 16); + if ( + auctions?.some( + (item) => traceObject(item) && traceOwn(item, 'schema_version') && item.schema_version !== 1 + ) + ) + return 'unsupported_auction_version'; + const correlations = traceItems(value.slot_correlations, 128); + if ( + correlations?.some( + (item) => traceObject(item) && traceOwn(item, 'schema_version') && item.schema_version !== 1 + ) + ) + return 'unsupported_correlation_version'; + return 'invalid_report'; + } catch { + return 'invalid_report'; + } +} + +/** Validates a browser-relative numeric clock without truncating fractional observations. */ +export function traceNumber(value: unknown, maximum = Number.MAX_SAFE_INTEGER): value is number { + return typeof value === 'number' && Number.isFinite(value) && value >= 0 && value <= maximum; +} + +/** Canonicalizes only a strict bare HTTP(S) origin, with no URL repair. */ +export function traceOrigin(value: unknown): string | undefined { + if (!traceText(value, 255) || !/^https?:\/\//i.test(value)) return undefined; + const authority = value.slice(value.indexOf('://') + 3); + if (!authority || /[\s/@?#,\\]/.test(authority)) return undefined; + const port = authority.startsWith('[') + ? authority.slice(authority.indexOf(']') + 1) + : authority.includes(':') + ? authority.slice(authority.lastIndexOf(':')) + : ''; + if (port !== '' && (!/^:\d+$/.test(port) || Number(port.slice(1)) > 65535)) return undefined; + try { + const parsed = new URL(value); + if ( + !['http:', 'https:'].includes(parsed.protocol) || + !parsed.hostname || + parsed.username || + parsed.password || + parsed.search || + parsed.hash || + parsed.pathname !== '/' + ) + return undefined; + return parsed.origin; + } catch { + return undefined; + } +} + +function members(value: unknown, limit: number, predicate: (item: unknown) => boolean): boolean { + const items = traceItems(value, limit); + return items !== undefined && items.every(predicate); +} +function optional( + value: Record, + key: string, + valid: (item: unknown) => boolean +): boolean { + return !traceOwn(value, key) || valid(value[key]); +} +function cycle(value: unknown): boolean { + if ( + !traceObject(value) || + !traceKeys(value, ['requestNumber', 'durations', 'incompleteSequence'], TRACE_CYCLE_KEYS) || + !traceInteger(value.requestNumber) || + typeof value.incompleteSequence !== 'boolean' + ) + return false; + if ( + !traceObject(value.durations) || + !traceKeys(value.durations, [], DURATION_KEYS) || + !Object.values(value.durations).every((item) => traceNumber(item)) + ) + return false; + if ( + !TIME_KEYS.every((key) => optional(value, key, traceNumber)) || + !BOOLEAN_KEYS.every((key) => optional(value, key, (item) => typeof item === 'boolean')) + ) + return false; + return ( + ['requestIntentId', 'replacedRequestNumber'].every((key) => + optional(value, key, traceInteger) + ) && + optional(value, 'trustedServerAuctionId', validDiagnosticAuctionId) && + optional(value, 'requestedSlotSizes', (item) => members(item, 16, (size) => traceSize(size))) && + optional(value, 'size', (item) => traceSize(item)) && + optional(value, 'observedSlotSize', (item) => traceSize(item, true)) && + optional(value, 'responseClass', (item) => traceEnum(item, TRACE_RESPONSE_CLASSES)) && + optional(value, 'requestPath', (item) => traceEnum(item, TRACE_REQUEST_PATHS)) && + optional(value, 'trustedServerOpportunity', (item) => traceEnum(item, TRACE_OPPORTUNITIES)) && + optional(value, 'delivery', (item) => traceEnum(item, TRACE_DELIVERIES)) && + optional(value, 'trustedServerCreativeFailures', (item) => + members(item, 16, (failure) => traceEnum(failure, TRACE_CREATIVE_FAILURES)) + ) + ); +} +function slot(value: unknown): boolean { + if ( + !traceObject(value) || + !traceKeys( + value, + ['runtimeSlotNumber', 'binding', 'requests'], + ['currentVisibilityPercentage', 'maximumVisibilityPercentage'] + ) || + !traceInteger(value.runtimeSlotNumber) + ) + return false; + if ( + !traceObject(value.binding) || + !traceKeys(value.binding, ['status'], ['reason']) || + !traceEnum(value.binding.status, ['bound', 'unbound', 'ambiguous']) || + !optional(value.binding, 'reason', (item) => traceEnum(item, TRACE_BINDING_REASONS)) + ) + return false; + return ( + optional(value, 'currentVisibilityPercentage', (item) => traceNumber(item, 100)) && + optional(value, 'maximumVisibilityPercentage', (item) => traceNumber(item, 100)) && + members(value.requests, 10, cycle) + ); +} +function callback(value: unknown): boolean { + return ( + traceObject(value) && + traceKeys(value, ['kind', 'runtimeSlotNumber', 'timestampMs', 'disposition', 'reason']) && + traceEnum(value.kind, TRACE_CALLBACK_KINDS) && + traceInteger(value.runtimeSlotNumber) && + traceNumber(value.timestampMs) && + traceEnum(value.disposition, ['matched', 'unmatched', 'ambiguous']) && + traceEnum(value.reason, TRACE_CALLBACK_REASONS) + ); +} +function attribution(value: unknown): boolean { + return ( + traceObject(value) && + traceKeys(value, ['reason', 'timestampMs'], ['runtimeSlotNumber']) && + traceEnum(value.reason, TRACE_ATTRIBUTION_REASONS) && + traceNumber(value.timestampMs) && + optional(value, 'runtimeSlotNumber', traceInteger) + ); +} +function coverageCounters(value: unknown): boolean { + return ( + traceObject(value) && + traceKeys(value, ['observed', 'matched', 'unmatched', 'ambiguous']) && + Object.values(value).every((item) => traceInteger(item)) + ); +} +function gpt(value: unknown, origin: string): value is TraceGptDiagnosticsV1 { + if ( + !traceObject(value) || + !traceKeys( + value, + [ + 'schema_version', + 'source_schema_version', + 'capturedAt', + 'page', + 'slots', + 'callbackIssues', + 'coverage', + 'metadata', + ], + ['attributionIssues'] + ) || + value.schema_version !== 1 || + value.source_schema_version !== 1 || + !traceTimestamp(value.capturedAt) + ) + return false; + if ( + !traceObject(value.page) || + !traceKeys(value.page, ['origin', 'pathname']) || + traceOrigin(value.page.origin) !== origin || + value.page.pathname !== '/[redacted]' + ) + return false; + if ( + !members(value.slots, 64, slot) || + !members(value.callbackIssues, 128, callback) || + !optional(value, 'attributionIssues', (item) => members(item, 128, attribution)) + ) + return false; + if ( + !traceObject(value.coverage) || + !traceKeys(value.coverage, TRACE_CALLBACK_KINDS) || + !Object.values(value.coverage).every(coverageCounters) + ) + return false; + return ( + traceObject(value.metadata) && + traceKeys( + value.metadata, + ['droppedCallbacks', 'evictedSlots', 'evictedRequestCycles'], + ['droppedAttributionIssues'] + ) && + Object.values(value.metadata).every((item) => traceInteger(item)) + ); +} + +/** Validates the supported public GPT projection before display or export. */ +export function validateTraceGptDiagnostics( + value: unknown, + origin: string +): value is TraceGptDiagnosticsV1 { + try { + const snapshot = traceJsonSnapshot(value, 10, 4 * 1024 * 1024); + return snapshot !== undefined && traceOrigin(origin) === origin && gpt(snapshot.value, origin); + } catch { + return false; + } +} + +function closeTimestamp(value: unknown, storedAtMs: number): boolean { + if (!traceTimestamp(value)) return false; + const time = Date.parse(value); + return Number.isFinite(time) && Math.abs(time - storedAtMs) <= 60000; +} +function report(value: unknown, origin: string, storedAtMs: number): value is TraceReportV1 { + if ( + !traceObject(value) || + !traceKeys(value, [ + 'schema_version', + 'captured_at', + 'request_context', + 'server_auctions', + 'slot_correlations', + 'gpt_diagnostics', + 'auction_coverage', + 'truncation', + ]) || + value.schema_version !== 1 || + !closeTimestamp(value.captured_at, storedAtMs) + ) + return false; + if ( + !validateTraceRequestContext(value.request_context) || + !members(value.server_auctions, 16, validateTraceAuctionEvidence) || + !members(value.slot_correlations, 128, validateTraceSlotCorrelation) || + !gpt(value.gpt_diagnostics, origin) || + !closeTimestamp(value.gpt_diagnostics.capturedAt, storedAtMs) + ) + return false; + if ( + !traceObject(value.truncation) || + !traceKeys(value.truncation, TRUNCATION_KEYS) || + !Object.values(value.truncation).every((item) => traceInteger(item, 65535)) + ) + return false; + const coverage = value.auction_coverage; + if (!traceObject(coverage) || !traceKeys(coverage, ['capture_status', 'issues'])) return false; + const issues = traceItems(coverage.issues, 16); + if (!issues || !issues.every((issue) => traceEnum(issue, TRACE_COVERAGE_ISSUES))) return false; + let previous = -1; + for (const issue of issues) { + const index = TRACE_COVERAGE_ISSUES.indexOf(issue as (typeof TRACE_COVERAGE_ISSUES)[number]); + if (index <= previous) return false; + previous = index; + } + const records = traceItems(value.server_auctions, 16); + if (!records) return false; + const correlations = traceItems(value.slot_correlations, 128); + if (!correlations) return false; + for (const correlation of correlations) { + if (!traceObject(correlation)) return false; + if ( + records.some( + (record) => + traceObject(record) && + record.source === 'auction_api' && + record.diagnostic_auction_id === correlation.diagnostic_auction_id + ) + ) + return false; + } + const failed = issues.some((issue) => + [ + 'evidence_projection_failed', + 'evidence_transport_failed', + 'evidence_validation_failed', + 'record_evicted', + ].includes(issue as string) + ); + const expected = + records.length > 0 + ? failed + ? 'partial' + : 'complete' + : failed + ? 'unavailable' + : 'not_observed'; + return coverage.capture_status === expected && boundedJsonShape(value, 10) !== undefined; +} + +/** Validates the exact combined report and its capture-clock relationship. */ +export function validateTraceReport( + value: unknown, + origin: string, + storedAtMs: number +): value is TraceReportV1 { + try { + const snapshot = traceJsonSnapshot(value); + return ( + snapshot !== undefined && + traceOrigin(origin) === origin && + traceNumber(storedAtMs) && + report(snapshot.value, origin, storedAtMs) + ); + } catch { + return false; + } +} +/** Validates the exact storage wrapper, bounds and expiry before report use. */ +export function validateTraceStoredReport( + value: unknown, + origin: string, + nowMs: number +): value is TraceStoredReportV1 { + try { + const snapshot = traceJsonSnapshot(value, 11); + if (!snapshot) return false; + const owned = snapshot.value; + return ( + traceObject(owned) && + traceKeys(owned, ['stored_at_ms', 'report']) && + traceNumber(nowMs) && + traceInteger(owned.stored_at_ms) && + owned.stored_at_ms - nowMs <= 60000 && + nowMs - owned.stored_at_ms <= 900000 && + validateTraceReport(owned.report, origin, owned.stored_at_ms) + ); + } catch { + return false; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/report-view.ts b/crates/trusted-server-js/lib/src/trace/report-view.ts new file mode 100644 index 000000000..5ef6fe100 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/report-view.ts @@ -0,0 +1,614 @@ +import { joinTraceEvidence } from './correlation'; +import { copyTraceReport, downloadTraceReport, shareTraceReport } from './export'; +import { endTraceSessionAndObserve, type TraceSessionChangeResult } from './lifecycle'; +import type { TraceGptRequestCycle, TraceReportV1, TraceStoredReportV1 } from './report-types'; +import { mountTraceSetup } from './setup'; +import { deleteTraceReport, readTraceReport } from './storage'; +import type { CookieHealth, TraceRequestContextV1 } from './types'; + +export interface TraceViewerOptions { + readonly origin?: string; + readonly now?: () => number; + readonly storage?: Pick; + readonly confirm?: (message: string) => boolean; + readonly copy?: typeof copyTraceReport; + readonly download?: typeof downloadTraceReport; + readonly share?: typeof shareTraceReport; +} + +function label(value: string): string { + const labels: Record = { + no_bid: 'No bid returned', + no_candidate: 'No candidate', + selected: 'Candidate selected', + selected_unrenderable: 'Selected candidate could not be rendered', + trusted_server_direct: 'Trusted Server request path observed', + prebid_refresh: 'Browser refresh observed; winner not determined', + publisher_refresh: 'Browser refresh observed; winner not determined', + competing: 'Multiple or unknown delivery paths', + unattributed: 'Multiple or unknown delivery paths', + trusted_server_response_sent: 'Trusted Server creative response sent', + trusted_server_selected: 'Trusted Server candidate selected; render unconfirmed', + candidate_unconfirmed: 'Candidate unconfirmed', + not_observed: 'Not observed', + unknown: 'Unknown', + unavailable: 'Unavailable', + }; + return ( + labels[value] ?? + value + .replace(/([a-z])([A-Z])/g, '$1 $2') + .replace(/_/g, ' ') + .replace(/^./, (first) => first.toUpperCase()) + ); +} +function valueText(value: unknown): string { + if (value === undefined) return 'Unavailable'; + if (typeof value === 'boolean') return value ? 'Yes' : 'No'; + if (typeof value === 'number') return String(value); + if (typeof value === 'string') return value; + return 'Unavailable'; +} +function healthText(health: CookieHealth): string { + if (health.state === 'absent') return 'Not present in this request'; + if (health.state === 'present_valid') return 'Valid shape observed'; + if (health.state === 'duplicate') return 'Multiple values observed'; + if (health.state === 'present_invalid') + return health.detail === 'oversized' + ? 'Invalid shape — too long' + : health.detail === 'unsupported_value' + ? 'Invalid shape — unsupported value' + : 'Invalid shape observed'; + if (health.detail === 'runtime_header_ambiguous') + return 'Unavailable — runtime-visible cookies could not be reliably inspected'; + return health.detail === 'header_too_large' + ? 'Unavailable — the visible cookie header was too large' + : 'Unavailable — the visible cookie header was not valid text'; +} +function element( + root: Document, + tag: K, + text?: string +): HTMLElementTagNameMap[K] { + const node = root.createElement(tag); + if (text !== undefined) node.textContent = text; + return node; +} +function section(root: Document, parent: Element, title: string, id?: string): HTMLElement { + const node = element(root, 'section'); + if (id) node.id = id; + node.append(element(root, 'h2', title)); + parent.append(node); + return node; +} +function facts( + root: Document, + parent: Element, + rows: readonly (readonly [string, unknown])[] +): void { + const list = element(root, 'dl'); + for (const [name, value] of rows) + list.append(element(root, 'dt', name), element(root, 'dd', valueText(value))); + parent.append(list); +} +function sizes(value?: readonly (readonly [number, number])[]): string { + return value === undefined + ? 'Unavailable' + : value.length === 0 + ? 'Not observed' + : value.map(([width, height]) => `${width} × ${height}`).join(', '); +} +function millis(value?: number): string { + return value === undefined ? 'Unavailable' : `${value} ms`; +} +const NETWORK_LABELS: Record = { + masked_client_ip: 'Approximate network identifier', + country: 'Country', + region: 'Region', + asn: 'ASN', + http_version: 'HTTP version', + tls_protocol: 'TLS protocol', + tls_cipher: 'TLS cipher', + edge_hostname: 'Edge hostname', + edge_region: 'Edge region', + edge_pop: 'Edge POP', +}; +const CYCLE_FACTS: Record< + Exclude< + keyof TraceGptRequestCycle, + | 'requestNumber' + | 'durations' + | 'requestedSlotSizes' + | 'size' + | 'observedSlotSize' + | 'trustedServerAuctionId' + | 'trustedServerCreativeFailures' + | 'isEmpty' + | 'requestPath' + >, + string +> = { + requestedAtMs: 'Requested (browser clock)', + responseAtMs: 'Response received (browser clock)', + renderAtMs: 'Rendered (browser clock)', + loadAtMs: 'Loaded (browser clock)', + viewableAtMs: 'Viewable (browser clock)', + isBackfill: 'Backfill observed', + slotContentChanged: 'Slot content changed', + incompleteSequence: 'Incomplete sequence', + responseClass: 'GPT response class', + requestIntentId: 'Browser request intent number', + opportunityToRequestMs: 'Opportunity to request', + replacedRequestNumber: 'Replaced request number', + previousRenderToRequestMs: 'Previous render to request', + creativeChanged: 'Creative changed', + loadObservedBeforeRender: 'Load observed before render', + trustedServerOpportunity: 'Trusted Server candidate opportunity', + trustedServerCreativeRequestAtMs: 'Creative bridge request (browser clock)', + trustedServerCreativeResponseAtMs: 'Creative bridge response (browser clock)', + delivery: 'Creative delivery observation', +}; + +function renderReport( + root: Document, + report: TraceReportV1, + origin: string, + storedAtMs: number +): HTMLElement { + const article = element(root, 'article'); + article.id = 'trace-report'; + article.append( + element(root, 'h1', 'Trusted Server trace results'), + element(root, 'p', 'Browser-carried, unverified diagnostic data') + ); + article.append( + element( + root, + 'p', + 'This browser snapshot helps troubleshoot rendering. It is not proof of a server event, identity, or security incident.' + ) + ); + const summary = section(root, article, 'Report summary'); + facts(root, summary, [ + ['Captured at', report.captured_at], + ['Publisher origin', report.gpt_diagnostics.page.origin], + ['Server auctions retained', report.server_auctions.length], + ['GPT slots retained', report.gpt_diagnostics.slots.length], + ]); + const network = section(root, article, 'Publisher request'); + network.append( + element(root, 'p', 'Produced by Trusted Server; copied through an untrusted browser snapshot') + ); + network.append( + element( + root, + 'p', + 'These facts describe the traced publisher document. Masked identifiers are approximate and may still identify a network.' + ) + ); + facts(root, network, [ + ['Document request captured at', report.request_context.captured_at], + ...Object.entries(NETWORK_LABELS).map( + ([key, name]) => + [ + name, + report.request_context.network[key as keyof TraceRequestContextV1['network']], + ] as const + ), + ]); + const cookies = section(root, article, 'Cookie health', 'trace-report-cookies'); + cookies.append( + element( + root, + 'p', + 'Produced by Trusted Server; copied through an untrusted browser snapshot. Only cookie shape visible in this request is inspected; values and browser attributes are excluded.' + ) + ); + facts(root, cookies, [ + ['Edge Cookie', healthText(report.request_context.cookies.ts_ec)], + ['External IDs', healthText(report.request_context.cookies.ts_eids)], + ['Tester', healthText(report.request_context.cookies.ts_tester)], + ['Diagnostics session', healthText(report.request_context.cookies.diagnostics_session)], + ]); + const auctions = section(root, article, 'Server auctions'); + auctions.append( + element( + root, + 'p', + 'Returned bid counts may overlap between slots and must not be summed as unique bids.' + ) + ); + const joined = joinTraceEvidence(report, origin, storedAtMs); + if (!report.server_auctions.length) + auctions.append(element(root, 'p', 'Not observed. This does not mean no server auction ran.')); + for (const [index, view] of (joined?.auctions ?? []).entries()) { + const entry = element(root, 'details'); + entry.open = true; + entry.append(element(root, 'summary', `Auction ${index + 1}: ${view.sourceLabel}`)); + entry.append( + element(root, 'p', 'Produced by Trusted Server; copied through an untrusted browser snapshot') + ); + facts(root, entry, [ + ['Terminal outcome', label(view.evidence.terminal_status)], + [ + 'Terminal reason', + view.evidence.terminal_reason === undefined + ? 'Unavailable' + : label(view.evidence.terminal_reason), + ], + ['Server auction-local elapsed time', millis(view.evidence.total_time_ms)], + ['Request-relative milestones', view.relativeMilestonesLabel], + ]); + entry.append(element(root, 'p', view.providerScopeLabel)); + for (const provider of view.evidence.provider_calls) + facts(root, entry, [ + [`Provider call ${provider.provider_number}`, label(provider.role)], + ['Call outcome', label(provider.status)], + ['Provider-local elapsed time', millis(provider.response_time_ms)], + ['Returned bid count', provider.returned_bid_count], + ]); + if (!view.evidence.provider_calls.length) + entry.append(element(root, 'p', 'Provider calls: Not observed')); + for (const slot of view.slots) { + const box = element(root, 'details'); + box.open = true; + box.append(element(root, 'summary', `Server slot ${slot.serverSlot.slot_number}`)); + facts(root, box, [ + ['Requested sizes', sizes(slot.serverSlot.requested_sizes)], + ['Returned bid count', slot.serverSlot.returned_bid_count], + ['Candidate', label(slot.serverSlot.candidate)], + [ + 'Selected creative size', + slot.serverSlot.selected_creative_size + ? sizes([slot.serverSlot.selected_creative_size]) + : 'Unavailable', + ], + [ + 'Correlation', + slot.correlation === 'matched' + ? `Matched GPT slot ${slot.runtimeSlotNumber}, request ${slot.requestNumber}` + : 'Correlation unknown', + ], + ['Browser request path', slot.pathLabel ?? 'Unknown'], + ['Creative participation', slot.creativeLabel ?? 'Participation unconfirmed'], + ]); + entry.append(box); + } + facts(root, entry, [ + ['Provider calls omitted', view.evidence.truncation.omitted_provider_calls], + ['Server slots omitted', view.evidence.truncation.omitted_slots], + ['Nested values omitted', view.evidence.truncation.omitted_nested_values], + ]); + auctions.append(entry); + } + const gpt = section(root, article, 'GPT delivery and creative rendering'); + gpt.append( + element( + root, + 'p', + 'Browser observed. Server auction → GPT request/response → creative render/load/viewability are separate observations. A filled slot does not identify an auction winner.' + ) + ); + facts(root, gpt, [['Browser snapshot captured at', report.gpt_diagnostics.capturedAt]]); + if (!report.gpt_diagnostics.slots.length) gpt.append(element(root, 'p', 'Not observed')); + for (const slot of report.gpt_diagnostics.slots) { + const box = element(root, 'details'); + box.open = true; + box.append(element(root, 'summary', `GPT slot ${slot.runtimeSlotNumber}`)); + facts(root, box, [ + ['Binding', label(slot.binding.status)], + [ + 'Binding reason', + slot.binding.reason === undefined ? 'Unavailable' : label(slot.binding.reason), + ], + ['Current visibility percentage', slot.currentVisibilityPercentage], + ['Maximum visibility percentage', slot.maximumVisibilityPercentage], + ]); + if (!slot.requests.length) box.append(element(root, 'p', 'Requests: Not observed')); + for (const cycle of slot.requests) { + const request = element(root, 'details'); + request.open = true; + request.append(element(root, 'summary', `Request ${cycle.requestNumber}`)); + const matches = (joined?.auctions ?? []) + .flatMap((auction) => auction.slots) + .filter( + (entry) => + entry.correlation === 'matched' && + entry.runtimeSlotNumber === slot.runtimeSlotNumber && + entry.requestNumber === cycle.requestNumber + ); + facts(root, request, [ + ['Correlation', matches.length === 1 ? 'Matched server slot' : 'Correlation unknown'], + [ + 'Creative participation', + matches.length === 1 ? matches[0].creativeLabel : 'Participation unconfirmed', + ], + [ + 'GPT fill observation', + cycle.isEmpty === undefined ? 'Unknown' : cycle.isEmpty ? 'Empty' : 'Filled', + ], + [ + 'Browser request path', + cycle.requestPath === undefined ? 'Unavailable' : label(cycle.requestPath), + ], + ['Requested sizes', sizes(cycle.requestedSlotSizes)], + ['Rendered size', cycle.size ? sizes([cycle.size]) : 'Unavailable'], + [ + 'Observed CSS box size', + cycle.observedSlotSize ? sizes([cycle.observedSlotSize]) : 'Unavailable', + ], + ]); + const rows = Object.entries(CYCLE_FACTS).map(([key, name]): readonly [string, unknown] => { + const value = cycle[key as keyof typeof CYCLE_FACTS]; + return [ + name, + typeof value === 'string' + ? label(value) + : key.endsWith('Ms') + ? millis(value as number | undefined) + : value, + ]; + }); + facts(root, request, rows); + facts(root, request, [ + ['Request to response', millis(cycle.durations.requestToResponseMs)], + ['Response to render', millis(cycle.durations.responseToRenderMs)], + ['Request to render', millis(cycle.durations.requestToRenderMs)], + ['Render to load', millis(cycle.durations.renderToLoadMs)], + ['Render to viewable', millis(cycle.durations.renderToViewableMs)], + [ + 'Creative bridge failures', + cycle.trustedServerCreativeFailures === undefined + ? 'Unavailable' + : cycle.trustedServerCreativeFailures.length === 0 + ? 'Not observed' + : cycle.trustedServerCreativeFailures.map(label).join(', '), + ], + ]); + box.append(request); + } + gpt.append(box); + } + const coverage = section(root, article, 'Coverage and ambiguity'); + facts(root, coverage, [ + ['Server capture', label(report.auction_coverage.capture_status)], + [ + 'Capture and interpretation limits', + report.auction_coverage.issues.length + ? report.auction_coverage.issues.map(label).join(', ') + : 'None recorded', + ], + ['Correlation sidecars retained', report.slot_correlations.length], + ]); + for (const [kind, counts] of Object.entries(report.gpt_diagnostics.coverage)) + facts(root, coverage, [ + [ + label(kind), + `${counts.observed} observed; ${counts.matched} matched; ${counts.unmatched} unmatched; ${counts.ambiguous} ambiguous`, + ], + ]); + for (const [key, count] of Object.entries(report.truncation)) + facts(root, coverage, [[label(key), count]]); + for (const [key, count] of Object.entries(report.gpt_diagnostics.metadata)) + facts(root, coverage, [[`GPT ${label(key)}`, count]]); + for (const issue of report.gpt_diagnostics.callbackIssues) + facts(root, coverage, [ + ['Callback issue', label(issue.kind)], + ['GPT slot', issue.runtimeSlotNumber], + ['Browser clock', millis(issue.timestampMs)], + ['Disposition', label(issue.disposition)], + ['Reason', label(issue.reason)], + ]); + for (const issue of report.gpt_diagnostics.attributionIssues ?? []) + facts(root, coverage, [ + ['Creative attribution issue', label(issue.reason)], + ['GPT slot', issue.runtimeSlotNumber], + ['Browser clock', millis(issue.timestampMs)], + ]); + for (const [index, record] of report.slot_correlations.entries()) { + const entry = element(root, 'details'); + entry.append(element(root, 'summary', `Correlation record ${index + 1}`)); + entry.append( + element( + root, + 'p', + 'Browser observed correlation. These opaque references permit a join only when unique and consistent; duplicate or conflicting records remain unknown.' + ) + ); + facts(root, entry, [ + ['Auction reference', record.diagnostic_auction_id], + ['Slot reference', record.slot_ref], + ['GPT slot number', record.runtime_slot_number], + ['GPT request number', record.request_number], + ]); + coverage.append(entry); + } + return article; +} + +/** Mounts the trace page using the current tab's validated report and setup controls. */ +export function mountTraceViewer( + root: Document = document, + options: TraceViewerOptions = {} +): { destroy(): void } { + const main = root.querySelector('main'); + if (!main) return { destroy() {} }; + const destroySetup = mountTraceSetup(root); + const origin = options.origin ?? window.location.origin; + let stored: TraceStoredReportV1 | undefined; + let destroyed = false; + const disposers: Array<() => void> = []; + const control = (parent: Element, text: string, action: () => void): HTMLButtonElement => { + const button = element(root, 'button', text); + button.type = 'button'; + const listener = (): void => { + if (!destroyed) action(); + }; + button.addEventListener('click', listener); + disposers.push(() => button.removeEventListener('click', listener)); + parent.append(button); + return button; + }; + const destroy = (): void => { + destroyed = true; + stored = undefined; + destroySetup(); + for (const dispose of disposers) dispose(); + }; + let read: ReturnType; + try { + read = readTraceReport(origin, (options.now ?? Date.now)(), options.storage); + } catch { + read = { status: 'unavailable' }; + } + if (read.status !== 'ready') { + const notice = element( + root, + 'p', + read.status === 'absent' + ? 'No saved report. Enable tracing, return to the affected page, reload once, reproduce the problem, then select View trace results.' + : 'The saved report is unavailable, expired, or unsupported. Return to the affected page in this same tab and exact hostname, reload once, reproduce the problem, then select View trace results.' + ); + notice.id = 'trace-report-notice'; + main.prepend(notice); + return { destroy }; + } + stored = read.value; + const setup = element(root, 'details'); + setup.id = 'trace-viewer-setup'; + setup.append(element(root, 'summary', 'Setup request and tracing controls')); + for (const child of Array.from(main.children)) setup.append(child); + const article = renderReport(root, stored.report, origin, stored.stored_at_ms); + main.append(article); + const exports = section(root, article, 'Export'); + exports.append( + element( + root, + 'p', + 'Copy, Download and Share use the same public report JSON. The selected app receives this JSON when you choose Share. Nothing is uploaded by this viewer.' + ) + ); + const exportControls = element(root, 'div'); + exportControls.className = 'controls'; + exports.append(exportControls); + const exportStatus = element(root, 'p'); + exportStatus.id = 'trace-export-status'; + exportStatus.setAttribute('role', 'status'); + exportStatus.setAttribute('aria-live', 'polite'); + exports.append(exportStatus); + let exporting = false; + const exportReport = async (kind: 'Copy' | 'Download' | 'Share'): Promise => { + if (!stored || destroyed || exporting) return; + exporting = true; + const captured = stored; + try { + const action = + kind === 'Copy' + ? (options.copy ?? copyTraceReport) + : kind === 'Share' + ? (options.share ?? shareTraceReport) + : (options.download ?? downloadTraceReport); + const result = await action(captured.report, origin, captured.stored_at_ms); + if (destroyed || stored !== captured) return; + exportStatus.textContent = + result.status === 'copied' + ? 'Copied JSON.' + : result.status === 'downloaded' + ? 'Download started; completion is managed by your browser.' + : result.status === 'shared' + ? 'The share request completed.' + : kind === 'Share' && result.status === 'unsupported' + ? 'File sharing is unavailable. Use Copy or Download.' + : `${kind} could not be completed. Your report remains available; retry or choose another export.`; + } catch { + if (!destroyed && stored === captured) + exportStatus.textContent = `${kind} could not be completed. Your report remains available.`; + } finally { + exporting = false; + } + }; + for (const kind of ['Copy', 'Download', 'Share'] as const) + control(exportControls, kind, () => { + void exportReport(kind); + }); + const cleanup = section(root, main, 'Report cleanup'); + const controls = element(root, 'div'); + controls.className = 'controls'; + cleanup.append(controls); + const localStatus = element(root, 'p', 'A local report is saved in this tab.'); + localStatus.id = 'trace-cleanup-local-status'; + localStatus.setAttribute('role', 'status'); + localStatus.setAttribute('aria-live', 'polite'); + cleanup.append(localStatus); + const serverStatus = element( + root, + 'p', + 'Server tracing state has not been changed by this report visit.' + ); + serverStatus.id = 'trace-cleanup-server-status'; + serverStatus.setAttribute('role', 'status'); + serverStatus.setAttribute('aria-live', 'polite'); + cleanup.append(serverStatus); + const removeLocal = (): void => { + let result: ReturnType; + try { + result = deleteTraceReport(options.storage); + } catch { + result = { status: 'unavailable' }; + } + if (destroyed) return; + if (result.status === 'deleted') { + stored = undefined; + article.remove(); + deleteButton.hidden = true; + localStatus.textContent = 'Local report deleted from this tab.'; + } else + localStatus.textContent = + 'Local report deletion failed. The report remains displayed; retry deletion.'; + }; + let ending = false; + const end = async (): Promise => { + if (destroyed || ending) return; + ending = true; + clearButton.disabled = retryButton.disabled = true; + serverStatus.textContent = 'Requesting tracing end and checking the next request…'; + let result: TraceSessionChangeResult; + try { + result = await endTraceSessionAndObserve(); + } catch { + result = { mutation: 'failed', observation: 'failed', confirmed: false }; + } + if (destroyed) return; + serverStatus.textContent = result.confirmed + ? 'Tracing is off — no valid diagnostics session observed.' + : result.observation === 'inactive' + ? 'End tracing unconfirmed. No valid diagnostics session was observed; tracing may remain active. Retry end tracing.' + : result.observation === 'active' + ? 'End tracing unconfirmed — a valid session is still observed. Tracing may remain active. Retry end tracing.' + : 'End tracing unconfirmed. Tracing may remain active. Retry end tracing.'; + retryButton.hidden = result.confirmed; + clearButton.disabled = retryButton.disabled = false; + ending = false; + }; + const clearButton = control(controls, 'Clear report and end tracing', () => { + if (ending) return; + let confirmed = false; + try { + confirmed = (options.confirm ?? ((message) => window.confirm(message)))( + 'Delete the report from this tab and request tracing end?' + ); + } catch { + /* No consent means no cleanup mutation. */ + } + if (!confirmed || destroyed) return; + removeLocal(); + if (!destroyed) void end(); + }); + const deleteButton = control(controls, 'Delete local report', removeLocal); + const retryButton = control(controls, 'Retry end tracing', () => { + void end(); + }); + retryButton.hidden = true; + main.append(setup); + return { destroy }; +} diff --git a/crates/trusted-server-js/lib/src/trace/report.ts b/crates/trusted-server-js/lib/src/trace/report.ts new file mode 100644 index 000000000..b83127714 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/report.ts @@ -0,0 +1,306 @@ +import { traceInteger, traceKeys, traceObject, validateTraceRequestContext } from './context'; +import type { TraceAuctionEvidenceV1, TraceSlotCorrelationV1 } from './types'; +import { addOmissions } from './omissions'; +import { projectTraceGptDiagnostics } from './projection'; +import { + TRACE_COVERAGE_ISSUES, + type TraceCoverageIssue, + type TraceReportV1, + type TraceStoredReportV1, +} from './report-types'; +import { parseTraceStoredReport, traceJsonSnapshot, traceOrigin } from './report-validation'; +import { + traceEnum, + traceItems, + parseTraceAuctionTransport, + parseTraceSlotCorrelation, +} from './validation'; + +/** Collector observations keep outer losses separate from inner evidence truncation. */ +export interface TraceCollectorSnapshot { + readonly serverAuctions: readonly TraceAuctionEvidenceV1[]; + readonly slotCorrelations: readonly TraceSlotCorrelationV1[]; + readonly issues: readonly TraceCoverageIssue[]; + readonly omittedServerAuctions: number; + readonly omittedSlotCorrelations: number; +} +export interface TraceCaptureInput { + readonly requestContext: unknown; + readonly gptSource: unknown; + readonly origin: string; + readonly capturedAtMs: number; + readonly collector: unknown; +} +export type TraceCaptureResult = + | { readonly ok: true; readonly value: TraceStoredReportV1 } + | { + readonly ok: false; + readonly reason: + | 'invalid_snapshot' + | 'unsupported_source_version' + | 'omission_counter_overflow' + | 'snapshot_too_large'; + }; +type Mutable = T extends readonly (infer Element)[] + ? Mutable[] + : T extends object + ? { -readonly [Key in keyof T]: Mutable } + : T; +const MAXIMUM_BYTES = 512 * 1024; + +/** Captures one owned report while protecting every nonempty slot's newest cycle. */ +export function buildTraceReport( + input: TraceCaptureInput, + maximumBytes = MAXIMUM_BYTES +): TraceCaptureResult { + let counterFailed = false; + const add = (current: number, added: number): number => { + try { + return addOmissions(current, added); + } catch { + counterFailed = true; + throw new Error('omission_counter_overflow'); + } + }; + try { + if ( + !traceObject(input) || + !traceKeys(input, ['requestContext', 'gptSource', 'origin', 'capturedAtMs', 'collector']) + ) + return { ok: false, reason: 'invalid_snapshot' }; + const descriptors = Object.getOwnPropertyDescriptors(input); + const captureClock: unknown = descriptors.capturedAtMs.value; + const origin: unknown = descriptors.origin.value; + const requestContext: unknown = descriptors.requestContext.value; + const gptSource: unknown = descriptors.gptSource.value; + const collector: unknown = descriptors.collector.value; + if ( + !traceInteger(captureClock) || + typeof origin !== 'string' || + traceOrigin(origin) !== origin || + !traceInteger(maximumBytes, MAXIMUM_BYTES) || + maximumBytes === 0 + ) + return { ok: false, reason: 'invalid_snapshot' }; + const capturedAt = new Date(captureClock).toISOString(); + const context = traceJsonSnapshot(requestContext); + if (!context || !validateTraceRequestContext(context.value)) + return { ok: false, reason: 'invalid_snapshot' }; + const projected = projectTraceGptDiagnostics(gptSource, origin); + if (!projected.ok) + return { + ok: false, + reason: projected.reason === 'invalid_source' ? 'invalid_snapshot' : projected.reason, + }; + if (Math.abs(Date.parse(projected.value.capturedAt) - captureClock) > 60000) + return { ok: false, reason: 'invalid_snapshot' }; + const collection = traceJsonSnapshot(collector, 10, 64 * 1024 * 1024)?.value; + if ( + !traceObject(collection) || + !traceKeys(collection, [ + 'serverAuctions', + 'slotCorrelations', + 'issues', + 'omittedServerAuctions', + 'omittedSlotCorrelations', + ]) + ) + return { ok: false, reason: 'invalid_snapshot' }; + const auctions = traceItems(collection.serverAuctions, 65535 + 16); + const sidecars = traceItems(collection.slotCorrelations, 65535 + 128); + const issueItems = traceItems(collection.issues, TRACE_COVERAGE_ISSUES.length); + if ( + !auctions || + !sidecars || + !issueItems || + !issueItems.every((issue) => traceEnum(issue, TRACE_COVERAGE_ISSUES)) + ) + return { ok: false, reason: 'invalid_snapshot' }; + const copiedAuctions = auctions.map( + (record) => parseTraceAuctionTransport({ schema_version: 1, evidence: record })?.evidence + ); + const copiedSidecars = sidecars.map(parseTraceSlotCorrelation); + if ( + copiedAuctions.some((record) => record === undefined) || + copiedSidecars.some((record) => record === undefined) + ) + return { ok: false, reason: 'invalid_snapshot' }; + const apiIds = new Set( + copiedAuctions + .filter((record) => record?.source === 'auction_api') + .map((record) => record!.diagnostic_auction_id) + ); + if (copiedSidecars.some((sidecar) => apiIds.has(sidecar!.diagnostic_auction_id))) + return { ok: false, reason: 'invalid_snapshot' }; + const omittedAuctions = add( + collection.omittedServerAuctions as number, + Math.max(0, auctions.length - 16) + ); + let omittedSidecars = add( + collection.omittedSlotCorrelations as number, + Math.max(0, sidecars.length - 128) + ); + const retainedAuctions = copiedAuctions.slice(-16); + const discardedIds = new Set( + copiedAuctions + .slice(0, Math.max(0, copiedAuctions.length - 16)) + .map((record) => record!.diagnostic_auction_id) + ); + const retainedIds = new Set(retainedAuctions.map((record) => record!.diagnostic_auction_id)); + const retainedSidecars = copiedSidecars + .slice(-128) + .filter( + (sidecar) => + !discardedIds.has(sidecar!.diagnostic_auction_id) || + retainedIds.has(sidecar!.diagnostic_auction_id) + ); + omittedSidecars = add( + omittedSidecars, + Math.min(128, sidecars.length) - retainedSidecars.length + ); + const draft = traceJsonSnapshot( + { + schema_version: 1, + captured_at: capturedAt, + request_context: context.value, + server_auctions: retainedAuctions, + slot_correlations: retainedSidecars, + gpt_diagnostics: projected.value, + auction_coverage: { capture_status: 'not_observed', issues: [] }, + truncation: { + omitted_server_auctions: omittedAuctions, + omitted_slot_correlations: omittedSidecars, + omitted_request_cycles: 0, + omitted_callback_issues: 0, + omitted_attribution_issues: 0, + omitted_nested_values: projected.omittedNestedValues, + }, + }, + 10, + 16 * 1024 * 1024 + )?.value as Mutable | undefined; + if (!draft) return { ok: false, reason: 'invalid_snapshot' }; + const issues = new Set(issueItems as TraceCoverageIssue[]); + const recompute = (): void => { + if (draft.truncation.omitted_server_auctions > 0) issues.add('record_evicted'); + if (draft.truncation.omitted_slot_correlations > 0) issues.add('correlation_unavailable'); + const ordered = TRACE_COVERAGE_ISSUES.filter((issue) => issues.has(issue)); + const failed = ordered.some((issue) => + [ + 'evidence_projection_failed', + 'evidence_transport_failed', + 'evidence_validation_failed', + 'record_evicted', + ].includes(issue) + ); + draft.auction_coverage = { + capture_status: draft.server_auctions.length + ? failed + ? 'partial' + : 'complete' + : failed + ? 'unavailable' + : 'not_observed', + issues: ordered, + }; + }; + recompute(); + const wrapper = { stored_at_ms: captureClock, report: draft }; + // Every draft member is already independently owned data. Serialize only that + // draft while measuring; final ingestion checks the complete schema and depth. + const encoder = new TextEncoder(); + const fits = (): boolean => encoder.encode(JSON.stringify(wrapper)).length <= maximumBytes; + const pruneSidecars = ( + predicate: (sidecar: Mutable['slot_correlations'][number]) => boolean + ): void => { + const retained = draft.slot_correlations.filter((sidecar) => !predicate(sidecar)); + draft.truncation.omitted_slot_correlations = add( + draft.truncation.omitted_slot_correlations, + draft.slot_correlations.length - retained.length + ); + draft.slot_correlations = retained; + }; + const floors = new Set( + draft.gpt_diagnostics.slots.flatMap((slot) => + slot.requests.length ? [slot.requests[slot.requests.length - 1]] : [] + ) + ); + const cycles = draft.gpt_diagnostics.slots.flatMap((slot) => + slot.requests.filter((cycle) => !floors.has(cycle)).map((cycle) => ({ slot, cycle })) + ); + cycles.sort((left, right) => { + const a = left.cycle.requestedAtMs; + const b = right.cycle.requestedAtMs; + return ( + (a === undefined ? (b === undefined ? 0 : -1) : b === undefined ? 1 : a - b) || + left.slot.runtimeSlotNumber - right.slot.runtimeSlotNumber || + left.cycle.requestNumber - right.cycle.requestNumber + ); + }); + for (const { slot, cycle } of cycles) { + if (fits()) break; + slot.requests = slot.requests.filter((retained) => retained !== cycle); + draft.truncation.omitted_request_cycles = add(draft.truncation.omitted_request_cycles, 1); + pruneSidecars( + (sidecar) => + sidecar.runtime_slot_number === slot.runtimeSlotNumber && + sidecar.request_number === cycle.requestNumber + ); + recompute(); + } + const removeIssues = ( + source: Issue[] | undefined, + counter: 'omitted_callback_issues' | 'omitted_attribution_issues' + ): void => { + if (!source) return; + const ordered = source + .map((issue, index) => ({ issue, index })) + .sort((a, b) => a.issue.timestampMs - b.issue.timestampMs || a.index - b.index); + for (const { issue } of ordered) { + if (fits()) break; + const index = source.indexOf(issue); + source.splice(index, 1); + draft.truncation[counter] = add(draft.truncation[counter], 1); + } + }; + removeIssues(draft.gpt_diagnostics.callbackIssues, 'omitted_callback_issues'); + removeIssues(draft.gpt_diagnostics.attributionIssues, 'omitted_attribution_issues'); + const remainingCycles = () => draft.gpt_diagnostics.slots.flatMap((slot) => slot.requests); + const removeAuction = (record: Mutable['server_auctions'][number]): void => { + const id = record.diagnostic_auction_id; + draft.server_auctions = draft.server_auctions.filter((retained) => retained !== record); + draft.truncation.omitted_server_auctions = add(draft.truncation.omitted_server_auctions, 1); + for (const slot of draft.gpt_diagnostics.slots) { + const retained = slot.requests.filter((cycle) => cycle.trustedServerAuctionId !== id); + draft.truncation.omitted_request_cycles = add( + draft.truncation.omitted_request_cycles, + slot.requests.length - retained.length + ); + slot.requests = retained; + } + pruneSidecars((sidecar) => sidecar.diagnostic_auction_id === id); + recompute(); + }; + for (const record of [...draft.server_auctions]) { + if (fits()) break; + if ( + !remainingCycles().some( + (cycle) => cycle.trustedServerAuctionId === record.diagnostic_auction_id + ) + ) + removeAuction(record); + } + for (const record of [...draft.server_auctions]) { + if (fits()) break; + if ( + ![...floors].some((cycle) => cycle.trustedServerAuctionId === record.diagnostic_auction_id) + ) + removeAuction(record); + } + if (!fits()) return { ok: false, reason: 'snapshot_too_large' }; + const result = parseTraceStoredReport(wrapper, origin, captureClock); + return result ? { ok: true, value: result } : { ok: false, reason: 'invalid_snapshot' }; + } catch { + return { ok: false, reason: counterFailed ? 'omission_counter_overflow' : 'invalid_snapshot' }; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/runtime.ts b/crates/trusted-server-js/lib/src/trace/runtime.ts new file mode 100644 index 000000000..7c15030bc --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/runtime.ts @@ -0,0 +1,160 @@ +import type { AdRequest } from '../core/auction'; +import type { TsjsApi } from '../core/types'; + +import { createTraceCollector, type TraceCollector } from './collector'; +import { createTraceGptBridge, type TraceGptBridge } from './gpt'; +import { parseTraceAuctionTransport, validTraceSlotRef } from './validation'; + +export interface TraceRuntimeScope { + tsjs?: TsjsApi; + __tsjs_trace_active?: unknown; + crypto?: { randomUUID?(): string }; +} +export interface TraceApiRequestCarry { + readonly request: AdRequest; + readonly slotRefs: readonly string[]; + readonly collector: TraceCollector; +} +function currentScope(): TraceRuntimeScope | undefined { + return typeof window === 'object' ? window : undefined; +} + +/** Installs one collector on the shared facade only for the literal document gate. */ +export function installTraceRuntime( + api: TsjsApi, + scope = currentScope() +): TraceCollector | undefined { + try { + if (scope?.__tsjs_trace_active !== true) return undefined; + const collector = (api.traceEvidence ??= createTraceCollector()); + api.traceGpt ??= createTraceGptBridge(collector); + return collector; + } catch { + return undefined; + } +} + +/** Reads the existing activated facade without creating a collector or listeners. */ +export function getActiveTraceCollector(scope = currentScope()): TraceCollector | undefined { + try { + return scope?.__tsjs_trace_active === true ? scope.tsjs?.traceEvidence : undefined; + } catch { + return undefined; + } +} + +/** Reads the single core-owned slot bridge only behind the literal document gate. */ +export function getActiveTraceGptBridge(scope = currentScope()): TraceGptBridge | undefined { + try { + return scope?.__tsjs_trace_active === true ? scope.tsjs?.traceGpt : undefined; + } catch { + return undefined; + } +} + +/** Decorates only final grouped units, retaining the ordinary request on token failure. */ +export function prepareTraceAuctionRequest( + request: AdRequest, + scope = currentScope() +): TraceApiRequestCarry | undefined { + const collector = getActiveTraceCollector(scope); + if (!collector) return undefined; + const fallback = { + request, + slotRefs: Object.freeze([] as string[]), + collector, + }; + try { + if (typeof scope?.crypto?.randomUUID !== 'function') return fallback; + const refs = request.adUnits.map(() => `ts-slot-${scope.crypto!.randomUUID!()}`); + if (!refs.every(validTraceSlotRef) || new Set(refs).size !== refs.length) return fallback; + const adUnits = request.adUnits.map((unit, index) => { + const prior = unit.ext?.trusted_server; + return { + ...unit, + ext: { + ...unit.ext, + trusted_server: { + ...(typeof prior === 'object' && prior !== null && !Array.isArray(prior) ? prior : {}), + trace_slot_ref: refs[index], + }, + }, + }; + }); + return { + request: { ...request, adUnits }, + slotRefs: Object.freeze(refs), + collector, + }; + } catch { + return fallback; + } +} + +function ownMember( + value: unknown, + key: string +): { present: boolean; valid: boolean; value?: unknown } { + if (value === null || typeof value !== 'object') return { present: false, valid: true }; + try { + const member = Object.getOwnPropertyDescriptor(value, key); + return member + ? { + present: true, + valid: Object.prototype.hasOwnProperty.call(member, 'value'), + value: member.value, + } + : { present: false, valid: true }; + } catch { + return { present: true, valid: false }; + } +} + +/** Consumes optional API evidence before bids, without inspecting unrelated extensions. */ +export function observeTraceApiResponse( + carry: Pick | undefined, + response: unknown +): void { + if (!carry) return; + try { + const ext = ownMember(response, 'ext'); + if (!ext.present) return; + if (!ext.valid) { + carry.collector.recordTransport(null); + return; + } + const trusted = ownMember(ext.value, 'trusted_server'); + if (!trusted.present) return; + if (!trusted.valid) { + carry.collector.recordTransport(null); + return; + } + const member = ownMember(trusted.value, 'trace_auction'); + if (!member.present) return; + const transport = member.valid ? parseTraceAuctionTransport(member.value) : undefined; + if (!transport || (transport.evidence && transport.evidence.source !== 'auction_api')) { + carry.collector.recordTransport(null); + return; + } + if (transport.evidence && carry.slotRefs.length) { + const expected = new Set(carry.slotRefs); + const returned = transport.evidence.slots.map((slot) => slot.slot_ref); + if (returned.some((ref) => !expected.has(ref)) || new Set(returned).size !== returned.length) + carry.collector.recordInterpretationIssue('correlation_unavailable'); + } + carry.collector.recordTransport(transport); + } catch { + /* Diagnostic callbacks never change ordinary bid parsing. */ + } +} + +/** Records only a bounded API transport failure, without errors or response bodies. */ +export function observeTraceApiFailure( + carry: Pick | undefined +): void { + try { + carry?.collector.recordTransportFailure(); + } catch { + /* Diagnostic failures never change ordinary bidding. */ + } +} diff --git a/crates/trusted-server-js/lib/src/trace/setup.ts b/crates/trusted-server-js/lib/src/trace/setup.ts new file mode 100644 index 000000000..01db45aa6 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/setup.ts @@ -0,0 +1,143 @@ +import { validateTraceRequestContext } from './context'; +import { changeTraceSession, type TraceSessionAction } from './lifecycle'; +import type { CookieHealth } from './types'; + +const REPRODUCE = + 'Return to the affected page, reload once, reproduce the problem, then select View trace results.'; +const RECOVERY = + 'Reopen the affected article on the exact same hostname and in this same tab, then reload once.'; +const ON = 'Tracing is on — cookie observed by server'; +const OFF = 'Tracing is off — no valid diagnostics session observed'; + +function healthLabel(health: CookieHealth): string { + switch (health.state) { + case 'absent': + return 'Not present in this request'; + case 'present_valid': + return 'Valid shape observed'; + case 'present_invalid': + return 'Invalid shape observed'; + case 'duplicate': + return 'Multiple values observed'; + case 'unavailable': + return health.detail === 'runtime_header_ambiguous' + ? 'Unavailable — runtime-visible cookies could not be reliably inspected' + : 'Unavailable — runtime-visible cookie inspection failed'; + } +} + +function fact(root: Document, list: Element, name: string, value: string): void { + const term = root.createElement('dt'); + term.textContent = name; + const description = root.createElement('dd'); + description.textContent = value; + list.append(term, description); +} + +function setupFacts(root: Document): void { + const source = root.getElementById('trace-request-context'); + const network = root.getElementById('trace-network-facts'); + const cookies = root.getElementById('trace-cookie-facts'); + if (!source || !network || !cookies) return; + let context: unknown; + try { + context = JSON.parse(source.textContent ?? ''); + } catch { + context = undefined; + } + source.textContent = ''; + network.replaceChildren(); + cookies.replaceChildren(); + if (!validateTraceRequestContext(context)) { + fact(root, network, 'Request facts', 'Unavailable'); + fact(root, cookies, 'Cookie health', 'Unavailable'); + return; + } + const facts = context.network; + fact(root, network, 'Approximate network identifier', facts.masked_client_ip ?? 'Unavailable'); + fact(root, network, 'Country', facts.country ?? 'Unavailable'); + fact(root, network, 'Region', facts.region ?? 'Unavailable'); + fact(root, network, 'ASN', facts.asn === undefined ? 'Unavailable' : String(facts.asn)); + fact(root, network, 'HTTP version', facts.http_version ?? 'Unavailable'); + fact(root, network, 'TLS protocol', facts.tls_protocol ?? 'Unavailable'); + fact(root, network, 'TLS cipher', facts.tls_cipher ?? 'Unavailable'); + fact(root, network, 'Edge hostname', facts.edge_hostname ?? 'Unavailable'); + fact(root, network, 'Edge region', facts.edge_region ?? 'Unavailable'); + fact(root, network, 'Edge POP', facts.edge_pop ?? 'Unavailable'); + fact(root, cookies, 'Edge Cookie', healthLabel(context.cookies.ts_ec)); + fact(root, cookies, 'External IDs', healthLabel(context.cookies.ts_eids)); + fact(root, cookies, 'Tester', healthLabel(context.cookies.ts_tester)); + fact(root, cookies, 'Diagnostics session', healthLabel(context.cookies.diagnostics_session)); +} + +/** Mounts explicit setup controls without changing cookies on page load. */ +export function mountTraceSetup(root: Document = document): () => void { + setupFacts(root); + const state = root.getElementById('trace-session-state'); + const status = root.getElementById('trace-status'); + const enable = root.getElementById('trace-enable'); + const end = root.getElementById('trace-end'); + const back = root.getElementById('trace-back'); + if ( + !state || + !status || + !(enable instanceof HTMLButtonElement) || + !(end instanceof HTMLButtonElement) || + !(back instanceof HTMLButtonElement) + ) + return () => undefined; + state.textContent = + state.dataset.observedActive === 'true' + ? ON + : state.dataset.observedActive === 'false' + ? OFF + : 'Tracing state unconfirmed'; + let pending = false; + let destroyed = false; + const action = async (requested: TraceSessionAction): Promise => { + if (destroyed || pending) return; + pending = true; + enable.disabled = end.disabled = true; + status.textContent = + requested === 'enable' ? 'Verifying activation…' : 'Verifying deactivation…'; + try { + const result = await changeTraceSession(requested); + if (destroyed) return; + state.textContent = + result.observation === 'active' + ? ON + : result.observation === 'inactive' + ? OFF + : 'Tracing state unconfirmed'; + if (result.confirmed) { + status.textContent = requested === 'enable' ? REPRODUCE : OFF; + back.classList.toggle('primary', requested === 'enable'); + } else { + status.textContent = `${requested === 'enable' ? 'Activation' : 'Deactivation'} unconfirmed. Try again.`; + } + } finally { + if (!destroyed) enable.disabled = end.disabled = false; + pending = false; + } + }; + const enableClick = (): void => { + void action('enable'); + }; + const endClick = (): void => { + void action('end'); + }; + const backClick = (): void => { + if (destroyed) return; + if (window.history.length > 1) window.history.back(); + else status.textContent = RECOVERY; + }; + enable.addEventListener('click', enableClick); + end.addEventListener('click', endClick); + back.addEventListener('click', backClick); + return () => { + destroyed = true; + enable.removeEventListener('click', enableClick); + end.removeEventListener('click', endClick); + back.removeEventListener('click', backClick); + }; +} diff --git a/crates/trusted-server-js/lib/src/trace/shape.ts b/crates/trusted-server-js/lib/src/trace/shape.ts new file mode 100644 index 000000000..888ec96d9 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/shape.ts @@ -0,0 +1,32 @@ +import { traceInteger, traceOwn } from './context'; + +/** Accepts a dense bounded JSON array with no added fields or accessors. */ +export function traceArray(value: unknown, limit: number): value is unknown[] { + return traceItems(value, limit) !== undefined; +} + +/** Copies bounded own array data without using caller methods or iterators. */ +export function traceItems(value: unknown, limit: number): unknown[] | undefined { + if (!Array.isArray(value) || Object.getPrototypeOf(value) !== Array.prototype) return undefined; + const length = Object.getOwnPropertyDescriptor(value, 'length')?.value; + if (!traceInteger(length, limit)) return undefined; + const keys = Reflect.ownKeys(value); + if (keys.length !== length + 1) return undefined; + if ( + !keys.every((key) => { + if (key === 'length') return true; + if (typeof key !== 'string' || !/^(?:0|[1-9]\d*)$/.test(key) || Number(key) >= length) + return false; + const property = Object.getOwnPropertyDescriptor(value, key); + return property?.enumerable === true && traceOwn(property, 'value'); + }) + ) + return undefined; + const owned: unknown[] = []; + for (let index = 0; index < length; index += 1) { + const property = Object.getOwnPropertyDescriptor(value, String(index)); + if (!property || property.enumerable !== true || !traceOwn(property, 'value')) return undefined; + owned.push(property.value); + } + return owned; +} diff --git a/crates/trusted-server-js/lib/src/trace/storage.ts b/crates/trusted-server-js/lib/src/trace/storage.ts new file mode 100644 index 000000000..b769c6a39 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/storage.ts @@ -0,0 +1,96 @@ +import { traceObject } from './context'; +import type { TraceStoredReportV1 } from './report-types'; +import { + parseTraceStoredReport, + traceJsonSnapshot, + traceReportRejection, + type TraceReportRejection, +} from './report-validation'; + +/** One explicit snapshot replaces the prior report in this browsing context. */ +export const TRACE_REPORT_STORAGE_KEY = 'trusted-server.trace.report.v1'; +type TraceStorage = Pick; +export type TraceStorageReadResult = + | { readonly status: 'ready'; readonly value: TraceStoredReportV1 } + | { readonly status: 'absent' | 'unavailable' } + | { readonly status: 'rejected'; readonly reason: TraceReportRejection }; +export type TraceStorageWriteResult = + | { readonly status: 'stored' | 'unavailable' } + | { readonly status: 'rejected'; readonly reason: TraceReportRejection }; + +function storageOrDefault(storage?: TraceStorage): TraceStorage { + return storage ?? window.sessionStorage; +} +function rejected(value: unknown, origin: string, nowMs: number): TraceReportRejection { + const owned = traceJsonSnapshot(value, 11)?.value; + if (!traceObject(owned)) return 'invalid_report'; + return traceReportRejection(owned.report, origin, nowMs) ?? 'invalid_report'; +} + +/** Reads and validates once; rejected entries are ignored even if deletion fails. */ +export function readTraceReport( + origin: string, + nowMs: number, + storage?: TraceStorage +): TraceStorageReadResult { + let available: TraceStorage; + let serialized: string | null; + try { + available = storageOrDefault(storage); + serialized = available.getItem(TRACE_REPORT_STORAGE_KEY); + } catch { + return { status: 'unavailable' }; + } + if (serialized === null) return { status: 'absent' }; + let value: unknown; + let result: TraceStoredReportV1 | undefined; + try { + // Bound serialized UTF-8 before parsing, including otherwise ignorable whitespace. + if ( + typeof serialized === 'string' && + new TextEncoder().encode(serialized).length <= 512 * 1024 + ) { + value = JSON.parse(serialized) as unknown; + result = parseTraceStoredReport(value, origin, nowMs); + } + } catch { + /* A bounded rejection is returned below without parser text. */ + } + if (result) return { status: 'ready', value: result }; + const reason = rejected(value, origin, nowMs); + try { + available.removeItem(TRACE_REPORT_STORAGE_KEY); + } catch { + /* Rejected data remains ignored. */ + } + return { status: 'rejected', reason }; +} + +/** Stores only a fresh validated wrapper after an explicit capture action. */ +export function storeTraceReport( + value: unknown, + origin: string, + nowMs: number, + storage?: TraceStorage +): TraceStorageWriteResult { + const owned = parseTraceStoredReport(value, origin, nowMs); + if (!owned) return { status: 'rejected', reason: rejected(value, origin, nowMs) }; + try { + storageOrDefault(storage).setItem(TRACE_REPORT_STORAGE_KEY, JSON.stringify(owned)); + return { status: 'stored' }; + } catch { + return { status: 'unavailable' }; + } +} + +/** Deletes only the trace-owned key on an explicit cleanup action. */ +export function deleteTraceReport(storage?: TraceStorage): { + readonly status: 'deleted' | 'unavailable'; +} { + try { + storageOrDefault(storage).removeItem(TRACE_REPORT_STORAGE_KEY); + return { status: 'deleted' }; + } catch { + return { status: 'unavailable' }; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/types.ts b/crates/trusted-server-js/lib/src/trace/types.ts new file mode 100644 index 000000000..789dd6898 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/types.ts @@ -0,0 +1,151 @@ +/** A bounded shape fact about one owned runtime-visible cookie, never its value. */ +export type CookieHealth = Readonly< + | { source: 'request'; state: 'absent' } + | { + source: 'request'; + state: 'present_valid'; + detail: + | 'valid_ec_format' + | 'valid_eids_format' + | 'valid_tester_value' + | 'valid_diagnostics_value'; + } + | { + source: 'request'; + state: 'present_invalid'; + detail: 'malformed' | 'oversized' | 'unsupported_value'; + } + | { source: 'request'; state: 'duplicate'; detail: 'multiple_values' } + | { + source: 'request'; + state: 'unavailable'; + detail: 'header_too_large' | 'header_not_utf8' | 'runtime_header_ambiguous'; + } +>; + +/** The server-produced allowlist of redacted request facts. */ +export interface TraceRequestContextV1 { + readonly schema_version: 1; + readonly captured_at: string; + readonly network: { + readonly masked_client_ip?: string; + readonly country?: string; + readonly region?: string; + readonly asn?: number; + readonly http_version?: string; + readonly tls_protocol?: string; + readonly tls_cipher?: string; + readonly edge_hostname?: string; + readonly edge_region?: string; + readonly edge_pop?: string; + }; + readonly cookies: { + readonly ts_ec: CookieHealth; + readonly ts_eids: CookieHealth; + readonly ts_tester: CookieHealth; + readonly diagnostics_session: CookieHealth; + }; +} + +/** Exact server-assigned call sites, independent of browser delivery intent. */ +export const TRACE_AUCTION_SOURCES = [ + 'initial_navigation_ssat', + 'spa_page_bids', + 'auction_api', +] as const; +export const TRACE_TERMINAL_STATUSES = [ + 'completed', + 'execution_failed', + 'dispatch_failed', + 'abandoned', + 'skipped', +] as const; +export const TRACE_TERMINAL_REASONS = [ + 'policy_skipped', + 'no_eligible_slots', + 'no_provider_launched', + 'provider_execution_failed', + 'collection_failed', + 'unknown', +] as const; +export const TRACE_PROVIDER_ROLES = ['bidder', 'mediator', 'unknown'] as const; +export const TRACE_PROVIDER_STATUSES = [ + 'success', + 'no_bid', + 'error', + 'pending', + 'abandoned', + 'unknown', +] as const; +export const TRACE_CANDIDATES = [ + 'selected', + 'no_candidate', + 'selected_unrenderable', + 'unknown', +] as const; + +/** Bounded auction-wide provider observations, without provider identity. */ +export interface TraceProviderCall { + readonly provider_number: number; + readonly role: (typeof TRACE_PROVIDER_ROLES)[number]; + readonly status: (typeof TRACE_PROVIDER_STATUSES)[number]; + readonly response_time_ms?: number; + readonly returned_bid_count: number; +} + +/** Bounded slot candidate facts using only an opaque auction-local reference. */ +export interface TraceAuctionSlot { + readonly slot_number: number; + readonly slot_ref: string; + readonly requested_sizes: readonly (readonly [number, number])[]; + readonly returned_bid_count: number; + readonly candidate: (typeof TRACE_CANDIDATES)[number]; + readonly selected_creative_size?: readonly [number, number]; +} + +/** Private SSAT/SPA identity carried into the existing GPT opportunity binding. */ +export interface TraceGptIdentity { + readonly diagnostic_auction_id: string; + readonly slot_ref: string; +} + +/** Server-produced public auction facts, distinct from auction/telemetry inputs. */ +export interface TraceAuctionEvidenceV1 { + readonly schema_version: 1; + readonly diagnostic_auction_id: string; + readonly source: (typeof TRACE_AUCTION_SOURCES)[number]; + readonly terminal_status: (typeof TRACE_TERMINAL_STATUSES)[number]; + readonly terminal_reason?: (typeof TRACE_TERMINAL_REASONS)[number]; + readonly total_time_ms?: number; + readonly provider_calls: readonly TraceProviderCall[]; + readonly slots: readonly TraceAuctionSlot[]; + readonly truncation: { + readonly omitted_provider_calls: number; + readonly omitted_slots: number; + readonly omitted_nested_values: number; + }; + readonly coverage: { readonly provider_to_slot_no_bid: 'unavailable' }; +} + +/** Exactly one server evidence record or a bounded projection failure marker. */ +export type TraceAuctionTransportV1 = Readonly< + | { + schema_version: 1; + evidence: TraceAuctionEvidenceV1; + unavailable_reason?: never; + } + | { + schema_version: 1; + unavailable_reason: 'evidence_projection_failed'; + evidence?: never; + } +>; + +/** Exact decision made by the existing GPT slot/request-cycle recorder. */ +export interface TraceSlotCorrelationV1 { + readonly schema_version: 1; + readonly diagnostic_auction_id: string; + readonly slot_ref: string; + readonly runtime_slot_number: number; + readonly request_number: number; +} diff --git a/crates/trusted-server-js/lib/src/trace/validation.ts b/crates/trusted-server-js/lib/src/trace/validation.ts new file mode 100644 index 000000000..56900ca7b --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/validation.ts @@ -0,0 +1,289 @@ +import { traceItems } from './shape'; +import { traceJsonSnapshot } from './json'; +export { traceArray, traceItems } from './shape'; +import { traceInteger, traceKeys, traceObject, traceOwn } from './context'; +import { + TRACE_AUCTION_SOURCES, + TRACE_TERMINAL_STATUSES, + TRACE_TERMINAL_REASONS, + TRACE_PROVIDER_ROLES, + TRACE_PROVIDER_STATUSES, + TRACE_CANDIDATES, + type TraceAuctionEvidenceV1, + type TraceAuctionTransportV1, + type TraceSlotCorrelationV1, + type TraceProviderCall, + type TraceAuctionSlot, +} from './types'; + +const AUCTION_TOKEN = /^ts-auc-[0-9a-f]{12}4[0-9a-f]{3}[89ab][0-9a-f]{15}$/; +const SLOT_TOKEN = /^ts-slot-[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; + +/** Checks the exact public auction token without normalization. */ +export function validDiagnosticAuctionId(value: unknown): value is string { + return typeof value === 'string' && value.length === 39 && AUCTION_TOKEN.test(value); +} + +/** Checks the exact opaque slot-reference token without normalization. */ +export function validTraceSlotRef(value: unknown): value is string { + return typeof value === 'string' && value.length === 44 && SLOT_TOKEN.test(value); +} + +/** Requires an exact enum member without coercion. */ +export function traceEnum(value: unknown, members: readonly T[]): value is T { + return typeof value === 'string' && members.includes(value as T); +} + +/** Validates a requested/creative dimension, or an explicitly zero-capable CSS box. */ +export function traceSize(value: unknown, allowZero = false): value is [number, number] { + const items = traceItems(value, 2); + return ( + items !== undefined && + items.length === 2 && + items.every((dimension) => traceInteger(dimension, 100_000) && (allowZero || dimension > 0)) + ); +} + +function optionalInteger(value: Record, name: string, maximum: number): boolean { + return !traceOwn(value, name) || traceInteger(value[name], maximum); +} + +function provider(value: unknown): boolean { + return ( + traceObject(value) && + traceKeys( + value, + ['provider_number', 'role', 'status', 'returned_bid_count'], + ['response_time_ms'] + ) && + traceInteger(value.provider_number, 65_535) && + value.provider_number > 0 && + traceEnum(value.role, TRACE_PROVIDER_ROLES) && + traceEnum(value.status, TRACE_PROVIDER_STATUSES) && + traceInteger(value.returned_bid_count, 65_535) && + optionalInteger(value, 'response_time_ms', 0xffff_ffff) + ); +} + +function slot(value: unknown): boolean { + return ( + traceObject(value) && + traceKeys( + value, + ['slot_number', 'slot_ref', 'requested_sizes', 'returned_bid_count', 'candidate'], + ['selected_creative_size'] + ) && + traceInteger(value.slot_number, 65_535) && + value.slot_number > 0 && + validTraceSlotRef(value.slot_ref) && + traceInteger(value.returned_bid_count, 65_535) && + traceEnum(value.candidate, TRACE_CANDIDATES) && + arrayMembers(value.requested_sizes, 16, (size) => traceSize(size)) && + (!traceOwn(value, 'selected_creative_size') || traceSize(value.selected_creative_size)) + ); +} + +function evidence(value: unknown): value is TraceAuctionEvidenceV1 { + return ( + traceObject(value) && + traceKeys( + value, + [ + 'schema_version', + 'diagnostic_auction_id', + 'source', + 'terminal_status', + 'provider_calls', + 'slots', + 'truncation', + 'coverage', + ], + ['terminal_reason', 'total_time_ms'] + ) && + value.schema_version === 1 && + validDiagnosticAuctionId(value.diagnostic_auction_id) && + traceEnum(value.source, TRACE_AUCTION_SOURCES) && + traceEnum(value.terminal_status, TRACE_TERMINAL_STATUSES) && + (!traceOwn(value, 'terminal_reason') || + traceEnum(value.terminal_reason, TRACE_TERMINAL_REASONS)) && + optionalInteger(value, 'total_time_ms', 0xffff_ffff) && + arrayMembers(value.provider_calls, 16, provider) && + arrayMembers(value.slots, 64, slot) && + traceObject(value.truncation) && + traceKeys(value.truncation, [ + 'omitted_provider_calls', + 'omitted_slots', + 'omitted_nested_values', + ]) && + Object.values(value.truncation).every((count) => traceInteger(count, 65_535)) && + traceObject(value.coverage) && + traceKeys(value.coverage, ['provider_to_slot_no_bid']) && + value.coverage.provider_to_slot_no_bid === 'unavailable' + ); +} + +function arrayMembers(value: unknown, limit: number, valid: (item: unknown) => boolean): boolean { + const items = traceItems(value, limit); + return items !== undefined && items.every(valid); +} + +/** Validates the exact server-produced evidence model without modifying it. */ +export function validateTraceAuctionEvidence(value: unknown): value is TraceAuctionEvidenceV1 { + try { + const snapshot = traceJsonSnapshot(value); + return snapshot !== undefined && evidence(snapshot.value); + } catch { + return false; + } +} + +/** Validates the exclusive evidence-or-unavailable transport envelope. */ +function transport(value: unknown): value is TraceAuctionTransportV1 { + try { + if (!traceObject(value) || value.schema_version !== 1) return false; + if (traceOwn(value, 'evidence')) + return traceKeys(value, ['schema_version', 'evidence']) && evidence(value.evidence); + return ( + traceKeys(value, ['schema_version', 'unavailable_reason']) && + value.unavailable_reason === 'evidence_projection_failed' + ); + } catch { + return false; + } +} + +/** Validates a descriptor-owned observation of the exact transport envelope. */ +export function validateTraceAuctionTransport(value: unknown): value is TraceAuctionTransportV1 { + const snapshot = traceJsonSnapshot(value); + return snapshot !== undefined && transport(snapshot.value); +} + +/** Validates the trace-only exact-token GPT correlation sidecar. */ +function correlation(value: unknown): value is TraceSlotCorrelationV1 { + try { + return ( + traceObject(value) && + traceKeys(value, [ + 'schema_version', + 'diagnostic_auction_id', + 'slot_ref', + 'runtime_slot_number', + 'request_number', + ]) && + value.schema_version === 1 && + validDiagnosticAuctionId(value.diagnostic_auction_id) && + validTraceSlotRef(value.slot_ref) && + traceInteger(value.runtime_slot_number) && + value.runtime_slot_number > 0 && + traceInteger(value.request_number) && + value.request_number > 0 + ); + } catch { + return false; + } +} + +/** Validates one descriptor-owned observation of an exact GPT sidecar. */ +export function validateTraceSlotCorrelation(value: unknown): value is TraceSlotCorrelationV1 { + const snapshot = traceJsonSnapshot(value); + return snapshot !== undefined && correlation(snapshot.value); +} + +/** Copies a validated envelope into an immutable public model. */ +export function parseTraceAuctionTransport(value: unknown): TraceAuctionTransportV1 | undefined { + try { + const snapshot = traceJsonSnapshot(value); + if (!snapshot) return undefined; + value = snapshot.value; + if (!transport(value)) return undefined; + if (value.evidence === undefined) + return Object.freeze({ + schema_version: 1, + unavailable_reason: 'evidence_projection_failed', + }); + const original = value.evidence; + const copied: TraceAuctionTransportV1 = Object.freeze({ + schema_version: 1, + evidence: Object.freeze({ + schema_version: 1, + diagnostic_auction_id: original.diagnostic_auction_id, + source: original.source, + terminal_status: original.terminal_status, + ...(traceOwn(original, 'terminal_reason') + ? { terminal_reason: original.terminal_reason } + : {}), + ...(traceOwn(original, 'total_time_ms') ? { total_time_ms: original.total_time_ms } : {}), + provider_calls: Object.freeze(copyItems(original.provider_calls, 16, copyProvider)), + slots: Object.freeze(copyItems(original.slots, 64, copySlot)), + truncation: Object.freeze({ + omitted_provider_calls: original.truncation.omitted_provider_calls, + omitted_slots: original.truncation.omitted_slots, + omitted_nested_values: original.truncation.omitted_nested_values, + }), + coverage: Object.freeze({ + provider_to_slot_no_bid: original.coverage.provider_to_slot_no_bid, + }), + }), + }); + return transport(copied) ? copied : undefined; + } catch { + return undefined; + } +} + +function copySize(value: readonly [number, number]): readonly [number, number] { + const items = traceItems(value, 2); + if (!items || items.length !== 2) throw new Error('Invalid trace size'); + return Object.freeze([items[0] as number, items[1] as number]); +} + +function copyItems(value: readonly T[], limit: number, copy: (item: T) => U): U[] { + const items = traceItems(value, limit); + if (!items) throw new Error('Invalid trace array'); + const owned: U[] = []; + for (let index = 0; index < items.length; index += 1) owned.push(copy(items[index] as T)); + return owned; +} + +function copyProvider(value: TraceProviderCall): TraceProviderCall { + return Object.freeze({ + provider_number: value.provider_number, + role: value.role, + status: value.status, + returned_bid_count: value.returned_bid_count, + ...(traceOwn(value, 'response_time_ms') ? { response_time_ms: value.response_time_ms } : {}), + }); +} + +function copySlot(value: TraceAuctionSlot): TraceAuctionSlot { + return Object.freeze({ + slot_number: value.slot_number, + slot_ref: value.slot_ref, + requested_sizes: Object.freeze(copyItems(value.requested_sizes, 16, copySize)), + returned_bid_count: value.returned_bid_count, + candidate: value.candidate, + ...(traceOwn(value, 'selected_creative_size') + ? { selected_creative_size: copySize(value.selected_creative_size!) } + : {}), + }); +} + +/** Copies a validated correlation decision into an immutable public model. */ +export function parseTraceSlotCorrelation(value: unknown): TraceSlotCorrelationV1 | undefined { + try { + const snapshot = traceJsonSnapshot(value); + if (!snapshot) return undefined; + value = snapshot.value; + if (!correlation(value)) return undefined; + const copied: TraceSlotCorrelationV1 = Object.freeze({ + schema_version: 1, + diagnostic_auction_id: value.diagnostic_auction_id, + slot_ref: value.slot_ref, + runtime_slot_number: value.runtime_slot_number, + request_number: value.request_number, + }); + return correlation(copied) ? copied : undefined; + } catch { + return undefined; + } +} diff --git a/crates/trusted-server-js/lib/src/trace/viewer.css b/crates/trusted-server-js/lib/src/trace/viewer.css new file mode 100644 index 000000000..7ca482803 --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/viewer.css @@ -0,0 +1,95 @@ +:root { + color-scheme: light dark; + font: + 1rem/1.5 system-ui, + sans-serif; + background: Canvas; + color: CanvasText; +} + +* { + box-sizing: border-box; +} + +body { + margin: 0; + padding: max(1rem, env(safe-area-inset-top)) max(1rem, env(safe-area-inset-right)) + max(1rem, env(safe-area-inset-bottom)) max(1rem, env(safe-area-inset-left)); + overflow-wrap: anywhere; +} + +main { + max-width: 60rem; + margin: auto; +} +h1 { + font-size: 1.6rem; + line-height: 1.3; +} +h2 { + font-size: 1.25rem; +} +section { + margin-block: 1.5rem; +} +dt { + font-weight: 650; + margin-top: 0.75rem; +} +dd { + margin-inline: 0; +} +.controls { + display: flex; + flex-wrap: wrap; + gap: 0.75rem; +} +button, +.button { + min-width: 44px; + min-height: 44px; + padding: 0.75rem 1rem; + font: inherit; + color: inherit; + background: Canvas; + border: 2px solid currentColor; + border-radius: 0.4rem; + cursor: pointer; + white-space: normal; +} + +.primary { + background: #17478b; + color: #fff; + border-color: #17478b; +} +:focus-visible { + outline: 3px solid #ca6800; + outline-offset: 3px; +} +button:disabled { + cursor: wait; + opacity: 0.65; +} +[role='status'] { + min-height: 3rem; +} +pre { + white-space: pre-wrap; + overflow-wrap: anywhere; +} +table { + width: 100%; + table-layout: fixed; + border-collapse: collapse; +} +th, +td { + padding: 0.5rem; + text-align: start; + vertical-align: top; + overflow-wrap: anywhere; +} +[hidden] { + display: none; +} diff --git a/crates/trusted-server-js/lib/src/trace/viewer.ts b/crates/trusted-server-js/lib/src/trace/viewer.ts new file mode 100644 index 000000000..4875c2c6b --- /dev/null +++ b/crates/trusted-server-js/lib/src/trace/viewer.ts @@ -0,0 +1,8 @@ +import { mountTraceViewer } from './report-view'; +import './viewer.css'; + +if (document.readyState === 'loading') { + document.addEventListener('DOMContentLoaded', () => mountTraceViewer(), { once: true }); +} else { + mountTraceViewer(); +} diff --git a/crates/trusted-server-js/lib/test/core/index.test.ts b/crates/trusted-server-js/lib/test/core/index.test.ts index a02082b59..d326199a6 100644 --- a/crates/trusted-server-js/lib/test/core/index.test.ts +++ b/crates/trusted-server-js/lib/test/core/index.test.ts @@ -9,6 +9,7 @@ describe('core/index', () => { await vi.resetModules(); document.body.innerHTML = ''; delete window.tsjs; + delete window.__tsjs_trace_active; }); afterEach(() => { @@ -28,6 +29,28 @@ describe('core/index', () => { expect(typeof api.getConfig).toBe('function'); expect(typeof api.requestAds).toBe('function'); }); + it('installs the literal gated shared trace facade before queued publisher callbacks', async () => { + window.__tsjs_trace_active = true; + let observed: unknown; + window.tsjs = { + que: [ + () => { + observed = window.tsjs?.traceEvidence; + }, + ], + } as TsjsApi; + await import('../../src/core/index'); + expect(observed).toBeDefined(); + expect(observed).toBe(window.tsjs?.traceEvidence); + }); + it.each([undefined, false, 'true', 1])( + 'keeps the trace facade absent for nonliteral gate %s', + async (active) => { + window.__tsjs_trace_active = active; + await import('../../src/core/index'); + expect(window.tsjs?.traceEvidence).toBeUndefined(); + } + ); it('defaults adSlots and bids so gated-off pages never see undefined', async () => { await import('../../src/core/index'); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts index 540d15229..4ffa564be 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts @@ -10,6 +10,10 @@ import { resolveFirstImpressionElement, } from '../../../src/core/first_impression'; import type { AuctionBidData, TsjsApi } from '../../../src/core/types'; +import { installTraceRuntime } from '../../../src/trace/runtime'; +import { GptDiagnosticsStore } from '../../../src/integrations/gpt_diagnostics/store'; +import { gptTransport, joinedGptStore } from '../../trace/gpt-fixtures'; +import { AUCTION_TOKEN, SLOT_TOKEN } from '../../trace/fixtures'; import { APS_PREBID_CREATIVE_RUNNER_URL, APS_RENDERING_MODE_ATTRIBUTE_NAME, @@ -215,6 +219,7 @@ describe('installTsAdInit', () => { }); afterEach(() => { + delete window.__tsjs_trace_active; document.getElementById('div-atf-sidebar')?.remove(); document.getElementById('div-new-slot')?.remove(); document.getElementById('div-atf-sidebar-2')?.remove(); @@ -224,6 +229,115 @@ describe('installTsAdInit', () => { document.querySelectorAll('[data-responsive-slot-test]').forEach((element) => element.remove()); }); + it.each(['bootstrap', 'bundle'] as const)( + 'binds the actual GPT cycle to the exact delivered slot in %s adInit', + async (implementation) => { + const { mockSlot } = configureOpportunityDiagnostics(undefined, vi.fn()); + const ts = (window as TestWindow).tsjs as TsjsApi; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + ts.adSlots![0]!.ext = { trusted_server: { trace_slot_ref: SLOT_TOKEN } }; + const store = new GptDiagnosticsStore({ + onTraceCorrelation: (value) => collector.recordCorrelation(value), + }); + ts.gptDiagnosticsRecorder = store; + ts.traceGpt!.observeTransport(ts.adSlots, gptTransport(), 'initial_navigation_ssat'); + await installHandoff(implementation); + ts.adInit!(); + expect(collector.snapshot().value?.slotCorrelations).toEqual([]); + store.recordSlotRequested(mockSlot); + expect(collector.snapshot().value?.slotCorrelations).toEqual([ + { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }, + ]); + expect(store.snapshot().slots[0]!.requests[0]).toMatchObject({ + requestPath: 'trusted_server_direct', + trustedServerOpportunity: 'no_candidate', + trustedServerAuctionId: AUCTION_TOKEN, + }); + expect(joinedGptStore(store, collector)?.auctions[0]?.slots[0]?.correlation).toBe('matched'); + expect(JSON.stringify(store.snapshot())).not.toContain(SLOT_TOKEN); + } + ); + + it.each(['bootstrap', 'bundle'] as const)( + 'retains the validated binding through the %s publisher first-impression fallback', + async (implementation) => { + vi.useFakeTimers(); + try { + const { mockPubads, mockSlot } = configureOpportunityDiagnostics( + { hb_pb: '1.10', hb_adid: 'example-creative', adm: '
Example
' }, + vi.fn() + ); + const ts = (window as TestWindow).tsjs as TsjsApi; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + ts.adSlots![0]!.ext = { trusted_server: { trace_slot_ref: SLOT_TOKEN } }; + const store = new GptDiagnosticsStore({ + onTraceCorrelation: (value) => collector.recordCorrelation(value), + }); + ts.gptDiagnosticsRecorder = store; + ts.traceGpt!.observeTransport(ts.adSlots, gptTransport(), 'initial_navigation_ssat'); + registerPublisherFirstImpressionAuctions(ts, ['div-atf-sidebar']); + await installHandoff(implementation); + ts.adInit!(); + expect(mockPubads.refresh).not.toHaveBeenCalled(); + expect(collector.snapshot().value?.slotCorrelations).toEqual([]); + vi.advanceTimersByTime(5001); + expect(mockPubads.refresh).toHaveBeenCalledOnce(); + store.recordSlotRequested(mockSlot); + expect(collector.snapshot().value?.slotCorrelations).toEqual([ + { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }, + ]); + expect(store.snapshot().slots[0]!.requests[0]).toMatchObject({ + requestPath: 'trusted_server_direct', + trustedServerOpportunity: 'renderable_candidate', + trustedServerAuctionId: AUCTION_TOKEN, + }); + expect(joinedGptStore(store, collector)?.auctions[0]?.slots[0]?.correlation).toBe( + 'matched' + ); + } finally { + vi.useRealTimers(); + } + } + ); + + it.each(['bootstrap', 'bundle'] as const)( + 'preserves a conflicting ordinary auction marker without a fabricated sidecar in %s', + async (implementation) => { + const marker = 'ts-auc-2234567812344abc8def123456789abc'; + const { mockSlot } = configureOpportunityDiagnostics({ hb_auction_id: marker }, vi.fn()); + const ts = (window as TestWindow).tsjs as TsjsApi; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + ts.adSlots![0]!.ext = { trusted_server: { trace_slot_ref: SLOT_TOKEN } }; + const store = new GptDiagnosticsStore({ + onTraceCorrelation: (value) => collector.recordCorrelation(value), + }); + ts.gptDiagnosticsRecorder = store; + ts.traceGpt!.observeTransport(ts.adSlots, gptTransport(), 'initial_navigation_ssat'); + await installHandoff(implementation); + ts.adInit!(); + store.recordSlotRequested(mockSlot); + expect(store.snapshot().slots[0]!.requests[0]!.trustedServerAuctionId).toBe(marker); + expect(collector.snapshot().value?.slotCorrelations).toEqual([]); + expect(collector.snapshot().value?.issues).toEqual(['correlation_unavailable']); + expect(joinedGptStore(store, collector)?.auctions[0]?.slots[0]?.correlation).toBe('unknown'); + } + ); + function configureOpportunityDiagnostics( bid: AuctionBidData | undefined, recordTrustedServerOpportunity: ReturnType, diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts index 6ca610547..a68b41ffe 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts @@ -4,6 +4,8 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import type { TsjsApi } from '../../../src/core/types'; import { GPT_BOOTSTRAP_PATH } from '../../fixtures/paths'; +import { installTraceRuntime } from '../../../src/trace/runtime'; +import { AUCTION_TOKEN, SLOT_TOKEN } from '../../trace/fixtures'; type TestWindow = Window & { googletag?: unknown; @@ -83,6 +85,7 @@ describe('scheduleInitialAdInit', () => { }); afterEach(() => { + delete window.__tsjs_trace_active; history.pushState = originalPushState; history.replaceState = originalReplaceState; // Reset jsdom location back to root for the next test. @@ -98,6 +101,126 @@ describe('scheduleInitialAdInit', () => { vi.unstubAllGlobals(); }); + it('records owned initial evidence after the final generation guard and before adInit', async () => { + readyState = 'complete'; + await importGptModule(); + window.__tsjs_trace_active = true; + const ts = (window as TestWindow).tsjs!; + installTraceRuntime(ts); + const evidence = { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + source: 'initial_navigation_ssat', + terminal_status: 'completed', + provider_calls: [], + slots: [ + { + slot_number: 1, + slot_ref: SLOT_TOKEN, + requested_sizes: [[300, 250]], + candidate: 'no_candidate', + returned_bid_count: 0, + }, + ], + truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }; + const slots = [ + { + id: 'example', + gam_unit_path: '/example/ad', + div_id: 'example', + formats: [[300, 250]] as Array<[number, number]>, + ext: { trusted_server: { trace_slot_ref: SLOT_TOKEN } }, + }, + ]; + const bids = { example: { hb_pb: '1.00' } }; + ts.adInit = vi.fn(() => { + expect(ts.traceEvidence!.snapshot().value?.serverAuctions).toEqual([evidence]); + expect(ts.adSlots).toBe(slots); + expect(ts.bids).toBe(bids); + }); + ts.scheduleInitialAdInit!(bids, slots, { schema_version: 1, evidence }); + expect(ts.traceEvidence!.snapshot().value?.serverAuctions).toEqual([]); + flushFrame(); + flushFrame(); + expect(ts.adInit).toHaveBeenCalledOnce(); + expect(ts.traceEvidence!.snapshot().value?.serverAuctions).toEqual([evidence]); + }); + + it('does not capture an initial transport abandoned before hydration', async () => { + readyState = 'complete'; + await importGptModule(); + window.__tsjs_trace_active = true; + const ts = (window as TestWindow).tsjs!; + installTraceRuntime(ts); + ts.adInit = vi.fn(); + ts.scheduleInitialAdInit!(undefined, [], { + schema_version: 1, + unavailable_reason: 'evidence_projection_failed', + }); + ts.navGeneration = 1; + flushFrame(); + flushFrame(); + expect(ts.traceEvidence!.snapshot().value?.serverAuctions).toEqual([]); + expect(ts.adInit).not.toHaveBeenCalled(); + }); + + it.each(['absent', 'malformed', 'throwing-bridge', 'inactive'] as const)( + 'preserves ordinary adInit for an %s optional trace seam', + async (kind) => { + readyState = 'complete'; + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + window.__tsjs_trace_active = true; + installTraceRuntime(ts); + const observe = vi.fn(() => { + throw new Error('private-bridge'); + }); + if (kind === 'throwing-bridge' || kind === 'inactive') + ts.traceGpt = { ...ts.traceGpt!, observeTransport: observe }; + if (kind === 'inactive') window.__tsjs_trace_active = 'true'; + ts.adInit = vi.fn(); + ts.scheduleInitialAdInit!( + undefined, + [], + kind === 'absent' ? undefined : { schema_version: 1 } + ); + flushFrame(); + flushFrame(); + expect(ts.adInit).toHaveBeenCalledOnce(); + expect(ts.traceEvidence!.snapshot().value?.issues).toEqual( + kind === 'malformed' ? ['evidence_validation_failed'] : [] + ); + if (kind === 'inactive') expect(observe).not.toHaveBeenCalled(); + } + ); + + it.each(['bootstrap', 'bootstrap-to-bundle'] as const)( + 'captures the first scheduler claim through the shared bridge (%s)', + async (mode) => { + readyState = 'complete'; + runBootstrap(); + window.__tsjs_trace_active = true; + const ts = (window as TestWindow).tsjs!; + installTraceRuntime(ts); + ts.adInit = vi.fn(); + ts.scheduleInitialAdInit!(undefined, [], { + schema_version: 1, + unavailable_reason: 'evidence_projection_failed', + }); + if (mode === 'bootstrap-to-bundle') { + await importGptModule(); + ts.adInit = vi.fn(); + ts.scheduleInitialAdInit!(undefined, [], { schema_version: 1 }); + } + flushFrame(); + flushFrame(); + expect(ts.adInit).toHaveBeenCalledOnce(); + expect(ts.traceEvidence!.snapshot().value?.issues).toEqual(['evidence_projection_failed']); + } + ); + it('applies the SSR payload and defers adInit until window load plus two animation frames', async () => { await importGptModule(); const ts = (window as TestWindow).tsjs!; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts index 647de8967..3474be84b 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts @@ -4,6 +4,10 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import type { TsjsApi } from '../../../src/core/types'; import { GPT_BOOTSTRAP_PATH } from '../../fixtures/paths'; +import { installTraceRuntime } from '../../../src/trace/runtime'; +import { GptDiagnosticsStore } from '../../../src/integrations/gpt_diagnostics/store'; +import { gptTransport, joinedGptStore } from '../../trace/gpt-fixtures'; +import { AUCTION_TOKEN, SLOT_TOKEN } from '../../trace/fixtures'; type TestWindow = Window & { googletag?: unknown; @@ -54,6 +58,7 @@ describe('installSpaAuctionHook', () => { }); afterEach(() => { + delete window.__tsjs_trace_active; history.pushState = originalPushState; history.replaceState = originalReplaceState; // Reset jsdom location back to root for the next test. @@ -67,6 +72,158 @@ describe('installSpaAuctionHook', () => { vi.unstubAllGlobals(); }); + it.each(['canonical', 'legacy', 'malformed', 'absent', 'throwing-bridge', 'oversized'] as const)( + 'consumes optional %s transport only after an accepted navigation and before adInit', + async (kind) => { + const { installSpaAuctionHook } = await importGptModule(); + installSpaAuctionHook(); + const ts = (window as TestWindow).tsjs!; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + const slots = [ + { + id: 'example', + gam_unit_path: '/example/ad', + div_id: 'example', + formats: [[300, 250]] as Array<[number, number]>, + ext: { trusted_server: { trace_slot_ref: SLOT_TOKEN } }, + }, + ]; + if (kind === 'oversized') { + slots.push( + ...Array.from({ length: 64 }, () => ({ + ...slots[0]!, + ext: { trusted_server: { trace_slot_ref: 'private-invalid' } }, + })) + ); + } + document.body.innerHTML = '
'; + const transport = gptTransport('spa_page_bids'); + const payload = { + slots, + bids: {}, + ...(kind === 'absent' + ? {} + : { trace_auction: kind === 'malformed' ? { schema_version: 1 } : transport }), + }; + if (kind === 'legacy') fetchStub.mockResolvedValueOnce({ ok: false, status: 404 }); + fetchStub.mockResolvedValue({ ok: true, json: async () => payload }); + if (kind === 'throwing-bridge') + ts.traceGpt = { + ...ts.traceGpt!, + observePageBids: () => { + throw new Error('private-bridge'); + }, + }; + ts.adInit = vi.fn(() => { + expect(ts.adSlots).toBe(slots); + expect(collector.snapshot().value?.serverAuctions).toHaveLength( + ['canonical', 'legacy', 'oversized'].includes(kind) ? 1 : 0 + ); + }); + history.pushState({}, '', '/trace-route'); + await flushAsync(); + expect(ts.adInit).toHaveBeenCalledOnce(); + expect(collector.snapshot().value?.issues).toEqual( + kind === 'malformed' + ? ['evidence_validation_failed'] + : kind === 'oversized' + ? ['correlation_unavailable'] + : [] + ); + if (kind === 'oversized') { + expect(ts.adSlots).toHaveLength(65); + expect(ts.traceGpt!.identity(slots[0]!)).toBeUndefined(); + } + if (kind === 'legacy') expect(fetchStub).toHaveBeenCalledTimes(2); + } + ); + + it('joins the actual no-bid SPA GPT request with its validated pre-dispatch token', async () => { + const { installSpaAuctionHook, installTsAdInit } = await importGptModule(); + installSpaAuctionHook(); + const ts = (window as TestWindow).tsjs!; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + const slot = { + addService: vi.fn().mockReturnThis(), + setTargeting: vi.fn().mockReturnThis(), + clearTargeting: vi.fn().mockReturnThis(), + getSlotElementId: () => 'example', + getTargeting: () => [], + }; + const pubads = { + enableSingleRequest: vi.fn(), + getSlots: () => [slot], + addEventListener: vi.fn(), + refresh: vi.fn(), + }; + vi.stubGlobal('googletag', { + cmd: { push: (callback: () => void) => callback() }, + defineSlot: () => slot, + pubads: () => pubads, + enableServices: vi.fn(), + }); + const store = new GptDiagnosticsStore({ + onTraceCorrelation: (value) => collector.recordCorrelation(value), + }); + ts.gptDiagnosticsRecorder = store; + document.body.innerHTML = '
'; + fetchStub.mockResolvedValue({ + ok: true, + json: async () => ({ + slots: [ + { + id: 'example', + gam_unit_path: '/example/ad', + div_id: 'example', + formats: [[300, 250]], + ext: { trusted_server: { trace_slot_ref: SLOT_TOKEN } }, + }, + ], + bids: {}, + trace_auction: gptTransport('spa_page_bids'), + }), + }); + installTsAdInit(); + history.pushState({}, '', '/trace-no-bid'); + await flushAsync(); + store.recordSlotRequested(slot); + expect(store.snapshot().slots[0]!.requests[0]).toMatchObject({ + requestPath: 'trusted_server_direct', + trustedServerOpportunity: 'no_candidate', + trustedServerAuctionId: AUCTION_TOKEN, + }); + expect(joinedGptStore(store, collector)?.auctions[0]?.slots[0]?.correlation).toBe('matched'); + }); + + it('does not capture a superseded SPA response', async () => { + const { installSpaAuctionHook } = await importGptModule(); + installSpaAuctionHook(); + const ts = (window as TestWindow).tsjs!; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(ts)!; + ts.adInit = vi.fn(); + let resolveOld!: (value: unknown) => void; + fetchStub.mockImplementationOnce( + () => + new Promise((resolve) => { + resolveOld = resolve; + }) + ); + fetchStub.mockResolvedValueOnce({ ok: true, json: async () => ({ slots: [], bids: {} }) }); + history.pushState({}, '', '/trace-old'); + history.pushState({}, '', '/trace-new'); + await flushAsync(); + resolveOld({ + ok: true, + json: async () => ({ slots: [], bids: {}, trace_auction: gptTransport('spa_page_bids') }), + }); + await flushAsync(); + expect(collector.snapshot().value?.serverAuctions).toEqual([]); + expect(collector.snapshot().value?.issues).toEqual([]); + }); + it('increments navGeneration only when a pathname navigation is accepted', async () => { // The deferred initial-adInit bootstrap keys off this counter, so it must // move in lockstep with the hook's own navigation identity: bumped diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/index.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/index.test.ts index fa50d6366..0056300de 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/index.test.ts @@ -1,6 +1,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { TsjsApi } from '../../../src/core/types'; +import * as traceHandoff from '../../../src/trace/handoff'; import { installGptDiagnosticsRuntime, isGptDiagnosticsActive, @@ -81,12 +82,14 @@ beforeEach(() => { delete target.googletag; delete target.__tsjs_gpt_diagnostics_active; delete target.__tsjs_gpt_diagnostics_runtime; + delete target.__tsjs_trace_active; }); afterEach(() => { target.__tsjs_gpt_diagnostics_runtime?.destroy(); delete target.__tsjs_gpt_diagnostics_active; delete target.__tsjs_gpt_diagnostics_runtime; + delete target.__tsjs_trace_active; delete target.googletag; delete target.tsjs; vi.unstubAllGlobals(); @@ -95,6 +98,54 @@ afterEach(() => { }); describe('GPT diagnostics integration composition', () => { + it.each([undefined, false, 'true', 1])( + 'installs no trace handoff for nonliteral trace flag %s', + (active) => { + const create = vi.spyOn(traceHandoff, 'createTraceHandoff'); + const shadow = vi.spyOn(Element.prototype, 'attachShadow'); + target.__tsjs_gpt_diagnostics_active = true; + target.__tsjs_trace_active = active; + installGptStub(); + installGptDiagnosticsRuntime(target); + const root = shadow.mock.results.at(-1)?.value as ShadowRoot | undefined; + expect(create).not.toHaveBeenCalled(); + expect(root?.textContent).not.toContain('View trace results'); + expect(target.tsjs?.gptDiagnostics).toBeDefined(); + } + ); + it('connects the gated trace controls after the public snapshot API and destroys recovery with the runtime', () => { + const view = vi.fn(); + const download = vi.fn(); + const destroy = vi.fn(); + const create = vi.spyOn(traceHandoff, 'createTraceHandoff').mockImplementation((options) => { + expect(options.target.tsjs?.gptDiagnostics?.snapshot).toBeTypeOf('function'); + return { view, download, destroy }; + }); + const shadow = vi.spyOn(Element.prototype, 'attachShadow'); + target.__tsjs_gpt_diagnostics_active = true; + target.__tsjs_trace_active = true; + installGptStub(); + installGptDiagnosticsRuntime(target); + const root = shadow.mock.results.at(-1)?.value as ShadowRoot; + const traceButton = Array.from(root.querySelectorAll('button')).find( + (item) => item.textContent === 'View trace results' + ); + expect(traceButton).toBeDefined(); + traceButton?.click(); + expect(view).toHaveBeenCalledTimes(1); + const options = create.mock.calls[0][0]; + options.onChange?.({ kind: 'storage_unavailable', downloadAvailable: true }); + const recovery = Array.from(root.querySelectorAll('button')).find( + (item) => item.textContent === 'Download trace report' + ); + expect(recovery).toBeDefined(); + recovery?.click(); + expect(download).toHaveBeenCalledTimes(1); + target.__tsjs_gpt_diagnostics_runtime?.destroy(); + expect(destroy).toHaveBeenCalledTimes(1); + traceButton?.click(); + expect(view).toHaveBeenCalledTimes(1); + }); it('has no inactive side effects', () => { const originalMutationObserver = window.MutationObserver; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index 41cad667a..7ff4faead 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -86,6 +86,52 @@ afterEach(() => { }); describe('GptDiagnosticsOverlay', () => { + it('adds an optional prominent trace action and independent storage recovery without replacing GPT export', () => { + let root: ShadowRoot | undefined; + const onViewTrace = vi.fn(); + const onDownloadTrace = vi.fn(); + const onExport = vi.fn(); + const overlay = new GptDiagnosticsOverlay(new GptDiagnosticsStore(), new FakeBindings(), { + scheduleFrame: (callback) => callback(), + onShadowRoot: (created) => { + root = created; + }, + onViewTrace, + onDownloadTrace, + onExport, + }); + if (!root) throw new Error('should mount diagnostics'); + const view = button(root, 'View trace results'); + expect(view.className).toContain('tsgd-trace-action'); + expect(root.querySelector('style')?.textContent).toContain('min-height: 44px'); + view.click(); + expect(onViewTrace).toHaveBeenCalledTimes(1); + button(root, 'Export JSON').click(); + expect(onExport).toHaveBeenCalledTimes(1); + expect(root.textContent).not.toContain('Download trace report'); + overlay.setTraceState({ kind: 'storage_unavailable', downloadAvailable: true }); + expect(root.querySelector('[aria-live="polite"]')?.textContent).toContain('could not be saved'); + button(root, 'Download trace report').click(); + expect(onDownloadTrace).toHaveBeenCalledTimes(1); + overlay.setTraceState({ kind: 'capture_failed', downloadAvailable: false }); + expect(root.textContent).not.toContain('Download trace report'); + expect(root.querySelector('[aria-live="polite"]')?.textContent).toContain( + 'could not be captured' + ); + overlay.destroy(); + }); + it('does not create trace controls when optional trace callbacks are absent', () => { + let root: ShadowRoot | undefined; + const overlay = new GptDiagnosticsOverlay(new GptDiagnosticsStore(), new FakeBindings(), { + scheduleFrame: (callback) => callback(), + onShadowRoot: (created) => { + root = created; + }, + }); + expect(root?.textContent).not.toContain('View trace results'); + expect(root?.querySelector('[aria-live="polite"]')).toBeNull(); + overlay.destroy(); + }); it('waits for document completion and two animation frames before mounting', () => { const frames: Array<() => void> = []; const store = new GptDiagnosticsStore({ schedule: (callback) => callback() }); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts index 52aef6a7f..30b585809 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it, vi } from 'vitest'; +import { AUCTION_TOKEN, SLOT_TOKEN } from '../../trace/fixtures'; import { CREATIVE_ATTEMPT_WINDOW_MS, GptDiagnosticsStore, @@ -67,12 +68,177 @@ function last(values: readonly T[]): T | undefined { } describe('GptDiagnosticsStore', () => { + it.each([undefined, 'private-malformed', 'ts-auc-2234567812344abc8def123456789abc'])( + 'retains ordinary marker %s without fabricating a trace sidecar', + (marker) => { + const onTraceCorrelation = vi.fn(); + const store = new GptDiagnosticsStore({ now: () => 1, onTraceCorrelation }); + const slot = fakeSlot('example-slot'); + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + marker, + undefined, + { + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + } + ); + store.recordSlotRequested(slot); + expect(onTraceCorrelation).not.toHaveBeenCalled(); + expect(store.snapshot().slots[0]!.requests[0]).toMatchObject({ + requestPath: 'trusted_server_direct', + }); + expect(store.snapshot().slots[0]!.requests[0]!.trustedServerAuctionId).toBe(marker); + } + ); + + it('emits a sidecar for an exactly matching marker after ordinary trimming', () => { + const onTraceCorrelation = vi.fn(); + const store = new GptDiagnosticsStore({ now: () => 1, onTraceCorrelation }); + const slot = fakeSlot('example-slot'); + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + ` ${AUCTION_TOKEN} `, + undefined, + { + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + } + ); + store.recordSlotRequested(slot); + expect(onTraceCorrelation).toHaveBeenCalledOnce(); + expect(store.snapshot().slots[0]!.requests[0]!.trustedServerAuctionId).toBe(AUCTION_TOKEN); + }); + it.each(['renderable_candidate', 'unrenderable_candidate', 'no_candidate'] as const)( + 'emits a trace identity only at the concrete %s request binding', + (opportunity) => { + const onTraceCorrelation = vi.fn(); + const store = new GptDiagnosticsStore({ now: () => 10, onTraceCorrelation }); + const slot = fakeSlot('private-slot'); + store.recordTrustedServerOpportunity( + slot, + 'private-auction-slot', + opportunity, + AUCTION_TOKEN, + [[300, 250]], + { diagnostic_auction_id: AUCTION_TOKEN, slot_ref: SLOT_TOKEN } + ); + expect(onTraceCorrelation).not.toHaveBeenCalled(); + store.recordSlotRequested(slot); + expect(onTraceCorrelation).toHaveBeenCalledExactlyOnceWith({ + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }); + store.recordSlotRequested(slot); + expect(onTraceCorrelation).toHaveBeenCalledOnce(); + const cycles = store.snapshot().slots[0]!.requests; + expect(cycles[0]).toMatchObject({ + requestPath: 'trusted_server_direct', + trustedServerOpportunity: opportunity, + trustedServerAuctionId: AUCTION_TOKEN, + }); + expect(cycles[1]).toMatchObject({ requestPath: 'unattributed' }); + expect(JSON.stringify(store.snapshot())).not.toContain(SLOT_TOKEN); + } + ); + it('uses the explicit creative-attempt and attribution retention bounds', () => { expect(CREATIVE_ATTEMPT_WINDOW_MS).toBe(30_000); expect(MAX_CREATIVE_ATTEMPTS).toBe(128); expect(MAX_ATTRIBUTION_ISSUES).toBe(128); }); + it('keeps sidecars on the consumed source decision with competing refreshes and exact expiry', () => { + let now = 0; + const onTraceCorrelation = vi.fn(); + const store = new GptDiagnosticsStore({ now: () => now, onTraceCorrelation }); + const slot = fakeSlot('example-slot'); + const identity = { diagnostic_auction_id: AUCTION_TOKEN, slot_ref: SLOT_TOKEN }; + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + AUCTION_TOKEN, + undefined, + identity + ); + store.recordPrebidRefresh([slot]); + now = 4_999; + store.recordSlotRequested(slot); + expect(onTraceCorrelation).toHaveBeenCalledOnce(); + expect(store.snapshot().slots[0]!.requests[0]!.requestPath).toBe('competing'); + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + AUCTION_TOKEN, + undefined, + identity + ); + now += REQUEST_PATH_ATTRIBUTION_WINDOW_MS; + store.recordSlotRequested(slot); + expect(onTraceCorrelation).toHaveBeenCalledOnce(); + expect(store.snapshot().slots[0]!.requests[1]!.requestPath).toBe('unattributed'); + }); + + it('owns valid identity data and keeps ordinary cycles intact when a trace callback throws', () => { + const onTraceCorrelation = vi.fn(() => { + throw new Error('private-callback'); + }); + const store = new GptDiagnosticsStore({ now: () => 1, onTraceCorrelation }); + const slot = fakeSlot('example-slot'); + const identity = { diagnostic_auction_id: AUCTION_TOKEN, slot_ref: SLOT_TOKEN }; + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + AUCTION_TOKEN, + undefined, + identity + ); + identity.slot_ref = 'private-mutated'; + expect(() => store.recordSlotRequested(slot)).not.toThrow(); + expect(onTraceCorrelation).toHaveBeenCalledExactlyOnceWith({ + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }); + expect(store.snapshot().coverage.slotRequested.matched).toBe(1); + expect(store.snapshot().slots[0]!.requests).toHaveLength(1); + }); + + it('ignores unbound/invalid identities without reading caller token getters', () => { + const onTraceCorrelation = vi.fn(); + const store = new GptDiagnosticsStore({ now: () => 1, onTraceCorrelation }); + const slot = fakeSlot('example-slot'); + const getter = vi.fn(() => { + throw new Error('private-token'); + }); + const identity = { diagnostic_auction_id: AUCTION_TOKEN, slot_ref: SLOT_TOKEN }; + Object.defineProperty(identity, 'slot_ref', { get: getter }); + store.recordTrustedServerOpportunity( + slot, + 'private-slot', + 'no_candidate', + undefined, + undefined, + identity + ); + store.recordSlotRequested(slot); + expect(getter).not.toHaveBeenCalled(); + expect(onTraceCorrelation).not.toHaveBeenCalled(); + expect(store.snapshot().coverage.slotRequested.matched).toBe(1); + }); + it('records a complete filled lifecycle with valid timings and visibility', () => { let now = 10; const store = new GptDiagnosticsStore({ now: () => now }); diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index 9ab8f53ae..02ccb06b8 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -213,6 +213,11 @@ type RequestBidsArg = Parameters['requestBid /** The bid adapter spec object registered via `pbjs.registerBidAdapter`. */ interface TestAdapterSpec { + onTimeout?: (bids: Array>) => void; + onBidderError?: (value: { + error: unknown; + bidderRequest: { bids: Array> }; + }) => void; code: string; supportedMediaTypes: string[]; isBidRequestValid: (bid: Record) => boolean; @@ -364,6 +369,9 @@ import type { TsjsApi } from '../../../src/core/types'; import { GptDiagnosticsObserver } from '../../../src/integrations/gpt_diagnostics/observer'; import { GptDiagnosticsStore } from '../../../src/integrations/gpt_diagnostics/store'; import envelope from '../../fixtures/aps-renderer-v1.json'; +import { installTraceRuntime } from '../../../src/trace/runtime'; +import { gptTransport } from '../../trace/gpt-fixtures'; +import { SLOT_TOKEN } from '../../trace/fixtures'; // installPrebidNpm is a per-page no-op once the sentinel is set (the module // self-init above already set it), so every test starts from a clean page. @@ -7966,3 +7974,362 @@ describe('prebid self-init user ID module timing', () => { expect(userSyncCallCount()).toBe(1); }); }); + +describe('Prebid registered trace transport hooks', () => { + function setup(...flags: unknown[]) { + const flag = flags.length ? flags[0] : true; + window.tsjs = {} as TsjsApi; + window.__tsjs_trace_active = flag as boolean; + if (flag === true) installTraceRuntime(window.tsjs); + installPrebidNpm(); + return mockRegisterBidAdapter.mock.calls[ + mockRegisterBidAdapter.mock.calls.length - 1 + ]![2] as TestAdapterSpec; + } + function original(id: string, code = 'example-unit', bidderRequestId = 'example-request') { + return { + bidId: id, + bidderRequestId, + adUnitCode: code, + bidder: 'trustedServer', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + params: {}, + }; + } + function result() { + return { + body: { + ext: { + trusted_server: { + trace_auction: { + ...gptTransport(), + evidence: { ...gptTransport().evidence!, source: 'auction_api' }, + }, + }, + }, + }, + }; + } + beforeEach(() => { + vi.useFakeTimers(); + mockRegisterBidAdapter.mockClear(); + mockGetConfig.mockReturnValue(3000); + vi.spyOn(window.crypto, 'randomUUID').mockReturnValue('12345678-1234-4abc-8def-123456789abc'); + }); + afterEach(() => { + window.dispatchEvent(new PageTransitionEvent('pagehide', { persisted: false })); + vi.useRealTimers(); + vi.restoreAllMocks(); + delete window.tsjs; + delete window.__tsjs_trace_active; + }); + it.each([undefined, false, 'true', 1])( + 'adds no hooks, listeners, tokens or timers for gate %s', + (flag) => { + const listen = vi.spyOn(window, 'addEventListener'); + const uuid = window.crypto.randomUUID; + const spec = setup(flag); + expect(spec).not.toHaveProperty('onTimeout'); + expect(spec).not.toHaveProperty('onBidderError'); + const request = spec.buildRequests([original('first')]); + expect(JSON.parse(request.data as unknown as string).adUnits[0]).not.toHaveProperty('ext'); + expect(uuid).not.toHaveBeenCalled(); + expect(listen.mock.calls.filter(([name]) => name === 'pagehide')).toHaveLength(0); + expect(vi.getTimerCount()).toBe(0); + } + ); + it('uses final grouped units and consumes API transport before ordinary bid parsing exactly once', () => { + const spec = setup(); + const input = [original('first'), original('second')]; + const request = spec.buildRequests(input); + const payload = JSON.parse(request.data as unknown as string); + expect(payload.adUnits).toHaveLength(1); + expect(payload.adUnits[0].ext.trusted_server.trace_slot_ref).toBe(SLOT_TOKEN); + expect(input[0]).not.toHaveProperty('ext'); + const received = result(); + Object.defineProperty(received.body, 'seatbid', { + get() { + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('complete'); + return []; + }, + }); + expect(spec.interpretResponse(received, request)).toEqual([]); + spec.interpretResponse(received, request); + spec.onTimeout!([original('first')]); + spec.onBidderError!({ error: new Error('private'), bidderRequest: { bids: input } }); + expect(window.tsjs!.traceEvidence!.snapshot().value?.serverAuctions).toHaveLength(1); + expect(window.tsjs!.traceEvidence!.snapshot().value?.slotCorrelations).toEqual([]); + expect(vi.getTimerCount()).toBe(0); + }); + it('isolates concurrent request IDs and consumes timeout/error only once before callbacks', () => { + const spec = setup(); + const first = spec.buildRequests([original('first')]); + const second = spec.buildRequests([original('second')]); + const collector = window.tsjs!.traceEvidence!; + expect(first).toBeDefined(); + spec.onTimeout!([original('first')]); + spec.onBidderError!({ + error: { + get message() { + throw Error('private'); + }, + }, + bidderRequest: { bids: [original('first')] }, + }); + spec.interpretResponse(result(), second); + expect(collector.captureStatus()).toBe('partial'); + expect(collector.snapshot().value?.serverAuctions).toHaveLength(1); + expect(vi.getTimerCount()).toBe(0); + }); + it('retains ordinary payload on throwing token generator and still observes independent evidence', () => { + const spec = setup(); + vi.mocked(window.crypto.randomUUID).mockImplementation(() => { + throw Error('private'); + }); + const request = spec.buildRequests([original('first')]); + expect(JSON.parse(request.data as unknown as string).adUnits[0]).not.toHaveProperty('ext'); + spec.interpretResponse(result(), request); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('complete'); + }); + it('keeps missing optional evidence unobserved and malformed evidence bounded without changing bids', () => { + const spec = setup(); + const absent = spec.buildRequests([original('first')]); + expect(spec.interpretResponse({ body: {} }, absent)).toEqual([]); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('not_observed'); + const malformed = spec.buildRequests([original('second')]); + expect( + spec.interpretResponse( + { body: { ext: { trusted_server: { trace_auction: null } } } }, + malformed + ) + ).toEqual([]); + expect(window.tsjs!.traceEvidence!.snapshot().value?.issues).toEqual([ + 'evidence_validation_failed', + ]); + }); + it('clears BFCache markers, permits resumed requests, and destroys nonpersisted page timers', () => { + const spec = setup(); + spec.buildRequests([original('first')]); + expect(vi.getTimerCount()).toBe(1); + window.dispatchEvent(new PageTransitionEvent('pagehide', { persisted: true })); + expect(vi.getTimerCount()).toBe(0); + spec.onTimeout!([original('first')]); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('not_observed'); + const resumed = spec.buildRequests([original('second')]); + expect(vi.getTimerCount()).toBe(1); + window.dispatchEvent(new PageTransitionEvent('pagehide', { persisted: false })); + expect(vi.getTimerCount()).toBe(0); + spec.interpretResponse(result(), resumed); + spec.buildRequests([original('third')]); + expect(vi.getTimerCount()).toBe(0); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('not_observed'); + }); + it('uses distinct final unit refs while retaining all original bid IDs for one request', () => { + const spec = setup(); + vi.mocked(window.crypto.randomUUID) + .mockReturnValueOnce('12345678-1234-4abc-8def-123456789abc') + .mockReturnValueOnce('22345678-1234-4abc-8def-123456789abc'); + const request = spec.buildRequests([ + original('first', 'unit-a'), + original('second', 'unit-a'), + original('third', 'unit-b'), + ]); + const payload = JSON.parse(request.data as unknown as string); + expect( + payload.adUnits.map( + (unit: { ext: { trusted_server: { trace_slot_ref: string } } }) => + unit.ext.trusted_server.trace_slot_ref + ) + ).toEqual([SLOT_TOKEN, 'ts-slot-22345678-1234-4abc-8def-123456789abc']); + spec.onTimeout!([original('second')]); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('unavailable'); + spec.interpretResponse(result(), request); + expect(window.tsjs!.traceEvidence!.snapshot().value?.serverAuctions).toEqual([]); + expect(vi.getTimerCount()).toBe(0); + }); + it('swallows throwing diagnostic callbacks while returning ordinary auction bids unchanged', () => { + window.tsjs = {} as TsjsApi; + window.__tsjs_trace_active = true; + const collector = installTraceRuntime(window.tsjs)!; + window.tsjs.traceEvidence = { + ...collector, + recordTransport: () => { + throw Error('private-transport'); + }, + recordTransportFailure: () => { + throw Error('private-failure'); + }, + }; + installPrebidNpm(); + const spec = mockRegisterBidAdapter.mock.calls[ + mockRegisterBidAdapter.mock.calls.length - 1 + ]![2] as TestAdapterSpec; + const request = spec.buildRequests([original('first')]); + const received = result(); + Object.assign(received.body, { + seatbid: [ + { + seat: 'example-bidder', + bid: [ + { impid: 'example-unit', adm: '
Example creative
', price: 1, w: 300, h: 250 }, + ], + }, + ], + }); + expect(spec.interpretResponse(received, request)).toHaveLength(1); + spec.buildRequests([original('second')]); + expect(() => spec.onTimeout!([original('second')])).not.toThrow(); + expect(vi.getTimerCount()).toBe(0); + }); + it('retires colliding live IDs while ordinary bid delivery and earlier capture remain intact', () => { + const spec = setup(); + const collector = window.tsjs!.traceEvidence!; + collector.recordTransport(result().body.ext.trusted_server.trace_auction); + const first = spec.buildRequests([original('shared')]); + const collision = spec.buildRequests([original('shared')]); + expect(vi.getTimerCount()).toBe(2); + spec.onTimeout!([original('shared')]); + spec.onBidderError!({ error: Error('private'), bidderRequest: { bids: [original('shared')] } }); + const received = result(); + Object.assign(received.body, { + seatbid: [ + { + seat: 'example-bidder', + bid: [{ impid: 'example-unit', adm: '
Example
', price: 1, w: 300, h: 250 }], + }, + ], + }); + expect(spec.interpretResponse(received, first)).toHaveLength(1); + expect(spec.interpretResponse(received, collision)).toHaveLength(1); + expect(collector.captureStatus()).toBe('complete'); + expect(collector.snapshot().value?.serverAuctions).toHaveLength(3); + }); + it('keeps ordinary bid parsing when the optional request trace handle getter throws', () => { + const spec = setup(); + const request = spec.buildRequests([original('first')]); + Object.defineProperty(request, 'tracePending', { + get: () => { + throw Error('private-handle'); + }, + }); + const received = result(); + Object.assign(received.body, { + seatbid: [ + { + seat: 'example-bidder', + bid: [{ impid: 'example-unit', adm: '
Example
', price: 1, w: 300, h: 250 }], + }, + ], + }); + expect(() => spec.interpretResponse(received, request)).not.toThrow(); + expect(spec.interpretResponse(received, request)).toHaveLength(1); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('not_observed'); + }); + it('declines trace setup if its lifecycle listener cannot install while registering the ordinary adapter', () => { + const listen = window.addEventListener.bind(window); + vi.spyOn(window, 'addEventListener').mockImplementation((type, listener, options) => { + if (type === 'pagehide') throw Error('private-listener'); + listen(type, listener, options); + }); + let spec!: TestAdapterSpec; + expect(() => { + spec = setup(); + }).not.toThrow(); + expect(spec).not.toHaveProperty('onTimeout'); + expect(spec).not.toHaveProperty('onBidderError'); + const request = spec.buildRequests([original('first')]); + expect(JSON.parse(request.data as unknown as string).adUnits[0]).not.toHaveProperty('ext'); + expect(vi.getTimerCount()).toBe(0); + }); + it('retains valid capped server evidence from a 65-unit request without changing ordinary bids', () => { + const spec = setup(); + let generated = 0; + vi.mocked(window.crypto.randomUUID).mockImplementation( + () => + `${(++generated).toString(16).padStart(8, '0')}-1234-4abc-8def-123456789abc` as ReturnType< + Crypto['randomUUID'] + > + ); + const requests = Array.from({ length: 65 }, (_, index) => + original(`bid-${index}`, `unit-${index}`) + ); + const request = spec.buildRequests(requests); + const payload = JSON.parse(request.data as unknown as string); + const envelope = gptTransport(); + envelope.evidence.source = 'auction_api'; + envelope.evidence.truncation.omitted_slots = 1; + envelope.evidence.slots = payload.adUnits + .slice(0, 64) + .map((unit: { ext: { trusted_server: { trace_slot_ref: string } } }, index: number) => ({ + ...envelope.evidence.slots[0]!, + slot_number: index + 1, + slot_ref: unit.ext.trusted_server.trace_slot_ref, + })); + const received = { + body: { + ext: { trusted_server: { trace_auction: envelope } }, + seatbid: [ + { + seat: 'example-bidder', + bid: [{ impid: 'unit-64', adm: '
Example
', price: 1, w: 300, h: 250 }], + }, + ], + }, + }; + expect(payload.adUnits).toHaveLength(65); + expect(spec.interpretResponse(received, request)).toHaveLength(1); + const capture = window.tsjs!.traceEvidence!.snapshot().value!; + expect(capture.serverAuctions).toHaveLength(1); + expect(capture.serverAuctions[0]!.slots).toHaveLength(64); + expect(capture.serverAuctions[0]!.truncation.omitted_slots).toBe(1); + expect(capture.slotCorrelations).toEqual([]); + expect(window.tsjs!.traceEvidence!.captureStatus()).toBe('complete'); + expect(vi.getTimerCount()).toBe(0); + }); + it('keeps over-cap exact response capture and never attributes unseen colliding timeout IDs', () => { + const spec = setup(); + const first = spec.buildRequests([original('bid-2048')]); + const large = spec.buildRequests( + Array.from({ length: 2049 }, (_, index) => original(`bid-${index}`)) + ); + const later = spec.buildRequests([original('bid-2047')]); + spec.onTimeout!([original('bid-2048'), original('bid-2047')]); + spec.onBidderError!({ + error: Error('private'), + bidderRequest: { bids: [original('bid-2048')] }, + }); + const collector = window.tsjs!.traceEvidence!; + expect(collector.captureStatus()).toBe('not_observed'); + expect(collector.snapshot().value?.issues).toEqual(['correlation_unavailable']); + for (const request of [first, large, later]) + expect(spec.interpretResponse(result(), request)).toEqual([]); + expect(collector.captureStatus()).toBe('complete'); + expect(collector.snapshot().value?.serverAuctions).toHaveLength(3); + expect(vi.getTimerCount()).toBe(0); + }); + it('does not add a second lifecycle listener on repeated shim installation', () => { + const listen = vi.spyOn(window, 'addEventListener'); + setup(); + installPrebidNpm(); + expect(listen.mock.calls.filter(([name]) => name === 'pagehide')).toHaveLength(1); + }); + it.each(['timeout', 'error'])( + 'ignores a retired %s hook when a newer request reuses the original bid ID', + (mode) => { + const spec = setup(); + const old = original('reused', 'example-unit', 'request-old'); + const first = spec.buildRequests([old]); + spec.interpretResponse(result(), first); + const next = spec.buildRequests([original('reused', 'example-unit', 'request-new')]); + if (mode === 'timeout') spec.onTimeout!([old]); + else spec.onBidderError!({ error: Error('private'), bidderRequest: { bids: [old] } }); + expect(vi.getTimerCount()).toBe(1); + expect(window.tsjs!.traceEvidence!.snapshot().value?.issues).toEqual([ + 'correlation_unavailable', + ]); + spec.interpretResponse(result(), next); + expect(window.tsjs!.traceEvidence!.snapshot().value?.serverAuctions).toHaveLength(2); + expect(vi.getTimerCount()).toBe(0); + } + ); +}); diff --git a/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs b/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs index f0283f0a7..8ddf69fa0 100644 --- a/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs +++ b/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs @@ -24,6 +24,7 @@ let analyticsArtifact; let noAnalyticsArtifact; let managedUserIdArtifact; let shimCode; +let coreCode; async function buildArtifact(modules) { await main(['--modules-json', JSON.stringify(modules), '--out', outputDirectory]); @@ -78,6 +79,31 @@ beforeAll(async () => { logLevel: 'warn', }); shimCode = fs.readFileSync(path.join(outputDirectory, 'tsjs-prebid.js'), 'utf8'); + await build({ + configFile: false, + root: libDir, + build: { + emptyOutDir: false, + outDir: outputDirectory, + assetsDir: '.', + sourcemap: false, + minify: 'esbuild', + rollupOptions: { + input: path.join(libDir, 'src', 'core', 'index.ts'), + output: { + format: 'iife', + dir: outputDirectory, + entryFileNames: 'tsjs-core.js', + inlineDynamicImports: true, + extend: false, + name: 'tsjs_core', + }, + }, + }, + logLevel: 'warn', + }); + coreCode = fs.readFileSync(path.join(outputDirectory, 'tsjs-core.js'), 'utf8'); + console.log('[trace-prebid-artifact] shim characters:', shimCode.length); }, 240_000); afterAll(() => { @@ -320,12 +346,11 @@ describe('tsjs-prebid production artifacts', () => { expect(shimCode).not.toContain(analyticsArtifact.manifest.prebidVersion); expect(shimCode).not.toContain('_pbjsGlobals'); expect(analyticsArtifact.bundleCode.length).toBeGreaterThan(200_000); - // A value-import of Prebid or a private rendering helper would multiply - // the shim size. The bound sits just above the normal compact shim output, - // which is roughly 40 KB: tight enough that material growth has to be - // noticed and re-justified here, and far enough below a multiplication - // that one still fails loudly. The bundle beside it is 200 KB and up. - expect(shimCode.length).toBeLessThan(41_000); + // The original shim measures 40,354 characters. Shared strict trace JSON + // validation, request-ID hook association, and bounded pending lifecycle add + // 11,676 characters (52,030 total). Keep a narrow 52.5 KB guard: Prebid remains a separate artifact above 200 KB, and importing + // its runtime or private rendering helpers must still fail this regression. + expect(shimCode.length).toBeLessThan(52_500); expect(shimCode).toContain('markWinningBidAsUsed'); }); @@ -562,3 +587,218 @@ describe('tsjs-prebid production artifacts', () => { expect(noAnalyticsArtifact.manifest.sri).toBe(sri); }); }); + +describe('active trace through real Prebid 10.26 registered adapters', () => { + function tracedPage(mode) { + const dom = createPage(); + const pageWindow = dom.window; + const stubs = installNetworkAndConsoleStubs(pageWindow); + installServerState(pageWindow); + pageWindow.TextEncoder = TextEncoder; + pageWindow.__tsjs_trace_active = true; + pageWindow.eval(coreCode); + pageWindow.eval(noAnalyticsArtifact.bundleCode); + pageWindow.pbjs.setConfig({ bidderTimeout: 25 }); + const register = pageWindow.pbjs.registerBidAdapter.bind(pageWindow.pbjs); + let spec; + pageWindow.pbjs.registerBidAdapter = (adapter, code, registered) => { + if (code === 'trustedServer') { + spec = registered; + if (typeof registered.onTimeout === 'function') + registered.onTimeout = vi.fn(registered.onTimeout); + if (typeof registered.onBidderError === 'function') + registered.onBidderError = vi.fn(registered.onBidderError); + } + return register(adapter, code, registered); + }; + if (mode === 'timeout') stubs.fetchSpy.mockImplementation(() => new Promise(() => {})); + if (mode === 'error') stubs.fetchSpy.mockRejectedValue(new Error('private-network-error')); + pageWindow.eval(shimCode); + return { dom, pageWindow, stubs, spec }; + } + function auction(pageWindow, timeout = 25, bidId) { + return new Promise((resolve) => + pageWindow.pbjs.requestBids({ + adUnits: [ + { + code: 'example-trace-unit', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bids: [{ bidder: 'trustedServer', params: {}, ...(bidId ? { bid_id: bidId } : {}) }], + }, + ], + timeout, + bidsBackHandler: resolve, + }) + ); + } + it.each(['timeout', 'error'])( + 'routes actual %s through registerBidAdapter/newBidder/adapterManager exactly once', + async (mode) => { + const { dom, pageWindow, stubs, spec } = tracedPage(mode); + try { + expect(spec.onTimeout).toEqual(expect.any(Function)); + expect(spec.onBidderError).toEqual(expect.any(Function)); + const bidderEvents = []; + pageWindow.pbjs.onEvent(mode === 'timeout' ? 'bidTimeout' : 'bidderError', (event) => + bidderEvents.push(event) + ); + const completed = auction(pageWindow); + await vi.waitFor( + () => expect(mode === 'timeout' ? spec.onTimeout : spec.onBidderError).toHaveBeenCalled(), + { timeout: 5000 } + ); + await completed; + const captured = pageWindow.tsjs.traceEvidence.snapshot(); + expect(pageWindow.tsjs.traceEvidence.captureStatus()).toBe('unavailable'); + expect(captured.value.issues).toEqual(['evidence_transport_failed']); + expect(captured.value.slotCorrelations).toEqual([]); + expect(bidderEvents.length).toBeGreaterThan(0); + expect(JSON.stringify(captured)).not.toContain('private-network-error'); + const callback = mode === 'timeout' ? spec.onTimeout : spec.onBidderError; + callback(...callback.mock.calls[0]); + expect(pageWindow.tsjs.traceEvidence.snapshot()).toEqual(captured); + expectNoUnexpectedNetworkActivity(stubs); + } finally { + dom.window.dispatchEvent(new dom.window.PageTransitionEvent('pagehide')); + dom.window.close(); + } + }, + 10000 + ); + it('consumes readable optional evidence in the real registered interpretResponse before the ordinary no-bid callback', async () => { + const { dom, pageWindow, stubs } = tracedPage('response'); + try { + stubs.fetchSpy.mockImplementation(async (_resource, init) => { + const payload = JSON.parse(init?.body ?? (await _resource.text())); + const ref = payload.adUnits[0].ext.trusted_server.trace_slot_ref; + return new Response( + JSON.stringify({ + seatbid: [], + ext: { + trusted_server: { + trace_auction: { + schema_version: 1, + evidence: { + schema_version: 1, + diagnostic_auction_id: 'ts-auc-1234567812344abc8def123456789abc', + source: 'auction_api', + terminal_status: 'completed', + provider_calls: [], + slots: [ + { + slot_number: 1, + slot_ref: ref, + requested_sizes: [[300, 250]], + returned_bid_count: 0, + candidate: 'no_candidate', + }, + ], + truncation: { + omitted_provider_calls: 0, + omitted_slots: 0, + omitted_nested_values: 0, + }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + }, + }, + }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } } + ); + }); + await auction(pageWindow, 1000); + expect(pageWindow.tsjs.traceEvidence.captureStatus()).toBe('complete'); + const captured = pageWindow.tsjs.traceEvidence.snapshot().value; + expect(captured.serverAuctions).toHaveLength(1); + expect(captured.issues).toEqual(['correlation_unavailable']); + expect(captured.slotCorrelations).toEqual([]); + expectNoUnexpectedNetworkActivity(stubs); + } finally { + dom.window.dispatchEvent(new dom.window.PageTransitionEvent('pagehide')); + dom.window.close(); + } + }, 10000); + it.each(['timeout', 'error'])( + 'preserves a newer reused bid ID through a repeated old actual %s hook', + async (mode) => { + const { dom, pageWindow, stubs, spec } = tracedPage(mode); + try { + const callback = mode === 'timeout' ? spec.onTimeout : spec.onBidderError; + const first = auction(pageWindow, 25, 'example-reused-bid'); + await vi.waitFor(() => expect(callback).toHaveBeenCalled(), { timeout: 5000 }); + await first; + const oldArguments = callback.mock.calls[0]; + const oldBids = mode === 'timeout' ? oldArguments[0] : oldArguments[0].bidderRequest.bids; + expect(oldBids[0].bidId).toBe('example-reused-bid'); + expect(oldBids[0].bidderRequestId).toEqual(expect.any(String)); + let deliver; + let nextPayload; + stubs.fetchSpy.mockImplementation(async (resource, init) => { + nextPayload = JSON.parse(init?.body ?? (await resource.text())); + return await new Promise((resolve) => { + deliver = resolve; + }); + }); + const build = spec.buildRequests; + let nextBids; + spec.buildRequests = (...args) => { + nextBids = args[0]; + return build(...args); + }; + const next = auction(pageWindow, 2000, 'example-reused-bid'); + await vi.waitFor(() => expect(deliver).toEqual(expect.any(Function))); + expect(nextBids[0].bidId).toBe(oldBids[0].bidId); + expect(nextBids[0].bidderRequestId).not.toBe(oldBids[0].bidderRequestId); + callback(...oldArguments); + deliver( + new Response( + JSON.stringify({ + seatbid: [], + ext: { + trusted_server: { + trace_auction: { + schema_version: 1, + evidence: { + schema_version: 1, + diagnostic_auction_id: 'ts-auc-1234567812344abc8def123456789abc', + source: 'auction_api', + terminal_status: 'completed', + provider_calls: [], + slots: [ + { + slot_number: 1, + slot_ref: nextPayload.adUnits[0].ext.trusted_server.trace_slot_ref, + requested_sizes: [[300, 250]], + returned_bid_count: 0, + candidate: 'no_candidate', + }, + ], + truncation: { + omitted_provider_calls: 0, + omitted_slots: 0, + omitted_nested_values: 0, + }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + }, + }, + }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } } + ) + ); + await next; + const capture = pageWindow.tsjs.traceEvidence.snapshot().value; + expect(capture.serverAuctions).toHaveLength(1); + expect(capture.issues).toEqual(['evidence_transport_failed', 'correlation_unavailable']); + expect(capture.slotCorrelations).toEqual([]); + expectNoUnexpectedNetworkActivity(stubs); + } finally { + dom.window.dispatchEvent(new dom.window.PageTransitionEvent('pagehide')); + dom.window.close(); + } + }, + 10000 + ); +}); diff --git a/crates/trusted-server-js/lib/test/trace-assets.test.mjs b/crates/trusted-server-js/lib/test/trace-assets.test.mjs new file mode 100644 index 000000000..4df1d6994 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace-assets.test.mjs @@ -0,0 +1,34 @@ +// @vitest-environment node +import { createHash } from 'node:crypto'; +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { traceSourceDigest } from '../trace-asset-sources.mjs'; + +const library = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const sha256 = (bytes) => createHash('sha256').update(bytes).digest('hex'); + +describe('versioned trace assets', () => { + it('retains independently embedded committed bytes with exact URLs and strong digests', () => { + const manifestFile = path.join(library, 'trace-assets-manifest.json'); + expect(fs.existsSync(manifestFile), 'should commit the trace asset manifest').toBe(true); + const manifest = JSON.parse(fs.readFileSync(manifestFile, 'utf8')); + expect(manifest.schema_version).toBe(1); + expect(manifest.source_sha256).toBe(traceSourceDigest(library)); + expect(manifest.assets.map((asset) => asset.path)).toEqual([ + '/_ts/trace/assets/v1.js', + '/_ts/trace/assets/v1.css', + ]); + for (const asset of manifest.assets) { + const frozen = fs.readFileSync(path.join(library, 'trace-assets', asset.file)); + const built = fs.readFileSync(path.join(library, '..', 'dist', 'trace', asset.file)); + expect(built).toEqual(frozen); + expect(sha256(frozen)).toBe(asset.sha256); + expect(asset.sha256).toMatch(/^[a-f\d]{64}$/); + } + expect(fs.existsSync(path.join(library, '..', 'dist', 'tsjs-trace.js'))).toBe(false); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/collector.test.ts b/crates/trusted-server-js/lib/test/trace/collector.test.ts new file mode 100644 index 000000000..3751e4c53 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/collector.test.ts @@ -0,0 +1,172 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { createTraceCollector } from '../../src/trace/collector'; + +import { SLOT_TOKEN } from './fixtures'; + +function transport(number = 1, source = 'initial_navigation_ssat') { + return { + schema_version: 1, + evidence: { + schema_version: 1, + diagnostic_auction_id: token(number), + source, + terminal_status: 'completed', + provider_calls: [], + slots: [], + truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + }; +} +function token(number: number) { + return `ts-auc-${number.toString(16).padStart(8, '0')}12344abc8def123456789abc`; +} +function sidecar(number = 1) { + return { + schema_version: 1, + diagnostic_auction_id: token(number), + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: number, + }; +} +function collector() { + const result = createTraceCollector(); + if (typeof result !== 'object' || result === null) throw new Error('should create collector'); + return result; +} +describe('bounded memory-only trace collector', () => { + it('cannot capture an invalid own enum disguised by a proxy getter', () => { + const value = collector(); + const original = transport(); + original.evidence.terminal_status = 'private-invalid-enum'; + let reads = 0; + original.evidence = new Proxy(original.evidence, { + get(target, key, receiver) { + reads += 1; + if (key === 'terminal_status') return 'completed'; + return Reflect.get(target, key, receiver); + }, + }); + value.recordTransport(original); + expect(value.captureStatus()).toBe('unavailable'); + expect(value.snapshot().value?.issues).toEqual(['evidence_validation_failed']); + expect(reads).toBe(0); + }); + it('does not touch storage or start network requests during collection', () => { + const get = vi.spyOn(Storage.prototype, 'getItem'); + const set = vi.spyOn(Storage.prototype, 'setItem'); + const remove = vi.spyOn(Storage.prototype, 'removeItem'); + const fetch = vi.fn(); + vi.stubGlobal('fetch', fetch); + const value = collector(); + value.recordTransport(transport()); + value.recordCorrelation(sidecar()); + value.snapshot(); + expect(get).not.toHaveBeenCalled(); + expect(set).not.toHaveBeenCalled(); + expect(remove).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + }); + it('treats absent optional transport as no observation without resetting earlier evidence', () => { + const value = collector(); + value.recordTransport(undefined); + expect(value.captureStatus()).toBe('not_observed'); + value.recordTransport(transport()); + value.recordTransport(undefined); + expect(value.captureStatus()).toBe('complete'); + expect(value.snapshot().value?.serverAuctions).toHaveLength(1); + }); + it('records bounded capture failures in unique enum order without retaining input text', () => { + const value = collector(); + value.recordTransportFailure(); + value.recordTransport(null); + value.recordTransport({ schema_version: 1, unavailable_reason: 'evidence_projection_failed' }); + value.recordTransportFailure(); + expect(value.captureStatus()).toBe('unavailable'); + expect(value.snapshot().value?.issues).toEqual([ + 'evidence_projection_failed', + 'evidence_transport_failed', + 'evidence_validation_failed', + ]); + value.recordTransport(transport()); + expect(value.captureStatus()).toBe('partial'); + expect(JSON.stringify(value.snapshot())).not.toContain('private'); + }); + it('retains newest sixteen records in observation order and exact eviction counts', () => { + const value = collector(); + for (let number = 1; number <= 18; number += 1) value.recordTransport(transport(number)); + const snapshot = value.snapshot().value; + expect(snapshot?.serverAuctions.map((record) => record.diagnostic_auction_id)).toEqual( + Array.from({ length: 16 }, (_, i) => token(i + 3)) + ); + expect(snapshot?.omittedServerAuctions).toBe(2); + expect(snapshot?.issues).toEqual(['record_evicted']); + expect(value.captureStatus()).toBe('partial'); + }); + it('keeps sidecar-only loss and interpretation limits separate from server coverage', () => { + const value = collector(); + value.recordTransport(transport()); + for (let number = 1; number <= 130; number += 1) value.recordCorrelation(sidecar(number)); + value.recordInterpretationIssue('external_client_side_unobservable'); + value.recordCorrelation({ private_value: 'private-sidecar' }); + expect(value.snapshot().value?.slotCorrelations).toHaveLength(128); + expect(value.snapshot().value?.omittedSlotCorrelations).toBe(2); + expect(value.snapshot().value?.issues).toEqual([ + 'correlation_unavailable', + 'external_client_side_unobservable', + ]); + expect(value.captureStatus()).toBe('complete'); + }); + it('prunes and counts sidecars when their known unique auction is evicted', () => { + const value = collector(); + value.recordTransport(transport(1)); + value.recordCorrelation(sidecar(1)); + for (let number = 2; number <= 17; number += 1) value.recordTransport(transport(number)); + expect(value.snapshot().value?.slotCorrelations).toEqual([]); + expect(value.snapshot().value?.omittedSlotCorrelations).toBe(1); + expect(value.snapshot().value?.issues).toEqual(['record_evicted', 'correlation_unavailable']); + }); + it('keeps API records independent and declines any supplied API sidecar', () => { + const value = collector(); + value.recordTransport(transport(1, 'auction_api')); + value.recordCorrelation(sidecar(1)); + expect(value.snapshot().value?.serverAuctions).toHaveLength(1); + expect(value.snapshot().value?.slotCorrelations).toEqual([]); + expect(value.captureStatus()).toBe('complete'); + expect(value.snapshot().value?.issues).toEqual(['correlation_unavailable']); + }); + it('returns fresh frozen arrays while caller objects remain mutable and isolated', () => { + const value = collector(); + const original = transport(); + value.recordTransport(original); + const first = value.snapshot(); + const second = value.snapshot(); + expect(first).toEqual(second); + expect(first.value).not.toBe(second.value); + expect(Object.isFrozen(first.value?.serverAuctions)).toBe(true); + expect(Object.isFrozen(original.evidence)).toBe(false); + original.evidence.diagnostic_auction_id = token(2); + expect(first.value?.serverAuctions[0].diagnostic_auction_id).toBe(token(1)); + }); + it('invalidates capture on checked eviction overflow without wrapping or throwing into ads', () => { + const value = collector(); + const record = transport(); + for (let count = 0; count < 65535 + 17; count += 1) value.recordTransport(record); + expect(value.snapshot()).toEqual({ ok: false, reason: 'omission_counter_overflow' }); + }, 15000); + it('ignores late callbacks after destruction and removes retained local evidence', () => { + const value = collector(); + value.recordTransport(transport()); + value.destroy(); + value.recordTransport(transport()); + value.recordCorrelation(sidecar()); + value.recordTransportFailure(); + expect(value.snapshot().value?.serverAuctions).toEqual([]); + expect(value.snapshot().value?.slotCorrelations).toEqual([]); + expect(value.captureStatus()).toBe('not_observed'); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/context.test.ts b/crates/trusted-server-js/lib/test/trace/context.test.ts new file mode 100644 index 000000000..4bf51b897 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/context.test.ts @@ -0,0 +1,205 @@ +import { describe, expect, it } from 'vitest'; + +import { validateTraceRequestContext } from '../../src/trace/context'; + +function context() { + return { + schema_version: 1, + captured_at: '2026-10-05T10:15:30.123Z', + network: {}, + cookies: { + ts_ec: { source: 'request', state: 'absent' }, + ts_eids: { source: 'request', state: 'absent' }, + ts_tester: { source: 'request', state: 'absent' }, + diagnostics_session: { source: 'request', state: 'absent' }, + }, + }; +} + +describe('redacted trace request context validation', () => { + it('accepts missing optional platform facts and all specified cookie states', () => { + expect(validateTraceRequestContext(context())).toBe(true); + const value = context(); + Object.assign(value.cookies, { + ts_ec: { source: 'request', state: 'present_valid', detail: 'valid_ec_format' }, + ts_eids: { source: 'request', state: 'present_invalid', detail: 'malformed' }, + ts_tester: { source: 'request', state: 'duplicate', detail: 'multiple_values' }, + diagnostics_session: { + source: 'request', + state: 'unavailable', + detail: 'runtime_header_ambiguous', + }, + }); + expect(validateTraceRequestContext(value)).toBe(true); + }); + + it.each(['192.0.2.0/24', '2001:db8:1234::/48', '::/48'])( + 'accepts the display-only masked prefix %s', + (masked_client_ip) => { + const value = context(); + Object.assign(value.network, { masked_client_ip }); + expect(validateTraceRequestContext(value)).toBe(true); + } + ); + + it.each([ + '192.0.2.129', + '192.0.2.129/24', + '192.0.2.0/32', + '256.0.2.0/24', + '01.0.2.0/24', + '2001:db8:1234:5678::1', + '2001:db8:1234:1::/48', + 'example.com', + ])('rejects a full address or incorrectly masked prefix %s', (masked_client_ip) => { + const value = context(); + Object.assign(value.network, { masked_client_ip }); + expect(validateTraceRequestContext(value)).toBe(false); + }); + + it.each([ + '2026-02-30T10:00:00Z', + '2026-10-05T25:00:00Z', + '2026-10-05T10:00:00+00:00', + '2026-10-05', + '2026-10-05T10:00:60Z', + '2026-10-05t10:00:00z', + '2026-10-05T10:15:30Z\n', + '2026-10-05T10:15:30Z\r', + '2026-10-05T10:15:30Z\u2028', + '2026-10-05T10:15:30Z\u2029', + ])('rejects an invalid or noncanonical UTC capture time %s', (captured_at) => { + expect(validateTraceRequestContext({ ...context(), captured_at })).toBe(false); + }); + + it.each([ + { country: 'USA' }, + { country: 'é' }, + { asn: -1 }, + { asn: 4_294_967_296 }, + { asn: 1.5 }, + { region: 'é'.repeat(17) }, + { tls_cipher: 'x'.repeat(33) }, + { edge_hostname: 'x'.repeat(129) }, + { region: 'a\n' }, + { region: 'a\u0085' }, + { region: 'a\u202e' }, + { region: '\ud800' }, + { ja4: 'secret-fingerprint' }, + { full_client_ip: '192.0.2.129' }, + ])('rejects forbidden or invalid platform facts %j', (network) => { + expect(validateTraceRequestContext({ ...context(), network })).toBe(false); + }); + + it.each([ + { state: 'absent', detail: 'malformed' }, + { state: 'present_valid', detail: 'valid_ec_format' }, + { state: 'present_invalid', detail: 'runtime_header_ambiguous' }, + { state: 'unavailable', detail: 'multiple_values' }, + { state: 'duplicate', detail: 'header_too_large' }, + { state: 'absent', value: 'raw-cookie-sentinel' }, + { state: 'unavailable', detail: 'unknown_reason' }, + { state: 'present_valid' }, + ])('rejects invalid session state/detail pairs and added values %j', (health) => { + const value = context(); + Object.assign(value.cookies, { diagnostics_session: { source: 'request', ...health } }); + expect(validateTraceRequestContext(value)).toBe(false); + }); + + it('requires exactly the four owned cookie names and rejects extra envelope fields', () => { + const value = context(); + expect(validateTraceRequestContext({ ...value, path: '/secret-sentinel' })).toBe(false); + expect(validateTraceRequestContext({ ...value, schema_version: 2 })).toBe(false); + expect( + validateTraceRequestContext({ ...value, cookies: { ...value.cookies, other: {} } }) + ).toBe(false); + expect(validateTraceRequestContext({ ...value, cookies: {} })).toBe(false); + }); + + it.each([ + ['ts_ec', 'valid_ec_format'], + ['ts_eids', 'valid_eids_format'], + ['ts_tester', 'valid_tester_value'], + ['diagnostics_session', 'valid_diagnostics_value'], + ])('binds the valid detail to its exact owned cookie %s', (name, detail) => { + const value = context(); + Object.assign(value.cookies, { [name]: { source: 'request', state: 'present_valid', detail } }); + expect(validateTraceRequestContext(value)).toBe(true); + Object.assign(value.cookies, { + [name]: { + source: 'request', + state: 'present_valid', + detail: detail === 'valid_ec_format' ? 'valid_diagnostics_value' : 'valid_ec_format', + }, + }); + expect(validateTraceRequestContext(value)).toBe(false); + }); + + it.each(['header_too_large', 'header_not_utf8', 'runtime_header_ambiguous'])( + 'preserves the aggregate unavailable reason on every owned cookie: %s', + (detail) => { + const value = context(); + for (const name of Object.keys(value.cookies)) + Object.assign(value.cookies, { + [name]: { source: 'request', state: 'unavailable', detail }, + }); + expect(validateTraceRequestContext(value)).toBe(true); + } + ); + + it('never coerces objects into a permitted detail or invokes getters', () => { + const value = context(); + Object.assign(value.cookies, { + diagnostics_session: { + source: 'request', + state: 'unavailable', + detail: { toString: () => 'runtime_header_ambiguous' }, + }, + }); + expect(validateTraceRequestContext(value)).toBe(false); + expect( + validateTraceRequestContext( + Object.defineProperty(context(), 'network', { + enumerable: true, + get: () => { + throw new Error('should not invoke getters'); + }, + }) + ) + ).toBe(false); + expect( + validateTraceRequestContext( + new Proxy( + {}, + { + getPrototypeOf: () => { + throw new Error('should fail closed'); + }, + } + ) + ) + ).toBe(false); + }); + + it('preserves valid boundary-sized Unicode and unsigned ASN facts', () => { + const value = context(); + Object.assign(value.network, { + country: 'US', + region: 'é'.repeat(16), + edge_hostname: 'x'.repeat(128), + edge_region: 'x'.repeat(128), + edge_pop: 'x'.repeat(32), + http_version: 'HTTP/2', + tls_protocol: 'TLSv1.3', + tls_cipher: 'example-cipher', + asn: 4_294_967_295, + }); + expect(validateTraceRequestContext(value)).toBe(true); + expect(validateTraceRequestContext({ ...value, captured_at: '2024-02-29T00:00:00Z' })).toBe( + true + ); + expect(validateTraceRequestContext({ ...value, captured_at: '1900-02-29T00:00:00Z' })).toBe( + false + ); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/correlation.test.ts b/crates/trusted-server-js/lib/test/trace/correlation.test.ts new file mode 100644 index 000000000..1a0c3a680 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/correlation.test.ts @@ -0,0 +1,160 @@ +import { describe, expect, it } from 'vitest'; + +import { joinTraceEvidence } from '../../src/trace/correlation'; + +import { reportFixture, TRACE_NOW, TRACE_ORIGIN, AUCTION_TOKEN, SLOT_TOKEN } from './fixtures'; + +function fixture(source = 'initial_navigation_ssat') { + const report = reportFixture(); + return { + ...report, + server_auctions: [ + { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + source, + terminal_status: 'completed', + provider_calls: [ + { provider_number: 1, role: 'bidder', status: 'no_bid', returned_bid_count: 0 }, + ], + slots: [ + { + slot_number: 1, + slot_ref: SLOT_TOKEN, + requested_sizes: [[300, 250]], + returned_bid_count: 0, + candidate: 'no_candidate', + }, + ], + truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + ], + slot_correlations: + source === 'auction_api' + ? [] + : [ + { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }, + ], + auction_coverage: { + capture_status: 'complete', + issues: source === 'auction_api' ? ['correlation_unavailable'] : [], + }, + }; +} +function view(report: ReturnType) { + const result = joinTraceEvidence(report, TRACE_ORIGIN, TRACE_NOW); + if (typeof result !== 'object' || result === null || !('auctions' in result)) + throw new Error('should join report'); + return result; +} +describe('exact trace evidence associations', () => { + it.each([ + ['initial_navigation_ssat', 'Initial-page server auction (SSAT)'], + ['spa_page_bids', 'Trusted Server page-refresh auction'], + ])( + 'joins a supported %s record only with both tokens and exact GPT identity', + (source, label) => { + const report = fixture(source); + const result = view(report); + expect(result.auctions[0].sourceLabel).toBe(label); + expect(result.auctions[0].slots[0].correlation).toBe('matched'); + expect(result.auctions[0].slots[0].cycle).toEqual( + report.gpt_diagnostics.slots[0].requests[0] + ); + expect(result.auctions[0].relativeMilestonesLabel).toBe('Unavailable in v1'); + expect(result.auctions[0].providerScopeLabel).toBe( + 'Auction-wide provider status; per-slot no-bid reason unavailable' + ); + expect(report).toEqual(fixture(source)); + } + ); + it('keeps API evidence independent of GPT and rejects API sidecars', () => { + const report = fixture('auction_api'); + expect(view(report).auctions[0]).toMatchObject({ + sourceLabel: 'Trusted Server auction API', + slots: [{ correlation: 'unknown' }], + }); + Object.assign(report, { slot_correlations: fixture().slot_correlations }); + expect(joinTraceEvidence(report, TRACE_ORIGIN, TRACE_NOW)).toBeUndefined(); + }); + it.each([ + 'missing-sidecar', + 'missing-cycle', + 'wrong-auction', + 'duplicate-sidecar', + 'duplicate-record', + 'duplicate-slot', + 'conflicting-sidecar', + 'duplicate-cycle', + ])('leaves %s as Correlation unknown', (kind) => { + const report = fixture(); + const raw = report as unknown as { + server_auctions: { slots: unknown[] }[]; + slot_correlations: { diagnostic_auction_id: string; slot_ref: string }[]; + }; + if (kind === 'missing-sidecar') raw.slot_correlations = []; + if (kind === 'missing-cycle') report.gpt_diagnostics.slots[0].requests = []; + if (kind === 'wrong-auction') + report.gpt_diagnostics.slots[0].requests[0].trustedServerAuctionId = + 'ts-auc-0000000112344abc8def123456789abc'; + if (kind === 'duplicate-sidecar') + raw.slot_correlations.push(structuredClone(raw.slot_correlations[0])); + if (kind === 'duplicate-record') + raw.server_auctions.push(structuredClone(raw.server_auctions[0])); + if (kind === 'duplicate-slot') + raw.server_auctions[0].slots.push(structuredClone(raw.server_auctions[0].slots[0])); + if (kind === 'conflicting-sidecar') + raw.slot_correlations.push({ + ...raw.slot_correlations[0], + slot_ref: 'ts-slot-00000001-1234-4abc-8def-123456789abc', + }); + if (kind === 'duplicate-cycle') + report.gpt_diagnostics.slots[0].requests.push( + structuredClone(report.gpt_diagnostics.slots[0].requests[0]) + ); + expect(view(report).auctions[0].slots[0].correlation).toBe('unknown'); + }); + it.each([ + ['prebid_refresh', 'Browser refresh observed; winner not determined'], + ['publisher_refresh', 'Browser refresh observed; winner not determined'], + ['competing', 'Multiple or unknown delivery paths'], + ['unattributed', 'Multiple or unknown delivery paths'], + ])('preserves %s path meaning rather than inferring a winner', (path, label) => { + const report = fixture(); + Object.assign(report.gpt_diagnostics.slots[0].requests[0], { + requestPath: path, + trustedServerCreativeResponseAtMs: undefined, + }); + Reflect.deleteProperty( + report.gpt_diagnostics.slots[0].requests[0], + 'trustedServerCreativeResponseAtMs' + ); + const slot = view(report).auctions[0].slots[0]; + expect(slot.pathLabel).toBe(label); + expect(slot.creativeLabel).not.toBe('Trusted Server creative rendered'); + }); + it('requires matched nonempty render and creative-bridge evidence for the participation label', () => { + const report = fixture(); + expect(view(report).auctions[0].slots[0].creativeLabel).toBe( + 'Trusted Server creative rendered' + ); + Object.assign(report.gpt_diagnostics.slots[0].requests[0], { isEmpty: true }); + expect(view(report).auctions[0].slots[0].creativeLabel).toBe('Participation unconfirmed'); + }); + it('cannot claim participation from a selected candidate and GPT fill alone', () => { + const report = fixture(); + report.server_auctions[0].slots[0].candidate = 'selected'; + Reflect.deleteProperty( + report.gpt_diagnostics.slots[0].requests[0], + 'trustedServerCreativeResponseAtMs' + ); + expect(view(report).auctions[0].slots[0].creativeLabel).toBe('Participation unconfirmed'); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/direct-api.test.ts b/crates/trusted-server-js/lib/test/trace/direct-api.test.ts new file mode 100644 index 000000000..b3214b5d8 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/direct-api.test.ts @@ -0,0 +1,233 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { buildAdRequest, sendAuction } from '../../src/core/auction'; +import type { TsjsApi } from '../../src/core/types'; +import { installTraceRuntime } from '../../src/trace/runtime'; + +import { SLOT_TOKEN, AUCTION_TOKEN } from './fixtures'; + +function request() { + return buildAdRequest([ + { code: 'example-slot', mediaTypes: { banner: { sizes: [[300, 250]] } } }, + ]); +} +function bids() { + return { + seatbid: [ + { + seat: 'example-bidder', + bid: [ + { + impid: 'example-slot', + adm: '
Example creative
', + price: 1, + w: 300, + h: 250, + }, + ], + }, + ], + }; +} +function transport() { + return { + schema_version: 1, + evidence: { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + source: 'auction_api', + terminal_status: 'completed', + provider_calls: [], + slots: [ + { + slot_number: 1, + slot_ref: SLOT_TOKEN, + requested_sizes: [[300, 250]], + returned_bid_count: 1, + candidate: 'selected', + selected_creative_size: [300, 250], + }, + ], + truncation: { + omitted_provider_calls: 0, + omitted_slots: 0, + omitted_nested_values: 0, + }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + }; +} +describe('direct API trace observer preserves ordinary bids', () => { + const fetch = vi.fn(); + beforeEach(() => { + window.tsjs = {} as TsjsApi; + window.__tsjs_trace_active = true; + vi.stubGlobal('fetch', fetch); + fetch.mockReset(); + vi.spyOn(window.crypto, 'randomUUID').mockReturnValue('12345678-1234-4abc-8def-123456789abc'); + installTraceRuntime(window.tsjs); + }); + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + delete window.tsjs; + delete window.__tsjs_trace_active; + }); + it('decorates final grouped units and records API evidence before parsing ordinary bids', async () => { + let parsedBids = false; + const result = { + ...bids(), + ext: { trusted_server: { trace_auction: transport() } }, + }; + Object.defineProperty(result, 'seatbid', { + get() { + parsedBids = true; + expect(window.tsjs?.traceEvidence?.snapshot().value?.serverAuctions).toHaveLength(1); + return bids().seatbid; + }, + }); + fetch.mockResolvedValueOnce({ + ok: true, + headers: new Headers({ 'content-type': 'application/json' }), + json: async () => result, + } as Response); + const ordinary = request(); + const received = await sendAuction('/auction', ordinary); + expect(parsedBids).toBe(true); + expect(received[0]).toMatchObject({ + impid: 'example-slot', + adm: '
Example creative
', + width: 300, + height: 250, + }); + const body = JSON.parse(fetch.mock.calls[0][1]?.body as string); + expect(body.adUnits[0].ext.trusted_server.trace_slot_ref).toBe(SLOT_TOKEN); + expect(ordinary.adUnits[0]).not.toHaveProperty('ext'); + expect(window.tsjs?.traceEvidence?.captureStatus()).toBe('complete'); + expect(window.tsjs?.traceEvidence?.snapshot().value?.issues).toEqual([ + 'correlation_unavailable', + ]); + expect(window.tsjs?.traceEvidence?.snapshot().value?.slotCorrelations).toEqual([]); + }); + it('records a supplied unreadable namespace as validation failure while preserving bids', async () => { + const trusted = new Proxy( + { trace_auction: transport() }, + { + getOwnPropertyDescriptor() { + throw new Error('private-descriptor-error'); + }, + } + ); + const response = { ...bids(), ext: { trusted_server: trusted } }; + fetch.mockResolvedValueOnce({ + ok: true, + headers: new Headers({ 'content-type': 'application/json' }), + json: async () => response, + } as Response); + expect(await sendAuction('/auction', request())).toHaveLength(1); + expect(window.tsjs?.traceEvidence?.snapshot().value?.issues).toEqual([ + 'evidence_validation_failed', + ]); + expect(JSON.stringify(window.tsjs?.traceEvidence?.snapshot())).not.toContain('private'); + }); + it.each([null, { schema_version: 1, unavailable_reason: 'evidence_projection_failed' }])( + 'preserves ordinary bid results when supplied evidence is invalid or unavailable', + async (value) => { + fetch.mockResolvedValueOnce( + new Response( + JSON.stringify({ + ...bids(), + ext: { trusted_server: { trace_auction: value } }, + }), + { headers: { 'content-type': 'application/json' } } + ) + ); + const received = await sendAuction('/auction', request()); + expect(received).toHaveLength(1); + expect(window.tsjs?.traceEvidence?.captureStatus()).toBe('unavailable'); + expect(window.tsjs?.traceEvidence?.snapshot().value?.issues).toEqual([ + value === null ? 'evidence_validation_failed' : 'evidence_projection_failed', + ]); + } + ); + it('adds no issue for absent optional response evidence and does not reset earlier records', async () => { + window.tsjs?.traceEvidence?.recordTransport(transport()); + fetch.mockResolvedValueOnce( + new Response(JSON.stringify(bids()), { + headers: { 'content-type': 'application/json' }, + }) + ); + expect(await sendAuction('/auction', request())).toHaveLength(1); + expect(window.tsjs?.traceEvidence?.captureStatus()).toBe('complete'); + }); + it.each(['non-ok', 'unreadable-json', 'fetch-rejected', 'non-json'])( + 'records only a bounded transport failure for %s', + async (failure) => { + if (failure === 'non-ok') + fetch.mockResolvedValueOnce(new Response('private-body', { status: 500 })); + if (failure === 'unreadable-json') + fetch.mockResolvedValueOnce( + new Response('private-invalid-json', { + headers: { 'content-type': 'application/json' }, + }) + ); + if (failure === 'fetch-rejected') + fetch.mockRejectedValueOnce(new Error('private-fetch-error')); + if (failure === 'non-json') + fetch.mockResolvedValueOnce( + new Response('private-body', { + headers: { 'content-type': 'text/plain' }, + }) + ); + expect(await sendAuction('/auction', request())).toEqual([]); + expect(window.tsjs?.traceEvidence?.snapshot().value?.issues).toEqual([ + 'evidence_transport_failed', + ]); + expect(JSON.stringify(window.tsjs?.traceEvidence?.snapshot())).not.toContain('private'); + } + ); + it('keeps the ordinary request and bids intact when token generation or collector callbacks throw', async () => { + vi.mocked(window.crypto.randomUUID).mockImplementation(() => { + throw new Error('private-random-error'); + }); + const ordinary = request(); + const prior = window.tsjs!.traceEvidence!; + window.tsjs!.traceEvidence = { + ...prior, + recordTransport() { + throw new Error('private-callback-error'); + }, + }; + fetch.mockResolvedValueOnce( + new Response( + JSON.stringify({ + ...bids(), + ext: { trusted_server: { trace_auction: transport() } }, + }), + { headers: { 'content-type': 'application/json' } } + ) + ); + expect(await sendAuction('/auction', ordinary)).toHaveLength(1); + expect(JSON.parse(fetch.mock.calls[0][1]?.body as string)).toEqual(ordinary); + }); + it.each([undefined, false, 'true', 1])( + 'adds no tokens or observers for nonliteral gate %s', + async (active) => { + window.__tsjs_trace_active = active; + fetch.mockResolvedValueOnce( + new Response( + JSON.stringify({ + ...bids(), + ext: { trusted_server: { trace_auction: transport() } }, + }), + { headers: { 'content-type': 'application/json' } } + ) + ); + const ordinary = request(); + expect(await sendAuction('/auction', ordinary)).toHaveLength(1); + expect(window.crypto.randomUUID).not.toHaveBeenCalled(); + expect(JSON.parse(fetch.mock.calls[0][1]?.body as string)).toEqual(ordinary); + expect(window.tsjs?.traceEvidence?.captureStatus()).toBe('not_observed'); + } + ); +}); diff --git a/crates/trusted-server-js/lib/test/trace/evidence.test.ts b/crates/trusted-server-js/lib/test/trace/evidence.test.ts new file mode 100644 index 000000000..c3960238e --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/evidence.test.ts @@ -0,0 +1,484 @@ +import { describe, expect, it } from 'vitest'; + +import type { TraceAuctionTransportV1 } from '../../src/trace/types'; +import { + validateTraceAuctionEvidence, + validateTraceAuctionTransport, + validateTraceSlotCorrelation, + parseTraceAuctionTransport, + parseTraceSlotCorrelation, +} from '../../src/trace/validation'; + +const AUCTION = 'ts-auc-1234567812344abc8def123456789abc'; +const SLOT = 'ts-slot-12345678-1234-4abc-8def-123456789abc'; + +describe('immutable evidence ingestion', () => { + it('rejects invalid own fields masked by caller get traps', () => { + const original = evidence(); + Object.assign(original, { terminal_status: 'private-invalid-enum' }); + let reads = 0; + const wrapped = new Proxy(original, { + get(target, key, receiver) { + reads += 1; + if (key === 'terminal_status') return 'completed'; + return Reflect.get(target, key, receiver); + }, + }); + expect(parseTraceAuctionTransport({ schema_version: 1, evidence: wrapped })).toBeUndefined(); + expect(reads).toBe(0); + const invalidSidecar = { + schema_version: 1, + diagnostic_auction_id: 'private-invalid-token', + slot_ref: SLOT, + runtime_slot_number: 1, + request_number: 1, + }; + expect( + parseTraceSlotCorrelation( + new Proxy(invalidSidecar, { + get(target, key, receiver) { + reads += 1; + if (key === 'diagnostic_auction_id') return AUCTION; + return Reflect.get(target, key, receiver); + }, + }) + ) + ).toBeUndefined(); + expect(reads).toBe(0); + }); + it('ingests recursively wrapped own data without reading caller properties', () => { + let reads = 0; + const wrap = (value: unknown): unknown => { + if (value === null || typeof value !== 'object') return value; + const data = Array.isArray(value) + ? value.map(wrap) + : Object.fromEntries(Object.entries(value).map(([key, item]) => [key, wrap(item)])); + return new Proxy(data, { + get() { + reads += 1; + throw new Error('private-get'); + }, + }); + }; + const value = parseTraceAuctionTransport(wrap({ schema_version: 1, evidence: evidence() })); + expect(value).toEqual({ schema_version: 1, evidence: evidence() }); + expect( + parseTraceSlotCorrelation( + wrap({ + schema_version: 1, + diagnostic_auction_id: AUCTION, + slot_ref: SLOT, + runtime_slot_number: 1, + request_number: 1, + }) + ) + ).toBeDefined(); + expect(reads).toBe(0); + }); + it('ignores caller-controlled array iteration methods when validating', () => { + const value = evidence(); + const invalid = [{ private_provider_name: 'private-provider' }]; + Object.assign(value, { + provider_calls: new Proxy(invalid, { + get(target, name, receiver) { + if (name === 'every') return () => true; + return Reflect.get(target, name, receiver); + }, + }), + }); + expect(validateTraceAuctionEvidence(value)).toBe(false); + expect(parseTraceAuctionTransport({ schema_version: 1, evidence: value })).toBeUndefined(); + }); + + it('copies array elements without invoking caller-controlled map or length', () => { + const value = evidence(); + const original = value.provider_calls; + let customCalls = 0; + value.provider_calls = new Proxy(original, { + get(target, name, receiver) { + if (name === 'map') + return () => { + customCalls += 1; + return original; + }; + if (name === 'length') { + customCalls += 1; + return 0; + } + return Reflect.get(target, name, receiver); + }, + }); + const result = parseTraceAuctionTransport({ schema_version: 1, evidence: value }); + expect(result).toBeDefined(); + expect(customCalls).toBe(0); + expect(Object.isFrozen(original)).toBe(false); + expect(Object.isFrozen(original[0])).toBe(false); + if (!result || result.evidence === undefined) throw new Error('should parse evidence'); + expect(result.evidence.provider_calls).not.toBe(original); + expect(result.evidence.provider_calls[0]).not.toBe(original[0]); + original[0].returned_bid_count = 99; + expect(result.evidence.provider_calls[0].returned_bid_count).toBe(0); + }); + it('keeps transport members exclusive at the public type boundary', () => { + const record = evidence(); + if (!validateTraceAuctionEvidence(record)) throw new Error('should validate fixture'); + const both = { + schema_version: 1, + evidence: record, + unavailable_reason: 'evidence_projection_failed', + } as const; + // @ts-expect-error Both transport alternatives cannot be present together. + const invalid: TraceAuctionTransportV1 = both; + expect(validateTraceAuctionTransport(invalid)).toBe(false); + }); + it('copies and freezes every retained field without retaining source arrays', () => { + const original = evidence(); + Object.assign(original, { total_time_ms: 0, terminal_reason: 'unknown' }); + Object.assign(original.provider_calls[0], { response_time_ms: 0 }); + Object.assign(original.slots[0], { selected_creative_size: [300, 250] }); + const envelope = { schema_version: 1, evidence: original }; + const result = parseTraceAuctionTransport(envelope); + expect(result).toEqual(envelope); + expect(result).not.toBe(envelope); + if (!result || result.evidence === undefined) throw new Error('should parse evidence'); + expect(result.evidence).not.toBe(original); + const assertFrozen = (value: unknown): void => { + if (typeof value !== 'object' || value === null) return; + expect(Object.isFrozen(value)).toBe(true); + Object.values(value).forEach(assertFrozen); + }; + assertFrozen(result); + original.slots[0].requested_sizes[0][0] = 999; + original.provider_calls[0].returned_bid_count = 99; + expect(result.evidence.slots[0].requested_sizes[0]).toEqual([300, 250]); + expect(result.evidence.provider_calls[0].returned_bid_count).toBe(0); + expect(Object.isFrozen(original)).toBe(false); + }); + + it('copies an explicit unavailable transport and the exact correlation decision', () => { + const unavailable = { + schema_version: 1, + unavailable_reason: 'evidence_projection_failed', + }; + const result = parseTraceAuctionTransport(unavailable); + expect(result).toEqual(unavailable); + expect(Object.isFrozen(result)).toBe(true); + const original = { + schema_version: 1, + diagnostic_auction_id: AUCTION, + slot_ref: SLOT, + runtime_slot_number: 1, + request_number: 2, + }; + const sidecar = parseTraceSlotCorrelation(original); + expect(sidecar).toEqual(original); + expect(Object.isFrozen(sidecar)).toBe(true); + original.request_number = 3; + expect(sidecar?.request_number).toBe(2); + }); + + it('rejects invalid inputs without serialization or getter invocation', () => { + let calls = 0; + const value = { + schema_version: 1, + evidence: evidence(), + toJSON: () => { + calls += 1; + throw new Error('should not call'); + }, + }; + expect(parseTraceAuctionTransport(value)).toBeUndefined(); + expect(parseTraceSlotCorrelation(value)).toBeUndefined(); + expect(calls).toBe(0); + expect( + parseTraceAuctionTransport( + new Proxy( + {}, + { + getPrototypeOf() { + throw new Error('should reject'); + }, + } + ) + ) + ).toBeUndefined(); + }); + + it('copies stable own facts without invoking changing property reads', () => { + let reads = 0; + const source = new Proxy(evidence(), { + get(target, name, receiver) { + if (name === 'diagnostic_auction_id' && ++reads > 1) return 'private-id'; + return Reflect.get(target, name, receiver); + }, + }); + expect(parseTraceAuctionTransport({ schema_version: 1, evidence: source })).toEqual({ + schema_version: 1, + evidence: evidence(), + }); + expect(reads).toBe(0); + }); +}); + +function evidence() { + return { + schema_version: 1, + diagnostic_auction_id: AUCTION, + source: 'initial_navigation_ssat', + terminal_status: 'completed', + provider_calls: [ + { + provider_number: 1, + role: 'bidder', + status: 'success', + returned_bid_count: 0, + }, + ], + slots: [ + { + slot_number: 1, + slot_ref: SLOT, + requested_sizes: [[300, 250]], + returned_bid_count: 0, + candidate: 'no_candidate', + }, + ], + truncation: { + omitted_provider_calls: 0, + omitted_slots: 0, + omitted_nested_values: 0, + }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }; +} + +describe('bounded server auction evidence contracts', () => { + it('accepts zero-bid evidence, empty observations and numeric boundaries', () => { + const value = evidence(); + expect(validateTraceAuctionEvidence(value)).toBe(true); + expect(validateTraceAuctionEvidence({ ...value, provider_calls: [], slots: [] })).toBe(true); + Object.assign(value, { + total_time_ms: 4_294_967_295, + terminal_reason: 'unknown', + }); + Object.assign(value.provider_calls[0], { + response_time_ms: 4_294_967_295, + returned_bid_count: 65_535, + }); + Object.assign(value.slots[0], { + selected_creative_size: [100_000, 1], + returned_bid_count: 65_535, + }); + expect(validateTraceAuctionEvidence(value)).toBe(true); + }); + + it.each(['initial_navigation_ssat', 'spa_page_bids', 'auction_api'])( + 'accepts source %s', + (source) => { + expect(validateTraceAuctionEvidence({ ...evidence(), source })).toBe(true); + } + ); + it.each(['completed', 'execution_failed', 'dispatch_failed', 'abandoned', 'skipped'])( + 'accepts terminal status %s', + (terminal_status) => { + expect(validateTraceAuctionEvidence({ ...evidence(), terminal_status })).toBe(true); + } + ); + it.each([ + 'policy_skipped', + 'no_eligible_slots', + 'no_provider_launched', + 'provider_execution_failed', + 'collection_failed', + 'unknown', + ])('accepts terminal reason %s', (terminal_reason) => { + expect(validateTraceAuctionEvidence({ ...evidence(), terminal_reason })).toBe(true); + }); + it.each(['bidder', 'mediator', 'unknown'])('accepts provider role %s', (role) => { + const value = evidence(); + Object.assign(value.provider_calls[0], { role }); + expect(validateTraceAuctionEvidence(value)).toBe(true); + }); + it.each(['success', 'no_bid', 'error', 'pending', 'abandoned', 'unknown'])( + 'accepts provider status %s', + (status) => { + const value = evidence(); + Object.assign(value.provider_calls[0], { status }); + expect(validateTraceAuctionEvidence(value)).toBe(true); + } + ); + it.each(['selected', 'no_candidate', 'selected_unrenderable', 'unknown'])( + 'accepts candidate %s', + (candidate) => { + const value = evidence(); + Object.assign(value.slots[0], { candidate }); + expect(validateTraceAuctionEvidence(value)).toBe(true); + } + ); + + it.each([ + { schema_version: 2 }, + { diagnostic_auction_id: 'internal-id-sentinel' }, + { source: 'initial_navigation' }, + { terminal_status: 'timeout' }, + { terminal_reason: 'raw-error-sentinel' }, + { total_time_ms: 4_294_967_296 }, + { total_time_ms: -1 }, + { total_time_ms: 0.5 }, + { total_time_ms: Infinity }, + { provider_calls: undefined }, + { coverage: { provider_to_slot_no_bid: 'known' } }, + { provider_name: 'fictional-private-provider' }, + { price: 1 }, + { + truncation: { + omitted_provider_calls: 65_536, + omitted_slots: 0, + omitted_nested_values: 0, + }, + }, + ])('rejects extra fields, unsafe numbers and unknown enums %j', (fields) => { + expect(validateTraceAuctionEvidence({ ...evidence(), ...fields })).toBe(false); + }); + + it('rejects unknown properties at every nested boundary', () => { + for (const target of ['provider', 'slot', 'truncation', 'coverage'] as const) { + const value = evidence(); + Object.assign( + target === 'provider' + ? value.provider_calls[0] + : target === 'slot' + ? value.slots[0] + : value[target], + { secret: 'private-sentinel' } + ); + expect(validateTraceAuctionEvidence(value)).toBe(false); + } + }); + + it.each([ + { provider_number: 0 }, + { provider_number: 65_536 }, + { provider_number: 1.5 }, + { role: 'named-provider' }, + { status: 'raw-error' }, + { returned_bid_count: -1 }, + { returned_bid_count: 65_536 }, + { response_time_ms: 4_294_967_296 }, + ])('rejects invalid provider fields %j', (fields) => { + const value = evidence(); + Object.assign(value.provider_calls[0], fields); + expect(validateTraceAuctionEvidence(value)).toBe(false); + }); + + it.each([ + { slot_number: 0 }, + { slot_number: 65_536 }, + { slot_ref: 'raw-slot-sentinel' }, + { candidate: 'filled' }, + { requested_sizes: [[0, 250]] }, + { requested_sizes: [[300, 0]] }, + { requested_sizes: [[100_001, 1]] }, + { requested_sizes: [[300.5, 250]] }, + { requested_sizes: [[300, 250, 1]] }, + { selected_creative_size: [0, 250] }, + { selected_creative_size: [300, 250, 1] }, + { returned_bid_count: 65_536 }, + ])('rejects invalid slot fields %j', (fields) => { + const value = evidence(); + Object.assign(value.slots[0], fields); + expect(validateTraceAuctionEvidence(value)).toBe(false); + }); + + it('rejects oversized inner models without truncating them', () => { + const value = evidence(); + expect( + validateTraceAuctionEvidence({ + ...value, + provider_calls: Array.from({ length: 17 }, () => value.provider_calls[0]), + }) + ).toBe(false); + expect( + validateTraceAuctionEvidence({ + ...value, + slots: Array.from({ length: 65 }, () => value.slots[0]), + }) + ).toBe(false); + Object.assign(value.slots[0], { + requested_sizes: Array.from({ length: 17 }, () => [300, 250]), + }); + expect(validateTraceAuctionEvidence(value)).toBe(false); + }); + + it('does not invoke forbidden accessors or throw on proxy inspection', () => { + const value = Object.defineProperty(evidence(), 'source', { + enumerable: true, + get: () => { + throw new Error('private-sentinel'); + }, + }); + expect(validateTraceAuctionEvidence(value)).toBe(false); + expect( + validateTraceAuctionEvidence( + new Proxy( + {}, + { + getPrototypeOf: () => { + throw new Error('private-sentinel'); + }, + } + ) + ) + ).toBe(false); + }); + + it('accepts exactly one evidence-or-unavailable transport member', () => { + expect( + validateTraceAuctionTransport({ + schema_version: 1, + evidence: evidence(), + }) + ).toBe(true); + expect( + validateTraceAuctionTransport({ + schema_version: 1, + unavailable_reason: 'evidence_projection_failed', + }) + ).toBe(true); + for (const value of [ + {}, + { schema_version: 1 }, + { schema_version: 2, evidence: evidence() }, + { + schema_version: 1, + evidence: evidence(), + unavailable_reason: 'evidence_projection_failed', + }, + { schema_version: 1, unavailable_reason: 'private-parser-error' }, + { schema_version: 1, evidence: evidence(), secret: 'private-sentinel' }, + ]) + expect(validateTraceAuctionTransport(value)).toBe(false); + }); + + it('accepts an exact trace-only sidecar with safe positive browser sequence numbers', () => { + const value = { + schema_version: 1, + diagnostic_auction_id: AUCTION, + slot_ref: SLOT, + runtime_slot_number: 1, + request_number: Number.MAX_SAFE_INTEGER, + }; + expect(validateTraceSlotCorrelation(value)).toBe(true); + for (const fields of [ + { schema_version: 2 }, + { diagnostic_auction_id: 'private-auction-id' }, + { slot_ref: 'private-dom-id' }, + { runtime_slot_number: 0 }, + { request_number: 0 }, + { request_number: 0.5 }, + { request_number: Number.MAX_SAFE_INTEGER + 1 }, + { slotElementId: 'private-slot-sentinel' }, + ]) + expect(validateTraceSlotCorrelation({ ...value, ...fields })).toBe(false); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/export.test.ts b/crates/trusted-server-js/lib/test/trace/export.test.ts new file mode 100644 index 000000000..202b38aee --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/export.test.ts @@ -0,0 +1,181 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { + copyTraceReport, + downloadTraceReport, + formatTraceReport, + shareTraceReport, +} from '../../src/trace/export'; + +import { reportFixture, TRACE_NOW, TRACE_ORIGIN } from './fixtures'; + +describe('equivalent explicit local trace exports', () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + document.body.replaceChildren(); + }); + it('formats only the validated identical public report for all export paths', () => { + const report = reportFixture(); + const json = formatTraceReport(report, TRACE_ORIGIN, TRACE_NOW); + expect(json).toBe(JSON.stringify(report, null, 2)); + expect(JSON.parse(json!)).toEqual(report); + expect(json).not.toContain('stored_at_ms'); + expect(JSON.parse(json!).request_context.cookies.diagnostics_session.detail).toBe( + 'runtime_header_ambiguous' + ); + expect( + formatTraceReport({ ...report, private_value: 'secret' }, TRACE_ORIGIN, TRACE_NOW) + ).toBeUndefined(); + }); + it('downloads the same report with deferred independent cleanup for repeated taps', () => { + const created: Blob[] = []; + const create = vi.fn((blob: Blob) => { + created.push(blob); + return `blob:trace-${created.length}`; + }); + const revoke = vi.fn(); + vi.stubGlobal( + 'URL', + class extends URL { + static createObjectURL = create; + static revokeObjectURL = revoke; + } + ); + const clicks: { href: string; download: string; connected: boolean }[] = []; + vi.spyOn(HTMLAnchorElement.prototype, 'click').mockImplementation(function ( + this: HTMLAnchorElement + ) { + clicks.push({ href: this.href, download: this.download, connected: this.isConnected }); + expect(revoke).not.toHaveBeenCalled(); + }); + expect(downloadTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'downloaded', + }); + expect(downloadTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'downloaded', + }); + expect(clicks).toEqual([ + { href: 'blob:trace-1', download: 'trusted-server-trace-v1.json', connected: true }, + { href: 'blob:trace-2', download: 'trusted-server-trace-v1.json', connected: true }, + ]); + expect(document.querySelector('a')).toBeNull(); + expect(created.map((blob) => blob.type)).toEqual(['application/json', 'application/json']); + vi.advanceTimersByTime(999); + expect(revoke).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1); + expect(revoke.mock.calls).toEqual([['blob:trace-1'], ['blob:trace-2']]); + }); + it('still schedules URL cleanup when the browser download click throws', () => { + const revoke = vi.fn(); + vi.stubGlobal( + 'URL', + class extends URL { + static createObjectURL = vi.fn(() => 'blob:failed-click'); + static revokeObjectURL = revoke; + } + ); + vi.spyOn(HTMLAnchorElement.prototype, 'click').mockImplementation(() => { + throw new Error('private-click-error'); + }); + expect(downloadTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'failed', + }); + expect(document.querySelector('a')).toBeNull(); + expect(revoke).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1000); + expect(revoke).toHaveBeenCalledWith('blob:failed-click'); + }); + it('copies formatted report JSON only when the explicit action runs', async () => { + const writeText = vi.fn(async () => undefined); + vi.stubGlobal('navigator', { clipboard: { writeText } }); + expect(writeText).not.toHaveBeenCalled(); + expect(await copyTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'copied', + }); + expect(writeText).toHaveBeenCalledWith(JSON.stringify(reportFixture(), null, 2)); + writeText.mockRejectedValueOnce(new Error('private-clipboard-error')); + expect(await copyTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'failed', + }); + }); + it('returns unsupported without attempting clipboard or URL fallback', async () => { + const fetch = vi.fn(); + vi.stubGlobal('fetch', fetch); + vi.stubGlobal('navigator', {}); + expect(await copyTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'unsupported', + }); + expect(await shareTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'unsupported', + }); + expect(fetch).not.toHaveBeenCalled(); + }); + it('shares a JSON File with exactly the formatted report and no URL', async () => { + let offered: File | undefined; + const canShare = vi.fn((data: ShareData) => { + offered = data.files?.[0]; + return true; + }); + const share = vi.fn(async (_data: ShareData) => undefined); + vi.stubGlobal('navigator', { canShare, share }); + expect(await shareTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'shared', + }); + expect(offered?.name).toBe('trusted-server-trace-v1.json'); + expect(offered?.type).toBe('application/json'); + expect(Object.keys(share.mock.calls[0][0])).toEqual(['files']); + expect(share.mock.calls[0][0].files?.[0]).toBe(offered); + expect(offered?.size).toBe( + new TextEncoder().encode(JSON.stringify(reportFixture(), null, 2)).length + ); + }); + it('leaves the valid model available after share refusal or rejection', async () => { + const report = reportFixture(); + const share = vi.fn(async () => undefined); + vi.stubGlobal('navigator', { canShare: () => false, share }); + expect(await shareTraceReport(report, TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'unsupported', + }); + expect(share).not.toHaveBeenCalled(); + vi.stubGlobal('navigator', { + canShare: () => true, + share: async () => { + throw new DOMException('User cancelled', 'AbortError'); + }, + }); + expect(await shareTraceReport(report, TRACE_ORIGIN, TRACE_NOW)).toEqual({ status: 'failed' }); + expect(formatTraceReport(report, TRACE_ORIGIN, TRACE_NOW)).toBe( + JSON.stringify(report, null, 2) + ); + }); + it('declines every action before browser side effects for an invalid report', async () => { + const writeText = vi.fn(); + const share = vi.fn(); + const create = vi.fn(); + vi.stubGlobal('navigator', { clipboard: { writeText }, canShare: () => true, share }); + vi.stubGlobal( + 'URL', + class extends URL { + static createObjectURL = create; + } + ); + const invalid = { ...reportFixture(), private_value: 'secret' }; + expect(downloadTraceReport(invalid, TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'invalid_report', + }); + expect(await copyTraceReport(invalid, TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'invalid_report', + }); + expect(await shareTraceReport(invalid, TRACE_ORIGIN, TRACE_NOW)).toEqual({ + status: 'invalid_report', + }); + expect(writeText).not.toHaveBeenCalled(); + expect(share).not.toHaveBeenCalled(); + expect(create).not.toHaveBeenCalled(); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/fixtures.ts b/crates/trusted-server-js/lib/test/trace/fixtures.ts new file mode 100644 index 000000000..57a82355d --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/fixtures.ts @@ -0,0 +1,218 @@ +import type { GptDiagnosticsExportV1 } from '../../src/core/types'; + +export const TRACE_ORIGIN = 'https://publisher.example.com'; +export const TRACE_TIME = '2026-10-05T10:15:30.123Z'; +export const TRACE_NOW = Date.parse(TRACE_TIME); +export const AUCTION_TOKEN = 'ts-auc-1234567812344abc8def123456789abc'; +export const SLOT_TOKEN = 'ts-slot-12345678-1234-4abc-8def-123456789abc'; + +/** Complete current TS Console fixture with separate forbidden identifier sentinels. */ +export function gptSourceFixture(): GptDiagnosticsExportV1 { + return { + version: 1, + capturedAt: TRACE_TIME, + page: { origin: TRACE_ORIGIN, pathname: '/private-page-secret?private-query=secret' }, + slots: [ + { + runtimeSlotNumber: 1, + slotElementId: 'private-slot-element', + adUnitPath: 'private-ad-unit', + binding: { status: 'bound', reason: 'missing_element' }, + currentVisibilityPercentage: 12.5, + maximumVisibilityPercentage: 100, + requests: [ + { + requestNumber: 1, + requestedAtMs: 1.5, + responseAtMs: 2, + renderAtMs: 3, + loadAtMs: 4, + viewableAtMs: 5, + durations: { + requestToResponseMs: 0.5, + responseToRenderMs: 1, + requestToRenderMs: 1.5, + renderToLoadMs: 1, + renderToViewableMs: 2, + }, + isEmpty: false, + requestedSlotSizes: [ + [300, 250], + [320, 50], + ], + size: [300, 250], + observedSlotSize: [0, 0], + isBackfill: true, + slotContentChanged: true, + incompleteSequence: false, + adManager: { + lineItemId: 900001, + creativeId: 900002, + campaignId: 900003, + advertiserId: 900004, + sourceAgnosticLineItemId: 900005, + sourceAgnosticCreativeId: 900006, + yieldGroupIds: [900007], + companyIds: [900008], + }, + responseClass: 'backfill', + requestPath: 'trusted_server_direct', + requestIntentId: 1, + trustedServerAuctionId: AUCTION_TOKEN, + opportunityToRequestMs: 0.5, + replacedRequestNumber: 0, + previousRenderToRequestMs: 0.5, + creativeChanged: true, + previousCreativeId: 900009, + loadObservedBeforeRender: false, + trustedServerOpportunity: 'renderable_candidate', + trustedServerCreativeRequestAtMs: 1, + trustedServerCreativeResponseAtMs: 2, + trustedServerCreativeFailures: [ + 'missing_render_source', + 'cache_fetch_failed', + 'invalid_cache_payload', + 'response_post_failed', + ], + delivery: 'trusted_server_response_sent', + }, + ], + }, + ], + callbackIssues: [ + { + kind: 'slotRenderEnded', + runtimeSlotNumber: 1, + slotElementId: 'private-callback-element', + timestampMs: 0.5, + disposition: 'matched', + reason: 'invalid_event_order', + }, + ], + attributionIssues: [ + { + reason: 'creative_request_without_slot', + timestampMs: 0.5, + runtimeSlotNumber: 1, + slotElementId: 'private-attribution-element', + }, + ], + coverage: { + slotRequested: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + slotResponseReceived: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + slotRenderEnded: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + slotOnload: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + impressionViewable: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + slotVisibilityChanged: { observed: 1, matched: 1, unmatched: 0, ambiguous: 0 }, + }, + metadata: { + droppedCallbacks: 0, + droppedAttributionIssues: 0, + evictedSlots: 0, + evictedRequestCycles: 0, + }, + }; +} + +/** Valid projected fixture built explicitly without any excluded source fields. */ +export function projectedGptFixture() { + const source = gptSourceFixture(); + const cycle = source.slots[0].requests[0]; + return { + schema_version: 1, + source_schema_version: 1, + capturedAt: TRACE_TIME, + page: { origin: TRACE_ORIGIN, pathname: '/[redacted]' }, + slots: [ + { + runtimeSlotNumber: 1, + binding: { status: 'bound', reason: 'missing_element' }, + currentVisibilityPercentage: 12.5, + maximumVisibilityPercentage: 100, + requests: [ + { + requestNumber: 1, + requestedAtMs: 1.5, + responseAtMs: 2, + renderAtMs: 3, + loadAtMs: 4, + viewableAtMs: 5, + durations: cycle.durations, + isEmpty: false, + requestedSlotSizes: [ + [300, 250], + [320, 50], + ], + size: [300, 250], + observedSlotSize: [0, 0], + isBackfill: true, + slotContentChanged: true, + incompleteSequence: false, + responseClass: 'backfill', + requestPath: 'trusted_server_direct', + requestIntentId: 1, + trustedServerAuctionId: AUCTION_TOKEN, + opportunityToRequestMs: 0.5, + replacedRequestNumber: 0, + previousRenderToRequestMs: 0.5, + creativeChanged: true, + loadObservedBeforeRender: false, + trustedServerOpportunity: 'renderable_candidate', + trustedServerCreativeRequestAtMs: 1, + trustedServerCreativeResponseAtMs: 2, + trustedServerCreativeFailures: cycle.trustedServerCreativeFailures, + delivery: 'trusted_server_response_sent', + }, + ], + }, + ], + callbackIssues: [ + { + kind: 'slotRenderEnded', + runtimeSlotNumber: 1, + timestampMs: 0.5, + disposition: 'matched', + reason: 'invalid_event_order', + }, + ], + attributionIssues: [ + { reason: 'creative_request_without_slot', timestampMs: 0.5, runtimeSlotNumber: 1 }, + ], + coverage: source.coverage, + metadata: source.metadata, + }; +} + +export function reportFixture() { + return { + schema_version: 1, + captured_at: TRACE_TIME, + request_context: { + schema_version: 1, + captured_at: '2026-10-05T09:15:30Z', + network: {}, + cookies: { + ts_ec: { source: 'request', state: 'absent' }, + ts_eids: { source: 'request', state: 'absent' }, + ts_tester: { source: 'request', state: 'absent' }, + diagnostics_session: { + source: 'request', + state: 'unavailable', + detail: 'runtime_header_ambiguous', + }, + }, + }, + server_auctions: [], + slot_correlations: [], + gpt_diagnostics: projectedGptFixture(), + auction_coverage: { capture_status: 'not_observed', issues: [] }, + truncation: { + omitted_server_auctions: 0, + omitted_slot_correlations: 0, + omitted_request_cycles: 0, + omitted_callback_issues: 0, + omitted_attribution_issues: 0, + omitted_nested_values: 0, + }, + }; +} diff --git a/crates/trusted-server-js/lib/test/trace/gpt-fixtures.ts b/crates/trusted-server-js/lib/test/trace/gpt-fixtures.ts new file mode 100644 index 000000000..1d25e6128 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/gpt-fixtures.ts @@ -0,0 +1,61 @@ +import type { GptDiagnosticsStore } from '../../src/integrations/gpt_diagnostics/store'; +import { GptDiagnosticsApiController } from '../../src/integrations/gpt_diagnostics/api'; +import type { TraceCollector } from '../../src/trace/collector'; +import { joinTraceEvidence } from '../../src/trace/correlation'; +import { buildTraceReport } from '../../src/trace/report'; + +import { AUCTION_TOKEN, SLOT_TOKEN, TRACE_NOW, TRACE_ORIGIN, reportFixture } from './fixtures'; + +/** Returns the actual public API snapshot, retaining its optional undefined members. */ +export function observedGptSource(store: GptDiagnosticsStore) { + const controller = new GptDiagnosticsApiController( + store, + { + exportBinding: () => ({ status: 'bound' }), + subscribe: () => () => {}, + }, + { show: () => {}, hide: () => {} }, + { now: () => new Date(TRACE_NOW) } + ); + const source = controller.snapshot(); + controller.destroy(); + return { ...source, page: { ...source.page, origin: TRACE_ORIGIN } }; +} + +/** Exercises the real public-cycle projection and exact V4 join with store observations. */ +export function joinedGptStore(store: GptDiagnosticsStore, collector: TraceCollector) { + const source = observedGptSource(store); + const captured = buildTraceReport({ + requestContext: reportFixture().request_context, + gptSource: source, + origin: TRACE_ORIGIN, + capturedAtMs: TRACE_NOW, + collector: collector.snapshot().value, + }); + if (!captured.ok) throw new Error(`should capture observed GPT cycles: ${captured.reason}`); + return joinTraceEvidence(captured.value.report, TRACE_ORIGIN, TRACE_NOW); +} + +export function gptTransport(source = 'initial_navigation_ssat') { + return { + schema_version: 1, + evidence: { + schema_version: 1, + diagnostic_auction_id: AUCTION_TOKEN, + source, + terminal_status: 'completed', + provider_calls: [], + slots: [ + { + slot_number: 1, + slot_ref: SLOT_TOKEN, + requested_sizes: [[300, 250]], + returned_bid_count: 0, + candidate: 'no_candidate', + }, + ], + truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }, + }; +} diff --git a/crates/trusted-server-js/lib/test/trace/gpt-transport.test.ts b/crates/trusted-server-js/lib/test/trace/gpt-transport.test.ts new file mode 100644 index 000000000..ddc0e8525 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/gpt-transport.test.ts @@ -0,0 +1,176 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import type { AuctionSlot, TsjsApi } from '../../src/core/types'; +import { installTraceRuntime } from '../../src/trace/runtime'; + +import { AUCTION_TOKEN, SLOT_TOKEN } from './fixtures'; +import { gptTransport } from './gpt-fixtures'; + +function slot(ref: unknown = SLOT_TOKEN): AuctionSlot { + return { + id: 'private-id', + gam_unit_path: '/private-path', + div_id: 'private-div', + formats: [[300, 250]], + ext: { trusted_server: { trace_slot_ref: ref } }, + }; +} +function setup(active: unknown = true) { + const api = {} as TsjsApi; + const scope = { tsjs: api, __tsjs_trace_active: active }; + const collector = installTraceRuntime(api, scope); + return { api, collector }; +} +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe('shared SSAT/SPA slot identity bridge', () => { + it('retains evidence but binds no prefix when delivered slots exceed the 64-slot cap', () => { + const { api, collector } = setup(); + const first = slot(); + const slots = [first, ...Array.from({ length: 64 }, () => slot('private-invalid'))]; + // The tail may hide a duplicate, so a bounded inspection cannot bind even a valid prefix. + api.traceGpt!.observeTransport(slots, gptTransport(), 'initial_navigation_ssat'); + expect(collector!.captureStatus()).toBe('complete'); + expect(collector!.snapshot().value?.serverAuctions).toHaveLength(1); + expect(collector!.snapshot().value?.issues).toEqual(['correlation_unavailable']); + expect(api.traceGpt!.identity(first)).toBeUndefined(); + expect(slots).toHaveLength(65); + expect(slots[0]).toBe(first); + }); + it('retains validated server capture when only delivered array descriptors are unreadable', () => { + const { api, collector } = setup(); + const slots = new Proxy([slot()], { + getOwnPropertyDescriptor: () => { + throw new Error('private-descriptor'); + }, + }); + expect(() => + api.traceGpt!.observeTransport(slots, gptTransport(), 'initial_navigation_ssat') + ).not.toThrow(); + expect(collector!.captureStatus()).toBe('complete'); + expect(collector!.snapshot().value?.issues).toEqual(['correlation_unavailable']); + }); + it.each([false, undefined, 'true', 1])( + 'allocates no bridge or bindings for gate %s', + (active) => { + const api = {} as TsjsApi; + const collector = installTraceRuntime(api, { tsjs: api, __tsjs_trace_active: active }); + expect(collector).toBeUndefined(); + expect(api.traceGpt).toBeUndefined(); + } + ); + it('retains owned evidence and exact object bindings without changing slots or reading excluded fields', () => { + const { api, collector } = setup(); + const original = slot(); + const unsafe = vi.fn(() => { + throw new Error('private-field'); + }); + Object.defineProperty(original.ext!, 'private_extension', { get: unsafe }); + Object.defineProperty(original, 'private_field', { get: unsafe }); + const transport = gptTransport(); + const reads = vi.fn(() => { + throw new Error('private-get'); + }); + const owned = new Proxy(transport, { get: reads }); + api.traceGpt!.observeTransport([original], owned, 'initial_navigation_ssat'); + expect(collector!.captureStatus()).toBe('complete'); + expect(api.traceGpt!.identity(original)).toEqual({ + diagnostic_auction_id: AUCTION_TOKEN, + slot_ref: SLOT_TOKEN, + }); + expect(api.traceGpt!.identity({ ...original })).toBeUndefined(); + expect(unsafe).not.toHaveBeenCalled(); + expect(reads).not.toHaveBeenCalled(); + expect(Object.isFrozen(original)).toBe(false); + expect(Object.isFrozen(transport)).toBe(false); + const identity = api.traceGpt!.identity(original)!; + expect(Object.isFrozen(identity)).toBe(true); + original.ext!.trusted_server!.trace_slot_ref = 'private-mutated'; + transport.evidence.slots[0]!.slot_ref = 'private-mutated'; + expect(identity.slot_ref).toBe(SLOT_TOKEN); + expect(collector!.snapshot().value?.serverAuctions[0]!.slots[0]!.slot_ref).toBe(SLOT_TOKEN); + }); + it.each([ + 'missing', + 'malformed', + 'duplicate-delivered', + 'duplicate-evidence', + 'conflicting', + 'accessor', + 'throwing-descriptor', + ] as const)('limits only correlation for a %s delivered reference', (kind) => { + const { api, collector } = setup(); + const original = slot(); + const slots = [original]; + const transport = gptTransport(); + if (kind === 'missing') delete original.ext; + if (kind === 'malformed') original.ext!.trusted_server!.trace_slot_ref = 'private-token'; + if (kind === 'duplicate-delivered') slots.push(slot()); + if (kind === 'duplicate-evidence') + transport.evidence.slots.push({ ...transport.evidence.slots[0]!, slot_number: 2 }); + if (kind === 'conflicting') + original.ext!.trusted_server!.trace_slot_ref = 'ts-slot-12345679-1234-4abc-8def-123456789abc'; + const getter = vi.fn(() => { + throw new Error('private-reference'); + }); + if (kind === 'accessor') + Object.defineProperty(original.ext!.trusted_server!, 'trace_slot_ref', { get: getter }); + if (kind === 'throwing-descriptor') + original.ext = new Proxy(original.ext!, { getOwnPropertyDescriptor: getter }); + api.traceGpt!.observeTransport(slots, transport, 'initial_navigation_ssat'); + expect(collector!.captureStatus()).toBe('complete'); + expect(collector!.snapshot().value?.issues).toEqual(['correlation_unavailable']); + expect(api.traceGpt!.identity(original)).toBeUndefined(); + expect(slots[0]).toBe(original); + if (kind === 'accessor') expect(getter).not.toHaveBeenCalled(); + }); + it('preserves earlier collector capture but clears stale bindings when the next optional member is absent', () => { + const { api, collector } = setup(); + const original = slot(); + api.traceGpt!.observeTransport([original], gptTransport(), 'initial_navigation_ssat'); + api.traceGpt!.observeTransport([original], undefined, 'initial_navigation_ssat'); + expect(api.traceGpt!.identity(original)).toBeUndefined(); + expect(collector!.snapshot().value?.serverAuctions).toHaveLength(1); + expect(collector!.snapshot().value?.issues).toEqual([]); + }); + it.each(['malformed', 'projection', 'api', 'wrong-source'] as const)( + 'keeps %s transport handling separate from correlation', + (kind) => { + const { api, collector } = setup(); + const original = slot(); + const value = + kind === 'projection' + ? { schema_version: 1, unavailable_reason: 'evidence_projection_failed' } + : kind === 'malformed' + ? { schema_version: 1 } + : gptTransport(kind === 'api' ? 'auction_api' : 'spa_page_bids'); + api.traceGpt!.observeTransport([original], value, 'initial_navigation_ssat'); + expect(api.traceGpt!.identity(original)).toBeUndefined(); + expect(collector!.snapshot().value?.issues).toEqual([ + kind === 'projection' ? 'evidence_projection_failed' : 'evidence_validation_failed', + ]); + } + ); + it('reads only an own data SPA transport member without caller getters or unrelated extensions', () => { + const { api, collector } = setup(); + const original = slot(); + const getter = vi.fn(() => { + throw new Error('private-member'); + }); + api.traceGpt!.observePageBids(Object.create({ trace_auction: gptTransport('spa_page_bids') }), [ + original, + ]); + expect(collector!.snapshot().value?.issues).toEqual([]); + const response = { trace_auction: gptTransport('spa_page_bids') }; + api.traceGpt!.observePageBids(new Proxy(response, { get: getter }), [original]); + expect(collector!.captureStatus()).toBe('complete'); + expect(getter).not.toHaveBeenCalled(); + api.traceGpt!.observePageBids(Object.defineProperty({}, 'trace_auction', { get: getter }), [ + original, + ]); + expect(collector!.snapshot().value?.issues).toEqual(['evidence_validation_failed']); + expect(getter).not.toHaveBeenCalled(); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/handoff.test.ts b/crates/trusted-server-js/lib/test/trace/handoff.test.ts new file mode 100644 index 000000000..a9a9b5eb7 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/handoff.test.ts @@ -0,0 +1,196 @@ +import { describe, expect, it, vi } from 'vitest'; + +import type { TsjsApi } from '../../src/core/types'; +import { createTraceCollector } from '../../src/trace/collector'; +import type { downloadTraceReport } from '../../src/trace/export'; +import { createTraceHandoff } from '../../src/trace/handoff'; + +import { gptSourceFixture, reportFixture, TRACE_NOW, TRACE_ORIGIN } from './fixtures'; + +function setup(active: unknown = true) { + const order: string[] = []; + const snapshot = vi.fn(() => { + order.push('snapshot'); + return gptSourceFixture(); + }); + const collector = createTraceCollector(); + const target = { + __tsjs_trace_active: active, + __tsjs_trace_request_context: reportFixture().request_context, + tsjs: { + traceEvidence: collector, + gptDiagnostics: { snapshot }, + } as unknown as TsjsApi, + location: { + origin: TRACE_ORIGIN, + assign: vi.fn((path: string) => { + order.push(`navigate:${path}`); + }), + }, + sessionStorage: { + getItem: vi.fn(() => null), + setItem: vi.fn((_key: string, _value: string) => { + order.push('store'); + }), + removeItem: vi.fn(), + }, + }; + const states: unknown[] = []; + const download = vi.fn(() => ({ status: 'downloaded' })); + const options = { + target, + now: () => TRACE_NOW, + onChange: (state: unknown): void => { + states.push(state); + }, + download, + }; + return { order, snapshot, collector, target, states, download, options }; +} +function handoff(options: ReturnType['options']) { + const result = createTraceHandoff(options); + if (typeof result !== 'object' || result === null) throw new Error('should create handoff'); + return result; +} +describe('explicit same-tab trace handoff', () => { + it.each([undefined, false, 'true', 1])( + 'creates no action or storage access for nonliteral gate %s', + (active) => { + const fixture = setup(); + fixture.target.__tsjs_trace_active = active; + expect(createTraceHandoff(fixture.options)).toBeUndefined(); + expect(fixture.snapshot).not.toHaveBeenCalled(); + expect(fixture.target.sessionStorage.getItem).not.toHaveBeenCalled(); + expect(fixture.target.sessionStorage.setItem).not.toHaveBeenCalled(); + } + ); + it('captures only on tap then stores a validated wrapper before same-tab navigation', () => { + const fixture = setup(); + const action = handoff(fixture.options); + expect(fixture.order).toEqual([]); + expect(fixture.target.sessionStorage.getItem).not.toHaveBeenCalled(); + action.view(); + expect(fixture.order).toEqual(['snapshot', 'store', 'navigate:/_ts/trace']); + const wrapper = JSON.parse(fixture.target.sessionStorage.setItem.mock.calls[0][1]); + expect(wrapper.report.request_context).toEqual(fixture.target.__tsjs_trace_request_context); + expect(wrapper.report.gpt_diagnostics.page.pathname).toBe('/[redacted]'); + expect(wrapper.stored_at_ms).toBe(TRACE_NOW); + }); + it('preserves a valid immutable combined report for explicit direct download after storage failure', () => { + const fixture = setup(); + fixture.target.sessionStorage.setItem.mockImplementation(() => { + throw new Error('private-quota-error'); + }); + const action = handoff(fixture.options); + action.view(); + expect(fixture.target.location.assign).not.toHaveBeenCalled(); + expect(fixture.download).not.toHaveBeenCalled(); + expect(fixture.states.at(-1)).toMatchObject({ + kind: 'storage_unavailable', + downloadAvailable: true, + }); + fixture.target.__tsjs_trace_request_context.network = { + private_value: 'changed-after-capture', + } as object; + action.download(); + const report = fixture.download.mock.calls[0][0]; + expect(report).toMatchObject({ request_context: reportFixture().request_context }); + expect(Object.isFrozen(report)).toBe(true); + expect(fixture.download.mock.calls[0].slice(1)).toEqual([TRACE_ORIGIN, TRACE_NOW]); + }); + it.each([ + 'missing-context', + 'invalid-context', + 'bad-gpt-version', + 'gpt-throws', + 'collector-overflow', + 'clock-invalid', + ])('never writes, navigates or offers a combined artifact after %s', (failure) => { + const fixture = setup(); + if (failure === 'missing-context') + Object.assign(fixture.target, { + __tsjs_trace_request_context: undefined, + }); + if (failure === 'invalid-context') + Object.assign(fixture.target, { + __tsjs_trace_request_context: { private_value: 'secret' }, + }); + if (failure === 'bad-gpt-version') + fixture.snapshot.mockReturnValue({ + ...gptSourceFixture(), + version: 2, + } as never); + if (failure === 'gpt-throws') + fixture.snapshot.mockImplementation(() => { + throw new Error('private-source-error'); + }); + if (failure === 'collector-overflow') + Object.assign(fixture.target.tsjs, { + traceEvidence: { + snapshot: () => ({ + ok: false, + reason: 'omission_counter_overflow', + }), + }, + }); + if (failure === 'clock-invalid') fixture.options.now = () => Number.NaN; + const action = handoff(fixture.options); + action.view(); + action.download(); + expect(fixture.target.sessionStorage.setItem).not.toHaveBeenCalled(); + expect(fixture.target.location.assign).not.toHaveBeenCalled(); + expect(fixture.download).not.toHaveBeenCalled(); + expect(JSON.stringify(fixture.states)).not.toContain('private'); + expect(fixture.states.at(-1)).toMatchObject({ + kind: 'capture_failed', + downloadAvailable: false, + }); + }); + it('remains on the publisher page if its navigation attempt fails after storage succeeds', () => { + const fixture = setup(); + fixture.target.location.assign.mockImplementation(() => { + throw new Error('private-navigation-error'); + }); + const action = handoff(fixture.options); + action.view(); + expect(fixture.target.sessionStorage.setItem).toHaveBeenCalledTimes(1); + expect(fixture.states.at(-1)).toMatchObject({ + kind: 'navigation_unavailable', + downloadAvailable: true, + }); + }); + it.each(['capturing', 'snapshot', 'storage'])( + 'stops capture when destroyed from %s callback', + (phase) => { + const fixture = setup(); + if (phase === 'capturing') fixture.options.onChange = () => action.destroy(); + if (phase === 'snapshot') + fixture.snapshot.mockImplementation(() => { + action.destroy(); + return gptSourceFixture(); + }); + if (phase === 'storage') + fixture.target.sessionStorage.setItem.mockImplementation(() => action.destroy()); + const action = handoff(fixture.options); + action.view(); + action.download(); + expect(fixture.target.location.assign).not.toHaveBeenCalled(); + expect(fixture.download).not.toHaveBeenCalled(); + if (phase === 'capturing') expect(fixture.snapshot).not.toHaveBeenCalled(); + if (phase !== 'storage') expect(fixture.target.sessionStorage.setItem).not.toHaveBeenCalled(); + } + ); + it('destroys retained recovery data and ignores later retry callbacks', () => { + const fixture = setup(); + fixture.target.sessionStorage.setItem.mockImplementation(() => { + throw new Error('private-quota-error'); + }); + const action = handoff(fixture.options); + action.view(); + action.destroy(); + action.download(); + action.view(); + expect(fixture.snapshot).toHaveBeenCalledTimes(1); + expect(fixture.download).not.toHaveBeenCalled(); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/lifecycle.test.ts b/crates/trusted-server-js/lib/test/trace/lifecycle.test.ts new file mode 100644 index 000000000..d7a4e7327 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/lifecycle.test.ts @@ -0,0 +1,153 @@ +import { beforeEach, afterEach, describe, expect, it, vi } from 'vitest'; + +import { changeTraceSession, endTraceSessionAndObserve } from '../../src/trace/lifecycle'; + +describe('deliberate trace session changes', () => { + const request = vi.fn(); + + beforeEach(() => { + request.mockReset(); + vi.stubGlobal('fetch', request); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + }); + + it.each(['enable', 'end'] as const)( + 'verifies %s through a separate state GET', + async (action) => { + const active = action === 'enable'; + request.mockResolvedValueOnce(new Response('{}', { status: 200 })); + request.mockResolvedValueOnce(new Response(JSON.stringify({ observed_active: active }))); + const historyLength = window.history.length; + const result = await changeTraceSession(action); + + expect(request.mock.calls).toEqual([ + [ + `/_ts/trace/${action}`, + { + method: 'POST', + credentials: 'same-origin', + cache: 'no-store', + headers: { 'X-TS-Trace-Action': action }, + }, + ], + ['/_ts/trace/state', { method: 'GET', credentials: 'same-origin', cache: 'no-store' }], + ]); + expect(result).toEqual({ + mutation: 'requested', + observation: active ? 'active' : 'inactive', + confirmed: true, + }); + expect(window.history.length).toBe(historyLength); + } + ); + + it('does not confirm activation when the browser fails to return a valid session', async () => { + request.mockResolvedValueOnce(new Response('{}')); + request.mockResolvedValueOnce(new Response('{"observed_active":false}')); + expect(await changeTraceSession('enable')).toEqual({ + mutation: 'requested', + observation: 'inactive', + confirmed: false, + }); + }); + + it('does not confirm deactivation when a valid session is still observed', async () => { + request.mockResolvedValueOnce(new Response('{}')); + request.mockResolvedValueOnce(new Response('{"observed_active":true}')); + expect(await changeTraceSession('end')).toEqual({ + mutation: 'requested', + observation: 'active', + confirmed: false, + }); + }); + + it.each([ + '{}', + '{"observed_active":"false"}', + '{"observed_active":false,"extra":true}', + '[]', + 'null', + '{invalid-json', + ])( + 'rejects an invalid state response without discarding the mutation result: %s', + async (body) => { + request.mockResolvedValueOnce(new Response('{}')); + request.mockResolvedValueOnce(new Response(body)); + expect(await changeTraceSession('enable')).toEqual({ + mutation: 'requested', + observation: 'failed', + confirmed: false, + }); + } + ); + + it('retains requested mutation when the verification request fails', async () => { + request.mockResolvedValueOnce(new Response('{}')); + request.mockRejectedValueOnce(new TypeError('offline')); + expect(await changeTraceSession('end')).toEqual({ + mutation: 'requested', + observation: 'failed', + confirmed: false, + }); + }); + + it('rejects a non-successful state response', async () => { + request.mockResolvedValueOnce(new Response('{}')); + request.mockResolvedValueOnce(new Response('{"observed_active":false}', { status: 500 })); + expect((await changeTraceSession('end')).observation).toBe('failed'); + }); + + it.each([403, 404, 413, 500])( + 'keeps a rejected mutation (%s) separate from state observation', + async (status) => { + request.mockResolvedValueOnce(new Response('{}', { status })); + expect(await changeTraceSession('enable')).toEqual({ + mutation: 'failed', + observation: 'not_attempted', + confirmed: false, + }); + expect(request).toHaveBeenCalledTimes(1); + } + ); + + it('permits an explicit retry after a failed mutation', async () => { + request.mockRejectedValueOnce(new TypeError('offline')); + expect((await changeTraceSession('enable')).mutation).toBe('failed'); + request.mockResolvedValueOnce(new Response('{}')); + request.mockResolvedValueOnce(new Response('{"observed_active":true}')); + expect((await changeTraceSession('enable')).confirmed).toBe(true); + expect(request).toHaveBeenCalledTimes(3); + }); + + it.each(['rejected', 'offline'] as const)( + 'observes state independently after an end POST is %s', + async (failure) => { + if (failure === 'offline') request.mockRejectedValueOnce(new Error('private-network-error')); + else request.mockResolvedValueOnce(new Response('{}', { status: 500 })); + request.mockResolvedValueOnce(new Response('{"observed_active":false}')); + expect(await endTraceSessionAndObserve()).toEqual({ + mutation: 'failed', + observation: 'inactive', + confirmed: false, + }); + expect(request.mock.calls.map(([path]) => path)).toEqual([ + '/_ts/trace/end', + '/_ts/trace/state', + ]); + } + ); + + it('keeps both failed end mutation and failed independent observation bounded', async () => { + request.mockRejectedValueOnce(new Error('private-post-error')); + request.mockRejectedValueOnce(new Error('private-state-error')); + expect(await endTraceSessionAndObserve()).toEqual({ + mutation: 'failed', + observation: 'failed', + confirmed: false, + }); + expect(request).toHaveBeenCalledTimes(2); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/pending.test.ts b/crates/trusted-server-js/lib/test/trace/pending.test.ts new file mode 100644 index 000000000..3fcf506a8 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/pending.test.ts @@ -0,0 +1,257 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { createTraceCollector } from '../../src/trace/collector'; +import { createTracePending, pendingExpiry } from '../../src/trace/pending'; + +import { gptTransport } from './gpt-fixtures'; +import { SLOT_TOKEN } from './fixtures'; + +function carry() { + return { collector: createTraceCollector(), slotRefs: [SLOT_TOKEN] }; +} +function response() { + return { + ext: { + trusted_server: { + trace_auction: { + ...gptTransport(), + evidence: { ...gptTransport().evidence!, source: 'auction_api' }, + }, + }, + }, + }; +} +function bindings(bidId = 'example-bid', bidderRequestId = 'example-request') { + return [{ bidId, bidderRequestId, slotRef: SLOT_TOKEN }]; +} +function identities(bidIds: readonly string[], bidderRequestId = 'example-request') { + return bidIds.map((bidId) => ({ bidId, bidderRequestId })); +} +afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); +}); +describe('bounded Prebid trace pending records', () => { + it.each([0, 2000, 2 ** 31 - 1 - 5000])('captures timeout %s once', (timeout) => { + expect(pendingExpiry(100, timeout)).toBe(100 + timeout + 5000); + }); + it.each([undefined, -1, 1.5, NaN, Infinity, '2000', 2 ** 31 - 5000])( + 'defaults invalid timeout %s to 3000', + (timeout) => { + expect(pendingExpiry(100, timeout)).toBe(8100); + } + ); + it.each([-1, 0.5, NaN, Infinity, Number.MAX_SAFE_INTEGER])('declines unsafe clock %s', (now) => { + expect(pendingExpiry(now, 3000)).toBeUndefined(); + }); + it.each([0, 2000, 2 ** 31 - 1 - 5000])( + 'arms one bounded browser timer for configured timeout %s', + (timeout) => { + vi.useFakeTimers(); + vi.setSystemTime(0); + const schedule = vi.fn((callback: () => void, delay: number) => setTimeout(callback, delay)); + const pending = createTracePending({ timeout: () => timeout, schedule }); + const value = carry(); + const handle = pending.add(bindings(), value); + expect(schedule.mock.calls[0]![1]).toBe(timeout + 5000); + vi.advanceTimersByTime(timeout + 4999); + pending.response(handle, response()); + expect(value.collector.captureStatus()).toBe('complete'); + expect(vi.getTimerCount()).toBe(0); + } + ); + it('records readable API evidence only once and never emits GPT sidecars', () => { + const pending = createTracePending({ now: () => 0 }); + const value = carry(); + const handle = pending.add(bindings(), value); + pending.response(handle, response()); + pending.response(handle, response()); + pending.failure(identities(['example-bid'])); + expect(value.collector.snapshot().value?.serverAuctions).toHaveLength(1); + expect(value.collector.snapshot().value?.slotCorrelations).toEqual([]); + expect(value.collector.captureStatus()).toBe('complete'); + pending.destroy(); + }); + it('consumes matching original IDs before callback and isolates concurrent requests', () => { + const pending = createTracePending({ now: () => 0 }); + const a = carry(); + const b = carry(); + pending.add(bindings('first'), a); + const second = pending.add(bindings('second'), b); + pending.failure(identities(['first'])); + pending.failure(identities(['first'])); + pending.response(second, response()); + expect(a.collector.captureStatus()).toBe('unavailable'); + expect(b.collector.captureStatus()).toBe('complete'); + pending.destroy(); + }); + it('retires expiry without inventing a transport outcome and captures timeout only at creation', () => { + vi.useFakeTimers(); + let timeout = 0; + const readTimeout = vi.fn(() => timeout); + const pending = createTracePending({ now: () => Date.now(), timeout: readTimeout }); + const value = carry(); + const handle = pending.add(bindings(), value); + timeout = 100000; + vi.advanceTimersByTime(5000); + pending.failure(identities(['example-bid'])); + pending.response(handle, response()); + expect(readTimeout).toHaveBeenCalledTimes(1); + expect(value.collector.captureStatus()).toBe('not_observed'); + expect(vi.getTimerCount()).toBe(0); + }); + it('checks expiry synchronously when a delayed callback beats its timer', () => { + let now = 0; + const value = carry(); + const pending = createTracePending({ now: () => now, timeout: () => 0 }); + const handle = pending.add(bindings(), value); + now = 5000; + pending.response(handle, response()); + expect(value.collector.captureStatus()).toBe('not_observed'); + pending.destroy(); + }); + it('evicts the oldest of 128 records without failures and destroys all timers', () => { + vi.useFakeTimers(); + const pending = createTracePending({ now: () => Date.now() }); + const values = Array.from({ length: 129 }, carry); + const handles = values.map((value, index) => pending.add(bindings(`bid-${index}`), value)); + expect(vi.getTimerCount()).toBe(128); + pending.response(handles[0], response()); + expect(values[0].collector.captureStatus()).toBe('not_observed'); + pending.destroy(); + expect(vi.getTimerCount()).toBe(0); + pending.failure(identities(['bid-1'])); + expect(values[1].collector.captureStatus()).toBe('not_observed'); + expect(pending.add(bindings(), carry())).toBeUndefined(); + }); + it('owns bindings/refs and removes state before a throwing/reentrant diagnostic callback', () => { + const pending = createTracePending({ now: () => 0 }); + const collector = createTraceCollector(); + const failure = vi.fn(() => { + pending.failure(identities(['original'])); + throw new Error('private-error'); + }); + const value = { + collector: { ...collector, recordTransportFailure: failure }, + slotRefs: [SLOT_TOKEN], + }; + const list = bindings('original'); + pending.add(list, value); + list[0].bidId = 'mutated'; + value.slotRefs[0] = 'mutated'; + expect(() => pending.failure(identities(['original']))).not.toThrow(); + expect(failure).toHaveBeenCalledTimes(1); + pending.destroy(); + }); + it('declines clock overflow and ambiguous IDs without disturbing previous capture', () => { + let now = 0; + const pending = createTracePending({ now: () => now }); + const value = carry(); + value.collector.recordTransport(response().ext.trusted_server.trace_auction); + pending.add(bindings(), value); + expect(pending.add(bindings(), carry())).toBeDefined(); + now = Number.MAX_SAFE_INTEGER; + expect(pending.add(bindings('new'), value)).toBeUndefined(); + now = 0; + pending.failure(identities(['example-bid'])); + expect(value.collector.captureStatus()).toBe('complete'); + pending.destroy(); + }); + it('keeps exact response handles for every collided request while declining ambiguous hooks', () => { + vi.useFakeTimers(); + const pending = createTracePending({ now: () => Date.now() }); + const a = carry(); + const b = carry(); + const c = carry(); + const first = pending.add(bindings('first'), a); + const second = pending.add(bindings('second'), b); + const collision = pending.add([...bindings('first'), ...bindings('second')], c); + expect(collision).toBeDefined(); + expect(vi.getTimerCount()).toBe(3); + pending.failure(identities(['first', 'second'])); + expect([a, b, c].map((value) => value.collector.captureStatus())).toEqual([ + 'not_observed', + 'not_observed', + 'not_observed', + ]); + pending.response(first, response()); + pending.response(second, response()); + pending.response(collision, response()); + expect([a, b, c].map((value) => value.collector.captureStatus())).toEqual([ + 'complete', + 'complete', + 'complete', + ]); + expect(vi.getTimerCount()).toBe(0); + pending.destroy(); + }); + it('keeps over-cap responses and disables current and later hooks that could collide with unseen IDs', () => { + vi.useFakeTimers(); + const pending = createTracePending({ now: () => Date.now() }); + const a = carry(); + const b = carry(); + const c = carry(); + const first = pending.add(bindings('bid-2048'), a); + const overflow = pending.add( + Array.from({ length: 2049 }, (_, index) => bindings(`bid-${index}`)[0]!), + b + ); + const later = pending.add(bindings('bid-2047'), c); + expect(overflow).toBeDefined(); + expect(later).toBeDefined(); + pending.failure(identities(['bid-2048', 'bid-2047'])); + expect([a, b, c].map((value) => value.collector.captureStatus())).toEqual([ + 'not_observed', + 'not_observed', + 'not_observed', + ]); + pending.response(first, response()); + pending.response(overflow, response()); + pending.response(later, response()); + expect([a, b, c].map((value) => value.collector.captureStatus())).toEqual([ + 'complete', + 'complete', + 'complete', + ]); + expect(vi.getTimerCount()).toBe(0); + }); + it.each(['response', 'failure'])( + 'isolates a reused original ID after old %s retirement', + (retire) => { + vi.useFakeTimers(); + const pending = createTracePending({ now: () => Date.now() }); + const old = carry(); + const next = carry(); + const first = pending.add(bindings('reused', 'request-old'), old); + if (retire === 'response') pending.response(first, response()); + else pending.failure(identities(['reused'], 'request-old')); + const second = pending.add(bindings('reused', 'request-new'), next); + pending.failure(identities(['reused'], 'request-old')); + expect(next.collector.captureStatus()).toBe('not_observed'); + expect(vi.getTimerCount()).toBe(1); + pending.response(second, response()); + expect(next.collector.captureStatus()).toBe('complete'); + expect(vi.getTimerCount()).toBe(0); + } + ); + it.each([undefined, '', 1, 'x'.repeat(129)])( + 'keeps malformed request ID %s response-only', + (bidderRequestId) => { + vi.useFakeTimers(); + const pending = createTracePending({ now: () => Date.now() }); + const value = carry(); + const handle = pending.add( + [{ bidId: 'reused', bidderRequestId } as ReturnType[number]], + value + ); + pending.failure([ + { bidId: 'reused', bidderRequestId } as ReturnType[number], + ]); + expect(value.collector.captureStatus()).toBe('not_observed'); + expect(value.collector.snapshot().value?.issues).toEqual(['correlation_unavailable']); + pending.response(handle, response()); + expect(value.collector.captureStatus()).toBe('complete'); + expect(vi.getTimerCount()).toBe(0); + } + ); +}); diff --git a/crates/trusted-server-js/lib/test/trace/projection.test.ts b/crates/trusted-server-js/lib/test/trace/projection.test.ts new file mode 100644 index 000000000..9e388052f --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/projection.test.ts @@ -0,0 +1,331 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { GptDiagnosticsStore } from '../../src/integrations/gpt_diagnostics/store'; +import { addOmissions, projectTraceGptDiagnostics } from '../../src/trace/projection'; + +import { gptSourceFixture, projectedGptFixture, TRACE_ORIGIN } from './fixtures'; +import { observedGptSource } from './gpt-fixtures'; + +describe('explicit GPT public projection', () => { + const optionalPaths = [ + ['attributionIssues'], + ['metadata', 'droppedAttributionIssues'], + ['slots', '0', 'binding', 'reason'], + ['slots', '0', 'currentVisibilityPercentage'], + ['slots', '0', 'requests', '0', 'requestedAtMs'], + ['slots', '0', 'requests', '0', 'durations', 'requestToResponseMs'], + ['slots', '0', 'requests', '0', 'size'], + ['slots', '0', 'requests', '0', 'observedSlotSize'], + ['slots', '0', 'requests', '0', 'requestedSlotSizes'], + ['slots', '0', 'requests', '0', 'trustedServerCreativeFailures'], + ['slots', '0', 'requests', '0', 'trustedServerAuctionId'], + ['attributionIssues', '0', 'runtimeSlotNumber'], + ]; + function parentAt(source: unknown, path: string[]): Record { + let parent = source as Record; + for (const key of path.slice(0, -1)) parent = parent[key] as Record; + return parent; + } + it.each(optionalPaths)( + 'omits optional own undefined like an absent JSON member: %j', + (...path) => { + const absent = gptSourceFixture(); + const present = gptSourceFixture(); + const key = path[path.length - 1]!; + delete parentAt(absent, path)[key]; + parentAt(present, path)[key] = undefined; + const expected = projectTraceGptDiagnostics(absent, TRACE_ORIGIN); + expect(expected.ok).toBe(true); + expect(projectTraceGptDiagnostics(present, TRACE_ORIGIN)).toEqual(expected); + } + ); + it.each(['required', 'unknown', 'null', 'accessor', 'malformed'] as const)( + 'keeps %s public-source members strict while omitting optional undefined', + (kind) => { + const source = gptSourceFixture(); + const cycle = source.slots[0]!.requests[0]! as unknown as Record; + const getter = vi.fn(() => undefined); + if (kind === 'required') cycle.requestNumber = undefined; + if (kind === 'unknown') cycle.privateFutureField = undefined; + if (kind === 'null') cycle.size = null; + if (kind === 'accessor') Object.defineProperty(cycle, 'size', { get: getter }); + if (kind === 'malformed') cycle.size = ['300', 250]; + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + expect(getter).not.toHaveBeenCalled(); + } + ); + it.each(['no_candidate', 'renderable_candidate'] as const)( + 'projects actual pending %s API snapshots with optional undefined values', + (opportunity) => { + const store = new GptDiagnosticsStore({ now: () => 1 }); + const slot = { getSlotElementId: () => 'example' }; + store.recordTrustedServerOpportunity(slot, 'example', opportunity); + store.recordSlotRequested(slot); + const source = observedGptSource(store); + expect( + Object.getOwnPropertyDescriptor(source.slots[0]!.requests[0]!, 'size')?.value + ).toBeUndefined(); + const projected = projectTraceGptDiagnostics(source, TRACE_ORIGIN); + expect(projected.ok).toBe(true); + if (!projected.ok) throw new Error('should project actual pending cycles'); + expect(projected.value.slots[0]!.requests[0]!.trustedServerOpportunity).toBe(opportunity); + expect(Object.hasOwn(projected.value.slots[0]!.requests[0]!, 'size')).toBe(false); + } + ); + it('classifies inspection failures without reading caller-controlled errors', () => { + let reads = 0; + const failure = Object.defineProperty(new Error('private-message'), 'message', { + get() { + reads += 1; + throw new Error('should not inspect'); + }, + }); + const source = new Proxy(gptSourceFixture(), { + ownKeys() { + throw failure; + }, + }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + expect(reads).toBe(0); + const proxyError = new Proxy(new Error('private-message'), { + getPrototypeOf() { + throw new Error('should not inspect'); + }, + }); + expect( + projectTraceGptDiagnostics( + new Proxy(gptSourceFixture(), { + ownKeys() { + throw proxyError; + }, + }), + TRACE_ORIGIN + ) + ).toEqual({ ok: false, reason: 'invalid_source' }); + }); + it('copies every public source member and excludes every private sentinel', () => { + const source = gptSourceFixture(); + const result = projectTraceGptDiagnostics(source, TRACE_ORIGIN); + expect(result.ok).toBe(true); + if (!result.ok) throw new Error('should project source'); + expect(result.value).toEqual(projectedGptFixture()); + expect(result.omittedNestedValues).toBe(0); + const json = JSON.stringify(result.value); + for (const sentinel of [ + 'private-page-secret', + 'private-slot-element', + 'private-ad-unit', + 'private-callback-element', + 'private-attribution-element', + '900001', + '900002', + '900003', + '900004', + '900005', + '900006', + '900007', + '900008', + '900009', + 'adManager', + 'previousCreativeId', + 'slotElementId', + 'adUnitPath', + ]) + expect(json).not.toContain(sentinel); + source.slots[0].requests[0].durations.requestToResponseMs = 999; + source.slots[0].requests[0].requestedSlotSizes = [[1, 1]]; + expect(result.value.slots[0].requests[0].durations.requestToResponseMs).toBe(0.5); + expect(result.value.slots[0].requests[0].requestedSlotSizes).toEqual([ + [300, 250], + [320, 50], + ]); + const frozen = (value: unknown): void => { + if (typeof value !== 'object' || value === null) return; + expect(Object.isFrozen(value)).toBe(true); + Object.values(value).forEach(frozen); + }; + frozen(result.value); + expect(Object.isFrozen(source.slots[0].requests[0])).toBe(false); + }); + it('does not inspect excluded contents and rejects source accessors without invoking them', () => { + const source = gptSourceFixture(); + let calls = 0; + Object.assign(source.slots[0].requests[0], { + adManager: new Proxy( + {}, + { + get() { + calls += 1; + throw new Error('should not call'); + }, + getPrototypeOf() { + calls += 1; + throw new Error('should not call'); + }, + } + ), + previousCreativeId: { private: 'ignored' }, + }); + Object.assign(source.slots[0], { slotElementId: '\ud800', adUnitPath: 42 }); + source.page.pathname = '\u0000' + 'x'.repeat(1024); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN).ok).toBe(true); + expect(calls).toBe(0); + Object.defineProperty(source.slots[0].requests[0], 'adManager', { + enumerable: true, + get() { + calls += 1; + return {}; + }, + }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + expect(calls).toBe(0); + }); + it('rejects every extra source key at each source boundary', () => { + const source = gptSourceFixture(); + const targets = [ + source, + source.page, + source.slots[0], + source.slots[0].binding, + source.slots[0].requests[0], + source.slots[0].requests[0].durations, + source.callbackIssues[0], + source.attributionIssues![0], + source.coverage, + source.coverage.slotRequested, + source.metadata, + ]; + for (const target of targets) { + Object.assign(target, { future: 'private-value' }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + Reflect.deleteProperty(target, 'future'); + } + }); + it('rejects invalid retained values and incompatible source versions without repairs', () => { + const source = gptSourceFixture(); + Object.assign(source, { version: 2 }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'unsupported_source_version', + }); + Object.assign(source, { version: 1 }); + Object.assign(source.slots[0].requests[0], { requestedAtMs: '1' }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + source.slots[0].requests[0].requestedAtMs = 0.5; + source.page.origin = 'https://attacker.example.com'; + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN).ok).toBe(false); + }); + it('preserves source optionality and canonicalizes origin during literal path redaction', () => { + const source = gptSourceFixture(); + delete source.attributionIssues; + delete source.metadata.droppedAttributionIssues; + source.page.origin = 'https://PUBLISHER.EXAMPLE.COM:443'; + const result = projectTraceGptDiagnostics(source, TRACE_ORIGIN); + expect(result.ok).toBe(true); + if (!result.ok) throw new Error('should project source'); + expect(result.value.page).toEqual({ origin: TRACE_ORIGIN, pathname: '/[redacted]' }); + expect(result.value).not.toHaveProperty('attributionIssues'); + expect(result.value.metadata).not.toHaveProperty('droppedAttributionIssues'); + }); + it('retains first sixteen nested values, validates omitted values and counts each omission once', () => { + const source = gptSourceFixture(); + source.slots[0].requests[0].requestedSlotSizes = Array.from({ length: 17 }, (_, index) => [ + index + 1, + 1, + ]); + source.slots[0].requests[0].trustedServerCreativeFailures = Array.from( + { length: 17 }, + () => 'missing_render_source' + ); + const result = projectTraceGptDiagnostics(source, TRACE_ORIGIN); + expect(result.ok).toBe(true); + if (!result.ok) throw new Error('should project source'); + expect(result.omittedNestedValues).toBe(2); + expect(result.value.slots[0].requests[0].requestedSlotSizes).toHaveLength(16); + expect(result.value.slots[0].requests[0].requestedSlotSizes?.[15]).toEqual([16, 1]); + Object.assign(source.slots[0].requests[0], { + trustedServerCreativeFailures: [ + ...source.slots[0].requests[0].trustedServerCreativeFailures!, + 'private-error', + ], + }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + }); + it('rejects unsupported source cardinality and omission overflow', () => { + const source = gptSourceFixture(); + source.slots = Array.from({ length: 65 }, () => gptSourceFixture().slots[0]); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'invalid_source', + }); + const cycles = gptSourceFixture(); + cycles.slots[0].requests = Array.from( + { length: 11 }, + () => gptSourceFixture().slots[0].requests[0] + ); + expect(projectTraceGptDiagnostics(cycles, TRACE_ORIGIN).ok).toBe(false); + const oversized = gptSourceFixture(); + oversized.slots[0].requests[0].trustedServerCreativeFailures = Array.from( + { length: 65552 }, + () => 'missing_render_source' + ); + expect(projectTraceGptDiagnostics(oversized, TRACE_ORIGIN)).toEqual({ + ok: false, + reason: 'omission_counter_overflow', + }); + }); + it('never invokes source get traps, array methods or iterators', () => { + const source = gptSourceFixture(); + let calls = 0; + source.slots[0].requests[0] = new Proxy(source.slots[0].requests[0], { + get() { + calls += 1; + throw new Error('should not call'); + }, + }); + source.slots = new Proxy(source.slots, { + get() { + calls += 1; + throw new Error('should not call'); + }, + }); + expect(projectTraceGptDiagnostics(source, TRACE_ORIGIN).ok).toBe(true); + expect(calls).toBe(0); + }); +}); +describe('checked public omission counters', () => { + it('adds exact zero and u16-boundary counts', () => { + expect(addOmissions(0, 0)).toBe(0); + expect(addOmissions(65534, 1)).toBe(65535); + expect(addOmissions(65535, 0)).toBe(65535); + }); + it.each([ + [65535, 1], + [-1, 1], + [0, -1], + [0, 0.5], + [NaN, 0], + [0, Infinity], + [65536, 0], + ])('rejects invalid or overflowing %s + %s', (current, added) => { + expect(() => addOmissions(current, added)).toThrow('omission_counter_overflow'); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/report.test.ts b/crates/trusted-server-js/lib/test/trace/report.test.ts new file mode 100644 index 000000000..ff0760aa2 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/report.test.ts @@ -0,0 +1,332 @@ +import { describe, expect, it } from 'vitest'; + +import { buildTraceReport } from '../../src/trace/report'; + +import { + gptSourceFixture, + reportFixture, + TRACE_NOW, + TRACE_ORIGIN, + AUCTION_TOKEN, + SLOT_TOKEN, +} from './fixtures'; + +function auction(number = 1) { + return { + schema_version: 1, + diagnostic_auction_id: token(number), + source: 'initial_navigation_ssat', + terminal_status: 'completed', + provider_calls: [], + slots: [], + truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 }, + coverage: { provider_to_slot_no_bid: 'unavailable' }, + }; +} +function token(number: number) { + return `ts-auc-${number.toString(16).padStart(8, '0')}12344abc8def123456789abc`; +} +function sidecar(number = 1) { + return { + schema_version: 1, + diagnostic_auction_id: token(number), + slot_ref: SLOT_TOKEN, + runtime_slot_number: 1, + request_number: 1, + }; +} +function input() { + return { + requestContext: reportFixture().request_context, + gptSource: gptSourceFixture(), + origin: TRACE_ORIGIN, + capturedAtMs: TRACE_NOW, + collector: { + serverAuctions: [], + slotCorrelations: [], + issues: [], + omittedServerAuctions: 0, + omittedSlotCorrelations: 0, + }, + }; +} +function success(result: ReturnType) { + if ( + typeof result !== 'object' || + result === null || + !('ok' in result) || + result.ok !== true || + !('value' in result) + ) + throw new Error('should build report'); + return result.value; +} +describe('combined trace capture', () => { + it('rejects a caller capture-clock getter before reading it or invoking serialization', () => { + let reads = 0; + let serialized = 0; + const source = input(); + Object.defineProperty(source, 'capturedAtMs', { + enumerable: true, + get() { + reads += 1; + return reads === 4 + ? { + toJSON() { + serialized += 1; + return TRACE_NOW; + }, + } + : TRACE_NOW; + }, + }); + expect(buildTraceReport(source)).toEqual({ ok: false, reason: 'invalid_snapshot' }); + expect(reads).toBe(0); + expect(serialized).toBe(0); + }); + it('uses a frozen top-level own data observation without caller get traps', () => { + let reads = 0; + const source = new Proxy(input(), { + get() { + reads += 1; + throw new Error('private-get'); + }, + }); + const result = success(buildTraceReport(source)); + expect(result.stored_at_ms).toBe(TRACE_NOW); + expect(reads).toBe(0); + }); + it('projects an immutable report with independent capture time and original cookie observations', () => { + const source = input(); + const result = success(buildTraceReport(source)); + expect(result.stored_at_ms).toBe(TRACE_NOW); + expect(result.report.captured_at).toBe(new Date(TRACE_NOW).toISOString()); + expect(result.report.request_context).toEqual(source.requestContext); + expect(result.report.gpt_diagnostics.page.pathname).toBe('/[redacted]'); + expect(JSON.stringify(result)).not.toContain('private-'); + expect(Object.isFrozen(result.report.gpt_diagnostics.slots[0].requests[0])).toBe(true); + source.gptSource.slots[0].requests[0].requestNumber = 99; + expect(result.report.gpt_diagnostics.slots[0].requests[0].requestNumber).toBe(1); + }); + it('retains newest cardinality records and merges collector losses once', () => { + const source = input(); + Object.assign(source.collector, { + serverAuctions: Array.from({ length: 18 }, (_, i) => auction(i + 1)), + slotCorrelations: Array.from({ length: 130 }, (_, i) => sidecar(i + 1)), + omittedServerAuctions: 4, + omittedSlotCorrelations: 7, + }); + const result = success(buildTraceReport(source)).report; + expect(result.server_auctions.map((record) => record.diagnostic_auction_id)).toEqual( + Array.from({ length: 16 }, (_, i) => token(i + 3)) + ); + expect(result.slot_correlations).toHaveLength(128); + expect(result.truncation.omitted_server_auctions).toBe(6); + expect(result.truncation.omitted_slot_correlations).toBe(9); + expect(result.truncation.omitted_request_cycles).toBe(0); + expect(result.auction_coverage).toEqual({ + capture_status: 'partial', + issues: ['record_evicted', 'correlation_unavailable'], + }); + }); + it('rejects invalid inner evidence even if that record would be discarded', () => { + const source = input(); + Object.assign(source.collector, { + serverAuctions: [ + { ...auction(), private_value: 'secret' }, + ...Array.from({ length: 16 }, (_, i) => auction(i + 2)), + ], + }); + expect(buildTraceReport(source)).toEqual({ ok: false, reason: 'invalid_snapshot' }); + }); + it('rejects checked omission overflow', () => { + const source = input(); + Object.assign(source.collector, { + serverAuctions: Array.from({ length: 17 }, (_, i) => auction(i + 1)), + omittedServerAuctions: 65535, + }); + expect(buildTraceReport(source)).toEqual({ ok: false, reason: 'omission_counter_overflow' }); + }); + it('removes oldest non-floor cycles and their sidecars while preserving output order', () => { + const source = input(); + const original = source.gptSource.slots[0].requests[0]; + source.gptSource.slots[0].requests = [ + { ...original, requestNumber: 1, requestedAtMs: 20 }, + { ...original, requestNumber: 2, requestedAtMs: undefined }, + { ...original, requestNumber: 3, requestedAtMs: 1 }, + ]; + delete source.gptSource.slots[0].requests[1].requestedAtMs; + Object.assign(source.collector, { + slotCorrelations: [{ ...sidecar(), diagnostic_auction_id: AUCTION_TOKEN, request_number: 2 }], + }); + const full = success(buildTraceReport(source)); + const budget = new TextEncoder().encode(JSON.stringify(full)).length - 1; + const result = success(buildTraceReport(source, budget)).report; + expect(result.gpt_diagnostics.slots[0].requests.map((cycle) => cycle.requestNumber)).toEqual([ + 1, 3, + ]); + expect(result.slot_correlations).toEqual([]); + expect(result.truncation.omitted_request_cycles).toBe(1); + expect(result.truncation.omitted_slot_correlations).toBe(1); + expect(result.auction_coverage.capture_status).toBe('not_observed'); + }); + it('never removes a protected floor or substitutes an empty GPT snapshot', () => { + expect(buildTraceReport(input(), 100)).toEqual({ ok: false, reason: 'snapshot_too_large' }); + expect(buildTraceReport(input(), 512 * 1024 + 1)).toEqual({ + ok: false, + reason: 'invalid_snapshot', + }); + }); + it('removes an uncorrelated auction and recomputes coverage when no evidence remains', () => { + const source = input(); + source.gptSource.callbackIssues = []; + source.gptSource.attributionIssues = []; + Object.assign(source.collector, { serverAuctions: [auction()], slotCorrelations: [sidecar()] }); + const full = success(buildTraceReport(source)); + const result = success( + buildTraceReport(source, new TextEncoder().encode(JSON.stringify(full)).length - 1) + ).report; + expect(result.server_auctions).toEqual([]); + expect(result.slot_correlations).toEqual([]); + expect(result.auction_coverage).toEqual({ + capture_status: 'unavailable', + issues: ['record_evicted', 'correlation_unavailable'], + }); + }); + it('protects any floor auction token even without a valid joining sidecar', () => { + const source = input(); + source.gptSource.callbackIssues = []; + source.gptSource.attributionIssues = []; + Object.assign(source.collector, { + serverAuctions: [{ ...auction(), diagnostic_auction_id: AUCTION_TOKEN }], + }); + const full = success(buildTraceReport(source)); + expect( + buildTraceReport(source, new TextEncoder().encode(JSON.stringify(full)).length - 1) + ).toEqual({ ok: false, reason: 'snapshot_too_large' }); + }); + it('rejects a fractional capture clock and an API sidecar', () => { + const source = input(); + expect(buildTraceReport({ ...source, capturedAtMs: TRACE_NOW + 0.5 })).toEqual({ + ok: false, + reason: 'invalid_snapshot', + }); + Object.assign(source.collector, { + serverAuctions: [{ ...auction(), source: 'auction_api' }], + slotCorrelations: [sidecar()], + }); + expect(buildTraceReport(source)).toEqual({ ok: false, reason: 'invalid_snapshot' }); + }); + it('counts sidecars pruned when their known auction is discarded at initial cardinality', () => { + const source = input(); + Object.assign(source.collector, { + serverAuctions: Array.from({ length: 17 }, (_, i) => auction(i + 1)), + slotCorrelations: [sidecar(1)], + }); + const result = success(buildTraceReport(source)).report; + expect(result.slot_correlations).toEqual([]); + expect(result.truncation.omitted_slot_correlations).toBe(1); + }); + it('rejects a capture clock outside the four-digit UTC range', () => { + expect(buildTraceReport({ ...input(), capturedAtMs: Date.UTC(10000, 0, 1) })).toEqual({ + ok: false, + reason: 'invalid_snapshot', + }); + }); + it('removes oldest callback issues before attribution issues without reordering survivors', () => { + const source = input(); + const issue = source.gptSource.callbackIssues[0]; + source.gptSource.callbackIssues = [ + { ...issue, timestampMs: 20 }, + { ...issue, timestampMs: 1 }, + { ...issue, timestampMs: 10 }, + ]; + const full = success(buildTraceReport(source)); + const result = success( + buildTraceReport(source, new TextEncoder().encode(JSON.stringify(full)).length - 1) + ).report; + expect(result.gpt_diagnostics.callbackIssues.map((value) => value.timestampMs)).toEqual([ + 20, 10, + ]); + expect(result.gpt_diagnostics.attributionIssues).toHaveLength(1); + expect(result.truncation.omitted_callback_issues).toBe(1); + expect(result.truncation.omitted_attribution_issues).toBe(0); + }); + it('bounds a fully populated real-size fixture to 512 KiB while retaining every slot floor', () => { + const source = input(); + source.requestContext.network = { region: 'é'.repeat(16), edge_region: '界'.repeat(42) }; + const slot = source.gptSource.slots[0]; + source.gptSource.slots = Array.from({ length: 64 }, (_, i) => ({ + ...structuredClone(slot), + runtimeSlotNumber: i + 1, + requests: Array.from({ length: 10 }, (_, j) => ({ + ...structuredClone(slot.requests[0]), + requestNumber: j + 1, + requestedAtMs: j + 1, + requestedSlotSizes: Array.from({ length: 16 }, () => [100000, 100000] as [number, number]), + })), + })); + source.gptSource.callbackIssues = Array.from({ length: 128 }, () => + structuredClone(source.gptSource.callbackIssues[0]) + ); + source.gptSource.attributionIssues = Array.from({ length: 128 }, () => + structuredClone(source.gptSource.attributionIssues![0]) + ); + Object.assign(source.collector, { + serverAuctions: Array.from({ length: 16 }, (_, i) => ({ + ...auction(i + 1), + provider_calls: Array.from({ length: 16 }, (_, j) => ({ + provider_number: j + 1, + role: 'bidder', + status: 'success', + returned_bid_count: 65535, + response_time_ms: 4294967295, + })), + slots: Array.from({ length: 64 }, (_, j) => ({ + slot_number: j + 1, + slot_ref: SLOT_TOKEN, + requested_sizes: Array.from({ length: 16 }, () => [100000, 100000]), + returned_bid_count: 65535, + candidate: 'selected', + selected_creative_size: [100000, 100000], + })), + })), + slotCorrelations: Array.from({ length: 128 }, (_, i) => ({ + ...sidecar(i + 1), + request_number: 10, + runtime_slot_number: (i % 64) + 1, + })), + }); + const result = success(buildTraceReport(source)); + expect(new TextEncoder().encode(JSON.stringify(result)).length).toBeLessThanOrEqual(512 * 1024); + expect(result.report.gpt_diagnostics.slots).toHaveLength(64); + expect( + result.report.gpt_diagnostics.slots.every( + (value) => value.requests.at(-1)?.requestNumber === 10 + ) + ).toBe(true); + expect(result.report.truncation.omitted_request_cycles).toBeGreaterThan(0); + expect(result.report.truncation).toEqual({ + omitted_server_auctions: 3, + omitted_slot_correlations: 3, + omitted_request_cycles: 64 * 9, + omitted_callback_issues: 128, + omitted_attribution_issues: 128, + omitted_nested_values: 0, + }); + expect(result.report.server_auctions.map((record) => record.diagnostic_auction_id)).toEqual( + Array.from({ length: 13 }, (_, i) => token(i + 4)) + ); + expect(result.report.slot_correlations.map((sidecar) => sidecar.diagnostic_auction_id)).toEqual( + Array.from({ length: 125 }, (_, i) => token(i + 4)) + ); + expect(new TextEncoder().encode(JSON.stringify(result)).length).toBeGreaterThan( + JSON.stringify(result).length + ); + expect(result.report.request_context.cookies.diagnostics_session).toEqual({ + source: 'request', + state: 'unavailable', + detail: 'runtime_header_ambiguous', + }); + }, 15000); +}); diff --git a/crates/trusted-server-js/lib/test/trace/runtime.test.ts b/crates/trusted-server-js/lib/test/trace/runtime.test.ts new file mode 100644 index 000000000..d8a1da487 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/runtime.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, it, vi } from 'vitest'; + +import type { TsjsApi } from '../../src/core/types'; +import { buildAdRequest } from '../../src/core/auction'; +import { + getActiveTraceCollector, + installTraceRuntime, + prepareTraceAuctionRequest, +} from '../../src/trace/runtime'; + +function setup(active: unknown = true) { + const api = {} as TsjsApi; + const randomUUID = vi.fn(() => '12345678-1234-4abc-8def-123456789abc'); + const scope = { tsjs: api, __tsjs_trace_active: active, crypto: { randomUUID } }; + return { api, scope, randomUUID }; +} +function request() { + return buildAdRequest([ + { + code: 'example-slot', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bidder: 'example-bidder', + }, + { + adUnitCode: 'example-slot', + mediaTypes: { banner: { sizes: [[320, 50]] } }, + bidder: 'example-other', + }, + ]); +} +describe('one strictly gated page trace facade', () => { + it.each([undefined, false, 'true', 1, null])( + 'does not activate or mint tokens for nonliteral gate %s', + (active) => { + const fixture = setup(); + fixture.scope.__tsjs_trace_active = active; + expect(installTraceRuntime(fixture.api, fixture.scope)).toBeUndefined(); + expect(getActiveTraceCollector(fixture.scope)).toBeUndefined(); + expect(prepareTraceAuctionRequest(request(), fixture.scope)).toBeUndefined(); + expect(fixture.randomUUID).not.toHaveBeenCalled(); + expect(Object.keys(fixture.api)).toEqual([]); + } + ); + it('reuses the page facade across independent module instances without storage access', async () => { + const fixture = setup(); + const storage = vi.spyOn(Storage.prototype, 'getItem'); + const first = installTraceRuntime(fixture.api, fixture.scope); + expect(first).toBeDefined(); + expect(getActiveTraceCollector(fixture.scope)).toBe(first); + vi.resetModules(); + const independent = await import('../../src/trace/runtime'); + expect(independent.installTraceRuntime(fixture.api, fixture.scope)).toBe(first); + expect(storage).not.toHaveBeenCalled(); + storage.mockRestore(); + }); + it('mints one opaque ref after final grouping and preserves original input payload', () => { + const fixture = setup(); + installTraceRuntime(fixture.api, fixture.scope); + const original = request(); + const before = structuredClone(original); + const result = prepareTraceAuctionRequest(original, fixture.scope); + expect(result).toMatchObject({ + request: { + adUnits: [ + { + code: 'example-slot', + ext: { + trusted_server: { trace_slot_ref: 'ts-slot-12345678-1234-4abc-8def-123456789abc' }, + }, + }, + ], + }, + slotRefs: ['ts-slot-12345678-1234-4abc-8def-123456789abc'], + }); + expect(fixture.randomUUID).toHaveBeenCalledTimes(1); + expect(original).toEqual(before); + }); + it('returns an observer without client refs if Web Crypto is unavailable', () => { + const fixture = setup(); + installTraceRuntime(fixture.api, fixture.scope); + const scope = { ...fixture.scope, crypto: undefined }; + const original = request(); + expect(prepareTraceAuctionRequest(original, scope)).toMatchObject({ + request: original, + slotRefs: [], + }); + }); + it('fails open when token generation throws or returns malformed or duplicate tokens', () => { + const fixture = setup(); + installTraceRuntime(fixture.api, fixture.scope); + const original = request(); + fixture.randomUUID.mockImplementation(() => { + throw new Error('private-random-error'); + }); + expect(prepareTraceAuctionRequest(original, fixture.scope)).toMatchObject({ + request: original, + slotRefs: [], + }); + fixture.randomUUID.mockReturnValue('private-invalid-token'); + expect(prepareTraceAuctionRequest(original, fixture.scope)).toMatchObject({ + request: original, + slotRefs: [], + }); + fixture.randomUUID.mockReturnValue('12345678-1234-4abc-8def-123456789abc'); + const two = buildAdRequest([{ code: 'example-one' }, { code: 'example-two' }]); + expect(prepareTraceAuctionRequest(two, fixture.scope)).toMatchObject({ + request: two, + slotRefs: [], + }); + expect(original.adUnits[0]).not.toHaveProperty('ext'); + }); + it('does not create a new collector from an auction caller before core initialization', () => { + const fixture = setup(); + expect(prepareTraceAuctionRequest(request(), fixture.scope)).toBeUndefined(); + expect(fixture.randomUUID).not.toHaveBeenCalled(); + expect(Object.keys(fixture.api)).toEqual([]); + }); +}); diff --git a/crates/trusted-server-js/lib/test/trace/setup.test.ts b/crates/trusted-server-js/lib/test/trace/setup.test.ts new file mode 100644 index 000000000..b41bfb587 --- /dev/null +++ b/crates/trusted-server-js/lib/test/trace/setup.test.ts @@ -0,0 +1,162 @@ +import { beforeEach, afterEach, describe, expect, it, vi } from 'vitest'; + +import { mountTraceSetup } from '../../src/trace/setup'; + +const setupContext = { + schema_version: 1, + captured_at: '2026-10-05T10:15:30Z', + network: { masked_client_ip: '192.0.2.0/24' }, + cookies: Object.fromEntries( + ['ts_ec', 'ts_eids', 'ts_tester', 'diagnostics_session'].map((name) => [ + name, + { source: 'request', state: 'unavailable', detail: 'runtime_header_ambiguous' }, + ]) + ), +}; + +function fixture(active = false) { + document.body.innerHTML = ` +

+

+ 

+    

+    
+    
+    `;
+  document.querySelector('#trace-request-context')!.textContent = JSON.stringify(setupContext);
+}
+
+async function flush() {
+  await vi.waitFor(() =>
+    expect(document.querySelector('#trace-enable')?.disabled).toBe(false)
+  );
+}
+
+describe('mobile trace setup', () => {
+  const request = vi.fn();
+  beforeEach(() => {
+    request.mockReset();
+    vi.stubGlobal('fetch', request);
+    fixture();
+  });
+  afterEach(() => {
+    vi.unstubAllGlobals();
+    vi.restoreAllMocks();
+    document.body.replaceChildren();
+  });
+
+  it('renders setup facts without automatically activating or claiming history', () => {
+    mountTraceSetup();
+    expect(request).not.toHaveBeenCalled();
+    expect(document.querySelector('#trace-session-state')?.textContent).toBe(
+      'Tracing is off — no valid diagnostics session observed'
+    );
+    expect(document.querySelector('#trace-network-facts')?.textContent).toContain('192.0.2.0/24');
+    expect(document.querySelector('#trace-cookie-facts')?.textContent).toContain(
+      'runtime-visible cookies could not be reliably inspected'
+    );
+  });
+
+  it('renders existing server observation without a mutation request', () => {
+    fixture(true);
+    mountTraceSetup();
+    expect(document.querySelector('#trace-session-state')?.textContent).toBe(
+      'Tracing is on — cookie observed by server'
+    );
+    expect(request).not.toHaveBeenCalled();
+  });
+
+  it.each([null, '', 'yes', '1', 'False'])(
+    'does not invent an inactive observation from invalid setup state %s',
+    (attribute) => {
+      const state = document.querySelector('#trace-session-state')!;
+      if (attribute === null) state.removeAttribute('data-observed-active');
+      else state.setAttribute('data-observed-active', attribute);
+      mountTraceSetup();
+      expect(state.textContent).toBe('Tracing state unconfirmed');
+      expect(request).not.toHaveBeenCalled();
+    }
+  );
+
+  it('confirms activation only after state verification and gives reload instructions', async () => {
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockResolvedValueOnce(new Response('{"observed_active":true}'));
+    mountTraceSetup();
+    document.querySelector('#trace-enable')!.click();
+    expect(document.querySelector('#trace-end')?.disabled).toBe(true);
+    await flush();
+    expect(document.querySelector('#trace-session-state')?.textContent).toContain('Tracing is on');
+    expect(document.querySelector('#trace-status')?.textContent).toContain('reload once');
+    expect(document.querySelector('#trace-status')?.textContent).toContain('View trace results');
+  });
+
+  it('keeps ambiguous activation unconfirmed and allows explicit retry', async () => {
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockResolvedValueOnce(new Response('{"observed_active":false}'));
+    mountTraceSetup();
+    document.querySelector('#trace-enable')!.click();
+    await flush();
+    expect(document.querySelector('#trace-status')?.textContent).toContain(
+      'Activation unconfirmed'
+    );
+    expect(document.querySelector('#trace-session-state')?.textContent).toContain(
+      'no valid diagnostics session observed'
+    );
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockResolvedValueOnce(new Response('{"observed_active":true}'));
+    document.querySelector('#trace-enable')!.click();
+    await flush();
+    expect(document.querySelector('#trace-session-state')?.textContent).toContain('Tracing is on');
+  });
+
+  it('allows end with ambiguous cookies and describes observation without claiming absence', async () => {
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockResolvedValueOnce(new Response('{"observed_active":false}'));
+    mountTraceSetup();
+    document.querySelector('#trace-end')!.click();
+    await flush();
+    expect(document.querySelector('#trace-status')?.textContent).toContain(
+      'no valid diagnostics session observed'
+    );
+    expect(document.body.textContent).not.toContain('cookie absent');
+  });
+
+  it('reports failed observation and restores action controls', async () => {
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockRejectedValueOnce(new Error('secret-error-sentinel'));
+    mountTraceSetup();
+    document.querySelector('#trace-end')!.click();
+    await flush();
+    expect(document.querySelector('#trace-status')?.textContent).toContain(
+      'Deactivation unconfirmed'
+    );
+    expect(document.querySelector('#trace-session-state')?.textContent).toBe(
+      'Tracing state unconfirmed'
+    );
+    expect(document.body.textContent).not.toContain('secret-error-sentinel');
+  });
+
+  it('fails closed on malformed setup facts without displaying a forbidden value', () => {
+    document.querySelector('#trace-request-context')!.textContent =
+      '{"secret":"raw-cookie-sentinel"}';
+    mountTraceSetup();
+    expect(document.querySelector('#trace-network-facts')?.textContent).toContain('Unavailable');
+    expect(document.querySelector('#trace-cookie-facts')?.textContent).not.toContain(
+      'raw-cookie-sentinel'
+    );
+  });
+
+  it('uses history only on explicit back and supplies a same-host same-tab fallback', () => {
+    const back = vi.spyOn(window.history, 'back').mockImplementation(() => {});
+    vi.spyOn(window.history, 'length', 'get').mockReturnValue(1);
+    mountTraceSetup();
+    document.querySelector('#trace-back')!.click();
+    expect(back).not.toHaveBeenCalled();
+    expect(document.querySelector('#trace-status')?.textContent).toContain(
+      'exact same hostname and in this same tab'
+    );
+    vi.spyOn(window.history, 'length', 'get').mockReturnValue(2);
+    document.querySelector('#trace-back')!.click();
+    expect(back).toHaveBeenCalledOnce();
+  });
+});
diff --git a/crates/trusted-server-js/lib/test/trace/storage.test.ts b/crates/trusted-server-js/lib/test/trace/storage.test.ts
new file mode 100644
index 000000000..c234b404c
--- /dev/null
+++ b/crates/trusted-server-js/lib/test/trace/storage.test.ts
@@ -0,0 +1,146 @@
+import { describe, expect, it, vi } from 'vitest';
+
+import {
+  readTraceReport,
+  storeTraceReport,
+  deleteTraceReport,
+  TRACE_REPORT_STORAGE_KEY,
+} from '../../src/trace/storage';
+import { validateTraceStoredReport } from '../../src/trace/report-validation';
+
+import { reportFixture, TRACE_ORIGIN, TRACE_NOW } from './fixtures';
+
+function fixture() {
+  return { stored_at_ms: TRACE_NOW, report: reportFixture() };
+}
+function memory(initial: string | null = null) {
+  let value = initial;
+  return {
+    getItem: vi.fn((key: string) => (key === TRACE_REPORT_STORAGE_KEY ? value : null)),
+    setItem: vi.fn((key: string, next: string) => {
+      if (key === TRACE_REPORT_STORAGE_KEY) value = next;
+    }),
+    removeItem: vi.fn((key: string) => {
+      if (key === TRACE_REPORT_STORAGE_KEY) value = null;
+    }),
+  };
+}
+describe('one origin-local validated trace report', () => {
+  it('requires a safe integer capture timestamp in the wrapper validator', () => {
+    expect(
+      validateTraceStoredReport(
+        { ...fixture(), stored_at_ms: TRACE_NOW + 0.5 },
+        TRACE_ORIGIN,
+        TRACE_NOW
+      )
+    ).toBe(false);
+  });
+  it('stores only the validated exact wrapper and replaces the previous explicit capture', () => {
+    const storage = memory();
+    expect(storeTraceReport(fixture(), TRACE_ORIGIN, TRACE_NOW, storage)).toEqual({
+      status: 'stored',
+    });
+    expect(storage.setItem).toHaveBeenCalledWith(
+      TRACE_REPORT_STORAGE_KEY,
+      JSON.stringify(fixture())
+    );
+    const next = fixture();
+    next.report.request_context.network = {};
+    expect(storeTraceReport(next, TRACE_ORIGIN, TRACE_NOW, storage)).toEqual({ status: 'stored' });
+    expect(storage.setItem).toHaveBeenCalledTimes(2);
+  });
+  it('reads an independently owned frozen model without writing it back', () => {
+    const storage = memory(JSON.stringify(fixture()));
+    const result = readTraceReport(TRACE_ORIGIN, TRACE_NOW, storage);
+    expect(result).toEqual({ status: 'ready', value: fixture() });
+    expect(storage.setItem).not.toHaveBeenCalled();
+    expect(storage.removeItem).not.toHaveBeenCalled();
+    if (typeof result !== 'object' || result === null || !('value' in result))
+      throw new Error('should read report');
+    expect(Object.isFrozen(result.value)).toBe(true);
+  });
+  it('distinguishes absent data from unavailable storage without writes', () => {
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, memory())).toEqual({ status: 'absent' });
+    const storage = memory();
+    storage.getItem.mockImplementation(() => {
+      throw new Error('private-read-error');
+    });
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, storage)).toEqual({ status: 'unavailable' });
+    expect(storage.setItem).not.toHaveBeenCalled();
+  });
+  it.each([
+    '{',
+    JSON.stringify({ stored_at_ms: TRACE_NOW }),
+    JSON.stringify({ ...fixture(), private_value: 'secret' }),
+    JSON.stringify({ ...fixture(), stored_at_ms: TRACE_NOW + 0.5 }),
+    JSON.stringify({ ...fixture(), stored_at_ms: TRACE_NOW + 60001 }),
+    JSON.stringify({ ...fixture(), stored_at_ms: TRACE_NOW - 900001 }),
+    JSON.stringify({ ...fixture(), stored_at_ms: -1 }),
+  ])('removes and ignores rejected storage %#', (serialized) => {
+    const storage = memory(serialized);
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, storage)).toMatchObject({ status: 'rejected' });
+    expect(storage.removeItem).toHaveBeenCalledWith(TRACE_REPORT_STORAGE_KEY);
+    expect(storage.setItem).not.toHaveBeenCalled();
+  });
+  it('accepts exactly fifteen minutes and sixty seconds of future clock skew', () => {
+    expect(
+      readTraceReport(TRACE_ORIGIN, TRACE_NOW + 900000, memory(JSON.stringify(fixture())))
+    ).toMatchObject({ status: 'ready' });
+    expect(
+      readTraceReport(TRACE_ORIGIN, TRACE_NOW - 60000, memory(JSON.stringify(fixture())))
+    ).toMatchObject({ status: 'ready' });
+    expect(
+      readTraceReport(TRACE_ORIGIN, TRACE_NOW - 60001, memory(JSON.stringify(fixture())))
+    ).toMatchObject({ status: 'rejected' });
+  });
+  it('rejects another origin and an unsupported public version with a bounded reason', () => {
+    expect(
+      readTraceReport('https://other.example.com', TRACE_NOW, memory(JSON.stringify(fixture())))
+    ).toMatchObject({ status: 'rejected', reason: 'invalid_report' });
+    const value = fixture();
+    value.report.schema_version = 2;
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, memory(JSON.stringify(value)))).toMatchObject({
+      status: 'rejected',
+      reason: 'unsupported_report_version',
+    });
+  });
+  it('ignores rejected entries even when removal throws', () => {
+    const storage = memory('{');
+    storage.removeItem.mockImplementation(() => {
+      throw new Error('private-remove-error');
+    });
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, storage)).toMatchObject({ status: 'rejected' });
+  });
+  it('rejects oversized serialized UTF-8 before parsing and never writes invalid models', () => {
+    const storage = memory(' '.repeat(512 * 1024 + 1));
+    expect(readTraceReport(TRACE_ORIGIN, TRACE_NOW, storage)).toEqual({
+      status: 'rejected',
+      reason: 'invalid_report',
+    });
+    expect(
+      storeTraceReport({ ...fixture(), private_value: 'secret' }, TRACE_ORIGIN, TRACE_NOW, storage)
+    ).toEqual({ status: 'rejected', reason: 'invalid_report' });
+    expect(storage.setItem).not.toHaveBeenCalled();
+  });
+  it('preserves the report model when writing fails and exposes no exception text', () => {
+    const storage = memory();
+    storage.setItem.mockImplementation(() => {
+      throw new Error('private-write-error');
+    });
+    const source = fixture();
+    expect(storeTraceReport(source, TRACE_ORIGIN, TRACE_NOW, storage)).toEqual({
+      status: 'unavailable',
+    });
+    expect(source).toEqual(fixture());
+    expect(storage.removeItem).not.toHaveBeenCalled();
+  });
+  it('deletes explicitly and reports failures without modifying another storage key', () => {
+    const storage = memory(JSON.stringify(fixture()));
+    expect(deleteTraceReport(storage)).toEqual({ status: 'deleted' });
+    expect(storage.removeItem).toHaveBeenCalledWith(TRACE_REPORT_STORAGE_KEY);
+    storage.removeItem.mockImplementation(() => {
+      throw new Error('private-remove-error');
+    });
+    expect(deleteTraceReport(storage)).toEqual({ status: 'unavailable' });
+  });
+});
diff --git a/crates/trusted-server-js/lib/test/trace/tokens.test.ts b/crates/trusted-server-js/lib/test/trace/tokens.test.ts
new file mode 100644
index 000000000..83987fe1d
--- /dev/null
+++ b/crates/trusted-server-js/lib/test/trace/tokens.test.ts
@@ -0,0 +1,51 @@
+import { describe, expect, it } from 'vitest';
+
+import { validDiagnosticAuctionId, validTraceSlotRef } from '../../src/trace/validation';
+
+export const AUCTION = 'ts-auc-1234567812344abc8def123456789abc';
+export const SLOT = 'ts-slot-12345678-1234-4abc-8def-123456789abc';
+
+describe('exact opaque trace tokens', () => {
+  it('accepts only the producer-specific UUID v4 formats', () => {
+    expect(validDiagnosticAuctionId(AUCTION)).toBe(true);
+    expect(validTraceSlotRef(SLOT)).toBe(true);
+  });
+
+  it.each([
+    null,
+    1,
+    '',
+    'internal-auction-id',
+    AUCTION.toUpperCase(),
+    ` ${AUCTION}`,
+    `${AUCTION} `,
+    AUCTION.replace('4abc', '5abc'),
+    AUCTION.replace('8def', '7def'),
+    AUCTION.replace('8def', 'cdef'),
+    'ts-auc-12345678-1234-4abc-8def-123456789abc',
+    `${AUCTION}\n`,
+    `${AUCTION}\u202e`,
+    SLOT,
+  ])('rejects every alternate auction-token spelling %j', (value) => {
+    expect(validDiagnosticAuctionId(value)).toBe(false);
+  });
+
+  it.each([
+    null,
+    1,
+    '',
+    'raw-slot-element-id',
+    SLOT.toUpperCase(),
+    ` ${SLOT}`,
+    `${SLOT} `,
+    SLOT.replace('4abc', '3abc'),
+    SLOT.replace('8def', '7def'),
+    SLOT.replace('8def', 'cdef'),
+    'ts-slot-1234567812344abc8def123456789abc',
+    `${SLOT}\n`,
+    `${SLOT}\u202e`,
+    AUCTION,
+  ])('rejects every alternate slot-token spelling %j', (value) => {
+    expect(validTraceSlotRef(value)).toBe(false);
+  });
+});
diff --git a/crates/trusted-server-js/lib/test/trace/types.test.ts b/crates/trusted-server-js/lib/test/trace/types.test.ts
new file mode 100644
index 000000000..97a9c2ccc
--- /dev/null
+++ b/crates/trusted-server-js/lib/test/trace/types.test.ts
@@ -0,0 +1,149 @@
+import { describe, expect, it } from 'vitest';
+
+import type {
+  GptDiagnosticsExportV1,
+  GptDiagnosticsSlotExport,
+  GptDiagnosticsRequestCycle,
+  GptDiagnosticsBinding,
+  GptDiagnosticsDurations,
+  GptDiagnosticsCallbackIssue,
+  GptDiagnosticsAttributionIssue,
+  GptDiagnosticsCoverageCounters,
+  GptDiagnosticsAdManagerIdentity,
+} from '../../src/core/types';
+
+type Treatment = 'copy' | 'exclude' | 'transform';
+
+const root = {
+  version: 'transform',
+  capturedAt: 'copy',
+  page: 'transform',
+  slots: 'transform',
+  callbackIssues: 'transform',
+  attributionIssues: 'transform',
+  coverage: 'transform',
+  metadata: 'transform',
+} satisfies Record;
+const page = { origin: 'copy', pathname: 'transform' } satisfies Record<
+  keyof GptDiagnosticsExportV1['page'],
+  Treatment
+>;
+const slot = {
+  runtimeSlotNumber: 'copy',
+  slotElementId: 'exclude',
+  adUnitPath: 'exclude',
+  binding: 'transform',
+  currentVisibilityPercentage: 'copy',
+  maximumVisibilityPercentage: 'copy',
+  requests: 'transform',
+} satisfies Record;
+const cycle = {
+  requestNumber: 'copy',
+  requestedAtMs: 'copy',
+  responseAtMs: 'copy',
+  renderAtMs: 'copy',
+  loadAtMs: 'copy',
+  viewableAtMs: 'copy',
+  durations: 'transform',
+  isEmpty: 'copy',
+  requestedSlotSizes: 'copy',
+  size: 'copy',
+  observedSlotSize: 'copy',
+  isBackfill: 'copy',
+  slotContentChanged: 'copy',
+  incompleteSequence: 'copy',
+  adManager: 'exclude',
+  responseClass: 'copy',
+  requestPath: 'copy',
+  requestIntentId: 'copy',
+  trustedServerAuctionId: 'copy',
+  opportunityToRequestMs: 'copy',
+  replacedRequestNumber: 'copy',
+  previousRenderToRequestMs: 'copy',
+  creativeChanged: 'copy',
+  previousCreativeId: 'exclude',
+  loadObservedBeforeRender: 'copy',
+  trustedServerOpportunity: 'copy',
+  trustedServerCreativeRequestAtMs: 'copy',
+  trustedServerCreativeResponseAtMs: 'copy',
+  trustedServerCreativeFailures: 'copy',
+  delivery: 'copy',
+} satisfies Record;
+const binding = { status: 'copy', reason: 'copy' } satisfies Record<
+  keyof GptDiagnosticsBinding,
+  Treatment
+>;
+const durations = {
+  requestToResponseMs: 'copy',
+  responseToRenderMs: 'copy',
+  requestToRenderMs: 'copy',
+  renderToLoadMs: 'copy',
+  renderToViewableMs: 'copy',
+} satisfies Record;
+const callback = {
+  kind: 'copy',
+  runtimeSlotNumber: 'copy',
+  slotElementId: 'exclude',
+  timestampMs: 'copy',
+  disposition: 'copy',
+  reason: 'copy',
+} satisfies Record;
+const attribution = {
+  reason: 'copy',
+  timestampMs: 'copy',
+  runtimeSlotNumber: 'copy',
+  slotElementId: 'exclude',
+} satisfies Record;
+const counters = {
+  observed: 'copy',
+  matched: 'copy',
+  unmatched: 'copy',
+  ambiguous: 'copy',
+} satisfies Record;
+const coverage = {
+  slotRequested: 'transform',
+  slotResponseReceived: 'transform',
+  slotRenderEnded: 'transform',
+  slotOnload: 'transform',
+  impressionViewable: 'transform',
+  slotVisibilityChanged: 'transform',
+} satisfies Record;
+const metadata = {
+  droppedCallbacks: 'copy',
+  droppedAttributionIssues: 'copy',
+  evictedSlots: 'copy',
+  evictedRequestCycles: 'copy',
+} satisfies Record;
+const adManager = {
+  lineItemId: 'exclude',
+  creativeId: 'exclude',
+  campaignId: 'exclude',
+  advertiserId: 'exclude',
+  sourceAgnosticLineItemId: 'exclude',
+  sourceAgnosticCreativeId: 'exclude',
+  yieldGroupIds: 'exclude',
+  companyIds: 'exclude',
+} satisfies Record;
+
+// @ts-expect-error Future or missing source members must be classified explicitly.
+const incomplete: Record = { requestNumber: 'copy' };
+
+describe('explicit current GPT source member classification', () => {
+  it('records every excluded identifier and each deliberate transformation', () => {
+    expect(cycle.adManager).toBe('exclude');
+    expect(cycle.previousCreativeId).toBe('exclude');
+    expect(slot.slotElementId).toBe('exclude');
+    expect(slot.adUnitPath).toBe('exclude');
+    expect(callback.slotElementId).toBe('exclude');
+    expect(attribution.slotElementId).toBe('exclude');
+    expect(Object.values(adManager).every((value) => value === 'exclude')).toBe(true);
+    expect(root.version).toBe('transform');
+    expect(page.pathname).toBe('transform');
+    expect(Object.keys(binding)).toHaveLength(2);
+    expect(Object.keys(durations)).toHaveLength(5);
+    expect(Object.keys(counters)).toHaveLength(4);
+    expect(Object.keys(coverage)).toHaveLength(6);
+    expect(Object.keys(metadata)).toHaveLength(4);
+    expect(incomplete.requestNumber).toBe('copy');
+  });
+});
diff --git a/crates/trusted-server-js/lib/test/trace/validation.test.ts b/crates/trusted-server-js/lib/test/trace/validation.test.ts
new file mode 100644
index 000000000..8a00041ae
--- /dev/null
+++ b/crates/trusted-server-js/lib/test/trace/validation.test.ts
@@ -0,0 +1,596 @@
+import { describe, expect, it } from 'vitest';
+
+import {
+  parseTraceReport,
+  parseTraceStoredReport,
+  traceReportRejection,
+  boundedJsonShape,
+  validateTraceGptDiagnostics,
+  validateTraceReport,
+  validateTraceStoredReport,
+} from '../../src/trace/report-validation';
+import {
+  TRACE_BINDING_REASONS,
+  TRACE_CALLBACK_KINDS,
+  TRACE_CALLBACK_REASONS,
+  TRACE_ATTRIBUTION_REASONS,
+  TRACE_RESPONSE_CLASSES,
+  TRACE_REQUEST_PATHS,
+  TRACE_OPPORTUNITIES,
+  TRACE_DELIVERIES,
+} from '../../src/trace/report-types';
+
+import {
+  projectedGptFixture,
+  reportFixture,
+  TRACE_ORIGIN,
+  TRACE_TIME,
+  TRACE_NOW,
+} from './fixtures';
+import { AUCTION_TOKEN, SLOT_TOKEN } from './fixtures';
+
+function valid(value: unknown) {
+  return validateTraceGptDiagnostics(value, TRACE_ORIGIN);
+}
+
+describe('strict combined report ingestion', () => {
+  it('returns a frozen owned model instead of retaining a proxy that changes later reads', () => {
+    const value = reportFixture();
+    const original = value.gpt_diagnostics.metadata;
+    let traps = 0;
+    value.gpt_diagnostics.metadata = new Proxy(original, {
+      get() {
+        traps += 1;
+        throw new Error('should not call');
+      },
+    });
+    const model = parseTraceReport(value, TRACE_ORIGIN, TRACE_NOW);
+    expect(model).toBeDefined();
+    expect(traps).toBe(0);
+    expect(Object.is(model, value)).toBe(false);
+    expect(model?.gpt_diagnostics.metadata).not.toBe(original);
+    expect(Object.isFrozen(original)).toBe(false);
+    original.droppedCallbacks = 99;
+    expect(model?.gpt_diagnostics.metadata.droppedCallbacks).toBe(0);
+    const frozen = (object: unknown): void => {
+      if (typeof object !== 'object' || object === null) return;
+      expect(Object.isFrozen(object)).toBe(true);
+      Object.values(object).forEach(frozen);
+    };
+    frozen(model);
+    const wrapper = parseTraceStoredReport(
+      { stored_at_ms: TRACE_NOW, report: reportFixture() },
+      TRACE_ORIGIN,
+      TRACE_NOW
+    );
+    expect(Object.isFrozen(wrapper)).toBe(true);
+    expect(wrapper?.stored_at_ms).toBe(TRACE_NOW);
+    expect(
+      parseTraceStoredReport(
+        { stored_at_ms: TRACE_NOW, report: reportFixture() },
+        TRACE_ORIGIN,
+        TRACE_NOW + 900001
+      )
+    ).toBeUndefined();
+  });
+  it('validates the same own counter and enum data that is measured for storage', () => {
+    const counter = reportFixture();
+    Object.assign(counter.gpt_diagnostics.metadata, { droppedCallbacks: 'private-counter' });
+    counter.gpt_diagnostics.metadata = new Proxy(counter.gpt_diagnostics.metadata, {
+      get(target, name, receiver) {
+        return name === 'droppedCallbacks' ? 0 : Reflect.get(target, name, receiver);
+      },
+    });
+    expect(validateTraceReport(counter, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    const enumValue = projectedGptFixture();
+    enumValue.slots[0].requests[0].delivery = 'private-delivery';
+    enumValue.slots[0].requests[0] = new Proxy(enumValue.slots[0].requests[0], {
+      get(target, name, receiver) {
+        return name === 'delivery' ? 'unknown' : Reflect.get(target, name, receiver);
+      },
+    });
+    expect(valid(enumValue)).toBe(false);
+    const token = projectedGptFixture();
+    token.slots[0].requests[0].trustedServerAuctionId = 'private-id';
+    token.slots[0].requests[0] = new Proxy(token.slots[0].requests[0], {
+      get(target, name, receiver) {
+        return name === 'trustedServerAuctionId'
+          ? AUCTION_TOKEN
+          : Reflect.get(target, name, receiver);
+      },
+    });
+    expect(valid(token)).toBe(false);
+  });
+  it('accepts numeric maxima and rejects fractional identifiers without changing fractional timing', () => {
+    const value = projectedGptFixture();
+    const cycle = value.slots[0].requests[0];
+    Object.assign(cycle, {
+      requestNumber: Number.MAX_SAFE_INTEGER,
+      requestIntentId: 0,
+      replacedRequestNumber: Number.MAX_SAFE_INTEGER,
+      requestedAtMs: Number.MAX_SAFE_INTEGER,
+      size: [100000, 100000],
+      observedSlotSize: [0, 100000],
+    });
+    Object.assign(value.metadata, { droppedCallbacks: Number.MAX_SAFE_INTEGER });
+    expect(valid(value)).toBe(true);
+    cycle.requestNumber = 0.5;
+    expect(valid(value)).toBe(false);
+  });
+
+  it('enforces the 255 UTF-8 byte origin cap at the exact boundary', () => {
+    const prefix = `https://${'a'.repeat(63)}.${'a'.repeat(63)}.${'a'.repeat(63)}.`;
+    const origin = prefix + 'b'.repeat(55);
+    expect(new TextEncoder().encode(origin).length).toBe(255);
+    const value = projectedGptFixture();
+    value.page.origin = origin;
+    expect(validateTraceGptDiagnostics(value, origin)).toBe(true);
+    value.page.origin = origin + 'b';
+    expect(validateTraceGptDiagnostics(value, origin + 'b')).toBe(false);
+  });
+
+  it('validates full nested report containers and the runtime ambiguity detail pair', () => {
+    const value = reportFixture();
+    expect(boundedJsonShape(value, 8)).toBeDefined();
+    expect(boundedJsonShape(value, 7)).toBeUndefined();
+    Object.assign(value.request_context.cookies.diagnostics_session, { state: 'present_invalid' });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+  });
+  it('returns bounded actionable categories for incompatible versions', () => {
+    expect(traceReportRejection(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toBeUndefined();
+    expect(
+      traceReportRejection({ ...reportFixture(), schema_version: 2 }, TRACE_ORIGIN, TRACE_NOW)
+    ).toBe('unsupported_report_version');
+    const gpt = reportFixture();
+    Object.assign(gpt.gpt_diagnostics, { schema_version: 2 });
+    expect(traceReportRejection(gpt, TRACE_ORIGIN, TRACE_NOW)).toBe('unsupported_gpt_version');
+    Object.assign(gpt.gpt_diagnostics, { schema_version: 1, source_schema_version: 2 });
+    expect(traceReportRejection(gpt, TRACE_ORIGIN, TRACE_NOW)).toBe(
+      'unsupported_gpt_source_version'
+    );
+    const auction = reportFixture();
+    Object.assign(auction, { server_auctions: [{ schema_version: 2 }] });
+    expect(traceReportRejection(auction, TRACE_ORIGIN, TRACE_NOW)).toBe(
+      'unsupported_auction_version'
+    );
+    const correlation = reportFixture();
+    Object.assign(correlation, { slot_correlations: [{ schema_version: 2 }] });
+    expect(traceReportRejection(correlation, TRACE_ORIGIN, TRACE_NOW)).toBe(
+      'unsupported_correlation_version'
+    );
+    expect(traceReportRejection({ private: 'private-error' }, TRACE_ORIGIN, TRACE_NOW)).toBe(
+      'invalid_report'
+    );
+  });
+  it('accepts every supported enum at its own boundary', () => {
+    for (const reason of TRACE_BINDING_REASONS) {
+      const value = projectedGptFixture();
+      value.slots[0].binding.reason = reason;
+      expect(valid(value)).toBe(true);
+    }
+    for (const kind of TRACE_CALLBACK_KINDS) {
+      const value = projectedGptFixture();
+      value.callbackIssues[0].kind = kind;
+      expect(valid(value)).toBe(true);
+    }
+    for (const reason of TRACE_CALLBACK_REASONS) {
+      const value = projectedGptFixture();
+      value.callbackIssues[0].reason = reason;
+      expect(valid(value)).toBe(true);
+    }
+    for (const reason of TRACE_ATTRIBUTION_REASONS) {
+      const value = projectedGptFixture();
+      value.attributionIssues[0].reason = reason;
+      expect(valid(value)).toBe(true);
+    }
+    for (const [key, members] of [
+      ['responseClass', TRACE_RESPONSE_CLASSES],
+      ['requestPath', TRACE_REQUEST_PATHS],
+      ['trustedServerOpportunity', TRACE_OPPORTUNITIES],
+      ['delivery', TRACE_DELIVERIES],
+    ] as const) {
+      for (const member of members) {
+        const value = projectedGptFixture();
+        Object.assign(value.slots[0].requests[0], { [key]: member });
+        expect(valid(value)).toBe(true);
+      }
+    }
+  });
+
+  it('counts actual nested containers from the report root and enforces compact UTF-8 bytes', () => {
+    let depthTen: unknown = null;
+    for (let index = 0; index < 10; index += 1) depthTen = { child: depthTen };
+    expect(boundedJsonShape(depthTen, 10)).toBeDefined();
+    expect(boundedJsonShape({ child: depthTen }, 10)).toBeUndefined();
+    const text = { value: 'é😀' };
+    const bytes = new TextEncoder().encode(JSON.stringify(text)).length;
+    expect(boundedJsonShape(text, 10, bytes)).toBe(bytes);
+    expect(boundedJsonShape(text, 10, bytes - 1)).toBeUndefined();
+    const cyclical: { child?: unknown } = {};
+    cyclical.child = cyclical;
+    expect(boundedJsonShape(cyclical)).toBeUndefined();
+    let calls = 0;
+    expect(
+      boundedJsonShape({
+        toJSON() {
+          calls += 1;
+          return {};
+        },
+      })
+    ).toBeUndefined();
+    expect(calls).toBe(0);
+  });
+
+  it('rejects unsafe Unicode and controls at text-bearing boundaries', () => {
+    for (const suffix of ['\u0000', '\u001f', '\u007f', '\u009f', '\u202e', '\u2066', '\ud800']) {
+      const value = projectedGptFixture();
+      value.page.origin = TRACE_ORIGIN + suffix;
+      expect(valid(value)).toBe(false);
+      value.page.origin = TRACE_ORIGIN;
+      value.capturedAt = TRACE_TIME + suffix;
+      expect(valid(value)).toBe(false);
+    }
+    const value = projectedGptFixture();
+    value.page.pathname = '/private-path';
+    expect(valid(value)).toBe(false);
+  });
+
+  it('enforces server/sidecar caps, status semantics and the API correlation exclusion', () => {
+    const auction = {
+      schema_version: 1,
+      diagnostic_auction_id: AUCTION_TOKEN,
+      source: 'initial_navigation_ssat',
+      terminal_status: 'completed',
+      provider_calls: [],
+      slots: [],
+      truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 },
+      coverage: { provider_to_slot_no_bid: 'unavailable' },
+    };
+    const correlation = {
+      schema_version: 1,
+      diagnostic_auction_id: AUCTION_TOKEN,
+      slot_ref: SLOT_TOKEN,
+      runtime_slot_number: 1,
+      request_number: 1,
+    };
+    const value = reportFixture();
+    Object.assign(value, {
+      server_auctions: Array.from({ length: 16 }, () => auction),
+      slot_correlations: Array.from({ length: 128 }, () => correlation),
+    });
+    value.auction_coverage.capture_status = 'complete';
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+    Object.assign(value, { server_auctions: Array.from({ length: 17 }, () => auction) });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    Object.assign(value, {
+      server_auctions: [auction],
+      slot_correlations: Array.from({ length: 129 }, () => correlation),
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    Object.assign(value, { slot_correlations: [correlation] });
+    Object.assign(value.auction_coverage, {
+      capture_status: 'partial',
+      issues: ['evidence_transport_failed'],
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+    auction.source = 'auction_api';
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+  });
+
+  it('rejects a shape-valid combined report whose compact wrapper exceeds 512 KiB', () => {
+    const gpt = projectedGptFixture();
+    gpt.slots = Array.from({ length: 64 }, () => {
+      const slot = projectedGptFixture().slots[0];
+      slot.requests = Array.from({ length: 10 }, () => projectedGptFixture().slots[0].requests[0]);
+      return slot;
+    });
+    expect(valid(gpt)).toBe(true);
+    const report = reportFixture();
+    report.gpt_diagnostics = gpt;
+    const wrapper = { stored_at_ms: TRACE_NOW, report };
+    expect(new TextEncoder().encode(JSON.stringify(wrapper)).length).toBeGreaterThan(512 * 1024);
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+  });
+  it('accepts the explicit projection and complete wrapper, including zero CSS size and fractional browser durations', () => {
+    expect(valid(projectedGptFixture())).toBe(true);
+    expect(validateTraceReport(reportFixture(), TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+    expect(
+      validateTraceStoredReport(
+        { stored_at_ms: TRACE_NOW, report: reportFixture() },
+        TRACE_ORIGIN,
+        TRACE_NOW
+      )
+    ).toBe(true);
+  });
+  it('retains source optionality without requiring later optional fields', () => {
+    const value = projectedGptFixture();
+    Reflect.deleteProperty(value, 'attributionIssues');
+    Reflect.deleteProperty(value.metadata, 'droppedAttributionIssues');
+    value.slots = [
+      {
+        runtimeSlotNumber: 0,
+        binding: { status: 'bound' },
+        requests: [{ requestNumber: 0, durations: {}, incompleteSequence: true }],
+      },
+    ] as typeof value.slots;
+    expect(valid(value)).toBe(true);
+  });
+  it.each([
+    'adManager',
+    'previousCreativeId',
+    'slotElementId',
+    'adUnitPath',
+    '__proto__',
+    'constructor',
+    'future',
+  ])('rejects forbidden request-cycle property %s', (key) => {
+    const value = projectedGptFixture();
+    Object.defineProperty(value.slots[0].requests[0], key, {
+      enumerable: true,
+      value: 'private-sentinel',
+    });
+    expect(valid(value)).toBe(false);
+  });
+  it('rejects extra keys at every nested boundary', () => {
+    const value = projectedGptFixture();
+    const objects = [
+      value,
+      value.page,
+      value.slots[0],
+      value.slots[0].binding,
+      value.slots[0].requests[0],
+      value.slots[0].requests[0].durations,
+      value.callbackIssues[0],
+      value.attributionIssues[0],
+      value.coverage,
+      value.coverage.slotRequested,
+      value.metadata,
+    ];
+    for (const target of objects) {
+      Object.assign(target, { future: 'private-sentinel' });
+      expect(valid(value)).toBe(false);
+      Reflect.deleteProperty(target, 'future');
+    }
+  });
+  it.each([0, 2, '1', null])('rejects unsupported GPT/source/report versions %s', (version) => {
+    expect(valid({ ...projectedGptFixture(), schema_version: version })).toBe(false);
+    expect(valid({ ...projectedGptFixture(), source_schema_version: version })).toBe(false);
+    expect(
+      validateTraceReport({ ...reportFixture(), schema_version: version }, TRACE_ORIGIN, TRACE_NOW)
+    ).toBe(false);
+  });
+  it.each([
+    'https://attacker.example.com',
+    'ftp://publisher.example.com',
+    'https://publisher.example.com/',
+    'https://user@publisher.example.com',
+    'https://@publisher.example.com',
+    'https://publisher.example.com?',
+    'https://publisher.example.com#',
+    'https://publisher.example.com\\',
+    'https://publisher.example.com\n',
+    'https://publisher.example.com,https://publisher.example.com',
+  ])('rejects a foreign or repaired origin %s', (origin) => {
+    const value = projectedGptFixture();
+    value.page.origin = origin;
+    expect(valid(value)).toBe(false);
+  });
+  it('accepts a canonical equivalent origin without changing the input', () => {
+    const value = projectedGptFixture();
+    value.page.origin = 'https://PUBLISHER.EXAMPLE.COM:443';
+    expect(valid(value)).toBe(true);
+    expect(value.page.origin).toBe('https://PUBLISHER.EXAMPLE.COM:443');
+  });
+  it.each(['adManager', 'previousCreativeId', 'slotElementId', 'adUnitPath'])(
+    'rejects a prohibited slot/issue field %s',
+    (key) => {
+      for (const where of ['slot', 'callback', 'attribution']) {
+        const value = projectedGptFixture();
+        const target =
+          where === 'slot'
+            ? value.slots[0]
+            : where === 'callback'
+              ? value.callbackIssues[0]
+              : value.attributionIssues[0];
+        Object.assign(target, { [key]: 'private-sentinel' });
+        expect(valid(value)).toBe(false);
+      }
+    }
+  );
+  it.each([NaN, Infinity, -1, Number.MAX_SAFE_INTEGER + 1, '1', null])(
+    'rejects invalid browser timing %s',
+    (time) => {
+      const value = projectedGptFixture();
+      Object.assign(value.slots[0].requests[0], { requestedAtMs: time });
+      expect(valid(value)).toBe(false);
+    }
+  );
+  it.each([-1, 100.1, NaN, Infinity, '1'])('rejects invalid visibility %s', (percentage) => {
+    const value = projectedGptFixture();
+    Object.assign(value.slots[0], { currentVisibilityPercentage: percentage });
+    expect(valid(value)).toBe(false);
+  });
+  it('enforces each array and nested numeric bound at its edge', () => {
+    const value = projectedGptFixture();
+    value.slots = Array.from({ length: 64 }, () => projectedGptFixture().slots[0]);
+    expect(valid(value)).toBe(true);
+    value.slots.push(value.slots[0]);
+    expect(valid(value)).toBe(false);
+    const cycles = projectedGptFixture();
+    cycles.slots[0].requests = Array.from(
+      { length: 10 },
+      () => projectedGptFixture().slots[0].requests[0]
+    );
+    expect(valid(cycles)).toBe(true);
+    cycles.slots[0].requests.push(cycles.slots[0].requests[0]);
+    expect(valid(cycles)).toBe(false);
+    for (const key of ['callbackIssues', 'attributionIssues'] as const) {
+      const issues = projectedGptFixture();
+      Object.assign(issues, { [key]: Array.from({ length: 128 }, () => issues[key][0]) });
+      expect(valid(issues)).toBe(true);
+      issues[key].push(issues[key][0] as never);
+      expect(valid(issues)).toBe(false);
+    }
+    const sizes = projectedGptFixture();
+    sizes.slots[0].requests[0].requestedSlotSizes = Array.from({ length: 16 }, () => [1, 100000]);
+    expect(valid(sizes)).toBe(true);
+    sizes.slots[0].requests[0].requestedSlotSizes.push([1, 1]);
+    expect(valid(sizes)).toBe(false);
+    const failures = projectedGptFixture();
+    failures.slots[0].requests[0].trustedServerCreativeFailures = Array.from(
+      { length: 16 },
+      () => 'missing_render_source'
+    );
+    expect(valid(failures)).toBe(true);
+    failures.slots[0].requests[0].trustedServerCreativeFailures?.push('missing_render_source');
+    expect(valid(failures)).toBe(false);
+  });
+  it('rejects invalid dimension shapes and retains zero-capable observed CSS boxes only', () => {
+    for (const key of ['size', 'observedSlotSize'] as const) {
+      for (const dimensions of [[-1, 1], [1.5, 1], [100001, 1], [1], [1, 1, 1], ['1', 1]]) {
+        const value = projectedGptFixture();
+        Object.assign(value.slots[0].requests[0], { [key]: dimensions });
+        expect(valid(value)).toBe(false);
+      }
+    }
+    const zero = projectedGptFixture();
+    zero.slots[0].requests[0].size = [0, 0];
+    expect(valid(zero)).toBe(false);
+  });
+  it('rejects unknown enums and malformed tokens without coercion', () => {
+    const updates = [
+      { responseClass: 'future' },
+      { requestPath: '/private/path' },
+      { trustedServerOpportunity: 'winner' },
+      { delivery: 'rendered' },
+      { trustedServerCreativeFailures: ['raw-error'] },
+      { trustedServerAuctionId: 'internal-id' },
+      { incompleteSequence: 1 },
+    ];
+    for (const update of updates) {
+      const value = projectedGptFixture();
+      Object.assign(value.slots[0].requests[0], update);
+      expect(valid(value)).toBe(false);
+    }
+    for (const update of [{ status: 'future' }, { reason: 'private-error' }]) {
+      const value = projectedGptFixture();
+      Object.assign(value.slots[0].binding, update);
+      expect(valid(value)).toBe(false);
+    }
+    for (const update of [
+      { kind: 'future' },
+      { disposition: 'future' },
+      { reason: 'private-error' },
+    ]) {
+      const value = projectedGptFixture();
+      Object.assign(value.callbackIssues[0], update);
+      expect(valid(value)).toBe(false);
+    }
+    const attribution = projectedGptFixture();
+    attribution.attributionIssues[0].reason = 'future';
+    expect(valid(attribution)).toBe(false);
+  });
+  it('enforces finite integer counters and omission counters', () => {
+    for (const count of [-1, 0.5, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1]) {
+      const value = projectedGptFixture();
+      value.metadata.droppedCallbacks = count;
+      expect(valid(value)).toBe(false);
+    }
+    for (const count of [65536, -1, 0.5]) {
+      const value = reportFixture();
+      value.truncation.omitted_nested_values = count;
+      expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    }
+    const value = reportFixture();
+    value.truncation.omitted_nested_values = 65535;
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+  });
+  it('requires capture times close to wrapper time and leaves earlier request time eligible', () => {
+    const value = reportFixture();
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW + 60000)).toBe(true);
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW + 60001)).toBe(false);
+    value.gpt_diagnostics.capturedAt = '2026-02-30T00:00:00Z';
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    value.gpt_diagnostics.capturedAt = TRACE_TIME;
+    value.request_context.captured_at = '2025-01-01T00:00:00Z';
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+  });
+  it('requires exact storage keys, finite times, bounded age and rollback rejection', () => {
+    const wrapper = { stored_at_ms: TRACE_NOW, report: reportFixture() };
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW + 900000)).toBe(true);
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW + 900001)).toBe(false);
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW - 1)).toBe(true);
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW - 60000)).toBe(true);
+    expect(validateTraceStoredReport(wrapper, TRACE_ORIGIN, TRACE_NOW - 60001)).toBe(false);
+    expect(validateTraceStoredReport({ ...wrapper, future: 1 }, TRACE_ORIGIN, TRACE_NOW)).toBe(
+      false
+    );
+    for (const stored_at_ms of [-1, NaN, Infinity, '1', null])
+      expect(validateTraceStoredReport({ ...wrapper, stored_at_ms }, TRACE_ORIGIN, TRACE_NOW)).toBe(
+        false
+      );
+  });
+  it('recomputes capture status and requires distinct enum-order coverage issues', () => {
+    const value = reportFixture();
+    Object.assign(value.auction_coverage, {
+      capture_status: 'unavailable',
+      issues: ['record_evicted'],
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+    value.auction_coverage.capture_status = 'not_observed';
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    Object.assign(value.auction_coverage, {
+      capture_status: 'not_observed',
+      issues: ['correlation_unavailable', 'external_client_side_unobservable'],
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(true);
+    Object.assign(value.auction_coverage, {
+      issues: ['external_client_side_unobservable', 'correlation_unavailable'],
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+    Object.assign(value.auction_coverage, {
+      issues: ['record_evicted', 'record_evicted'],
+      capture_status: 'unavailable',
+    });
+    expect(validateTraceReport(value, TRACE_ORIGIN, TRACE_NOW)).toBe(false);
+  });
+  it('rejects accessors and throwing proxies without invoking user serialization', () => {
+    let calls = 0;
+    const value = projectedGptFixture();
+    Object.defineProperty(value.slots[0], 'binding', {
+      enumerable: true,
+      get() {
+        calls += 1;
+        throw new Error('should not call');
+      },
+    });
+    expect(valid(value)).toBe(false);
+    expect(calls).toBe(0);
+    expect(
+      valid(
+        new Proxy(
+          {},
+          {
+            getPrototypeOf() {
+              throw new Error('should reject');
+            },
+          }
+        )
+      )
+    ).toBe(false);
+    expect(
+      validateTraceStoredReport(
+        new Proxy(
+          {},
+          {
+            ownKeys() {
+              throw new Error('should reject');
+            },
+          }
+        ),
+        TRACE_ORIGIN,
+        TRACE_NOW
+      )
+    ).toBe(false);
+  });
+});
diff --git a/crates/trusted-server-js/lib/test/trace/viewer.test.ts b/crates/trusted-server-js/lib/test/trace/viewer.test.ts
new file mode 100644
index 000000000..6c2223110
--- /dev/null
+++ b/crates/trusted-server-js/lib/test/trace/viewer.test.ts
@@ -0,0 +1,382 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+import { mountTraceViewer } from '../../src/trace/report-view';
+import { TRACE_REPORT_STORAGE_KEY } from '../../src/trace/storage';
+import { formatTraceReport } from '../../src/trace/export';
+import type {
+  copyTraceReport,
+  downloadTraceReport,
+  shareTraceReport,
+} from '../../src/trace/export';
+
+import { reportFixture, TRACE_NOW, TRACE_ORIGIN, AUCTION_TOKEN, SLOT_TOKEN } from './fixtures';
+
+function page() {
+  document.body.innerHTML =
+    '
Trusted Server ad diagnostics
Setup request
';
+  document.getElementById('trace-request-context')!.textContent = JSON.stringify({
+    ...reportFixture().request_context,
+    network: { edge_hostname: 'setup-only.example.com' },
+  });
+}
+function setup(value: unknown = reportFixture()) {
+  let serialized: string | null =
+    value === null ? null : JSON.stringify({ stored_at_ms: TRACE_NOW, report: value });
+  const storage = {
+    getItem: vi.fn(() => serialized),
+    setItem: vi.fn(),
+    removeItem: vi.fn((_key: string) => {
+      serialized = null;
+    }),
+  };
+  const copy = vi.fn(async () => ({ status: 'copied' }));
+  const download = vi.fn(() => ({ status: 'downloaded' }));
+  const share = vi.fn(async () => ({ status: 'shared' }));
+  const options = {
+    storage,
+    now: () => TRACE_NOW,
+    origin: TRACE_ORIGIN,
+    confirm: vi.fn(() => true),
+    copy,
+    download,
+    share,
+  };
+  return { storage, options, copy, download, share };
+}
+function button(label: string): HTMLButtonElement {
+  const result = Array.from(document.querySelectorAll('button')).find(
+    (candidate) => candidate.textContent === label
+  );
+  if (!result) throw new Error(`should find ${label} control`);
+  return result;
+}
+const request = vi.fn();
+beforeEach(() => {
+  page();
+  request.mockReset();
+  vi.stubGlobal('fetch', request);
+});
+afterEach(() => {
+  vi.unstubAllGlobals();
+  vi.restoreAllMocks();
+  document.body.replaceChildren();
+});
+
+describe('consolidated trace report viewer', () => {
+  it('keeps setup read-only when no report exists', () => {
+    const fixture = setup(null);
+    mountTraceViewer(document, fixture.options);
+    expect(document.getElementById('trace-report')).toBeNull();
+    expect(document.body.textContent).toContain('No saved report');
+    expect(document.body.textContent).toContain('setup-only.example.com');
+    expect(request).not.toHaveBeenCalled();
+    expect(fixture.storage.setItem).not.toHaveBeenCalled();
+  });
+  it.each(['expired', 'hostile', 'unsupported', 'wrong-origin'])(
+    'ignores and removes a %s report with actionable reproduction guidance',
+    (reason) => {
+      const report = reportFixture();
+      if (reason === 'hostile')
+        Object.assign(report, { private_value: '' });
+      if (reason === 'unsupported') report.schema_version = 2;
+      if (reason === 'wrong-origin')
+        report.gpt_diagnostics.page.origin = 'https://other.example.com';
+      const fixture = setup(report);
+      if (reason === 'expired') fixture.options.now = () => TRACE_NOW + 15 * 60 * 1000 + 1;
+      fixture.storage.removeItem.mockImplementation(() => {
+        throw new Error('private-storage-error');
+      });
+      mountTraceViewer(document, fixture.options);
+      expect(document.getElementById('trace-report')).toBeNull();
+      expect(document.body.textContent).toContain('reload once');
+      expect(document.body.textContent).not.toContain('private');
+      expect(fixture.storage.removeItem).toHaveBeenCalledWith(TRACE_REPORT_STORAGE_KEY);
+      expect(request).not.toHaveBeenCalled();
+    }
+  );
+  it('renders all sections with provenance and keeps setup facts out of publisher facts', () => {
+    const fixture = setup();
+    mountTraceViewer(document, fixture.options);
+    const report = document.getElementById('trace-report');
+    expect(report?.textContent).toContain('Browser-carried, unverified diagnostic data');
+    for (const title of [
+      'Report summary',
+      'Publisher request',
+      'Cookie health',
+      'Server auctions',
+      'GPT delivery and creative rendering',
+      'Coverage and ambiguity',
+      'Export',
+    ])
+      expect(report?.textContent).toContain(title);
+    expect(report?.textContent).toContain('Browser observed');
+    expect(report?.textContent).toContain('Correlation unknown');
+    expect(report?.textContent).toContain('0 × 0');
+    expect(report?.textContent).toContain('Unavailable');
+    expect(report?.textContent).not.toContain('setup-only.example.com');
+    expect(document.body.textContent).toContain('setup-only.example.com');
+    expect(report?.querySelector('style, [style], script, iframe, img')).toBeNull();
+    expect(request).not.toHaveBeenCalled();
+  });
+  it('displays all four ambiguous cookie rows as unavailable for reliable inspection', () => {
+    const report = reportFixture();
+    for (const name of ['ts_ec', 'ts_eids', 'ts_tester', 'diagnostics_session'])
+      Object.assign(report.request_context.cookies, {
+        [name]: { source: 'request', state: 'unavailable', detail: 'runtime_header_ambiguous' },
+      });
+    const fixture = setup(report);
+    mountTraceViewer(document, fixture.options);
+    const text = document.getElementById('trace-report-cookies')?.textContent ?? '';
+    expect(text.match(/could not be reliably inspected/g)).toHaveLength(4);
+    expect(text).not.toContain('invalid UTF');
+    expect(text).not.toContain('Not present');
+  });
+  it('renders exact joined server evidence and retains plain refresh path labels without inferring a winner', () => {
+    const report = {
+      ...reportFixture(),
+      server_auctions: [
+        {
+          schema_version: 1,
+          diagnostic_auction_id: AUCTION_TOKEN,
+          source: 'initial_navigation_ssat',
+          terminal_status: 'completed',
+          total_time_ms: 10,
+          provider_calls: [
+            { provider_number: 1, role: 'bidder', status: 'no_bid', returned_bid_count: 0 },
+          ],
+          slots: [
+            {
+              slot_number: 1,
+              slot_ref: SLOT_TOKEN,
+              requested_sizes: [[300, 250]],
+              returned_bid_count: 0,
+              candidate: 'no_candidate',
+            },
+          ],
+          truncation: { omitted_provider_calls: 0, omitted_slots: 0, omitted_nested_values: 0 },
+          coverage: { provider_to_slot_no_bid: 'unavailable' },
+        },
+      ],
+      slot_correlations: [
+        {
+          schema_version: 1,
+          diagnostic_auction_id: AUCTION_TOKEN,
+          slot_ref: SLOT_TOKEN,
+          runtime_slot_number: 1,
+          request_number: 1,
+        },
+      ],
+      auction_coverage: { capture_status: 'complete', issues: [] },
+    };
+    report.gpt_diagnostics.slots[0].requests[0].requestPath = 'prebid_refresh';
+    const fixture = setup(report);
+    mountTraceViewer(document, fixture.options);
+    const text = document.getElementById('trace-report')?.textContent;
+    expect(text).toContain('Initial-page server auction (SSAT)');
+    expect(text).toContain(
+      'Produced by Trusted Server; copied through an untrusted browser snapshot'
+    );
+    expect(text).toContain('Auction-wide provider status; per-slot no-bid reason unavailable');
+    expect(text).toContain('Unavailable in v1');
+    expect(text).toContain('Browser refresh observed; winner not determined');
+    expect(text).toContain('Trusted Server creative rendered');
+    expect(text).toContain('must not be summed as unique bids');
+    expect(text).toContain('Correlation record 1');
+    expect(text).toContain('Browser observed correlation');
+    expect(text).not.toContain('client-side auction won');
+  });
+  it('uses one immutable public model for explicit Copy, Share and Download and retains it after failures', async () => {
+    const fixture = setup();
+    fixture.copy.mockResolvedValue({ status: 'failed' });
+    fixture.share.mockResolvedValue({ status: 'unsupported' });
+    mountTraceViewer(document, fixture.options);
+    expect(fixture.copy).not.toHaveBeenCalled();
+    expect(fixture.share).not.toHaveBeenCalled();
+    expect(fixture.download).not.toHaveBeenCalled();
+    button('Copy').click();
+    await vi.waitFor(() =>
+      expect(document.getElementById('trace-export-status')?.textContent).toContain(
+        'Copy could not'
+      )
+    );
+    button('Share').click();
+    await vi.waitFor(() =>
+      expect(document.getElementById('trace-export-status')?.textContent).toContain(
+        'File sharing is unavailable'
+      )
+    );
+    button('Download').click();
+    const copied = fixture.copy.mock.calls[0];
+    const shared = fixture.share.mock.calls[0];
+    const downloaded = fixture.download.mock.calls[0];
+    expect(copied).toEqual(shared);
+    expect(copied).toEqual(downloaded);
+    expect(formatTraceReport(...copied)).toBe(JSON.stringify(reportFixture(), null, 2));
+    expect(Object.isFrozen(copied[0])).toBe(true);
+    expect(document.getElementById('trace-report')).not.toBeNull();
+    expect(document.body.textContent).toContain('selected app receives this JSON');
+  });
+  it('requires confirmation before any cleanup mutation', () => {
+    const fixture = setup();
+    fixture.options.confirm.mockReturnValue(false);
+    mountTraceViewer(document, fixture.options);
+    button('Clear report and end tracing').click();
+    expect(fixture.storage.removeItem).not.toHaveBeenCalled();
+    expect(request).not.toHaveBeenCalled();
+    expect(document.getElementById('trace-report')).not.toBeNull();
+  });
+  it('deletes the local report independently without confirmation or any server request', () => {
+    const fixture = setup();
+    mountTraceViewer(document, fixture.options);
+    button('Delete local report').click();
+    expect(fixture.options.confirm).not.toHaveBeenCalled();
+    expect(fixture.storage.removeItem).toHaveBeenCalledWith(TRACE_REPORT_STORAGE_KEY);
+    expect(request).not.toHaveBeenCalled();
+    expect(document.getElementById('trace-report')).toBeNull();
+    expect(document.body.textContent).toContain('Local report deleted');
+  });
+  for (const local of ['deleted', 'failed'] as const)
+    for (const mutation of ['requested', 'failed'] as const)
+      for (const observation of ['inactive', 'active', 'failed'] as const) {
+        it(`separates local ${local}, end ${mutation}, observed ${observation}`, async () => {
+          const fixture = setup();
+          if (local === 'failed')
+            fixture.storage.removeItem.mockImplementation(() => {
+              throw new Error('private-deletion-error');
+            });
+          request.mockResolvedValueOnce(
+            new Response('{}', { status: mutation === 'requested' ? 200 : 500 })
+          );
+          if (observation === 'failed')
+            request.mockRejectedValueOnce(new Error('private-state-error'));
+          else
+            request.mockResolvedValueOnce(
+              new Response(JSON.stringify({ observed_active: observation === 'active' }))
+            );
+          mountTraceViewer(document, fixture.options);
+          button('Clear report and end tracing').click();
+          await vi.waitFor(() => expect(request).toHaveBeenCalledTimes(2));
+          await vi.waitFor(() =>
+            expect(document.getElementById('trace-cleanup-server-status')?.textContent).toContain(
+              mutation === 'requested' && observation === 'inactive'
+                ? 'Tracing is off'
+                : 'unconfirmed'
+            )
+          );
+          expect(document.getElementById('trace-report') === null).toBe(local === 'deleted');
+          expect(document.getElementById('trace-cleanup-local-status')?.textContent).toContain(
+            local === 'deleted' ? 'Local report deleted' : 'deletion failed'
+          );
+          expect(request.mock.calls.map(([path]) => path)).toEqual([
+            '/_ts/trace/end',
+            '/_ts/trace/state',
+          ]);
+          expect(document.body.textContent).not.toContain('private-');
+          expect(document.body.textContent).not.toContain('cookie absent');
+          if (local === 'failed') expect(button('Delete local report').disabled).toBe(false);
+          if (!(mutation === 'requested' && observation === 'inactive'))
+            expect(button('Retry end tracing').hidden).toBe(false);
+        });
+      }
+  it('ignores removed controls and late async results after viewer destruction', async () => {
+    const fixture = setup();
+    let resolveCopy: ((result: Awaited>) => void) | undefined;
+    fixture.copy.mockImplementation(
+      () =>
+        new Promise((resolve) => {
+          resolveCopy = resolve;
+        })
+    );
+    const viewer = mountTraceViewer(document, fixture.options);
+    const copy = button('Copy');
+    const enable = button('Enable tracing');
+    copy.click();
+    viewer.destroy();
+    copy.click();
+    enable.click();
+    resolveCopy?.({ status: 'copied' });
+    await Promise.resolve();
+    expect(fixture.copy).toHaveBeenCalledTimes(1);
+    expect(request).not.toHaveBeenCalled();
+    expect(document.getElementById('trace-export-status')?.textContent).not.toBe('Copied JSON.');
+  });
+  it('allows separate local deletion and server retry after both earlier cleanup steps fail', async () => {
+    const fixture = setup();
+    fixture.storage.removeItem.mockImplementationOnce(() => {
+      throw new Error('private-storage-error');
+    });
+    request.mockResolvedValueOnce(new Response('{}', { status: 500 }));
+    request.mockResolvedValueOnce(new Response('{"observed_active":true}'));
+    mountTraceViewer(document, fixture.options);
+    button('Clear report and end tracing').click();
+    await vi.waitFor(() => expect(button('Retry end tracing').hidden).toBe(false));
+    button('Delete local report').click();
+    expect(document.getElementById('trace-report')).toBeNull();
+    request.mockResolvedValueOnce(new Response('{}'));
+    request.mockResolvedValueOnce(new Response('{"observed_active":false}'));
+    button('Retry end tracing').click();
+    await vi.waitFor(() =>
+      expect(document.getElementById('trace-cleanup-server-status')?.textContent).toContain(
+        'Tracing is off'
+      )
+    );
+    expect(fixture.options.confirm).toHaveBeenCalledTimes(1);
+    expect(fixture.storage.removeItem).toHaveBeenCalledTimes(2);
+    expect(request).toHaveBeenCalledTimes(4);
+    expect(document.getElementById('trace-cleanup-local-status')?.textContent).toContain(
+      'Local report deleted'
+    );
+  });
+  it('still deletes the report and attempts state verification when both network steps are offline', async () => {
+    const fixture = setup();
+    request.mockRejectedValueOnce(new Error('private-post-error'));
+    request.mockRejectedValueOnce(new Error('private-get-error'));
+    mountTraceViewer(document, fixture.options);
+    button('Clear report and end tracing').click();
+    await vi.waitFor(() =>
+      expect(document.getElementById('trace-cleanup-server-status')?.textContent).toContain(
+        'may remain active'
+      )
+    );
+    expect(request).toHaveBeenCalledTimes(2);
+    expect(document.getElementById('trace-report')).toBeNull();
+    expect(document.body.textContent).not.toContain('private-');
+  });
+  it('keeps local deletion available while the independent end request is pending', async () => {
+    const fixture = setup();
+    fixture.storage.removeItem.mockImplementationOnce(() => {
+      throw new Error('private-storage-error');
+    });
+    let resolvePost: ((value: Response) => void) | undefined;
+    request.mockImplementationOnce(
+      () =>
+        new Promise((resolve) => {
+          resolvePost = resolve;
+        })
+    );
+    request.mockResolvedValueOnce(new Response('{"observed_active":false}'));
+    mountTraceViewer(document, fixture.options);
+    button('Clear report and end tracing').click();
+    expect(button('Delete local report').disabled).toBe(false);
+    button('Delete local report').click();
+    expect(document.getElementById('trace-report')).toBeNull();
+    resolvePost?.(new Response('{}'));
+    await vi.waitFor(() =>
+      expect(document.getElementById('trace-cleanup-server-status')?.textContent).toContain(
+        'Tracing is off'
+      )
+    );
+    expect(request).toHaveBeenCalledTimes(2);
+  });
+  it('renders an allowed hostile-looking network value as text without executable content or inline style', () => {
+    const report = reportFixture();
+    Object.assign(report.request_context.network, { tls_cipher: '' });
+    const fixture = setup(report);
+    mountTraceViewer(document, fixture.options);
+    expect(document.getElementById('trace-report')?.textContent).toContain(
+      ''
+    );
+    expect(document.querySelector('img, iframe, script, [style]')).toBeNull();
+    expect(request).not.toHaveBeenCalled();
+  });
+});
diff --git a/crates/trusted-server-js/lib/trace-asset-sources.mjs b/crates/trusted-server-js/lib/trace-asset-sources.mjs
new file mode 100644
index 000000000..c9d399894
--- /dev/null
+++ b/crates/trusted-server-js/lib/trace-asset-sources.mjs
@@ -0,0 +1,27 @@
+import { createHash } from 'node:crypto';
+import fs from 'node:fs';
+import path from 'node:path';
+
+/** Hashes the explicit trace build inputs to reject stale assets when building is skipped. */
+export function traceSourceDigest(library) {
+  const sources = ['build-all.mjs', 'trace-asset-sources.mjs', 'package-lock.json'];
+  function discover(directory) {
+    for (const entry of fs.readdirSync(path.join(library, directory), { withFileTypes: true })) {
+      const relative = `${directory}/${entry.name}`;
+      if (entry.isDirectory()) discover(relative);
+      else if (entry.isFile() && /\.(?:ts|css)$/.test(entry.name)) sources.push(relative);
+    }
+  }
+  discover('src/trace');
+  const hash = createHash('sha256');
+  for (const relative of sources.sort((left, right) =>
+    Buffer.compare(Buffer.from(left), Buffer.from(right))
+  )) {
+    hash
+      .update(relative)
+      .update('\0')
+      .update(fs.readFileSync(path.join(library, relative)))
+      .update('\0');
+  }
+  return hash.digest('hex');
+}
diff --git a/crates/trusted-server-js/lib/trace-assets-manifest.json b/crates/trusted-server-js/lib/trace-assets-manifest.json
new file mode 100644
index 000000000..8a660a3ad
--- /dev/null
+++ b/crates/trusted-server-js/lib/trace-assets-manifest.json
@@ -0,0 +1,16 @@
+{
+  "schema_version": 1,
+  "source_sha256": "8c7b4768299d57175576b663cbb57f54d5bce521af1794676ca8e850c623fb7f",
+  "assets": [
+    {
+      "path": "/_ts/trace/assets/v1.js",
+      "file": "v1.js",
+      "sha256": "9ddbde76323a2af09a2c20019f550f9bb917eb0c5ac76e0471852ade6a526bc5"
+    },
+    {
+      "path": "/_ts/trace/assets/v1.css",
+      "file": "v1.css",
+      "sha256": "b3a7f77b5190c36919cd9a151dabf055784e5757e22a26e64b56d48b0aafb980"
+    }
+  ]
+}
diff --git a/crates/trusted-server-js/lib/trace-assets/v1.css b/crates/trusted-server-js/lib/trace-assets/v1.css
new file mode 100644
index 000000000..22519eaf4
--- /dev/null
+++ b/crates/trusted-server-js/lib/trace-assets/v1.css
@@ -0,0 +1 @@
+:root{color-scheme:light dark;font:1rem/1.5 system-ui,sans-serif;background:Canvas;color:CanvasText}*{box-sizing:border-box}body{margin:0;padding:max(1rem,env(safe-area-inset-top)) max(1rem,env(safe-area-inset-right)) max(1rem,env(safe-area-inset-bottom)) max(1rem,env(safe-area-inset-left));overflow-wrap:anywhere}main{max-width:60rem;margin:auto}h1{font-size:1.6rem;line-height:1.3}h2{font-size:1.25rem}section{margin-block:1.5rem}dt{font-weight:650;margin-top:.75rem}dd{margin-inline:0}.controls{display:flex;flex-wrap:wrap;gap:.75rem}button,.button{min-width:44px;min-height:44px;padding:.75rem 1rem;font:inherit;color:inherit;background:Canvas;border:2px solid currentColor;border-radius:.4rem;cursor:pointer;white-space:normal}.primary{background:#17478b;color:#fff;border-color:#17478b}:focus-visible{outline:3px solid #ca6800;outline-offset:3px}button:disabled{cursor:wait;opacity:.65}[role=status]{min-height:3rem}pre{white-space:pre-wrap;overflow-wrap:anywhere}table{width:100%;table-layout:fixed;border-collapse:collapse}th,td{padding:.5rem;text-align:start;vertical-align:top;overflow-wrap:anywhere}[hidden]{display:none}
diff --git a/crates/trusted-server-js/lib/trace-assets/v1.js b/crates/trusted-server-js/lib/trace-assets/v1.js
new file mode 100644
index 000000000..599cfdad1
--- /dev/null
+++ b/crates/trusted-server-js/lib/trace-assets/v1.js
@@ -0,0 +1 @@
+(function(){"use strict";function C(e,t){return Object.prototype.hasOwnProperty.call(e,t)}function m(e){if(e===null||typeof e!="object"||Array.isArray(e))return!1;const t=Object.getPrototypeOf(e);return t!==Object.prototype&&t!==null?!1:Reflect.ownKeys(e).every(n=>{if(typeof n!="string")return!1;const r=Object.getOwnPropertyDescriptor(e,n);return r?.enumerable===!0&&C(r,"value")})}function g(e,t,n=[]){return t.every(r=>C(e,r))&&Object.keys(e).every(r=>t.includes(r)||n.includes(r))}function $(e,t=128){if(typeof e!="string")return!1;for(const n of e){const r=n.codePointAt(0);if(r<=31||r>=127&&r<=159||r>=55296&&r<=57343||r===1564||r===8206||r===8207||r>=8234&&r<=8238||r>=8294&&r<=8297)return!1}return new TextEncoder().encode(e).length<=t}function H(e){if(typeof e!="string")return!1;const t=/^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,9})?Z$/.exec(e);if(!t)return!1;const[,n,r,s,i,c,l]=t,o=Number(n),S=Number(r),p=Number(s),d=[31,o%4===0&&(o%100!==0||o%400===0)?29:28,31,30,31,30,31,31,30,31,30,31];return S>=1&&S<=12&&p>=1&&p<=d[S-1]&&Number(i)<=23&&Number(c)<=59&&Number(l)<=59}function T(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isSafeInteger(e)&&e>=0&&e<=t}function Re(e){if(!$(e))return!1;const t=/^((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.0\/24$/.exec(e);if(t)return t.slice(1).every(s=>Number(s)<=255);if(!e.endsWith("::/48"))return!1;const n=e.slice(0,-3),r=n.slice(0,-2);if(r!==""&&!/^[\da-f]{1,4}(?::[\da-f]{1,4}){0,2}$/.test(r))return!1;try{return new URL(`http://[${n}]/`).hostname===`[${n}]`}catch{return!1}}function F(e,t){if(!m(e)||e.source!=="request")return!1;if(e.state==="absent")return g(e,["source","state"]);if(!g(e,["source","state","detail"])||typeof e.detail!="string")return!1;switch(e.state){case"present_valid":return e.detail===t;case"present_invalid":return["malformed","oversized","unsupported_value"].includes(e.detail);case"duplicate":return e.detail==="multiple_values";case"unavailable":return["header_too_large","header_not_utf8","runtime_header_ambiguous"].includes(e.detail);default:return!1}}function ne(e){try{return we(e)}catch{return!1}}function we(e){if(!m(e)||!g(e,["schema_version","captured_at","network","cookies"])||e.schema_version!==1||!H(e.captured_at)||!m(e.network)||!m(e.cookies))return!1;const t=e.network,n={country:2,region:32,http_version:32,tls_protocol:32,tls_cipher:32,edge_hostname:128,edge_region:128,edge_pop:32};if(!g(t,[],["masked_client_ip","asn",...Object.keys(n)])||C(t,"masked_client_ip")&&!Re(t.masked_client_ip)||C(t,"asn")&&!T(t.asn,4294967295))return!1;for(const[s,i]of Object.entries(n))if(C(t,s)&&!$(t[s],i))return!1;if(typeof t.country=="string"&&!/^[\x20-\x7e]{0,2}$/.test(t.country))return!1;const r=e.cookies;return g(r,["ts_ec","ts_eids","ts_tester","diagnostics_session"])&&F(r.ts_ec,"valid_ec_format")&&F(r.ts_eids,"valid_eids_format")&&F(r.ts_tester,"valid_tester_value")&&F(r.diagnostics_session,"valid_diagnostics_value")}function x(e,t){if(!Array.isArray(e)||Object.getPrototypeOf(e)!==Array.prototype)return;const n=Object.getOwnPropertyDescriptor(e,"length")?.value;if(!T(n,t))return;const r=Reflect.ownKeys(e);if(r.length!==n+1||!r.every(i=>{if(i==="length")return!0;if(typeof i!="string"||!/^(?:0|[1-9]\d*)$/.test(i)||Number(i)>=n)return!1;const c=Object.getOwnPropertyDescriptor(e,i);return c?.enumerable===!0&&C(c,"value")}))return;const s=[];for(let i=0;i(s+=r.encode(o).length,s<=n),l=(o,S)=>{if(o===null||typeof o=="boolean"||typeof o=="number")return(typeof o!="number"||Number.isFinite(o))&&c(JSON.stringify(o))?{value:o}:void 0;if(typeof o=="string")return $(o,n)&&c(JSON.stringify(o))?{value:o}:void 0;if(S>t||typeof o!="object"||i.has(o))return;i.add(o);let p=!0,b;if(Array.isArray(o)){const d=[];b=d;const u=x(o,n);if(!u||!c("["))p=!1;else for(let a=0;aT(r,1e5)&&(t||r>0))}function ce(e,t,n){return!C(e,t)||T(e[t],n)}function De(e){return m(e)&&g(e,["provider_number","role","status","returned_bid_count"],["response_time_ms"])&&T(e.provider_number,65535)&&e.provider_number>0&&E(e.role,Le)&&E(e.status,Ue)&&T(e.returned_bid_count,65535)&&ce(e,"response_time_ms",4294967295)}function Ve(e){return m(e)&&g(e,["slot_number","slot_ref","requested_sizes","returned_bid_count","candidate"],["selected_creative_size"])&&T(e.slot_number,65535)&&e.slot_number>0&&ae(e.slot_ref)&&T(e.returned_bid_count,65535)&&E(e.candidate,Be)&&X(e.requested_sizes,16,t=>D(t))&&(!C(e,"selected_creative_size")||D(e.selected_creative_size))}function Fe(e){return m(e)&&g(e,["schema_version","diagnostic_auction_id","source","terminal_status","provider_calls","slots","truncation","coverage"],["terminal_reason","total_time_ms"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&E(e.source,Me)&&E(e.terminal_status,Pe)&&(!C(e,"terminal_reason")||E(e.terminal_reason,je))&&ce(e,"total_time_ms",4294967295)&&X(e.provider_calls,16,De)&&X(e.slots,64,Ve)&&m(e.truncation)&&g(e.truncation,["omitted_provider_calls","omitted_slots","omitted_nested_values"])&&Object.values(e.truncation).every(t=>T(t,65535))&&m(e.coverage)&&g(e.coverage,["provider_to_slot_no_bid"])&&e.coverage.provider_to_slot_no_bid==="unavailable"}function X(e,t,n){const r=x(e,t);return r!==void 0&&r.every(n)}function Ge(e){try{const t=I(e);return t!==void 0&&Fe(t.value)}catch{return!1}}function Ke(e){try{return m(e)&&g(e,["schema_version","diagnostic_auction_id","slot_ref","runtime_slot_number","request_number"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&ae(e.slot_ref)&&T(e.runtime_slot_number)&&e.runtime_slot_number>0&&T(e.request_number)&&e.request_number>0}catch{return!1}}function Ye(e){const t=I(e);return t!==void 0&&Ke(t.value)}const Je=["requestNumber","requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","durations","isEmpty","requestedSlotSizes","size","observedSlotSize","isBackfill","slotContentChanged","incompleteSequence","responseClass","requestPath","requestIntentId","trustedServerAuctionId","opportunityToRequestMs","replacedRequestNumber","previousRenderToRequestMs","creativeChanged","loadObservedBeforeRender","trustedServerOpportunity","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs","trustedServerCreativeFailures","delivery"],He=["requestToResponseMs","responseToRenderMs","requestToRenderMs","renderToLoadMs","renderToViewableMs"],We=["requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","opportunityToRequestMs","previousRenderToRequestMs","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs"],Xe=["isEmpty","isBackfill","slotContentChanged","creativeChanged","loadObservedBeforeRender"],Ze=["omitted_server_auctions","omitted_slot_correlations","omitted_request_cycles","omitted_callback_issues","omitted_attribution_issues","omitted_nested_values"];function de(e,t,n){const r=I(e);if(!(!r||Q(t)!==t||!M(n)||!le(r.value,t,n)))return Z(r.value),r.value}function Qe(e,t,n){const r=I(e,11);if(!(!r||!at(r.value,t,n)))return Z(r.value),r.value}function Z(e){typeof e!="object"||e===null||(Object.values(e).forEach(Z),Object.freeze(e))}function et(e,t,n){try{const r=I(e);if(!r)return"invalid_report";if(e=r.value,pe(e,t,n))return;if(!m(e))return"invalid_report";if(C(e,"schema_version")&&e.schema_version!==1)return"unsupported_report_version";const s=e.gpt_diagnostics;if(m(s)){if(C(s,"schema_version")&&s.schema_version!==1)return"unsupported_gpt_version";if(C(s,"source_schema_version")&&s.source_schema_version!==1)return"unsupported_gpt_source_version"}return x(e.server_auctions,16)?.some(l=>m(l)&&C(l,"schema_version")&&l.schema_version!==1)?"unsupported_auction_version":x(e.slot_correlations,128)?.some(l=>m(l)&&C(l,"schema_version")&&l.schema_version!==1)?"unsupported_correlation_version":"invalid_report"}catch{return"invalid_report"}}function M(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isFinite(e)&&e>=0&&e<=t}function Q(e){if(!$(e,255)||!/^https?:\/\//i.test(e))return;const t=e.slice(e.indexOf("://")+3);if(!t||/[\s/@?#,\\]/.test(t))return;const n=t.startsWith("[")?t.slice(t.indexOf("]")+1):t.includes(":")?t.slice(t.lastIndexOf(":")):"";if(!(n!==""&&(!/^:\d+$/.test(n)||Number(n.slice(1))>65535)))try{const r=new URL(e);return!["http:","https:"].includes(r.protocol)||!r.hostname||r.username||r.password||r.search||r.hash||r.pathname!=="/"?void 0:r.origin}catch{return}}function P(e,t,n){const r=x(e,t);return r!==void 0&&r.every(n)}function A(e,t,n){return!C(e,t)||n(e[t])}function tt(e){return!m(e)||!g(e,["requestNumber","durations","incompleteSequence"],Je)||!T(e.requestNumber)||typeof e.incompleteSequence!="boolean"||!m(e.durations)||!g(e.durations,[],He)||!Object.values(e.durations).every(t=>M(t))||!We.every(t=>A(e,t,M))||!Xe.every(t=>A(e,t,n=>typeof n=="boolean"))?!1:["requestIntentId","replacedRequestNumber"].every(t=>A(e,t,T))&&A(e,"trustedServerAuctionId",W)&&A(e,"requestedSlotSizes",t=>P(t,16,n=>D(n)))&&A(e,"size",t=>D(t))&&A(e,"observedSlotSize",t=>D(t,!0))&&A(e,"responseClass",t=>E(t,ke))&&A(e,"requestPath",t=>E(t,Oe))&&A(e,"trustedServerOpportunity",t=>E(t,Ne))&&A(e,"delivery",t=>E(t,Ie))&&A(e,"trustedServerCreativeFailures",t=>P(t,16,n=>E(n,xe)))}function rt(e){return!m(e)||!g(e,["runtimeSlotNumber","binding","requests"],["currentVisibilityPercentage","maximumVisibilityPercentage"])||!T(e.runtimeSlotNumber)||!m(e.binding)||!g(e.binding,["status"],["reason"])||!E(e.binding.status,["bound","unbound","ambiguous"])||!A(e.binding,"reason",t=>E(t,Ce))?!1:A(e,"currentVisibilityPercentage",t=>M(t,100))&&A(e,"maximumVisibilityPercentage",t=>M(t,100))&&P(e.requests,10,tt)}function nt(e){return m(e)&&g(e,["kind","runtimeSlotNumber","timestampMs","disposition","reason"])&&E(e.kind,ie)&&T(e.runtimeSlotNumber)&&M(e.timestampMs)&&E(e.disposition,["matched","unmatched","ambiguous"])&&E(e.reason,Ae)}function st(e){return m(e)&&g(e,["reason","timestampMs"],["runtimeSlotNumber"])&&E(e.reason,qe)&&M(e.timestampMs)&&A(e,"runtimeSlotNumber",T)}function it(e){return m(e)&&g(e,["observed","matched","unmatched","ambiguous"])&&Object.values(e).every(t=>T(t))}function ot(e,t){return!m(e)||!g(e,["schema_version","source_schema_version","capturedAt","page","slots","callbackIssues","coverage","metadata"],["attributionIssues"])||e.schema_version!==1||e.source_schema_version!==1||!H(e.capturedAt)||!m(e.page)||!g(e.page,["origin","pathname"])||Q(e.page.origin)!==t||e.page.pathname!=="/[redacted]"||!P(e.slots,64,rt)||!P(e.callbackIssues,128,nt)||!A(e,"attributionIssues",n=>P(n,128,st))||!m(e.coverage)||!g(e.coverage,ie)||!Object.values(e.coverage).every(it)?!1:m(e.metadata)&&g(e.metadata,["droppedCallbacks","evictedSlots","evictedRequestCycles"],["droppedAttributionIssues"])&&Object.values(e.metadata).every(n=>T(n))}function ue(e,t){if(!H(e))return!1;const n=Date.parse(e);return Number.isFinite(n)&&Math.abs(n-t)<=6e4}function le(e,t,n){if(!m(e)||!g(e,["schema_version","captured_at","request_context","server_auctions","slot_correlations","gpt_diagnostics","auction_coverage","truncation"])||e.schema_version!==1||!ue(e.captured_at,n)||!ne(e.request_context)||!P(e.server_auctions,16,Ge)||!P(e.slot_correlations,128,Ye)||!ot(e.gpt_diagnostics,t)||!ue(e.gpt_diagnostics.capturedAt,n)||!m(e.truncation)||!g(e.truncation,Ze)||!Object.values(e.truncation).every(p=>T(p,65535)))return!1;const r=e.auction_coverage;if(!m(r)||!g(r,["capture_status","issues"]))return!1;const s=x(r.issues,16);if(!s||!s.every(p=>E(p,oe)))return!1;let i=-1;for(const p of s){const b=oe.indexOf(p);if(b<=i)return!1;i=b}const c=x(e.server_auctions,16);if(!c)return!1;const l=x(e.slot_correlations,128);if(!l)return!1;for(const p of l)if(!m(p)||c.some(b=>m(b)&&b.source==="auction_api"&&b.diagnostic_auction_id===p.diagnostic_auction_id))return!1;const o=s.some(p=>["evidence_projection_failed","evidence_transport_failed","evidence_validation_failed","record_evicted"].includes(p)),S=c.length>0?o?"partial":"complete":o?"unavailable":"not_observed";return r.capture_status===S&&Ee(e,10)!==void 0}function pe(e,t,n){try{const r=I(e);return r!==void 0&&Q(t)===t&&M(n)&&le(r.value,t,n)}catch{return!1}}function at(e,t,n){try{const r=I(e,11);if(!r)return!1;const s=r.value;return m(s)&&g(s,["stored_at_ms","report"])&&M(n)&&T(s.stored_at_ms)&&s.stored_at_ms-n<=6e4&&n-s.stored_at_ms<=9e5&&pe(s.report,t,s.stored_at_ms)}catch{return!1}}function ct(e){switch(e.requestPath){case"prebid_refresh":case"publisher_refresh":return"Browser refresh observed; winner not determined";case"competing":case"unattributed":return"Multiple or unknown delivery paths";case"trusted_server_direct":return"Trusted Server request path observed";default:return"Request path unavailable"}}const dt={initial_navigation_ssat:"Initial-page server auction (SSAT)",spa_page_bids:"Trusted Server page-refresh auction",auction_api:"Trusted Server auction API"};function ut(e,t,n){const r=de(e,t,n);if(!r)return;const s=r.gpt_diagnostics.slots.flatMap(l=>l.requests.map(o=>({runtimeSlotNumber:l.runtimeSlotNumber,cycle:o}))),i=r.server_auctions.flatMap(l=>l.slots.map(o=>({id:l.diagnostic_auction_id,slot:o}))),c=r.server_auctions.map(l=>{const o=l.diagnostic_auction_id,S=r.server_auctions.filter(b=>b.diagnostic_auction_id===o).length===1,p=l.slots.map(b=>{const d=Object.freeze({serverSlot:b,correlation:"unknown"});if(!S||l.source==="auction_api"||i.filter(y=>y.id===o&&y.slot.slot_ref===b.slot_ref).length!==1)return d;const u=r.slot_correlations.filter(y=>y.diagnostic_auction_id===o&&y.slot_ref===b.slot_ref);if(u.length!==1)return d;const a=u[0];if(r.slot_correlations.filter(y=>y.runtime_slot_number===a.runtime_slot_number&&y.request_number===a.request_number).length!==1)return d;const f=s.filter(y=>y.runtimeSlotNumber===a.runtime_slot_number&&y.cycle.requestNumber===a.request_number);if(f.length!==1||f[0].cycle.trustedServerAuctionId!==o)return d;const h=f[0].cycle,L=h.isEmpty===!1&&h.renderAtMs!==void 0&&h.trustedServerCreativeResponseAtMs!==void 0&&h.delivery==="trusted_server_response_sent";return Object.freeze({serverSlot:b,correlation:"matched",runtimeSlotNumber:a.runtime_slot_number,requestNumber:a.request_number,cycle:h,pathLabel:ct(h),creativeLabel:L?"Trusted Server creative rendered":"Participation unconfirmed"})});return Object.freeze({evidence:l,sourceLabel:dt[l.source],relativeMilestonesLabel:"Unavailable in v1",providerScopeLabel:"Auction-wide provider status; per-slot no-bid reason unavailable",slots:Object.freeze(p)})});return Object.freeze({auctions:Object.freeze(c)})}const fe="trusted-server-trace-v1.json";function ee(e,t,n){const r=de(e,t,n);return r?JSON.stringify(r,null,2):void 0}function lt(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};let s;try{s=URL.createObjectURL(new Blob([r],{type:"application/json"}))}catch{return{status:"failed"}}let i;try{return i=document.createElement("a"),i.href=s,i.download=fe,document.body.append(i),i.click(),{status:"downloaded"}}catch{return{status:"failed"}}finally{try{i?.remove()}catch{}window.setTimeout(()=>URL.revokeObjectURL(s),1e3)}}async function pt(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{return typeof navigator.clipboard?.writeText!="function"?{status:"unsupported"}:(await navigator.clipboard.writeText(r),{status:"copied"})}catch{return{status:"failed"}}}async function ft(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{if(typeof navigator.canShare!="function"||typeof navigator.share!="function")return{status:"unsupported"};const i={files:[new File([r],fe,{type:"application/json"})]};return navigator.canShare(i)?(await navigator.share(i),{status:"shared"}):{status:"unsupported"}}catch{return{status:"failed"}}}async function _t(e){return _e(e,!1)}async function bt(){return _e("end",!0)}async function _e(e,t){const n={mutation:"failed",observation:"not_attempted",confirmed:!1};if(e!=="enable"&&e!=="end")return n;let r="failed";try{(await fetch(`/_ts/trace/${e}`,{method:"POST",credentials:"same-origin",cache:"no-store",headers:{"X-TS-Trace-Action":e}})).ok&&(r="requested")}catch{}if(r==="failed"&&!t)return n;const s={mutation:r,observation:"failed",confirmed:!1};try{const i=await fetch("/_ts/trace/state",{method:"GET",credentials:"same-origin",cache:"no-store"});if(!i.ok)return s;const c=await i.json();if(c===null||typeof c!="object"||Array.isArray(c))return s;const l=Object.keys(c);if(l.length!==1||l[0]!=="observed_active")return s;const o=Object.getOwnPropertyDescriptor(c,"observed_active");if(!o||typeof o.value!="boolean")return s;const S=o.value;return{mutation:r,observation:S?"active":"inactive",confirmed:r==="requested"&&S===(e==="enable")}}catch{return s}}const mt="Return to the affected page, reload once, reproduce the problem, then select View trace results.",vt="Reopen the affected article on the exact same hostname and in this same tab, then reload once.",be="Tracing is on — cookie observed by server",te="Tracing is off — no valid diagnostics session observed";function G(e){switch(e.state){case"absent":return"Not present in this request";case"present_valid":return"Valid shape observed";case"present_invalid":return"Invalid shape observed";case"duplicate":return"Multiple values observed";case"unavailable":return e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":"Unavailable — runtime-visible cookie inspection failed"}}function q(e,t,n,r){const s=e.createElement("dt");s.textContent=n;const i=e.createElement("dd");i.textContent=r,t.append(s,i)}function gt(e){const t=e.getElementById("trace-request-context"),n=e.getElementById("trace-network-facts"),r=e.getElementById("trace-cookie-facts");if(!t||!n||!r)return;let s;try{s=JSON.parse(t.textContent??"")}catch{s=void 0}if(t.textContent="",n.replaceChildren(),r.replaceChildren(),!ne(s)){q(e,n,"Request facts","Unavailable"),q(e,r,"Cookie health","Unavailable");return}const i=s.network;q(e,n,"Approximate network identifier",i.masked_client_ip??"Unavailable"),q(e,n,"Country",i.country??"Unavailable"),q(e,n,"Region",i.region??"Unavailable"),q(e,n,"ASN",i.asn===void 0?"Unavailable":String(i.asn)),q(e,n,"HTTP version",i.http_version??"Unavailable"),q(e,n,"TLS protocol",i.tls_protocol??"Unavailable"),q(e,n,"TLS cipher",i.tls_cipher??"Unavailable"),q(e,n,"Edge hostname",i.edge_hostname??"Unavailable"),q(e,n,"Edge region",i.edge_region??"Unavailable"),q(e,n,"Edge POP",i.edge_pop??"Unavailable"),q(e,r,"Edge Cookie",G(s.cookies.ts_ec)),q(e,r,"External IDs",G(s.cookies.ts_eids)),q(e,r,"Tester",G(s.cookies.ts_tester)),q(e,r,"Diagnostics session",G(s.cookies.diagnostics_session))}function ht(e=document){gt(e);const t=e.getElementById("trace-session-state"),n=e.getElementById("trace-status"),r=e.getElementById("trace-enable"),s=e.getElementById("trace-end"),i=e.getElementById("trace-back");if(!t||!n||!(r instanceof HTMLButtonElement)||!(s instanceof HTMLButtonElement)||!(i instanceof HTMLButtonElement))return()=>{};t.textContent=t.dataset.observedActive==="true"?be:t.dataset.observedActive==="false"?te:"Tracing state unconfirmed";let c=!1,l=!1;const o=async d=>{if(!(l||c)){c=!0,r.disabled=s.disabled=!0,n.textContent=d==="enable"?"Verifying activation…":"Verifying deactivation…";try{const u=await _t(d);if(l)return;t.textContent=u.observation==="active"?be:u.observation==="inactive"?te:"Tracing state unconfirmed",u.confirmed?(n.textContent=d==="enable"?mt:te,i.classList.toggle("primary",d==="enable")):n.textContent=`${d==="enable"?"Activation":"Deactivation"} unconfirmed. Try again.`}finally{l||(r.disabled=s.disabled=!1),c=!1}}},S=()=>{o("enable")},p=()=>{o("end")},b=()=>{l||(window.history.length>1?window.history.back():n.textContent=vt)};return r.addEventListener("click",S),s.addEventListener("click",p),i.addEventListener("click",b),()=>{l=!0,r.removeEventListener("click",S),s.removeEventListener("click",p),i.removeEventListener("click",b)}}const re="trusted-server.trace.report.v1";function me(e){return e??window.sessionStorage}function yt(e,t,n){const r=I(e,11)?.value;return m(r)?et(r.report,t,n)??"invalid_report":"invalid_report"}function St(e,t,n){let r,s;try{r=me(n),s=r.getItem(re)}catch{return{status:"unavailable"}}if(s===null)return{status:"absent"};let i,c;try{typeof s=="string"&&new TextEncoder().encode(s).length<=512*1024&&(i=JSON.parse(s),c=Qe(i,e,t))}catch{}if(c)return{status:"ready",value:c};const l=yt(i,e,t);try{r.removeItem(re)}catch{}return{status:"rejected",reason:l}}function Tt(e){try{return me(e).removeItem(re),{status:"deleted"}}catch{return{status:"unavailable"}}}function R(e){return{no_bid:"No bid returned",no_candidate:"No candidate",selected:"Candidate selected",selected_unrenderable:"Selected candidate could not be rendered",trusted_server_direct:"Trusted Server request path observed",prebid_refresh:"Browser refresh observed; winner not determined",publisher_refresh:"Browser refresh observed; winner not determined",competing:"Multiple or unknown delivery paths",unattributed:"Multiple or unknown delivery paths",trusted_server_response_sent:"Trusted Server creative response sent",trusted_server_selected:"Trusted Server candidate selected; render unconfirmed",candidate_unconfirmed:"Candidate unconfirmed",not_observed:"Not observed",unknown:"Unknown",unavailable:"Unavailable"}[e]??e.replace(/([a-z])([A-Z])/g,"$1 $2").replace(/_/g," ").replace(/^./,n=>n.toUpperCase())}function Rt(e){return e===void 0?"Unavailable":typeof e=="boolean"?e?"Yes":"No":typeof e=="number"?String(e):typeof e=="string"?e:"Unavailable"}function K(e){return e.state==="absent"?"Not present in this request":e.state==="present_valid"?"Valid shape observed":e.state==="duplicate"?"Multiple values observed":e.state==="present_invalid"?e.detail==="oversized"?"Invalid shape — too long":e.detail==="unsupported_value"?"Invalid shape — unsupported value":"Invalid shape observed":e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":e.detail==="header_too_large"?"Unavailable — the visible cookie header was too large":"Unavailable — the visible cookie header was not valid text"}function _(e,t,n){const r=e.createElement(t);return n!==void 0&&(r.textContent=n),r}function j(e,t,n,r){const s=_(e,"section");return r&&(s.id=r),s.append(_(e,"h2",n)),t.append(s),s}function w(e,t,n){const r=_(e,"dl");for(const[s,i]of n)r.append(_(e,"dt",s),_(e,"dd",Rt(i)));t.append(r)}function V(e){return e===void 0?"Unavailable":e.length===0?"Not observed":e.map(([t,n])=>`${t} × ${n}`).join(", ")}function O(e){return e===void 0?"Unavailable":`${e} ms`}const wt={masked_client_ip:"Approximate network identifier",country:"Country",region:"Region",asn:"ASN",http_version:"HTTP version",tls_protocol:"TLS protocol",tls_cipher:"TLS cipher",edge_hostname:"Edge hostname",edge_region:"Edge region",edge_pop:"Edge POP"},Et={requestedAtMs:"Requested (browser clock)",responseAtMs:"Response received (browser clock)",renderAtMs:"Rendered (browser clock)",loadAtMs:"Loaded (browser clock)",viewableAtMs:"Viewable (browser clock)",isBackfill:"Backfill observed",slotContentChanged:"Slot content changed",incompleteSequence:"Incomplete sequence",responseClass:"GPT response class",requestIntentId:"Browser request intent number",opportunityToRequestMs:"Opportunity to request",replacedRequestNumber:"Replaced request number",previousRenderToRequestMs:"Previous render to request",creativeChanged:"Creative changed",loadObservedBeforeRender:"Load observed before render",trustedServerOpportunity:"Trusted Server candidate opportunity",trustedServerCreativeRequestAtMs:"Creative bridge request (browser clock)",trustedServerCreativeResponseAtMs:"Creative bridge response (browser clock)",delivery:"Creative delivery observation"};function Ct(e,t,n,r){const s=_(e,"article");s.id="trace-report",s.append(_(e,"h1","Trusted Server trace results"),_(e,"p","Browser-carried, unverified diagnostic data")),s.append(_(e,"p","This browser snapshot helps troubleshoot rendering. It is not proof of a server event, identity, or security incident."));const i=j(e,s,"Report summary");w(e,i,[["Captured at",t.captured_at],["Publisher origin",t.gpt_diagnostics.page.origin],["Server auctions retained",t.server_auctions.length],["GPT slots retained",t.gpt_diagnostics.slots.length]]);const c=j(e,s,"Publisher request");c.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),c.append(_(e,"p","These facts describe the traced publisher document. Masked identifiers are approximate and may still identify a network.")),w(e,c,[["Document request captured at",t.request_context.captured_at],...Object.entries(wt).map(([d,u])=>[u,t.request_context.network[d]])]);const l=j(e,s,"Cookie health","trace-report-cookies");l.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot. Only cookie shape visible in this request is inspected; values and browser attributes are excluded.")),w(e,l,[["Edge Cookie",K(t.request_context.cookies.ts_ec)],["External IDs",K(t.request_context.cookies.ts_eids)],["Tester",K(t.request_context.cookies.ts_tester)],["Diagnostics session",K(t.request_context.cookies.diagnostics_session)]]);const o=j(e,s,"Server auctions");o.append(_(e,"p","Returned bid counts may overlap between slots and must not be summed as unique bids."));const S=ut(t,n,r);t.server_auctions.length||o.append(_(e,"p","Not observed. This does not mean no server auction ran."));for(const[d,u]of(S?.auctions??[]).entries()){const a=_(e,"details");a.open=!0,a.append(_(e,"summary",`Auction ${d+1}: ${u.sourceLabel}`)),a.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),w(e,a,[["Terminal outcome",R(u.evidence.terminal_status)],["Terminal reason",u.evidence.terminal_reason===void 0?"Unavailable":R(u.evidence.terminal_reason)],["Server auction-local elapsed time",O(u.evidence.total_time_ms)],["Request-relative milestones",u.relativeMilestonesLabel]]),a.append(_(e,"p",u.providerScopeLabel));for(const f of u.evidence.provider_calls)w(e,a,[[`Provider call ${f.provider_number}`,R(f.role)],["Call outcome",R(f.status)],["Provider-local elapsed time",O(f.response_time_ms)],["Returned bid count",f.returned_bid_count]]);u.evidence.provider_calls.length||a.append(_(e,"p","Provider calls: Not observed"));for(const f of u.slots){const h=_(e,"details");h.open=!0,h.append(_(e,"summary",`Server slot ${f.serverSlot.slot_number}`)),w(e,h,[["Requested sizes",V(f.serverSlot.requested_sizes)],["Returned bid count",f.serverSlot.returned_bid_count],["Candidate",R(f.serverSlot.candidate)],["Selected creative size",f.serverSlot.selected_creative_size?V([f.serverSlot.selected_creative_size]):"Unavailable"],["Correlation",f.correlation==="matched"?`Matched GPT slot ${f.runtimeSlotNumber}, request ${f.requestNumber}`:"Correlation unknown"],["Browser request path",f.pathLabel??"Unknown"],["Creative participation",f.creativeLabel??"Participation unconfirmed"]]),a.append(h)}w(e,a,[["Provider calls omitted",u.evidence.truncation.omitted_provider_calls],["Server slots omitted",u.evidence.truncation.omitted_slots],["Nested values omitted",u.evidence.truncation.omitted_nested_values]]),o.append(a)}const p=j(e,s,"GPT delivery and creative rendering");p.append(_(e,"p","Browser observed. Server auction → GPT request/response → creative render/load/viewability are separate observations. A filled slot does not identify an auction winner.")),w(e,p,[["Browser snapshot captured at",t.gpt_diagnostics.capturedAt]]),t.gpt_diagnostics.slots.length||p.append(_(e,"p","Not observed"));for(const d of t.gpt_diagnostics.slots){const u=_(e,"details");u.open=!0,u.append(_(e,"summary",`GPT slot ${d.runtimeSlotNumber}`)),w(e,u,[["Binding",R(d.binding.status)],["Binding reason",d.binding.reason===void 0?"Unavailable":R(d.binding.reason)],["Current visibility percentage",d.currentVisibilityPercentage],["Maximum visibility percentage",d.maximumVisibilityPercentage]]),d.requests.length||u.append(_(e,"p","Requests: Not observed"));for(const a of d.requests){const f=_(e,"details");f.open=!0,f.append(_(e,"summary",`Request ${a.requestNumber}`));const h=(S?.auctions??[]).flatMap(y=>y.slots).filter(y=>y.correlation==="matched"&&y.runtimeSlotNumber===d.runtimeSlotNumber&&y.requestNumber===a.requestNumber);w(e,f,[["Correlation",h.length===1?"Matched server slot":"Correlation unknown"],["Creative participation",h.length===1?h[0].creativeLabel:"Participation unconfirmed"],["GPT fill observation",a.isEmpty===void 0?"Unknown":a.isEmpty?"Empty":"Filled"],["Browser request path",a.requestPath===void 0?"Unavailable":R(a.requestPath)],["Requested sizes",V(a.requestedSlotSizes)],["Rendered size",a.size?V([a.size]):"Unavailable"],["Observed CSS box size",a.observedSlotSize?V([a.observedSlotSize]):"Unavailable"]]);const L=Object.entries(Et).map(([y,B])=>{const k=a[y];return[B,typeof k=="string"?R(k):y.endsWith("Ms")?O(k):k]});w(e,f,L),w(e,f,[["Request to response",O(a.durations.requestToResponseMs)],["Response to render",O(a.durations.responseToRenderMs)],["Request to render",O(a.durations.requestToRenderMs)],["Render to load",O(a.durations.renderToLoadMs)],["Render to viewable",O(a.durations.renderToViewableMs)],["Creative bridge failures",a.trustedServerCreativeFailures===void 0?"Unavailable":a.trustedServerCreativeFailures.length===0?"Not observed":a.trustedServerCreativeFailures.map(R).join(", ")]]),u.append(f)}p.append(u)}const b=j(e,s,"Coverage and ambiguity");w(e,b,[["Server capture",R(t.auction_coverage.capture_status)],["Capture and interpretation limits",t.auction_coverage.issues.length?t.auction_coverage.issues.map(R).join(", "):"None recorded"],["Correlation sidecars retained",t.slot_correlations.length]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.coverage))w(e,b,[[R(d),`${u.observed} observed; ${u.matched} matched; ${u.unmatched} unmatched; ${u.ambiguous} ambiguous`]]);for(const[d,u]of Object.entries(t.truncation))w(e,b,[[R(d),u]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.metadata))w(e,b,[[`GPT ${R(d)}`,u]]);for(const d of t.gpt_diagnostics.callbackIssues)w(e,b,[["Callback issue",R(d.kind)],["GPT slot",d.runtimeSlotNumber],["Browser clock",O(d.timestampMs)],["Disposition",R(d.disposition)],["Reason",R(d.reason)]]);for(const d of t.gpt_diagnostics.attributionIssues??[])w(e,b,[["Creative attribution issue",R(d.reason)],["GPT slot",d.runtimeSlotNumber],["Browser clock",O(d.timestampMs)]]);for(const[d,u]of t.slot_correlations.entries()){const a=_(e,"details");a.append(_(e,"summary",`Correlation record ${d+1}`)),a.append(_(e,"p","Browser observed correlation. These opaque references permit a join only when unique and consistent; duplicate or conflicting records remain unknown.")),w(e,a,[["Auction reference",u.diagnostic_auction_id],["Slot reference",u.slot_ref],["GPT slot number",u.runtime_slot_number],["GPT request number",u.request_number]]),b.append(a)}return s}function ve(e=document,t={}){const n=e.querySelector("main");if(!n)return{destroy(){}};const r=ht(e),s=t.origin??window.location.origin;let i,c=!1;const l=[],o=(v,U,Se)=>{const N=_(e,"button",U);N.type="button";const Te=()=>{c||Se()};return N.addEventListener("click",Te),l.push(()=>N.removeEventListener("click",Te)),v.append(N),N},S=()=>{c=!0,i=void 0,r();for(const v of l)v()};let p;try{p=St(s,(t.now??Date.now)(),t.storage)}catch{p={status:"unavailable"}}if(p.status!=="ready"){const v=_(e,"p",p.status==="absent"?"No saved report. Enable tracing, return to the affected page, reload once, reproduce the problem, then select View trace results.":"The saved report is unavailable, expired, or unsupported. Return to the affected page in this same tab and exact hostname, reload once, reproduce the problem, then select View trace results.");return v.id="trace-report-notice",n.prepend(v),{destroy:S}}i=p.value;const b=_(e,"details");b.id="trace-viewer-setup",b.append(_(e,"summary","Setup request and tracing controls"));for(const v of Array.from(n.children))b.append(v);const d=Ct(e,i.report,s,i.stored_at_ms);n.append(d);const u=j(e,d,"Export");u.append(_(e,"p","Copy, Download and Share use the same public report JSON. The selected app receives this JSON when you choose Share. Nothing is uploaded by this viewer."));const a=_(e,"div");a.className="controls",u.append(a);const f=_(e,"p");f.id="trace-export-status",f.setAttribute("role","status"),f.setAttribute("aria-live","polite"),u.append(f);let h=!1;const L=async v=>{if(!i||c||h)return;h=!0;const U=i;try{const N=await(v==="Copy"?t.copy??pt:v==="Share"?t.share??ft:t.download??lt)(U.report,s,U.stored_at_ms);if(c||i!==U)return;f.textContent=N.status==="copied"?"Copied JSON.":N.status==="downloaded"?"Download started; completion is managed by your browser.":N.status==="shared"?"The share request completed.":v==="Share"&&N.status==="unsupported"?"File sharing is unavailable. Use Copy or Download.":`${v} could not be completed. Your report remains available; retry or choose another export.`}catch{!c&&i===U&&(f.textContent=`${v} could not be completed. Your report remains available.`)}finally{h=!1}};for(const v of["Copy","Download","Share"])o(a,v,()=>{L(v)});const y=j(e,n,"Report cleanup"),B=_(e,"div");B.className="controls",y.append(B);const k=_(e,"p","A local report is saved in this tab.");k.id="trace-cleanup-local-status",k.setAttribute("role","status"),k.setAttribute("aria-live","polite"),y.append(k);const z=_(e,"p","Server tracing state has not been changed by this report visit.");z.id="trace-cleanup-server-status",z.setAttribute("role","status"),z.setAttribute("aria-live","polite"),y.append(z);const ge=()=>{let v;try{v=Tt(t.storage)}catch{v={status:"unavailable"}}c||(v.status==="deleted"?(i=void 0,d.remove(),At.hidden=!0,k.textContent="Local report deleted from this tab."):k.textContent="Local report deletion failed. The report remains displayed; retry deletion.")};let Y=!1;const he=async()=>{if(c||Y)return;Y=!0,ye.disabled=J.disabled=!0,z.textContent="Requesting tracing end and checking the next request…";let v;try{v=await bt()}catch{v={mutation:"failed",observation:"failed",confirmed:!1}}c||(z.textContent=v.confirmed?"Tracing is off — no valid diagnostics session observed.":v.observation==="inactive"?"End tracing unconfirmed. No valid diagnostics session was observed; tracing may remain active. Retry end tracing.":v.observation==="active"?"End tracing unconfirmed — a valid session is still observed. Tracing may remain active. Retry end tracing.":"End tracing unconfirmed. Tracing may remain active. Retry end tracing.",J.hidden=v.confirmed,ye.disabled=J.disabled=!1,Y=!1)},ye=o(B,"Clear report and end tracing",()=>{if(Y)return;let v=!1;try{v=(t.confirm??(U=>window.confirm(U)))("Delete the report from this tab and request tracing end?")}catch{}!v||c||(ge(),c||he())}),At=o(B,"Delete local report",ge),J=o(B,"Retry end tracing",()=>{he()});return J.hidden=!0,n.append(b),{destroy:S}}document.readyState==="loading"?document.addEventListener("DOMContentLoaded",()=>ve(),{once:!0}):ve()})();
diff --git a/crates/trusted-server-js/src/lib.rs b/crates/trusted-server-js/src/lib.rs
index 2c816b154..d4504da82 100644
--- a/crates/trusted-server-js/src/lib.rs
+++ b/crates/trusted-server-js/src/lib.rs
@@ -4,6 +4,7 @@
 )]
 
 pub mod bundle;
+pub mod trace_assets;
 
 pub use bundle::{
     all_module_ids, concatenate_modules, concatenated_hash, module_bundle, single_module_hash,
diff --git a/crates/trusted-server-js/src/trace_assets.rs b/crates/trusted-server-js/src/trace_assets.rs
new file mode 100644
index 000000000..b75a06b43
--- /dev/null
+++ b/crates/trusted-server-js/src/trace_assets.rs
@@ -0,0 +1,70 @@
+//! Immutable, independently embedded assets for the full-document trace page.
+
+/// Verified asset bytes and their byte-derived SHA-256 digest.
+///
+/// Publisher integration bundle discovery does not include these assets.
+#[derive(Clone, Copy, Debug)]
+pub struct TraceAsset {
+    /// Exact pathname of the versioned asset.
+    pub path: &'static str,
+    /// Immutable bytes verified against the committed manifest during the build.
+    pub bytes: &'static [u8],
+    /// Lowercase hexadecimal SHA-256 digest of [`Self::bytes`].
+    pub sha256: &'static str,
+}
+
+include!(concat!(env!("OUT_DIR"), "/trace_assets.rs"));
+
+/// Look up an asset only by its exact versioned pathname.
+///
+/// # Examples
+///
+/// ```
+/// use trusted_server_js::trace_assets::trace_asset;
+/// assert!(trace_asset("/_ts/trace/assets/v1.js").is_some());
+/// assert!(trace_asset("/_ts/trace/assets/v1.js?data=example").is_none());
+/// ```
+#[must_use]
+pub fn trace_asset(path: &str) -> Option {
+    TRACE_ASSETS
+        .iter()
+        .find(|asset| asset.path == path)
+        .copied()
+}
+
+#[cfg(test)]
+mod tests {
+    use sha2::{Digest as _, Sha256};
+
+    use super::*;
+
+    #[test]
+    fn exact_trace_asset_lookup_matches_verified_bytes() {
+        for path in ["/_ts/trace/assets/v1.js", "/_ts/trace/assets/v1.css"] {
+            let asset = trace_asset(path).expect("should embed the fixed versioned trace asset");
+            assert_eq!(
+                asset.path, path,
+                "should retain the exact public asset path"
+            );
+            assert_eq!(
+                hex::encode(Sha256::digest(asset.bytes)),
+                asset.sha256,
+                "should embed only bytes matching the manifest digest"
+            );
+        }
+        for path in [
+            "/_ts/trace/assets/v2.js",
+            "/_ts/trace/assets/v1.js?data=example",
+            "/_ts/trace/assets/../v1.js",
+        ] {
+            assert!(
+                trace_asset(path).is_none(),
+                "should reject every unregistered asset spelling"
+            );
+        }
+        assert!(
+            !crate::all_module_ids().contains(&"trace"),
+            "should keep trace assets outside the publisher integration pipeline"
+        );
+    }
+}
diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md
index 09a59a01f..3b2cec920 100644
--- a/docs/guide/configuration.md
+++ b/docs/guide/configuration.md
@@ -1598,9 +1598,18 @@ See [GPT](/guide/integrations/gpt).
 
 **Section**: `[integrations.gpt_diagnostics]`
 
-The only field is `enabled`, a Boolean that defaults to `false`. When enabled,
-the standalone diagnostics tag is available, but individual browser sessions
-still require the activation flow in [GPT diagnostics](/guide/integrations/gpt-diagnostics).
+Both fields default to `false`:
+
+| Field                | Type    | Contract                                                         |
+| -------------------- | ------- | ---------------------------------------------------------------- |
+| `enabled`            | Boolean | Make the GPT diagnostics module available for activated sessions |
+| `trace_page_enabled` | Boolean | Enable the mobile trace endpoint; requires `enabled = true`      |
+
+Individual browser sessions still require the activation flow in
+[GPT diagnostics](/guide/integrations/gpt-diagnostics). The mobile endpoint does
+not activate tracing on a GET. Before enabling it, accept the documented
+same-origin visibility and privacy implications of cookie health, masked network
+facts and browser-carried auction evidence.
 
 ### JS Asset Proxy Integration
 
diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md
index af6f2e324..88e73f0b0 100644
--- a/docs/guide/integrations/gpt-diagnostics.md
+++ b/docs/guide/integrations/gpt-diagnostics.md
@@ -45,8 +45,8 @@ and non-storeable.
 
 ### Auction correlation token
 
-Enabling the integration has one further server-side effect, beyond module
-availability, that does not depend on browser activation. For each server-side auction
+The existing `hb_auction_id` publication has one further server-side effect, beyond module
+availability, that does not depend on browser activation. With mobile tracing inactive, for each server-side auction
 that produced winning bids, Trusted Server mints a fresh correlation token and publishes
 it as `hb_auction_id` on each winning bid in `window.tsjs.bids`:
 
@@ -67,6 +67,11 @@ ts-auc-2f8c1d5a4b7e4c0f9a3d6b1e8c5f2a7d
   documents with no active console session, because the console reads it from the same
   page bid state the GPT integration already consumes.
 
+Active mobile tracing instead mints its opaque identity before dispatch and reuses
+that token in deliverable auction evidence and GPT opportunity markers. Zero-bid,
+skipped and failed auctions can carry evidence independently of winning bids.
+Disabling mobile tracing retains the existing console-only publication behavior.
+
 ## Activate or Deactivate a Browser Session
 
 Open a page with one of these exact, case-sensitive query directives:
@@ -94,14 +99,146 @@ An exact directive establishes or clears the host-only, `Secure`, `HttpOnly`,
 `SameSite=Lax` `__Host-ts-console` session cookie. The server removes every reserved
 `ts_console` pair before origin, cookie, or auction handling, and the response removes
 the directive from the visible URL while preserving the path, unrelated query pairs,
-and fragment. Activation applies to the same origin across tabs until the browser
-session ends or an exact deactivation directive clears it.
+and fragment. Explicit activation lasts 30 minutes; ordinary requests do not
+extend that lifetime. The host-only cookie is shared across tabs on that hostname
+and does not isolate different ports. An exact deactivation directive clears it.
 
 Duplicate directives, unrecognized values, and duplicate activation cookies fail
 closed for the current response. Active and directive-bearing HTML responses use
 `Cache-Control: private, no-store` and omit surrogate cache headers. The cookie is
 never forwarded to the publisher origin and is unrelated to `ts-tester`.
 
+## Mobile Trace Page
+
+The mobile report is a separate opt-in surface. Its deployment option defaults
+to `false` and requires the diagnostics integration:
+
+```toml
+[integrations.gpt_diagnostics]
+enabled = true
+trace_page_enabled = true
+```
+
+Before enabling it, the operator must accept that same-origin scripts can read
+the projected health of four owned cookies, opaque auction outcomes, a masked
+network identifier and available coarse network metadata. A masked identifier
+can still identify a network. The `HttpOnly` cookie value remains inaccessible
+to scripts; the report deliberately exposes only validated shape and observation
+categories. No cookie values, exact publisher path, provider identity, creative
+markup, bid price or external user identifier belongs in a trace report.
+
+Open `/_ts/trace` on the exact publisher hostname and in the same browser tab.
+The initial visit is read-only and describes its own **Setup request**, not an
+earlier ad failure. Tap **Enable tracing**. A successful POST requests the cookie
+change; a separate state request must observe a valid session before the page
+says tracing is on. Return to the affected page, explicitly reload it once,
+reproduce the issue, then tap **View trace results** in the TS Console. A history
+Back or restored page alone does not prove a new capture. If history is not useful,
+reopen the article on that same hostname in that tab and reload once.
+
+The publisher action validates one combined report and saves it before navigating
+in the same tab. If storage alone fails, it stays on the publisher page and offers
+an explicit download of that same valid report. A failed capture does not offer a
+GPT-only export as an equivalent fallback. The viewer keeps traced-document facts
+separate from its setup request. Server-auction and browser/GPT clocks remain
+separate; candidate selection or a filled slot does not establish an auction winner.
+
+### Storage, export and cleanup
+
+One versioned `sessionStorage` key holds at most 512 KiB of compact UTF-8 data.
+A new explicit capture replaces it. Reports expire after 15 minutes, but tabs with
+an opener and browser session restoration may copy or retain the entry. Same-origin
+scripts and service workers can read or replace it. The viewer labels it
+**Browser-carried, unverified diagnostic data**; it is troubleshooting information,
+not cryptographic proof, identity evidence or security evidence.
+
+**Copy**, **Download** and file **Share** use the same validated public report JSON.
+The selected share app receives that file. The viewer does not upload a report or
+substitute URL sharing when file sharing is unavailable. Native download or share
+initiation does not prove that a recipient saved the file.
+
+**Delete local report** removes only this tab's owned report key. After confirmation,
+**Clear report and end tracing** attempts local deletion, the end POST and a separate
+state check independently. Offline requests cannot prevent local deletion. A failed
+deletion retains its own retry; unconfirmed server end retains a separate retry and
+may leave tracing active. An inactive state means no valid session was observed in
+that request; it does not prove that every cookie is absent. Exported files and
+copies in other tabs or apps are not removed by local cleanup.
+
+### Deployment and runtime boundary
+
+Use a suitable same-origin deployment where the browser accepts the unchanged
+`Secure`, `HttpOnly`, host-only `SameSite=Lax` cookie. Its explicit activation lifetime
+is 1800 seconds; ordinary requests do not refresh it. Do not weaken cookie attributes
+or infer HTTPS from an untrusted forwarding header to make a test pass.
+
+A pre-existing session cookie cannot be retroactively assigned this lifetime by
+the server. End tracing and explicitly enable it again to adopt the bounded cookie.
+Expiry or disabling the deployment flag does not unload an already-running page;
+reload documents after rollback. Subsequent gated server requests stop producing
+new evidence immediately when the deployment option is disabled.
+
+Ordered Basic-auth handlers remain first-match-wins. They match the literal path
+visible after runtime conversion, before trace feature flags or method checks.
+Protect the shell and its fixed assets consistently if operator authentication is
+required. Without a matching handler, this opt-in surface is available to users of
+that publisher deployment. The script and stylesheet are fixed same-origin versioned
+assets. Dynamic endpoints and evidence-bearing responses are `private, no-store`;
+successful assets are publicly immutable only when unprotected.
+
+The boundary distinguishes runtime rejection before invocation, adapter
+bootstrap/conversion failure, and successfully converted application-visible
+handling. Fastly, Cloudflare or Spin can normalize a path or change visible header
+representation before Trusted Server sees it. A path normalized outside the trace
+namespace follows ordinary health/routing behavior. A path normalized into a literal
+trace path receives full trace authentication and handling. Aliases still visible
+encoded after conversion are reserved and rejected under authentication for their literal visible path; decoding does not
+grant different authentication coverage. Original wire-target reconstruction is not
+a release prerequisite or a source of authentication decisions.
+
+The visible Cookie header is capped at 16384 bytes. Invalid visible text or size
+prevents health inspection. When runtime fidelity cannot guarantee preservation,
+ambiguous comma-folded values or replacement characters conservatively make all
+four cookie rows unavailable and suppress capture; marker-free headers with unknown
+fidelity can still activate tracing. This can produce false negatives rather than
+guessing a session. End remains available for ambiguous cookie observations.
+
+Locally handled authentication, routing and control failures use bounded
+400/401/403/404/405/413 responses. A runtime rejection has no fabricated local trace
+response. Adapters may buffer a request body before the hook runs; the hook supplies
+no wall-clock timeout, memory-allocation guarantee or earlier raw-header access.
+Unsupported metadata and request-relative server milestones display **Unavailable**.
+There is no historical recovery or server-side report store.
+
+### Release acceptance
+
+Keep `trace_page_enabled` off until the required automated checks in the
+[implementation plan](../../superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md)
+pass. The browser checks must exercise the complete setup, real cookie observation,
+publisher reload, capture, viewer, export and end flow on each supported adapter.
+Real bidder transport and creative rendering require their separate live checks;
+an auction-disabled browser fixture does not establish those behaviors.
+
+Before production rollout, also complete the following checks on real iOS Safari
+and Android Chrome. Desktop Chromium emulation does not establish native mobile
+behavior:
+
+- Reproduce from the same hostname and tab using the displayed return/reload
+  instructions. Verify browser Back alone is not described as a fresh capture.
+- Check 320 CSS px layout, text zoom, safe-area placement, readable status messages
+  and usable primary controls without horizontal page scrolling.
+- Download and share the JSON through the native browser/app picker. Compare the
+  saved file with the copied report, and verify cancellation or rejection retains
+  the visible report and its other export actions.
+- Verify local deletion independently of an offline or failed end request, then
+  reconnect and use the separate end/state retry.
+- Check viewer reload, tab/opener copies, expiry and any browser session restoration
+  behavior being supported. Record session-restoration checks separately from
+  automated storage fixtures.
+
+Record the device/browser versions and results in the same implementation plan.
+Until these checks are complete, physical mobile acceptance remains pending.
+
 ## What the Console Shows
 
 The panel opens expanded after document startup and provides filters for All,
@@ -410,9 +547,9 @@ unsubscribe()
 | `hide()`              | Dismisses presentation without stopping capture                                                      |
 | `show()`              | Clears dismissal and remounts presentation without resetting data                                    |
 
-## V1 Export, Storage, and Privacy
+## GPT-only V1 Export, Observation Memory, and Privacy
 
-The allowlisted export contains:
+The existing GPT-only `snapshot()` and `export()` model contains:
 
 - `version: 1` and an ISO `capturedAt` timestamp.
 - Current page origin and pathname, excluding query parameters and fragments.
@@ -441,12 +578,16 @@ identifier, and never repeated across auctions, so it cannot be joined back to a
 visitor. Diagnostics retain it only after trimming to a non-empty value of at most
 256 UTF-8 bytes.
 
-Captured records are memory-only. Diagnostics do not add an upload, diagnostics
+The GPT-only observation store remains memory-only. Its capture and `export()` do not add an upload, diagnostics
 network request, `localStorage`, `sessionStorage`, IndexedDB, or other persistence.
 `export()` creates only the user-requested local JSON download; it sends nothing to a
 server. The `__Host-ts-console` session cookie contains only the activation bit and is
 inaccessible to JavaScript.
 
+The explicit mobile handoff separately writes the bounded combined trace report
+to `sessionStorage`. Its stricter GPT projection excludes the exact pathname,
+GAM identifiers and `previousCreativeId`, as described under [Mobile Trace Page](#mobile-trace-page).
+
 ## Timing and Retention Bounds
 
 - Direct, Prebid, and publisher request-path markers: five seconds, one-shot.
diff --git a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
new file mode 100644
index 000000000..f2a461bb3
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
@@ -0,0 +1,1072 @@
+# Mobile Ad Rendering Trace Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use @superpowers:subagent-driven-development or @superpowers:executing-plans to implement this plan task by task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Implement the approved mobile trace journey from deliberate activation through real publisher reproduction to a bounded, redacted, browser-local report.
+
+**Architecture:** Core owns early authenticated trace responses, request facts, and live auction evidence. The existing GPT recorder supplies browser observations and exact binding sidecars; a trace-owned browser projection supplies the same model to the mobile viewer and every export. Implement the four boundaries in spec section 17 as phases within this single plan, with optional network enrichment outside the v1 release gate.
+
+**Tech Stack:** Rust 1.95.0/edition 2024, EdgeZero, Fastly Compute, Axum, Cloudflare Workers, Spin, TypeScript, the existing Vite/esbuild pipeline, Vitest, Prebid 10.26.0, Playwright, and native DOM/Web APIs.
+
+**Runtime-boundary amendment:** This plan tracks the approved v1 amendment in
+spec sections 8/9.2/16. Classify application-visible paths; inspect frozen
+runtime-visible cookies with explicit ambiguity suppression; record runtime
+rejection and adapter conversion failure separately. Original-target recovery
+and original-wire reconstruction are not v1 prerequisites. Existing auth,
+same-origin actions and response privacy remain required. Keep this one plan.
+
+---
+
+## Source and execution boundary
+
+- Approved source: [Mobile ad-rendering trace design](../specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md), especially sections 5, 8, 9, 12–17.
+- Approval reference: [PR #1107](https://github.com/IABTechLab/trusted-server/pull/1107).
+- Planning baseline: `20f4a0cc0139eac842d1d6ff1410a1af6ee8eba8` on `spec/mobile-ad-render-trace-endpoint`.
+- The original spec and runtime-boundary amendment are approved. The user authorized implementation on 2026-10-05 after independent document review.
+- Keep this branch and checkout. Do not create a worktree or branch. Implement incrementally against this single plan, recording verification and independent review evidence. Publish only the complete v1 feature after its release gates.
+- Existing contributor rules apply to implementation: `error-stack`, `derive_more`, no local imports, no sensitive fixture data, documented public APIs, and sentence-case imperative commit messages.
+- Proposed new filenames and function names below are implementation decisions, not claims that those symbols already exist. Locate existing code by symbol; line numbers will move.
+
+## Execution order and dependency map
+
+This is one implementation plan for one approved spec. The user requested a single plan; the four implementation boundaries in spec section 17 remain phases here rather than separate documents or mandatory separate PRs. All mandatory work stays on the existing branch and ships as one complete default-off v1 feature. Intermediate commits are review checkpoints, not independently published setup releases.
+
+| Phase                       | Tasks | Deliverable                                                                                                                                      | Dependency                                                                    |
+| --------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
+| 1. Route/privacy foundation | F0–F8 | Local authenticated application-visible routes on all four adapters, setup/session lifecycle, cookie/network facts and private publisher context | Reviewed immutable EdgeZero hook/metadata pin and runtime-boundary acceptance |
+| 2. Live auction evidence    | E1–E7 | Bounded server records through all transports, API unit tokens and exact SSAT/SPA recorder sidecars                                              | Foundation gates, immutable request facts and private response policy         |
+| 3. Browser handoff/viewer   | V1–V7 | Strict combined report, deterministic bounds, same-tab handoff, mobile presentation and equivalent local exports                                 | Foundation plus live collector/public GPT export                              |
+| 4. Optional enrichment      | N1–N3 | SDK-verified optional protocol/POP/ASN facts only                                                                                                | Complete v1; not a release gate                                               |
+
+Implementation may progress on pure F1–F3/F5–F6 and E1/V1 contract work while F0 is unavailable, but no adapter-ordering success may be claimed without F0. After each task, run its focused verification; checkpoint the completed phase against the acceptance matrix. Keep full release gates at the final handoff, and run the full repository gates before any earlier external PR handoff required by AGENTS.md.
+
+No trace assets exist at the planning baseline. Build and evolve one unpublished v1 JS/CSS set through phases 1–3, keeping the committed digest/byte fixtures synchronized in each implementation commit. Freeze that complete set at first publication after V7; later changed bytes require a new asset-set URL/digest. Do not create a v2 set merely because the setup and viewer were implemented in different tasks.
+
+## Staff review of implementation choices
+
+Reuse the four adapter entry paths and existing authentication/cache mechanisms. Add focused trace modules rather than splitting the large publisher/orchestrator files or replacing the existing GPT attribution engine. The server projection reads live observations, never telemetry rows. The viewer imports the public GPT export contract, never the private store.
+
+Pinned EdgeZero v0.0.8 (`567964158e4f8bd0d52321b9801de44966422e1b`) selects a method/path before middleware. Its concrete RouterService runner seams therefore require the upstream pre-dispatch hook and a reviewed immutable dependency pin. The EdgeZero feature branch now supplies that API, honest metadata and converter fixes. Verify method preservation for requests the runtime accepts, classify the SDK/runtime-visible pathname, and prove required cookie/control semantics at this boundary. Do not require an original-target SDK accessor or impossible original-wire reconstruction for v1. Fastly paths normalized outside trace retain their existing health/JA4 behavior; paths exposed within trace must be intercepted. Cloudflare's converter must preserve exposed extension tokens even though its tested wire parser rejects them before invocation. A finite method list or Fastly-only bypass does not satisfy application-visible adapter parity.
+
+## Implementation invariants
+
+1. Authenticate every reserved trace request before setup facts, body inspection, or mutation, including disabled routes and malformed paths; preserve first-match-wins handlers.
+2. Trace dispatch creates no ordinary event/EC state, invokes no configured filters, performs no auction/EID processing, and makes no publisher or telemetry request.
+3. Freeze runtime-visible CookieHealth before `gpt_diagnostics::prepare_request` sanitizes headers; retain no raw values. Base capture requires the flag plus exactly one valid inspectable session, with ambiguity suppressing it; document capture additionally requires the effective diagnostics navigation decision.
+4. Preserve console-only diagnostics when trace is off. Query activation deliberately adopts the shared 1800-second lifetime even when trace is disabled.
+5. Gate every new token, extension, listener, mapping, sidecar, and handoff action. TSJS requires the literal `window.__tsjs_trace_active === true`; the server independently evaluates each request.
+6. Keep the three clocks and three evidence layers separate. Correlation failure never downgrades complete server capture. `/auction` has no GPT sidecars in v1.
+7. Strictly allowlist both projection and ingestion. No page path, cookie value, identity, full IP, fingerprint, provider name, price, targeting, creative identity/payload, or internal request ID enters a trace artifact or trace-specific log.
+8. Reuse terminal private/no-store protection, including Fastly's final guard after late effects. Only successful unprotected fixed assets can be public immutable.
+9. Asset URLs are byte contracts. Freeze the complete v1 set at first publication; any later changed bytes require new URLs/digests while retaining published bytes/digests and exact routes.
+10. All trace failures fail open for advertising. No diagnostic failure changes ordinary ad acceptance, ordering, targeting, rendering, or status.
+
+## Acceptance evidence matrix
+
+| Spec criterion                                      | Owning work                                      | Required evidence                                                                                                                                          |
+| --------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 1: off by default, ordinary traffic unchanged       | Foundation F1/F4/F7; auction E2/E5/E6; viewer V5 | Config validation, inactive gates, before/after bid fixtures, rollback reload                                                                              |
+| 2–3: deliberate mobile setup and reproduction       | Foundation F2–F6                                 | Auth/method/path matrix, same-origin controls, GET read-only, observed-state verification, history/reload instructions                                     |
+| 4: real publisher capture                           | Foundation F7; auction E1–E6                     | Traced reload, SSAT/SPA/API transports, unchanged ads                                                                                                      |
+| 5: bounded same-tab snapshot                        | Viewer V1–V3/V5                                  | Strict validation, 512 KiB UTF-8 wrapper, deterministic truncation/floor rejection, successful write before navigation                                     |
+| 6–7: separated evidence and equivalent export       | Viewer V4/V6/V7                                  | Semantic labels, exact joins, downloaded/copied/shared report equality                                                                                     |
+| 8: public-safe fields only                          | All phases                                       | Distinct forbidden-data sentinels across HTML, transport, storage, logs, copy/share/export; hostile stored keys rejected                                   |
+| 9: private/no-store                                 | Foundation F4/F7; auction E3/E4; viewer V7       | Hostile operator overrides, final Fastly effects, ESI/shared-template absence                                                                              |
+| 10: honest degradation                              | Auction E5–E7; viewer V2–V7                      | Coverage truth table, transport/projection/storage failures, optional API absence                                                                          |
+| 11: mobile/accessibility                            | Foundation F6; viewer V6/V7                      | 320 CSS px, 44 px controls, keyboard/focus/aria-live, iOS Safari and Android Chrome                                                                        |
+| 12–13: exact schemas, provenance, hostile storage   | Viewer V1/V4/V6/V7                               | Source-member coverage/type gate, all unknown versions/keys rejected, CSP/XSS/depth/DOM-bound checks                                                       |
+| 14: independent cleanup and verification            | Foundation F5/F6; viewer V3/V6/V7                | Offline end, failed deletion, mismatch and retry with separate statuses                                                                                    |
+| 15: server entry point and honest joins             | Auction E1–E7; viewer V4/V6/V7                   | Zero-bid/failure records, SSAT/SPA exact joins, API independent, no inferred client winner                                                                 |
+| 16: explicit runtime boundary and adapter parity    | Foundation F0/F2–F5/F8; viewer V7                | Separate pre-invocation rejection/conversion outcomes, visible path/auth/method tests, complete browser journey on all four adapters with suitable origins |
+| 17: conservative Cookie ambiguity and public detail | Foundation F3/F4/F6; viewer V1–V3/V6/V7          | Per-Cookie fidelity-axis fixtures, cap/UTF-8/ambiguity precedence, inactive capture, exact reason/state validation and storage/export equality             |
+
+## Shared verification commands
+
+Run focused red/green tests in each task. A red run must fail for the intended behavior, not because a dependency is missing. Name new Rust regression functions/modules with the task filter shown in its command and assert each focused run actually selects tests; a zero-test run is not verification. Keep tests and the minimum implementation together in each planned commit. Use @superpowers:test-driven-development and @superpowers:verification-before-completion during execution. No implementation or runtime test pass is claimed by this plan.
+
+Before the final implementation/PR handoff, run the complete repository gate list from the root unless a working directory is shown. Earlier internal task/phase checkpoints use focused tests and affected-target checks; if an earlier checkpoint is submitted as its own external PR, run the complete gates at that handoff as AGENTS.md requires:
+
+```bash
+cargo fmt --all -- --check
+cargo clippy-fastly
+cargo clippy-axum
+cargo clippy-cloudflare
+cargo clippy-cloudflare-wasm
+cargo clippy-spin-native
+cargo clippy-spin-wasm
+cargo clippy-cli
+cargo clippy-codegen
+cargo test-fastly
+cargo test-axum
+cargo test-cloudflare
+cargo test-spin
+./scripts/test-cli.sh
+cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity
+npm --prefix crates/trusted-server-js/lib run build
+npm --prefix crates/trusted-server-js/lib run test
+npm --prefix crates/trusted-server-js/lib run format
+npm --prefix docs run format
+docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md"
+git diff --check
+```
+
+Expected: every command exits 0; all suites have zero unexpected failures. Install locked npm dependencies with `npm ci` in the respective package only when missing. Use the pinned Rust/Node/runtime tools. Rust adapter aliases already select the proper targets; bare workspace `cargo test` is incorrect.
+
+For runtime/build changes, also run `cargo build-fastly`, `cargo build-axum`, `cargo build-cloudflare`, and `cargo build --package trusted-server-adapter-spin --target wasm32-wasip1 --features spin --release`. Verify changed public Rust documentation with `cargo doc -p trusted-server-core --no-deps --all-features --target wasm32-wasip1`; do not run the incompatible workspace all-feature command.
+
+Browser changes additionally run `./scripts/integration-tests-browser.sh`, which builds artifacts, generates Viceroy config, builds the Docker fixtures, and executes Next.js and WordPress Playwright suites. It requires Docker, Viceroy, the WASM target, and Playwright Chromium. Manual mobile checks are required release evidence; an unavailable phone or Docker daemon is an explicitly pending check, never a pass.
+
+## Execution checklist
+
+### Execution evidence (2026-10-06)
+
+Entries below record successive checkpoints. Later verification supersedes an
+earlier checkpoint's pending status; the final status table records remaining
+release work.
+
+- User approved the amended contract and authorized implementation on the
+  existing branch.
+- F0: EdgeZero independently reviewed and published as PR #403, closing Task
+  issue #402 through its PR-creator workflow. Trusted Server pins immutable
+  revision `499d5c93d597f01b5e0497e332af82f2aa633277`. All four adapter check
+  aliases pass. Independent probes using the resolved consumer versions pass
+  47 hook/converter tests and all 54 freshly rebuilt raw-wire cases, with zero
+  differences from committed observations. All reported upstream GitHub checks
+  pass. Project-board association is pending token project permissions.
+  Full F0 acceptance still requires F4 integration and V7 supported-origin
+  browser workflows; converter evidence alone does not close that gate.
+- F1: Default-off config, dependency validation on both startup paths and shared
+  1800-second cookie policy implemented. Red tests demonstrated the missing
+  dependency validation and literal missing cookie lifetime before implementation.
+  `cargo test-fastly trace_config` selects two passing core tests;
+  `cargo test-fastly gpt_diagnostics` selects fourteen passing core tests.
+  Independent spec and quality reviews approved after corrections.
+  Combined F1/F2 target-matched Fastly clippy passes; endpoint/query sequence
+  proof follows F5.
+- F2: Classifier, authentication and response preflight implemented. Spec review
+  approved. Quality review identified combined separator/dot namespace erasure;
+  a behavioral red regression and fix now pass all twelve route tests, with
+  independent re-review approved. Twenty-seven existing auth tests pass.
+  Successful shell/state/action/asset behavior depends on F3/F5/F6; no complete
+  endpoint or runtime acceptance is claimed yet.
+- F3: Runtime-visible cookie scanner and frozen request context implemented;
+  ten cookie and seven context tests pass, with independent spec and quality
+  approvals. Focused clippy, documentation build and formatting pass.
+- F5: Deliberate action and frozen state handlers implemented. Fifteen focused
+  action tests and forty-six combined trace tests pass. Both independent reviews
+  approved; clippy, documentation build and formatting pass. Production callers
+  and adapter body parity remain part of F4 integration.
+- F6: Session helper, setup controls, strict request-context validator, escaped
+  Rust shell, separate versioned assets and build verification are implemented
+  and independently approved for their current unit scopes. Eighty-five browser
+  foundation/asset tests and two Rust shell tests pass. Five compiled build-script
+  probes verify missing/stale assets, stale source, unknown manifest fields and
+  valid skipped builds. Complete report viewer and browser workflows remain pending.
+- E1 browser contract slice: exact opaque tokens, bounded evidence and sidecar
+  validators, exclusive transport types and immutable owned ingestion implemented.
+  Independent review caught and corrected type exclusivity and caller-controlled
+  array-method gaps; both re-reviews approved. One hundred and three focused tests
+  pass; combined browser foundation and evidence tests total one hundred and
+  eighty-eight. Scoped strict TypeScript and ESLint checks pass. The Rust contract
+  also received independent spec and quality approvals: twenty-six focused native
+  and Viceroy tests, nine documentation examples, all six adapter Clippy aliases
+  and native core lint pass. Strict map/string decoding and required-fact validation
+  before truncating tails are pinned by regressions.
+- E2 private live carries received independent spec approval and architectural
+  review. Forty-seven focused native and Viceroy tests pass, covering actual
+  provider/mediator launch order, empty-plan versus split-dispatch semantics,
+  checked tail counts, duplicate accepted instances and telemetry-independent
+  cancellation. All four adapter suites, six adapter Clippy gates, formatting and
+  committed assets pass; docs compile with the existing thirty-one warnings.
+  Response slot-token adoption, final delivery disposition and public transports
+  remain E3/E4 work.
+- V1 public report contract is implemented and independently approved for its
+  pure validation/type scope. Fifty-nine behavioral validation tests and source
+  member classification/type tests pass. Validation and byte measurement consume
+  the same owned descriptor snapshot; changing getters, serialization hooks,
+  extra fields and unsupported schemas cannot enter accepted reports. The scoped
+  TypeScript gate exposes only four pre-existing duplicate `w`/`h` declarations
+  in `core/types.ts`, reproduced against the planning baseline; an in-memory
+  future-source-field probe correctly breaks the exhaustive classification map.
+- V2 projection and report builder are implemented, with separate spec and
+  quality approvals. Seventeen projection and fifteen report tests pass, including
+  checked omission overflow, stable capture-clock observations, dangling sidecar
+  pruning, deterministic removal order and protected slot floors. The populated
+  512 KiB fixture includes multibyte facts and exact six-counter/survivor checks.
+  Collector integration and mobile capture responsiveness remain pending.
+- V3 storage and export primitives are implemented and independently approved.
+  Seventeen storage and eight export tests pass, covering exact origin/expiry,
+  serialized UTF-8 bounds, safe integer capture time, unavailable storage,
+  identical public-report formatting and independent deferred download cleanup.
+  Actual branch trace tests now total three hundred and six across thirteen
+  files. User action wiring, handoff recovery and browser acceptance remain pending.
+- F4 integration received independent spec and quality approvals. The actual
+  Fastly, Cloudflare and Spin runtime boundary suite passes all three selected
+  tests, including each runtime's 198-case flag/auth matrix, normalized-in and
+  normalized-out paths, encoded aliases, terminal headers and distinct runtime
+  rejection outcomes. Native Axum parity and all six adapter lint gates pass.
+  This closes the scoped hook/terminal integration; F0 browser acceptance still
+  requires V7. No pre-invocation rejection is credited with an application response.
+- The E1 owned-ingress correction, E5 pure collector and V4 pure correlation
+  model received independent spec and quality approvals. Public validators now
+  snapshot own data before validation, and collector imports do not pull in the
+  report/GPT runtime graph. Actual trace tests total 336 across fifteen files.
+  Live caller, sidecar and viewer integration remain separate tasks.
+- F7 publisher bootstrap received independent spec and quality approvals. Six actual Viceroy
+  tests cover the gate matrix, HTML-only injection, safe head ordering, failure
+  privacy, ESI exclusion and distinct visitors using a warm template cache. An
+  emitted-script Node test proves hostile-string roundtrip and deep freezing of
+  all context containers. All four adapter suites and all six adapter lint gates
+  pass; documentation retains the same 31 pre-existing warnings.
+- E5 direct API wiring and V5 handoff received independent spec and quality
+  approvals and are copied into the branch. Direct evidence is consumed before ordinary
+  bids; supplied unreadable namespaces record a bounded validation issue. Handoff
+  saves one validated report before same-tab navigation, keeps an explicit download
+  after storage failure, and stops on destruction from callbacks. The actual branch
+  passes 587 tests across thirty trace/asset/core/GPT files after rebuilding and
+  refreshing draft source digests. Prebid hooks, live SSAT/SPA sidecars and V7
+  browser acceptance remain separate tasks.
+- V6 viewer and independent cleanup received independent spec and quality approvals
+  and are copied into the branch. Sixty-one focused tests pass, including twelve local-delete
+  × end-request × state-observation combinations, offline requests, independent
+  retries, local deletion during a pending end request and listener destruction.
+  Report facts and setup facts remain separate; exports use the same immutable
+  public model. After rebuilding and refreshing draft assets, the actual branch
+  passes 658 tests across thirty-seven selected trace/asset/core/GPT files.
+  An independently reviewed browser-carried fixture passes the served viewer in
+  actual Chromium against each publisher framework: 320 px layout, 44 px controls,
+  text-only hostile content, real keyboard Clipboard and Blob download with equal
+  report-only JSON, and independent local/server cleanup. This proves viewer
+  behavior under its CSP; it does not substitute for live auction capture/handoff,
+  physical devices or file-sharing acceptance.
+- F8 uses two dedicated trace configurations alongside the unchanged default-off
+  browser baseline, with explicit owned runtime cleanup in local and CI runners.
+  Two process-cleanup regressions pass. Seven actual Chromium foundation tests
+  pass against each of Next.js and WordPress with a freshly rebuilt Fastly artifact:
+  explicit session activation/end, authenticated routes/assets, served asset digests,
+  default-off behavior, cross-site mutation rejection and history/reload activation.
+  Publisher reload exposed origin-304 reuse when the ad stack was disabled. Two
+  meaningful failing regressions preceded a fix that strips validators/ranges for
+  diagnostics document changes and rejects unexpected origin 304 responses with a
+  private 502. Both regressions and eighteen existing cache-policy tests now pass.
+  Browser TypeScript checks pass with the existing shared Node type definitions.
+  Operator guide additions received scoped spec approval; full report workflows,
+  other adapter browser workflows and physical mobile acceptance remain pending.
+- E6 browser binding and the related real-snapshot V2 correction received
+  independent spec and quality approvals and are copied into the branch. Accepted
+  SSAT/SPA batches bind exact slot objects; sidecars emit only for the consumed
+  opportunity and its actual GPT cycle. Zero-bid opportunities reuse the validated
+  auction token, conflicting markers remain unknown, and oversized delivery lists
+  keep server evidence without prefix joins. Optional own-data undefined members
+  in the existing GPT snapshot are omitted during projection; other schema checks
+  remain strict. Actual bootstrap, bundle, fallback and SPA snapshots pass report
+  construction and exact correlation. The branch passes 990 selected tests across
+  forty-four files, including committed asset verification. Live Rust transport
+  wiring remains E3/E4 work.
+- E7 Prebid transport is copied into the branch after independent spec and
+  quality approvals. Review exposed sequential publisher bid-ID reuse; the
+  corrected private association pairs the original bid ID with the actual SDK
+  bidder-request ID. Late old timeout/error hooks cannot consume a newer request;
+  ambiguous or missing associations still retain exact readable responses.
+  Bounds, expiry and page lifecycle cleanup preserve ordinary bidding. Eleven
+  actual registered SDK artifact tests pass; the branch passes 852 selected tests
+  across twenty-seven files, including asset verification. The measured external
+  Prebid shim is 52,030 characters with a narrowly adjusted 52,500 guard; strict
+  source and changed-test checks introduce no diagnostics against the existing
+  baseline.
+- V7's local bidder fixture reaches the real Rust HTTP client through test-only
+  Viceroy aliases derived from the production backend naming policy. Thirteen
+  generator regressions pass, including rejection of non-loopback overrides and
+  bounded timeout enumeration; helper Clippy passes. The fixture returns selected,
+  empty and failed responses, counts actual invocations without retaining bodies,
+  and rejects malformed/null controls with fixed 400 responses. A meaningful
+  failing HTTP regression preceded that correction. Nine Chromium checks pass
+  with the enabled Next.js configuration and eight shared checks pass against
+  WordPress. A form-navigation race was corrected by waiting for the actual
+  rejected navigation to commit. The live zero-bid journey proves two actual
+  bidder calls before reaching its expected missing-transport failure; it is
+  still incomplete until E4 delivers the live envelope and final assertions run.
+- E3's typed `/auction` transport received independent spec and architecture
+  approvals. The raw extension visitor tracks accepted conversion occurrences
+  without changing public ad models or acceptance. Review caught oversized
+  optional numeric values discarding unrelated valid references; a meaningful
+  failing regression preceded borrowed per-unit isolation. Eleven focused cases
+  pass on native and Fastly, all four full adapter suites and six adapter lint
+  gates pass after the correction, and documentation/format/asset checks pass
+  with the same pre-existing documentation warnings. API responses preserve
+  ordinary ad content and add private evidence only under the frozen request gate.
+  Publisher SSAT/SPA transport remains E4 work.
+- V7's all-four browser harness reuses the Rust runtime launchers and owns its
+  browser subprocess. Its source received independent review, TypeScript,
+  discovery and baseline-skip checks pass, and a real hanging Chromium probe
+  proves the owned Node/worker signal cleanup closes the observed detached
+  browser processes locally. The normal browser workflows remain unexecuted
+  pending fresh artifacts; this is not evidence of full adapter journey success.
+- E4's request-scoped SSAT and both SPA transports received independent spec
+  and architecture approvals. The document tail reuses the pre-dispatch auction
+  identity and exact slot references, projects actual delivered dispositions,
+  and keeps templates/ESI free of private transport. Skipped and empty auctions
+  remain observable when the tail is reached; an unreached tail invents no
+  evidence. Two meaningful failing regressions preceded implementation; six
+  focused native and six focused Fastly cases pass. All four adapter suites,
+  six adapter lint gates, formatting and asset checks pass. Final lint corrections
+  reuse the ordinary inactive wrapper and replace a serializer panic with the
+  exclusive unavailable marker and a fixed log message. Fresh production
+  artifacts and complete live browser assertions remain pending.
+- The integration workflow now gives raw runtime and normal browser trace
+  acceptance a dedicated job with explicit Chromium, Wrangler and Spin
+  prerequisites. Ordinary integration runs exclude these opt-in families.
+  Independent review and six actual Bash 3.2 argument cases verify default,
+  positive-filter and negative-skip behavior. The prepared CI job has not run
+  remotely; its source is not credited as runtime acceptance evidence.
+- Independent full mandatory-phase source/spec and architecture reviews found
+  no actionable findings. Seventeen shared Chromium checks pass against the
+  reviewed foundation artifact, including real CSP blocking, opener-cloned
+  storage separation and simulated expiry, rejected report removal, blocked
+  deletion with retained export, and offline cleanup followed by actual server
+  retry/observation. These carried-report fixtures are not credited as live
+  auction capture or a real browser restart. Three additional reviewed live
+  fixtures cover a real failed-provider SSAT response and unchanged core API
+  creatives after only the optional trace transport is removed or made invalid;
+  their fresh-artifact execution remains pending.
+- Final fresh production artifacts build on all four platforms. The complete
+  repository browser runner passes 49 Next.js and 28 WordPress checks, with only
+  framework-selection and separate-runtime skips. Actual SSAT zero-bid and
+  failed-provider evidence, selected creative delivery through the real PUC
+  bridge, both core API outcomes, the pinned real Prebid caller, both SPA aliases,
+  and unchanged advertising after altered optional trace transport all pass.
+  The first live handoff failure exposed a test locator that could not reach the
+  intentionally closed diagnostics shadow root. A reviewed Chromium-only helper
+  now finds the exact accessible button and sends a real pointer click; production
+  shadow behavior and asset bytes are unchanged. The failing case and complete
+  suites then pass.
+- Fresh normal browser workflows pass separately on Axum, Cloudflare, Fastly and
+  Spin: real Secure/HttpOnly cookie storage, separate observed state, publisher
+  reload/capture, same-tab handoff, equivalent clipboard/download JSON, independent
+  local deletion, server end and subsequent state verification. Auctions are
+  deliberately disabled in this cross-runtime fixture; live bidder and creative
+  transport proof comes from the dedicated Next.js/Viceroy suite above. Fresh raw
+  boundary suites also pass against Cloudflare, Fastly and Spin, keeping runtime
+  rejection separate from converted-path/header behavior. These are local runtime
+  results, not deployed-platform or physical-device acceptance.
+- Remaining repository gates pass: CLI and codegen lint, host CLI tests including
+  actual Chrome fixtures, 21 cross-adapter parity cases, integration lint, JS build,
+  pinned external Prebid build, 1,733 Vitest tests in 69 files with no type errors,
+  JS/docs/Markdown format and browser TypeScript checks. Final current-source full
+  adapter suites pass after the last E4 lint corrections: Fastly core 3,011,
+  adapter 199, JS crate 4, OpenRTB 21 and doctests 23; Axum 44, Cloudflare 54 and
+  Spin 89, with only existing ignored cases. Generator tests pass 13/0; core
+  target-matched documentation and format checks pass with the same 31 existing
+  documentation warnings. Physical mobile, actual session restoration and staging
+  rollout remain pending release checks.
+- Existing unpublished v1 draft bytes and source digests are deliberately
+  refreshed during implementation. First complete publication remains gated on
+  every mandatory phase and final verification.
+
+- [x] Resolve and verify the EdgeZero prerequisite before foundation integration.
+- [x] Execute and review foundation tasks F0–F8.
+- [x] Execute and review live-evidence tasks E1–E7.
+- [ ] Execute and review browser tasks V1–V7.
+- [ ] Record all acceptance evidence, full CI gates, and manual mobile results.
+- [ ] Enable only a controlled staging fixture; verify CDN/private responses and operator privacy acceptance.
+- [x] Keep enrichment and future schemas separately scheduled.
+- [x] Present the resulting implementation diff for independent review before external publishing.
+
+| Scope                      | Current status                                                                          |
+| -------------------------- | --------------------------------------------------------------------------------------- |
+| Mandatory production code  | Implemented, independently reviewed; complete automated browser journeys pass           |
+| Final repository gates     | All required automated gates pass                                                       |
+| Physical mobile acceptance | iOS Safari and Android Chrome checks pending; see operator guide release checklist      |
+| Real session restoration   | Manual check pending; actual opener cloning and simulated future-load expiry are tested |
+| Staging and deployment     | Pending operator rollout/privacy/CDN acceptance; feature remains default off            |
+| Optional phase 4           | Unscheduled; unavailable fields remain omitted rather than inferred                     |
+
+## Independent plan review
+
+Reviewed on 2026-10-05 by three independent read-only subagents against the approved spec and current source. Each reviewer assessed the consolidated document without relying on the earlier multi-document reviews; the findings were corrected centrally and the changed tasks were reviewed again.
+
+| Reviewer                | Coverage                                                                                                                                                                             | Final result                     |
+| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
+| Spec alignment          | All normative sections, acceptance criteria, activation, exact schemas, privacy, storage, failure behavior, rollout and optional scope                                               | Approved                         |
+| Server and all adapters | Rust ownership, upstream API, runtime conversion, Fastly native shortcuts/finalization, authentication, cookies, controls, live outcomes, transports and parity                      | Approved after F0/F4 corrections |
+| Browser, UI and build   | Both API callers, registered Prebid hooks, GPT binding, IIFE sharing, validation/truncation, local storage/export, CSP/mobile/accessibility, frozen assets and real fixture plumbing | Approved                         |
+
+Baseline review corrections included early Fastly capture, actual converter/runtime probes, the browser-gated Cloudflare runner, successful empty-plan versus unsuccessful split-dispatch outcomes, throwing diagnostic-callback fail-open tests, dedicated fixture scoping and measured Prebid artifact-size handling. The runtime-boundary amendment supersedes its original-target/original-wire acceptance assumptions. File maps were checked against the checkout and consolidated phase paths.
+
+This historical review approves the baseline plan's alignment and task ownership, not completed implementation or the subsequent amendment. Under the proposed runtime-boundary amendment, F0 still requires a reviewed upstream revision and actual resolved-runtime evidence, but original-path/header recovery is no longer a release gate. Full runtime/CI and supported-origin browser acceptance remain required during execution.
+
+### Independent runtime-boundary amendment review
+
+Two independent read-only subagents reviewed the amended spec and this single
+plan together on 2026-10-05. The spec reviewer covered the complete normative
+contract, path/auth precedence, runtime failure boundary, cookies and acceptance
+criteria. The plan reviewer checked the actual EdgeZero ingress API and task
+coverage across adapters, capture gates, lifecycle, schema validation, UI,
+storage, export and size bounds.
+
+The review found one status alignment gap: encoded namespace aliases needed an
+explicit enabled `400` in the spec to match F2. That was corrected with the
+existing authentication and disabled-feature precedence preserved. Document
+cleanup added acceptance matrix rows 16–17, corrected the source/execution
+boundary and stale raw-wire/first-PR wording, and clarified that unrelated-pair
+tolerance applies after aggregate cookie checks. Both reviewers re-read the
+corrections and approved with no remaining findings.
+
+This is approval of document/API alignment. The user subsequently accepted the
+amendment and authorized implementation on 2026-10-05; no implementation,
+immutable dependency pin, runtime/browser acceptance or CI pass is claimed by
+this review.
+
+## Phase 1: Foundation
+
+### Inputs, scope, and dependency
+
+Read the shared sections above and approved [spec](../specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md) sections 5.1–5.4, 6.1, 8, 9.1–9.2, 11–14.2, and 17.1. Execute in the current branch. This phase supplies setup/lifecycle behavior; the feature is released only after combined reports and the viewer are complete in phase 3.
+
+Pinned EdgeZero v0.0.8 cannot satisfy method-independent pre-router dispatch. F0 is an actual prerequisite, not a suggested optimization. The EdgeZero feature branch implements the required hook and metadata API; a reviewed immutable upstream revision and resolved-runtime verification are still needed before adapter integration. If that revision is unavailable, finish the pure core/browser tasks and report adapter integration blocked. Original-target SDK recovery is not required. Never substitute ordinary middleware, enumerate only standard HTTP methods, edit the Cargo cache, or claim parity from Fastly alone.
+
+### File map
+
+| Action | Exact path                                                                                                                                                                                                                                                                                                                                                                 | Responsibility                                                                                                                             |
+| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
+| Modify | `Cargo.toml`, `Cargo.lock`                                                                                                                                                                                                                                                                                                                                                 | Reviewed EdgeZero revision supporting pre-dispatch, trusted origin and conservative request metadata                                       |
+| Modify | `crates/trusted-server-core/src/lib.rs`                                                                                                                                                                                                                                                                                                                                    | Export only adapter-facing `trace` entry points                                                                                            |
+| Create | `crates/trusted-server-core/src/trace/mod.rs`                                                                                                                                                                                                                                                                                                                              | Base/document gates, frozen runtime-visible cookie inspection, adapter-facing responder and typed terminal trace-response marker           |
+| Create | `crates/trusted-server-core/src/trace/types.rs`                                                                                                                                                                                                                                                                                                                            | Versioned public network/context/cookie types; no request structs                                                                          |
+| Create | `crates/trusted-server-core/src/trace/cookies.rs`                                                                                                                                                                                                                                                                                                                          | Read-only bounded exact-name scanner                                                                                                       |
+| Create | `crates/trusted-server-core/src/trace/context.rs`                                                                                                                                                                                                                                                                                                                          | IP masking and bounded optional network projection                                                                                         |
+| Create | `crates/trusted-server-core/src/trace/routes.rs`                                                                                                                                                                                                                                                                                                                           | Reserved-path classifier, POST controls/body validation, response hardening                                                                |
+| Create | `crates/trusted-server-core/src/trace/shell.rs`                                                                                                                                                                                                                                                                                                                            | Escaped fixed shell/setup facts, data-URL favicon, external asset references                                                               |
+| Modify | `crates/trusted-server-core/src/integrations/gpt_diagnostics.rs`                                                                                                                                                                                                                                                                                                           | Default-off option, shared set/clear cookie policy, effective document gate integration                                                    |
+| Modify | `crates/trusted-server-core/src/config.rs`                                                                                                                                                                                                                                                                                                                                 | Raw-config validation on deploy/runtime paths                                                                                              |
+| Modify | `crates/trusted-server-core/src/auth.rs`                                                                                                                                                                                                                                                                                                                                   | Bounded trace-auth failure logging without changing canonical first-match handler selection                                                |
+| Modify | `crates/trusted-server-core/src/publisher.rs`                                                                                                                                                                                                                                                                                                                              | Request-scoped early trace bootstrap, never cached template/ESI data                                                                       |
+| Modify | `crates/trusted-server-core/src/html_processor.rs`                                                                                                                                                                                                                                                                                                                         | Place request-scoped trace bootstrap before synchronous TSJS initialization                                                                |
+| Modify | `crates/trusted-server-adapter-fastly/src/app.rs`, `crates/trusted-server-adapter-fastly/src/main.rs`                                                                                                                                                                                                                                                                      | Hook registration and native ingress snapshot capture before conversion                                                                    |
+| Modify | `crates/trusted-server-adapter-axum/src/app.rs`, `crates/trusted-server-adapter-cloudflare/src/app.rs`, `crates/trusted-server-adapter-spin/src/app.rs`                                                                                                                                                                                                                    | Same hook for production and `routes_with_settings` seams                                                                                  |
+| Modify | `crates/trusted-server-adapter-fastly/src/platform.rs`, `crates/trusted-server-adapter-axum/src/platform.rs`, `crates/trusted-server-adapter-cloudflare/src/platform.rs`, `crates/trusted-server-adapter-spin/src/platform.rs`                                                                                                                                             | Read-only existing client/geo sources, trusted scheme/authority adapter mapping                                                            |
+| Create | `crates/trusted-server-js/lib/src/trace/types.ts`, `crates/trusted-server-js/lib/src/trace/lifecycle.ts`, `crates/trusted-server-js/lib/src/trace/viewer.ts`, `crates/trusted-server-js/lib/src/trace/viewer.css`                                                                                                                                                          | Setup/state/retry UI and shared lifecycle contracts, no report parser yet                                                                  |
+| Create | `crates/trusted-server-js/lib/test/trace/lifecycle.test.ts`, `crates/trusted-server-js/lib/test/trace/setup.test.ts`                                                                                                                                                                                                                                                       | Setup actions and observed-state tests                                                                                                     |
+| Modify | `crates/trusted-server-js/lib/build-all.mjs`, `crates/trusted-server-js/build.rs`, `crates/trusted-server-js/src/lib.rs`                                                                                                                                                                                                                                                   | Separate fixed trace build/embedding pipeline                                                                                              |
+| Modify | `crates/trusted-server-js/Cargo.toml`                                                                                                                                                                                                                                                                                                                                      | Reuse workspace serde_json as a build dependency for the committed JSON asset manifest                                                     |
+| Create | `crates/trusted-server-js/src/trace_assets.rs`, `crates/trusted-server-js/lib/trace-assets-manifest.json`, `crates/trusted-server-js/lib/test/trace-assets.test.mjs`                                                                                                                                                                                                       | Versioned immutable bytes, digest verification, asset lookup independent of integration discovery                                          |
+| Modify | `crates/trusted-server-js/lib/.prettierignore`, `crates/trusted-server-js/lib/eslint.config.js`                                                                                                                                                                                                                                                                            | Exclude frozen generated asset bytes from source format/lint rewrites                                                                      |
+| Create | `crates/trusted-server-js/lib/trace-assets/v1.js`, `crates/trusted-server-js/lib/trace-assets/v1.css`                                                                                                                                                                                                                                                                      | Frozen release bytes retained across later asset-set builds                                                                                |
+| Create | `crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts`, `crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts`, `crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml`                                                                                                                      | Dedicated setup endpoint/browser runtime, independent of shared disabled fixtures                                                          |
+| Modify | `crates/trusted-server-integration-tests/browser/helpers/infra.ts`, `crates/trusted-server-integration-tests/browser/helpers/state.ts`, `crates/trusted-server-integration-tests/browser/global-setup.ts`, `crates/trusted-server-integration-tests/browser/global-teardown.ts`, `scripts/integration-tests-browser.sh`, `scripts/generate-integration-viceroy-configs.sh` | Launch and clean up dedicated enabled/disabled trace test runtime                                                                          |
+| Modify | `crates/trusted-server-integration-tests/tests/parity.rs`                                                                                                                                                                                                                                                                                                                  | Common route/auth/order/body contract across native adapter seams                                                                          |
+| Modify | `crates/trusted-server-integration-tests/tests/integration.rs`, `crates/trusted-server-integration-tests/tests/common/config.rs`                                                                                                                                                                                                                                           | Isolated real Workers transport/config regression using `crates/trusted-server-integration-tests/tests/environments/cloudflare.rs` harness |
+| Modify | `trusted-server.example.toml`, `docs/guide/integrations/gpt-diagnostics.md`, `docs/guide/configuration.md`                                                                                                                                                                                                                                                                 | Default-off config, disclosure/transport/session/auth limits                                                                               |
+
+Rust unit tests live beside each changed module. Do not add a new crate, UI framework, cookie parser dependency, or generic diagnostics subsystem.
+
+### F0: Supply the missing EdgeZero pre-dispatch seam
+
+**External files:** `crates/edgezero-core/src/router.rs`, `crates/edgezero-core/src/request.rs` or the existing request-metadata module, and `crates/edgezero-adapter-{fastly,axum,cloudflare,spin}/src/request.rs` in the upstream EdgeZero repository. These are not Trusted Server paths. Use a separate temporary source checkout only for dependency investigation; leave the Trusted Server branch intact.
+
+- [ ] Confirm the existing pinned router with `cargo metadata --format-version 1 --no-deps` and `rg -n 'edgezero|v0.0.8' Cargo.toml Cargo.lock`. Expected: all EdgeZero packages resolve consistently to the current tag/revision before changes.
+- [ ] Verify the reviewed upstream implementation of the hook API below. The EdgeZero feature branch now supplies `RouterBuilder::pre_dispatch_hook`, `PreDispatchHook` and bounded `RequestIngress` metadata; it has not yet supplied an immutable reviewed dependency revision. Retain `RouterService` as the concrete return type so existing adapter runners and test seams use it.
+
+```rust
+#[async_trait::async_trait(?Send)]
+pub trait PreDispatchHook: Send + Sync + 'static {
+    async fn handle(
+        &self,
+        request: &mut Request,
+    ) -> Result, EdgeError>;
+}
+```
+
+- [ ] Verify upstream regressions for no hook, continuation mutation, early response, async empty stream inspection, shared hook across router clones, and an unregistered extension method. Both method and path must be inspected before `find_route`, state/introspection insertion, `RequestContext::new`, and ordinary middleware. Hook errors stop routing. Trace policy failures must instead return `Ok(Some(hardened_response))` to retain the exact response contract through the generic router.
+- [ ] Verify `RouterBuilder::pre_dispatch_hook` stores `Arc` and runs first in `RouterInner::dispatch`. `Some(response)` terminates; `None` preserves the possibly updated request for ordinary dispatch. Existing routers remain behavior-identical when no hook is installed.
+- [ ] Verify runtime-visible method preservation in every converter. Cloudflare must use `req.inner().method()` and parse it with `http::Method::from_bytes`, not Workers `req.method()` (which maps unknown tokens to GET). Its browser contract constructs a `web_sys::Request` extension token and proves conversion. Application-seam tests require authenticated hardened 405 and Allow for that token on every exact trace route. Real Worker wire tests instead record pre-invocation rejection if the parser returns 501; never claim those two tests prove the same outcome.
+- [ ] Consume `RequestIngress::{origin,header_fidelity}` without a parallel metadata type. Route decisions use the converted `Request::uri().path()`, not `CapturedTarget` or guessed original bytes. Original-target metadata may remain unavailable/transformed/over cap without rejecting an inspectable pathname. Canonical origin still requires adapter-owned `RequestIngress.origin()`, never `RequestInfo` forwarding-header fallbacks. Verify literal/encoded dot/separator and absolute/origin-form transport observations, including normalization into and out of trace.
+- [ ] Verify Fastly's early snapshot captures origin before native mutations and is carried through `into_core_request_with_ingress`. Classify its SDK-visible pathname before shortcuts. Raw original target remains NotExposed and no SDK getter work is required for v1. Test `/_ts/trace/../../health` and `/_ts/trace/../debug/ja4` as ordinary paths when the runtime normalizes them outside trace; test normalizations into trace and visible reserved ambiguities against full local auth/hardening. Record existing health/JA4 auth limitations rather than asserting trace authentication on those ordinary requests.
+- [ ] Verify runtime-visible cookie/control semantics against the resolved graph. Worker strings are UTF-8 and must copy via `HeaderValue::from_bytes(value.as_bytes())`; do not cast scalar values to bytes. Cookie-specific non-preserved octets plus U+FFFD, or non-preserved multiplicity plus comma, make all four states unavailable/runtime_header_ambiguous. Unknown/missing metadata alone permits marker-free sessions. Check exact visible reserved-name counts/duplicate precedence and visible byte bounds; global/same-name order and original-wire reconstruction are not prerequisites. Validate complete Origin/action/Fetch Metadata values and reject duplicates/folded values, never comma-split controls. Expose no raw bytes or parser messages.
+- [ ] Run exact raw-byte Cookie/control tests: valid session plus original FF versus original valid EF BF BD, comma-coalesced repeated reserved cookies, visible duplicates, and visible header caps. Include actual isolated Workers transport beyond synthetic HeaderValue tests. Record runtime-rejected, conversion-failed and application-handled/transformed cases separately; only the latter can assert CookieHealth and hardened trace responses. Use Rust or existing native/shell tooling, with no Python shipped.
+- [ ] In the upstream checkout run `cargo test -p edgezero-core pre_dispatch` and its adapter request-conversion tests. Expected: all new cases pass and old no-hook routing tests remain green. For Cloudflare’s browser-gated `tests/contract.rs`, reproduce the pinned upstream workflow: install `wasm-bindgen-cli` at the version resolved from its Cargo.lock, establish the required browser/WebDriver, then run `CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner cargo test -p edgezero-adapter-cloudflare --features cloudflare --target wasm32-unknown-unknown --test contract`. A host-only run or a missing browser/runner is not conversion evidence; record it as pending.
+- [ ] Obtain a reviewed immutable upstream release/revision exposing that API, update all workspace EdgeZero dependencies consistently in `Cargo.toml`, and regenerate `Cargo.lock` with `cargo update`. Do not invent a future tag/hash or ship a local Cargo-cache edit. Record the resolved commit and dependency graph. Current local upstream evidence uses worker 0.8.3 / wasm-bindgen 0.2.122; this consumer currently resolves 0.8.5 / 0.2.126. Repeat browser converter and real Worker ingress tests against the repinned graph; do not downgrade merely to match prior evidence.
+- [ ] Run `cargo check-fastly`, `cargo check-axum`, `cargo check-cloudflare`, and `cargo check-spin`. Expected: all adapters compile against the same pin. Planned commit: `Add EdgeZero pre-dispatch support for reserved trace routes`.
+
+**Amended capability gate:** Keep runtime rejection, adapter bootstrap/conversion failure and application-visible handling distinct. The proposed spec now accepts normalization outside trace as ordinary traffic and defines conservative Cookie ambiguity suppression. Original-target support remains documented external work, not a v1 blocker. F0 remains open until a reviewed immutable EdgeZero revision is adopted, converters and control/cookie semantics pass on the resolved graph, and safe normal requests work on every adapter. F4 additionally proves the trace terminal path bypasses ordinary lifecycle/finalizers. The hook does not precede adapter bootstrap/body buffering; no runtime rejection is falsely credited with a local response.
+
+### F1: Validate default-off configuration and unify cookie policy
+
+**Files:** `integrations/gpt_diagnostics.rs`, `config.rs`, `trusted-server.example.toml` under the paths in the file map.
+
+- [ ] Write `trace_config_requires_enabled_on_both_validation_paths`: trace=true with enabled=false or omitted must fail both `validate_settings_for_deploy` and `validate_settings_for_runtime`. Add accepted flag=false/true cases and disabled unknown-field rejection.
+- [ ] Run `cargo test-fastly trace_config`. Expected red: field rejected as unknown or the disabled-gate combination is not validated; inspect the actual failure.
+- [ ] Add `#[serde(default)] pub trace_page_enabled: bool` to `GptDiagnosticsConfig`. Add a schema check for the dependency and a raw-config hook next to `validate_js_asset_proxy_config`; explicitly deserialize/validate even when `enabled` is false. Invoke it in both validation paths before enabled integration lookup.
+- [ ] Write query/endpoint policy tests for enable→query enable, query enable→enable, repeated explicit activation, either clear surface, no ordinary-request refresh, and feature-disabled query activation. Expected set header: `__Host-ts-console=1; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800`; clear uses `Max-Age=0`, never Domain.
+- [ ] Run `cargo test-fastly gpt_diagnostics`. Confirm a lifetime regression fails, then centralize the set/clear policy in that module for both writers. Keep the existing action enum/API where possible; update its obsolete browser-session docs.
+- [ ] Re-run both filters and `cargo clippy-fastly`. Expected: green; config validation cannot silently log-and-disable invalid enabled settings. Planned commit: `Add validated trace configuration and bounded diagnostics cookie policy`.
+
+### F2: Implement exact reserved classification and hardened responses
+
+**Files:** `trace/routes.rs`, `trace/mod.rs`, `trace/shell.rs`, `lib.rs`; reuse `ec/admin.rs` and `response_privacy.rs`; make only the bounded trace-auth logging adjustment in `auth.rs`, without broad refactoring.
+
+- [ ] Write table tests covering shell, state, enable, end, exact v1 assets, trailing slash, extra segments, lookalikes, repeated separators, encoded/repeatedly encoded separators, dot segments, decode-budget exhaustion, and unrelated paths. Reuse the four-round fixed-point decoding pattern in `deny_admin_diagnostic_fallback`.
+- [ ] Run `cargo test-fastly trace::routes`. Expected red: missing classifier/responder, then specific status mismatches once compiled.
+- [ ] Implement classification returning `NotTrace`, a supported route, or a reserved-path rejection from the application-visible `req.uri().path()`. Use bounded decoded variants only to reserve/reject aliases, never to serve them. Still-visible malformed encodings/dot ambiguities return 400 after auth/flag precedence; other reserved lookalikes return 404. Normalized-out paths continue as ordinary traffic; normalized-in exact trace paths receive full validation. Ignore original-target availability for decisions.
+- [ ] Define the shared responder sequence: classify → `enforce_basic_auth` with the canonical request path/settings → harden challenge/error → flag check → reserved-path error → method check → route-specific handling. No body read or setup projection precedes authentication. Disabled requests are 404 after auth.
+- [ ] Preserve existing first-match `Settings::handler_for_path(req.uri().path())` selection; neither decoded aliases nor original-target metadata creates a second auth-policy interpretation. Pin `/%5Fts/trace`: under `^/`, auth challenges before the local error; under a lone `^/_ts`, the unmatched alias is rejected without setup data/actions (400 when enabled, 404 when disabled). Set a diagnostic auth-log-policy extension before synchronous `enforce_basic_auth`; its wrong-credential log otherwise prints the full path. Emit a fixed trace-auth failure category while ordinary callers retain logging/authorization-marker behavior. Test a fictional secret-path sentinel never enters body/logs.
+- [ ] Implement the complete local route table below. HEAD uses GET status/headers but always drops the body; HEAD on actions returns bodyless 405.
+
+| Path                       | Methods   | Enabled response                           | Allow on 405 |
+| -------------------------- | --------- | ------------------------------------------ | ------------ |
+| `/_ts/trace`               | GET, HEAD | 200 escaped setup shell                    | `GET, HEAD`  |
+| `/_ts/trace/state`         | GET, HEAD | 200 exact `{ "observed_active": boolean }` | `GET, HEAD`  |
+| `/_ts/trace/enable`        | POST      | 200 bounded mutation-requested result      | `POST`       |
+| `/_ts/trace/end`           | POST      | 200 bounded mutation-requested result      | `POST`       |
+| `/_ts/trace/assets/v1.js`  | GET, HEAD | Fixed JS bytes                             | `GET, HEAD`  |
+| `/_ts/trace/assets/v1.css` | GET, HEAD | Fixed CSS bytes                            | `GET, HEAD`  |
+
+- [ ] Harden every local dynamic/error/challenge response, including assets errors: private/no-store, removal of shared/edge cache directives, nosniff, no-referrer, Permissions-Policy and exact spec CSP. Dynamic shell is HTML UTF-8; state/actions are JSON UTF-8. Build only bounded errors, no raw `Report` text in bodies/logs. Retain `WWW-Authenticate` on 401 and `Allow` on 405.
+- [ ] Serve successful assets with correct JS/CSS MIME, nosniff, strong byte-derived ETag. If the selected handler has auth, use private/no-store even with accepted credentials; otherwise use public/max-age=31536000/immutable. Do not reflect query/report data into assets.
+- [ ] Add regression tests for broad auth, earlier narrow-handler shadowing, disabled/auth order, protected assets, extension methods, HEAD bodylessness, and query nonreflection. Re-run filter and clippy. Planned commit: `Add authenticated local trace routing and response hardening`.
+
+### F3: Inspect runtime-visible cookies and project request facts
+
+**Files:** `trace/types.rs`, `trace/cookies.rs`, `trace/context.rs`, `trace/mod.rs`; canonical validators in `ec/generation.rs` and `ec/prebid_eids.rs` are reused.
+
+- [ ] Write scanner tests for all four reserved names, absent/valid/invalid, duplicate precedence, multiple header fields, additional equals in values, unrelated malformed names, exact malformed reserved tokens, oversized values, exactly/over 16 KiB aggregate, and non-UTF-8 headers.
+- [ ] Add a table covering each non-preserved status plus missing metadata: marker-free session and empty collection remain inspectable; comma anywhere with non-preserved multiplicity or U+FFFD anywhere with non-preserved octets makes all four unavailable/runtime_header_ambiguous. Test independent per-Cookie overrides, unrelated/quoted markers, exact visible duplicates, Preserved-axis marker semantics, valid other multibyte UTF-8 and original invalid bytes. Current adapters' common Unknown is not a reason to disable normal sessions.
+- [ ] Run `cargo test-fastly trace::cookies`. Expected red; implement a read-only frozen visible-field scan, not CookieJar. Sum as_bytes lengths (including introduced separators); above 16,384 yields header_too_large, then actual invalid UTF-8 yields header_not_utf8, then markers yield runtime_header_ambiguous. Use str::from_utf8, not HeaderValue::to_str. Get per-Cookie axes from RequestIngress::header_fidelity(&COOKIE); missing metadata is unproved. Do not comma-split. Preserve exact names/count malformed reserved occurrences before canonical value parsing. Order does not affect counts. Test combined failures pin this precedence and suppress the capture gate.
+- [ ] Enforce visible UTF-8 per-value byte caps 512 EC, 8192 EIDs, 16 tester/session. Reuse canonical validators and literal tester=true/session=1. Add runtime_header_ambiguous to the exact Rust detail enum, legal only with unavailable; emit source: request, no values/parser text/fidelity metadata. After aggregate checks, duplicate wins over individual validity; absent has no detail. These bounds/counts do not reconstruct discarded original wire information.
+- [ ] Write `trace_context_masks_and_omits_forbidden_fields` using `192.0.2.129` and `2001:db8:1234:5678::1`: expect a deterministic /24 and /48 display mask, never the full addresses. Put distinct fictional sentinels in fingerprints, path/query, user/auction IDs, headers and geo city/coordinates; serialized output must exclude them.
+- [ ] Run `cargo test-fastly trace::context`. Implement the exact section 9.1 schema with strict RFC3339 UTC capture time and only optional allowlisted network members. Country ≤2 ASCII; region/POP/protocol/cipher ≤32 UTF-8 bytes; edge hostname/region ≤128. Reject invalid Unicode/control/bidi values, omit bad optional fields with bounded categories; do not silently shorten or log values.
+- [ ] Project existing `RuntimeServices.client_info` and read-only geo: Fastly IP/TLS/server metadata/country/region, Axum trusted IP, Cloudflare trusted IP/country, Spin trusted IP. ASN stays absent unless actually populated. No speculative HTTP/POP mapping in this increment.
+- [ ] Re-run both filters. Tests use panic/counting KV/auction/filter/sink mocks to prove inspection has no identity or telemetry side effects. Planned commit: `Add bounded read-only trace request and cookie projections`.
+
+### F4: Integrate the hook on every adapter before ordinary processing
+
+**Files:** all four `src/app.rs`, Fastly `src/main.rs`, platform metadata mapping, adapter-local tests, `tests/parity.rs`.
+
+- [ ] Write dispatch tests using each adapter's actual `routes_with_settings` or equivalent build seam with injected counting/panic services. Include exact/malformed/disabled routes, HEAD, extension methods, and auth failures. All ordinary lifecycle/filter/KV/auction/origin/sink counters must remain zero. Optional read-only geo is allowed only after auth.
+- [ ] Run `cargo test-fastly trace_dispatch`, `cargo test-axum trace_dispatch`, `cargo test-cloudflare trace_dispatch`, and `cargo test-spin trace_dispatch`. Expected red until every route construction registers the hook.
+- [ ] In Fastly main.rs capture RequestIngress before mutations and classify the SDK-visible pathname before native shortcuts. Only NotTrace may use health/JA4; runtime-normalized paths outside trace explicitly retain ordinary behavior. Carry the snapshot and consistent visible-path classification into the core responder. Real Viceroy raw-client tests cover normalizations out to health/JA4 and into trace, still-visible reserved ambiguity, exact routes, disabled/auth precedence and zero ordinary effects for trace. Do not wait for a raw-target SDK accessor or assert auth on normalized-out health/JA4.
+- [ ] Register one hook capturing each adapter's `Arc`, before EdgeZero method routing. Consume `RequestIngress` facts and a lazy read-only metadata supplier; canonical origin must not use `RequestInfo` forwarding-header fallbacks. Capture Fastly metadata before native mutation/shortcuts and pass the same snapshot through `into_core_request_with_ingress` into the existing direct `oneshot` seam. Return every trace policy rejection as `Ok(Some(hardened_response))`, not a propagated generic `EdgeError`. Do not construct the full EC/event pipeline for setup facts. Preserve harmless transport metadata needed for trusted IP resolution; sanitize trust-secret headers on continuing ordinary requests using existing policy.
+- [ ] On a non-trace continuation only when trace config is enabled, freeze CookieHealth and incoming-session validity before prepare_request. Compute the base gate once from config plus runtime-visible inspection; unavailable/invalid/duplicate/absent gives false. After existing diagnostics decision, compute document eligibility without rereading sanitized cookies. Retain no raw values; suppress trace tokens/evidence/sidecars on ambiguity while ordinary advertising/console behavior remains unchanged.
+- [ ] Mark all reserved local responses, including 401/disabled/path/method errors and successful assets, with a core `TraceTerminalResponse` extension. Fastly main must detect it before `apply_entry_point_finalize_headers` and bypass that ordinary finalizer: it performs geo lookup and applies operator headers that could otherwise replace CSP/MIME or add cookies after the early responder. Use only a trace-specific terminal pass to preserve the complete route header/cache/body contract through native conversion; do not insert `EcFinalizeState` or `RequestFilterEffects`. Add entry-point regressions with hostile CSP/Content-Type/Set-Cookie/cache overrides: auth failure performs zero geo lookups, GET/HEAD never gain Set-Cookie, action cookies remain exactly the accepted policy, and Allow/WWW-Authenticate/security headers survive. Preserve native adapter startup/runtime config validation and `routes_with_settings_and_services` seams.
+- [ ] Add native parity tests exercising arbitrary methods, malformed paths, auth matrices, missing optional facts, and no-Content-Type streamed POSTs. Fastly is checked separately under Viceroy; the native parity test does not execute Fastly.
+- [ ] Run all four trace_dispatch filters and `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity trace`. Add isolated actual-runtime tests in tests/integration.rs using CloudflareWorkers and CLOUDFLARE_WRANGLER_DIR; parameterize tests/common/config.rs without changing disabled baseline fixtures. Run the Cloudflare build.sh and `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test integration trace_runtime_boundary -- --ignored --test-threads=1`. A raw client preserves duplicate/invalid bytes where reqwest/fetch cannot. Pin extension-method wire rejection (local 501), FF replacement, comma joining and normal safe GET/POST flows separately from application-seam hardened 405 tests. Add equivalent Fastly/Spin evidence, including pre-component invalid-byte rejection and conversion-failure counters. Only successful conversion can assert local status/Allow/hardening/CookieHealth; preserve no trace-specific writes/capture for failures. Planned commit: `Intercept trace requests before all adapter lifecycles`.
+
+### F5: Enforce deliberate actions and observe state separately
+
+**Files:** `trace/routes.rs`, `trace/mod.rs`, adapter body-parity tests.
+
+- [ ] Write same-origin action tests: missing/duplicate/conflicting action, Origin, Fetch Metadata, Host/authority/scheme; cross-site Origin; credentials/path/query/fragment in Origin; case/default-port canonicalization; forbidden action query; valid enable/end and repeated actions.
+- [ ] Run `cargo test-fastly trace_actions`. Implement exact single control-header validation: correct `X-TS-Trace-Action`, exact `Sec-Fetch-Site: same-origin`, and one HTTP(S) Origin canonically equal to trusted inbound origin. Lowercase host/remove default ports for both; never trust arbitrary forwarding headers. Rejection is 403 with no mutation.
+- [ ] Write body tests for missing/zero/invalid/positive Content-Length, any application-visible Transfer-Encoding, nonempty `Body::Once`, clean streamed EOF, empty chunks before EOF/nonempty chunk, and stream error. Assert 413 for invalid/positive length, encoding or actual body bytes, 400 for stream error, hardening and no Set-Cookie. Test identical zero lengths folded by the transport as one zero length plus an actually empty body; the spec does not require transport-level 413 for identical duplicates. Record conflicting-framing parser rejection separately rather than claiming the application returned a hardened response.
+- [ ] Implement header prechecks, then explicitly match Body::Once/Stream. Only empty Once or clean stream EOF is accepted. Reject the first nonempty streamed chunk without continuing consumption. Do not use `into_bytes().unwrap_or_default()` or forward `into_bytes_bounded(0)` errors; those cannot implement the prescribed 413 behavior.
+- [ ] For accepted action return the shared set/clear header independently of incoming CookieHealth; ambiguous cookies must not prevent a valid end action. State GET uses the shared frozen inspection and returns true only for exactly one present_valid/valid_diagnostics_value session. Absent/invalid/duplicate/unavailable is false, never cookie-absence proof. Test accepted enable/end with ambiguous cookies and separate unconfirmed activation observation. GET/HEAD never mutate cookies or identity; strict action/Origin/Fetch Metadata checks remain unchanged.
+- [ ] Run `cargo test-fastly trace_actions`, all four adapter `trace_dispatch` filters, and parity. Expected: no mutation on any rejection, including bodiless fetch without Content-Type through Axum streaming. Planned commit: `Add same-origin trace actions and observed session state`.
+
+No transport deadline/one-byte read/408 guarantee is added. Fastly, Cloudflare, and Spin pre-buffer; Axum streams non-JSON; body acceptance here cannot bound earlier transport allocation. Document that deployment limitation.
+
+### F6: Build versioned setup assets and verified lifecycle UI
+
+**Files:** `trace/shell.rs`, JS `src/trace/{types,lifecycle,viewer}.ts`, `viewer.css`, setup tests, asset build/embedding files from the map.
+
+- [ ] Write lifecycle tests for activation success followed by active state, mismatch, failed verification, deactivation success/inactive, missing cookie, retry, rejected POST, and no added history entry. Assert POST is same-origin, bodyless, `credentials: 'same-origin'`, cache=no-store, with fixed action header; state GET is a distinct no-store request.
+- [ ] Add setup/context rendering for unavailable/runtime_header_ambiguous with plain text explaining unreliable runtime-visible cookie inspection. An ambiguous follow-up state stays inactive and activation unconfirmed; successful end with inactive observation says no valid session observed, not cookie absent. Retain independent local cleanup/retry behavior.
+- [ ] From `crates/trusted-server-js/lib` run `npx vitest run test/trace/lifecycle.test.ts test/trace/setup.test.ts`. Expected red. Implement `changeTraceSession(action: 'enable' | 'end')` returning mutation and observation results separately. Never equate POST success with confirmed state.
+- [ ] Implement shell/setup UI from section 6.1: title, setup-request network/cookie facts, on/off state, reproduce-after-enable instructions, same-host/same-tab/history fallback, explicit reload guidance, 44 px enable/end/back controls and aria-live status. No previous-page URL inference, target parameter, automatic activation, or GET cookie mutation. Facts are escaped text nodes, not inline executable JSON; viewer reads fixed element attributes/text.
+- [ ] Emit viewer JS as `dist/trace/v1.js` and CSS as `dist/trace/v1.css` separately from `tsjs-*.js`. Extend the existing Vite build with a standalone trace entry; no new integration registration or concatenation of viewer code into inactive publisher bundles.
+- [ ] Add workspace `serde_json` to `crates/trusted-server-js/Cargo.toml` build-dependencies to parse the committed JSON manifest; this reuses an existing dependency rather than introducing another parser. Extend `build.rs` to embed assets independently, verify committed SHA-256 digests from `lib/trace-assets-manifest.json`, and fail closed for missing/stale assets under `TSJS_SKIP_BUILD=1`. Expose narrow static asset metadata through `src/trace_assets.rs`. Exclude only generated frozen `trace-assets/**` from JS Prettier and ESLint source rules via `lib/.prettierignore` and `lib/eslint.config.js`; digest tests govern those bytes and source TypeScript/CSS remain formatted/linted. Store published bytes in versioned source-controlled files `crates/trusted-server-js/lib/trace-assets/v1.js` and `v1.css`; manifest/digests alone cannot preserve old routes' bytes. Build output must match these frozen bytes at release, and future changes add a new version.
+- [ ] Add a Vitest `test/trace-assets.test.mjs` with `// @vitest-environment node`, following the existing build-artifact tests, checking rebuilt bytes/digests, exact lookup URLs, no dynamic/request data and shell references. Run `node build-all.mjs`, `npx vitest run test/trace-assets.test.mjs`, setup Vitest tests, and `cargo test-fastly trace::routes`. Expected green. Planned commit: `Add versioned mobile trace setup assets and session controls`.
+
+Keep v1 unpublished through all mandatory phases and update its draft digest/byte fixture with each source change. Freeze only the complete viewer set at first publication after V7. There is no planned interim setup publication or automatic v2 transition.
+
+### F7: Inject the literal document gate and request context privately
+
+**Files:** `publisher.rs`, `html_processor.rs`, `trace/mod.rs`, `integrations/gpt_diagnostics.rs`, publisher/unit cache regressions.
+
+- [ ] Write gate matrix tests: valid incoming cookie + trace flag + active decision succeeds; missing cookie/query-enable only, query-disable, duplicate/invalid directives, prefetch, bot/ineligible navigation, trace flag=false, and nonboolean browser globals cannot activate document tracing.
+- [ ] Run `cargo test-fastly trace_document`. Thread a request-scoped optional trace bootstrap through `ProcessResponseParams`, `OwnedProcessResponseParams` and `HtmlProcessorConfig`; use the existing head injection point in `html_processor.rs` beside the diagnostics bootstrap, before the unified TSJS tag. After the effective diagnostics decision, inject `window.__tsjs_trace_active=true` and the immutable redacted context before TSJS initializes, only on eligible documents. Expose context as `window.__tsjs_trace_request_context` with the `TraceRequestContextV1` contract; use the existing script-safe serializer/escaper, not raw executable concatenation. Missing/failed context projection leaves advertising intact and causes a bounded capture failure later.
+- [ ] Keep tokens/context out of shared HTML templates and ESI fragments. Add them only in request-scoped injection/body seams. On inactive publisher responses, emit no trace assets/bootstrap/listeners/cache-policy change; existing explicit console activation retains its own behavior.
+- [ ] Test projection/serialization failure, hostile `` strings, unsupported metadata, shared-template reuse, and request-specific values for two users. Test hostile response_headers and late filter overrides; retain `late_filter_effects_cannot_make_an_assembled_response_public` and reuse `apply_response_headers_with_cache_privacy`/Fastly terminal guards.
+- [ ] Run `cargo test-fastly trace_document`, `cargo test-fastly late_filter_effects_cannot_make_an_assembled_response_public`, all four adapter tests, and relevant clippy gates. Expected: private/no-store terminally, no shared bytes contain trace context, advertising preserved. Planned commit: `Inject eligible request-scoped trace context with terminal privacy`.
+
+### F8: Document and verify the foundation review unit
+
+**Files:** example TOML and operator docs in the map; all changed files for verification.
+
+- [ ] Document default-off option beside `enabled`, same-origin script visibility of HttpOnly cookie health/opaque auction outcomes, potentially personal masked IP/coarse geo, and explicit operator disclosure acceptance.
+- [ ] Document first-match-wins auth and public mobile deployment scope, cookie/session caveats, host-only origin scope, browser/service-worker trust limits, transport pre-buffering, local 400/401/403/404/405/413, and no timeout guarantee.
+- [ ] Document the three runtime-boundary outcomes, normalized-out ordinary health/JA4 behavior, literal visible-path auth for rejected encoded aliases, visible cookie bounds and conservative comma/U+FFFD false negatives. Require a suitable same-origin test deployment for Secure host-only cookies; never weaken cookie attributes or infer HTTPS from untrusted headers. F0 no longer depends on external raw-target/header reconstruction work.
+- [ ] Create a dedicated setup test runtime/helper and trace TOML without changing the shared disabled integration config. Parameterize the existing `scripts/generate-integration-viceroy-configs.sh` app-config input and generate isolated trace output with its existing `generate-viceroy-config --app-config` binary; preserve baseline input/output defaults. Track extra process IDs in browser state and extend both failed-setup cleanup and `global-teardown.ts` to stop every runtime/container; preserve baseline runtime teardown. Add foundation-specific browser tests to `crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts` for read-only GET, cross-site actions, history, observed state, exact CSP/favicon, auth and inactive traffic. Run `./scripts/integration-tests-browser.sh tests/shared/mobile-trace.spec.ts` for this foundation checkpoint, plus affected adapter builds/clippy/docs and the trace parity suite. V7 owns the single complete release gate run. The later viewer phase extends these fixtures; foundation verification does not wait for the viewer.
+- [ ] Review diff for accidental real data, raw logs, unrelated refactors, and unimplemented upstream promises. Planned commit: `Document and verify trace setup privacy and adapter parity`.
+- [ ] Checkpoint the default-off setup phase with its test evidence and explicit dependency/mobile gaps, then continue E1; do not publish interim assets or declare the full feature complete.
+
+## Phase 2: Live auction evidence
+
+### Inputs and invariants
+
+Depends on foundation tasks F0–F8. Read [spec](../specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md) sections 5.2, 9.3–9.5, 10, 12, 13, and 14.1–14.4. Keep the existing branch and follow the shared verification's verification/commit rules.
+
+Maintain the existing console-only `diagnostics_auction_id` behavior when trace is off; only the new trace carry/evidence/slot tokens are gated by trace. For active trace, mint the public auction token once before dispatch and use that same token in both public evidence and the existing GPT opportunity marker, including zero-bid/no-candidate paths.
+
+### File map
+
+| Action | Exact path                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | Responsibility                                                                                                      |
+| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| Create | `crates/trusted-server-core/src/trace/auction.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Typed opaque tokens, private observation carry, public evidence projection/bounds                                   |
+| Modify | `crates/trusted-server-core/src/trace/mod.rs`, `crates/trusted-server-core/src/trace/types.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Export the narrow auction contracts/hooks                                                                           |
+| Modify | `crates/trusted-server-core/src/auction/orchestrator.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | Trace-only launch-order/terminal-fact carry through dispatch/collect/abandon                                        |
+| Modify | `crates/trusted-server-core/src/auction/formats.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Permissive trace extension extraction along exact accepted-slot conversion; actual response conversion dispositions |
+| Modify | `crates/trusted-server-core/src/auction/endpoints.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | API base gate, pre-dispatch identity, response evidence and privacy                                                 |
+| Modify | `crates/trusted-server-core/src/openrtb.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Optional namespaced public response extension                                                                       |
+| Modify | `crates/trusted-server-core/src/publisher.rs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | SSAT/SPA request-scoped carry, token-bearing slot definitions, held-tail scheduler transport, legacy alias          |
+| Modify | `crates/trusted-server-js/lib/src/trace/types.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Evidence/transport/correlation types                                                                                |
+| Create | `crates/trusted-server-js/lib/src/trace/validation.ts`, `crates/trusted-server-js/lib/src/trace/collector.ts`, `crates/trusted-server-js/lib/src/trace/runtime.ts`, `crates/trusted-server-js/lib/src/trace/pending.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Strict public evidence checks, one collector facade, API unit mapping and bounded pending lifecycle                 |
+| Modify | `crates/trusted-server-js/lib/src/core/types.ts`, `crates/trusted-server-js/lib/src/core/global.d.ts`, `crates/trusted-server-js/lib/src/core/index.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Optional typed trace facade/slot extension and scheduler third argument                                             |
+| Modify | `crates/trusted-server-js/lib/src/core/auction.ts`, `crates/trusted-server-js/lib/src/core/request.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Final grouped unit tokens, direct response collection before bid parsing                                            |
+| Modify | `crates/trusted-server-js/lib/src/integrations/gpt/index.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Initial/SPA transport validation, slot identity to existing recorder                                                |
+| Modify | `crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts`, `crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts`, `crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                  | Trace-only recorder input/callback at concrete cycle binding                                                        |
+| Modify | `crates/trusted-server-js/lib/src/integrations/prebid/index.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Actual registered timeout/error hooks and request-associated evidence                                               |
+| Create | `crates/trusted-server-js/lib/test/trace/evidence.test.ts`, `crates/trusted-server-js/lib/test/trace/collector.test.ts`, `crates/trusted-server-js/lib/test/trace/pending.test.ts`, `crates/trusted-server-js/lib/test/trace/tokens.test.ts` in the same directory                                                                                                                                                                                                                                                                                                                                                                                                          | Strict bounds, coverage truth table, token/mapping/pending behavior                                                 |
+| Modify | `crates/trusted-server-js/lib/test/core/auction.test.ts`, `crates/trusted-server-js/lib/test/core/request.test.ts`; `crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts`, `crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts`, `crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts`; `crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts`, `crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/api.test.ts`, `crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/types.test.ts`; `crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts` | Behavior compatibility at existing seams                                                                            |
+| Modify | `crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | Real pinned Prebid register/newBidder callback routing                                                              |
+| Modify | `crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Live transport/attribution regression fixtures added to foundation suite                                            |
+
+New Rust tests stay in module `#[cfg(test)]` blocks. Do not serialize the telemetry observation, add fields to public GPT export, rename telemetry source vocabulary, or change provider protocols.
+
+### E1: Define opaque tokens and the strict public auction contract
+
+**Files:** `trace/auction.rs`, `trace/types.rs`, `trace/mod.rs`, JS trace types/validation, token/evidence tests.
+
+- [ ] Add tests that accept exactly the existing unhyphenated lowercase auction UUID-v4 shape and new hyphenated lowercase slot UUID-v4 shape, enforce RFC variant, and reject uppercase, whitespace, wrong UUID versions, internal IDs, and normalized alternatives.
+- [ ] Run `cargo test-fastly trace::auction` and, from `crates/trusted-server-js/lib`, `npx vitest run test/trace/tokens.test.ts test/trace/evidence.test.ts`. Expected red.
+- [ ] Add newtypes `DiagnosticAuctionId` and `TraceSlotRef` with checked parsing and strong internal ownership. Generate auction UUID via `Uuid::new_v4().simple()` and slot UUID via canonical hyphenated UUID v4. Compare exact stored bytes. Browser token validators are:
+
+```typescript
+const auctionToken = /^ts-auc-[0-9a-f]{12}4[0-9a-f]{3}[89ab][0-9a-f]{15}$/
+const slotToken =
+  /^ts-slot-[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/
+```
+
+- [ ] Define the exact `TraceAuctionEvidenceV1`/`TraceAuctionTransportV1` types from spec 9.4/9.4.1. Explicit enums cover all source, terminal reason/status, provider role/status and candidate members. Transport is a discriminated one-of: schema_version=1 plus evidence OR unavailable_reason=`evidence_projection_failed`, never both/neither.
+- [ ] Write table tests for every enum member and rejection, unknown keys at every boundary, provider ≤16, auction slots ≤64, requested sizes ≤16, token/string bounds, dimensions 1..100000, u16 ordinals/counts/counters, u32 durations, required positive ordinals and exact fixed coverage=`unavailable`.
+- [ ] Implement strict validators that copy accepted values into trace-owned models, reject extra keys/unsafe numbers/invalid Unicode/control/bidi strings, and never modify a supplied ad object. Optional duration conversion failure omits/counts; required ordinal/count or checked u16 omission overflow rejects that evidence. Browser rejects invalid core-bounded inner evidence without truncating it again.
+- [ ] Re-run tests and `cargo clippy-fastly`; expected green. Planned commit: `Define bounded public trace auction evidence and opaque tokens`.
+
+### E2: Preserve live auction facts before dispatch and through terminal paths
+
+**Files:** `trace/auction.rs`, `auction/orchestrator.rs`, publisher request/stream carries, endpoint carries.
+
+- [ ] Write `trace_auction_identity_is_stable_across_terminal_outcomes` and `trace_auction_provider_numbers_follow_launch_order` with completed, zero-bid, skipped, dispatch-failed, execution-failed and abandoned cases. Randomize completion order and legacy HashMap insertion; output order must remain launch order.
+- [ ] Run `cargo test-fastly trace_auction`. Expected red. Introduce a private optional `TraceAuctionCarry` owned by the request/dispatch carry with source, pre-dispatch token, exact accepted-slot token mapping, ordered provider observations and monotonic clock start. Instantiate only under the applicable server gate; do not use `AuctionRequest.id` or telemetry UUID.
+- [ ] Extend the existing private dispatch/collect outcomes to preserve trace facts on failures. Observe calls at their actual launch point, including dispatch failures and mediator calls; never infer missing provider observations from a `Report`. Number providers in deterministic dispatch order, not completion/HashMap order; pass only redacted summaries into projection.
+- [ ] Map terminal observations explicitly:
+
+| Live condition                                    | Status                                | Reason                                            |
+| ------------------------------------------------- | ------------------------------------- | ------------------------------------------------- |
+| Successful execution, including zero bids         | completed                             | Omit unless an actual allowlisted reason is known |
+| Consent/policy skip                               | skipped                               | policy_skipped                                    |
+| Definitive converted slot list empty              | skipped                               | no_eligible_slots                                 |
+| Unsuccessful split-path NotStarted without launch | dispatch_failed                       | no_provider_launched                              |
+| Execution failed with provider failure known      | execution_failed                      | provider_execution_failed                         |
+| Collection failure                                | execution_failed                      | collection_failed                                 |
+| Dispatched but cannot be collected/delivered      | abandoned                             | unknown or a directly observed allowlisted reason |
+| Other terminal failure                            | Corresponding observed failure status | unknown                                           |
+
+- [ ] Distinguish path semantics before mapping `NotStarted`: `run_planned_auction` already returns successful `OrchestrationResult::no_bid()` for an enabled empty plan. Successful API/SPA empty-plan execution is completed with zero providers/no candidates; it is not a dispatch failure merely because internal dispatch was NotStarted. Add separate empty-plan API/SPA success and unsuccessful split-dispatch regressions, preserving ordinary response/status behavior and explicit consent/no-slot skips.
+- [ ] Treat actual `DispatchAuctionOutcome::DispatchFailed` as dispatch_failed with provider_execution_failed when launch failures prove it, otherwise unknown; do not label NotStarted as a launch failure. Keep no-slot/policy-skipped trace observations independent of whether current telemetry constructs `AuctionObservationContext`. Do not create telemetry rows solely to get a trace identity. Abandoned telemetry's existing clamped duration and unordered lists are not a valid trace projection source; perform separate checked duration conversions.
+- [ ] Derive provider statuses/counts and role from actual live responses/pending/disposition. Provider numbers never expose names. Compute per-slot bid counts from returned bids; no provider-to-slot no-bid attribution is inferred. Preserve final `provider_to_slot_no_bid: 'unavailable'`.
+- [ ] Test that diagnostics disabled allocates no trace carry/tokens and launches exactly the same provider work. Re-run all four adapter suites after changing shared carries/constructors, plus focused filter/clippy. Planned commit: `Carry trace auction identity and live outcomes across dispatch`.
+
+### E3: Preserve exact slot associations and API acceptance compatibility
+
+**Files:** `auction/formats.rs`, `auction/endpoints.rs`, `openrtb.rs`, `trace/auction.rs`, `publisher.rs` request-scoped slot construction.
+
+- [ ] Write converted-slot mapping tests for grouped multi-bidder units, duplicate codes, skipped non-banner units, filtered inputs, missing/invalid/duplicate token values, duplicate extension members and wrong scalar/object shapes. Ordinary AdRequest success/bids must equal the baseline for every trace-only malformed input.
+- [ ] Run `cargo test-fastly trace_slot_conversion`. Extract only `adUnits[].ext.trusted_server.trace_slot_ref` permissively while retaining occurrence validity; do not add a strict serde string field that rejects requests whose unknown extension was previously ignored. Preserve duplicate-member detection with a narrow deserialize visitor if needed; do not reparse with `serde_json::Value` alone and silently lose duplicate evidence.
+- [ ] Thread the accepted unit's token alongside the exact unit-to-`AuctionRequest.slots` conversion. One-based `slot_number` uses only definitive post-conversion order, never client response association. Invalid/missing/duplicate tokens get server-generated refs; ordinary input validity is unchanged.
+- [ ] Keep trace association in a private sidecar rather than serialized `AdSlot`/provider input. Add bidder and mediator outbound JSON sentinel tests proving `trace_slot_ref`/diagnostic auction tokens never reach their requests, even if supplied as input extensions.
+- [ ] For duplicate accepted routing keys preserve distinct instance ordinals/refs. Count actual returned records matching the ordinary key, documenting shared, non-disjoint counts rather than a unique-bid total. Since winner/delivery maps cannot prove an instance-specific disposition, emit `unknown`, omit selected size and never infer an instance/GPT join. Pin this with duplicate-code regressions and viewer wording.
+- [ ] Use `OpenRtbResponseConversion` delivered-winner dispositions and actual bid-map inclusion to distinguish selected, selected_unrenderable, no_candidate and unknown. Selection alone is insufficient; malformed dimensions/creative rejection/missing price/cache fallback handling stay unchanged.
+- [ ] Add `ext.trusted_server.trace_auction` to the response only under the base gate; preserve existing orchestrator extension, status, seats/bids and headers apart from deliberate private/no-store. Projection failure returns the exact unavailable envelope, no raw error.
+- [ ] Run focused tests, `cargo test-fastly auction::endpoints`, native adapter tests, and clippy. Planned commit: `Associate trace slots through auction conversion without changing bids`.
+
+### E4: Transport SSAT and SPA evidence at request-scoped seams
+
+**Files:** `publisher.rs`, trace carry/projection, existing HTML processing; JS scheduler/SPA hooks in E6.
+
+- [ ] Write initial-navigation and both canonical/legacy page-bids tests for matching slot tokens, single shared opportunity/evidence auction token, every terminal outcome, zero bids, context gate suppression, and projection failure with unchanged normal bids.
+- [ ] Run `cargo test-fastly trace_transport`. Add tokens to exact request-scoped slot definitions while constructing the corresponding auction slots. Do not add random tokens inside generic `build_slot_json` when it is called for shared templates; use an optional request-scoped argument/map, leaving template output token-free.
+- [ ] Extend `build_seam_script`/held-tail path to serialize the optional transport with the existing script-safe JSON method and call `scheduleInitialAdInit(bids, slots?, traceAuctionTransport?)`. Record available failure/skip evidence even when no winning bid exists. If tail injection never reaches the browser, do not fabricate a failure marker; collector remains not_observed absent other evidence.
+- [ ] Add optional top-level `trace_auction` next to `slots` and `bids` in both `/_ts/page-bids` and `/__ts/page-bids`. Strip both envelope and slot extensions when gate=false; base gate applies without navigation eligibility. Initial documents additionally require effective navigation decision.
+- [ ] Assert current GPT-only marker behavior when trace flag=false. When active, reuse the pre-dispatch trace token in the existing opportunity marker instead of minting another token at collection. No candidate/failed auctions still carry the same identity when deliverable.
+- [ ] Mark every evidence-bearing response terminal private/no-store. Test late operator/cache overrides, ESI/shared-template absence, malformed HTML/serialization fail-open and old scheduler that ignores the third arg.
+- [ ] Re-run focused Rust tests, all four adapters and cache guard regression; expected unchanged baseline bids/slots/render status except optional gated public members. Planned commit: `Carry live trace evidence through initial and SPA ad responses`.
+
+### E5: Install one gated browser collector and instrument direct API requests
+
+**Files:** JS trace runtime/collector/validation, core facade/globals, `core/auction.ts`, `core/request.ts`, collector/core tests.
+
+- [ ] Write gate tests with missing/false/string/number globals while GPT diagnostics=true: no token generation, listeners, unit mappings, pending records, collector facade activation, or storage access. Test repeated integration IIFE installs share one active facade, not separate imported singletons.
+- [ ] From `crates/trusted-server-js/lib`, run `npx vitest run test/trace/collector.test.ts test/core/auction.test.ts test/core/request.test.ts`. Expected red. Instantiate the collector once through a typed `window.tsjs` facade only behind the strict literal gate. Pure validation imports are allowed; import-time trace side effects are not.
+- [ ] Accept immutable validated transport before normal bid parsing. Absent optional member adds no issue and never resets earlier records/issues; malformed supplied member adds evidence_validation_failed; supplied unavailable marker adds evidence_projection_failed. Direct non-OK/unreadable JSON/rejected fetch paths add evidence_transport_failed without retaining errors/bodies.
+- [ ] Retain newest 16 server records/newest 128 sidecars, stable arrival/emission order and checked omission counters. Server eviction adds record_evicted; sidecar-only eviction adds correlation_unavailable. No sessionStorage writes during collection.
+- [ ] Implement capture status independently of interpretation issues:
+
+```typescript
+type CaptureStatus = 'complete' | 'partial' | 'unavailable' | 'not_observed'
+
+function captureStatus(
+  recordCount: number,
+  issues: readonly string[]
+): CaptureStatus {
+  const hasCaptureIssue = issues.some((issue) =>
+    [
+      'evidence_projection_failed',
+      'evidence_transport_failed',
+      'evidence_validation_failed',
+      'record_evicted',
+    ].includes(issue)
+  )
+  if (recordCount > 0) return hasCaptureIssue ? 'partial' : 'complete'
+  return hasCaptureIssue ? 'unavailable' : 'not_observed'
+}
+```
+
+- [ ] Deduplicate/sort the six exact issue enums in their documented order, max 16. `correlation_unavailable` and `external_client_side_unobservable` never change capture status. Mark external refresh limits only when the existing browser source actually observes them.
+- [ ] Assign `crypto.randomUUID()` refs after `buildAdRequest` groups/deduplicates final units, then keep the exact request-scoped token-to-unit mapping through `sendAuction`. No Web Crypto means no client refs; ordinary request still runs. Validate echoed tokens/association without indices/raw unit codes in public evidence; record API evidence with correlation_unavailable and no GPT opportunity/sidecar.
+- [ ] Add throwing token-generator and collector-callback fixtures for both actual `/auction` callers and the scheduler/SPA boundary. Assert the ordinary request, ad initialization, bid acceptance/order and targeting still complete unchanged; diagnostic failures remain bounded and never escape into advertising.
+- [ ] Re-run focused tests; add unchanged bid-result assertions on absent, invalid and failed evidence. Planned commit: `Collect gated trace evidence without changing direct auction delivery`.
+
+### E6: Emit sidecars only at the existing SSAT/SPA recorder binding
+
+**Files:** `core/types.ts`, `integrations/gpt/index.ts`, diagnostics store/API/index, existing scheduler/SPA/recorder tests.
+
+- [ ] Write initial scheduler third-argument and SPA parser tests: validate/record before adInit, absent optional member leaves old behavior, malformed transport still initializes ads, legacy retry preserves transport, stale/superseded SPA result does not join a current cycle.
+- [ ] Run `npx vitest run test/integrations/gpt/schedule_initial_ad_init.test.ts test/integrations/gpt/spa_hook.test.ts test/integrations/gpt/ad_init.test.ts`. Add optional AuctionSlot.ext and the exact transport argument/member from spec; do not spread arbitrary extensions into trace objects.
+- [ ] Validate a delivered slot ref occurs exactly once in both delivered slots and matching validated auction slots before associating it with the concrete slot object. Missing/malformed/duplicate/conflicting refs affect only correlation: preserve slot/bid order/content, add correlation_unavailable, and keep complete server capture. Absent optional envelope does not start token validation.
+- [ ] Add an optional trace identity parameter to the existing opportunity forwarding and a trace-only store callback. Keep the current GPT export type unchanged. Pass trace identities only for validated SSAT/SPA opportunities; preserve existing attribution/source selection logic.
+- [ ] Emit `TraceSlotCorrelationV1` inside `GptDiagnosticsStore.recordSlotRequested` after `consumeRequestIntent` selects evidence and concrete runtimeSlotNumber/requestNumber exist. It contains only schema/version, two tokens, two positive safe integers. No second attribution engine, timestamps, raw DOM/ad-unit IDs or best-effort joins.
+- [ ] Write recorder tests for candidate/no-candidate/unrenderable, expired opportunities, competing sources, ambiguous/no concrete cycles, repeated callbacks, exact token equality and type assertions that GptDiagnosticsExportV1 has no new fields. Inject a throwing trace-sidecar callback and assert the recorder still completes ordinary binding and public GPT export unchanged. API evidence cannot call this binding.
+- [ ] Run `npx vitest run test/integrations/gpt test/integrations/gpt_diagnostics test/trace/collector.test.ts`. Expected green, including source export type gate. Planned commit: `Record exact SSAT and SPA trace sidecars at GPT cycle binding`.
+
+### E7: Consume Prebid transport exactly once through real registered hooks
+
+**Files:** Prebid integration, `trace/pending.ts`, pending/Prebid tests and real artifact test.
+
+- [ ] Write buildRequests/interpretResponse/onTimeout/onBidderError tests keyed by original bid IDs and final grouped unit refs, including multi-unit outcomes, concurrent requests, duplicate callbacks, absent response extension, malformed evidence and server-gate absence. Pending refs are request-local, not a global current request.
+- [ ] Run `npx vitest run test/trace/pending.test.ts test/integrations/prebid/index.test.ts`. Add trace-only optional hooks to the actual spec passed to `pbjs.registerBidAdapter(undefined, ADAPTER_CODE, spec)` only when trace is active. Preserve all ordinary bidder spec behavior when inactive.
+- [ ] Create a pending record after final buildRequests with maximum 128 retained entries. interpretResponse consumes before bids; `onTimeout(bidRequestsWithTimeout)` and `onBidderError({ error, bidderRequest })` consume the same matching pending record and add transport failure exactly once. Never store error text/XHR responses; capped/expired records without supported outcome hooks invent no failure.
+- [ ] Implement and test this checked expiry helper; capture effective browser bidderTimeout once per record, unrelated to server `[auction]` timers:
+
+```typescript
+const MAX_CAPTURED_BIDDER_TIMEOUT = 2 ** 31 - 1 - 5000
+
+function pendingExpiry(
+  createdAtMs: number,
+  configuredTimeout: unknown
+): number | undefined {
+  if (!Number.isSafeInteger(createdAtMs) || createdAtMs < 0) return undefined
+  const timeout =
+    typeof configuredTimeout === 'number' &&
+    Number.isInteger(configuredTimeout) &&
+    configuredTimeout >= 0 &&
+    configuredTimeout <= MAX_CAPTURED_BIDDER_TIMEOUT
+      ? configuredTimeout
+      : 3000
+  const expiresAtMs = createdAtMs + timeout + 5000
+  return Number.isSafeInteger(expiresAtMs) ? expiresAtMs : undefined
+}
+```
+
+- [ ] Use an integer wall-clock `Date.now()` (or its injected test clock) for record creation and read `pbjs.getConfig('bidderTimeout')` at creation. Configured `merged.timeout` already supplies that browser setting. Use a bounded timer delay of captured timeout+5000; future config changes do not alter existing expiry. Clock/checked-add failure declines only that pending attempt with no capture issue or reset; expiry alone removes the marker and remains not_observed.
+- [ ] Cover configured 0/normal/max, missing/negative/fractional/NaN/infinite/string/over-max values, default 3000, safe-integer clock overflow, callback after expiry, cap eviction, timer cleanup on runtime destruction and repeated hooks. Use fake timers, not wall-clock waiting.
+- [ ] Inject throwing token-generation/collector hooks in the registered Prebid path and prove bid callbacks/results remain unchanged. Measure the built shim against its existing 41,000-character guard in `test/prebid-artifact-integration.test.mjs`; if trace code requires an increase, record the measured before/after size and justify a narrow new bound while preserving the Prebid-free regression guard.
+- [ ] Extend `test/prebid-artifact-integration.test.mjs` using its real external bundle+shim setup. Drive real adapterManager timeout/error callbacks and prove `registerBidAdapter` → `newBidder` → registered spec hooks. Do not stop at calling mock spec methods directly.
+- [ ] Run `npx vitest run test/trace/pending.test.ts test/integrations/prebid/index.test.ts test/prebid-artifact-integration.test.mjs`, full JS tests/build/format, and affected Rust adapter checks. Verify API evidence stays independent and neither hook emits GPT sidecars or `trusted_server_direct`. Planned commit: `Capture Prebid trace transport through registered timeout and error hooks`.
+
+### Review-unit verification and handoff
+
+- [ ] Extend dedicated foundation browser fixtures with initial/SPA/API normal, zero-bid and failure results, token conversion filtering, old bundles, malformed members, legacy page-bids retry, disabled cookie-bearing traffic and fail-open ad assertions. The fixture must exercise actual built integration bundles, not fabricated evidence alone.
+- [ ] Run `./scripts/integration-tests-browser.sh tests/shared/mobile-trace.spec.ts`, focused evidence tests and affected-target build/clippy/docs checks; V7 owns the single full release gate run. Expected: advertising assertions stay identical to baseline; all evidence/privacy/token cases pass.
+- [ ] Search serialized trace fixture output for distinct forbidden sentinels covering IDs, names, prices, payloads, consent and raw errors. Assert no trace-specific logs contain sentinels or cookie/network values.
+- [ ] Review deterministic provider numbering and every terminal path centrally; preserve existing telemetry tests without adopting its private fields or saturation rules. Planned commit: `Verify live trace transports and advertising compatibility`.
+- [ ] Checkpoint evidence collection as memory-only, then continue V1. User snapshot/storage/viewer behavior remains phase 3; do not imply retained observations already survive navigation.
+
+## Phase 3: Browser handoff and viewer
+
+### Inputs and scope
+
+Depends on foundation tasks F0–F8 and live-evidence tasks E1–E7. Read [spec](../specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md) sections 6, 9.1–9.5, 10, 12, 13, 14.3–14.5, and 16. Follow the shared verification's same-branch and verification rules.
+
+Use `window.tsjs.gptDiagnostics.snapshot()` (the public API in `integrations/gpt_diagnostics/api.ts`) as the only GPT input. No private-store export traversal, second GPT attribution engine, server upload, telemetry query, target URL, report ID, or new UI dependency.
+
+### File map
+
+| Action | Exact path                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | Responsibility                                                                                  |
+| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
+| Modify | `crates/trusted-server-js/lib/src/trace/types.ts`, `crates/trusted-server-js/lib/src/trace/validation.ts`, `crates/trusted-server-js/lib/src/trace/runtime.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                | Complete trace/report/storage types and strict ingestion                                        |
+| Create | `crates/trusted-server-js/lib/src/trace/projection.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Exhaustive explicit GPT source-to-public projection                                             |
+| Create | `crates/trusted-server-js/lib/src/trace/report.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Snapshot assembly, checked omissions and deterministic bounds                                   |
+| Create | `crates/trusted-server-js/lib/src/trace/storage.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Single key, exact wrapper, age/size/origin validation and independent deletion                  |
+| Create | `crates/trusted-server-js/lib/src/trace/export.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | One validated formatted artifact for copy/download/share                                        |
+| Create | `crates/trusted-server-js/lib/src/trace/correlation.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Exact-token joins and explicit ambiguous/unmatched results                                      |
+| Create | `crates/trusted-server-js/lib/src/trace/handoff.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Explicit capture → validate → write → same-tab navigation and recovery                          |
+| Modify | `crates/trusted-server-js/lib/src/trace/viewer.ts`, `crates/trusted-server-js/lib/src/trace/viewer.css`, `crates/trusted-server-js/lib/src/trace/lifecycle.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                | Full report DOM/accessibility and independent cleanup/state verification                        |
+| Modify | `crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts`, `crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Optional gated prominent handoff action; keep old GPT export intact                             |
+| Create | `crates/trusted-server-js/lib/test/trace/fixtures.ts`, `crates/trusted-server-js/lib/test/trace/types.test.ts`, `crates/trusted-server-js/lib/test/trace/validation.test.ts`, `crates/trusted-server-js/lib/test/trace/projection.test.ts`, `crates/trusted-server-js/lib/test/trace/report.test.ts`, `crates/trusted-server-js/lib/test/trace/storage.test.ts`, `crates/trusted-server-js/lib/test/trace/export.test.ts`, `crates/trusted-server-js/lib/test/trace/correlation.test.ts`, `crates/trusted-server-js/lib/test/trace/handoff.test.ts`, `crates/trusted-server-js/lib/test/trace/viewer.test.ts` | Complete source and hostile/bounded fixtures, type/runtime/DOM/action tests                     |
+| Modify | `crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts`, `crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/index.test.ts`                                                                                                                                                                                                                                                                                                                                                                                                                                              | Publisher action gating/presentation compatibility                                              |
+| Modify | `crates/trusted-server-js/lib/trace-assets-manifest.json`, `crates/trusted-server-js/lib/test/trace-assets.test.mjs`, `crates/trusted-server-js/lib/build-all.mjs`; `crates/trusted-server-js/src/trace_assets.rs`; `crates/trusted-server-core/src/trace/routes.rs`, `crates/trusted-server-core/src/trace/shell.rs`                                                                                                                                                                                                                                                                                         | Complete unpublished v1 asset set/shell lookup; preserve byte contracts after first publication |
+| Modify | `crates/trusted-server-js/lib/trace-assets/v1.js`, `crates/trusted-server-js/lib/trace-assets/v1.css`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Finalize the complete unpublished v1 bytes created in F6 before first release                   |
+| Modify | `crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts`, `crates/trusted-server-integration-tests/browser/helpers/gpt-stub.ts`, `crates/trusted-server-integration-tests/browser/helpers/state.ts`, `crates/trusted-server-integration-tests/browser/helpers/infra.ts`, `crates/trusted-server-integration-tests/browser/global-setup.ts`, `crates/trusted-server-integration-tests/browser/global-teardown.ts`, `crates/trusted-server-integration-tests/browser/playwright.config.ts`                                                                                           | Dedicated trace runtime and realistic server→GPT→creative fixtures                              |
+| Modify | `crates/trusted-server-integration-tests/browser/helpers/trace-fixture.ts`, `crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml`                                                                                                                                                                                                                                                                                                                                                                                                                                              | Extend foundation runtime/settings with live auction cases                                      |
+| Create | `crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/app/mobile-trace/page.tsx`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Real publisher slot fixture for the complete viewer journey                                     |
+| Modify | `scripts/integration-tests-browser.sh`, `scripts/generate-integration-viceroy-configs.sh`, `.github/workflows/integration-tests.yml`                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Build/install any new trace fixture dependencies and browser projects consistently              |
+| Modify | `docs/guide/integrations/gpt-diagnostics.md`, `docs/guide/configuration.md`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Full support journey, trust/privacy limits and rollout                                          |
+
+`test/trace/types.test.ts` joins the existing Vitest typecheck include. Preserve its scoped source-error behavior; do not claim package-wide `tsc --noEmit` passes because Vitest passes.
+
+### V1: Define exact report ingestion and classify every GPT source member
+
+**Files:** trace types/validation/fixtures/type tests; source interface reference `src/core/types.ts`.
+
+- [ ] Create a complete fictional source fixture exercising every current `GptDiagnosticsExportV1` property, every request-cycle optional value, all callback/attribution/coverage metadata and all relevant enums. Use distinct forbidden sentinels for every `adManager` member, previousCreativeId, slotElementId, adUnitPath, issue slotElementId and a secret-bearing example.com pathname.
+- [ ] Write type-level member coverage using `satisfies Record` and equivalent checks for the source root, slot, issues, coverage counters, metadata and nested durations/binding. All current keys must be classified and no later source key passes automatically. Source additions fail typecheck until classified.
+- [ ] From `crates/trusted-server-js/lib`, run `npx vitest run test/trace/types.test.ts test/trace/validation.test.ts`. Expected red, including a real type assertion failure when a field is deliberately unclassified.
+- [ ] Define only the compatibility matrix `TraceReportV1`, `TraceAuctionEvidenceV1`, `TraceSlotCorrelationV1`, `TraceGptDiagnosticsV1` and source GPT v1. Unknown versions at any boundary fail with an actionable bounded category. All stored objects have exact allowed/required keys, not optional passthrough fields.
+- [ ] Implement request context/CookieHealth validators from foundation contracts. Validate masked addresses as the chosen /24 or /48 display representation, not arbitrary full IP strings; only canonical HTTP(S) origins equal to location.origin are accepted, with no credentials/path/query/fragment. Validate real UTC RFC3339 calendar times without permissive Date normalization.
+- [ ] Propagate runtime_header_ambiguous through the exact TypeScript CookieHealthDetail union and setup/context/TraceReportV1 validation. It is legal only with unavailable; test wrong state/detail pairs and unknown reason rejection. Retain supported schema versions and bounds; expose no fidelity metadata or original header values.
+- [ ] Implement exact GPT request-cycle allowlist below, with source optionality and enum membership from the current interfaces. Never accept adManager/previousCreativeId/slot or issue DOM IDs/ad-unit paths. The viewer accepts only pathname=`/[redacted]`.
+
+```typescript
+const TRACE_CYCLE_KEYS = [
+  'requestNumber',
+  'requestedAtMs',
+  'responseAtMs',
+  'renderAtMs',
+  'loadAtMs',
+  'viewableAtMs',
+  'durations',
+  'isEmpty',
+  'requestedSlotSizes',
+  'size',
+  'observedSlotSize',
+  'isBackfill',
+  'slotContentChanged',
+  'incompleteSequence',
+  'responseClass',
+  'requestPath',
+  'requestIntentId',
+  'trustedServerAuctionId',
+  'opportunityToRequestMs',
+  'replacedRequestNumber',
+  'previousRenderToRequestMs',
+  'creativeChanged',
+  'loadObservedBeforeRender',
+  'trustedServerOpportunity',
+  'trustedServerCreativeRequestAtMs',
+  'trustedServerCreativeResponseAtMs',
+  'trustedServerCreativeFailures',
+  'delivery',
+] as const
+
+const CALLBACK_REASONS = [
+  'invalid_event_order',
+  'missing_response_before_render',
+  'invalid_visibility_percentage',
+  'evicted_slot',
+  'no_compatible_request_cycle',
+  'overlapping_request_cycles',
+] as const
+```
+
+- [ ] Validate remaining exact objects: root GPT schema/source version/capturedAt/page/slots/callbackIssues/attributionIssues/coverage/metadata; slot runtimeSlotNumber/binding/current+max visibility/requests; binding status/reason; five duration members; callback kind/runtimeSlotNumber/timestampMs/disposition/reason; attribution reason/timestampMs/runtimeSlotNumber; six callback coverage keys with observed/matched/unmatched/ambiguous; metadata droppedCallbacks/droppedAttributionIssues/evictedSlots/evictedRequestCycles.
+- [ ] Apply all bounds: 64 GPT slots, ≤10 cycles/slot, 128 callback/attribution issues, ≤16 sizes/failure enums, origin ≤255 bytes, other allowed strings ≤128 bytes, exact tokens, finite nonnegative safe integer sequences/counters, browser timestamp/duration finite 0..MAX_SAFE_INTEGER, visibility 0..100, positive integer requested/selected/fill dimensions ≤100000; observed CSS dimensions integer 0..100000 including zeros. Outer/inner u16 omission counters retain their stricter bound.
+- [ ] Count nested containers from report=1, not wrapper; objects and arrays each increment, max 10. Reject level 11, unknown properties/prototype keys, unpaired surrogates, C0/C1 and bidi override/isolate controls before any DOM/export. Validate wrapper separately as exactly stored_at_ms/report; precheck compact UTF-8 bytes before rendering/JSON parsing where input is serialized.
+- [ ] Add one test per bound edge, forbidden member, enum/version/extra-key boundary, Unicode/control class, true complete-depth fixture, numeric overflow and zero observed dimensions. Re-run the filtered suites/type gate. Planned commit: `Define strict combined trace report validation`.
+
+### V2: Project the public GPT model and implement deterministic report bounds
+
+**Files:** projection/report/fixtures and their tests; collector interface from phase 2.
+
+- [ ] Write field-by-field projection tests for the complete fixture. Expected root transforms: source `version`→source_schema_version=1, outer schema_version=1, source capturedAt retained, canonical origin retained, pathname replaced with literal. Every forbidden sentinel must be absent; no source object spreads/dynamic property inheritance.
+- [ ] Run `npx vitest run test/trace/projection.test.ts`. Implement a new owned projection with explicit assignments for all V1-classified copied fields, checked source values and new nested objects/arrays. Invalid source values reject snapshot creation, not stringify/coerce/truncate strings. Mutating source after capture cannot mutate the projected report.
+- [ ] Validate exact source container/key/own-data contracts and every value consumed by the projection. Never inspect excluded identity contents; their malformed contents cannot contaminate the report. Require the source pathname to be an own string, then replace it without reading its content or imposing a raw-path cap. Reject source arrays beyond their existing 64-slot/10-cycle/128-issue bounds; validate all supplied bounded sizes/failure enums before retaining the first sixteen and counting omissions.
+- [ ] Write table tests for initial cardinality projection: newest 16 server records/128 sidecars, first 16 requested sizes/failure enums, ≤64 slots/10 cycles/128 issue records as defined by the source contract. Merge collector auction/sidecar evictions into outer omission counters exactly once; retain existing GPT metadata as its source meaning, not an extra outer omission count. Reject invalid inner server evidence rather than re-truncating it.
+- [ ] Add the checked counter primitive and use it for every initial discard, later cycle/issue/auction/sidecar removal and optional invalid numeric discard from core:
+
+```typescript
+function addOmissions(current: number, added: number): number {
+  if (
+    !Number.isInteger(current) ||
+    !Number.isInteger(added) ||
+    current < 0 ||
+    added < 0 ||
+    current > 65535 ||
+    added > 65535 - current
+  ) {
+    throw new Error('omission_counter_overflow')
+  }
+  return current + added
+}
+```
+
+- [ ] Define stored_at_ms from the capture clock independently of source content. Outer report captured_at and GPT capturedAt must be within 60 seconds of that clock; request_context.captured_at may be older and is not overwritten by setup/time data. Validate the complete compact wrapper with `TextEncoder`, limit 512\*1024 bytes.
+- [ ] Write truncation-order tests with deterministic IDs/timestamps: missing requestedAtMs sorts first; otherwise cycle order is requestedAtMs/runtimeSlotNumber/requestNumber. Callback/attribution order is timestampMs then original index. Array output retains original relative order after removals; server/provider/slot and sidecar observation orders remain stable.
+- [ ] Implement the complete removal sequence, measuring the whole compact wrapper after each eligible removal:
+  1. Protect each GPT slot's newest retained cycle for the entire algorithm.
+  2. Remove globally oldest non-floor GPT cycles. Remove/count every associated sidecar in the same operation; never retain a claimed dangling join.
+  3. Remove oldest callback issues, then oldest attribution issues.
+  4. Remove oldest uncorrelated server auctions. Treat any remaining GPT-cycle auction-token reference as correlated even if no sidecar validates; do not orphan that cycle just because the join is unknown. Remove/count all referencing sidecars with the auction.
+  5. Remove oldest eligible correlated server auctions together with all retained cycles that reference them and their sidecars. Skip any auction referenced by a protected floor cycle. Eligibility must preserve every GPT slot's newest cycle; no stage can empty a slot that started with cycles.
+  6. Recompute capture_status/issues after each auction removal; auction omission adds record_evicted, sidecar-only omission adds correlation_unavailable. Deduplicate/sort issues and recalculate compact wrapper bytes including changed counters/issues.
+  7. If still oversized after all eligible removals, fail with a bounded snapshot-size category. Do not remove the protected floor, return an empty GPT section, store, navigate, or offer a combined-report fallback.
+
+- [ ] Add worst-case successful ≤512 KiB fixture with all source fields populated, multi-byte strings and exact six outer counters; counter-overflow fixture; full server eviction→unavailable/record_evicted; partial eviction→partial; sidecar-only eviction preserves complete; protected-floor+auction set >budget rejects. Use bounded valid model construction or a test-only injected smaller budget to exercise deterministic branches, plus an actual 512 KiB worst-case fixture. Do not weaken production v1 bounds.
+- [ ] Include runtime_header_ambiguous in valid worst-case request-context/cookie fixtures and reassert exact encoded size, truncation/protected-floor behavior and existing depth/string bounds. Adding a valid reason does not authorize expanding the report envelope.
+- [ ] Run `npx vitest run test/trace/projection.test.ts test/trace/report.test.ts test/trace/collector.test.ts`. Expected green and exact byte/counter assertions. Planned commit: `Project and deterministically bound combined trace snapshots`.
+
+### V3: Store one validated report and export the identical model
+
+**Files:** storage/export and unit tests.
+
+- [ ] Write storage tests for one namespaced versioned key `trusted-server.trace.report.v1`; exact `{ stored_at_ms, report }` wrapper; replacement; absent/get/set/remove exceptions; malformed JSON; future, rollback and expiry; origin mismatch; hostile keys; oversized compact UTF-8 input. No storage write occurs during observation or viewer load.
+- [ ] Run `npx vitest run test/trace/storage.test.ts`. Implement explicit read/write/delete results with bounded categories. Validate before setItem and after every getItem; remove rejected/expired entries best-effort, ignore them even when deletion fails. Enforce finite nonnegative safe stored_at_ms, max age 15 minutes, >60-second future skew rejection, capture-time consistency. Exactly 15 minutes remains within the age cap; older expires.
+- [ ] Write export tests requiring the same TraceReportV1 bytes/model for download/copy/share/direct storage-failure recovery. Export the report, not private source objects or a page URL; use deterministic filename `trusted-server-trace-v1.json`, MIME `application/json`, formatted JSON and visible browser-carried/unverified labeling in the UI. Do not add an unknown provenance field to the approved schema.
+- [ ] Pin unavailable/runtime_header_ambiguous through storage roundtrip, download, formatted copy, shared File and direct export. Wrong state/detail pairs reject on ingestion; every accepted output retains the same reason and no raw values.
+- [ ] Implement one `formatTraceReport(report)` after validation used by all actions. Copy invokes clipboard only on explicit user action. Share supplies a `File` with the same formatted JSON only after `navigator.canShare({ files })` support and a tap; cancellation/rejection/absence leaves Copy and Download available with status. Never fall back to URL share/upload.
+- [ ] Implement the download primitive with independent deferred URL cleanup:
+
+```typescript
+function downloadJson(json: string, filename: string): void {
+  const url = URL.createObjectURL(
+    new Blob([json], { type: 'application/json' })
+  )
+  const anchor = document.createElement('a')
+  anchor.href = url
+  anchor.download = filename
+  document.body.append(anchor)
+  try {
+    anchor.click()
+  } finally {
+    anchor.remove()
+    window.setTimeout(() => URL.revokeObjectURL(url), 1000)
+  }
+}
+```
+
+- [ ] Use fake timers/mocked URLs to assert click before scheduling, no synchronous revoke, each repeated download owns its URL, exact 1000 ms cleanup and accessible failure reporting. Do not change the pre-existing GPT-only export's synchronous cleanup in `api.ts`; that is outside this feature.
+- [ ] Re-run storage/export tests. Expected: failure preserves visible valid report, no report upload/continuous storage. Planned commit: `Add validated trace storage and equivalent local exports`.
+
+### V4: Join exact evidence without inventing winners or mixing clocks
+
+**Files:** correlation, tests and renderer-facing result types.
+
+- [ ] Write one valid SSAT and one SPA fixture joining exact diagnostic_auction_id/slot_ref and runtime_slot_number/request_number. Include no_candidate, unrenderable, empty/filled, creative bridge render/load/viewability and conflicting browser refresh paths.
+- [ ] Run `npx vitest run test/trace/correlation.test.ts`. Implement joins only when one validated sidecar uniquely matches one server slot and one exported GPT cycle with consistent auction token. Runtime numbers/slot ordinals alone, timestamps, ad-unit/DOM paths and implicit array position never join.
+- [ ] Add unmatched/duplicate/conflicting/forged/missing token and evicted-cycle fixtures. Keep structurally valid ambiguous records independent and show Correlation unknown; never choose a best match or assert no auction ran. Prune known dangling joins during builder truncation, not by fabricating replacement sidecars.
+- [ ] Reject any stored sidecar referencing an auction_api record: unsupported in v1. API evidence displays independently with correlation_unavailable, retains complete server capture when otherwise valid, and never changes prebid_refresh to trusted_server_direct/competing.
+- [ ] Map labels exactly: SSAT initial-page server auction, SPA Trusted Server page-refresh auction, API Trusted Server auction API; publisher/prebid_refresh=`Browser refresh observed; winner not determined`; competing/unattributed=`Multiple or unknown delivery paths`. Show `Trusted Server creative rendered` only when matched existing nonempty creative-bridge evidence proves participation, not candidate selection or GPT fill alone.
+- [ ] Present server auction-local duration, request-relative milestones=`Unavailable in v1`, and browser/GPT timings in separate groups. Never subtract/add timestamps across clocks. Provider status remains auction-wide, not a per-slot no-bid reason.
+- [ ] Re-run tests and assert source models are unchanged by joining. Planned commit: `Join exact trace evidence with explicit coverage and ambiguity`.
+
+### V5: Add the explicit publisher handoff and storage-failure recovery
+
+**Files:** handoff/runtime, diagnostics index/overlay, handoff/overlay/index tests.
+
+- [ ] Write gated action tests: missing/false/nonboolean trace flag with console=true exposes no action, listeners, mappings, storage reads/writes; active=true supplies one prominent `View trace results` action with ≥44 px target. Existing GPT-only export remains separate.
+- [ ] Run `npx vitest run test/trace/handoff.test.ts test/integrations/gpt_diagnostics/overlay.test.ts test/integrations/gpt_diagnostics/index.test.ts`. Add an optional overlay callback/options entry rather than rewriting the overlay or moving its internals into the trace viewer.
+- [ ] On tap, obtain public GPT snapshot, immutable request context and collector snapshot; build/project/bound/validate the combined wrapper, then setItem, then `location.assign('/_ts/trace')`. No continuous persistence, new tab, URL payload, target URL, or extra diagnostic network request.
+- [ ] If context/projection/validation/bounds fail, keep publisher page, announce bounded capture failure, do not navigate/write or offer a combined artifact. A GPT-only export is not an equivalent fallback.
+- [ ] If only storage write fails after a valid report exists, keep publisher page, announce storage failure, retain that immutable combined report for an explicit direct Download action using V3. Navigation never happens after rejected storage. Destroy retry listeners/recovery state with the diagnostics runtime.
+- [ ] Test exact action order, no navigation before successful write, snapshot source mutation isolation, serialization errors, quota/unavailable storage and direct-download equality. Re-run focused tests plus all GPT diagnostics tests. Planned commit: `Add explicit same-tab mobile trace handoff and recovery`.
+
+### V6: Render the mobile report and independent cleanup outcomes
+
+**Files:** viewer/styles/lifecycle, viewer/setup/export tests, Rust shell/routes and immutable asset metadata.
+
+- [ ] Write DOM tests for no report→setup, rejected/expired report→reproduce message, valid report→summary/network/cookies/server/GPT/coverage/export/cleanup. Setup-request facts stay separately labeled and never fill missing publisher facts.
+- [ ] Run `npx vitest run test/trace/viewer.test.ts test/trace/setup.test.ts test/trace/lifecycle.test.ts`. Render only the validated bounded model with `createElement`, safe attributes and textContent. No report innerHTML, inline style attributes/blocks or JS style writes under CSP; use classes/hidden.
+- [ ] Add browser-carried/unverified heading and per-entry server-produced/copied-through-browser versus browser-observed provenance. All unavailable fields/states/ambiguities have plain text labels; no winner inference, exact page/path, or provider identity appears. Preserve zero observed box sizes without changing isEmpty.
+- [ ] Render runtime_header_ambiguous as cookies unavailable for reliable inspection, never as actual invalid UTF-8 or absence. Cover all four affected cookie rows and retain export equality; no parsing detail or runtime fidelity metadata is displayed.
+- [ ] Add formatted Copy, deterministic Download, and progressive file Share from V3. Status is aria-live; errors retain report. Disclosure says the selected share app receives the JSON. Keyboard focus and visible focus styles work independently of color/hover.
+- [ ] Add `Clear report and end tracing` with explicit confirmation and distinct always-available `Delete local report`. After confirmation, attempt local deletion and end POST independently even if either fails; after POST attempt state verification independently. Successful deletion removes on-screen saved-report claims; failed deletion retains retry; failed/mismatched server observation uses unconfirmed/may remain active wording and independent retry. Never undo deletion because network failed.
+- [ ] Cover the entire local-delete × mutation × state-verification outcome matrix with explicit UI assertions, including offline, successful mutation/unconfirmed observation, inactive-invalid/duplicate cookie, successful server end/failed deletion and retries. A false observed_active says no valid session observed, not cookie absent.
+- [ ] Implement 320 px full-document layout with ≥44×44 primary controls, no horizontal page scroll, content-safe action placement, semantic headings/lists, zoom enabled, safe-area insets and readable long values. Tests check no content is covered by controls and report sections remain navigable by keyboard.
+- [ ] Finalize the existing unpublished v1 JS/CSS bytes and committed digests with the complete viewer. Keep the exact v1 routes and shell references from F2/F6. At the V7 release checkpoint freeze this asset set; tests reject any later changed published bytes without a new URL/digest. No intermediate setup asset version is published as part of this plan.
+- [ ] Run `node build-all.mjs`, `npx vitest run test/trace-assets.test.mjs`, `npx vitest run test/trace`, `cargo test-fastly trace::routes`, adapter parity and format checks. Expected: manifest pins every retained frozen version and current rebuilt bytes; shell references current URLs; altered bytes fail without a new version. Planned commit: `Add accessible consolidated mobile trace viewer and independent cleanup`.
+
+### V7: Prove the complete journey with real endpoints and built bundles
+
+**Files:** dedicated browser fixtures/helper/config/spec and build/CI paths in the file map; operator docs.
+
+- [ ] Extend the foundation trace suite with a dedicated enabled config/runtime and real publisher slots; keep existing baseline config unchanged (it disables GPT/auction). Build images/artifacts through existing scripts/global setup. The runner executes both Next.js and WordPress: explicitly scope the Next.js-only publisher journey with the existing framework setting, or add equivalent WordPress content; run shared endpoint/setup scenarios against both. Generate the dedicated trace Viceroy config from the trace TOML through F8’s parameterized generator, never reuse the disabled baseline output. The existing observer-only GPT stub has no working defineSlot/refresh and cannot prove the chain; implement realistic callbacks in the dedicated fixture or extend the helper without changing existing test behavior.
+- [ ] Write browser test: open publisher → navigate trace → assert read-only/no cookie → tap enable → separate state confirms → history Back → explicit real reload → collect SSAT and multiple GPT cycles → tap handoff → same-tab viewer → parse export and compare every displayed model section. Include SPA canonical/legacy page-bids and both actual `/auction` callers.
+- [ ] Execute normal safe-cookie setup/enable/state/reload/capture/view/export/end/state across all four adapter fixtures on suitable same-origin deployments. Unknown fidelity alone must not suppress capture. Assert exact privacy/action/session contracts while documenting runtime rejection separately. Browser tests must prove cookie storage/observation, not inject a synthetic active-cookie header and call it a browser journey. Record unsuitable origins or unavailable runtime/browser environments as pending acceptance.
+- [ ] Add raw-transport ambiguity fixtures alongside the browser workflow: valid session plus unrelated FF, original EF BF BD, comma-folded repeats and visible duplicates. For converted ambiguous requests assert state false, no trace tokens/evidence/sidecars, ordinary advertising unchanged, exact CookieHealth reason in setup/capture where observable, and valid end remains possible. Runtime-rejected requests produce no invented report. Normalization-in/out and encoded-alias auth tests follow F2/F4's actual visible path.
+- [ ] Run `./scripts/integration-tests-browser.sh tests/shared/mobile-trace.spec.ts` from root. Expected red for missing journey fixtures/behavior, not artifact or Docker setup failure. Then fill the concrete fixture behavior and re-run after each scenario group.
+- [ ] Add initial-navigation fixtures for selected, no-candidate, selected-unrenderable, skipped, dispatch-failed, execution-failed and abandoned evidence where delivery is possible; stopped/unreached tail remains not_observed. Test correlated SSAT/SPA chain, API independent, browser/client intent unknown, empty/ambiguous/unattributed states and malformed/absent transport with unchanged ads.
+- [ ] Test genuine second-origin top-level GET, form and fetch activation/end attempts; configured auth; history unchanged by actions; BFCache guidance does not claim new capture; same-origin service-worker-forged exchange/report remains labeled unverified and real server endpoints still reject invalid requests.
+- [ ] Test unexpired viewer reload, replacement, expired/hostile storage removal, removal failure, opener-cloned entry and expiry, restoration simulation, future-clock rollback, origin/hostname changes and same-host recovery instructions. Do not claim real browser restart coverage from a unit fixture; document any manual session-restore check still pending.
+- [ ] Under the exact CSP test complete Blob JSON download with 1000 ms cleanup, repeated download, storage-failure direct export, clipboard/share absence/rejection/cancellation, no URL sharing, favicon without publisher `/favicon.ico`, framing/inline/third-party/report-derived injection blocked. Assert no upload/beacon or third-party trace request.
+- [ ] Populate forbidden sentinels across the entire source fixture and assert absence in server trace HTML/JSON, stored wrapper, DOM, formatted copy, shared File and both downloads; hostile stored forbidden properties are rejected. Check output model/version equality, not just filename/download event.
+- [ ] Add 320 px/mobile viewport and keyboard/focus/status tests to the existing Chromium runner. Keep real iOS Safari and Android Chrome manual acceptance as a separate checklist for zoom, safe areas, native file share/download, layperson instructions and retention of visible report after failures. Any optional automated WebKit project requires matching Playwright install changes in script and CI.
+- [ ] Test rollback fresh loads with valid diagnostics cookie and console query activation retained: no trace gate/action/listeners/storage access/tokens/response evidence/new cache change. Already-running pages cannot be remotely revoked; later server requests immediately stop evidence. Previously cached static assets remain inert.
+- [ ] Document reproduction/support/export instructions and browser-carried trust limits, no historic recovery, no client winner inference, host-only scope, independent cleanup retries and controlled staging rollout. Mark #1081/#1074/#1076 fields unavailable, not missing implementation.
+- [ ] Run the full shared verification gates/builds/docs and the complete browser runner, plus manual mobile checklist. Review final diff and acceptance matrix. Planned commit: `Verify the mobile trace journey and document release acceptance`.
+- [ ] Handoff v1 feature only after the first three phases meet all mandatory acceptance criteria; record pending environment/manual checks separately. Publishing/deployment is a subsequent explicit action.
+
+## Phase 4: Optional network enrichment
+
+### Scheduling boundary
+
+This is the optional fourth phase from approved [spec](../specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md) sections 11 and 17.4. It does not block the first three phases or #1050 acceptance. Follow the shared sections above, keep the current branch while executing authorized work, and use the already strict optional field validators rather than expanding the report.
+
+Do not adopt request-relative #1074/#1076 milestones, bidder/price/creative-number/winner disclosures from #1081, resolver facts, speed probes, raw IP or JA4/H2 here. Each later public contract change needs its own approved design/compatibility plan; this document schedules no speculative schema implementation.
+
+### File map
+
+| Action                         | Exact path                                                                                                                                                             | Responsibility                                                      |
+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
+| Modify if a source is verified | `crates/trusted-server-core/src/platform/types.rs`                                                                                                                     | Optional adapter-owned HTTP version/POP metadata, default None      |
+| Modify                         | `crates/trusted-server-core/src/trace/context.rs`                                                                                                                      | Map only verified optional metadata through existing bounds         |
+| Modify if needed               | `crates/trusted-server-adapter-fastly/src/main.rs`, `crates/trusted-server-adapter-fastly/src/platform.rs`                                                             | Native HTTP metadata before conversion, optional SDK ASN/POP source |
+| Modify if a source is verified | `crates/trusted-server-adapter-axum/src/platform.rs`, `crates/trusted-server-adapter-cloudflare/src/platform.rs`, `crates/trusted-server-adapter-spin/src/platform.rs` | Explicit supported mapping, not fabricated cross-platform values    |
+| Modify                         | `crates/trusted-server-integration-tests/tests/parity.rs`                                                                                                              | Common schema/omission expectations with differing supported fields |
+| Modify                         | `crates/trusted-server-js/lib/test/trace/validation.test.ts`, `crates/trusted-server-js/lib/test/trace/viewer.test.ts`                                                 | Optional values accepted/bounded/rendered without default invention |
+| Modify                         | `docs/guide/integrations/gpt-diagnostics.md`                                                                                                                           | Source/support matrix and privacy/provenance documentation          |
+
+New Rust tests stay in the changed modules. Do not edit a dependency's Cargo-cache source. If upstream conversion must preserve a fact, release/pin that change through the foundation's dependency process.
+
+### N1: Prove source availability against the actual SDK pin
+
+- [ ] Record SDK revisions from `cargo metadata --format-version 1` and Cargo.lock and inspect the corresponding sources. Existing investigation found Fastly `Request::get_version()` and `Geo::as_number()` in the installed pinned SDK; verify again at execution, after the foundation EdgeZero upgrade.
+- [ ] Establish the source table below with a concrete native accessor/type per newly enabled cell. Official runtime documentation must support environment-derived facts; an available header name is not a trusted source. Do not read arbitrary forwarded headers for HTTP/TLS/POP facts.
+
+| Public field                  | Existing baseline                                             | Enrichment source to verify                                                                        | Failure behavior                          |
+| ----------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------- |
+| http_version/Fastly           | EdgeZero converter does not copy Fastly native version        | Capture native `req.get_version()` before conversion and carry adapter-owned fact                  | Omit unsupported/error value              |
+| http_version/Axum             | Native inbound request version is available before conversion | Prove version is copied from inbound `http::Request` parts, not default-created downstream request | Omit if original version cannot be proved |
+| http_version/Cloudflare, Spin | No current trace mapping                                      | Stable documented native runtime metadata only                                                     | Absent until verified                     |
+| asn/Fastly                    | `geo_from_fastly` sets None                                   | Pinned `Geo::as_number()` with documented unknown/sentinel handling                                | Unknown/sentinel/error remains None       |
+| asn/Cloudflare                | Geo adapter currently sets None                               | Typed documented Workers connection metadata with checked u32 conversion                           | Absent until verified                     |
+| edge_pop/Fastly, Cloudflare   | No existing mapping                                           | Stable documented runtime POP field/environment variable                                           | Absent until verified                     |
+| edge_pop/Axum, Spin           | Unavailable in v1                                             | No invented source                                                                                 | Absent                                    |
+
+- [ ] Add red tests only for mappings with established sources. Test unavailable cells remain absent and do not inherit a default HTTP/1.1 value, hostname-derived POP, region-as-POP, fabricated ASN or full IP.
+- [ ] Run adapter platform tests for selected mappings and `cargo test-fastly trace::context`. Expected red for supported new fields, existing omission tests remain green. If no source is established for a field, document it as unavailable and finish that field's discovery without runtime edits.
+
+### N2: Add protocol/POP metadata without broadening disclosure
+
+- [ ] For verified HTTP sources, add optional fields to `ClientInfo` (or the established adapter-owned inbound metadata extension) with defaults None. Preserve existing callers/constructors and do not infer protocol from a synthetic provider request or forwarding header.
+- [ ] Use a closed protocol mapping for supported HTTP versions, bounded ≤32 bytes. Store it before the conversion point that would lose it; core receives only that normalized fact. Other adapters omit it until they can supply the same provenance.
+- [ ] For a verified POP source, normalize printable Unicode/control rules and ≤32 UTF-8 bytes with the existing core helper. Invalid/unavailable/overlong values are omitted with only a bounded category; no source value enters logs.
+- [ ] Test getter failures, absent metadata, values at/over bounds, control/bidi strings, and spoofed headers that cannot overwrite the adapter fact. Retain all masking/JA4/H2 exclusions and setup-vs-publisher labeling.
+- [ ] Run `cargo test-fastly`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, and `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity`. Expected: common schema, intentional availability differences, unchanged route/auth/cache behavior. Planned commit, only if a mapping was added: `Add verified optional trace protocol and POP facts`.
+
+### N3: Populate verified ASN sources and verify the optional increment
+
+- [ ] For Fastly's proven getter, add a test distinguishing a valid documented ASN from unknown/sentinel values, then populate `GeoInfo.asn` without copying city/coordinates or adding an active lookup beyond existing geo behavior. For Cloudflare, add a mapping only after N1 establishes the exact typed source; checked conversion failure omits it.
+- [ ] Run selected geo tests to observe red, implement the minimal optional mapping and re-run. Update ordinary geo consumer tests if their intentionally observable GeoInfo now includes ASN; verify no identity/routing/consent behavior changes unintentionally.
+- [ ] Test public network projection accepts only u32 ASN and emits no unsupported fallback. Viewer preserves unavailable labels and existing report validity/expiry/export equality. From JS lib run `npx vitest run test/trace/validation.test.ts test/trace/viewer.test.ts`.
+- [ ] Document the actual resulting adapter support/provenance matrix and remaining unavailable cells. Verify masked IP/coarse geo privacy acceptance still applies and that optional field errors do not fail publisher delivery.
+- [ ] Run all shared verification CI/build/doc gates and the trace browser suite. Re-run cache/auth/inactive gates because extra metadata crosses the shared platform type; feature-disabled behavior remains unchanged. Planned commit, only if ASN was added: `Add verified trace ASN projection and document platform support`.
+- [ ] Handoff only implemented, verified optional facts. Future #1081/#1074/#1076 schemas remain separate work; do not mark them complete or add placeholders to the v1 report.
diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
index 7b220df24..7479ea919 100644
--- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
+++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
@@ -1,6 +1,14 @@
 # Mobile Ad-Rendering Trace Endpoint Design
 
-**Status:** Proposed
+**Status:** Approved, including runtime-boundary amendment (2026-10-05)
+
+**Runtime-boundary amendment:** Version one evaluates the method, pathname,
+headers and body exposed at application dispatch. It does not require recovery
+of information discarded by an SDK/runtime or application responses to requests
+rejected before dispatch. Cookie ambiguity suppresses trace capture under the
+explicit rules in section 9.2. See section 8 for response scope and section 16
+for acceptance tests. This amends the original-path and original-wire promises;
+authentication, deliberate actions and response privacy remain required.
 
 **Issue:** [#1050 — Create debug endpoint for mobile user to trace ad rendering](https://github.com/IABTechLab/trusted-server/issues/1050)
 
@@ -163,7 +171,10 @@ to adopt the new lifetime. Cookie expiry also does not unload a running page
 or clear an existing report; report expiry is defined separately in section 9.5.
 
 The base trace gate requires both `trace_page_enabled = true` and exactly one
-valid incoming diagnostics cookie, inspected before cookie sanitation.
+valid incoming diagnostics cookie, inspected from frozen runtime-visible
+fields before cookie sanitation under section 9.2. An unavailable result,
+including `runtime_header_ambiguous`, keeps this gate false. Unknown or missing
+fidelity metadata alone does not invalidate a readable marker-free cookie.
 Publisher-document tracing additionally requires the existing effective
 `GptDiagnosticsRequestDecision.active` decision. Query disable or invalid
 directives, prefetches, bots, and other ineligible navigations therefore
@@ -418,12 +429,15 @@ Report GET /_ts/trace
 - Convert `ClientInfo` and available geo data into the public network allowlist.
 - Inject request context only into an active private diagnostics document.
 - Apply response privacy and security headers.
-- Ensure trace requests never reach the publisher origin.
+- Ensure requests classified within the application-visible trace namespace
+  never reach the publisher origin, including visible rejected aliases.
 
 ### 7.2 Adapter responsibilities
 
 - Invoke the reserved-route classifier at the earliest adapter dispatch point
-  with exact path and method handling.
+  with exact application-visible path and method handling. Runtime/SDK
+  normalization before this point is not reconstructed; original-target
+  metadata is optional and does not govern a second route or auth policy.
 - Populate optional `ClientInfo` fields available on the platform.
 - Fastly may supply bounded POP, HTTP version, TLS, and edge-server data when
   the SDK exposes them. JA4 and H2 fingerprints are excluded from version one.
@@ -452,6 +466,38 @@ Report GET /_ts/trace
 
 ## 8. Route and configuration contract
 
+The version-one boundary is the request exposed by the runtime/SDK and
+successfully converted for application dispatch, before ordinary routing and
+middleware. Apply path classification to `Request::uri().path()` at that
+boundary, before additional Trusted Server normalization. Apply cookie
+inspection to the frozen incoming core headers, before sanitation. Distinguish
+three outcomes in tests and deployment evidence:
+
+| Outcome                                                                       | Version-one guarantee                                                                                                                                                                                 |
+| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Runtime rejects before application invocation                                 | No Trusted Server handler runs. Record runtime rejection separately; do not claim a local trace status, authentication challenge, `Allow` or hardening headers.                                       |
+| Runtime delivers but adapter bootstrap/conversion fails before trace dispatch | Preserve existing adapter failure behavior and record it separately. No successful trace action or capture is claimed; these failures must not perform trace-specific cookie writes or mint evidence. |
+| Conversion succeeds and trace dispatch runs                                   | Enforce this spec's authentication, method/action validation, local-response, privacy and no-publisher-fallback contracts against application-visible facts, including transformed facts.             |
+
+Normal accepted browser requests on a suitable same-origin deployment must
+support the complete workflow on Fastly, Axum, Cloudflare and Spin. The browser
+must permit the required Secure host-only cookie; plain HTTP development must
+not weaken its attributes or invent a trusted HTTPS scheme. Record an unsuitable
+test origin separately rather than claiming a passing browser journey.
+An unsupported malformed-wire case does not
+justify disabling an adapter's normal trace journey. Conversely, accepting a
+runtime rejection in the evidence does not prove a successful browser journey.
+Existing adapter startup/config/store initialization and body buffering remain
+outside the pre-dispatch hook; no new transport allocation/deadline promise is
+introduced. A deliberately broken converter is not a compliant normal path.
+
+Original target spelling and original header octets/counts may already be
+unavailable. Recovery is not a v1 prerequisite. `RequestIngress.target()` is
+optional capability evidence, never a second route/auth interpretation; a
+missing snapshot or whole-target capture cap does not invalidate an inspectable
+pathname. Trusted canonical origin for actions still comes from the adapter's
+validated `RequestIngress.origin()`, never from forwarded-header fallbacks.
+
 Add an explicit default-off option to the existing integration:
 
 ```toml
@@ -547,7 +593,8 @@ Rules (route responses below apply after configured authentication):
   the single parsed `Origin` header are serialized with lowercase host and
   default ports removed before exact comparison. Invalid or multi-valued host,
   authority, scheme, or origin input fails closed.
-- Unsupported methods on a shell or state-changing path return a local 405
+- Unsupported application-visible methods on shell/state/assets or a
+  state-changing path return a local 405
   Method Not Allowed response with the path-specific `Allow` header:
   `GET, HEAD` for shell/state/assets and `POST` for enable/end. Classification
   must intercept unsupported methods before router dispatch. The router's
@@ -556,27 +603,38 @@ Rules (route responses below apply after configured authentication):
   `dispatch_unregistered_method_returns_405_at_router_level` in the Fastly
   adapter. The trace responder itself supplies `Allow` and all section 12.3
   error-response hardening; it must not rely on router-generated errors.
+  If the runtime rejects a method before invocation, record that outcome as
+  runtime rejection instead of claiming an application-generated 405.
   Fastly entry-point finalization can add ordinary headers later, but does not
   establish this trace-specific contract on behalf of the router.
 - Disabled deployments return a local `404` for the complete trace route set,
   including assets, and never fall through to the publisher origin.
 - The `/_ts/trace` namespace is reserved. A trailing slash, extra path segment,
   unsupported asset name, repeated separator, or lookalike beneath that
-  namespace returns a local `404`; an encoded separator or ambiguous dot
-  segment returns a local `400`. None falls through to the publisher origin.
-  The adapter classifies from its canonical parsed path while retaining enough
-  raw-path information to reject ambiguous encodings consistently.
+  namespace returns a local `404`; an encoded namespace alias, encoded
+  separator, or ambiguous dot segment still visible at dispatch returns a
+  local `400` after authentication and the enabled-feature check. None of these
+  application-visible reserved requests falls through to the publisher origin.
+  Bounded percent-decoding identifies encoded aliases for rejection only; it
+  never turns an alias into a supported route or permits a trace action.
+  A path normalized by the runtime/SDK outside this namespace is ordinary
+  traffic, even if its unavailable original spelling mentioned trace. For
+  example, a Fastly request normalized to `/health` or `/_ts/debug/ja4` follows
+  that path's existing behavior, including its existing auth limitations.
+  A path normalized into an exact trace route must receive the full trace
+  authentication and validation contract. Additional original-target metadata
+  does not impose different v1 behavior on otherwise identical visible paths.
 
 Reuse the bounded percent-decode-to-fixed-point classification pattern from
 `deny_admin_diagnostic_fallback` in `crates/trusted-server-core/src/ec/admin.rs`
 (`MAX_PERCENT_DECODE_ROUNDS = 4`), moving trace classification before dispatch
 rather than relying on fallback. Register and intercept trace paths on Fastly,
-Axum, Cloudflare, and Spin from the first implementation PR; existing Fastly-only
+Axum, Cloudflare, and Spin from the first implementation phase; existing Fastly-only
 `/_ts/*` routes are not a parity precedent.
 
 Every adapter implements the following order:
 
-1. Parse the method, canonical host/origin, path, query, and bounded headers
+1. Read the application-visible method, trusted host/origin, path, query, and bounded headers
    required for route safety.
 2. Classify an exact Trusted Server reserved path.
 3. For a trace path, enforce the existing configured Basic Authentication
@@ -588,6 +646,13 @@ Every adapter implements the following order:
    read-only platform geo lookup, and terminate locally. This applies to the
    entire reserved namespace, assets, unsupported methods, and disabled routes;
    the route statuses below authentication are never an auth exemption.
+   Preserve existing first-match `Settings::handler_for_path(req.uri().path())`
+   selection. Do not decode the path again for authentication or combine rules
+   selected from alternative spellings. An encoded namespace alias such as
+   `/%5Fts/trace` is rejected, never served: a matching `^/` rule challenges it
+   first, while a `^/_ts` rule that does not match its visible spelling does not
+   create a challenge and the alias receives the hardened path error. This
+   grants neither setup data nor a trace action.
 4. For all other paths, continue through the adapter's ordinary event context,
    authentication, request filters, geo enrichment, EC/EID processing, named
    routes, auction handling, telemetry, and publisher fallback.
@@ -685,33 +750,61 @@ CookieHealth
     valid_ec_format | valid_eids_format | valid_tester_value
     | valid_diagnostics_value | malformed | oversized
     | unsupported_value | multiple_values
-    | header_too_large | header_not_utf8
+    | header_too_large | header_not_utf8 | runtime_header_ambiguous
 ```
 
 Details describe shape, never value. `absent` has no detail; `duplicate` uses
-`multiple_values`; `unavailable` uses `header_too_large` or `header_not_utf8`;
+`multiple_values`; `unavailable` uses `header_too_large`, `header_not_utf8`,
+or `runtime_header_ambiguous`;
 and a valid state uses its cookie-specific valid detail.
 
 The classifier uses this deterministic contract:
 
-- Inspect all `Cookie` header fields in wire order, up to a combined 16 KiB.
-  Exceeding the cap or encountering any non-UTF-8 header makes all four states
-  `unavailable`; no partial result is presented as authoritative.
+- Inspect all incoming runtime-visible `Cookie` fields, measuring a combined
+  16 KiB as the sum of their `HeaderValue::as_bytes().len()` values. Include
+  runtime-added separators; exactly 16,384 bytes is within the cap. This is a
+  visible-field bound, not a guarantee about discarded original bytes.
+- Apply aggregate failures in this fixed precedence: above the cap gives
+  `unavailable/header_too_large`; otherwise any actual invalid UTF-8 gives
+  `unavailable/header_not_utf8`; otherwise the ambiguity rules below give
+  `unavailable/runtime_header_ambiguous`. Each aggregate failure applies to
+  all four cookie states and prevents trace activation. Check the complete
+  bounded collection before presenting cookie-specific results. Decode with
+  `str::from_utf8(value.as_bytes())`, not `HeaderValue::to_str()`, which rejects
+  some valid non-ASCII UTF-8.
+- Read Cookie-specific fidelity with `RequestIngress::header_fidelity(&COOKIE)`.
+  When octets are not `Preserved`, any Unicode replacement character U+FFFD
+  anywhere in any Cookie field is ambiguous. When field multiplicity is not
+  `Preserved`, any literal comma anywhere in any Cookie field is ambiguous,
+  including unrelated or quoted values. Never comma-split, recover guessed
+  field counts, or select a valid-looking session from part of a folded field.
+  Apply these fallbacks to `Unknown`, `Transformed`, `Unavailable` and missing
+  metadata alike; never select them by platform name. A non-preserved status
+  alone does not reject readable marker-free fields. Same-name/global field
+  order is unnecessary for exact occurrence counting and duplicate precedence.
+- This deliberately treats original valid EF BF BD and runtime replacement of
+  invalid FF as the same ambiguity when preservation is unproved, rather than
+  falsely labeling valid UTF-8 `header_not_utf8`. If the applicable axis is
+  explicitly `Preserved`, its marker proceeds through the ordinary grammar;
+  a comma is never a field separator and canonical value validators still
+  apply. The per-name axes are independent. These rules cannot detect original
+  information erased without a visible marker, and make no such promise.
 - Split each readable header on semicolons and trim optional ASCII whitespace.
   A valid pair contains a non-empty RFC 6265 token name, one `=`, and the
   remaining bytes as its value; additional `=` bytes belong to the value. Empty
   segments and malformed pairs with an unrelated name are ignored. A segment
   with no `=` counts as one malformed reserved occurrence only when its first
   whitespace-delimited token is exactly a reserved name; a name such as
-  `ts-ec-extra` remains unrelated. No malformed unrelated pair poisons a
-  reserved-cookie result.
+  `ts-ec-extra` remains unrelated. After the aggregate size, UTF-8 and ambiguity
+  checks above, no malformed unrelated pair poisons a reserved-cookie result.
 - Count exact, case-sensitive reserved names before passing values to existing
   parsers. Zero occurrences is `absent`; more than one is `duplicate`,
   regardless of whether one value would otherwise be valid. Duplicate
   precedence is therefore diagnostic rather than first- or last-value
   selection.
 - Per-value limits are 512 bytes for `ts-ec`, 8 KiB for `ts-eids`, and 16 bytes
-  each for `ts-tester` and `__Host-ts-console`. Only the 8 KiB EID limit is
+  each for `ts-tester` and `__Host-ts-console`, measured in visible UTF-8 bytes.
+  Only the 8 KiB EID limit is
   inherited (`MAX_EIDS_COOKIE_BYTES` in `ec/prebid_eids.rs`); the other limits
   are new trace-inspection bounds. The 512-byte EC limit is an outer guard,
   above the exact 71-character format accepted by `is_valid_ec_id` in
@@ -728,13 +821,34 @@ The classifier uses this deterministic contract:
   details `malformed`, `oversized`, or `unsupported_value`. Parser error text
   and the value itself never enter the report or logs.
 
-The parser inspects the incoming request before diagnostics-cookie sanitation,
+The parser freezes incoming runtime-visible facts before diagnostics-cookie sanitation,
 while preserving existing authoritative-cookie and consent semantics. It must
 scan without using the current lossy `CookieJar` representation, which skips
 malformed pairs and cannot preserve duplicate evidence. Inspection is
 read-only: it must not generate an EC, touch the identity graph, sync partner
 IDs, or extend any cookie lifetime.
 
+State observation and capture eligibility both use this same frozen result:
+exactly one `present_valid/valid_diagnostics_value` session is active; absent,
+invalid, duplicate or unavailable is inactive. Never reread sanitized cookies.
+Cookie ambiguity does not change the deliberate enable/end action policy:
+valid authenticated same-origin empty-body actions may request a cookie write
+or clear, and a separate state request must observe the result. End remains
+available when cookies are ambiguous. Ambiguity may leave activation
+unconfirmed; the UI must not claim cookie absence or successful browser storage.
+Do not change ordinary EC/EID, consent or existing TS Console behavior because
+trace inspection was unavailable.
+
+Add `runtime_header_ambiguous` to the exact Rust/TypeScript cookie detail enums,
+setup/context/report validators and storage/export fixtures. It is allowed only
+with `unavailable`; unknown details and invalid state/detail pairs still reject.
+Render a plain explanation that the runtime-visible cookies could not be
+reliably inspected. Retain the same reason in download/copy/share and direct
+storage-failure export. Never expose fidelity metadata, raw values or parser
+errors. Existing report versions and byte/depth/string bounds remain unchanged;
+include the new detail in worst-case size fixtures before the unpublished v1
+assets are frozen.
+
 Only those four Trusted Server-owned cookie names are reported. Arbitrary
 cookie names and values are excluded. The endpoint cannot claim knowledge of
 browser attributes, expiry, or cookies the browser withheld from the request.
@@ -962,6 +1076,14 @@ telemetry and OpenRTB objects:
   called, timed out, or returned no bid for a particular slot. Only returned
   bids can contribute to a slot's `returned_bid_count`; the fixed coverage
   value makes the missing provider-to-slot no-bid relation explicit.
+- A slot's `returned_bid_count` counts actual returned bid records whose
+  ordinary internal routing key matches that accepted slot. Accepted slots with
+  duplicate routing keys retain distinct ordinals and opaque refs, but share
+  that observed count. These counts are non-disjoint and must not be summed as
+  unique bids. When existing winner/delivery data cannot distinguish those
+  accepted instances, their candidate is `unknown` and
+  `selected_creative_size` is omitted. No internal routing key is exposed and
+  no instance-specific winner or GPT association is inferred.
 - `total_time_ms` and `response_time_ms` use the server's auction-local monotonic
   durations. They are not request-relative milestones and are never
   arithmetically combined with browser timestamps.
@@ -1273,7 +1395,13 @@ characters or bidirectional override/isolate controls. This applies to browser
 source fields as well as platform fields and precedes rendering or export.
 
 The snapshot builder creates a new field-by-field projection and rejects an
-invalid source value rather than stringifying it. Server auctions are already
+invalid value consumed by that public projection rather than stringifying it.
+Source containers must have the exact current keys and own data properties;
+unknown fields and accessors are rejected. Contents explicitly excluded in
+section 9.3 are never accepted, copied, traversed or validated. The required
+source `page.pathname` is checked only as an own string property before literal
+replacement; its contents and raw length never enter the report.
+Server auctions are already
 bounded by core; the browser rejects an invalid inner model rather than
 truncating it. The builder retains the newest 16 server auctions in observation
 order, the newest 128 correlations in recorder emission order, and only the
@@ -1460,7 +1588,8 @@ event, user identity, or security incident.
 
 ### 12.3 Response hardening
 
-The HTML shell, enable/end responses, every active diagnostic publisher
+For application-generated trace handling within section 8's boundary, the
+HTML shell, enable/end responses, every active diagnostic publisher
 response, and every dynamic page-bids or `/auction` response carrying trace
 evidence are terminally `private, no-store`. The fixed versioned JS/CSS assets
 are the sole exception and may be publicly cached because they contain no
@@ -1557,20 +1686,31 @@ shell or actions.
 
 ## 13. Failure handling
 
+Local status/hardening rules below apply to application-visible trace handling
+as defined in section 8; earlier runtime and adapter failures are separate.
+
 - Configured authentication failure: local private/no-store `401` challenge
   before trace handling, including on disabled routes.
 - Disabled route after authentication: local privacy-safe `404`.
 - Unsupported method: local `405`; never publisher fallback.
+- Runtime rejects before invocation, or adapter bootstrap/conversion fails
+  before trace dispatch: record the separate outcome defined in section 8;
+  do not invent a local hardened trace response or cookie-health report.
 - Reserved-namespace path that is a trailing slash, extra segment, unsupported
   asset name, repeated separator, or lookalike: local `404`; an encoded
-  separator or ambiguous dot segment: local `400`. Never publisher fallback.
+  namespace alias, encoded separator, or ambiguous dot segment still visible
+  at dispatch: local `400` after authentication and the enabled-feature check.
+  Never publisher fallback for a classified reserved request. A runtime-normalized
+  path outside the namespace retains ordinary behavior.
 - Non-empty or unreadable activation/end body: local `413` for any body bytes,
   positive/invalid `Content-Length`, or `Transfer-Encoding`, and local `400`
   for a stream read error, both with no cookie mutation.
 - Rejected activation/end POST: local `403` with no state mutation.
 - Optional platform fact unavailable: omit the field and continue.
 - Bounded cookie inspection failure: report the contract-defined invalid or
-  unavailable state without a value or parser message.
+  unavailable state without a value or parser message. Runtime ambiguity uses
+  `runtime_header_ambiguous`, disables trace capture and leaves ordinary
+  advertising/console behavior intact; valid end actions remain available.
 - Diagnostics context serialization failure: omit the context, log a bounded
   server error, and preserve publisher delivery.
 - Server-auction evidence construction or serialization failure: omit only the
@@ -1661,6 +1801,12 @@ results, never a prerequisite for returning them.
 - Cookie-health scanner covers multiple header fields; zero, one, and duplicate
   occurrences; mixed valid/invalid duplicates; non-UTF-8; malformed pairs; and
   per-value and total-header limits without retaining values.
+  Cover actual invalid bytes separately from valid non-ASCII UTF-8; missing and
+  every non-preserved fidelity status; independent per-Cookie octet/multiplicity
+  overrides; commas/U+FFFD in unrelated and quoted values; marker-free normal
+  sessions under Unknown; explicit Preserved marker semantics; and aggregate
+  size > actual invalid UTF-8 > runtime-ambiguity precedence. Ambiguity makes
+  all four states unavailable and suppresses tokens, evidence and trace sidecars.
 - Request-context serializer masks IPv4/IPv6; enforces every string bound; and
   omits page paths, fingerprints, query, raw headers, IDs, and unsupported
   fields.
@@ -1711,16 +1857,25 @@ results, never a prerequisite for returning them.
   sources.
 - Axum, Cloudflare, and Spin return the common route/schema with unavailable
   fields omitted.
-- Trace-route failures never fall through to publisher origin, including HEAD
-  on each exact path, malformed reserved paths, and arbitrary unsupported
+- Application-visible trace-route failures never fall through to publisher origin, including HEAD
+  on each exact path, visible malformed reserved paths, and arbitrary unsupported
   methods intercepted before router dispatch. Assert path-specific `Allow`,
   bodyless HEAD errors, and hardening headers on local errors.
+  Separately exercise real runtime method/parser rejection, adapter conversion
+  failure and transformed application handling. Cloudflare wire 501 is not a
+  successful local 405; Spin invalid-byte rejection is not cookie-health output.
+  Paths normalized into trace routes require auth; paths normalized to health,
+  JA4 or publisher paths retain ordinary behavior. Missing/transformed/oversize
+  original-target metadata must not reject an inspectable normal pathname.
 - Broad `^/_ts` and `^/` authentication rules challenge every trace path
   (shell/state/actions/assets, disabled routes, and unsupported methods); valid
   credentials proceed to trace handling, while `^/_ts/admin` leaves trace
   routes public. Challenges expose no context or cookie mutation, and
   protected assets remain private/no-store. Also test an earlier narrow handler
   shadowing a broad rule to pin first-match-wins behavior.
+  For encoded namespace aliases, pin literal visible-path auth selection:
+  `/%5Fts/trace` challenges under `^/`, but under a lone `^/_ts` rule is locally
+  rejected without serving setup data or performing an action.
 - GET, HEAD, state-changing POST, and unsupported methods obey the same
   lifecycle contract across adapters.
 - Enable/end POSTs without `Content-Type` accept empty bodies and reject actual
@@ -1801,7 +1956,11 @@ results, never a prerequisite for returning them.
   correlated auctions, fails snapshot creation without dropping a slot's last
   cycle or offering a combined-report export.
 - Same-tab navigation occurs only after a successful write.
-- Viewer handles absent optional network facts and every cookie-health state.
+- Viewer handles absent optional network facts and every cookie-health state,
+  including the exact unavailable/runtime_header_ambiguous detail. Storage,
+  download, copy, share and direct export retain the same reason; invalid
+  state/detail pairs and unknown reasons reject. Worst-case size fixtures
+  include the new detail without expanding the existing report bounds.
 - Populate every excluded `adManager` field, `previousCreativeId`, slot
   `slotElementId`/`adUnitPath`, and callback/attribution issue `slotElementId`
   with distinct sentinel values, including a synthetic secret-bearing path.
@@ -1891,6 +2050,12 @@ fixture:
 observed`, `GPT filled/rendered`, and `Unknown` without understanding internal
   request-path names.
 - A failed share or download does not lose the visible report.
+- Record the all-adapter normal setup → enable → state observation → eligible
+  reload → capture → view/export → end → state observation journey on suitable
+  same-origin fixtures. Include marker-free cookies with Unknown fidelity.
+  Do not treat malformed-wire rejection or synthetic converter-only tests as
+  evidence that this browser journey succeeded. Mobile Safari/Chrome acceptance
+  uses the supported deployment origin and retains Secure cookie attributes.
 
 ## 15. Rollout and observability
 
@@ -1957,11 +2122,23 @@ observed`, `GPT filled/rendered`, and `Unknown` without understanding internal
     Trusted Server auction API ran; show its bounded provider and per-slot
     outcome; show subsequent GPT/creative evidence; and display `Unknown`
     rather than inventing a client-side winner or an unsupported correlation.
+16. The normal accepted journey works on all four adapters within the documented
+    application boundary. Visible reserved paths terminate locally with the
+    required authentication/hardening; runtime rejection, conversion failure and
+    normalized non-trace paths are recorded separately. No original-target SDK
+    accessor or byte-exact reconstruction is a v1 dependency.
+17. Cookie ambiguity is reported as unavailable/runtime_header_ambiguous and
+    cannot authorize capture. Unknown fidelity alone does not disable a normal
+    readable session. The reason survives strict Rust/JS validation, the viewer,
+    storage and every export path without exposing raw data or changing
+    ordinary advertising.
 
 ## 17. Implementation sequencing
 
-This design is one product flow, but its implementation is split into four
-independently reviewable plans and preferably four PRs:
+This design has one implementation plan with four reviewable phases. The first
+three together form the complete mandatory v1 release; phase four is optional.
+Keep one spec and one plan on the existing branch. Phase checkpoints do not
+authorize publishing a partial endpoint or creating separate plans:
 
 1. **Reserved route and privacy foundation:** configuration, shared early-route
    classification with operator authentication, same-origin enable/end lifecycle,
@@ -1970,7 +2147,7 @@ independently reviewable plans and preferably four PRs:
    `ClientInfo`/`GeoInfo` fields, response hardening, and adapter parity. Do not
    add speculative new platform fields in this change. Include the raw-config
    validation hook and method-independent dispatch on all four adapters in
-   this first PR; Fastly-only route registration does not satisfy the contract.
+   this first phase; Fastly-only route registration does not satisfy the contract.
 2. **Live server-auction evidence:** introduce the public diagnostic auction and
    slot tokens, project `TraceAuctionEvidenceV1` at the live observation
    boundary, transport it through initial navigation, page-bids, and both
@@ -1987,7 +2164,7 @@ independently reviewable plans and preferably four PRs:
    bounds. Adopt #1081 or #1074/#1076 later through a separately reviewed
    versioned compatibility change.
 
-Each plan must include its own adapter, privacy, cache, and failure tests. The
+Each phase must include its relevant adapter, privacy, cache, and failure tests. The
 implementation must not claim completion of #1081 or request-relative timing as
 part of #1050. Version one ships with minimal live server-auction evidence plus
 current GPT/render evidence and labels unavailable fields honestly.
@@ -2044,6 +2221,16 @@ length, history, logging, referrer, and accidental-sharing risks.
 
 ## 19. Known limitations
 
+- Original wire information is outside the v1 guarantee. Runtime/SDK path
+  normalization can turn an originally trace-like spelling into an ordinary
+  request, including Fastly health/JA4 behavior. Runtime rejection and adapter
+  conversion failures cannot provide application trace responses.
+- Cookie bounds and counts concern visible fields. Comma/replacement markers
+  conservatively suppress tracing when the applicable fidelity axis is not
+  Preserved, including valid unrelated values that happen to contain markers.
+  Transformations that erase ambiguity without a marker cannot be inferred.
+  Do not present an unavailable result as cookie absence or original-wire proof.
+  Resolved dependency/runtime changes require rerunning the semantic probes.
 - The user must reproduce the problem after enabling tracing.
 - Same-tab navigation is the supported workflow, but opener-created tabs and
   browser session restore may copy or retain session storage. JSON export is
diff --git a/scripts/generate-integration-viceroy-configs.sh b/scripts/generate-integration-viceroy-configs.sh
index 761d06926..eef2a0e3c 100755
--- a/scripts/generate-integration-viceroy-configs.sh
+++ b/scripts/generate-integration-viceroy-configs.sh
@@ -12,7 +12,7 @@ ORIGIN_PORT="${INTEGRATION_ORIGIN_PORT:-8888}"
 ARTIFACTS_DIR="${ARTIFACTS_DIR:-$REPO_ROOT/target/integration-test-artifacts}"
 CONFIG_DIR="$ARTIFACTS_DIR/configs"
 TEMPLATE_PATH="crates/trusted-server-integration-tests/fixtures/configs/viceroy-template.toml"
-APP_CONFIG_PATH="crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml"
+APP_CONFIG_PATH="${INTEGRATION_APP_CONFIG_PATH:-crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml}"
 INTEGRATION_TARGET_DIR="crates/trusted-server-integration-tests/target"
 ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT"
 HOST_TARGET="$(rustc -vV | sed -n 's/^host: //p')"
@@ -36,8 +36,14 @@ if [ ! -x "$GENERATOR_BIN" ]; then
     exit 1
 fi
 
-"$GENERATOR_BIN" \
-    --template "$TEMPLATE_PATH" \
-    --app-config "$APP_CONFIG_PATH" \
-    --output "$CONFIG_DIR/viceroy.toml" \
+GENERATOR_ARGS=(
+    --template "$TEMPLATE_PATH"
+    --app-config "$APP_CONFIG_PATH"
+    --output "$CONFIG_DIR/viceroy.toml"
     --origin-url "$ORIGIN_URL"
+)
+if [ -n "${INTEGRATION_BIDDER_ORIGIN_URL:-}" ]; then
+    GENERATOR_ARGS+=(--bidder-origin-url "$INTEGRATION_BIDDER_ORIGIN_URL")
+fi
+
+"$GENERATOR_BIN" "${GENERATOR_ARGS[@]}"
diff --git a/scripts/integration-tests-browser.sh b/scripts/integration-tests-browser.sh
index e81767be3..0d1e6eb51 100755
--- a/scripts/integration-tests-browser.sh
+++ b/scripts/integration-tests-browser.sh
@@ -19,6 +19,10 @@ cd "$REPO_ROOT"
 ORIGIN_PORT="${INTEGRATION_ORIGIN_PORT:-8888}"
 BROWSER_DIR="crates/trusted-server-integration-tests/browser"
 TSJS_LIB_DIR="crates/trusted-server-js/lib"
+BROWSER_ARTIFACTS_DIR="${ARTIFACTS_DIR:-$REPO_ROOT/target/integration-test-artifacts}"
+if [[ "$BROWSER_ARTIFACTS_DIR" != /* ]]; then
+    BROWSER_ARTIFACTS_DIR="$REPO_ROOT/$BROWSER_ARTIFACTS_DIR"
+fi
 NODE_VERSION="$(grep '^nodejs ' .tool-versions | awk '{print $2}')"
 
 if [ -z "$NODE_VERSION" ]; then
@@ -36,8 +40,16 @@ TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK=false \
     cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
 
 echo "==> Generating Viceroy configs..."
-INTEGRATION_ORIGIN_PORT="$ORIGIN_PORT" ./scripts/generate-integration-viceroy-configs.sh
-GENERATED_VICEROY_CONFIG_PATH="$REPO_ROOT/target/integration-test-artifacts/configs/viceroy.toml"
+INTEGRATION_ORIGIN_PORT="$ORIGIN_PORT" \
+ARTIFACTS_DIR="$BROWSER_ARTIFACTS_DIR" ./scripts/generate-integration-viceroy-configs.sh
+GENERATED_VICEROY_CONFIG_PATH="$BROWSER_ARTIFACTS_DIR/configs/viceroy.toml"
+INTEGRATION_ORIGIN_PORT="$ORIGIN_PORT" \
+INTEGRATION_APP_CONFIG_PATH="crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace.toml" \
+INTEGRATION_BIDDER_ORIGIN_URL="http://127.0.0.1:$ORIGIN_PORT" \
+ARTIFACTS_DIR="$BROWSER_ARTIFACTS_DIR/trace" ./scripts/generate-integration-viceroy-configs.sh
+INTEGRATION_ORIGIN_PORT="$ORIGIN_PORT" \
+INTEGRATION_APP_CONFIG_PATH="crates/trusted-server-integration-tests/fixtures/configs/trusted-server.trace-auth.toml" \
+ARTIFACTS_DIR="$BROWSER_ARTIFACTS_DIR/trace-auth" ./scripts/generate-integration-viceroy-configs.sh
 
 # --- Build Docker images ---
 echo "==> Building WordPress test container..."
@@ -68,6 +80,10 @@ cd "$REPO_ROOT/$BROWSER_DIR"
 export WASM_BINARY_PATH="$REPO_ROOT/target/wasm32-wasip1/release/trusted-server-adapter-fastly.wasm"
 export INTEGRATION_ORIGIN_PORT="$ORIGIN_PORT"
 export VICEROY_CONFIG_PATH="$GENERATED_VICEROY_CONFIG_PATH"
+export TRACE_VICEROY_CONFIG_PATH="$BROWSER_ARTIFACTS_DIR/trace/configs/viceroy.toml"
+export TRACE_AUTH_VICEROY_CONFIG_PATH="$BROWSER_ARTIFACTS_DIR/trace-auth/configs/viceroy.toml"
+
+node --test trace-fixture.test.cjs
 
 # Cleanup trap: stop any leftover containers on failure
 stop_matching_containers() {
diff --git a/scripts/integration-tests.sh b/scripts/integration-tests.sh
index 96e492f40..8053418d3 100755
--- a/scripts/integration-tests.sh
+++ b/scripts/integration-tests.sh
@@ -22,6 +22,8 @@ ORIGIN_PORT="${INTEGRATION_ORIGIN_PORT:-8888}"
 NODE_VERSION="$(grep '^nodejs ' .tool-versions | awk '{print $2}')"
 TEST_ARGS=("$@")
 SKIP_DUPLICATE_HELPERS=true
+TRACE_RUNTIME_REQUESTED=false
+TRACE_SKIP_VALUE=false
 
 if [ -z "$NODE_VERSION" ]; then
     echo "Failed to detect Node.js version from .tool-versions" >&2
@@ -34,10 +36,27 @@ for arg in "$@"; do
             SKIP_DUPLICATE_HELPERS=false
             ;;
     esac
+    if [ "$TRACE_SKIP_VALUE" = true ]; then
+        TRACE_SKIP_VALUE=false
+        continue
+    fi
+    case "$arg" in
+        --skip)
+            TRACE_SKIP_VALUE=true
+            ;;
+        trace_browser_workflow*|trace_runtime_boundary*)
+            TRACE_RUNTIME_REQUESTED=true
+            ;;
+    esac
 done
 
 if [ "$SKIP_DUPLICATE_HELPERS" = true ]; then
-    TEST_ARGS=(--skip test_wordpress_fastly --skip test_nextjs_fastly "${TEST_ARGS[@]}")
+    TEST_ARGS=(--skip test_wordpress_fastly --skip test_nextjs_fastly "$@")
+fi
+
+# These opt-in probes require separately prepared browser and Spin artifacts.
+if [ "$TRACE_RUNTIME_REQUESTED" = false ]; then
+    TEST_ARGS=(--skip trace_browser_workflow --skip trace_runtime_boundary "${TEST_ARGS[@]}")
 fi
 
 # Detect native target from rustc (handles all OS + arch combinations correctly)
diff --git a/trusted-server.example.toml b/trusted-server.example.toml
index f552c989e..c4e057598 100644
--- a/trusted-server.example.toml
+++ b/trusted-server.example.toml
@@ -629,9 +629,11 @@ gam_attribution_enabled = false
 
 # GPT runtime diagnostics browser overlay. Optional and enabled manually (not
 # flipped by `ts audit`); serves a diagnostics module gated behind an activation
-# query param + session cookie.
+# query param + host-only cookie. Explicit activation lasts 30 minutes;
+# ordinary requests do not extend it. Mobile trace remains off by default.
 # [integrations.gpt_diagnostics]
 # enabled = true
+# trace_page_enabled = false
 
 # APS browser renderer ownership. Server-side APS behavior belongs under an
 # [auction.providers] entry with profile = "aps".

From e076f77a10a1fd8b485a6b5875fe774b00336634 Mon Sep 17 00:00:00 2001
From: prk-Jr 
Date: Tue, 6 Oct 2026 12:06:21 +0530
Subject: [PATCH 131/230] Link trace release checklist to the repository plan

---
 docs/guide/integrations/gpt-diagnostics.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md
index 88e73f0b0..d65567766 100644
--- a/docs/guide/integrations/gpt-diagnostics.md
+++ b/docs/guide/integrations/gpt-diagnostics.md
@@ -213,7 +213,7 @@ There is no historical recovery or server-side report store.
 ### Release acceptance
 
 Keep `trace_page_enabled` off until the required automated checks in the
-[implementation plan](../../superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md)
+[implementation plan](https://github.com/IABTechLab/trusted-server/blob/750ba8dc470aa37f3eb62512cc2b5a025bab8f9d/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md)
 pass. The browser checks must exercise the complete setup, real cookie observation,
 publisher reload, capture, viewer, export and end flow on each supported adapter.
 Real bidder transport and creative rendering require their separate live checks;

From b2930d7b9131535722c2400fab1d0f4dbd479839 Mon Sep 17 00:00:00 2001
From: prk-Jr 
Date: Tue, 6 Oct 2026 13:03:26 +0530
Subject: [PATCH 132/230] Preserve advertising behavior and trace controls

---
 Cargo.lock                                    |  16 +-
 Cargo.toml                                    |  12 +-
 crates/trusted-server-core/src/publisher.rs   | 197 +++++++++++++++++-
 .../integrations/gpt_diagnostics/overlay.ts   |  97 +++++----
 .../gpt_diagnostics/overlay.test.ts           |  50 +++++
 .../lib/test/trace/handoff.test.ts            |   6 +-
 .../lib/test/trace/projection.test.ts         |   4 +-
 .../lib/test/trace/report.test.ts             |   2 +-
 docs/guide/integrations/gpt-diagnostics.md    |  27 ++-
 ...ile-ad-render-trace-implementation-plan.md |  64 +++++-
 ...-mobile-ad-render-trace-endpoint-design.md |  12 ++
 11 files changed, 421 insertions(+), 66 deletions(-)

diff --git a/Cargo.lock b/Cargo.lock
index 10dd15c38..f336094aa 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1428,7 +1428,7 @@ dependencies = [
 [[package]]
 name = "edgezero-adapter"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "toml",
 ]
@@ -1436,7 +1436,7 @@ dependencies = [
 [[package]]
 name = "edgezero-adapter-axum"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -1464,7 +1464,7 @@ dependencies = [
 [[package]]
 name = "edgezero-adapter-cloudflare"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -1487,7 +1487,7 @@ dependencies = [
 [[package]]
 name = "edgezero-adapter-fastly"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "anyhow",
  "async-stream",
@@ -1516,7 +1516,7 @@ dependencies = [
 [[package]]
 name = "edgezero-adapter-spin"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -1543,7 +1543,7 @@ dependencies = [
 [[package]]
 name = "edgezero-cli"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "chrono",
  "clap",
@@ -1568,7 +1568,7 @@ dependencies = [
 [[package]]
 name = "edgezero-core"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "anyhow",
  "async-compression",
@@ -1599,7 +1599,7 @@ dependencies = [
 [[package]]
 name = "edgezero-macros"
 version = "0.1.0"
-source = "git+https://github.com/stackpop/edgezero?rev=499d5c93d597f01b5e0497e332af82f2aa633277#499d5c93d597f01b5e0497e332af82f2aa633277"
+source = "git+https://github.com/stackpop/edgezero?rev=75067d2c9a3cf865591665e88a736db4c8a13be0#75067d2c9a3cf865591665e88a736db4c8a13be0"
 dependencies = [
  "log",
  "proc-macro2",
diff --git a/Cargo.toml b/Cargo.toml
index d82609a0f..471321a7e 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -55,12 +55,12 @@ cssparser = "0.36"
 derive_more = { version = "2.0", features = ["display", "error"] }
 directories = "5"
 ed25519-dalek = { version = "2.2", features = ["rand_core"] }
-edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false }
-edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false }
-edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false }
-edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false }
-edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277" }
-edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "499d5c93d597f01b5e0497e332af82f2aa633277", default-features = false }
+edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0", default-features = false }
+edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0", default-features = false }
+edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0", default-features = false }
+edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0", default-features = false }
+edgezero-cli = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0" }
+edgezero-core = { git = "https://github.com/stackpop/edgezero", rev = "75067d2c9a3cf865591665e88a736db4c8a13be0", default-features = false }
 env_logger = "0.11"
 error-stack = "0.6"
 esi = "0.7.2"
diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs
index a546b4b03..a4f07defa 100644
--- a/crates/trusted-server-core/src/publisher.rs
+++ b/crates/trusted-server-core/src/publisher.rs
@@ -3283,6 +3283,8 @@ pub(crate) struct AdBidsState {
     debug_prefix: Arc>,
     /// Optional trace carry belongs to this request, outside the ordinary bid map.
     trace: Option,
+    /// Withheld ad-slot injection permits only trace observation at the body seam.
+    trace_only: bool,
 }
 
 #[cfg(test)]
@@ -3305,7 +3307,11 @@ impl AdBidsState {
     /// Record one auction result, rendering the script from the same map that is
     /// stored, so the two representations cannot drift.
     fn set(&self, bid_map: serde_json::Map) {
-        let bids_script = build_bids_script_with_trace(&bid_map, self.trace());
+        let bids_script = if self.trace_only {
+            build_trace_only_script(self.trace())
+        } else {
+            build_bids_script_with_trace(&bid_map, self.trace())
+        };
         *self.script.lock().expect("should lock bid script") = Some(bids_script);
         *self.bids.lock().expect("should lock bid map") = bid_map;
     }
@@ -3327,6 +3333,11 @@ impl AdBidsState {
         self.set(self.bids());
     }
 
+    fn withhold_ad_initialization(&mut self) {
+        self.trace_only = true;
+        self.set(self.bids());
+    }
+
     /// Build the shared-template seam, retaining the same debug prefix as inline.
     fn build_seam_script(&self, slots_json: &str) -> String {
         let seam = match self.trace() {
@@ -5423,6 +5434,10 @@ pub async fn handle_publisher_request(
         )
     };
 
+    if ad_slots_script.is_none() && trace_auction.is_some() {
+        ad_bids_state.withhold_ad_initialization();
+    }
+
     // §4.7: HTML with synthesized per-navigation auction state must not be
     // stored or validated as an origin representation. Strip both browser and
     // surrogate validators/cache directives before returning it.
@@ -6209,6 +6224,26 @@ fn trace_transport_script(trace: Option<&TraceAuctionCarry>) -> Option {
         })
 }
 
+fn build_trace_only_script(trace: Option<&TraceAuctionCarry>) -> String {
+    let Some(transport) = trace_transport_script(trace) else {
+        return String::new();
+    };
+    format!(
+        "",
+        html_escape_for_script(&transport)
+    )
+}
+
 fn trace_transport_value(transport: &crate::trace::TraceAuctionTransportV1) -> serde_json::Value {
     serde_json::to_value(transport).unwrap_or_else(|_| trace_serialization_unavailable())
 }
@@ -7495,10 +7530,6 @@ pub async fn handle_page_bids(
             None,
         )
     });
-    if let Some(trace) = &trace_auction {
-        trace.observe_delivery(&winning_bids, &bid_map.keys().cloned().collect());
-    }
-
     // Gate slots on the ad-stack kill switch / consent: when disabled, return no
     // slots so the SPA hook does not call `adInit()` / create GPT slots.
     let slots_json: Vec = if ad_stack_enabled {
@@ -15734,8 +15765,8 @@ mod tests {
                             "should deliver the directly observed skipped auction even with zero bids"
                         );
                         assert!(
-                            document.contains("s(b,undefined,x)"),
-                            "should supply optional transport to the initial scheduler"
+                            !document.contains("s(b,undefined,x)"),
+                            "should withhold the ad scheduler when ordinary ad slots are absent"
                         );
                     } else {
                         assert!(
@@ -15746,6 +15777,158 @@ mod tests {
                 }
             }
 
+            #[cfg(not(target_arch = "wasm32"))]
+            #[tokio::test]
+            #[ignore = "requires the declared Node toolchain for emitted-script execution"]
+            async fn trace_document_skipped_emitted_script_leaves_ad_state_untouched() {
+                for gate in [
+                    "missing_opportunities",
+                    "auction_disabled",
+                    "consent_denied",
+                    "unmatched_path",
+                ] {
+                    for finalizer in [Finalizer::Buffered, Finalizer::Streaming] {
+                        let stub = Arc::new(StubHttpClient::new());
+                        let services = services_for_ip(
+                            Arc::clone(&stub),
+                            Arc::new(MemoryTemplateCache::default()),
+                            IpAddr::V4(Ipv4Addr::new(192, 0, 2, 99)),
+                        );
+                        queue_shareable_html(&stub);
+                        let mut settings = settings_with_mode("inline");
+                        settings.auction.enabled = gate != "auction_disabled";
+                        if gate == "missing_opportunities" {
+                            settings.creative_opportunities = None;
+                        }
+                        settings
+                            .integrations
+                            .insert_config(
+                                "gpt_diagnostics",
+                                &serde_json::json!({"enabled":true,"trace_page_enabled":true}),
+                            )
+                            .expect("should enable trace");
+                        let settings = Arc::new(settings);
+                        let mut request = conditional_trace_request(true);
+                        if gate == "unmatched_path" {
+                            *request.uri_mut() = "https://ts.example.com/unmatched"
+                                .parse()
+                                .expect("should build unmatched path");
+                        }
+                        TracePreDispatchHook::new(
+                            Arc::clone(&settings),
+                            Arc::new(|_| panic!("should not load setup metadata")),
+                        )
+                        .handle(&mut request)
+                        .await
+                        .expect("should freeze trace gate");
+                        crate::integrations::gpt_diagnostics::prepare_request(
+                            &settings,
+                            &mut request,
+                        )
+                        .expect("should freeze diagnostics decision");
+                        let mut ec_context = EcContext::new_for_test(
+                            None,
+                            crate::consent::ConsentContext {
+                                jurisdiction: if gate == "consent_denied" {
+                                    crate::consent::jurisdiction::Jurisdiction::Gdpr
+                                } else {
+                                    crate::consent::jurisdiction::Jurisdiction::NonRegulated
+                                },
+                                ..Default::default()
+                            },
+                        );
+                        let orchestrator =
+                            Arc::new(AuctionOrchestrator::new(settings.auction.clone()));
+                        let registry = IntegrationRegistry::new(&settings)
+                            .expect("should register integrations");
+                        let response = handle_publisher_request(
+                            &settings,
+                            &services,
+                            None,
+                            &mut ec_context,
+                            AuctionDispatch {
+                                orchestrator: &orchestrator,
+                                slots: &[article_slot()],
+                                registry: None,
+                            },
+                            request,
+                            EdgeCacheHeader::SMaxageFallback,
+                        )
+                        .await
+                        .expect("should preserve publisher response");
+                        let document = String::from_utf8(
+                            body_of(
+                                finalize_test_publisher_response(
+                                    response,
+                                    &settings,
+                                    &services,
+                                    ®istry,
+                                    orchestrator,
+                                    finalizer,
+                                )
+                                .await,
+                            )
+                            .await,
+                        )
+                        .expect("should render publisher HTML");
+                        let script = document
+                            .rsplit_once("")
+                            .expect("should close the body script")
+                            .0;
+                        let probe = r#"
+const assert = require('node:assert/strict');
+const vm = require('node:vm');
+const script = process.argv[1];
+for (const mode of ['ready', 'queued', 'stale', 'inactive', 'throws']) {
+  const bids = {example: 'publisher-owned'};
+  const slots = [{id: 'publisher-owned'}];
+  const records = [];
+  let schedulerCalls = 0;
+  let adInitCalls = 0;
+  const api = {bids, adSlots: slots, initialAdInitScheduled: false, navGeneration: 0,
+    scheduleInitialAdInit() {schedulerCalls++;},
+    adInit() {adInitCalls++;}};
+  const bridge = {observeTransport(delivered, transport, source) {
+    if (mode === 'throws') throw Error('private diagnostic failure');
+    assert.equal(delivered, undefined);
+    assert.equal(source, 'initial_navigation_ssat');
+    assert.equal(transport.evidence.terminal_status, 'skipped');
+    records.push(transport);
+  }};
+  if (mode !== 'queued' && mode !== 'stale') api.traceGpt = bridge;
+  const window = {tsjs: api, __tsjs_trace_active: mode !== 'inactive'};
+  vm.runInNewContext(script, {window});
+  if (mode === 'queued' || mode === 'stale') {
+    assert.equal(records.length, 0);
+    assert.equal(api.que.length, 1);
+    api.traceGpt = bridge;
+    if (mode === 'stale') api.navGeneration = 1;
+    api.que[0]();
+  }
+  assert.equal(records.length, mode === 'ready' || mode === 'queued' ? 1 : 0);
+  assert.equal(api.bids, bids);
+  assert.equal(api.adSlots, slots);
+  assert.equal(api.initialAdInitScheduled, false);
+  assert.equal(schedulerCalls, 0);
+  assert.equal(adInitCalls, 0);
+}
+"#;
+                        let output = std::process::Command::new("node")
+                            .args(["-e", probe, script])
+                            .output()
+                            .expect("should execute the emitted script with Node");
+                        assert!(
+                            output.status.success(),
+                            "should observe only trace evidence: {}",
+                            String::from_utf8_lossy(&output.stderr)
+                        );
+                    }
+                }
+            }
+
             #[tokio::test]
             async fn trace_document_reload_rejects_unexpected_origin_304_without_the_ad_stack() {
                 for content_type in [None, Some("text/html")] {
diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts
index f419b7aee..0042ed65f 100644
--- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts
+++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts
@@ -525,7 +525,16 @@ export class GptDiagnosticsOverlay {
         .map((details) => details.closest('.tsgd-slot')?.dataset.runtimeSlot)
         .filter((runtimeSlot): runtimeSlot is string => runtimeSlot !== undefined)
     );
-    panel.replaceChildren();
+    const retainedToolbar =
+      this.onViewTrace && !this.collapsed
+        ? panel.querySelector('.tsgd-toolbar')
+        : null;
+    const retainedTraceStatus = retainedToolbar
+      ? panel.querySelector('.tsgd-trace-status')
+      : null;
+    for (const child of Array.from(panel.children)) {
+      if (child !== retainedToolbar && child !== retainedTraceStatus) child.remove();
+    }
 
     const header = this.document.createElement('header');
     header.className = 'tsgd-header';
@@ -542,51 +551,62 @@ export class GptDiagnosticsOverlay {
     collapse.setAttribute('aria-expanded', String(!this.collapsed));
     const close = this.button('Close', () => this.hide());
     header.append(title, status, collapse, close);
-    panel.append(header);
+    panel.prepend(header);
 
     if (this.collapsed) return;
 
-    const toolbar = this.document.createElement('div');
+    const toolbar = retainedToolbar ?? this.document.createElement('div');
     toolbar.className = 'tsgd-toolbar';
-    const filterLabel = this.document.createElement('label');
-    filterLabel.textContent = 'Filter';
-    const select = this.document.createElement('select');
-    select.setAttribute('aria-label', 'Filter diagnostic slots');
-    const filters: Array<[GptDiagnosticsFilter, string]> = [
-      ['all', 'All'],
-      ['visible', 'Visible'],
-      ['filled', 'Filled'],
-      ['empty', 'Empty'],
-      ['pending', 'Pending/Incomplete'],
-      ['unbound', 'Unbound/Ambiguous'],
-    ];
-    for (const [value, label] of filters) {
-      const option = this.document.createElement('option');
-      option.value = value;
-      option.textContent = label;
-      option.selected = this.filter === value;
-      select.append(option);
+    if (!retainedToolbar) {
+      const filterLabel = this.document.createElement('label');
+      filterLabel.textContent = 'Filter';
+      const select = this.document.createElement('select');
+      select.setAttribute('aria-label', 'Filter diagnostic slots');
+      const filters: Array<[GptDiagnosticsFilter, string]> = [
+        ['all', 'All'],
+        ['visible', 'Visible'],
+        ['filled', 'Filled'],
+        ['empty', 'Empty'],
+        ['pending', 'Pending/Incomplete'],
+        ['unbound', 'Unbound/Ambiguous'],
+      ];
+      for (const [value, label] of filters) {
+        const option = this.document.createElement('option');
+        option.value = value;
+        option.textContent = label;
+        option.selected = this.filter === value;
+        select.append(option);
+      }
+      select.addEventListener('change', () => {
+        this.filter = select.value as GptDiagnosticsFilter;
+        this.render();
+      });
+      const exportButton = this.button('Export JSON', () => this.onExport());
+      filterLabel.append(select);
+      toolbar.append(filterLabel, exportButton);
     }
-    select.addEventListener('change', () => {
-      this.filter = select.value as GptDiagnosticsFilter;
-      this.render();
-    });
-    const exportButton = this.button('Export JSON', () => this.onExport());
-    filterLabel.append(select);
-    toolbar.append(filterLabel, exportButton);
     if (this.onViewTrace) {
-      const viewTrace = this.button('View trace results', this.onViewTrace);
+      const viewTrace =
+        toolbar.querySelector('[data-trace-action="view"]') ??
+        this.button('View trace results', this.onViewTrace);
       viewTrace.className = 'tsgd-trace-action';
-      viewTrace.disabled =
-        this.traceState.kind === 'capturing' || this.traceState.kind === 'navigating';
-      toolbar.prepend(viewTrace);
+      viewTrace.dataset.traceAction = 'view';
+      viewTrace.disabled = this.traceState.kind === 'capturing';
+      if (!viewTrace.parentNode) toolbar.prepend(viewTrace);
+      const existingDownload = toolbar.querySelector(
+        '[data-trace-action="download"]'
+      );
       if (this.traceState.downloadAvailable && this.onDownloadTrace) {
-        const download = this.button('Download trace report', this.onDownloadTrace);
+        const download =
+          existingDownload ?? this.button('Download trace report', this.onDownloadTrace);
         download.className = 'tsgd-trace-action';
-        toolbar.append(download);
+        download.dataset.traceAction = 'download';
+        if (!download.parentNode) toolbar.append(download);
+      } else {
+        existingDownload?.remove();
       }
     }
-    panel.append(toolbar);
+    if (!retainedToolbar) panel.append(toolbar);
     if (this.onViewTrace) {
       const messages: Record = {
         ready: 'Capture the current page, then view its trace in this tab.',
@@ -601,12 +621,13 @@ export class GptDiagnosticsOverlay {
           'The download could not be started. Your report remains available for retry.',
         downloaded: 'The download was started. Your report remains available.',
       };
-      const traceStatus = this.document.createElement('p');
+      const traceStatus = retainedTraceStatus ?? this.document.createElement('p');
       traceStatus.className = 'tsgd-trace-status';
       traceStatus.setAttribute('role', 'status');
       traceStatus.setAttribute('aria-live', 'polite');
-      traceStatus.textContent = messages[this.traceState.kind];
-      panel.append(traceStatus);
+      const message = messages[this.traceState.kind];
+      if (traceStatus.textContent !== message) traceStatus.textContent = message;
+      if (!retainedTraceStatus) panel.append(traceStatus);
     }
 
     const summary = this.document.createElement('div');
diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts
index 7ff4faead..722209893 100644
--- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts
+++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts
@@ -86,6 +86,56 @@ afterEach(() => {
 });
 
 describe('GptDiagnosticsOverlay', () => {
+  it.each(['View trace results', 'Download trace report'])(
+    'retains focused %s and the live region across GPT updates',
+    (label) => {
+      const store = new GptDiagnosticsStore({ schedule: (callback) => callback() });
+      let root: ShadowRoot | undefined;
+      const overlay = new GptDiagnosticsOverlay(store, new FakeBindings(), {
+        scheduleFrame: (callback) => callback(),
+        onShadowRoot: (created) => {
+          root = created;
+        },
+        onViewTrace: vi.fn(),
+        onDownloadTrace: vi.fn(),
+      });
+      if (!root) throw new Error('should mount diagnostics');
+      try {
+        overlay.setTraceState({ kind: 'storage_unavailable', downloadAvailable: true });
+        const control = button(root, label);
+        const status = root.querySelector('[aria-live="polite"]');
+        control.focus();
+        expect(root.activeElement).toBe(control);
+        store.recordSlotRequested(slot('example-slot'));
+        expect(button(root, label)).toBe(control);
+        expect(root.activeElement).toBe(control);
+        expect(root.querySelector('[aria-live="polite"]')).toBe(status);
+      } finally {
+        overlay.destroy();
+      }
+    }
+  );
+
+  it('allows an explicit retry when navigation did not depart from the document', () => {
+    let root: ShadowRoot | undefined;
+    const onViewTrace = vi.fn();
+    const overlay = new GptDiagnosticsOverlay(new GptDiagnosticsStore(), new FakeBindings(), {
+      scheduleFrame: (callback) => callback(),
+      onShadowRoot: (created) => {
+        root = created;
+      },
+      onViewTrace,
+    });
+    if (!root) throw new Error('should mount diagnostics');
+    try {
+      overlay.setTraceState({ kind: 'navigating', downloadAvailable: true });
+      button(root, 'View trace results').click();
+      expect(onViewTrace).toHaveBeenCalledTimes(1);
+    } finally {
+      overlay.destroy();
+    }
+  });
+
   it('adds an optional prominent trace action and independent storage recovery without replacing GPT export', () => {
     let root: ShadowRoot | undefined;
     const onViewTrace = vi.fn();
diff --git a/crates/trusted-server-js/lib/test/trace/handoff.test.ts b/crates/trusted-server-js/lib/test/trace/handoff.test.ts
index a9a9b5eb7..235620e0a 100644
--- a/crates/trusted-server-js/lib/test/trace/handoff.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/handoff.test.ts
@@ -85,7 +85,7 @@ describe('explicit same-tab trace handoff', () => {
     action.view();
     expect(fixture.target.location.assign).not.toHaveBeenCalled();
     expect(fixture.download).not.toHaveBeenCalled();
-    expect(fixture.states.at(-1)).toMatchObject({
+    expect(fixture.states[fixture.states.length - 1]).toMatchObject({
       kind: 'storage_unavailable',
       downloadAvailable: true,
     });
@@ -141,7 +141,7 @@ describe('explicit same-tab trace handoff', () => {
     expect(fixture.target.location.assign).not.toHaveBeenCalled();
     expect(fixture.download).not.toHaveBeenCalled();
     expect(JSON.stringify(fixture.states)).not.toContain('private');
-    expect(fixture.states.at(-1)).toMatchObject({
+    expect(fixture.states[fixture.states.length - 1]).toMatchObject({
       kind: 'capture_failed',
       downloadAvailable: false,
     });
@@ -154,7 +154,7 @@ describe('explicit same-tab trace handoff', () => {
     const action = handoff(fixture.options);
     action.view();
     expect(fixture.target.sessionStorage.setItem).toHaveBeenCalledTimes(1);
-    expect(fixture.states.at(-1)).toMatchObject({
+    expect(fixture.states[fixture.states.length - 1]).toMatchObject({
       kind: 'navigation_unavailable',
       downloadAvailable: true,
     });
diff --git a/crates/trusted-server-js/lib/test/trace/projection.test.ts b/crates/trusted-server-js/lib/test/trace/projection.test.ts
index 9e388052f..0cde5b2e1 100644
--- a/crates/trusted-server-js/lib/test/trace/projection.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/projection.test.ts
@@ -72,7 +72,9 @@ describe('explicit GPT public projection', () => {
       expect(projected.ok).toBe(true);
       if (!projected.ok) throw new Error('should project actual pending cycles');
       expect(projected.value.slots[0]!.requests[0]!.trustedServerOpportunity).toBe(opportunity);
-      expect(Object.hasOwn(projected.value.slots[0]!.requests[0]!, 'size')).toBe(false);
+      expect(
+        Object.prototype.hasOwnProperty.call(projected.value.slots[0]!.requests[0]!, 'size')
+      ).toBe(false);
     }
   );
   it('classifies inspection failures without reading caller-controlled errors', () => {
diff --git a/crates/trusted-server-js/lib/test/trace/report.test.ts b/crates/trusted-server-js/lib/test/trace/report.test.ts
index ff0760aa2..6c1bf6cc6 100644
--- a/crates/trusted-server-js/lib/test/trace/report.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/report.test.ts
@@ -302,7 +302,7 @@ describe('combined trace capture', () => {
     expect(result.report.gpt_diagnostics.slots).toHaveLength(64);
     expect(
       result.report.gpt_diagnostics.slots.every(
-        (value) => value.requests.at(-1)?.requestNumber === 10
+        (value) => value.requests[value.requests.length - 1]?.requestNumber === 10
       )
     ).toBe(true);
     expect(result.report.truncation.omitted_request_cycles).toBeGreaterThan(0);
diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md
index d65567766..18af11050 100644
--- a/docs/guide/integrations/gpt-diagnostics.md
+++ b/docs/guide/integrations/gpt-diagnostics.md
@@ -143,6 +143,10 @@ GPT-only export as an equivalent fallback. The viewer keeps traced-document fact
 separate from its setup request. Server-auction and browser/GPT clocks remain
 separate; candidate selection or a filled slot does not establish an auction winner.
 
+Slot returned-bid counts include records returned by bidder and mediator calls,
+including mediator echoes. They are observations across stages, not unique bids;
+do not sum slot counts to infer a unique bid total.
+
 ### Storage, export and cleanup
 
 One versioned `sessionStorage` key holds at most 512 KiB of compact UTF-8 data.
@@ -172,6 +176,17 @@ Use a suitable same-origin deployment where the browser accepts the unchanged
 is 1800 seconds; ordinary requests do not refresh it. Do not weaken cookie attributes
 or infer HTTPS from an untrusted forwarding header to make a test pass.
 
+On deployed Fastly and Spin staging services, verify HTTPS Enable returns a
+successful response, a separate state request observes the session, publisher
+reload captures evidence, and End followed by another state request observes
+inactivity. Local plain-HTTP runtime tests do not establish this HTTPS behavior.
+
+Publisher activation and context use injected inline scripts without an attached
+CSP nonce. A publisher nonce/hash policy that does not authorize those scripts
+can prevent diagnostics activation and capture. Check the literal activation gate
+and actual capture under the deployed publisher CSP; do not weaken that policy
+or the trace viewer's separate CSP to make the check pass.
+
 A pre-existing session cookie cannot be retroactively assigned this lifetime by
 the server. End tracing and explicitly enable it again to adopt the bounded cookie.
 Expiry or disabling the deployment flag does not unload an already-running page;
@@ -236,9 +251,19 @@ behavior:
   behavior being supported. Record session-restoration checks separately from
   automated storage fixtures.
 
-Record the device/browser versions and results in the same implementation plan.
+The linked plan is the immutable implementation baseline. Record device/browser
+versions and new acceptance results in the current rollout evidence, referencing
+that baseline and the exact deployed revision.
 Until these checks are complete, physical mobile acceptance remains pending.
 
+### Maintaining versioned assets
+
+The committed trace asset manifest covers source, compiler dependency locks and
+build options. After changing those inputs, rebuild and review the JS/CSS and
+refresh the source digest. Unchanged bytes keep their existing asset URL and
+digest. Once an asset set is published, changed bytes require a new versioned
+URL; the current feature's v1 set remains unpublished until release acceptance.
+
 ## What the Console Shows
 
 The panel opens expanded after document startup and provides filters for All,
diff --git a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
index f2a461bb3..27f3799b1 100644
--- a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
+++ b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
@@ -366,7 +366,8 @@ release work.
 - [x] Resolve and verify the EdgeZero prerequisite before foundation integration.
 - [x] Execute and review foundation tasks F0–F8.
 - [x] Execute and review live-evidence tasks E1–E7.
-- [ ] Execute and review browser tasks V1–V7.
+- [x] Execute and review browser implementation V1–V6 and automated V7 journeys.
+- [ ] Complete V7 physical-device, actual session restoration and staging acceptance.
 - [ ] Record all acceptance evidence, full CI gates, and manual mobile results.
 - [ ] Enable only a controlled staging fixture; verify CDN/private responses and operator privacy acceptance.
 - [x] Keep enrichment and future schemas separately scheduled.
@@ -1014,6 +1015,67 @@ function downloadJson(json: string, filename: string): void {
 - [ ] Run the full shared verification gates/builds/docs and the complete browser runner, plus manual mobile checklist. Review final diff and acceptance matrix. Planned commit: `Verify the mobile trace journey and document release acceptance`.
 - [ ] Handoff v1 feature only after the first three phases meet all mandatory acceptance criteria; record pending environment/manual checks separately. Publishing/deployment is a subsequent explicit action.
 
+## Approved review corrections — 2026-10-06
+
+The user authorized this focused corrective pass after independent review of
+Claude's findings. Keep the same feature branches, spec and plan.
+
+- [x] Add a gated trace-only initial seam when ordinary ad-slot injection is
+      withheld. Execute the emitted script and verify evidence is recorded once
+      without invoking the scheduler, changing bids/slots or setting the ad-init
+      latch; preserve queue initialization, generation guards and fail-open behavior.
+- [x] Remove the redundant unconditional page-bids delivery observation. Retain
+      the successful branch's definitive delivery observation and verify actual SPA
+      success/fallback projections. The proposed partial-bid `Err` scenario is not
+      reachable in current planned production execution; do not fabricate a
+      legacy-test-only regression or claim a reproduced production truth defect.
+- [x] Keep View usable after a navigation does not depart, and retain connected
+      trace controls/live status across GPT data updates. Add retry/focus regressions.
+- [x] Replace the five newly introduced ES2022 test APIs with ES2020-compatible
+      access, without broadening the existing TypeScript target.
+- [x] Correct EdgeZero's per-observation preservation metadata conservatively;
+      document Fastly Content-Length folding; verify and pin the new upstream commit.
+- [x] Clarify mediator-inclusive record counts, asset-manifest maintenance,
+      publisher CSP compatibility, completed automation versus pending manual
+      acceptance, and deployed HTTPS Enable/End checks.
+- [x] Run target-matched checks and required full gates, independently review
+      the corrections, and update both existing draft PRs without adding Python.
+
+### Corrective-pass verification
+
+Three independent reviewers approved the final corrections with no remaining
+actionable findings: publisher/spec alignment, browser controls/types, and
+EdgeZero metadata/architecture. EdgeZero commit
+`75067d2c9a3cf865591665e88a736db4c8a13be0` replaces the previous reviewed pin
+in all six workspace dependencies and eight lockfile sources, without unrelated
+dependency changes. Its full workspace gates, adapter contracts and 54 raw
+ingress cases pass; all checks on EdgeZero PR #403 pass.
+
+Trusted Server's current-source format, all eight target-matched clippy gates,
+four adapter test suites, host CLI tests and 21 parity cases pass against the new
+pin. The host-only emitted-script regression executes real publisher output for
+four skip conditions and buffered/streaming finalizers, each with five callback
+modes; no scheduler or ad initialization runs and publisher-owned state remains
+unchanged. The overlay regressions prove retained focus/live status and retry.
+The normal JS and external Prebid builds, formatting, lint and all 1,736 Vitest
+tests in 69 files pass. The five new ES2022 test errors are removed; the broader
+JS TypeScript check retains unrelated baseline errors. Browser TypeScript
+passes using the workspace's existing Node type declarations. Core documentation,
+docs lint/format/build and Markdown formatting pass with existing warnings.
+
+Fresh Fastly, Axum, Cloudflare worker-build and Spin artifacts pass the complete
+repository browser runner: Next.js 49 passed with two expected skips; WordPress
+28 passed with 23 expected skips. All four separate runtime browser workflows
+pass, as do the three raw Cloudflare/Fastly/Spin boundary suites. These verify
+local runtime behavior and do not replace physical-device, actual restoration or
+deployed HTTPS/CDN/CSP acceptance. Both existing draft PRs receive the corrections;
+neither feature branch contains Python files. Frozen standalone v1 asset bytes
+remain unchanged by this corrective pass.
+
+Bundle splitting remains a separate load-order/performance design. Preserve
+the build-input freshness guard and ordinary reserved-route/cache policy.
+Physical mobile and deployed CDN/HTTPS acceptance remain release gates.
+
 ## Phase 4: Optional network enrichment
 
 ### Scheduling boundary
diff --git a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
index 7479ea919..86ed31bf5 100644
--- a/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
+++ b/docs/superpowers/specs/2026-09-01-mobile-ad-render-trace-endpoint-design.md
@@ -1078,6 +1078,8 @@ telemetry and OpenRTB objects:
   value makes the missing provider-to-slot no-bid relation explicit.
 - A slot's `returned_bid_count` counts actual returned bid records whose
   ordinary internal routing key matches that accepted slot. Accepted slots with
+  matching keys count records from both bidder and mediator calls, including
+  mediator echoes; the count is not unique across provider stages. Slots with
   duplicate routing keys retain distinct ordinals and opaque refs, but share
   that observed count. These counts are non-disjoint and must not be summed as
   unique bids. When existing winner/delivery data cannot distinguish those
@@ -1169,6 +1171,9 @@ The three transport call shapes are exact v1 contracts:
 Initial seam:
   scheduleInitialAdInit(bids, slots?, traceAuctionTransport?)
 
+Initial seam when ordinary ad-slot injection is withheld:
+  traceGpt.observeTransport(undefined, traceAuctionTransport, 'initial_navigation_ssat')
+
 SPA JSON:
   { slots, bids, trace_auction?: TraceAuctionTransportV1 }
 
@@ -1186,6 +1191,13 @@ accepts only the exact optional top-level member and preserves its existing
 `slots` and `bids` behavior. These trace members never become required for a
 successful advertising response.
 
+When the ordinary initial page would not inject ad slots, deliver available
+trace evidence through the existing gated core bridge only. Queue that one
+observation until core initialization if necessary and retain the initial
+navigation-generation guard. This trace-only seam must not call the ad
+scheduler or `adInit`, set its latch, or assign bids or slots. Skipped-page
+evidence cannot activate an otherwise withheld ad stack.
+
 For initial-navigation SSAT and SPA page-bids only, when the existing GPT
 recorder consumes the matching Trusted Server opportunity for a concrete request
 cycle, it emits this trace-owned sidecar:

From fb45d1c9b8f993d87f3c103856e8f079439b0a18 Mon Sep 17 00:00:00 2001
From: prk-Jr 
Date: Tue, 6 Oct 2026 13:25:30 +0530
Subject: [PATCH 133/230] Run skipped-page trace regression in CI

---
 .github/workflows/test.yml                       |  3 +++
 ...mobile-ad-render-trace-implementation-plan.md | 16 ++++++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index a7fc78d08..cc43fa781 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -59,6 +59,9 @@ jobs:
       - name: Run tests
         run: cargo test-fastly
 
+      - name: Run skipped-page trace emitted-script regression
+        run: cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu --lib trace_document_skipped_emitted_script_leaves_ad_state_untouched -- --ignored
+
       - name: Run template cache ESI local harness
         run: BID_DELAY=3 ./scripts/template-cache-local-test.sh esi
 
diff --git a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
index 27f3799b1..66c3b07ca 100644
--- a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
+++ b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
@@ -1072,6 +1072,22 @@ deployed HTTPS/CDN/CSP acceptance. Both existing draft PRs receive the correctio
 neither feature branch contains Python files. Frozen standalone v1 asset bytes
 remain unchanged by this corrective pass.
 
+Claude's re-review found no new runtime defects and identified one automated
+coverage gap: the ignored host-only emitted-script regression was not selected
+by CI. The existing Node-equipped Rust job now runs that exact core library test
+with an explicit Linux host target and `--ignored`; other ignored tests remain
+excluded. The equivalent command on the macOS host selects one test and passes.
+An independent reviewer approved the step with no findings. Workflow validation
+passes; the nine pre-existing ShellCheck quoting notices are unchanged.
+
+All four integration jobs on `b2930d7b9`, including framework browser tests and
+four-runtime trace acceptance, have now passed remotely. The unrelated generated
+Python CodeQL job and its aggregate check still fail because this feature branch
+contains no Python. Cloudflare's new-pin worker-build artifact and actual local
+browser/raw-boundary suites use Trusted Server's resolved worker 0.8.5 and
+wasm-bindgen 0.2.126; that consumer runtime check is complete. Deployed-platform
+and physical-device acceptance remain separate release gates.
+
 Bundle splitting remains a separate load-order/performance design. Preserve
 the build-input freshness guard and ordinary reserved-route/cache policy.
 Physical mobile and deployed CDN/HTTPS acceptance remain release gates.

From f6c3bd5ca6c890c1df02eee25601208b8cd45674 Mon Sep 17 00:00:00 2001
From: prk-Jr 
Date: Tue, 6 Oct 2026 13:39:55 +0530
Subject: [PATCH 134/230] Record CodeQL test-fixture triage

---
 ...ile-ad-render-trace-implementation-plan.md | 31 +++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
index 66c3b07ca..6bd0f9c59 100644
--- a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
+++ b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
@@ -1082,12 +1082,39 @@ passes; the nine pre-existing ShellCheck quoting notices are unchanged.
 
 All four integration jobs on `b2930d7b9`, including framework browser tests and
 four-runtime trace acceptance, have now passed remotely. The unrelated generated
-Python CodeQL job and its aggregate check still fail because this feature branch
-contains no Python. Cloudflare's new-pin worker-build artifact and actual local
+Python CodeQL job fails because this feature branch contains no Python. The
+separate CodeQL security gate reported test-fixture alerts; their resolution is
+recorded below. Cloudflare's new-pin worker-build artifact and actual local
 browser/raw-boundary suites use Trusted Server's resolved worker 0.8.5 and
 wasm-bindgen 0.2.126; that consumer runtime check is complete. Deployed-platform
 and physical-device acceptance remain separate release gates.
 
+### CodeQL alert triage
+
+On 2026-10-06, the user authorized fixing CodeQL findings or dismissing verified
+false positives. Two independent read-only reviewers confirmed alerts #196–203
+are confined to fictional authentication fixtures in `#[cfg(test)]` modules or
+the parity integration-test target. Alerts #202–203 flag test-only passwords;
+alerts #196–201 flag `expect()` calls on `Handler` deserialization. That
+deserialization does not invoke the `Settings`/Tinybird secret validation named
+in the alleged logging flow. No production secret reaches these fixtures.
+
+All eight alerts were dismissed individually with GitHub's `used in tests`
+reason and per-alert evidence. A fresh API check confirms no open alerts on
+`refs/pull/1107/merge` and a successful CodeQL security gate. The earlier
+explanation attributing the aggregate gate solely to Python extraction was
+incomplete; the extraction error and these alerts are independent.
+
+The generated `dynamic/github-code-quality/codeql` workflow still attempts Python
+analysis on `refs/pull/1107/head` and exits 32 because that head contains no
+Python. The default branch contains an unrelated Python browser helper, which
+accounts for repository-wide language detection. This is an analysis failure,
+not a dismissible vulnerability alert. Reading Code Quality setup returned HTTP
+403 with the current maintainer account; configuration changes require a
+repository admin. Any Python language-selection change is repository-wide and
+must account for that default-branch helper. No scanner rules, source files or
+production behavior were weakened to dismiss the test-only alerts.
+
 Bundle splitting remains a separate load-order/performance design. Preserve
 the build-input freshness guard and ordinary reserved-route/cache policy.
 Physical mobile and deployed CDN/HTTPS acceptance remain release gates.

From 6e9e7c4851dde29cda845d2fe30e786bd427c939 Mon Sep 17 00:00:00 2001
From: prk-Jr 
Date: Tue, 6 Oct 2026 14:49:00 +0530
Subject: [PATCH 135/230] Bound trace capture cost and complete review
 corrections

---
 .github/workflows/test.yml                    |   6 +-
 CHANGELOG.md                                  |   4 +
 .../tests/nextjs/mobile-trace-live.spec.ts    |   7 ++
 .../browser/tests/shared/mobile-trace.spec.ts |   2 +
 .../lib/src/trace/handoff.ts                  |   2 +-
 .../lib/src/trace/report-view.ts              |  12 ++
 .../trusted-server-js/lib/src/trace/report.ts |  61 +++++++----
 .../lib/test/trace/handoff.test.ts            |  10 +-
 .../lib/test/trace/report.test.ts             | 103 +++++++++++++++++-
 .../lib/test/trace/viewer.test.ts             |  47 +++++++-
 .../lib/trace-assets-manifest.json            |   4 +-
 .../trusted-server-js/lib/trace-assets/v1.js  |   2 +-
 docs/guide/integrations/gpt-diagnostics.md    |  16 ++-
 ...ile-ad-render-trace-implementation-plan.md |  68 ++++++++++++
 14 files changed, 312 insertions(+), 32 deletions(-)

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index e390d7424..bc64c6ca2 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -59,8 +59,10 @@ jobs:
       - name: Run tests
         run: cargo test-fastly
 
-      - name: Run skipped-page trace emitted-script regression
-        run: cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu --lib trace_document_skipped_emitted_script_leaves_ad_state_untouched -- --ignored
+      - name: Run trace emitted-script regressions
+        run: |
+          cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu --lib trace_document_skipped_emitted_script_leaves_ad_state_untouched -- --ignored
+          cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu --lib trace_document_emitted_script_deep_freezes_owned_context_in_node -- --ignored
 
       - name: Run template cache ESI local harness
         run: BID_DELAY=3 ./scripts/template-cache-local-test.sh esi
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 101087b8d..699be4e43 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- Explicit TS Console activation with `?ts_console=1` now sets a 30-minute diagnostics cookie instead of a browser-session cookie, independently of the mobile trace flag. Ordinary requests do not refresh it; end and explicitly enable diagnostics again to adopt the lifetime for an existing session.
+- Publisher documents with diagnostics active now use private, no-store responses and strip conditional and range request headers before fetching the origin. An unexpected origin `304` becomes a private `502` because it cannot supply an instrumented document, including when mobile tracing is disabled.
 - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias.
 - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave.
 - **Breaking:** Auction providers and bidder routes now use the configuration-first `[auction.providers.]` and `[auction.bidders.]` maps. The removed `[auction].providers = [...]` list and removed server fields under `[integrations.prebid]` and `[integrations.aps]` are rejected even when those integrations are disabled, and `ts config push` rejects the old shape before publication. Move PBS `server_url` to provider `endpoint`, server timeout to provider `timeout_ms`, request controls and bidder-parameter overrides to the `prebid-server` `profile_config`, notification suppression to `notifications`, and each former server bidder to an `[auction.bidders.]` route. Move APS endpoint, timeout, account, inventory, debug, and creative controls to an `aps` provider and its `profile_config`. Browser Prebid settings remain under `[integrations.prebid]`; values such as timeout and debug that previously affected both browser and server behavior must now be configured for each owner. Provider endpoints must be absolute HTTPS URLs. Only bidder codes present in `[auction.bidders]` are folded into Trusted Server requests; unlisted publisher bids remain native browser demand. Provider response names now use the configured provider ID, such as `pbs-main`, instead of the legacy literal `prebid`; audit consumers that match `AuctionResponse.provider`. This schema has no mixed-version-safe deployment order: old binaries reject the maps and new binaries reject the retired fields, so activate the new binary and config blob together. Rollbacks must restore an old-schema blob together with the old binary.
@@ -27,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Security
 
+- Reserve the application-visible `/_ts/trace*` prefix locally on every adapter, including when mobile tracing is disabled. Existing authentication runs first; disabled or unknown trace routes then return a local `404` instead of forwarding to the publisher.
 - `/first-party/sign` now rejects valid targets outside `proxy.allowed_domains` before minting a proxy token. The creative runtime keeps image and iframe assignments blocked after this `403` policy response instead of loading the rejected URL directly; fetch-time checks still cover the initial target and every redirect.
 - Reserved the complete admin namespace at the publisher-fallback boundary. Percent-encoded separators (`/_ts/admin%2Fec`, `%2f`, and double-encoded forms) matched the `^/_ts/admin` Basic-auth handler but escaped the literal-slash namespace check, so an authenticated request fell through to publisher fallback and forwarded its `Authorization` header and body to the publisher origin. The reservation now spans the whole `/_ts/admin` prefix plus the retired `/admin/keys` aliases — including trailing, descendant, and encoded-separator forms — evaluated on the raw path and on each of its bounded percent-decodings, so multi-encoded separators such as `/admin%252Fkeys/rotate` cannot survive to fallback for a proxy or origin to decode again, and applies to every adapter.
 - Validate synthetic ID format on inbound values from the `x-synthetic-id` header and `synthetic_id` cookie; values that do not match the expected format (`64-hex-hmac.6-alphanumeric-suffix`) are discarded and a fresh ID is generated rather than forwarded to response headers, cookies, or third-party APIs
@@ -40,6 +43,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Add opt-in mobile ad-render tracing through `[integrations.gpt_diagnostics].trace_page_enabled` (default `false`, requiring GPT diagnostics). The same-tab `/_ts/trace` viewer presents bounded, redacted browser-local request, auction and GPT observations with copy, download, share and independent local/server cleanup controls. See the GPT diagnostics guide for deployment and rollout checks.
 - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.
diff --git a/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts b/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts
index 7e5ac4223..c6ef1bb6a 100644
--- a/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts
+++ b/crates/trusted-server-integration-tests/browser/tests/nextjs/mobile-trace-live.spec.ts
@@ -104,6 +104,13 @@ test('captures a real zero-bid SSAT auction and carries its GPT cycle through th
   expect(evidence.value.slotCorrelations[0].slot_ref).toBe(
     evidence.value.serverAuctions[0].slots[0].slot_ref
   )
+  await page.route('https://other.example/**', (route) => route.abort())
+  await page.evaluate(() => {
+    const base = document.createElement('base')
+    base.href = 'https://other.example/'
+    document.head.prepend(base)
+  })
+  expect(await page.evaluate(() => document.baseURI)).toBe('https://other.example/')
   await clickTraceHandoff(page)
   await page.waitForURL(traceRuntimeUrl('/_ts/trace'))
   await expect(
diff --git a/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts
index fa44e9fad..61d11cae6 100644
--- a/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts
+++ b/crates/trusted-server-integration-tests/browser/tests/shared/mobile-trace.spec.ts
@@ -495,6 +495,7 @@ test.describe('browser-carried trace report viewer', () => {
     await expect(page.locator('#trace-cleanup-local-status')).toHaveText(
       'Local report deleted from this tab.'
     )
+    await expect(page.locator('#trace-cleanup-local-status')).toBeFocused()
   })
 
   test('deletes the report offline and retries end with a separate server observation after reconnecting', async ({
@@ -549,6 +550,7 @@ test.describe('browser-carried trace report viewer', () => {
     await expect(page.locator('#trace-cleanup-server-status')).toContainText(
       'Tracing is off'
     )
+    await expect(page.locator('#trace-cleanup-server-status')).toBeFocused()
     expect(requests).toEqual(['POST /_ts/trace/end', 'GET /_ts/trace/state'])
     expect(
       (await page.context().cookies()).find((item) => item.name === SESSION)
diff --git a/crates/trusted-server-js/lib/src/trace/handoff.ts b/crates/trusted-server-js/lib/src/trace/handoff.ts
index 15fe60f65..fb516d4a5 100644
--- a/crates/trusted-server-js/lib/src/trace/handoff.ts
+++ b/crates/trusted-server-js/lib/src/trace/handoff.ts
@@ -108,7 +108,7 @@ export function createTraceHandoff(options: TraceHandoffOptions): TraceHandoff |
           return;
         }
         try {
-          options.target.location.assign('/_ts/trace');
+          options.target.location.assign(`${origin}/_ts/trace`);
           notify('navigating');
         } catch {
           notify('navigation_unavailable');
diff --git a/crates/trusted-server-js/lib/src/trace/report-view.ts b/crates/trusted-server-js/lib/src/trace/report-view.ts
index 5ef6fe100..4778229ff 100644
--- a/crates/trusted-server-js/lib/src/trace/report-view.ts
+++ b/crates/trusted-server-js/lib/src/trace/report-view.ts
@@ -539,6 +539,7 @@ export function mountTraceViewer(
   localStatus.id = 'trace-cleanup-local-status';
   localStatus.setAttribute('role', 'status');
   localStatus.setAttribute('aria-live', 'polite');
+  localStatus.tabIndex = -1;
   cleanup.append(localStatus);
   const serverStatus = element(
     root,
@@ -548,6 +549,7 @@ export function mountTraceViewer(
   serverStatus.id = 'trace-cleanup-server-status';
   serverStatus.setAttribute('role', 'status');
   serverStatus.setAttribute('aria-live', 'polite');
+  serverStatus.tabIndex = -1;
   cleanup.append(serverStatus);
   const removeLocal = (): void => {
     let result: ReturnType;
@@ -558,10 +560,13 @@ export function mountTraceViewer(
     }
     if (destroyed) return;
     if (result.status === 'deleted') {
+      const losingFocus =
+        root.activeElement === deleteButton || article.contains(root.activeElement);
       stored = undefined;
       article.remove();
       deleteButton.hidden = true;
       localStatus.textContent = 'Local report deleted from this tab.';
+      if (losingFocus) localStatus.focus();
     } else
       localStatus.textContent =
         'Local report deletion failed. The report remains displayed; retry deletion.';
@@ -569,6 +574,7 @@ export function mountTraceViewer(
   let ending = false;
   const end = async (): Promise => {
     if (destroyed || ending) return;
+    const retryHadFocus = root.activeElement === retryButton;
     ending = true;
     clearButton.disabled = retryButton.disabled = true;
     serverStatus.textContent = 'Requesting tracing end and checking the next request…';
@@ -588,6 +594,12 @@ export function mountTraceViewer(
           : 'End tracing unconfirmed. Tracing may remain active. Retry end tracing.';
     retryButton.hidden = result.confirmed;
     clearButton.disabled = retryButton.disabled = false;
+    if (
+      result.confirmed &&
+      retryHadFocus &&
+      (root.activeElement === retryButton || root.activeElement === root.body)
+    )
+      serverStatus.focus();
     ending = false;
   };
   const clearButton = control(controls, 'Clear report and end tracing', () => {
diff --git a/crates/trusted-server-js/lib/src/trace/report.ts b/crates/trusted-server-js/lib/src/trace/report.ts
index b83127714..fda0c0c07 100644
--- a/crates/trusted-server-js/lib/src/trace/report.ts
+++ b/crates/trusted-server-js/lib/src/trace/report.ts
@@ -206,18 +206,38 @@ export function buildTraceReport(
     };
     recompute();
     const wrapper = { stored_at_ms: captureClock, report: draft };
-    // Every draft member is already independently owned data. Serialize only that
-    // draft while measuring; final ingestion checks the complete schema and depth.
+    // Measure the owned draft once, then account for removed JSON payloads,
+    // commas, counter digits and coverage changes without serializing it again.
+    // A final independent measurement enforces the budget before ingestion.
     const encoder = new TextEncoder();
-    const fits = (): boolean => encoder.encode(JSON.stringify(wrapper)).length <= maximumBytes;
+    const bytes = (value: unknown): number => encoder.encode(JSON.stringify(value)).length;
+    let reportBytes = bytes(wrapper);
+    const fits = (): boolean => reportBytes <= maximumBytes;
+    const omit = (counter: keyof TraceReportV1['truncation'], count: number): void => {
+      const current = draft.truncation[counter];
+      const next = add(current, count);
+      reportBytes += String(next).length - String(current).length;
+      draft.truncation[counter] = next;
+    };
+    const retain = (items: Item[], keep: (item: Item) => boolean): Item[] => {
+      const retained: Item[] = [];
+      for (const item of items) {
+        if (keep(item)) retained.push(item);
+        else reportBytes -= bytes(item);
+      }
+      reportBytes -= Math.max(0, items.length - 1) - Math.max(0, retained.length - 1);
+      return retained;
+    };
+    const recomputeMeasured = (): void => {
+      const previous = bytes(draft.auction_coverage);
+      recompute();
+      reportBytes += bytes(draft.auction_coverage) - previous;
+    };
     const pruneSidecars = (
       predicate: (sidecar: Mutable['slot_correlations'][number]) => boolean
     ): void => {
-      const retained = draft.slot_correlations.filter((sidecar) => !predicate(sidecar));
-      draft.truncation.omitted_slot_correlations = add(
-        draft.truncation.omitted_slot_correlations,
-        draft.slot_correlations.length - retained.length
-      );
+      const retained = retain(draft.slot_correlations, (sidecar) => !predicate(sidecar));
+      omit('omitted_slot_correlations', draft.slot_correlations.length - retained.length);
       draft.slot_correlations = retained;
     };
     const floors = new Set(
@@ -239,14 +259,14 @@ export function buildTraceReport(
     });
     for (const { slot, cycle } of cycles) {
       if (fits()) break;
-      slot.requests = slot.requests.filter((retained) => retained !== cycle);
-      draft.truncation.omitted_request_cycles = add(draft.truncation.omitted_request_cycles, 1);
+      slot.requests = retain(slot.requests, (retained) => retained !== cycle);
+      omit('omitted_request_cycles', 1);
       pruneSidecars(
         (sidecar) =>
           sidecar.runtime_slot_number === slot.runtimeSlotNumber &&
           sidecar.request_number === cycle.requestNumber
       );
-      recompute();
+      recomputeMeasured();
     }
     const removeIssues = (
       source: Issue[] | undefined,
@@ -259,8 +279,9 @@ export function buildTraceReport(
       for (const { issue } of ordered) {
         if (fits()) break;
         const index = source.indexOf(issue);
+        reportBytes -= bytes(issue) + (source.length > 1 ? 1 : 0);
         source.splice(index, 1);
-        draft.truncation[counter] = add(draft.truncation[counter], 1);
+        omit(counter, 1);
       }
     };
     removeIssues(draft.gpt_diagnostics.callbackIssues, 'omitted_callback_issues');
@@ -268,18 +289,15 @@ export function buildTraceReport(
     const remainingCycles = () => draft.gpt_diagnostics.slots.flatMap((slot) => slot.requests);
     const removeAuction = (record: Mutable['server_auctions'][number]): void => {
       const id = record.diagnostic_auction_id;
-      draft.server_auctions = draft.server_auctions.filter((retained) => retained !== record);
-      draft.truncation.omitted_server_auctions = add(draft.truncation.omitted_server_auctions, 1);
+      draft.server_auctions = retain(draft.server_auctions, (retained) => retained !== record);
+      omit('omitted_server_auctions', 1);
       for (const slot of draft.gpt_diagnostics.slots) {
-        const retained = slot.requests.filter((cycle) => cycle.trustedServerAuctionId !== id);
-        draft.truncation.omitted_request_cycles = add(
-          draft.truncation.omitted_request_cycles,
-          slot.requests.length - retained.length
-        );
+        const retained = retain(slot.requests, (cycle) => cycle.trustedServerAuctionId !== id);
+        omit('omitted_request_cycles', slot.requests.length - retained.length);
         slot.requests = retained;
       }
       pruneSidecars((sidecar) => sidecar.diagnostic_auction_id === id);
-      recompute();
+      recomputeMeasured();
     };
     for (const record of [...draft.server_auctions]) {
       if (fits()) break;
@@ -297,7 +315,8 @@ export function buildTraceReport(
       )
         removeAuction(record);
     }
-    if (!fits()) return { ok: false, reason: 'snapshot_too_large' };
+    if (!fits() || bytes(wrapper) > maximumBytes)
+      return { ok: false, reason: 'snapshot_too_large' };
     const result = parseTraceStoredReport(wrapper, origin, captureClock);
     return result ? { ok: true, value: result } : { ok: false, reason: 'invalid_snapshot' };
   } catch {
diff --git a/crates/trusted-server-js/lib/test/trace/handoff.test.ts b/crates/trusted-server-js/lib/test/trace/handoff.test.ts
index 235620e0a..c51279e06 100644
--- a/crates/trusted-server-js/lib/test/trace/handoff.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/handoff.test.ts
@@ -70,12 +70,20 @@ describe('explicit same-tab trace handoff', () => {
     expect(fixture.order).toEqual([]);
     expect(fixture.target.sessionStorage.getItem).not.toHaveBeenCalled();
     action.view();
-    expect(fixture.order).toEqual(['snapshot', 'store', 'navigate:/_ts/trace']);
+    expect(fixture.order).toEqual(['snapshot', 'store', `navigate:${TRACE_ORIGIN}/_ts/trace`]);
     const wrapper = JSON.parse(fixture.target.sessionStorage.setItem.mock.calls[0][1]);
     expect(wrapper.report.request_context).toEqual(fixture.target.__tsjs_trace_request_context);
     expect(wrapper.report.gpt_diagnostics.page.pathname).toBe('/[redacted]');
     expect(wrapper.stored_at_ms).toBe(TRACE_NOW);
   });
+  it('keeps the handoff on the publisher origin when the document has an external base', () => {
+    const fixture = setup();
+    fixture.target.location.assign.mockImplementation((url) => {
+      fixture.order.push(`navigate:${new URL(url, 'https://other.example/').href}`);
+    });
+    handoff(fixture.options).view();
+    expect(fixture.order).toEqual(['snapshot', 'store', `navigate:${TRACE_ORIGIN}/_ts/trace`]);
+  });
   it('preserves a valid immutable combined report for explicit direct download after storage failure', () => {
     const fixture = setup();
     fixture.target.sessionStorage.setItem.mockImplementation(() => {
diff --git a/crates/trusted-server-js/lib/test/trace/report.test.ts b/crates/trusted-server-js/lib/test/trace/report.test.ts
index 6c1bf6cc6..ab816d684 100644
--- a/crates/trusted-server-js/lib/test/trace/report.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/report.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it } from 'vitest';
+import { describe, expect, it, vi } from 'vitest';
 
 import { buildTraceReport } from '../../src/trace/report';
 
@@ -177,6 +177,90 @@ describe('combined trace capture', () => {
       reason: 'invalid_snapshot',
     });
   });
+  it('accepts an exact protected-floor byte budget and rejects one byte less', () => {
+    const source = input();
+    Object.assign(source.requestContext.network, { region: 'é"\\' });
+    source.gptSource.callbackIssues = [];
+    source.gptSource.attributionIssues = [];
+    const full = success(buildTraceReport(source));
+    const bytes = new TextEncoder().encode(JSON.stringify(full)).length;
+    expect(success(buildTraceReport(source, bytes))).toEqual(full);
+    expect(buildTraceReport(source, bytes - 1)).toEqual({
+      ok: false,
+      reason: 'snapshot_too_large',
+    });
+  });
+  it('accounts exactly for issue commas and the omission counter growing to two digits', () => {
+    const source = input();
+    const issue = source.gptSource.callbackIssues[0];
+    source.gptSource.callbackIssues = Array.from({ length: 10 }, (_, i) => ({
+      ...issue,
+      timestampMs: i + 1,
+    }));
+    source.gptSource.attributionIssues = [];
+    const full = success(buildTraceReport(source));
+    for (const removed of [9, 10]) {
+      const expected = {
+        ...full,
+        report: {
+          ...full.report,
+          gpt_diagnostics: {
+            ...full.report.gpt_diagnostics,
+            callbackIssues: full.report.gpt_diagnostics.callbackIssues.slice(removed),
+          },
+          truncation: { ...full.report.truncation, omitted_callback_issues: removed },
+        },
+      };
+      const budget = new TextEncoder().encode(JSON.stringify(expected)).length;
+      expect(success(buildTraceReport(source, budget))).toEqual(expected);
+    }
+  });
+  for (const seed of [0, 9, 99, 999, 9999]) {
+    it(`accounts exactly for auction and sidecar removal with omission seed ${seed}`, () => {
+      const source = input();
+      Object.assign(source.requestContext.network, { region: 'é"\\' });
+      source.gptSource.callbackIssues = [];
+      source.gptSource.attributionIssues = [];
+      Object.assign(source.collector, {
+        serverAuctions: [auction(1), auction(2)],
+        slotCorrelations: [sidecar(1), sidecar(2)],
+        omittedServerAuctions: seed,
+        omittedSlotCorrelations: seed,
+      });
+      const full = success(buildTraceReport(source));
+      const expected = {
+        ...full,
+        report: {
+          ...full.report,
+          server_auctions: full.report.server_auctions.slice(1),
+          slot_correlations: full.report.slot_correlations.slice(1),
+          truncation: {
+            ...full.report.truncation,
+            omitted_server_auctions: seed + 1,
+            omitted_slot_correlations: seed + 1,
+          },
+          auction_coverage: {
+            capture_status: 'partial',
+            issues: ['record_evicted', 'correlation_unavailable'],
+          },
+        },
+      };
+      const budget = new TextEncoder().encode(JSON.stringify(expected)).length;
+      expect(success(buildTraceReport(source, budget))).toEqual(expected);
+    });
+  }
+  it('rejects checked omission overflow during byte-budget auction removal', () => {
+    const source = input();
+    source.gptSource.callbackIssues = [];
+    source.gptSource.attributionIssues = [];
+    Object.assign(source.collector, { serverAuctions: [auction()], omittedServerAuctions: 65535 });
+    const full = success(buildTraceReport(source));
+    const budget = new TextEncoder().encode(JSON.stringify(full)).length - 1;
+    expect(buildTraceReport(source, budget)).toEqual({
+      ok: false,
+      reason: 'omission_counter_overflow',
+    });
+  });
   it('removes an uncorrelated auction and recomputes coverage when no evidence remains', () => {
     const source = input();
     source.gptSource.callbackIssues = [];
@@ -297,7 +381,22 @@ describe('combined trace capture', () => {
         runtime_slot_number: (i % 64) + 1,
       })),
     });
-    const result = success(buildTraceReport(source));
+    const serialization = vi.spyOn(JSON, 'stringify');
+    let result: ReturnType;
+    let completeMeasurements: number;
+    try {
+      result = success(buildTraceReport(source));
+      completeMeasurements = serialization.mock.calls.filter(
+        ([value]) =>
+          value !== null &&
+          typeof value === 'object' &&
+          Object.prototype.hasOwnProperty.call(value, 'stored_at_ms') &&
+          Object.prototype.hasOwnProperty.call(value, 'report')
+      ).length;
+    } finally {
+      serialization.mockRestore();
+    }
+    expect(completeMeasurements).toBeLessThanOrEqual(2);
     expect(new TextEncoder().encode(JSON.stringify(result)).length).toBeLessThanOrEqual(512 * 1024);
     expect(result.report.gpt_diagnostics.slots).toHaveLength(64);
     expect(
diff --git a/crates/trusted-server-js/lib/test/trace/viewer.test.ts b/crates/trusted-server-js/lib/test/trace/viewer.test.ts
index 6c2223110..add543c25 100644
--- a/crates/trusted-server-js/lib/test/trace/viewer.test.ts
+++ b/crates/trusted-server-js/lib/test/trace/viewer.test.ts
@@ -235,6 +235,23 @@ describe('consolidated trace report viewer', () => {
     expect(document.getElementById('trace-report')).toBeNull();
     expect(document.body.textContent).toContain('Local report deleted');
   });
+  it('moves focus to the local status when deletion hides its focused control', () => {
+    const fixture = setup();
+    mountTraceViewer(document, fixture.options);
+    const remove = button('Delete local report');
+    remove.focus();
+    remove.click();
+    expect(remove.hidden).toBe(true);
+    expect(document.activeElement).toBe(document.getElementById('trace-cleanup-local-status'));
+  });
+  it('keeps another cleanup control focused during local deletion', () => {
+    const fixture = setup();
+    mountTraceViewer(document, fixture.options);
+    const clear = button('Clear report and end tracing');
+    clear.focus();
+    button('Delete local report').click();
+    expect(document.activeElement).toBe(clear);
+  });
   for (const local of ['deleted', 'failed'] as const)
     for (const mutation of ['requested', 'failed'] as const)
       for (const observation of ['inactive', 'active', 'failed'] as const) {
@@ -314,7 +331,9 @@ describe('consolidated trace report viewer', () => {
     expect(document.getElementById('trace-report')).toBeNull();
     request.mockResolvedValueOnce(new Response('{}'));
     request.mockResolvedValueOnce(new Response('{"observed_active":false}'));
-    button('Retry end tracing').click();
+    const retry = button('Retry end tracing');
+    retry.focus();
+    retry.click();
     await vi.waitFor(() =>
       expect(document.getElementById('trace-cleanup-server-status')?.textContent).toContain(
         'Tracing is off'
@@ -323,10 +342,36 @@ describe('consolidated trace report viewer', () => {
     expect(fixture.options.confirm).toHaveBeenCalledTimes(1);
     expect(fixture.storage.removeItem).toHaveBeenCalledTimes(2);
     expect(request).toHaveBeenCalledTimes(4);
+    expect(document.activeElement).toBe(document.getElementById('trace-cleanup-server-status'));
     expect(document.getElementById('trace-cleanup-local-status')?.textContent).toContain(
       'Local report deleted'
     );
   });
+  it('does not move focus back from another control when a pending retry succeeds', async () => {
+    const fixture = setup();
+    request.mockResolvedValueOnce(new Response('{}', { status: 500 }));
+    request.mockResolvedValueOnce(new Response('{"observed_active":true}'));
+    mountTraceViewer(document, fixture.options);
+    button('Clear report and end tracing').click();
+    await vi.waitFor(() => expect(button('Retry end tracing').hidden).toBe(false));
+    request.mockResolvedValueOnce(new Response('{}'));
+    let resolveState: ((response: Response) => void) | undefined;
+    request.mockImplementationOnce(
+      () =>
+        new Promise((resolve) => {
+          resolveState = resolve;
+        })
+    );
+    const retry = button('Retry end tracing');
+    retry.focus();
+    retry.click();
+    await vi.waitFor(() => expect(request).toHaveBeenCalledTimes(4));
+    const retained = button('Return to previous page');
+    retained.focus();
+    resolveState?.(new Response('{"observed_active":false}'));
+    await vi.waitFor(() => expect(retry.hidden).toBe(true));
+    expect(document.activeElement).toBe(retained);
+  });
   it('still deletes the report and attempts state verification when both network steps are offline', async () => {
     const fixture = setup();
     request.mockRejectedValueOnce(new Error('private-post-error'));
diff --git a/crates/trusted-server-js/lib/trace-assets-manifest.json b/crates/trusted-server-js/lib/trace-assets-manifest.json
index 8a660a3ad..77c293a4d 100644
--- a/crates/trusted-server-js/lib/trace-assets-manifest.json
+++ b/crates/trusted-server-js/lib/trace-assets-manifest.json
@@ -1,11 +1,11 @@
 {
   "schema_version": 1,
-  "source_sha256": "8c7b4768299d57175576b663cbb57f54d5bce521af1794676ca8e850c623fb7f",
+  "source_sha256": "9340982becac8571f2b414950388dc5d1f93fceca8ee5ef61f149c7ac6cd2748",
   "assets": [
     {
       "path": "/_ts/trace/assets/v1.js",
       "file": "v1.js",
-      "sha256": "9ddbde76323a2af09a2c20019f550f9bb917eb0c5ac76e0471852ade6a526bc5"
+      "sha256": "6af6294d8e3ae50817816debd7e5290fc1bb5b1ba31b72f310b6a2d588fc44e9"
     },
     {
       "path": "/_ts/trace/assets/v1.css",
diff --git a/crates/trusted-server-js/lib/trace-assets/v1.js b/crates/trusted-server-js/lib/trace-assets/v1.js
index 599cfdad1..7adeec167 100644
--- a/crates/trusted-server-js/lib/trace-assets/v1.js
+++ b/crates/trusted-server-js/lib/trace-assets/v1.js
@@ -1 +1 @@
-(function(){"use strict";function C(e,t){return Object.prototype.hasOwnProperty.call(e,t)}function m(e){if(e===null||typeof e!="object"||Array.isArray(e))return!1;const t=Object.getPrototypeOf(e);return t!==Object.prototype&&t!==null?!1:Reflect.ownKeys(e).every(n=>{if(typeof n!="string")return!1;const r=Object.getOwnPropertyDescriptor(e,n);return r?.enumerable===!0&&C(r,"value")})}function g(e,t,n=[]){return t.every(r=>C(e,r))&&Object.keys(e).every(r=>t.includes(r)||n.includes(r))}function $(e,t=128){if(typeof e!="string")return!1;for(const n of e){const r=n.codePointAt(0);if(r<=31||r>=127&&r<=159||r>=55296&&r<=57343||r===1564||r===8206||r===8207||r>=8234&&r<=8238||r>=8294&&r<=8297)return!1}return new TextEncoder().encode(e).length<=t}function H(e){if(typeof e!="string")return!1;const t=/^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,9})?Z$/.exec(e);if(!t)return!1;const[,n,r,s,i,c,l]=t,o=Number(n),S=Number(r),p=Number(s),d=[31,o%4===0&&(o%100!==0||o%400===0)?29:28,31,30,31,30,31,31,30,31,30,31];return S>=1&&S<=12&&p>=1&&p<=d[S-1]&&Number(i)<=23&&Number(c)<=59&&Number(l)<=59}function T(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isSafeInteger(e)&&e>=0&&e<=t}function Re(e){if(!$(e))return!1;const t=/^((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.0\/24$/.exec(e);if(t)return t.slice(1).every(s=>Number(s)<=255);if(!e.endsWith("::/48"))return!1;const n=e.slice(0,-3),r=n.slice(0,-2);if(r!==""&&!/^[\da-f]{1,4}(?::[\da-f]{1,4}){0,2}$/.test(r))return!1;try{return new URL(`http://[${n}]/`).hostname===`[${n}]`}catch{return!1}}function F(e,t){if(!m(e)||e.source!=="request")return!1;if(e.state==="absent")return g(e,["source","state"]);if(!g(e,["source","state","detail"])||typeof e.detail!="string")return!1;switch(e.state){case"present_valid":return e.detail===t;case"present_invalid":return["malformed","oversized","unsupported_value"].includes(e.detail);case"duplicate":return e.detail==="multiple_values";case"unavailable":return["header_too_large","header_not_utf8","runtime_header_ambiguous"].includes(e.detail);default:return!1}}function ne(e){try{return we(e)}catch{return!1}}function we(e){if(!m(e)||!g(e,["schema_version","captured_at","network","cookies"])||e.schema_version!==1||!H(e.captured_at)||!m(e.network)||!m(e.cookies))return!1;const t=e.network,n={country:2,region:32,http_version:32,tls_protocol:32,tls_cipher:32,edge_hostname:128,edge_region:128,edge_pop:32};if(!g(t,[],["masked_client_ip","asn",...Object.keys(n)])||C(t,"masked_client_ip")&&!Re(t.masked_client_ip)||C(t,"asn")&&!T(t.asn,4294967295))return!1;for(const[s,i]of Object.entries(n))if(C(t,s)&&!$(t[s],i))return!1;if(typeof t.country=="string"&&!/^[\x20-\x7e]{0,2}$/.test(t.country))return!1;const r=e.cookies;return g(r,["ts_ec","ts_eids","ts_tester","diagnostics_session"])&&F(r.ts_ec,"valid_ec_format")&&F(r.ts_eids,"valid_eids_format")&&F(r.ts_tester,"valid_tester_value")&&F(r.diagnostics_session,"valid_diagnostics_value")}function x(e,t){if(!Array.isArray(e)||Object.getPrototypeOf(e)!==Array.prototype)return;const n=Object.getOwnPropertyDescriptor(e,"length")?.value;if(!T(n,t))return;const r=Reflect.ownKeys(e);if(r.length!==n+1||!r.every(i=>{if(i==="length")return!0;if(typeof i!="string"||!/^(?:0|[1-9]\d*)$/.test(i)||Number(i)>=n)return!1;const c=Object.getOwnPropertyDescriptor(e,i);return c?.enumerable===!0&&C(c,"value")}))return;const s=[];for(let i=0;i(s+=r.encode(o).length,s<=n),l=(o,S)=>{if(o===null||typeof o=="boolean"||typeof o=="number")return(typeof o!="number"||Number.isFinite(o))&&c(JSON.stringify(o))?{value:o}:void 0;if(typeof o=="string")return $(o,n)&&c(JSON.stringify(o))?{value:o}:void 0;if(S>t||typeof o!="object"||i.has(o))return;i.add(o);let p=!0,b;if(Array.isArray(o)){const d=[];b=d;const u=x(o,n);if(!u||!c("["))p=!1;else for(let a=0;aT(r,1e5)&&(t||r>0))}function ce(e,t,n){return!C(e,t)||T(e[t],n)}function De(e){return m(e)&&g(e,["provider_number","role","status","returned_bid_count"],["response_time_ms"])&&T(e.provider_number,65535)&&e.provider_number>0&&E(e.role,Le)&&E(e.status,Ue)&&T(e.returned_bid_count,65535)&&ce(e,"response_time_ms",4294967295)}function Ve(e){return m(e)&&g(e,["slot_number","slot_ref","requested_sizes","returned_bid_count","candidate"],["selected_creative_size"])&&T(e.slot_number,65535)&&e.slot_number>0&&ae(e.slot_ref)&&T(e.returned_bid_count,65535)&&E(e.candidate,Be)&&X(e.requested_sizes,16,t=>D(t))&&(!C(e,"selected_creative_size")||D(e.selected_creative_size))}function Fe(e){return m(e)&&g(e,["schema_version","diagnostic_auction_id","source","terminal_status","provider_calls","slots","truncation","coverage"],["terminal_reason","total_time_ms"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&E(e.source,Me)&&E(e.terminal_status,Pe)&&(!C(e,"terminal_reason")||E(e.terminal_reason,je))&&ce(e,"total_time_ms",4294967295)&&X(e.provider_calls,16,De)&&X(e.slots,64,Ve)&&m(e.truncation)&&g(e.truncation,["omitted_provider_calls","omitted_slots","omitted_nested_values"])&&Object.values(e.truncation).every(t=>T(t,65535))&&m(e.coverage)&&g(e.coverage,["provider_to_slot_no_bid"])&&e.coverage.provider_to_slot_no_bid==="unavailable"}function X(e,t,n){const r=x(e,t);return r!==void 0&&r.every(n)}function Ge(e){try{const t=I(e);return t!==void 0&&Fe(t.value)}catch{return!1}}function Ke(e){try{return m(e)&&g(e,["schema_version","diagnostic_auction_id","slot_ref","runtime_slot_number","request_number"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&ae(e.slot_ref)&&T(e.runtime_slot_number)&&e.runtime_slot_number>0&&T(e.request_number)&&e.request_number>0}catch{return!1}}function Ye(e){const t=I(e);return t!==void 0&&Ke(t.value)}const Je=["requestNumber","requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","durations","isEmpty","requestedSlotSizes","size","observedSlotSize","isBackfill","slotContentChanged","incompleteSequence","responseClass","requestPath","requestIntentId","trustedServerAuctionId","opportunityToRequestMs","replacedRequestNumber","previousRenderToRequestMs","creativeChanged","loadObservedBeforeRender","trustedServerOpportunity","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs","trustedServerCreativeFailures","delivery"],He=["requestToResponseMs","responseToRenderMs","requestToRenderMs","renderToLoadMs","renderToViewableMs"],We=["requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","opportunityToRequestMs","previousRenderToRequestMs","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs"],Xe=["isEmpty","isBackfill","slotContentChanged","creativeChanged","loadObservedBeforeRender"],Ze=["omitted_server_auctions","omitted_slot_correlations","omitted_request_cycles","omitted_callback_issues","omitted_attribution_issues","omitted_nested_values"];function de(e,t,n){const r=I(e);if(!(!r||Q(t)!==t||!M(n)||!le(r.value,t,n)))return Z(r.value),r.value}function Qe(e,t,n){const r=I(e,11);if(!(!r||!at(r.value,t,n)))return Z(r.value),r.value}function Z(e){typeof e!="object"||e===null||(Object.values(e).forEach(Z),Object.freeze(e))}function et(e,t,n){try{const r=I(e);if(!r)return"invalid_report";if(e=r.value,pe(e,t,n))return;if(!m(e))return"invalid_report";if(C(e,"schema_version")&&e.schema_version!==1)return"unsupported_report_version";const s=e.gpt_diagnostics;if(m(s)){if(C(s,"schema_version")&&s.schema_version!==1)return"unsupported_gpt_version";if(C(s,"source_schema_version")&&s.source_schema_version!==1)return"unsupported_gpt_source_version"}return x(e.server_auctions,16)?.some(l=>m(l)&&C(l,"schema_version")&&l.schema_version!==1)?"unsupported_auction_version":x(e.slot_correlations,128)?.some(l=>m(l)&&C(l,"schema_version")&&l.schema_version!==1)?"unsupported_correlation_version":"invalid_report"}catch{return"invalid_report"}}function M(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isFinite(e)&&e>=0&&e<=t}function Q(e){if(!$(e,255)||!/^https?:\/\//i.test(e))return;const t=e.slice(e.indexOf("://")+3);if(!t||/[\s/@?#,\\]/.test(t))return;const n=t.startsWith("[")?t.slice(t.indexOf("]")+1):t.includes(":")?t.slice(t.lastIndexOf(":")):"";if(!(n!==""&&(!/^:\d+$/.test(n)||Number(n.slice(1))>65535)))try{const r=new URL(e);return!["http:","https:"].includes(r.protocol)||!r.hostname||r.username||r.password||r.search||r.hash||r.pathname!=="/"?void 0:r.origin}catch{return}}function P(e,t,n){const r=x(e,t);return r!==void 0&&r.every(n)}function A(e,t,n){return!C(e,t)||n(e[t])}function tt(e){return!m(e)||!g(e,["requestNumber","durations","incompleteSequence"],Je)||!T(e.requestNumber)||typeof e.incompleteSequence!="boolean"||!m(e.durations)||!g(e.durations,[],He)||!Object.values(e.durations).every(t=>M(t))||!We.every(t=>A(e,t,M))||!Xe.every(t=>A(e,t,n=>typeof n=="boolean"))?!1:["requestIntentId","replacedRequestNumber"].every(t=>A(e,t,T))&&A(e,"trustedServerAuctionId",W)&&A(e,"requestedSlotSizes",t=>P(t,16,n=>D(n)))&&A(e,"size",t=>D(t))&&A(e,"observedSlotSize",t=>D(t,!0))&&A(e,"responseClass",t=>E(t,ke))&&A(e,"requestPath",t=>E(t,Oe))&&A(e,"trustedServerOpportunity",t=>E(t,Ne))&&A(e,"delivery",t=>E(t,Ie))&&A(e,"trustedServerCreativeFailures",t=>P(t,16,n=>E(n,xe)))}function rt(e){return!m(e)||!g(e,["runtimeSlotNumber","binding","requests"],["currentVisibilityPercentage","maximumVisibilityPercentage"])||!T(e.runtimeSlotNumber)||!m(e.binding)||!g(e.binding,["status"],["reason"])||!E(e.binding.status,["bound","unbound","ambiguous"])||!A(e.binding,"reason",t=>E(t,Ce))?!1:A(e,"currentVisibilityPercentage",t=>M(t,100))&&A(e,"maximumVisibilityPercentage",t=>M(t,100))&&P(e.requests,10,tt)}function nt(e){return m(e)&&g(e,["kind","runtimeSlotNumber","timestampMs","disposition","reason"])&&E(e.kind,ie)&&T(e.runtimeSlotNumber)&&M(e.timestampMs)&&E(e.disposition,["matched","unmatched","ambiguous"])&&E(e.reason,Ae)}function st(e){return m(e)&&g(e,["reason","timestampMs"],["runtimeSlotNumber"])&&E(e.reason,qe)&&M(e.timestampMs)&&A(e,"runtimeSlotNumber",T)}function it(e){return m(e)&&g(e,["observed","matched","unmatched","ambiguous"])&&Object.values(e).every(t=>T(t))}function ot(e,t){return!m(e)||!g(e,["schema_version","source_schema_version","capturedAt","page","slots","callbackIssues","coverage","metadata"],["attributionIssues"])||e.schema_version!==1||e.source_schema_version!==1||!H(e.capturedAt)||!m(e.page)||!g(e.page,["origin","pathname"])||Q(e.page.origin)!==t||e.page.pathname!=="/[redacted]"||!P(e.slots,64,rt)||!P(e.callbackIssues,128,nt)||!A(e,"attributionIssues",n=>P(n,128,st))||!m(e.coverage)||!g(e.coverage,ie)||!Object.values(e.coverage).every(it)?!1:m(e.metadata)&&g(e.metadata,["droppedCallbacks","evictedSlots","evictedRequestCycles"],["droppedAttributionIssues"])&&Object.values(e.metadata).every(n=>T(n))}function ue(e,t){if(!H(e))return!1;const n=Date.parse(e);return Number.isFinite(n)&&Math.abs(n-t)<=6e4}function le(e,t,n){if(!m(e)||!g(e,["schema_version","captured_at","request_context","server_auctions","slot_correlations","gpt_diagnostics","auction_coverage","truncation"])||e.schema_version!==1||!ue(e.captured_at,n)||!ne(e.request_context)||!P(e.server_auctions,16,Ge)||!P(e.slot_correlations,128,Ye)||!ot(e.gpt_diagnostics,t)||!ue(e.gpt_diagnostics.capturedAt,n)||!m(e.truncation)||!g(e.truncation,Ze)||!Object.values(e.truncation).every(p=>T(p,65535)))return!1;const r=e.auction_coverage;if(!m(r)||!g(r,["capture_status","issues"]))return!1;const s=x(r.issues,16);if(!s||!s.every(p=>E(p,oe)))return!1;let i=-1;for(const p of s){const b=oe.indexOf(p);if(b<=i)return!1;i=b}const c=x(e.server_auctions,16);if(!c)return!1;const l=x(e.slot_correlations,128);if(!l)return!1;for(const p of l)if(!m(p)||c.some(b=>m(b)&&b.source==="auction_api"&&b.diagnostic_auction_id===p.diagnostic_auction_id))return!1;const o=s.some(p=>["evidence_projection_failed","evidence_transport_failed","evidence_validation_failed","record_evicted"].includes(p)),S=c.length>0?o?"partial":"complete":o?"unavailable":"not_observed";return r.capture_status===S&&Ee(e,10)!==void 0}function pe(e,t,n){try{const r=I(e);return r!==void 0&&Q(t)===t&&M(n)&&le(r.value,t,n)}catch{return!1}}function at(e,t,n){try{const r=I(e,11);if(!r)return!1;const s=r.value;return m(s)&&g(s,["stored_at_ms","report"])&&M(n)&&T(s.stored_at_ms)&&s.stored_at_ms-n<=6e4&&n-s.stored_at_ms<=9e5&&pe(s.report,t,s.stored_at_ms)}catch{return!1}}function ct(e){switch(e.requestPath){case"prebid_refresh":case"publisher_refresh":return"Browser refresh observed; winner not determined";case"competing":case"unattributed":return"Multiple or unknown delivery paths";case"trusted_server_direct":return"Trusted Server request path observed";default:return"Request path unavailable"}}const dt={initial_navigation_ssat:"Initial-page server auction (SSAT)",spa_page_bids:"Trusted Server page-refresh auction",auction_api:"Trusted Server auction API"};function ut(e,t,n){const r=de(e,t,n);if(!r)return;const s=r.gpt_diagnostics.slots.flatMap(l=>l.requests.map(o=>({runtimeSlotNumber:l.runtimeSlotNumber,cycle:o}))),i=r.server_auctions.flatMap(l=>l.slots.map(o=>({id:l.diagnostic_auction_id,slot:o}))),c=r.server_auctions.map(l=>{const o=l.diagnostic_auction_id,S=r.server_auctions.filter(b=>b.diagnostic_auction_id===o).length===1,p=l.slots.map(b=>{const d=Object.freeze({serverSlot:b,correlation:"unknown"});if(!S||l.source==="auction_api"||i.filter(y=>y.id===o&&y.slot.slot_ref===b.slot_ref).length!==1)return d;const u=r.slot_correlations.filter(y=>y.diagnostic_auction_id===o&&y.slot_ref===b.slot_ref);if(u.length!==1)return d;const a=u[0];if(r.slot_correlations.filter(y=>y.runtime_slot_number===a.runtime_slot_number&&y.request_number===a.request_number).length!==1)return d;const f=s.filter(y=>y.runtimeSlotNumber===a.runtime_slot_number&&y.cycle.requestNumber===a.request_number);if(f.length!==1||f[0].cycle.trustedServerAuctionId!==o)return d;const h=f[0].cycle,L=h.isEmpty===!1&&h.renderAtMs!==void 0&&h.trustedServerCreativeResponseAtMs!==void 0&&h.delivery==="trusted_server_response_sent";return Object.freeze({serverSlot:b,correlation:"matched",runtimeSlotNumber:a.runtime_slot_number,requestNumber:a.request_number,cycle:h,pathLabel:ct(h),creativeLabel:L?"Trusted Server creative rendered":"Participation unconfirmed"})});return Object.freeze({evidence:l,sourceLabel:dt[l.source],relativeMilestonesLabel:"Unavailable in v1",providerScopeLabel:"Auction-wide provider status; per-slot no-bid reason unavailable",slots:Object.freeze(p)})});return Object.freeze({auctions:Object.freeze(c)})}const fe="trusted-server-trace-v1.json";function ee(e,t,n){const r=de(e,t,n);return r?JSON.stringify(r,null,2):void 0}function lt(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};let s;try{s=URL.createObjectURL(new Blob([r],{type:"application/json"}))}catch{return{status:"failed"}}let i;try{return i=document.createElement("a"),i.href=s,i.download=fe,document.body.append(i),i.click(),{status:"downloaded"}}catch{return{status:"failed"}}finally{try{i?.remove()}catch{}window.setTimeout(()=>URL.revokeObjectURL(s),1e3)}}async function pt(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{return typeof navigator.clipboard?.writeText!="function"?{status:"unsupported"}:(await navigator.clipboard.writeText(r),{status:"copied"})}catch{return{status:"failed"}}}async function ft(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{if(typeof navigator.canShare!="function"||typeof navigator.share!="function")return{status:"unsupported"};const i={files:[new File([r],fe,{type:"application/json"})]};return navigator.canShare(i)?(await navigator.share(i),{status:"shared"}):{status:"unsupported"}}catch{return{status:"failed"}}}async function _t(e){return _e(e,!1)}async function bt(){return _e("end",!0)}async function _e(e,t){const n={mutation:"failed",observation:"not_attempted",confirmed:!1};if(e!=="enable"&&e!=="end")return n;let r="failed";try{(await fetch(`/_ts/trace/${e}`,{method:"POST",credentials:"same-origin",cache:"no-store",headers:{"X-TS-Trace-Action":e}})).ok&&(r="requested")}catch{}if(r==="failed"&&!t)return n;const s={mutation:r,observation:"failed",confirmed:!1};try{const i=await fetch("/_ts/trace/state",{method:"GET",credentials:"same-origin",cache:"no-store"});if(!i.ok)return s;const c=await i.json();if(c===null||typeof c!="object"||Array.isArray(c))return s;const l=Object.keys(c);if(l.length!==1||l[0]!=="observed_active")return s;const o=Object.getOwnPropertyDescriptor(c,"observed_active");if(!o||typeof o.value!="boolean")return s;const S=o.value;return{mutation:r,observation:S?"active":"inactive",confirmed:r==="requested"&&S===(e==="enable")}}catch{return s}}const mt="Return to the affected page, reload once, reproduce the problem, then select View trace results.",vt="Reopen the affected article on the exact same hostname and in this same tab, then reload once.",be="Tracing is on — cookie observed by server",te="Tracing is off — no valid diagnostics session observed";function G(e){switch(e.state){case"absent":return"Not present in this request";case"present_valid":return"Valid shape observed";case"present_invalid":return"Invalid shape observed";case"duplicate":return"Multiple values observed";case"unavailable":return e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":"Unavailable — runtime-visible cookie inspection failed"}}function q(e,t,n,r){const s=e.createElement("dt");s.textContent=n;const i=e.createElement("dd");i.textContent=r,t.append(s,i)}function gt(e){const t=e.getElementById("trace-request-context"),n=e.getElementById("trace-network-facts"),r=e.getElementById("trace-cookie-facts");if(!t||!n||!r)return;let s;try{s=JSON.parse(t.textContent??"")}catch{s=void 0}if(t.textContent="",n.replaceChildren(),r.replaceChildren(),!ne(s)){q(e,n,"Request facts","Unavailable"),q(e,r,"Cookie health","Unavailable");return}const i=s.network;q(e,n,"Approximate network identifier",i.masked_client_ip??"Unavailable"),q(e,n,"Country",i.country??"Unavailable"),q(e,n,"Region",i.region??"Unavailable"),q(e,n,"ASN",i.asn===void 0?"Unavailable":String(i.asn)),q(e,n,"HTTP version",i.http_version??"Unavailable"),q(e,n,"TLS protocol",i.tls_protocol??"Unavailable"),q(e,n,"TLS cipher",i.tls_cipher??"Unavailable"),q(e,n,"Edge hostname",i.edge_hostname??"Unavailable"),q(e,n,"Edge region",i.edge_region??"Unavailable"),q(e,n,"Edge POP",i.edge_pop??"Unavailable"),q(e,r,"Edge Cookie",G(s.cookies.ts_ec)),q(e,r,"External IDs",G(s.cookies.ts_eids)),q(e,r,"Tester",G(s.cookies.ts_tester)),q(e,r,"Diagnostics session",G(s.cookies.diagnostics_session))}function ht(e=document){gt(e);const t=e.getElementById("trace-session-state"),n=e.getElementById("trace-status"),r=e.getElementById("trace-enable"),s=e.getElementById("trace-end"),i=e.getElementById("trace-back");if(!t||!n||!(r instanceof HTMLButtonElement)||!(s instanceof HTMLButtonElement)||!(i instanceof HTMLButtonElement))return()=>{};t.textContent=t.dataset.observedActive==="true"?be:t.dataset.observedActive==="false"?te:"Tracing state unconfirmed";let c=!1,l=!1;const o=async d=>{if(!(l||c)){c=!0,r.disabled=s.disabled=!0,n.textContent=d==="enable"?"Verifying activation…":"Verifying deactivation…";try{const u=await _t(d);if(l)return;t.textContent=u.observation==="active"?be:u.observation==="inactive"?te:"Tracing state unconfirmed",u.confirmed?(n.textContent=d==="enable"?mt:te,i.classList.toggle("primary",d==="enable")):n.textContent=`${d==="enable"?"Activation":"Deactivation"} unconfirmed. Try again.`}finally{l||(r.disabled=s.disabled=!1),c=!1}}},S=()=>{o("enable")},p=()=>{o("end")},b=()=>{l||(window.history.length>1?window.history.back():n.textContent=vt)};return r.addEventListener("click",S),s.addEventListener("click",p),i.addEventListener("click",b),()=>{l=!0,r.removeEventListener("click",S),s.removeEventListener("click",p),i.removeEventListener("click",b)}}const re="trusted-server.trace.report.v1";function me(e){return e??window.sessionStorage}function yt(e,t,n){const r=I(e,11)?.value;return m(r)?et(r.report,t,n)??"invalid_report":"invalid_report"}function St(e,t,n){let r,s;try{r=me(n),s=r.getItem(re)}catch{return{status:"unavailable"}}if(s===null)return{status:"absent"};let i,c;try{typeof s=="string"&&new TextEncoder().encode(s).length<=512*1024&&(i=JSON.parse(s),c=Qe(i,e,t))}catch{}if(c)return{status:"ready",value:c};const l=yt(i,e,t);try{r.removeItem(re)}catch{}return{status:"rejected",reason:l}}function Tt(e){try{return me(e).removeItem(re),{status:"deleted"}}catch{return{status:"unavailable"}}}function R(e){return{no_bid:"No bid returned",no_candidate:"No candidate",selected:"Candidate selected",selected_unrenderable:"Selected candidate could not be rendered",trusted_server_direct:"Trusted Server request path observed",prebid_refresh:"Browser refresh observed; winner not determined",publisher_refresh:"Browser refresh observed; winner not determined",competing:"Multiple or unknown delivery paths",unattributed:"Multiple or unknown delivery paths",trusted_server_response_sent:"Trusted Server creative response sent",trusted_server_selected:"Trusted Server candidate selected; render unconfirmed",candidate_unconfirmed:"Candidate unconfirmed",not_observed:"Not observed",unknown:"Unknown",unavailable:"Unavailable"}[e]??e.replace(/([a-z])([A-Z])/g,"$1 $2").replace(/_/g," ").replace(/^./,n=>n.toUpperCase())}function Rt(e){return e===void 0?"Unavailable":typeof e=="boolean"?e?"Yes":"No":typeof e=="number"?String(e):typeof e=="string"?e:"Unavailable"}function K(e){return e.state==="absent"?"Not present in this request":e.state==="present_valid"?"Valid shape observed":e.state==="duplicate"?"Multiple values observed":e.state==="present_invalid"?e.detail==="oversized"?"Invalid shape — too long":e.detail==="unsupported_value"?"Invalid shape — unsupported value":"Invalid shape observed":e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":e.detail==="header_too_large"?"Unavailable — the visible cookie header was too large":"Unavailable — the visible cookie header was not valid text"}function _(e,t,n){const r=e.createElement(t);return n!==void 0&&(r.textContent=n),r}function j(e,t,n,r){const s=_(e,"section");return r&&(s.id=r),s.append(_(e,"h2",n)),t.append(s),s}function w(e,t,n){const r=_(e,"dl");for(const[s,i]of n)r.append(_(e,"dt",s),_(e,"dd",Rt(i)));t.append(r)}function V(e){return e===void 0?"Unavailable":e.length===0?"Not observed":e.map(([t,n])=>`${t} × ${n}`).join(", ")}function O(e){return e===void 0?"Unavailable":`${e} ms`}const wt={masked_client_ip:"Approximate network identifier",country:"Country",region:"Region",asn:"ASN",http_version:"HTTP version",tls_protocol:"TLS protocol",tls_cipher:"TLS cipher",edge_hostname:"Edge hostname",edge_region:"Edge region",edge_pop:"Edge POP"},Et={requestedAtMs:"Requested (browser clock)",responseAtMs:"Response received (browser clock)",renderAtMs:"Rendered (browser clock)",loadAtMs:"Loaded (browser clock)",viewableAtMs:"Viewable (browser clock)",isBackfill:"Backfill observed",slotContentChanged:"Slot content changed",incompleteSequence:"Incomplete sequence",responseClass:"GPT response class",requestIntentId:"Browser request intent number",opportunityToRequestMs:"Opportunity to request",replacedRequestNumber:"Replaced request number",previousRenderToRequestMs:"Previous render to request",creativeChanged:"Creative changed",loadObservedBeforeRender:"Load observed before render",trustedServerOpportunity:"Trusted Server candidate opportunity",trustedServerCreativeRequestAtMs:"Creative bridge request (browser clock)",trustedServerCreativeResponseAtMs:"Creative bridge response (browser clock)",delivery:"Creative delivery observation"};function Ct(e,t,n,r){const s=_(e,"article");s.id="trace-report",s.append(_(e,"h1","Trusted Server trace results"),_(e,"p","Browser-carried, unverified diagnostic data")),s.append(_(e,"p","This browser snapshot helps troubleshoot rendering. It is not proof of a server event, identity, or security incident."));const i=j(e,s,"Report summary");w(e,i,[["Captured at",t.captured_at],["Publisher origin",t.gpt_diagnostics.page.origin],["Server auctions retained",t.server_auctions.length],["GPT slots retained",t.gpt_diagnostics.slots.length]]);const c=j(e,s,"Publisher request");c.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),c.append(_(e,"p","These facts describe the traced publisher document. Masked identifiers are approximate and may still identify a network.")),w(e,c,[["Document request captured at",t.request_context.captured_at],...Object.entries(wt).map(([d,u])=>[u,t.request_context.network[d]])]);const l=j(e,s,"Cookie health","trace-report-cookies");l.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot. Only cookie shape visible in this request is inspected; values and browser attributes are excluded.")),w(e,l,[["Edge Cookie",K(t.request_context.cookies.ts_ec)],["External IDs",K(t.request_context.cookies.ts_eids)],["Tester",K(t.request_context.cookies.ts_tester)],["Diagnostics session",K(t.request_context.cookies.diagnostics_session)]]);const o=j(e,s,"Server auctions");o.append(_(e,"p","Returned bid counts may overlap between slots and must not be summed as unique bids."));const S=ut(t,n,r);t.server_auctions.length||o.append(_(e,"p","Not observed. This does not mean no server auction ran."));for(const[d,u]of(S?.auctions??[]).entries()){const a=_(e,"details");a.open=!0,a.append(_(e,"summary",`Auction ${d+1}: ${u.sourceLabel}`)),a.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),w(e,a,[["Terminal outcome",R(u.evidence.terminal_status)],["Terminal reason",u.evidence.terminal_reason===void 0?"Unavailable":R(u.evidence.terminal_reason)],["Server auction-local elapsed time",O(u.evidence.total_time_ms)],["Request-relative milestones",u.relativeMilestonesLabel]]),a.append(_(e,"p",u.providerScopeLabel));for(const f of u.evidence.provider_calls)w(e,a,[[`Provider call ${f.provider_number}`,R(f.role)],["Call outcome",R(f.status)],["Provider-local elapsed time",O(f.response_time_ms)],["Returned bid count",f.returned_bid_count]]);u.evidence.provider_calls.length||a.append(_(e,"p","Provider calls: Not observed"));for(const f of u.slots){const h=_(e,"details");h.open=!0,h.append(_(e,"summary",`Server slot ${f.serverSlot.slot_number}`)),w(e,h,[["Requested sizes",V(f.serverSlot.requested_sizes)],["Returned bid count",f.serverSlot.returned_bid_count],["Candidate",R(f.serverSlot.candidate)],["Selected creative size",f.serverSlot.selected_creative_size?V([f.serverSlot.selected_creative_size]):"Unavailable"],["Correlation",f.correlation==="matched"?`Matched GPT slot ${f.runtimeSlotNumber}, request ${f.requestNumber}`:"Correlation unknown"],["Browser request path",f.pathLabel??"Unknown"],["Creative participation",f.creativeLabel??"Participation unconfirmed"]]),a.append(h)}w(e,a,[["Provider calls omitted",u.evidence.truncation.omitted_provider_calls],["Server slots omitted",u.evidence.truncation.omitted_slots],["Nested values omitted",u.evidence.truncation.omitted_nested_values]]),o.append(a)}const p=j(e,s,"GPT delivery and creative rendering");p.append(_(e,"p","Browser observed. Server auction → GPT request/response → creative render/load/viewability are separate observations. A filled slot does not identify an auction winner.")),w(e,p,[["Browser snapshot captured at",t.gpt_diagnostics.capturedAt]]),t.gpt_diagnostics.slots.length||p.append(_(e,"p","Not observed"));for(const d of t.gpt_diagnostics.slots){const u=_(e,"details");u.open=!0,u.append(_(e,"summary",`GPT slot ${d.runtimeSlotNumber}`)),w(e,u,[["Binding",R(d.binding.status)],["Binding reason",d.binding.reason===void 0?"Unavailable":R(d.binding.reason)],["Current visibility percentage",d.currentVisibilityPercentage],["Maximum visibility percentage",d.maximumVisibilityPercentage]]),d.requests.length||u.append(_(e,"p","Requests: Not observed"));for(const a of d.requests){const f=_(e,"details");f.open=!0,f.append(_(e,"summary",`Request ${a.requestNumber}`));const h=(S?.auctions??[]).flatMap(y=>y.slots).filter(y=>y.correlation==="matched"&&y.runtimeSlotNumber===d.runtimeSlotNumber&&y.requestNumber===a.requestNumber);w(e,f,[["Correlation",h.length===1?"Matched server slot":"Correlation unknown"],["Creative participation",h.length===1?h[0].creativeLabel:"Participation unconfirmed"],["GPT fill observation",a.isEmpty===void 0?"Unknown":a.isEmpty?"Empty":"Filled"],["Browser request path",a.requestPath===void 0?"Unavailable":R(a.requestPath)],["Requested sizes",V(a.requestedSlotSizes)],["Rendered size",a.size?V([a.size]):"Unavailable"],["Observed CSS box size",a.observedSlotSize?V([a.observedSlotSize]):"Unavailable"]]);const L=Object.entries(Et).map(([y,B])=>{const k=a[y];return[B,typeof k=="string"?R(k):y.endsWith("Ms")?O(k):k]});w(e,f,L),w(e,f,[["Request to response",O(a.durations.requestToResponseMs)],["Response to render",O(a.durations.responseToRenderMs)],["Request to render",O(a.durations.requestToRenderMs)],["Render to load",O(a.durations.renderToLoadMs)],["Render to viewable",O(a.durations.renderToViewableMs)],["Creative bridge failures",a.trustedServerCreativeFailures===void 0?"Unavailable":a.trustedServerCreativeFailures.length===0?"Not observed":a.trustedServerCreativeFailures.map(R).join(", ")]]),u.append(f)}p.append(u)}const b=j(e,s,"Coverage and ambiguity");w(e,b,[["Server capture",R(t.auction_coverage.capture_status)],["Capture and interpretation limits",t.auction_coverage.issues.length?t.auction_coverage.issues.map(R).join(", "):"None recorded"],["Correlation sidecars retained",t.slot_correlations.length]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.coverage))w(e,b,[[R(d),`${u.observed} observed; ${u.matched} matched; ${u.unmatched} unmatched; ${u.ambiguous} ambiguous`]]);for(const[d,u]of Object.entries(t.truncation))w(e,b,[[R(d),u]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.metadata))w(e,b,[[`GPT ${R(d)}`,u]]);for(const d of t.gpt_diagnostics.callbackIssues)w(e,b,[["Callback issue",R(d.kind)],["GPT slot",d.runtimeSlotNumber],["Browser clock",O(d.timestampMs)],["Disposition",R(d.disposition)],["Reason",R(d.reason)]]);for(const d of t.gpt_diagnostics.attributionIssues??[])w(e,b,[["Creative attribution issue",R(d.reason)],["GPT slot",d.runtimeSlotNumber],["Browser clock",O(d.timestampMs)]]);for(const[d,u]of t.slot_correlations.entries()){const a=_(e,"details");a.append(_(e,"summary",`Correlation record ${d+1}`)),a.append(_(e,"p","Browser observed correlation. These opaque references permit a join only when unique and consistent; duplicate or conflicting records remain unknown.")),w(e,a,[["Auction reference",u.diagnostic_auction_id],["Slot reference",u.slot_ref],["GPT slot number",u.runtime_slot_number],["GPT request number",u.request_number]]),b.append(a)}return s}function ve(e=document,t={}){const n=e.querySelector("main");if(!n)return{destroy(){}};const r=ht(e),s=t.origin??window.location.origin;let i,c=!1;const l=[],o=(v,U,Se)=>{const N=_(e,"button",U);N.type="button";const Te=()=>{c||Se()};return N.addEventListener("click",Te),l.push(()=>N.removeEventListener("click",Te)),v.append(N),N},S=()=>{c=!0,i=void 0,r();for(const v of l)v()};let p;try{p=St(s,(t.now??Date.now)(),t.storage)}catch{p={status:"unavailable"}}if(p.status!=="ready"){const v=_(e,"p",p.status==="absent"?"No saved report. Enable tracing, return to the affected page, reload once, reproduce the problem, then select View trace results.":"The saved report is unavailable, expired, or unsupported. Return to the affected page in this same tab and exact hostname, reload once, reproduce the problem, then select View trace results.");return v.id="trace-report-notice",n.prepend(v),{destroy:S}}i=p.value;const b=_(e,"details");b.id="trace-viewer-setup",b.append(_(e,"summary","Setup request and tracing controls"));for(const v of Array.from(n.children))b.append(v);const d=Ct(e,i.report,s,i.stored_at_ms);n.append(d);const u=j(e,d,"Export");u.append(_(e,"p","Copy, Download and Share use the same public report JSON. The selected app receives this JSON when you choose Share. Nothing is uploaded by this viewer."));const a=_(e,"div");a.className="controls",u.append(a);const f=_(e,"p");f.id="trace-export-status",f.setAttribute("role","status"),f.setAttribute("aria-live","polite"),u.append(f);let h=!1;const L=async v=>{if(!i||c||h)return;h=!0;const U=i;try{const N=await(v==="Copy"?t.copy??pt:v==="Share"?t.share??ft:t.download??lt)(U.report,s,U.stored_at_ms);if(c||i!==U)return;f.textContent=N.status==="copied"?"Copied JSON.":N.status==="downloaded"?"Download started; completion is managed by your browser.":N.status==="shared"?"The share request completed.":v==="Share"&&N.status==="unsupported"?"File sharing is unavailable. Use Copy or Download.":`${v} could not be completed. Your report remains available; retry or choose another export.`}catch{!c&&i===U&&(f.textContent=`${v} could not be completed. Your report remains available.`)}finally{h=!1}};for(const v of["Copy","Download","Share"])o(a,v,()=>{L(v)});const y=j(e,n,"Report cleanup"),B=_(e,"div");B.className="controls",y.append(B);const k=_(e,"p","A local report is saved in this tab.");k.id="trace-cleanup-local-status",k.setAttribute("role","status"),k.setAttribute("aria-live","polite"),y.append(k);const z=_(e,"p","Server tracing state has not been changed by this report visit.");z.id="trace-cleanup-server-status",z.setAttribute("role","status"),z.setAttribute("aria-live","polite"),y.append(z);const ge=()=>{let v;try{v=Tt(t.storage)}catch{v={status:"unavailable"}}c||(v.status==="deleted"?(i=void 0,d.remove(),At.hidden=!0,k.textContent="Local report deleted from this tab."):k.textContent="Local report deletion failed. The report remains displayed; retry deletion.")};let Y=!1;const he=async()=>{if(c||Y)return;Y=!0,ye.disabled=J.disabled=!0,z.textContent="Requesting tracing end and checking the next request…";let v;try{v=await bt()}catch{v={mutation:"failed",observation:"failed",confirmed:!1}}c||(z.textContent=v.confirmed?"Tracing is off — no valid diagnostics session observed.":v.observation==="inactive"?"End tracing unconfirmed. No valid diagnostics session was observed; tracing may remain active. Retry end tracing.":v.observation==="active"?"End tracing unconfirmed — a valid session is still observed. Tracing may remain active. Retry end tracing.":"End tracing unconfirmed. Tracing may remain active. Retry end tracing.",J.hidden=v.confirmed,ye.disabled=J.disabled=!1,Y=!1)},ye=o(B,"Clear report and end tracing",()=>{if(Y)return;let v=!1;try{v=(t.confirm??(U=>window.confirm(U)))("Delete the report from this tab and request tracing end?")}catch{}!v||c||(ge(),c||he())}),At=o(B,"Delete local report",ge),J=o(B,"Retry end tracing",()=>{he()});return J.hidden=!0,n.append(b),{destroy:S}}document.readyState==="loading"?document.addEventListener("DOMContentLoaded",()=>ve(),{once:!0}):ve()})();
+(function(){"use strict";function A(e,t){return Object.prototype.hasOwnProperty.call(e,t)}function m(e){if(e===null||typeof e!="object"||Array.isArray(e))return!1;const t=Object.getPrototypeOf(e);return t!==Object.prototype&&t!==null?!1:Reflect.ownKeys(e).every(n=>{if(typeof n!="string")return!1;const r=Object.getOwnPropertyDescriptor(e,n);return r?.enumerable===!0&&A(r,"value")})}function v(e,t,n=[]){return t.every(r=>A(e,r))&&Object.keys(e).every(r=>t.includes(r)||n.includes(r))}function D(e,t=128){if(typeof e!="string")return!1;for(const n of e){const r=n.codePointAt(0);if(r<=31||r>=127&&r<=159||r>=55296&&r<=57343||r===1564||r===8206||r===8207||r>=8234&&r<=8238||r>=8294&&r<=8297)return!1}return new TextEncoder().encode(e).length<=t}function H(e){if(typeof e!="string")return!1;const t=/^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,9})?Z$/.exec(e);if(!t)return!1;const[,n,r,s,i,c,l]=t,o=Number(n),S=Number(r),p=Number(s),d=[31,o%4===0&&(o%100!==0||o%400===0)?29:28,31,30,31,30,31,31,30,31,30,31];return S>=1&&S<=12&&p>=1&&p<=d[S-1]&&Number(i)<=23&&Number(c)<=59&&Number(l)<=59}function T(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isSafeInteger(e)&&e>=0&&e<=t}function we(e){if(!D(e))return!1;const t=/^((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.((?:0|[1-9]\d{0,2}))\.0\/24$/.exec(e);if(t)return t.slice(1).every(s=>Number(s)<=255);if(!e.endsWith("::/48"))return!1;const n=e.slice(0,-3),r=n.slice(0,-2);if(r!==""&&!/^[\da-f]{1,4}(?::[\da-f]{1,4}){0,2}$/.test(r))return!1;try{return new URL(`http://[${n}]/`).hostname===`[${n}]`}catch{return!1}}function G(e,t){if(!m(e)||e.source!=="request")return!1;if(e.state==="absent")return v(e,["source","state"]);if(!v(e,["source","state","detail"])||typeof e.detail!="string")return!1;switch(e.state){case"present_valid":return e.detail===t;case"present_invalid":return["malformed","oversized","unsupported_value"].includes(e.detail);case"duplicate":return e.detail==="multiple_values";case"unavailable":return["header_too_large","header_not_utf8","runtime_header_ambiguous"].includes(e.detail);default:return!1}}function ne(e){try{return Ee(e)}catch{return!1}}function Ee(e){if(!m(e)||!v(e,["schema_version","captured_at","network","cookies"])||e.schema_version!==1||!H(e.captured_at)||!m(e.network)||!m(e.cookies))return!1;const t=e.network,n={country:2,region:32,http_version:32,tls_protocol:32,tls_cipher:32,edge_hostname:128,edge_region:128,edge_pop:32};if(!v(t,[],["masked_client_ip","asn",...Object.keys(n)])||A(t,"masked_client_ip")&&!we(t.masked_client_ip)||A(t,"asn")&&!T(t.asn,4294967295))return!1;for(const[s,i]of Object.entries(n))if(A(t,s)&&!D(t[s],i))return!1;if(typeof t.country=="string"&&!/^[\x20-\x7e]{0,2}$/.test(t.country))return!1;const r=e.cookies;return v(r,["ts_ec","ts_eids","ts_tester","diagnostics_session"])&&G(r.ts_ec,"valid_ec_format")&&G(r.ts_eids,"valid_eids_format")&&G(r.ts_tester,"valid_tester_value")&&G(r.diagnostics_session,"valid_diagnostics_value")}function I(e,t){if(!Array.isArray(e)||Object.getPrototypeOf(e)!==Array.prototype)return;const n=Object.getOwnPropertyDescriptor(e,"length")?.value;if(!T(n,t))return;const r=Reflect.ownKeys(e);if(r.length!==n+1||!r.every(i=>{if(i==="length")return!0;if(typeof i!="string"||!/^(?:0|[1-9]\d*)$/.test(i)||Number(i)>=n)return!1;const c=Object.getOwnPropertyDescriptor(e,i);return c?.enumerable===!0&&A(c,"value")}))return;const s=[];for(let i=0;i(s+=r.encode(o).length,s<=n),l=(o,S)=>{if(o===null||typeof o=="boolean"||typeof o=="number")return(typeof o!="number"||Number.isFinite(o))&&c(JSON.stringify(o))?{value:o}:void 0;if(typeof o=="string")return D(o,n)&&c(JSON.stringify(o))?{value:o}:void 0;if(S>t||typeof o!="object"||i.has(o))return;i.add(o);let p=!0,b;if(Array.isArray(o)){const d=[];b=d;const u=I(o,n);if(!u||!c("["))p=!1;else for(let a=0;aT(r,1e5)&&(t||r>0))}function ce(e,t,n){return!A(e,t)||T(e[t],n)}function Ve(e){return m(e)&&v(e,["provider_number","role","status","returned_bid_count"],["response_time_ms"])&&T(e.provider_number,65535)&&e.provider_number>0&&E(e.role,Ue)&&E(e.status,Be)&&T(e.returned_bid_count,65535)&&ce(e,"response_time_ms",4294967295)}function Fe(e){return m(e)&&v(e,["slot_number","slot_ref","requested_sizes","returned_bid_count","candidate"],["selected_creative_size"])&&T(e.slot_number,65535)&&e.slot_number>0&&ae(e.slot_ref)&&T(e.returned_bid_count,65535)&&E(e.candidate,ze)&&X(e.requested_sizes,16,t=>V(t))&&(!A(e,"selected_creative_size")||V(e.selected_creative_size))}function Ge(e){return m(e)&&v(e,["schema_version","diagnostic_auction_id","source","terminal_status","provider_calls","slots","truncation","coverage"],["terminal_reason","total_time_ms"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&E(e.source,Pe)&&E(e.terminal_status,je)&&(!A(e,"terminal_reason")||E(e.terminal_reason,Le))&&ce(e,"total_time_ms",4294967295)&&X(e.provider_calls,16,Ve)&&X(e.slots,64,Fe)&&m(e.truncation)&&v(e.truncation,["omitted_provider_calls","omitted_slots","omitted_nested_values"])&&Object.values(e.truncation).every(t=>T(t,65535))&&m(e.coverage)&&v(e.coverage,["provider_to_slot_no_bid"])&&e.coverage.provider_to_slot_no_bid==="unavailable"}function X(e,t,n){const r=I(e,t);return r!==void 0&&r.every(n)}function Ke(e){try{const t=M(e);return t!==void 0&&Ge(t.value)}catch{return!1}}function Ye(e){try{return m(e)&&v(e,["schema_version","diagnostic_auction_id","slot_ref","runtime_slot_number","request_number"])&&e.schema_version===1&&W(e.diagnostic_auction_id)&&ae(e.slot_ref)&&T(e.runtime_slot_number)&&e.runtime_slot_number>0&&T(e.request_number)&&e.request_number>0}catch{return!1}}function Je(e){const t=M(e);return t!==void 0&&Ye(t.value)}const He=["requestNumber","requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","durations","isEmpty","requestedSlotSizes","size","observedSlotSize","isBackfill","slotContentChanged","incompleteSequence","responseClass","requestPath","requestIntentId","trustedServerAuctionId","opportunityToRequestMs","replacedRequestNumber","previousRenderToRequestMs","creativeChanged","loadObservedBeforeRender","trustedServerOpportunity","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs","trustedServerCreativeFailures","delivery"],We=["requestToResponseMs","responseToRenderMs","requestToRenderMs","renderToLoadMs","renderToViewableMs"],Xe=["requestedAtMs","responseAtMs","renderAtMs","loadAtMs","viewableAtMs","opportunityToRequestMs","previousRenderToRequestMs","trustedServerCreativeRequestAtMs","trustedServerCreativeResponseAtMs"],Ze=["isEmpty","isBackfill","slotContentChanged","creativeChanged","loadObservedBeforeRender"],Qe=["omitted_server_auctions","omitted_slot_correlations","omitted_request_cycles","omitted_callback_issues","omitted_attribution_issues","omitted_nested_values"];function de(e,t,n){const r=M(e);if(!(!r||Q(t)!==t||!P(n)||!le(r.value,t,n)))return Z(r.value),r.value}function et(e,t,n){const r=M(e,11);if(!(!r||!ct(r.value,t,n)))return Z(r.value),r.value}function Z(e){typeof e!="object"||e===null||(Object.values(e).forEach(Z),Object.freeze(e))}function tt(e,t,n){try{const r=M(e);if(!r)return"invalid_report";if(e=r.value,pe(e,t,n))return;if(!m(e))return"invalid_report";if(A(e,"schema_version")&&e.schema_version!==1)return"unsupported_report_version";const s=e.gpt_diagnostics;if(m(s)){if(A(s,"schema_version")&&s.schema_version!==1)return"unsupported_gpt_version";if(A(s,"source_schema_version")&&s.source_schema_version!==1)return"unsupported_gpt_source_version"}return I(e.server_auctions,16)?.some(l=>m(l)&&A(l,"schema_version")&&l.schema_version!==1)?"unsupported_auction_version":I(e.slot_correlations,128)?.some(l=>m(l)&&A(l,"schema_version")&&l.schema_version!==1)?"unsupported_correlation_version":"invalid_report"}catch{return"invalid_report"}}function P(e,t=Number.MAX_SAFE_INTEGER){return typeof e=="number"&&Number.isFinite(e)&&e>=0&&e<=t}function Q(e){if(!D(e,255)||!/^https?:\/\//i.test(e))return;const t=e.slice(e.indexOf("://")+3);if(!t||/[\s/@?#,\\]/.test(t))return;const n=t.startsWith("[")?t.slice(t.indexOf("]")+1):t.includes(":")?t.slice(t.lastIndexOf(":")):"";if(!(n!==""&&(!/^:\d+$/.test(n)||Number(n.slice(1))>65535)))try{const r=new URL(e);return!["http:","https:"].includes(r.protocol)||!r.hostname||r.username||r.password||r.search||r.hash||r.pathname!=="/"?void 0:r.origin}catch{return}}function j(e,t,n){const r=I(e,t);return r!==void 0&&r.every(n)}function q(e,t,n){return!A(e,t)||n(e[t])}function rt(e){return!m(e)||!v(e,["requestNumber","durations","incompleteSequence"],He)||!T(e.requestNumber)||typeof e.incompleteSequence!="boolean"||!m(e.durations)||!v(e.durations,[],We)||!Object.values(e.durations).every(t=>P(t))||!Xe.every(t=>q(e,t,P))||!Ze.every(t=>q(e,t,n=>typeof n=="boolean"))?!1:["requestIntentId","replacedRequestNumber"].every(t=>q(e,t,T))&&q(e,"trustedServerAuctionId",W)&&q(e,"requestedSlotSizes",t=>j(t,16,n=>V(n)))&&q(e,"size",t=>V(t))&&q(e,"observedSlotSize",t=>V(t,!0))&&q(e,"responseClass",t=>E(t,Oe))&&q(e,"requestPath",t=>E(t,Ne))&&q(e,"trustedServerOpportunity",t=>E(t,xe))&&q(e,"delivery",t=>E(t,Me))&&q(e,"trustedServerCreativeFailures",t=>j(t,16,n=>E(n,Ie)))}function nt(e){return!m(e)||!v(e,["runtimeSlotNumber","binding","requests"],["currentVisibilityPercentage","maximumVisibilityPercentage"])||!T(e.runtimeSlotNumber)||!m(e.binding)||!v(e.binding,["status"],["reason"])||!E(e.binding.status,["bound","unbound","ambiguous"])||!q(e.binding,"reason",t=>E(t,Ae))?!1:q(e,"currentVisibilityPercentage",t=>P(t,100))&&q(e,"maximumVisibilityPercentage",t=>P(t,100))&&j(e.requests,10,rt)}function st(e){return m(e)&&v(e,["kind","runtimeSlotNumber","timestampMs","disposition","reason"])&&E(e.kind,ie)&&T(e.runtimeSlotNumber)&&P(e.timestampMs)&&E(e.disposition,["matched","unmatched","ambiguous"])&&E(e.reason,qe)}function it(e){return m(e)&&v(e,["reason","timestampMs"],["runtimeSlotNumber"])&&E(e.reason,ke)&&P(e.timestampMs)&&q(e,"runtimeSlotNumber",T)}function ot(e){return m(e)&&v(e,["observed","matched","unmatched","ambiguous"])&&Object.values(e).every(t=>T(t))}function at(e,t){return!m(e)||!v(e,["schema_version","source_schema_version","capturedAt","page","slots","callbackIssues","coverage","metadata"],["attributionIssues"])||e.schema_version!==1||e.source_schema_version!==1||!H(e.capturedAt)||!m(e.page)||!v(e.page,["origin","pathname"])||Q(e.page.origin)!==t||e.page.pathname!=="/[redacted]"||!j(e.slots,64,nt)||!j(e.callbackIssues,128,st)||!q(e,"attributionIssues",n=>j(n,128,it))||!m(e.coverage)||!v(e.coverage,ie)||!Object.values(e.coverage).every(ot)?!1:m(e.metadata)&&v(e.metadata,["droppedCallbacks","evictedSlots","evictedRequestCycles"],["droppedAttributionIssues"])&&Object.values(e.metadata).every(n=>T(n))}function ue(e,t){if(!H(e))return!1;const n=Date.parse(e);return Number.isFinite(n)&&Math.abs(n-t)<=6e4}function le(e,t,n){if(!m(e)||!v(e,["schema_version","captured_at","request_context","server_auctions","slot_correlations","gpt_diagnostics","auction_coverage","truncation"])||e.schema_version!==1||!ue(e.captured_at,n)||!ne(e.request_context)||!j(e.server_auctions,16,Ke)||!j(e.slot_correlations,128,Je)||!at(e.gpt_diagnostics,t)||!ue(e.gpt_diagnostics.capturedAt,n)||!m(e.truncation)||!v(e.truncation,Qe)||!Object.values(e.truncation).every(p=>T(p,65535)))return!1;const r=e.auction_coverage;if(!m(r)||!v(r,["capture_status","issues"]))return!1;const s=I(r.issues,16);if(!s||!s.every(p=>E(p,oe)))return!1;let i=-1;for(const p of s){const b=oe.indexOf(p);if(b<=i)return!1;i=b}const c=I(e.server_auctions,16);if(!c)return!1;const l=I(e.slot_correlations,128);if(!l)return!1;for(const p of l)if(!m(p)||c.some(b=>m(b)&&b.source==="auction_api"&&b.diagnostic_auction_id===p.diagnostic_auction_id))return!1;const o=s.some(p=>["evidence_projection_failed","evidence_transport_failed","evidence_validation_failed","record_evicted"].includes(p)),S=c.length>0?o?"partial":"complete":o?"unavailable":"not_observed";return r.capture_status===S&&Ce(e,10)!==void 0}function pe(e,t,n){try{const r=M(e);return r!==void 0&&Q(t)===t&&P(n)&&le(r.value,t,n)}catch{return!1}}function ct(e,t,n){try{const r=M(e,11);if(!r)return!1;const s=r.value;return m(s)&&v(s,["stored_at_ms","report"])&&P(n)&&T(s.stored_at_ms)&&s.stored_at_ms-n<=6e4&&n-s.stored_at_ms<=9e5&&pe(s.report,t,s.stored_at_ms)}catch{return!1}}function dt(e){switch(e.requestPath){case"prebid_refresh":case"publisher_refresh":return"Browser refresh observed; winner not determined";case"competing":case"unattributed":return"Multiple or unknown delivery paths";case"trusted_server_direct":return"Trusted Server request path observed";default:return"Request path unavailable"}}const ut={initial_navigation_ssat:"Initial-page server auction (SSAT)",spa_page_bids:"Trusted Server page-refresh auction",auction_api:"Trusted Server auction API"};function lt(e,t,n){const r=de(e,t,n);if(!r)return;const s=r.gpt_diagnostics.slots.flatMap(l=>l.requests.map(o=>({runtimeSlotNumber:l.runtimeSlotNumber,cycle:o}))),i=r.server_auctions.flatMap(l=>l.slots.map(o=>({id:l.diagnostic_auction_id,slot:o}))),c=r.server_auctions.map(l=>{const o=l.diagnostic_auction_id,S=r.server_auctions.filter(b=>b.diagnostic_auction_id===o).length===1,p=l.slots.map(b=>{const d=Object.freeze({serverSlot:b,correlation:"unknown"});if(!S||l.source==="auction_api"||i.filter(y=>y.id===o&&y.slot.slot_ref===b.slot_ref).length!==1)return d;const u=r.slot_correlations.filter(y=>y.diagnostic_auction_id===o&&y.slot_ref===b.slot_ref);if(u.length!==1)return d;const a=u[0];if(r.slot_correlations.filter(y=>y.runtime_slot_number===a.runtime_slot_number&&y.request_number===a.request_number).length!==1)return d;const f=s.filter(y=>y.runtimeSlotNumber===a.runtime_slot_number&&y.cycle.requestNumber===a.request_number);if(f.length!==1||f[0].cycle.trustedServerAuctionId!==o)return d;const h=f[0].cycle,U=h.isEmpty===!1&&h.renderAtMs!==void 0&&h.trustedServerCreativeResponseAtMs!==void 0&&h.delivery==="trusted_server_response_sent";return Object.freeze({serverSlot:b,correlation:"matched",runtimeSlotNumber:a.runtime_slot_number,requestNumber:a.request_number,cycle:h,pathLabel:dt(h),creativeLabel:U?"Trusted Server creative rendered":"Participation unconfirmed"})});return Object.freeze({evidence:l,sourceLabel:ut[l.source],relativeMilestonesLabel:"Unavailable in v1",providerScopeLabel:"Auction-wide provider status; per-slot no-bid reason unavailable",slots:Object.freeze(p)})});return Object.freeze({auctions:Object.freeze(c)})}const fe="trusted-server-trace-v1.json";function ee(e,t,n){const r=de(e,t,n);return r?JSON.stringify(r,null,2):void 0}function pt(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};let s;try{s=URL.createObjectURL(new Blob([r],{type:"application/json"}))}catch{return{status:"failed"}}let i;try{return i=document.createElement("a"),i.href=s,i.download=fe,document.body.append(i),i.click(),{status:"downloaded"}}catch{return{status:"failed"}}finally{try{i?.remove()}catch{}window.setTimeout(()=>URL.revokeObjectURL(s),1e3)}}async function ft(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{return typeof navigator.clipboard?.writeText!="function"?{status:"unsupported"}:(await navigator.clipboard.writeText(r),{status:"copied"})}catch{return{status:"failed"}}}async function _t(e,t,n){const r=ee(e,t,n);if(r===void 0)return{status:"invalid_report"};try{if(typeof navigator.canShare!="function"||typeof navigator.share!="function")return{status:"unsupported"};const i={files:[new File([r],fe,{type:"application/json"})]};return navigator.canShare(i)?(await navigator.share(i),{status:"shared"}):{status:"unsupported"}}catch{return{status:"failed"}}}async function bt(e){return _e(e,!1)}async function mt(){return _e("end",!0)}async function _e(e,t){const n={mutation:"failed",observation:"not_attempted",confirmed:!1};if(e!=="enable"&&e!=="end")return n;let r="failed";try{(await fetch(`/_ts/trace/${e}`,{method:"POST",credentials:"same-origin",cache:"no-store",headers:{"X-TS-Trace-Action":e}})).ok&&(r="requested")}catch{}if(r==="failed"&&!t)return n;const s={mutation:r,observation:"failed",confirmed:!1};try{const i=await fetch("/_ts/trace/state",{method:"GET",credentials:"same-origin",cache:"no-store"});if(!i.ok)return s;const c=await i.json();if(c===null||typeof c!="object"||Array.isArray(c))return s;const l=Object.keys(c);if(l.length!==1||l[0]!=="observed_active")return s;const o=Object.getOwnPropertyDescriptor(c,"observed_active");if(!o||typeof o.value!="boolean")return s;const S=o.value;return{mutation:r,observation:S?"active":"inactive",confirmed:r==="requested"&&S===(e==="enable")}}catch{return s}}const vt="Return to the affected page, reload once, reproduce the problem, then select View trace results.",gt="Reopen the affected article on the exact same hostname and in this same tab, then reload once.",be="Tracing is on — cookie observed by server",te="Tracing is off — no valid diagnostics session observed";function K(e){switch(e.state){case"absent":return"Not present in this request";case"present_valid":return"Valid shape observed";case"present_invalid":return"Invalid shape observed";case"duplicate":return"Multiple values observed";case"unavailable":return e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":"Unavailable — runtime-visible cookie inspection failed"}}function k(e,t,n,r){const s=e.createElement("dt");s.textContent=n;const i=e.createElement("dd");i.textContent=r,t.append(s,i)}function ht(e){const t=e.getElementById("trace-request-context"),n=e.getElementById("trace-network-facts"),r=e.getElementById("trace-cookie-facts");if(!t||!n||!r)return;let s;try{s=JSON.parse(t.textContent??"")}catch{s=void 0}if(t.textContent="",n.replaceChildren(),r.replaceChildren(),!ne(s)){k(e,n,"Request facts","Unavailable"),k(e,r,"Cookie health","Unavailable");return}const i=s.network;k(e,n,"Approximate network identifier",i.masked_client_ip??"Unavailable"),k(e,n,"Country",i.country??"Unavailable"),k(e,n,"Region",i.region??"Unavailable"),k(e,n,"ASN",i.asn===void 0?"Unavailable":String(i.asn)),k(e,n,"HTTP version",i.http_version??"Unavailable"),k(e,n,"TLS protocol",i.tls_protocol??"Unavailable"),k(e,n,"TLS cipher",i.tls_cipher??"Unavailable"),k(e,n,"Edge hostname",i.edge_hostname??"Unavailable"),k(e,n,"Edge region",i.edge_region??"Unavailable"),k(e,n,"Edge POP",i.edge_pop??"Unavailable"),k(e,r,"Edge Cookie",K(s.cookies.ts_ec)),k(e,r,"External IDs",K(s.cookies.ts_eids)),k(e,r,"Tester",K(s.cookies.ts_tester)),k(e,r,"Diagnostics session",K(s.cookies.diagnostics_session))}function yt(e=document){ht(e);const t=e.getElementById("trace-session-state"),n=e.getElementById("trace-status"),r=e.getElementById("trace-enable"),s=e.getElementById("trace-end"),i=e.getElementById("trace-back");if(!t||!n||!(r instanceof HTMLButtonElement)||!(s instanceof HTMLButtonElement)||!(i instanceof HTMLButtonElement))return()=>{};t.textContent=t.dataset.observedActive==="true"?be:t.dataset.observedActive==="false"?te:"Tracing state unconfirmed";let c=!1,l=!1;const o=async d=>{if(!(l||c)){c=!0,r.disabled=s.disabled=!0,n.textContent=d==="enable"?"Verifying activation…":"Verifying deactivation…";try{const u=await bt(d);if(l)return;t.textContent=u.observation==="active"?be:u.observation==="inactive"?te:"Tracing state unconfirmed",u.confirmed?(n.textContent=d==="enable"?vt:te,i.classList.toggle("primary",d==="enable")):n.textContent=`${d==="enable"?"Activation":"Deactivation"} unconfirmed. Try again.`}finally{l||(r.disabled=s.disabled=!1),c=!1}}},S=()=>{o("enable")},p=()=>{o("end")},b=()=>{l||(window.history.length>1?window.history.back():n.textContent=gt)};return r.addEventListener("click",S),s.addEventListener("click",p),i.addEventListener("click",b),()=>{l=!0,r.removeEventListener("click",S),s.removeEventListener("click",p),i.removeEventListener("click",b)}}const re="trusted-server.trace.report.v1";function me(e){return e??window.sessionStorage}function St(e,t,n){const r=M(e,11)?.value;return m(r)?tt(r.report,t,n)??"invalid_report":"invalid_report"}function Tt(e,t,n){let r,s;try{r=me(n),s=r.getItem(re)}catch{return{status:"unavailable"}}if(s===null)return{status:"absent"};let i,c;try{typeof s=="string"&&new TextEncoder().encode(s).length<=512*1024&&(i=JSON.parse(s),c=et(i,e,t))}catch{}if(c)return{status:"ready",value:c};const l=St(i,e,t);try{r.removeItem(re)}catch{}return{status:"rejected",reason:l}}function Rt(e){try{return me(e).removeItem(re),{status:"deleted"}}catch{return{status:"unavailable"}}}function R(e){return{no_bid:"No bid returned",no_candidate:"No candidate",selected:"Candidate selected",selected_unrenderable:"Selected candidate could not be rendered",trusted_server_direct:"Trusted Server request path observed",prebid_refresh:"Browser refresh observed; winner not determined",publisher_refresh:"Browser refresh observed; winner not determined",competing:"Multiple or unknown delivery paths",unattributed:"Multiple or unknown delivery paths",trusted_server_response_sent:"Trusted Server creative response sent",trusted_server_selected:"Trusted Server candidate selected; render unconfirmed",candidate_unconfirmed:"Candidate unconfirmed",not_observed:"Not observed",unknown:"Unknown",unavailable:"Unavailable"}[e]??e.replace(/([a-z])([A-Z])/g,"$1 $2").replace(/_/g," ").replace(/^./,n=>n.toUpperCase())}function wt(e){return e===void 0?"Unavailable":typeof e=="boolean"?e?"Yes":"No":typeof e=="number"?String(e):typeof e=="string"?e:"Unavailable"}function Y(e){return e.state==="absent"?"Not present in this request":e.state==="present_valid"?"Valid shape observed":e.state==="duplicate"?"Multiple values observed":e.state==="present_invalid"?e.detail==="oversized"?"Invalid shape — too long":e.detail==="unsupported_value"?"Invalid shape — unsupported value":"Invalid shape observed":e.detail==="runtime_header_ambiguous"?"Unavailable — runtime-visible cookies could not be reliably inspected":e.detail==="header_too_large"?"Unavailable — the visible cookie header was too large":"Unavailable — the visible cookie header was not valid text"}function _(e,t,n){const r=e.createElement(t);return n!==void 0&&(r.textContent=n),r}function L(e,t,n,r){const s=_(e,"section");return r&&(s.id=r),s.append(_(e,"h2",n)),t.append(s),s}function w(e,t,n){const r=_(e,"dl");for(const[s,i]of n)r.append(_(e,"dt",s),_(e,"dd",wt(i)));t.append(r)}function F(e){return e===void 0?"Unavailable":e.length===0?"Not observed":e.map(([t,n])=>`${t} × ${n}`).join(", ")}function N(e){return e===void 0?"Unavailable":`${e} ms`}const Et={masked_client_ip:"Approximate network identifier",country:"Country",region:"Region",asn:"ASN",http_version:"HTTP version",tls_protocol:"TLS protocol",tls_cipher:"TLS cipher",edge_hostname:"Edge hostname",edge_region:"Edge region",edge_pop:"Edge POP"},Ct={requestedAtMs:"Requested (browser clock)",responseAtMs:"Response received (browser clock)",renderAtMs:"Rendered (browser clock)",loadAtMs:"Loaded (browser clock)",viewableAtMs:"Viewable (browser clock)",isBackfill:"Backfill observed",slotContentChanged:"Slot content changed",incompleteSequence:"Incomplete sequence",responseClass:"GPT response class",requestIntentId:"Browser request intent number",opportunityToRequestMs:"Opportunity to request",replacedRequestNumber:"Replaced request number",previousRenderToRequestMs:"Previous render to request",creativeChanged:"Creative changed",loadObservedBeforeRender:"Load observed before render",trustedServerOpportunity:"Trusted Server candidate opportunity",trustedServerCreativeRequestAtMs:"Creative bridge request (browser clock)",trustedServerCreativeResponseAtMs:"Creative bridge response (browser clock)",delivery:"Creative delivery observation"};function At(e,t,n,r){const s=_(e,"article");s.id="trace-report",s.append(_(e,"h1","Trusted Server trace results"),_(e,"p","Browser-carried, unverified diagnostic data")),s.append(_(e,"p","This browser snapshot helps troubleshoot rendering. It is not proof of a server event, identity, or security incident."));const i=L(e,s,"Report summary");w(e,i,[["Captured at",t.captured_at],["Publisher origin",t.gpt_diagnostics.page.origin],["Server auctions retained",t.server_auctions.length],["GPT slots retained",t.gpt_diagnostics.slots.length]]);const c=L(e,s,"Publisher request");c.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),c.append(_(e,"p","These facts describe the traced publisher document. Masked identifiers are approximate and may still identify a network.")),w(e,c,[["Document request captured at",t.request_context.captured_at],...Object.entries(Et).map(([d,u])=>[u,t.request_context.network[d]])]);const l=L(e,s,"Cookie health","trace-report-cookies");l.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot. Only cookie shape visible in this request is inspected; values and browser attributes are excluded.")),w(e,l,[["Edge Cookie",Y(t.request_context.cookies.ts_ec)],["External IDs",Y(t.request_context.cookies.ts_eids)],["Tester",Y(t.request_context.cookies.ts_tester)],["Diagnostics session",Y(t.request_context.cookies.diagnostics_session)]]);const o=L(e,s,"Server auctions");o.append(_(e,"p","Returned bid counts may overlap between slots and must not be summed as unique bids."));const S=lt(t,n,r);t.server_auctions.length||o.append(_(e,"p","Not observed. This does not mean no server auction ran."));for(const[d,u]of(S?.auctions??[]).entries()){const a=_(e,"details");a.open=!0,a.append(_(e,"summary",`Auction ${d+1}: ${u.sourceLabel}`)),a.append(_(e,"p","Produced by Trusted Server; copied through an untrusted browser snapshot")),w(e,a,[["Terminal outcome",R(u.evidence.terminal_status)],["Terminal reason",u.evidence.terminal_reason===void 0?"Unavailable":R(u.evidence.terminal_reason)],["Server auction-local elapsed time",N(u.evidence.total_time_ms)],["Request-relative milestones",u.relativeMilestonesLabel]]),a.append(_(e,"p",u.providerScopeLabel));for(const f of u.evidence.provider_calls)w(e,a,[[`Provider call ${f.provider_number}`,R(f.role)],["Call outcome",R(f.status)],["Provider-local elapsed time",N(f.response_time_ms)],["Returned bid count",f.returned_bid_count]]);u.evidence.provider_calls.length||a.append(_(e,"p","Provider calls: Not observed"));for(const f of u.slots){const h=_(e,"details");h.open=!0,h.append(_(e,"summary",`Server slot ${f.serverSlot.slot_number}`)),w(e,h,[["Requested sizes",F(f.serverSlot.requested_sizes)],["Returned bid count",f.serverSlot.returned_bid_count],["Candidate",R(f.serverSlot.candidate)],["Selected creative size",f.serverSlot.selected_creative_size?F([f.serverSlot.selected_creative_size]):"Unavailable"],["Correlation",f.correlation==="matched"?`Matched GPT slot ${f.runtimeSlotNumber}, request ${f.requestNumber}`:"Correlation unknown"],["Browser request path",f.pathLabel??"Unknown"],["Creative participation",f.creativeLabel??"Participation unconfirmed"]]),a.append(h)}w(e,a,[["Provider calls omitted",u.evidence.truncation.omitted_provider_calls],["Server slots omitted",u.evidence.truncation.omitted_slots],["Nested values omitted",u.evidence.truncation.omitted_nested_values]]),o.append(a)}const p=L(e,s,"GPT delivery and creative rendering");p.append(_(e,"p","Browser observed. Server auction → GPT request/response → creative render/load/viewability are separate observations. A filled slot does not identify an auction winner.")),w(e,p,[["Browser snapshot captured at",t.gpt_diagnostics.capturedAt]]),t.gpt_diagnostics.slots.length||p.append(_(e,"p","Not observed"));for(const d of t.gpt_diagnostics.slots){const u=_(e,"details");u.open=!0,u.append(_(e,"summary",`GPT slot ${d.runtimeSlotNumber}`)),w(e,u,[["Binding",R(d.binding.status)],["Binding reason",d.binding.reason===void 0?"Unavailable":R(d.binding.reason)],["Current visibility percentage",d.currentVisibilityPercentage],["Maximum visibility percentage",d.maximumVisibilityPercentage]]),d.requests.length||u.append(_(e,"p","Requests: Not observed"));for(const a of d.requests){const f=_(e,"details");f.open=!0,f.append(_(e,"summary",`Request ${a.requestNumber}`));const h=(S?.auctions??[]).flatMap(y=>y.slots).filter(y=>y.correlation==="matched"&&y.runtimeSlotNumber===d.runtimeSlotNumber&&y.requestNumber===a.requestNumber);w(e,f,[["Correlation",h.length===1?"Matched server slot":"Correlation unknown"],["Creative participation",h.length===1?h[0].creativeLabel:"Participation unconfirmed"],["GPT fill observation",a.isEmpty===void 0?"Unknown":a.isEmpty?"Empty":"Filled"],["Browser request path",a.requestPath===void 0?"Unavailable":R(a.requestPath)],["Requested sizes",F(a.requestedSlotSizes)],["Rendered size",a.size?F([a.size]):"Unavailable"],["Observed CSS box size",a.observedSlotSize?F([a.observedSlotSize]):"Unavailable"]]);const U=Object.entries(Ct).map(([y,z])=>{const O=a[y];return[z,typeof O=="string"?R(O):y.endsWith("Ms")?N(O):O]});w(e,f,U),w(e,f,[["Request to response",N(a.durations.requestToResponseMs)],["Response to render",N(a.durations.responseToRenderMs)],["Request to render",N(a.durations.requestToRenderMs)],["Render to load",N(a.durations.renderToLoadMs)],["Render to viewable",N(a.durations.renderToViewableMs)],["Creative bridge failures",a.trustedServerCreativeFailures===void 0?"Unavailable":a.trustedServerCreativeFailures.length===0?"Not observed":a.trustedServerCreativeFailures.map(R).join(", ")]]),u.append(f)}p.append(u)}const b=L(e,s,"Coverage and ambiguity");w(e,b,[["Server capture",R(t.auction_coverage.capture_status)],["Capture and interpretation limits",t.auction_coverage.issues.length?t.auction_coverage.issues.map(R).join(", "):"None recorded"],["Correlation sidecars retained",t.slot_correlations.length]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.coverage))w(e,b,[[R(d),`${u.observed} observed; ${u.matched} matched; ${u.unmatched} unmatched; ${u.ambiguous} ambiguous`]]);for(const[d,u]of Object.entries(t.truncation))w(e,b,[[R(d),u]]);for(const[d,u]of Object.entries(t.gpt_diagnostics.metadata))w(e,b,[[`GPT ${R(d)}`,u]]);for(const d of t.gpt_diagnostics.callbackIssues)w(e,b,[["Callback issue",R(d.kind)],["GPT slot",d.runtimeSlotNumber],["Browser clock",N(d.timestampMs)],["Disposition",R(d.disposition)],["Reason",R(d.reason)]]);for(const d of t.gpt_diagnostics.attributionIssues??[])w(e,b,[["Creative attribution issue",R(d.reason)],["GPT slot",d.runtimeSlotNumber],["Browser clock",N(d.timestampMs)]]);for(const[d,u]of t.slot_correlations.entries()){const a=_(e,"details");a.append(_(e,"summary",`Correlation record ${d+1}`)),a.append(_(e,"p","Browser observed correlation. These opaque references permit a join only when unique and consistent; duplicate or conflicting records remain unknown.")),w(e,a,[["Auction reference",u.diagnostic_auction_id],["Slot reference",u.slot_ref],["GPT slot number",u.runtime_slot_number],["GPT request number",u.request_number]]),b.append(a)}return s}function ve(e=document,t={}){const n=e.querySelector("main");if(!n)return{destroy(){}};const r=yt(e),s=t.origin??window.location.origin;let i,c=!1;const l=[],o=(g,C,Te)=>{const x=_(e,"button",C);x.type="button";const Re=()=>{c||Te()};return x.addEventListener("click",Re),l.push(()=>x.removeEventListener("click",Re)),g.append(x),x},S=()=>{c=!0,i=void 0,r();for(const g of l)g()};let p;try{p=Tt(s,(t.now??Date.now)(),t.storage)}catch{p={status:"unavailable"}}if(p.status!=="ready"){const g=_(e,"p",p.status==="absent"?"No saved report. Enable tracing, return to the affected page, reload once, reproduce the problem, then select View trace results.":"The saved report is unavailable, expired, or unsupported. Return to the affected page in this same tab and exact hostname, reload once, reproduce the problem, then select View trace results.");return g.id="trace-report-notice",n.prepend(g),{destroy:S}}i=p.value;const b=_(e,"details");b.id="trace-viewer-setup",b.append(_(e,"summary","Setup request and tracing controls"));for(const g of Array.from(n.children))b.append(g);const d=At(e,i.report,s,i.stored_at_ms);n.append(d);const u=L(e,d,"Export");u.append(_(e,"p","Copy, Download and Share use the same public report JSON. The selected app receives this JSON when you choose Share. Nothing is uploaded by this viewer."));const a=_(e,"div");a.className="controls",u.append(a);const f=_(e,"p");f.id="trace-export-status",f.setAttribute("role","status"),f.setAttribute("aria-live","polite"),u.append(f);let h=!1;const U=async g=>{if(!i||c||h)return;h=!0;const C=i;try{const x=await(g==="Copy"?t.copy??ft:g==="Share"?t.share??_t:t.download??pt)(C.report,s,C.stored_at_ms);if(c||i!==C)return;f.textContent=x.status==="copied"?"Copied JSON.":x.status==="downloaded"?"Download started; completion is managed by your browser.":x.status==="shared"?"The share request completed.":g==="Share"&&x.status==="unsupported"?"File sharing is unavailable. Use Copy or Download.":`${g} could not be completed. Your report remains available; retry or choose another export.`}catch{!c&&i===C&&(f.textContent=`${g} could not be completed. Your report remains available.`)}finally{h=!1}};for(const g of["Copy","Download","Share"])o(a,g,()=>{U(g)});const y=L(e,n,"Report cleanup"),z=_(e,"div");z.className="controls",y.append(z);const O=_(e,"p","A local report is saved in this tab.");O.id="trace-cleanup-local-status",O.setAttribute("role","status"),O.setAttribute("aria-live","polite"),O.tabIndex=-1,y.append(O);const B=_(e,"p","Server tracing state has not been changed by this report visit.");B.id="trace-cleanup-server-status",B.setAttribute("role","status"),B.setAttribute("aria-live","polite"),B.tabIndex=-1,y.append(B);const ge=()=>{let g;try{g=Rt(t.storage)}catch{g={status:"unavailable"}}if(!c)if(g.status==="deleted"){const C=e.activeElement===Se||d.contains(e.activeElement);i=void 0,d.remove(),Se.hidden=!0,O.textContent="Local report deleted from this tab.",C&&O.focus()}else O.textContent="Local report deletion failed. The report remains displayed; retry deletion."};let J=!1;const he=async()=>{if(c||J)return;const g=e.activeElement===$;J=!0,ye.disabled=$.disabled=!0,B.textContent="Requesting tracing end and checking the next request…";let C;try{C=await mt()}catch{C={mutation:"failed",observation:"failed",confirmed:!1}}c||(B.textContent=C.confirmed?"Tracing is off — no valid diagnostics session observed.":C.observation==="inactive"?"End tracing unconfirmed. No valid diagnostics session was observed; tracing may remain active. Retry end tracing.":C.observation==="active"?"End tracing unconfirmed — a valid session is still observed. Tracing may remain active. Retry end tracing.":"End tracing unconfirmed. Tracing may remain active. Retry end tracing.",$.hidden=C.confirmed,ye.disabled=$.disabled=!1,C.confirmed&&g&&(e.activeElement===$||e.activeElement===e.body)&&B.focus(),J=!1)},ye=o(z,"Clear report and end tracing",()=>{if(J)return;let g=!1;try{g=(t.confirm??(C=>window.confirm(C)))("Delete the report from this tab and request tracing end?")}catch{}!g||c||(ge(),c||he())}),Se=o(z,"Delete local report",ge),$=o(z,"Retry end tracing",()=>{he()});return $.hidden=!0,n.append(b),{destroy:S}}document.readyState==="loading"?document.addEventListener("DOMContentLoaded",()=>ve(),{once:!0}):ve()})();
diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md
index 18af11050..72474a98a 100644
--- a/docs/guide/integrations/gpt-diagnostics.md
+++ b/docs/guide/integrations/gpt-diagnostics.md
@@ -176,11 +176,25 @@ Use a suitable same-origin deployment where the browser accepts the unchanged
 is 1800 seconds; ordinary requests do not refresh it. Do not weaken cookie attributes
 or infer HTTPS from an untrusted forwarding header to make a test pass.
 
-On deployed Fastly and Spin staging services, verify HTTPS Enable returns a
+On deployed Fastly, Cloudflare and Spin staging services, verify HTTPS Enable returns a
 successful response, a separate state request observes the session, publisher
 reload captures evidence, and End followed by another state request observes
 inactivity. Local plain-HTTP runtime tests do not establish this HTTPS behavior.
 
+Enable and End require the browser's canonical Origin to agree with the
+runtime-provided origin and request authority. A TLS-terminating proxy that
+forwards HTTPS traffic as HTTP, or replaces the public authority with an internal
+host, can therefore cause an intentional `403`, including on Spin and Axum.
+Use a deployment that preserves the public origin in the runtime's trusted
+request metadata and verify both actions through the actual proxy. Untrusted
+`Forwarded` or `X-Forwarded-*` headers cannot repair that mismatch.
+
+The current Axum entry point supplies trusted origin metadata only for its
+plain-HTTP listener. TLS offload does not supply a trusted HTTPS origin, so an
+HTTPS browser session cannot pass Enable/End through that arrangement. Axum
+HTTPS support needs a suitable trusted transport binding before it can meet
+the deployed HTTPS acceptance gate.
+
 Publisher activation and context use injected inline scripts without an attached
 CSP nonce. A publisher nonce/hash policy that does not authorize those scripts
 can prevent diagnostics activation and capture. Check the literal activation gate
diff --git a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
index 6bd0f9c59..5160341a4 100644
--- a/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
+++ b/docs/superpowers/plans/2026-10-05-mobile-ad-render-trace-implementation-plan.md
@@ -1119,6 +1119,74 @@ Bundle splitting remains a separate load-order/performance design. Preserve
 the build-input freshness guard and ordinary reserved-route/cache policy.
 Physical mobile and deployed CDN/HTTPS acceptance remain release gates.
 
+## Latest independent review follow-up — 2026-10-06
+
+Keep the approved report schema, exact truncation order, protected slot floors,
+omission overflow checks and 512 KiB UTF-8 budget. Implement the confirmed
+corrections centrally on the existing branches, then independently review them.
+
+- [x] Select both host-only Node emitted-script regressions explicitly in CI.
+- [x] Replace repeated whole-report measurement with exact byte accounting for
+      removed values, commas, counter digit changes and coverage updates. Retain
+      one final complete measurement and all existing survivor-order assertions.
+- [x] Bind the handoff URL to the captured publisher origin, including pages
+      with an external base element, and recover focus when cleanup hides the
+      focused control without stealing focus from another control.
+- [x] Add operator-facing changelog entries for the default-off trace workflow,
+      bounded diagnostics cookie, reserved paths and private document handling.
+- [x] Include Cloudflare in deployed HTTPS acceptance and document TLS-offload
+      constraints without trusting forwarded scheme/authority headers.
+- [x] Correct EdgeZero's Axum metadata wording and identify earlier probe counts
+      as historical. Preserve URI registered-name syntax, including valid
+      punctuation and trailing dots; a DNS-only policy is not this API's contract.
+- [x] Rebuild and review unpublished trace assets, run the required gates and
+      actual browser workflows, and obtain independent review of the final fixes.
+
+Bundle splitting and optional network enrichment remain separately scoped.
+Upstream merge and repinning to its merged revision remain integration steps;
+this corrective pass does not authorize merging either draft PR.
+
+### Executed corrective-pass evidence
+
+The initial complexity regression failed with 838 complete-wrapper JSON
+serializations. The corrected builder passes with at most two, retaining the
+existing worst-case survivor order, exact omission counters and protected slot
+floors. Additional exact-budget tests cover UTF-8/escaped strings, counter
+digit transitions, array commas, empty arrays, coverage changes and byte-stage
+omission overflow.
+
+Fresh verification after the main merge: all eight target-matched clippy gates,
+four adapter test aliases, the host CLI helper including its browser fixtures,
+21 parity tests, format, core rustdoc, native/WASM builds and both explicitly
+selected Node regressions pass. The normal JS and pinned external Prebid builds,
+1,748 Vitest tests in 69 files, final focused regressions, lint and formatting
+pass. The final asset guard confirms rebuilt bytes and manifest digests; v1 is
+still unpublished. Package-wide TypeScript retains unrelated existing errors,
+with none in the modified trace source/tests; browser TypeScript passes.
+
+The complete browser runner passes Next.js 49 and WordPress 28 tests, including
+a real handoff under an external base and real Delete/Retry focus checks. Its
+first attempt stopped before tests because Docker was inactive; starting Docker
+and retrying resolved that prerequisite. All-four-runtime trace workflows pass
+4/4, and fresh Cloudflare/Fastly/Spin raw boundary suites pass 3/3 using the
+current consumer dependency graph and freshly built artifacts.
+
+Three independent scopes reviewed report performance/byte bounds, remaining
+runtime/UI/CI/operator surfaces, and EdgeZero authority/provenance/docs. Review
+corrections include typed immutable expected-report fixtures, the actual config
+key and explicit Axum plain-HTTP limitations; final rereviews have no remaining
+actionable findings. EdgeZero docs format/lint/build pass under Node 24.12.0;
+its source is unchanged by this follow-up, and all checks pass at docs revision
+`4f221d257e3224c946326b949bb966ac822ae5a3`. Trusted Server keeps the reviewed
+implementation pin until the upstream merge/release handoff.
+
+Hosted Trusted Server checks passed for main-merge head `b7e48c7bc`; the latest
+corrective working tree passed the local gates recorded above. The PR merge
+reference has zero open CodeQL alerts. The earlier Python extraction
+failure cleared after that merge; prior CodeQL paragraphs are historical
+checkpoints. No Python is added by this feature. Physical-device, real browser
+session restoration and deployed HTTPS/CDN/CSP acceptance remain release gates.
+
 ## Phase 4: Optional network enrichment
 
 ### Scheduling boundary

From c052b59ca4d7384bdfb9d0aa0b5084ad0ca9c272 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Thu, 1 Oct 2026 20:34:25 +0530
Subject: [PATCH 136/230] Use web_time::Instant for Cloudflare timing

std::time::Instant::now() panics on the Cloudflare adapter's
wasm32-unknown-unknown target. Switch auction telemetry and the DataDome
IP CIDR source cache to web_time::Instant, which re-exports the standard
type on other targets.

Add a Cloudflare integration test that starts wrangler dev with an
auction-enabled config and posts to /auction. The request passes the
consent gate and completes through the orchestrator, so it creates an
AuctionObservationContext and its web_time::Instant in workerd. The
config drops the fixture's providers so the auction makes no outbound
calls. With std::time::Instant restored, the worker panics with "time
not implemented on this platform" and the test fails.

The config is built per test from the shared fixture, so the Fastly,
Axum, and other Cloudflare tests keep auctions disabled. The DataDome
IP CIDR source is not covered: only the Fastly adapter runs request
filters, so Cloudflare never evaluates the DataDome protection scope.

Closes #1075

Signed-off-by: dhruv8sh 
---
 .../.gitignore                                |  1 +
 .../src/auction/telemetry.rs                  |  2 +-
 .../integrations/datadome/protection_scope.rs |  3 +-
 .../tests/common/config.rs                    | 50 ++++++++++++++++-
 .../tests/environments/cloudflare.rs          | 47 +++++++++++-----
 .../tests/integration.rs                      | 54 +++++++++++++++++++
 6 files changed, 139 insertions(+), 18 deletions(-)

diff --git a/crates/trusted-server-adapter-cloudflare/.gitignore b/crates/trusted-server-adapter-cloudflare/.gitignore
index e3df6a859..be872c72e 100644
--- a/crates/trusted-server-adapter-cloudflare/.gitignore
+++ b/crates/trusted-server-adapter-cloudflare/.gitignore
@@ -5,3 +5,4 @@ build/
 .worker-build/
 .env.cloudflare.dev
 .dev.vars
+wrangler.integration.generated.toml
diff --git a/crates/trusted-server-core/src/auction/telemetry.rs b/crates/trusted-server-core/src/auction/telemetry.rs
index c6e37d02e..ab7de3c32 100644
--- a/crates/trusted-server-core/src/auction/telemetry.rs
+++ b/crates/trusted-server-core/src/auction/telemetry.rs
@@ -4,12 +4,12 @@
 //! adapters provide the concrete sink implementation.
 
 use std::collections::HashSet;
-use std::time::Instant;
 
 use chrono::Utc;
 use error_stack::Report;
 use serde::Serialize;
 use uuid::Uuid;
+use web_time::Instant;
 
 use crate::auction::orchestrator::OrchestrationResult;
 use crate::auction::types::{AuctionRequest, AuctionResponse, Bid, BidStatus, MediaType};
diff --git a/crates/trusted-server-core/src/integrations/datadome/protection_scope.rs b/crates/trusted-server-core/src/integrations/datadome/protection_scope.rs
index a83f73217..1d3646063 100644
--- a/crates/trusted-server-core/src/integrations/datadome/protection_scope.rs
+++ b/crates/trusted-server-core/src/integrations/datadome/protection_scope.rs
@@ -2,11 +2,12 @@ use std::collections::{HashMap, HashSet};
 use std::net::IpAddr;
 use std::str::FromStr;
 use std::sync::{LazyLock, Mutex};
-use std::time::{Duration, Instant};
+use std::time::Duration;
 
 use error_stack::Report;
 use regex::Regex;
 use serde::Deserialize;
+use web_time::Instant;
 
 use crate::error::TrustedServerError;
 use crate::platform::{RuntimeServices, StoreName};
diff --git a/crates/trusted-server-integration-tests/tests/common/config.rs b/crates/trusted-server-integration-tests/tests/common/config.rs
index 3bcbfe53e..0b793833f 100644
--- a/crates/trusted-server-integration-tests/tests/common/config.rs
+++ b/crates/trusted-server-integration-tests/tests/common/config.rs
@@ -9,12 +9,28 @@ const GENERATED_AT: &str = "2026-06-23T00:00:00Z";
 const APP_CONFIG: &str = include_str!("../../fixtures/configs/trusted-server.integration.toml");
 
 pub fn integration_app_config_envelope(origin_port: u16) -> TestResult {
+    app_config_envelope(origin_port, |_| Ok(()))
+}
+
+/// Builds the integration app-config envelope after `customize` edits the
+/// parsed fixture.
+fn app_config_envelope(
+    origin_port: u16,
+    customize: impl FnOnce(&mut toml::Table) -> TestResult<()>,
+) -> TestResult {
     let origin_url = format!("http://127.0.0.1:{origin_port}");
-    let app_config: TrustedServerAppConfig = toml::from_str(APP_CONFIG).map_err(|error| {
+    let mut table: toml::Table = toml::from_str(APP_CONFIG).map_err(|error| {
         Report::new(TestError::ConfigGeneration).attach(format!(
             "invalid Trusted Server integration config: {error}"
         ))
     })?;
+    customize(&mut table)?;
+    let app_config: TrustedServerAppConfig =
+        toml::Value::Table(table).try_into().map_err(|error| {
+            Report::new(TestError::ConfigGeneration).attach(format!(
+                "invalid customized Trusted Server integration config: {error}"
+            ))
+        })?;
     let mut settings = app_config.into_settings();
     settings.publisher.origin_url = origin_url;
     let app_config = TrustedServerAppConfig::new(settings).map_err(|report| {
@@ -36,7 +52,37 @@ pub fn integration_app_config_envelope(origin_port: u16) -> TestResult {
 
 pub fn cloudflare_config_json(origin_port: u16) -> TestResult {
     let envelope = integration_app_config_envelope(origin_port)?;
-    serde_json::to_string(&serde_json::json!({ CONFIG_BLOB_KEY: envelope })).map_err(|error| {
+    cloudflare_binding_json(&serde_json::json!({ CONFIG_BLOB_KEY: envelope }))
+}
+
+/// Builds a Cloudflare config binding with server-side auctions enabled.
+///
+/// The fixture's providers are removed so the enabled auction completes
+/// through the orchestrator and telemetry without any outbound provider call.
+pub fn cloudflare_auction_config_json(origin_port: u16) -> TestResult {
+    let envelope = app_config_envelope(origin_port, |table| {
+        let auction = table_mut(table, "auction")?;
+        auction.insert("enabled".to_string(), toml::Value::Boolean(true));
+        auction.remove("providers");
+        auction.remove("bidders");
+        Ok(())
+    })?;
+    cloudflare_binding_json(&serde_json::json!({ CONFIG_BLOB_KEY: envelope }))
+}
+
+fn table_mut<'a>(table: &'a mut toml::Table, key: &str) -> TestResult<&'a mut toml::Table> {
+    table
+        .get_mut(key)
+        .and_then(toml::Value::as_table_mut)
+        .ok_or_else(|| {
+            Report::new(TestError::ConfigGeneration).attach(format!(
+                "integration config should contain a `[{key}]` table"
+            ))
+        })
+}
+
+fn cloudflare_binding_json(binding: &serde_json::Value) -> TestResult {
+    serde_json::to_string(binding).map_err(|error| {
         Report::new(TestError::ConfigGeneration).attach(format!(
             "failed to serialize Cloudflare config binding: {error}"
         ))
diff --git a/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs b/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs
index 10d16c0a7..72328f42f 100644
--- a/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs
+++ b/crates/trusted-server-integration-tests/tests/environments/cloudflare.rs
@@ -27,7 +27,7 @@ const CI_CONFIG_TEMPLATE: &str = "wrangler.ci.toml";
 const GENERATED_CI_CONFIG: &str = "wrangler.integration.generated.toml";
 const TRUSTED_SERVER_CONFIG_PLACEHOLDER: &str = "TRUSTED_SERVER_CONFIG = \"{}\"";
 
-fn write_generated_ci_config(wrangler_dir: &Path) -> TestResult {
+fn write_generated_ci_config(wrangler_dir: &Path, config_json: &str) -> TestResult {
     let template_path = wrangler_dir.join(CI_CONFIG_TEMPLATE);
     let template = std::fs::read_to_string(&template_path)
         .change_context(TestError::RuntimeSpawn)
@@ -35,8 +35,7 @@ fn write_generated_ci_config(wrangler_dir: &Path) -> TestResult {
             "failed to read Cloudflare CI wrangler config at {}",
             template_path.display()
         ))?;
-    let config_json = cloudflare_config_json(origin_port())?;
-    let generated = inject_cloudflare_config(&template, &config_json)?;
+    let generated = inject_cloudflare_config(&template, config_json)?;
     let output_path = wrangler_dir.join(GENERATED_CI_CONFIG);
     std::fs::write(&output_path, generated)
         .change_context(TestError::RuntimeSpawn)
@@ -69,11 +68,37 @@ impl RuntimeEnvironment for CloudflareWorkers {
     fn spawn(&self, _wasm_path: &Path) -> TestResult {
         let wrangler_dir = self.wrangler_dir();
         let config = if std::env::var("CI").is_ok() {
-            write_generated_ci_config(&wrangler_dir)?
+            write_generated_ci_config(&wrangler_dir, &cloudflare_config_json(origin_port())?)?
         } else {
             "wrangler.toml".to_string()
         };
+        self.spawn_wrangler(&wrangler_dir, &config)
+    }
+
+    fn health_check_path(&self) -> &str {
+        "/.well-known/trusted-server.json"
+    }
+}
 
+impl CloudflareWorkers {
+    /// Start `wrangler dev` with a test-specific `TRUSTED_SERVER_CONFIG` binding.
+    ///
+    /// Unlike [`RuntimeEnvironment::spawn`], this always renders the CI
+    /// wrangler template, so tests can enable features the shared fixture
+    /// leaves off without changing the other runtimes' configuration.
+    ///
+    /// # Errors
+    ///
+    /// Returns [`TestError::RuntimeSpawn`] if the wrangler config cannot be
+    /// written or `wrangler dev` cannot be started, and
+    /// [`TestError::RuntimeNotReady`] if the health check times out.
+    pub fn spawn_with_config_json(&self, config_json: &str) -> TestResult {
+        let wrangler_dir = self.wrangler_dir();
+        let config = write_generated_ci_config(&wrangler_dir, config_json)?;
+        self.spawn_wrangler(&wrangler_dir, &config)
+    }
+
+    fn spawn_wrangler(&self, wrangler_dir: &Path, config: &str) -> TestResult {
         let port = super::find_available_port().unwrap_or(CLOUDFLARE_DEFAULT_PORT);
 
         #[cfg(unix)]
@@ -83,13 +108,13 @@ impl RuntimeEnvironment for CloudflareWorkers {
                 .args([
                     "dev",
                     "--config",
-                    config.as_str(),
+                    config,
                     "--port",
                     &port.to_string(),
                     "--ip",
                     "127.0.0.1",
                 ])
-                .current_dir(&wrangler_dir)
+                .current_dir(wrangler_dir)
                 .stdout(Stdio::null())
                 .stderr(Stdio::piped())
                 .process_group(0)
@@ -108,13 +133,13 @@ impl RuntimeEnvironment for CloudflareWorkers {
             .args([
                 "dev",
                 "--config",
-                config.as_str(),
+                config,
                 "--port",
                 &port.to_string(),
                 "--ip",
                 "127.0.0.1",
             ])
-            .current_dir(&wrangler_dir)
+            .current_dir(wrangler_dir)
             .stdout(Stdio::null())
             .stderr(Stdio::piped())
             .spawn()
@@ -149,12 +174,6 @@ impl RuntimeEnvironment for CloudflareWorkers {
         })
     }
 
-    fn health_check_path(&self) -> &str {
-        "/.well-known/trusted-server.json"
-    }
-}
-
-impl CloudflareWorkers {
     /// Resolve the Cloudflare adapter crate root.
     ///
     /// Respects `CLOUDFLARE_WRANGLER_DIR` for CI overrides; falls back to
diff --git a/crates/trusted-server-integration-tests/tests/integration.rs b/crates/trusted-server-integration-tests/tests/integration.rs
index ebd026410..ad8249307 100644
--- a/crates/trusted-server-integration-tests/tests/integration.rs
+++ b/crates/trusted-server-integration-tests/tests/integration.rs
@@ -214,6 +214,60 @@ fn test_cloudflare_dynamic_publisher_response_does_not_cross_cookie_boundaries()
     );
 }
 
+/// Exercises an enabled `/auction` request through auction telemetry in workerd.
+///
+/// `std::time::Instant::now()` traps on Cloudflare's `wasm32-unknown-unknown`
+/// target, so this guards the `web_time::Instant` timing in
+/// `AuctionObservationContext`. The auction config has no providers, so the
+/// enabled auction completes as a no-bid without outbound calls.
+#[test]
+#[ignore = "requires the `wrangler` CLI in $PATH and a prebuilt Cloudflare Workers bundle (run build.sh first); the test starts wrangler dev automatically"]
+fn test_cloudflare_enabled_auction_reaches_telemetry() {
+    init_logger();
+    // The readiness probe falls back to the proxied root page.
+    let _origin = common::ec::MinimalOrigin::start(origin_port());
+    let config_json = common::config::cloudflare_auction_config_json(origin_port())
+        .expect("should build auction-enabled Cloudflare config");
+    let process = environments::cloudflare::CloudflareWorkers
+        .spawn_with_config_json(&config_json)
+        .expect("should start Cloudflare Worker");
+    let request_body = serde_json::json!({
+        "adUnits": [{
+            "code": "integration-slot",
+            "mediaTypes": { "banner": { "sizes": [[300, 250]] } }
+        }]
+    });
+
+    // A non-regulated geo passes the server-side auction consent gate, so the
+    // request runs the enabled orchestration path instead of a consent skip.
+    let response = reqwest::blocking::Client::new()
+        .post(format!("{}/auction", process.base_url))
+        .header("cf-ipcountry", "US")
+        .json(&request_body)
+        .send()
+        .expect("should send auction request");
+
+    let status = response.status().as_u16();
+    let body = response.text().expect("should read auction response");
+    assert_eq!(
+        status, 200,
+        "enabled auction should complete without a worker exception: {body}"
+    );
+    let auction_response: serde_json::Value =
+        serde_json::from_str(&body).expect("should return an OpenRTB JSON response");
+    assert!(
+        auction_response["id"]
+            .as_str()
+            .is_some_and(|id| !id.is_empty()),
+        "auction response should carry the auction id: {body}"
+    );
+    assert_eq!(
+        auction_response["ext"]["orchestrator"]["providers"],
+        serde_json::json!(0),
+        "enabled auction without providers should complete as a no-bid: {body}"
+    );
+}
+
 #[test]
 #[ignore = "requires Docker and pre-built trusted-server-axum binary"]
 fn test_wordpress_axum() {

From 22268364250467cba708529c1d640f481381abde Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Tue, 6 Oct 2026 18:11:02 +0530
Subject: [PATCH 137/230] Use web_time clocks in publisher and lint core for
 wasm-unsafe clocks

The template cache and origin_shared_ttl in publisher.rs still called std::time::Instant::now and SystemTime::now, which panic on wasm32-unknown-unknown. Move them to web_time. origin_shared_ttl_at keeps taking a std SystemTime because httpdate parses into that type, so build it from UNIX_EPOCH plus the web_time elapsed duration.

Add scripts/lint-wasm-clock.sh, a clippy pass on wasm32-unknown-unknown with a dedicated config that denies std::time::Instant::now and std::time::SystemTime::now in trusted-server-core. It runs on that target only because elsewhere web_time re-exports the std types, so a workspace-wide rule would flag the correct web_time calls. Run it in CI and list it under the AGENTS.md CI gates.

Signed-off-by: dhruv8sh 
---
 .github/workflows/format.yml                |  3 +++
 AGENTS.md                                   |  1 +
 crates/trusted-server-core/src/publisher.rs | 13 +++++++++++--
 scripts/clippy-wasm-clock/clippy.toml       |  6 ++++++
 scripts/lint-wasm-clock.sh                  | 17 +++++++++++++++++
 5 files changed, 38 insertions(+), 2 deletions(-)
 create mode 100644 scripts/clippy-wasm-clock/clippy.toml
 create mode 100755 scripts/lint-wasm-clock.sh

diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml
index 93bc99229..521331ed7 100644
--- a/.github/workflows/format.yml
+++ b/.github/workflows/format.yml
@@ -73,6 +73,9 @@ jobs:
       - name: Run cargo clippy (Cloudflare — wasm32-unknown-unknown)
         run: cargo clippy-cloudflare-wasm
 
+      - name: Check wasm-safe clock usage in core (wasm32-unknown-unknown)
+        run: scripts/lint-wasm-clock.sh
+
       - name: Run cargo clippy (Spin — native)
         run: cargo clippy-spin-native
 
diff --git a/AGENTS.md b/AGENTS.md
index ee2ad3fd0..3c0895275 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -348,6 +348,7 @@ Every PR must pass:
 
 1. `cargo fmt --all -- --check`
 2. `cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasm && cargo clippy-cli && cargo clippy-codegen`
+   Also `scripts/lint-wasm-clock.sh` (core must use `web_time`, not `std::time::{Instant,SystemTime}::now`, which panic on Cloudflare).
 3. `cargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spin`
 4. `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity`
 5. JS build and test (`cd crates/trusted-server-js/lib && npx vitest run`)
diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs
index 61bc7efe5..01fb04daa 100644
--- a/crates/trusted-server-core/src/publisher.rs
+++ b/crates/trusted-server-core/src/publisher.rs
@@ -22,7 +22,7 @@ use std::borrow::Cow;
 use std::io::Write;
 use std::sync::atomic::{AtomicBool, Ordering};
 use std::sync::{Arc, Mutex};
-use std::time::{Duration, Instant, SystemTime};
+use std::time::{Duration, SystemTime};
 
 use brotli::Decompressor;
 use brotli::enc::BrotliEncoderParams;
@@ -34,6 +34,7 @@ use flate2::read::ZlibDecoder;
 use flate2::write::{GzEncoder, ZlibEncoder};
 use futures::StreamExt as _;
 use http::{HeaderValue, Method, Request, Response, StatusCode, Uri, header};
+use web_time::Instant;
 
 use crate::auction::endpoints::{
     merge_auction_eids, resolve_auction_eids, resolve_client_auction_eids,
@@ -6462,7 +6463,15 @@ fn origin_shared_ttl(
     headers: &edgezero_core::http::HeaderMap,
     max_age: Duration,
 ) -> Result {
-    origin_shared_ttl_at(headers, SystemTime::now(), max_age)
+    // `httpdate` parses `Date`/`Expires` into `std::time::SystemTime`, but
+    // `std::time::SystemTime::now()` panics on `wasm32-unknown-unknown`
+    // (Cloudflare Workers). Build the equivalent `std` value from the wasm-safe
+    // `web_time` clock: `UNIX_EPOCH + elapsed` never calls the panicking `now()`.
+    let now = SystemTime::UNIX_EPOCH
+        + web_time::SystemTime::now()
+            .duration_since(web_time::UNIX_EPOCH)
+            .unwrap_or_default();
+    origin_shared_ttl_at(headers, now, max_age)
 }
 
 fn origin_shared_ttl_at(
diff --git a/scripts/clippy-wasm-clock/clippy.toml b/scripts/clippy-wasm-clock/clippy.toml
new file mode 100644
index 000000000..eed3f09e5
--- /dev/null
+++ b/scripts/clippy-wasm-clock/clippy.toml
@@ -0,0 +1,6 @@
+# Dedicated clippy config for scripts/lint-wasm-clock.sh. See that script for
+# why this rule cannot live in the workspace-wide clippy.toml.
+disallowed-methods = [
+    { path = "std::time::Instant::now", reason = "panics on wasm32-unknown-unknown; use `web_time::Instant::now`" },
+    { path = "std::time::SystemTime::now", reason = "panics on wasm32-unknown-unknown; use `web_time::SystemTime::now`" },
+]
diff --git a/scripts/lint-wasm-clock.sh b/scripts/lint-wasm-clock.sh
new file mode 100755
index 000000000..2efc4a223
--- /dev/null
+++ b/scripts/lint-wasm-clock.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Fail if trusted-server-core calls std::time::Instant::now or
+# std::time::SystemTime::now. Both panic with `RuntimeError: unreachable` on
+# wasm32-unknown-unknown (Cloudflare Workers); use `web_time` instead.
+#
+# This cannot live in the workspace clippy.toml: on native and wasm32-wasip1,
+# `web_time::Instant` and `web_time::SystemTime` are re-exports of the std
+# types, so clippy would flag the compliant `web_time` calls too. On
+# wasm32-unknown-unknown the types are distinct, so only that target can tell
+# them apart. The dedicated config dir holds just the disallowed-methods rule.
+set -euo pipefail
+
+root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+CLIPPY_CONF_DIR="$root/scripts/clippy-wasm-clock" \
+    cargo clippy -p trusted-server-core --lib --target wasm32-unknown-unknown -- \
+    -A clippy::all -D clippy::disallowed_methods

From 3631a82932744c49e0e2e001da691171aec9a28f Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 14:48:36 +0530
Subject: [PATCH 138/230] Add design spec and plan for separate creative click
 rewrite switch

Spec and implementation plan for #1234: an [auction] rewrite_clicks switch that follows rewrite_creatives when unset, so click wrapping can be controlled independently of asset rewriting.

Signed-off-by: dhruv8sh 
---
 ...0-07-1234-creative-click-rewrite-switch.md | 1710 +++++++++++++++++
 ...34-creative-click-rewrite-switch-design.md |  511 +++++
 2 files changed, 2221 insertions(+)
 create mode 100644 docs/superpowers/plans/2026-10-07-1234-creative-click-rewrite-switch.md
 create mode 100644 docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md

diff --git a/docs/superpowers/plans/2026-10-07-1234-creative-click-rewrite-switch.md b/docs/superpowers/plans/2026-10-07-1234-creative-click-rewrite-switch.md
new file mode 100644
index 000000000..03f83b802
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-07-1234-creative-click-rewrite-switch.md
@@ -0,0 +1,1710 @@
+# Creative Click Rewrite Switch Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add `[auction] rewrite_clicks` so creative click wrapping (`` / `` → signed `/first-party/click`) can be turned on or off independently of asset rewriting, without changing output for any existing config.
+
+**Architecture:**
+
+- Replace `to_abs` with a pure URL normalizer plus host policy on `Rewrite`. This "shared step" is written identically in #1231's PR.
+- Add an `Option` field on `AuctionConfig` with two resolution methods.
+- Give the lol_html rewrite pass a `CreativeFeatures { assets, clicks }` parameter, and register the asset and anchor handlers only when their feature is on.
+- `` removal and TSJS injection run whenever the pass runs.
+
+**Tech Stack:** Rust 2024 (`trusted-server-core`, `trusted-server-cli`), `lol_html`, `url`, `serde`, EdgeZero typed app-config loader, Viceroy for `wasm32-wasip1` tests.
+
+**Spec:** `docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md` (issue [#1234](https://github.com/IABTechLab/trusted-server/issues/1234)). Read it first.
+
+**Status:** Validated by an exploratory implementation on 2026-10-07. Every code block below is the code that compiled, passed clippy and passed its tests. Each step's "Expected" line is the observed result.
+
+## Global Constraints
+
+- With `rewrite_clicks` unset, rewritten output must be byte-identical to today for both values of `rewrite_creatives` on `/auction` and inline SSAT/page-bids, and unchanged for `/first-party/proxy` HTML.
+- `rewrite_clicks` is `Option` with `#[serde(default, skip_serializing_if = "Option::is_none")]`. `AuctionConfig` keeps `deny_unknown_fields`.
+- Unset resolution: auction paths use `rewrite_clicks.unwrap_or(rewrite_creatives)`, and proxied HTML uses `rewrite_clicks.unwrap_or(true)`.
+- `exclude_domains` applies to anchors. The #1231 include list never does.
+- The shared matcher is `crate::proxy::is_host_allowed` (`proxy.rs:1242`), which is case-insensitive.
+- No JavaScript, adapter, routing or dependency changes. `/first-party/click` and `/first-party/proxy-rebuild` stay routed.
+- AGENTS.md rules apply:
+  - example.com-style domains only;
+  - `expect("should ...")` messages;
+  - comments on their own line;
+  - no local `use` inside functions;
+  - every public item documented;
+  - at most 7 function arguments.
+- Commits use sentence case, imperative mood, and `git commit --signoff -S` with multiple `-m` flags. No prefixes, no attribution trailers, never a heredoc.
+
+## Environment notes (learned during validation)
+
+- `cargo test-fastly` runs the wasm test binary through Viceroy. If `viceroy` is installed under `~/.cargo/bin` but not on `PATH`, prefix commands with `PATH=$HOME/.cargo/bin:$PATH`. Validation used Viceroy 0.21.1; `.tool-versions` pins 0.17.0.
+- Under Viceroy, one panicking test aborts the whole wasm test binary (exit 134) and hides the assertion message. To watch a test fail, run it alone:
+
+  ```bash
+  cargo test-fastly -p trusted-server-core --lib -- --exact  --nocapture
+  ```
+
+  Read the `panicked at` line.
+
+- A test that only adds a new symbol fails at compile time (`E0425`/`E0599`/`E0609`). That is the red step for API-shaped tests. Tests that only use existing APIs must be run against the old code to see a runtime red.
+- The worktree may have no `docs/node_modules`. Use a checkout that has one:
+
+  ```bash
+  /path/to/checkout/docs/node_modules/.bin/prettier --config docs/.prettierrc ...
+  ```
+
+  `cd docs && npm run format` (check mode) still works.
+
+## File Map
+
+| File                                                                                  | Responsibility in this change                                                                                                               |
+| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
+| `crates/trusted-server-core/src/creative.rs`                                          | `normalize_creative_url`, `asset_target`/`click_target`, `CreativeFeatures`, conditional handler registration, caller wiring, matrix tests. |
+| `crates/trusted-server-core/src/settings.rs`                                          | `Rewrite::should_proxy_asset` / `should_wrap_click` / `normalize`, removal of `is_excluded`, TOML tests.                                    |
+| `crates/trusted-server-core/src/proxy.rs`                                             | `/first-party/sign` call-site migration, sign and proxied-HTML tests.                                                                       |
+| `crates/trusted-server-core/src/auction_config_types.rs`                              | `rewrite_clicks`, `rewrites_auction_clicks` / `rewrites_proxied_clicks`, serialization tests.                                               |
+| `crates/trusted-server-core/src/config_payload.rs`                                    | Blob round-trip tests.                                                                                                                      |
+| `crates/trusted-server-core/src/auction/orchestrator.rs`                              | `rewrite_clicks: None` in the one exhaustive `AuctionConfig` literal.                                                                       |
+| `crates/trusted-server-core/src/auction/formats.rs`                                   | Doc comment, debug log, `/auction` end-to-end test.                                                                                         |
+| `crates/trusted-server-core/src/publisher.rs`                                         | Inline end-to-end test.                                                                                                                     |
+| `crates/trusted-server-core/src/auction/endpoints.rs`                                 | Response doc comment.                                                                                                                       |
+| `crates/trusted-server-cli/tests/config_env_overlay.rs`                               | Overlay tests for `TRUSTED_SERVER__AUCTION__REWRITE_CLICKS`.                                                                                |
+| `trusted-server.example.toml`                                                         | Reworded `rewrite_creatives` comment and commented-out `rewrite_clicks` leaf.                                                               |
+| `docs/guide/*.md`, `crates/trusted-server-core/src/auction/README.md`, `CHANGELOG.md` | Operator documentation.                                                                                                                     |
+
+## Shared step
+
+This section is identical in the #1231 and #1234 plans. Both PRs write this production code and this test set verbatim; whichever PR lands first ships them, and the other skips its shared-step task. `proxy::is_host_permitted` stays private in the shared step: #1231 makes it `pub(crate)` in the task that adds the include clause, and #1234 never needs it.
+
+`crates/trusted-server-core/src/settings.rs`, imports (`:26`). The shared step uses only `is_host_allowed`, so it imports only that; importing `is_host_permitted` here would fail `-D warnings` as an unused import. #1231's include-clause task widens the import.
+
+```rust
+use crate::proxy::is_host_allowed;
+```
+
+`settings.rs`, replaces `impl Rewrite` (`:645-670`), removing `is_excluded` and its `#[allow(dead_code)]`:
+
+```rust
+impl Rewrite {
+    /// Returns `true` when an asset URL on `host` should be rewritten to
+    /// `/first-party/proxy`.
+    ///
+    /// The host must not match [`Self::exclude_domains`]. Matching is
+    /// case-insensitive; see [`is_host_allowed`] for the pattern rules.
+    #[must_use]
+    pub fn should_proxy_asset(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
+
+    /// Returns `true` when a click-through URL on `host` should be wrapped in
+    /// `/first-party/click`.
+    ///
+    /// Only [`Self::exclude_domains`] applies to click-through links.
+    #[must_use]
+    pub fn should_wrap_click(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
+
+    fn is_excluded_host(&self, host: &str) -> bool {
+        self.exclude_domains
+            .iter()
+            .any(|pattern| is_host_allowed(host, pattern))
+    }
+
+    /// Trims and lowercases host patterns in place.
+    ///
+    /// Empty and bare `*` entries in `exclude_domains` are dropped with a
+    /// warning: neither can match a host, so dropping them changes no behavior.
+    fn normalize(&mut self) {
+        let before = self.exclude_domains.len();
+        self.exclude_domains = self
+            .exclude_domains
+            .iter()
+            .map(|pattern| pattern.trim().to_ascii_lowercase())
+            .filter(|pattern| !pattern.is_empty() && pattern != "*")
+            .collect();
+        if self.exclude_domains.len() < before {
+            log::warn!(
+                "rewrite.exclude_domains: removed empty or bare \"*\" entries, which never match a host"
+            );
+        }
+    }
+}
+```
+
+`settings.rs`, in `Settings::normalize_deserialized` (`:2988`), right after `self.proxy.normalize();`:
+
+```rust
+        self.rewrite.normalize();
+```
+
+`crates/trusted-server-core/src/creative.rs`, replaces `to_abs` and its comment (`:52-75`):
+
+```rust
+/// Normalizes a creative URL to an absolute HTTP(S) [`url::Url`].
+///
+/// Trims surrounding whitespace, resolves a protocol-relative `//host/...`
+/// against `https:`, and accepts only `http://` and `https://` input (ASCII
+/// case-insensitive). Returns `None` for empty, relative, non-network-scheme or
+/// unparseable input. Applies no host policy; see
+/// [`crate::settings::Rewrite::should_proxy_asset`] and
+/// [`crate::settings::Rewrite::should_wrap_click`].
+pub(super) fn normalize_creative_url(url: &str) -> Option {
+    let trimmed = url.trim();
+    let is_http = trimmed
+        .get(..7)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("http://"));
+    let is_https = trimmed
+        .get(..8)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("https://"));
+    if trimmed.starts_with("//") {
+        url::Url::parse(&format!("https:{trimmed}")).ok()
+    } else if is_http || is_https {
+        url::Url::parse(trimmed).ok()
+    } else {
+        None
+    }
+}
+
+/// Normalizes `raw` and returns it when asset policy allows proxying its host.
+fn asset_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_proxy_asset(host))
+    })
+}
+
+/// Normalizes `raw` and returns it when click policy allows wrapping its host.
+fn click_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_wrap_click(host))
+    })
+}
+```
+
+`creative.rs` call sites (`:402`, `:409`, `:639-641`, `:726-727`, `:1285-1287`):
+
+```rust
+        // CssUrlRewriter::rewrite
+        let Some(target) = asset_target(self.settings, value) else {
+            return;
+        };
+        // ... later in the same function:
+            let proxied = build_proxy_url(self.settings, target.as_str(), self.base_origin);
+
+// proxy_if_abs
+pub(super) fn proxy_if_abs(settings: &Settings, val: &str, base_origin: &str) -> Option {
+    asset_target(settings, val).map(|url| build_proxy_url(settings, url.as_str(), base_origin))
+}
+
+        // rewrite_srcset
+        let rewritten = if let Some(target) = asset_target(settings, url) {
+            build_proxy_url(settings, target.as_str(), base_origin)
+        } else {
+            url.to_owned()
+        };
+
+                // Anchor handler
+                element!("a[href], area[href]", |el| {
+                    if let Some(href) = el.get_attribute("href")
+                        && let Some(target) = click_target(settings, &href)
+                    {
+                        let click = build_click_url(settings, target.as_str(), base_origin);
+                        let _ = el.set_attribute("href", &click);
+                        let _ = el.set_attribute("data-tsclick", &click);
+                    }
+                    Ok(())
+                }),
+```
+
+`creative.rs` module doc (`:25-27`):
+
+```rust
+//! - `normalize_creative_url(&str) -> Option`: Normalizes an absolute or
+//!   protocol-relative http(s) URL; returns `None` for relative input,
+//!   non-network schemes and unparseable values. Host policy lives on
+//!   [`crate::settings::Rewrite`].
+```
+
+`proxy.rs`, `handle_first_party_proxy_sign` (`:1625`): replace `:1672-1703` (from `let trimmed` up to the `is_host_permitted` check) with the block below, then at `:1723` and `:1726` use `target` in place of `parsed` and `target.as_str()` in place of `&abs`. The old parse, scheme and duplicate exclusion checks go away because the normalizer already guarantees an `http(s)` URL.
+
+```rust
+    let trimmed = payload.url.trim();
+    let protocol_relative;
+    // A protocol-relative target inherits the signing request's scheme before
+    // normalization, so `//` and absolute input reach the same policy check.
+    let candidate = if trimmed.starts_with("//") {
+        protocol_relative = format!("{request_scheme}:{trimmed}");
+        protocol_relative.as_str()
+    } else {
+        trimmed
+    };
+    let target = crate::creative::normalize_creative_url(candidate).ok_or_else(|| {
+        Report::new(TrustedServerError::Proxy {
+            message: "unsupported url".to_string(),
+        })
+    })?;
+    let host = target.host_str().ok_or_else(|| {
+        Report::new(TrustedServerError::Proxy {
+            message: "missing host".to_string(),
+        })
+    })?;
+    if !settings.rewrite.should_proxy_asset(host) {
+        log::debug!("sign request for `{host}` declined by rewrite policy");
+        return Err(Report::new(TrustedServerError::Proxy {
+            message: "unsupported url".to_string(),
+        }));
+    }
+```
+
+```rust
+    let mut base = target.clone();
+    base.set_query(None);
+    base.set_fragment(None);
+    let proxied = crate::creative::build_proxy_url_with_extras(settings, target.as_str(), &extras);
+```
+
+### Shared tests
+
+Every shared test builds `Rewrite` with `Rewrite::default()` and `.extend`/`.push` on its lists, or sets `settings.rewrite.exclude_domains` on a full `Settings`. None uses a `Rewrite { .. }` literal or `#[allow(clippy::field_reassign_with_default)]`. The code for each test is in the task that adds it in each plan.
+
+| File          | Test                                                                 | Replaces or placement                                                                                    |
+| ------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `settings.rs` | `rewrite_policy_matches_exclude_patterns_case_insensitively`         | replaces `test_rewrite_is_excluded`                                                                      |
+| `settings.rs` | `rewrite_normalize_trims_lowercases_and_drops_inert_exclude_entries` | after the previous test                                                                                  |
+| `settings.rs` | `rewrite_exclude_domains_match_mixed_case_entries_from_toml`         | after the previous test                                                                                  |
+| `settings.rs` | `settings_load_normalizes_rewrite_exclude_domains`                   | after the previous test                                                                                  |
+| `creative.rs` | `normalized` helper and `normalize_creative_url_conversions`         | replace `to_abs_conversions`, `to_abs_preserves_port_in_protocol_relative` and `to_abs_additional_cases` |
+| `creative.rs` | `proxy_if_abs_respects_exclude_domains`                              | replaces `to_abs_respects_exclude_domains` and `to_abs_respects_wildcard_domains`                        |
+| `creative.rs` | `unparseable_absolute_url_is_left_byte_identical`                    | after the previous test                                                                                  |
+| `creative.rs` | `unparseable_absolute_click_url_is_left_byte_identical`              | after the previous test                                                                                  |
+| `creative.rs` | `exclude_domains_match_case_insensitively_in_the_rewrite_pass`       | after the previous test                                                                                  |
+| `proxy.rs`    | `proxy_sign_rejects_excluded_urls`                                   | rewritten in place: GET and POST, plus an uppercase target host                                          |
+| `proxy.rs`    | `proxy_sign_rejects_excluded_urls_case_insensitively`                | after the previous test: mixed-case config entry, absolute and `//` input, GET and POST                  |
+
+The `creative.rs` test module's `use super::{...}` list gains `normalize_creative_url` and `proxy_if_abs` and loses `to_abs`.
+
+CHANGELOG: the shared step's entry goes under `## [Unreleased]` › `### Fixed`.
+
+---
+
+### Task 0: Pre-flight, check whether #1231 already landed the shared step
+
+**Files:** none modified.
+
+- [ ] **Step 1: Check main**
+
+```bash
+git fetch origin
+git grep -n 'fn normalize_creative_url' origin/main -- crates/trusted-server-core/src/creative.rs
+git grep -n 'fn should_wrap_click' origin/main -- crates/trusted-server-core/src/settings.rs
+```
+
+- [ ] **Step 2: Decide**
+
+- Both greps print nothing: run **Task 1**.
+- Both print matches: `git rebase origin/main`, **skip Task 1**, and confirm that `click_target` exists with `git grep -n 'fn click_target' -- crates/trusted-server-core/src/creative.rs`. It is part of the shared step, so it should. Note in the PR description that the shared step came from #1231.
+
+---
+
+### Task 1: Shared step (skip if Task 0 found it on main)
+
+This task writes the [Shared step](#shared-step) code and tests verbatim. They are identical to #1231's Tasks 1–2.
+
+**Files:**
+
+- Modify: `crates/trusted-server-core/src/settings.rs` (import, `impl Rewrite`, `normalize_deserialized`, tests)
+- Modify: `crates/trusted-server-core/src/creative.rs` (module doc, `to_abs`, `CssUrlRewriter::rewrite`, `proxy_if_abs`, `rewrite_srcset`, anchor handler, test imports, tests)
+- Modify: `crates/trusted-server-core/src/proxy.rs` (`handle_first_party_proxy_sign`, tests)
+- Modify: `CHANGELOG.md`
+
+**Interfaces:**
+
+- Produces:
+  - `pub(super) fn normalize_creative_url(url: &str) -> Option`;
+  - private `asset_target(&Settings, &str) -> Option` and `click_target(&Settings, &str) -> Option`;
+  - `pub fn Rewrite::should_proxy_asset(&self, host: &str) -> bool`;
+  - `pub fn Rewrite::should_wrap_click(&self, host: &str) -> bool`;
+  - private `Rewrite::is_excluded_host` and `Rewrite::normalize`.
+- Removes: `to_abs`, `Rewrite::is_excluded`. `proxy::is_host_permitted` stays private.
+
+- [ ] **Step 1: Settings tests**
+
+Replace `test_rewrite_is_excluded` in `settings.rs` with:
+
+```rust
+    #[test]
+    fn rewrite_policy_matches_exclude_patterns_case_insensitively() {
+        let mut rewrite = Rewrite::default();
+        rewrite
+            .exclude_domains
+            .extend(["cdn.example.com", "*.example.org"].map(str::to_owned));
+
+        for (host, expected) in [
+            ("cdn.example.com", false),
+            ("CDN.EXAMPLE.COM", false),
+            ("example.org", false),
+            ("a.b.example.org", false),
+            ("evil-example.org", true),
+            ("sub.cdn.example.com", true),
+            ("other.example.com", true),
+        ] {
+            assert_eq!(
+                rewrite.should_proxy_asset(host),
+                expected,
+                "should_proxy_asset(`{host}`) should be {expected}"
+            );
+            assert_eq!(
+                rewrite.should_wrap_click(host),
+                expected,
+                "should_wrap_click(`{host}`) should be {expected}"
+            );
+        }
+    }
+
+    #[test]
+    fn rewrite_normalize_trims_lowercases_and_drops_inert_exclude_entries() {
+        let mut rewrite = Rewrite::default();
+        rewrite
+            .exclude_domains
+            .extend(["  CDN.Example.com ", "", "*", "*.Example.ORG"].map(str::to_owned));
+
+        rewrite.normalize();
+
+        assert_eq!(
+            rewrite.exclude_domains,
+            vec!["cdn.example.com".to_owned(), "*.example.org".to_owned()],
+            "should trim, lowercase, and drop empty and bare `*` entries"
+        );
+    }
+
+    #[test]
+    fn rewrite_exclude_domains_match_mixed_case_entries_from_toml() {
+        let toml_str = crate_test_settings_str()
+            + r#"
+            [rewrite]
+            exclude_domains = ["CDN.Example.com"]
+            "#;
+
+        let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
+
+        assert!(
+            !settings.rewrite.should_proxy_asset("cdn.example.com"),
+            "should exclude a host whose config entry was written in mixed case"
+        );
+    }
+
+    #[test]
+    fn settings_load_normalizes_rewrite_exclude_domains() {
+        let toml_str = crate_test_settings_str()
+            + r#"
+            [rewrite]
+            exclude_domains = ["  CDN.Example.com ", "*", ""]
+            "#;
+
+        let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
+
+        assert_eq!(
+            settings.rewrite.exclude_domains,
+            vec!["cdn.example.com".to_owned()],
+            "should trim, lowercase, and drop inert entries when settings load"
+        );
+    }
+```
+
+`settings_load_normalizes_rewrite_exclude_domains` is the only test that proves `normalize` is wired into settings loading. The matcher is case-insensitive on its own, so the mixed-case TOML test would pass even without the `normalize_deserialized` call.
+
+- [ ] **Step 2: Creative tests**
+
+Change the `creative.rs` test module's import list to:
+
+```rust
+    use super::{
+        CreativeCssProcessor, StreamProcessor as _, normalize_creative_url,
+        process_auction_creative, proxy_if_abs, rewrite_creative_html,
+        rewrite_inline_creative_html, rewrite_srcset, rewrite_style_urls, sanitize_creative_html,
+    };
+```
+
+Replace `to_abs_conversions` and `to_abs_preserves_port_in_protocol_relative` with:
+
+```rust
+    fn normalized(raw: &str) -> Option {
+        normalize_creative_url(raw).map(|url| url.as_str().to_owned())
+    }
+
+    #[test]
+    fn normalize_creative_url_conversions() {
+        for (raw, expected) in [
+            ("//cdn.example/x", Some("https://cdn.example/x")),
+            ("HTTPS://cdn.example/x", Some("https://cdn.example/x")),
+            ("http://cdn.example/x", Some("http://cdn.example/x")),
+            ("   //cdn.example/y  ", Some("https://cdn.example/y")),
+            ("   https://cdn.example/a   ", Some("https://cdn.example/a")),
+            (
+                "//cdn.example.com:8080/asset.js",
+                Some("https://cdn.example.com:8080/asset.js"),
+            ),
+            (
+                "//cdn.example.com:9443/img.png",
+                Some("https://cdn.example.com:9443/img.png"),
+            ),
+            ("/local/x", None),
+            ("", None),
+            ("data:image/png;base64,abcd", None),
+            ("javascript:alert(1)", None),
+            ("mailto:test@example.com", None),
+            ("blob:xyz", None),
+            ("tel:+123", None),
+            ("about:blank", None),
+            ("https://exa mple.example/x", None),
+        ] {
+            assert_eq!(
+                normalized(raw).as_deref(),
+                expected,
+                "should normalize `{raw}` to {expected:?}"
+            );
+        }
+    }
+```
+
+Delete `to_abs_additional_cases`; its cases are in the table above. Replace `to_abs_respects_exclude_domains` and `to_abs_respects_wildcard_domains` with:
+
+```rust
+    #[test]
+    fn proxy_if_abs_respects_exclude_domains() {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings.rewrite.exclude_domains = vec![
+            "trusted-cdn.example.com".to_owned(),
+            "*.example.org".to_owned(),
+        ];
+
+        for excluded in [
+            "https://trusted-cdn.example.com/lib.js",
+            "//trusted-cdn.example.com/lib.js",
+            "https://example.org/cdn.js",
+            "//cdnjs.example.org/lib.js",
+        ] {
+            assert_eq!(
+                proxy_if_abs(&settings, excluded, ""),
+                None,
+                "should leave excluded URL `{excluded}` unproxied"
+            );
+        }
+        for proxied in [
+            "https://other-cdn.example.com/lib.js",
+            "//other-cdn.example.com/lib.js",
+            "https://notexample.org/lib.js",
+        ] {
+            assert!(
+                proxy_if_abs(&settings, proxied, "")
+                    .is_some_and(|url| url.starts_with("/first-party/proxy?tsurl=")),
+                "should proxy non-excluded URL `{proxied}`"
+            );
+        }
+    }
+
+    #[test]
+    fn unparseable_absolute_url_is_left_byte_identical() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = "";
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(html),
+            "should leave an unparseable absolute URL untouched, including its quoting: {out}"
+        );
+    }
+
+    #[test]
+    fn unparseable_absolute_click_url_is_left_byte_identical() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = "x";
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(html),
+            "should leave an unparseable click URL untouched, including its quoting: {out}"
+        );
+        assert!(
+            !out.contains("data-tsclick"),
+            "should not mark an unparseable link for the click guard: {out}"
+        );
+    }
+
+    #[test]
+    fn exclude_domains_match_case_insensitively_in_the_rewrite_pass() {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings
+            .rewrite
+            .exclude_domains
+            .extend(["Landing.Example.com", "CDN.example.com"].map(str::to_owned));
+        let html = r#"x"#;
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(r#""#),
+            "should leave a link raw when its host matches a mixed-case entry: {out}"
+        );
+        assert!(
+            out.contains(r#""#),
+            "should leave an asset raw when its host matches a mixed-case entry: {out}"
+        );
+    }
+```
+
+The unparseable tests assert `contains`, not equality: `rewrite_creative_html` injects the TSJS script tag even into markup with no ``.
+
+- [ ] **Step 3: Sign tests**
+
+In `proxy.rs`, replace `proxy_sign_rejects_excluded_urls`, and add the case test directly after it:
+
+```rust
+    #[test]
+    fn proxy_sign_rejects_excluded_urls() {
+        futures::executor::block_on(async {
+            let mut settings = create_test_settings();
+            settings.rewrite.exclude_domains = vec!["cdn.example".to_owned()];
+
+            for method in [&Method::GET, &Method::POST] {
+                for url in [
+                    "https://cdn.example/asset.js",
+                    "//cdn.example/asset.js",
+                    "https://CDN.Example/asset.js",
+                ] {
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example/first-party/sign",
+                        url,
+                    );
+                    let err: Report =
+                        handle_first_party_proxy_sign(&settings, &noop_services(), req)
+                            .await
+                            .expect_err("should reject excluded URL");
+
+                    assert_eq!(
+                        err.current_context().status_code(),
+                        StatusCode::BAD_GATEWAY,
+                        "{} should reject excluded URL `{url}` as unsupported",
+                        method.as_str()
+                    );
+                }
+            }
+        });
+    }
+
+    #[test]
+    fn proxy_sign_rejects_excluded_urls_case_insensitively() {
+        futures::executor::block_on(async {
+            let mut settings = create_test_settings();
+            settings.rewrite.exclude_domains = vec!["CDN.Example".to_owned()];
+
+            for method in [&Method::GET, &Method::POST] {
+                for url in ["https://cdn.example/asset.js", "//cdn.example/asset.js"] {
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example/first-party/sign",
+                        url,
+                    );
+                    let err: Report =
+                        handle_first_party_proxy_sign(&settings, &noop_services(), req)
+                            .await
+                            .expect_err("should reject a host excluded by a mixed-case entry");
+
+                    assert_eq!(
+                        err.current_context().status_code(),
+                        StatusCode::BAD_GATEWAY,
+                        "{} should reject `{url}` excluded by a mixed-case entry as unsupported",
+                        method.as_str()
+                    );
+                }
+            }
+        });
+    }
+```
+
+- [ ] **Step 4: Verify red**
+
+Run: `cargo test-fastly --no-run`
+
+Expected: compile errors, including `error[E0432]: unresolved import super::normalize_creative_url` and `no method named should_proxy_asset found for struct settings::Rewrite` (also `should_wrap_click` and `normalize`).
+
+Five tests use only existing APIs. Run each alone against the unmodified code, as described in the environment notes. Each panics:
+
+- `unparseable_absolute_url_is_left_byte_identical`: the output re-quotes the value as ``.
+- `unparseable_absolute_click_url_is_left_byte_identical`: the output is ``.
+- `exclude_domains_match_case_insensitively_in_the_rewrite_pass`: the link is wrapped in `/first-party/click`.
+- `proxy_sign_rejects_excluded_urls_case_insensitively`: sign returns `Response { status: 200 ... }`.
+- `settings_load_normalizes_rewrite_exclude_domains`: the loaded list is not normalized.
+
+- [ ] **Step 5: Implement**
+
+Apply the [Shared step](#shared-step) code to `settings.rs`, `creative.rs` and `proxy.rs`. Delete `Rewrite::is_excluded` with its `#[allow(dead_code)]`.
+
+Run: `git grep -n "to_abs\|is_excluded(" crates/`
+
+Expected: no matches.
+
+- [ ] **Step 6: Verify green**
+
+Run: `cargo fmt --all && cargo test-fastly -- normalize_creative_url proxy_if_abs_respects unparseable_absolute exclude_domains_match_case proxy_sign_ rewrite_policy_matches rewrite_normalize_trims rewrite_exclude_domains_match settings_load_normalizes`
+
+Expected: `16 passed` in `trusted-server-core`: the eleven shared tests plus the five existing `proxy_sign_*` tests.
+
+- [ ] **Step 7: Lint and full suite**
+
+Run: `cargo clippy-fastly && cargo test-fastly`
+
+Expected: clippy exits 0 with no `#[allow]` attributes. Every crate passes `test-fastly`, and no existing rewrite-output assertion needs editing.
+
+- [ ] **Step 8: CHANGELOG**
+
+Under `## [Unreleased]` › `### Fixed` (create the heading if absent), as the first entry:
+
+```markdown
+- `rewrite.exclude_domains` matching is now case-insensitive and ignores surrounding whitespace. Entries written with uppercase letters previously never matched and now take effect, so those hosts stop being proxied and click-wrapped. Audit `exclude_domains` for mixed-case entries before upgrading.
+```
+
+- [ ] **Step 9: Commit**
+
+```bash
+git add crates/trusted-server-core/src/creative.rs crates/trusted-server-core/src/settings.rs crates/trusted-server-core/src/proxy.rs CHANGELOG.md
+git commit --signoff -S -m "Split creative URL normalization from rewrite host policy" -m "Replace to_abs with normalize_creative_url and move exclusion policy to Rewrite::should_proxy_asset and Rewrite::should_wrap_click, sharing the case-insensitive proxy host matcher. Normalize exclude_domains at load. This is the shared step for #1231 and #1234."
+```
+
+---
+
+### Task 2: `rewrite_clicks` setting and resolution
+
+**Files:**
+
+- Modify: `crates/trusted-server-core/src/auction_config_types.rs` (field docs and field, `Default`, `impl AuctionConfig`, tests)
+- Modify: `crates/trusted-server-core/src/auction/orchestrator.rs` (`test_no_providers_configured` literal)
+- Modify: `crates/trusted-server-core/src/config_payload.rs` (tests)
+- Modify: `crates/trusted-server-core/src/settings.rs` (tests)
+
+**Interfaces:**
+
+- Produces:
+  - `pub rewrite_clicks: Option` on `AuctionConfig`;
+  - `pub fn AuctionConfig::rewrites_auction_clicks(&self) -> bool`;
+  - `pub fn AuctionConfig::rewrites_proxied_clicks(&self) -> bool`.
+
+- [ ] **Step 1: Write the tests**
+
+`auction_config_types.rs`, before `default_sanitize_creatives_is_not_serialized`:
+
+```rust
+    #[test]
+    fn default_rewrite_clicks_is_unset_and_not_serialized() {
+        let config = AuctionConfig::default();
+
+        assert_eq!(
+            config.rewrite_clicks, None,
+            "should leave click rewriting unset by default"
+        );
+        let serialized = serde_json::to_value(&config).expect("should serialize defaults");
+        assert!(
+            serialized.get("rewrite_clicks").is_none(),
+            "should omit the unset click setting from serialized config"
+        );
+    }
+
+    #[test]
+    fn explicit_rewrite_clicks_is_serialized() {
+        for value in [true, false] {
+            let config = AuctionConfig {
+                rewrite_clicks: Some(value),
+                ..AuctionConfig::default()
+            };
+
+            let serialized =
+                serde_json::to_value(config).expect("should serialize explicit click setting");
+
+            assert_eq!(
+                serialized.get("rewrite_clicks"),
+                Some(&serde_json::Value::Bool(value)),
+                "should serialize explicit rewrite_clicks = {value}"
+            );
+        }
+    }
+
+    #[test]
+    fn click_rewriting_resolves_per_entry_point() {
+        // (rewrite_creatives, rewrite_clicks, auction clicks, proxied clicks)
+        let cases = [
+            (true, None, true, true),
+            (false, None, false, true),
+            (true, Some(false), false, false),
+            (false, Some(true), true, true),
+            (true, Some(true), true, true),
+            (false, Some(false), false, false),
+        ];
+
+        for (rewrite_creatives, rewrite_clicks, auction, proxied) in cases {
+            let config = AuctionConfig {
+                rewrite_creatives,
+                rewrite_clicks,
+                ..AuctionConfig::default()
+            };
+
+            assert_eq!(
+                config.rewrites_auction_clicks(),
+                auction,
+                "auction clicks for rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
+            );
+            assert_eq!(
+                config.rewrites_proxied_clicks(),
+                proxied,
+                "proxied clicks for rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
+            );
+        }
+    }
+```
+
+`config_payload.rs`, before `strings_that_look_like_json_scalars_round_trip_as_strings`:
+
+```rust
+    #[test]
+    fn legacy_blob_without_rewrite_clicks_follows_rewrite_creatives() {
+        for rewrite_creatives in [true, false] {
+            let mut original = test_settings();
+            original.auction.rewrite_creatives = rewrite_creatives;
+            let data =
+                serde_json::to_value(&original).expect("should serialize settings to JSON");
+            assert!(
+                data["auction"].get("rewrite_clicks").is_none(),
+                "should omit unset rewrite_clicks from the payload"
+            );
+
+            let reconstructed = load_settings(&envelope_json(&original))
+                .expect("should reconstruct settings without rewrite_clicks");
+
+            assert_eq!(
+                reconstructed.auction.rewrite_clicks, None,
+                "should load a blob without rewrite_clicks as unset"
+            );
+            assert_eq!(
+                reconstructed.auction.rewrites_auction_clicks(),
+                rewrite_creatives,
+                "should follow rewrite_creatives = {rewrite_creatives} when unset"
+            );
+        }
+    }
+
+    #[test]
+    fn explicit_rewrite_clicks_survives_blob_round_trip() {
+        for (rewrite_creatives, rewrite_clicks) in [(true, false), (false, true)] {
+            let mut original = test_settings();
+            original.auction.rewrite_creatives = rewrite_creatives;
+            original.auction.rewrite_clicks = Some(rewrite_clicks);
+
+            let reconstructed = load_settings(&envelope_json(&original))
+                .expect("should reconstruct explicit rewrite_clicks");
+
+            assert_eq!(
+                reconstructed.auction.rewrite_clicks,
+                Some(rewrite_clicks),
+                "should preserve rewrite_clicks = {rewrite_clicks} with rewrite_creatives = {rewrite_creatives}"
+            );
+        }
+    }
+```
+
+`settings.rs`, before `test_auction_rewrite_creatives_accepts_explicit_false`:
+
+```rust
+    #[test]
+    fn test_auction_rewrite_clicks_accepts_explicit_values() {
+        for value in [true, false] {
+            let toml_str = crate_test_settings_str()
+                + &format!(
+                    r#"
+            [auction]
+            enabled = true
+            rewrite_clicks = {value}
+            "#
+                );
+
+            let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
+
+            assert_eq!(
+                settings.auction.rewrite_clicks,
+                Some(value),
+                "should parse explicit rewrite_clicks = {value}"
+            );
+        }
+    }
+```
+
+Append this assertion to `test_auction_creative_processing_defaults_when_omitted`:
+
+```rust
+        assert_eq!(
+            settings.auction.rewrite_clicks, None,
+            "click rewriting stays unset when the setting is omitted"
+        );
+```
+
+- [ ] **Step 2: Verify red**
+
+Run: `cargo test-fastly -p trusted-server-core --lib --no-run`
+
+Expected: 11 errors:
+
+- `E0560` (`AuctionConfig` has no field named `rewrite_clicks`), twice;
+- `E0599` (`no method named rewrites_auction_clicks`), twice;
+- `E0599` (`rewrites_proxied_clicks`), once;
+- `E0609` (`no field rewrite_clicks`), six times.
+
+- [ ] **Step 3: Implement**
+
+`auction_config_types.rs`: replace the first paragraph of the `rewrite_creatives` doc comment, and add the field after `pub rewrite_creatives: bool,`:
+
+```rust
+    /// Rewrite winning-bid creative asset URLs (images, scripts, styles,
+    /// media, iframes, CSS `url()`) to first-party `/first-party/proxy`
+    /// endpoints, applied after sanitization when
+    /// [`Self::sanitize_creatives`] is enabled.
+    ///
+    /// Bidder `` removal and creative TSJS injection run whenever this
+    /// or click rewriting ([`Self::rewrites_auction_clicks`]) is on.
+    ///
+    /// The default stays omitted from serialized config blobs to avoid adding
+    /// this field when it has no effect. Any rollback across schema versions
+    /// still requires restoring the matching old-schema blob with the old
+    /// binary.
+    #[serde(
+        default = "default_rewrite_creatives",
+        skip_serializing_if = "is_default_rewrite_creatives"
+    )]
+    pub rewrite_creatives: bool,
+
+    /// Wrap creative click-through links (``, ``) in signed
+    /// `/first-party/click` redirects.
+    ///
+    /// Unset keeps each path's existing behavior: auction creatives follow
+    /// [`Self::rewrite_creatives`], and HTML fetched through
+    /// `/first-party/proxy` keeps wrapping. An explicit value applies to every
+    /// path. Unset is omitted from serialized config blobs, so older binaries
+    /// keep loading them; any explicit value is serialized and rejected by
+    /// binaries that predate this field.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub rewrite_clicks: Option,
+```
+
+In `impl Default for AuctionConfig`, after `rewrite_creatives: default_rewrite_creatives(),`, add `rewrite_clicks: None,`. At the top of `impl AuctionConfig`:
+
+```rust
+    /// Whether auction creatives (`POST /auction` and inline SSAT/page-bids)
+    /// wrap click-through links.
+    ///
+    /// Unset [`Self::rewrite_clicks`] follows [`Self::rewrite_creatives`].
+    #[must_use]
+    pub fn rewrites_auction_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(self.rewrite_creatives)
+    }
+
+    /// Whether HTML fetched through `/first-party/proxy` wraps click-through
+    /// links.
+    ///
+    /// Unset [`Self::rewrite_clicks`] keeps wrapping, as before the setting
+    /// existed.
+    #[must_use]
+    pub fn rewrites_proxied_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(true)
+    }
+```
+
+In `auction/orchestrator.rs`, `test_no_providers_configured`, add `rewrite_clicks: None,` after `rewrite_creatives: true,`. It is the only `AuctionConfig` literal without `..Default::default()`.
+
+- [ ] **Step 4: Verify green**
+
+Run: `cargo test-fastly -p trusted-server-core --lib -- auction_config_types:: rewrite_clicks test_auction_creative_processing_defaults`
+
+Expected: `15 passed; 0 failed`.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add crates/trusted-server-core/src/auction_config_types.rs crates/trusted-server-core/src/auction/orchestrator.rs crates/trusted-server-core/src/config_payload.rs crates/trusted-server-core/src/settings.rs
+git commit --signoff -S -m "Add an auction rewrite_clicks setting that follows rewrite_creatives when unset"
+```
+
+---
+
+### Task 3: Gate asset and click handlers in the rewrite pass
+
+**Files:**
+
+- Modify: `crates/trusted-server-core/src/creative.rs` (`process_*`, wrappers and their docs, `CreativeFeatures`, `rewrite_creative_html_impl`, tests)
+- Modify: `crates/trusted-server-core/src/auction/formats.rs` (doc comment, debug log, test)
+- Modify: `crates/trusted-server-core/src/publisher.rs` (test)
+- Modify: `crates/trusted-server-core/src/auction/endpoints.rs` (doc comment)
+
+**Interfaces:**
+
+- Consumes: `AuctionConfig::rewrites_auction_clicks()` (Task 2), `asset_target` and `click_target` (Task 1).
+- Produces, private to `creative.rs`:
+
+```rust
+struct CreativeFeatures { assets: bool, clicks: bool }
+impl CreativeFeatures { const ALL: Self; fn any(self) -> bool; }
+fn rewrite_creative_html_impl(
+    settings: &Settings,
+    markup: &str,
+    base_origin: &str,
+    inject_tsjs: bool,
+    max_output_size: usize,
+    features: CreativeFeatures,
+) -> String;
+```
+
+- [ ] **Step 1: Matrix fixtures and helpers**
+
+At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
+
+```rust
+    use crate::settings::Settings;
+
+    const MATRIX_FIXTURE_WITH_BODY: &str = r#"
AdExcludedMail"#;
+    const MATRIX_FIXTURE_FRAGMENT: &str = r#"AdExcluded"#;
+    const INLINE_ORIGIN: &str = "https://www.example.com";
+
+    #[derive(Debug, Clone, Copy)]
+    enum CreativePath {
+        Auction,
+        Inline,
+    }
+
+    fn process_for_path(settings: &Settings, path: CreativePath, html: &str) -> String {
+        match path {
+            CreativePath::Auction => process_auction_creative(settings, html),
+            CreativePath::Inline => {
+                super::process_inline_auction_creative(settings, INLINE_ORIGIN, html)
+            }
+        }
+    }
+
+    fn matrix_settings(assets: bool, clicks: Option) -> Settings {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings.auction.sanitize_creatives = false;
+        settings.auction.rewrite_creatives = assets;
+        settings.auction.rewrite_clicks = clicks;
+        settings.rewrite.exclude_domains = vec!["excluded.example.com".to_owned()];
+        settings
+    }
+```
+
+- [ ] **Step 2: Matrix and unset-pin tests**
+
+```rust
+    #[test]
+    fn asset_and_click_switches_combine_on_auction_and_inline_paths() {
+        for path in [CreativePath::Auction, CreativePath::Inline] {
+            for (fixture, has_body) in [
+                (MATRIX_FIXTURE_WITH_BODY, true),
+                (MATRIX_FIXTURE_FRAGMENT, false),
+            ] {
+                for (assets, clicks) in [(true, true), (true, false), (false, true), (false, false)]
+                {
+                    let settings = matrix_settings(assets, Some(clicks));
+                    let label =
+                        format!("{path:?} assets={assets} clicks={clicks} body={has_body}");
+                    let prefix = match path {
+                        CreativePath::Auction => "",
+                        CreativePath::Inline => INLINE_ORIGIN,
+                    };
+
+                    let out = process_for_path(&settings, path, fixture);
+
+                    let expected_proxied = match (assets, has_body) {
+                        (false, _) => 0,
+                        (true, true) => 4,
+                        (true, false) => 1,
+                    };
+                    assert_eq!(
+                        out.matches("/first-party/proxy?tsurl=").count(),
+                        expected_proxied,
+                        "{label}: proxied asset URL count: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(&format!("src=\"{prefix}/first-party/proxy?tsurl=")),
+                        assets,
+                        "{label}: image proxied only when assets are on: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(r#"src="https://cdn.example.com/ad.png""#),
+                        !assets,
+                        "{label}: image raw only when assets are off: {out}"
+                    );
+
+                    // Each wrapped anchor carries the click URL in href and data-tsclick.
+                    let expected_click_urls = match (clicks, has_body) {
+                        (false, _) => 0,
+                        (true, true) => 4,
+                        (true, false) => 2,
+                    };
+                    assert_eq!(
+                        out.matches("/first-party/click?tsurl=").count(),
+                        expected_click_urls,
+                        "{label}: click URL count: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(&format!("href=\"{prefix}/first-party/click?tsurl=")),
+                        clicks,
+                        "{label}: anchor wrapped only when clicks are on: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(r#"href="https://landing.example.com/page""#),
+                        !clicks,
+                        "{label}: anchor raw only when clicks are off: {out}"
+                    );
+                    assert_eq!(
+                        out.contains("data-tsclick"),
+                        clicks,
+                        "{label}: data-tsclick only when clicks are on: {out}"
+                    );
+                    assert!(
+                        out.contains(r#"href="https://excluded.example.com/page""#),
+                        "{label}: excluded anchor always stays raw: {out}"
+                    );
+                    if has_body {
+                        assert!(
+                            out.contains(r#"href="mailto:ads@example.com""#),
+                            "{label}: mailto anchor always stays raw: {out}"
+                        );
+                    }
+
+                    assert_eq!(
+                        out.contains(" removed whenever either switch is on: {out}"
+                    );
+                    let expected_tsjs =
+                        usize::from(matches!(path, CreativePath::Auction) && (assets || clicks));
+                    assert_eq!(
+                        out.matches("/static/tsjs=").count(),
+                        expected_tsjs,
+                        "{label}: TSJS injected once on /auction when either switch is on: {out}"
+                    );
+                    if !assets && !clicks {
+                        assert_eq!(
+                            out, fixture,
+                            "{label}: should pass through byte for byte with both switches off"
+                        );
+                    }
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn unset_rewrite_clicks_preserves_existing_output() {
+        for path in [CreativePath::Auction, CreativePath::Inline] {
+            let unset = matrix_settings(true, None);
+            let explicit = matrix_settings(true, Some(true));
+            assert_eq!(
+                process_for_path(&unset, path, MATRIX_FIXTURE_WITH_BODY),
+                process_for_path(&explicit, path, MATRIX_FIXTURE_WITH_BODY),
+                "{path:?}: unset rewrite_clicks with rewrite_creatives = true should match full rewriting"
+            );
+
+            let disabled = matrix_settings(false, None);
+            assert_eq!(
+                process_for_path(&disabled, path, MATRIX_FIXTURE_WITH_BODY),
+                MATRIX_FIXTURE_WITH_BODY,
+                "{path:?}: unset rewrite_clicks with rewrite_creatives = false should pass through"
+            );
+        }
+    }
+```
+
+- [ ] **Step 3: End-to-end tests**
+
+`auction/formats.rs`, before `sanitize_creatives_defaults_to_disabled`:
+
+```rust
+    #[test]
+    fn convert_to_openrtb_response_wraps_clicks_without_rewriting_assets() {
+        let mut settings = make_settings();
+        settings.auction.sanitize_creatives = false;
+        settings.auction.rewrite_creatives = false;
+        settings.auction.rewrite_clicks = Some(true);
+        let auction_request = make_auction_request();
+        let result = make_result(make_complete_creative_bid());
+
+        let response = convert_to_openrtb_response(&result, &settings, &auction_request, false)
+            .expect("should convert creative with click rewriting only");
+        let adm = response_adm(response);
+
+        assert!(
+            adm.contains("/first-party/click?tsurl=") && adm.contains("data-tsclick"),
+            "should wrap the landing link: {adm}"
+        );
+        assert!(
+            !adm.contains("/first-party/proxy?tsurl="),
+            "should not proxy any asset: {adm}"
+        );
+        assert!(
+            adm.contains(r#"src="https://cdn.example.com/ad.png""#),
+            "should keep the image URL direct: {adm}"
+        );
+        assert!(
+            adm.contains("tsjs-unified.min.js"),
+            "should inject the creative runtime for the click guard: {adm}"
+        );
+    }
+```
+
+`publisher.rs`, before `build_bid_map_uses_request_origin_for_inline_urls`:
+
+```rust
+        #[test]
+        fn build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off() {
+            let mut settings = test_settings();
+            settings.auction.rewrite_creatives = true;
+            settings.auction.rewrite_clicks = Some(false);
+            settings.publisher.domain = "example.com".to_string();
+
+            let mut winning_bids = HashMap::new();
+            let mut bid = make_bid(
+                "atf_sidebar_ad",
+                1.50,
+                "examplessp",
+                "abc123",
+                "https://ssp.example.com/win",
+                "https://ssp.example.com/bill",
+            );
+            bid.creative = Some(
+                ""
+                    .to_string(),
+            );
+            winning_bids.insert("atf_sidebar_ad".to_string(), bid);
+
+            let map = build_bid_map(&winning_bids, PriceGranularity::Dense, &settings, "", false);
+            let adm = map
+                .get("atf_sidebar_ad")
+                .and_then(|v| v.as_object())
+                .and_then(|o| o.get("adm"))
+                .and_then(|v| v.as_str())
+                .expect("should include a rewritten adm");
+
+            assert!(
+                adm.contains("https://example.com/first-party/proxy?tsurl="),
+                "should still proxy the image with an absolute URL, got: {adm}"
+            );
+            assert!(
+                adm.contains("href=\"https://landing.example.com/page\""),
+                "should leave the landing link raw, got: {adm}"
+            );
+            assert!(
+                !adm.contains("data-tsclick") && !adm.contains("/first-party/click"),
+                "should not wrap the landing link, got: {adm}"
+            );
+        }
+```
+
+- [ ] **Step 4: Verify red**
+
+Run each test alone, as described in the environment notes. Expected:
+
+- `asset_and_click_switches_combine_on_auction_and_inline_paths` panics with "Auction assets=true clicks=false body=true: click URL count".
+- `convert_to_openrtb_response_wraps_clicks_without_rewriting_assets` panics with "should wrap the landing link". Nothing was rewritten.
+- `build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off` panics with "should leave the landing link raw". The anchor was wrapped.
+- `unset_rewrite_clicks_preserves_existing_output` **passes**. It pins today's behavior.
+
+- [ ] **Step 5: Add `CreativeFeatures`**
+
+Above the `rewrite_creative_html_impl` doc comment:
+
+```rust
+/// Which rewrite features one creative pass applies.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct CreativeFeatures {
+    /// Proxy eligible asset URLs through `/first-party/proxy`.
+    assets: bool,
+    /// Wrap eligible click-through links in `/first-party/click`.
+    clicks: bool,
+}
+
+impl CreativeFeatures {
+    /// Both features on: the behavior of the public rewrite wrappers.
+    const ALL: Self = Self {
+        assets: true,
+        clicks: true,
+    };
+
+    /// Whether the pass needs to run at all.
+    fn any(self) -> bool {
+        self.assets || self.clicks
+    }
+}
+```
+
+Extend the `rewrite_creative_html_impl` doc comment's sentence to read "...`max_output_size` bounds the rewritten result; `features` selects which handlers run. `` removal and TSJS injection always run, since callers only invoke the pass when at least one feature is on." Add `features: CreativeFeatures,` as the last parameter.
+
+- [ ] **Step 6: Build the handler vector conditionally**
+
+Inside `rewrite_creative_html_impl`, after `let overflowed = ...;`, build the vector, then hand it to `HtmlRewriter::new`. Each block below is the existing handler moved verbatim with its comment. Only the indentation changes, so review with `git diff -w`.
+
+```rust
+    let mut element_content_handlers = vec![
+        // (existing `` removal handler with its comment)
+        // (existing `` TSJS handler)
+    ];
+    if features.assets {
+        element_content_handlers.extend([
+            // (existing handlers, unchanged and in their current order: img,
+            //  script[src], link[href], video/audio/source, object[data],
+            //  embed[src], input[src], SVG image/use, [style], text!("style"),
+            //  iframe, [srcset], [imagesrcset])
+        ]);
+    }
+    if features.clicks {
+        element_content_handlers.push(
+            // Click-through links
+            element!("a[href], area[href]", |el| {
+                if let Some(href) = el.get_attribute("href")
+                    && let Some(target) = click_target(settings, &href)
+                {
+                    let click = build_click_url(settings, target.as_str(), base_origin);
+                    let _ = el.set_attribute("href", &click);
+                    let _ = el.set_attribute("data-tsclick", &click);
+                }
+                Ok(())
+            }),
+        );
+    }
+    let mut rewriter = HtmlRewriter::new(
+        HtmlSettings {
+            element_content_handlers,
+            ..HtmlSettings::default()
+        },
+        // (existing output sink closure, unchanged)
+    );
+```
+
+`extend([...])` with an array of `element!`/`text!` invocations type-checks with no annotations, because every invocation yields the same tuple type. The body-less TSJS fallback keeps `if inject_tsjs && !injected_ts_creative.get() && !rewritten.is_empty()`.
+
+- [ ] **Step 7: Wire the callers**
+
+```rust
+pub(crate) fn process_auction_creative(settings: &Settings, raw: &str) -> String {
+    process_auction_creative_with_rewriter(settings, raw, |sanitized, features| {
+        rewrite_creative_html_impl(settings, sanitized, "", true, MAX_CREATIVE_SIZE, features)
+    })
+}
+
+pub(crate) fn process_inline_auction_creative(
+    settings: &Settings,
+    base_origin: &str,
+    raw: &str,
+) -> String {
+    process_auction_creative_with_rewriter(settings, raw, |sanitized, features| {
+        rewrite_creative_html_impl(
+            settings,
+            sanitized,
+            base_origin,
+            false,
+            MAX_CREATIVE_SIZE,
+            features,
+        )
+    })
+}
+```
+
+In `process_auction_creative_with_rewriter`, change the closure type to `rewrite: impl FnOnce(&str, CreativeFeatures) -> String`, and replace the `if settings.auction.rewrite_creatives { ... }` tail with:
+
+```rust
+    let features = CreativeFeatures {
+        assets: settings.auction.rewrite_creatives,
+        clicks: settings.auction.rewrites_auction_clicks(),
+    };
+    if features.any() {
+        rewrite(&sanitized, features)
+    } else {
+        sanitized
+    }
+```
+
+Public wrappers: pass `CreativeFeatures::ALL` as the sixth argument in `rewrite_creative_html` (`"", true, MAX_CREATIVE_SIZE`), `rewrite_inline_creative_html` (`base_origin, false, MAX_CREATIVE_SIZE`) and, for now, `rewrite_proxied_html` (`"", true, MAX_REWRITABLE_BODY_SIZE`). Task 4 changes the last one.
+
+- [ ] **Step 8: Doc comments**
+
+- `process_auction_creative`: "Sanitization is controlled by [`...sanitize_creatives`], asset rewriting by [`...rewrite_creatives`], and click wrapping by [`...rewrites_auction_clicks`]. With all three disabled the creative is returned exactly as the bidder sent it."
+- `process_inline_auction_creative`: "Applies the same opt-in sanitization and the same asset and click switches as [`process_auction_creative`]. Proxy and click URLs are emitted as absolute URLs against `base_origin` without injecting the creative TSJS bundle."
+- `rewrite_creative_html`: replace the stale `tsjs-creative` bullet with:
+  - "`` / `` → `/first-party/click?tsurl=…`, copied into `data-tsclick`";
+  - "Injects the unified TSJS bundle (`/static/tsjs=tsjs-unified.min.js`), whose creative runtime installs the click guard, once at the top of ``."
+
+  Add: "Rewrites assets and clicks unconditionally; the auction settings are applied by the auction processing entry points, not here."
+
+- `rewrite_inline_creative_html`: "its click guard is unnecessary for click URLs that are already absolute here".
+- `auction/endpoints.rs` response doc: "Creative HTML is inlined in each bid's `adm` field after optional sanitization ([`auction.sanitize_creatives`][...]). First-party asset rewriting ([`auction.rewrite_creatives`][...]) and click wrapping ([`auction.rewrite_clicks`][...]) are enabled by default. Bidder `` removal and creative TSJS injection run when either is on." This replaces the stale "mandatory server-side sanitization".
+- `auction/formats.rs`, on `convert_to_openrtb_response`: name `rewrite_clicks` next to `rewrite_creatives` and add the intra-doc link target `/// [`AuctionConfig::rewrite_clicks`]: crate::auction_config_types::AuctionConfig::rewrite_clicks`. Add `clicks {}` to the "Processed creative" debug log, fed by `settings.auction.rewrites_auction_clicks()`.
+
+- [ ] **Step 9: Verify green**
+
+Run: `cargo fmt --all && cargo test-fastly -p trusted-server-core --lib -- creative:: auction::formats publisher::`
+
+Expected: `551 passed; 0 failed`, including every pre-existing rewrite test.
+
+Run: `cargo clippy-fastly`
+
+Expected: exit 0.
+
+- [ ] **Step 10: Commit**
+
+```bash
+git add crates/trusted-server-core/src/creative.rs crates/trusted-server-core/src/auction/formats.rs crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/auction/endpoints.rs
+git commit --signoff -S -m "Gate creative asset and click rewriting on separate switches" -m "The rewrite pass registers asset handlers only when rewrite_creatives is on and the anchor handler only when click rewriting resolves on. Base removal and TSJS injection run whenever either is on."
+```
+
+---
+
+### Task 4: Apply an explicit `rewrite_clicks` to HTML fetched through `/first-party/proxy`
+
+**Files:**
+
+- Modify: `crates/trusted-server-core/src/creative.rs` (`rewrite_proxied_html`)
+- Test: `crates/trusted-server-core/src/proxy.rs`
+
+- [ ] **Step 1: Write the test**
+
+Before `html_response_rewrite_preserves_non_standard_port`:
+
+```rust
+    #[test]
+    fn proxied_html_click_wrapping_follows_explicit_rewrite_clicks() {
+        let html = r#""#;
+        // (rewrite_creatives, rewrite_clicks, expect wrapped clicks)
+        let cases = [
+            (true, None, true),
+            (false, None, true),
+            (true, Some(false), false),
+            (false, Some(false), false),
+            (false, Some(true), true),
+        ];
+
+        for (rewrite_creatives, rewrite_clicks, expect_clicks) in cases {
+            let mut settings = create_test_settings();
+            settings.auction.rewrite_creatives = rewrite_creatives;
+            settings.auction.rewrite_clicks = rewrite_clicks;
+            let label = format!(
+                "rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
+            );
+            let req = build_http_request(Method::GET, "https://edge.example.com/first-party/proxy");
+            let mut response = build_http_response(StatusCode::OK, EdgeBody::from(html));
+            response.headers_mut().insert(
+                header::CONTENT_TYPE,
+                HeaderValue::from_static("text/html; charset=utf-8"),
+            );
+
+            let body = response_body_string(
+                finalize(
+                    &settings,
+                    &req,
+                    "https://cdn.example.com/creative.html",
+                    response,
+                )
+                .expect("should finalize proxied HTML"),
+            );
+
+            assert!(
+                body.contains("/first-party/proxy?tsurl="),
+                "{label}: proxied HTML always proxies assets: {body}"
+            );
+            assert_eq!(
+                body.contains("/first-party/click?tsurl="),
+                expect_clicks,
+                "{label}: click wrapping in proxied HTML: {body}"
+            );
+            assert_eq!(
+                body.contains("data-tsclick"),
+                expect_clicks,
+                "{label}: data-tsclick in proxied HTML: {body}"
+            );
+            assert!(
+                body.contains("/static/tsjs="),
+                "{label}: proxied HTML always receives the runtime: {body}"
+            );
+        }
+    }
+```
+
+- [ ] **Step 2: Verify red**
+
+Run the test alone. Expected: a panic with "rewrite_creatives=true rewrite_clicks=Some(false): click wrapping in proxied HTML".
+
+- [ ] **Step 3: Implement**
+
+In `rewrite_proxied_html`, replace `CreativeFeatures::ALL` with:
+
+```rust
+        CreativeFeatures {
+            assets: true,
+            clicks: settings.auction.rewrites_proxied_clicks(),
+        },
+```
+
+Append this paragraph to its doc comment: "Asset rewriting here is unconditional. Click wrapping follows an explicit `[auction] rewrite_clicks`; when that is unset, links keep being wrapped regardless of `rewrite_creatives`."
+
+- [ ] **Step 4: Verify green**
+
+Run: `cargo test-fastly -p trusted-server-core --lib -- proxy::tests::proxied_html proxy::tests::auction_rewrite_setting_does_not_change`
+
+Expected: `2 passed; 0 failed`.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add crates/trusted-server-core/src/creative.rs crates/trusted-server-core/src/proxy.rs
+git commit --signoff -S -m "Apply an explicit rewrite_clicks to HTML fetched through the first-party proxy"
+```
+
+---
+
+### Task 5: Example config and CLI overlay coverage
+
+**Files:**
+
+- Modify: `trusted-server.example.toml`
+- Test: `crates/trusted-server-cli/tests/config_env_overlay.rs`
+
+- [ ] **Step 1: Write the overlay tests**
+
+Add a constant after `SANITIZE_ENV`:
+
+```rust
+const CLICKS_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CLICKS";
+```
+
+Before `config_validate_explains_legacy_provider_list_migration`:
+
+```rust
+fn pushed_auction_with_env(
+    project: &MigratedProject,
+    key: &str,
+    raw_value: &str,
+) -> serde_json::Value {
+    let output = Command::new(env!("CARGO_BIN_EXE_ts"))
+        .args(["config", "push", "--adapter", "axum", "--manifest"])
+        .arg(&project.manifest_path)
+        .arg("--app-config")
+        .arg(&project.config_path)
+        .args(["--yes", "--no-diff"])
+        .current_dir(project.directory.path())
+        .env(key, raw_value)
+        .output()
+        .expect("should run ts config push");
+    assert!(
+        output.status.success(),
+        "config push should succeed: {}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+
+    let local_store_path = project
+        .directory
+        .path()
+        .join(".edgezero/local-config-trusted_server_config.json");
+    let local_store: serde_json::Value = serde_json::from_str(
+        &fs::read_to_string(local_store_path).expect("should read pushed local config"),
+    )
+    .expect("should parse local config store");
+    let envelope_json = local_store
+        .as_object()
+        .and_then(|entries| entries.values().next())
+        .and_then(serde_json::Value::as_str)
+        .expect("should contain a blob envelope");
+    let envelope: serde_json::Value =
+        serde_json::from_str(envelope_json).expect("should parse blob envelope");
+    envelope["data"]["auction"].clone()
+}
+
+#[test]
+fn rewrite_clicks_environment_override_applies_when_leaf_present() {
+    let project = migrated_project();
+    let mut document = fs::read_to_string(&project.config_path)
+        .expect("should read migrated config")
+        .parse::()
+        .expect("should parse migrated config");
+    document["auction"]["rewrite_clicks"] = value(true);
+    fs::write(&project.config_path, document.to_string()).expect("should write config");
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert_eq!(
+        auction["rewrite_clicks"],
+        serde_json::Value::Bool(false),
+        "pushed config should contain the rewrite_clicks environment override"
+    );
+}
+
+#[test]
+fn rewrite_clicks_environment_override_is_ignored_without_leaf() {
+    let project = migrated_project();
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert!(
+        auction.get("rewrite_clicks").is_none(),
+        "an override for a missing leaf should be ignored and the unset default omitted"
+    );
+}
+```
+
+- [ ] **Step 2: Run**
+
+Run: `cargo test --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --test config_env_overlay rewrite_clicks`
+
+Expected: `2 passed; 0 failed`.
+
+These pin existing EdgeZero behavior, so there is no red step. The leaf-present test proves the override is read. The leaf-absent test proves an override for a missing leaf is ignored, so the spec's overlay caveat holds. If the leaf-absent test ever fails, update the spec and the configuration guide before continuing.
+
+- [ ] **Step 3: Example TOML**
+
+Replace the `rewrite_creatives` comment and line in `[auction]`:
+
+```toml
+# Rewrite winning-bid creative asset URLs (images, scripts, styles, media,
+# iframes, CSS url()) to first-party /first-party/proxy endpoints (default
+# true). Set false to leave asset URLs direct. Bidder  removal and
+# creative TSJS injection run when this or rewrite_clicks is on. Sanitization
+# is controlled separately by `sanitize_creatives` below. Restore true before
+# rolling back to an older binary that rejects unknown fields.
+rewrite_creatives = true
+# Wrap creative click-through links (, ) in signed
+# /first-party/click redirects. Unset (the default) follows rewrite_creatives
+# for auction creatives and keeps wrapping links in HTML fetched through
+# /first-party/proxy; a set value applies to both. Leave it commented out to
+# keep that behavior: any explicit value is stored in the pushed config, and
+# binaries older than this setting reject it. Uncomment it before relying on
+# TRUSTED_SERVER__AUCTION__REWRITE_CLICKS, which cannot create a missing leaf.
+# rewrite_clicks = true
+```
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add trusted-server.example.toml crates/trusted-server-cli/tests/config_env_overlay.rs
+git commit --signoff -S -m "Document rewrite_clicks in the example config and pin its overlay behavior"
+```
+
+---
+
+### Task 6: Operator documentation and CHANGELOG
+
+**Files:**
+
+- Modify: `docs/guide/configuration.md`
+- Modify: `docs/guide/creative-processing.md`
+- Modify: `docs/guide/auction-orchestration.md`
+- Modify: `docs/guide/api-reference.md`
+- Modify: `crates/trusted-server-core/src/auction/README.md`
+- Modify: `CHANGELOG.md`
+
+- [ ] **Step 1: `configuration.md` `[auction]`**
+
+- Table: change the `rewrite_creatives` description to "Rewrite winning-bid asset URLs through first-party endpoints". Add a row:
+
+  ```markdown
+  | `rewrite_clicks` | Boolean | unset | Wrap creative links in `/first-party/click`; unset follows `rewrite_creatives` |
+  ```
+
+  Prettier re-pads the whole table.
+
+- Processing paragraph:
+  - assets go to `/first-party/proxy`;
+  - `rewrite_clicks` controls ``/`` and follows `rewrite_creatives` when unset;
+  - `` is removed, and TSJS injected on `POST /auction`, whenever either setting is on;
+  - "With all three disabled" replaces "With both disabled";
+  - an explicit `rewrite_clicks` applies to links in proxied HTML, and unset keeps wrapping them.
+- Warning block:
+  - add "any explicit `rewrite_clicks`" to the serialized values;
+  - rollback note: "Older binaries cannot express clicks off with assets on; rolling back from that configuration re-enables click wrapping.";
+  - overlay note: "`rewrite_clicks` is unset by default and so cannot appear as a TOML leaf until you set it; add `rewrite_clicks = true` or `false` under `[auction]` before relying on `TRUSTED_SERVER__AUCTION__REWRITE_CLICKS`."
+
+- [ ] **Step 2: `creative-processing.md`**
+
+- Processing Triggers, item 1, append: "Click-through links are wrapped according to `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset."
+- Item 2, append: "An explicitly set `[auction].rewrite_clicks` also applies to links in proxied HTML."
+- After the sanitization warning in "Auction Rewrite Control", add an `### Assets and clicks` subsection containing:
+  - the TOML snippet (`rewrite_creatives = true`, `# rewrite_clicks = false`);
+  - the four-row table (assets × clicks → asset URLs, links, ``, TSJS);
+  - "SSAT/page-bids follows the same table with absolute URLs and never injects TSJS. `exclude_domains` applies to both assets and links.";
+  - the `clickGuard` versus `rewrite_clicks` paragraph.
+- "Anchors (Click Tracking)": add `**Controlled by**: [auction].rewrite_clicks (follows rewrite_creatives when unset)`, switch the example to `advertiser.example.com`, and show the `data-tsclick` attribute.
+
+- [ ] **Step 3: Remaining guides**
+
+- `auction-orchestration.md`:
+  - mermaid note: `rewrite_creatives=false: asset URLs direct; clicks follow rewrite_clicks`;
+  - flow: `├─[rewrite_creatives or rewrite_clicks] Rewrite assets and/or links, inject creative TSJS`;
+  - Creative Processing paragraph: link to `/guide/creative-processing#assets-and-clicks`.
+- `api-reference.md`: list `[auction].rewrite_clicks` with the other two settings.
+- `auction/README.md`: add the `rewrite_clicks` bullet.
+
+- [ ] **Step 4: CHANGELOG**
+
+Under `### Added`, as the first entry:
+
+```markdown
+- Added `[auction].rewrite_clicks` to control creative click-through wrapping (``/`` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `` removal and creative TSJS injection run when either setting is on. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary; older binaries cannot express clicks off with assets on.
+```
+
+- [ ] **Step 5: Format**
+
+```bash
+PRETTIER=/path/to/checkout/docs/node_modules/.bin/prettier
+$PRETTIER --config docs/.prettierrc --write docs/guide/configuration.md docs/guide/creative-processing.md docs/guide/auction-orchestration.md docs/guide/api-reference.md
+(cd docs && npm run format)
+$PRETTIER --config docs/.prettierrc --check "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md"
+```
+
+Expected: the first `npm run format` before `--write` flags `guide/configuration.md`, because of the new table row width. After `--write`, both checks print "All matched files use Prettier code style!".
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add docs/guide CHANGELOG.md crates/trusted-server-core/src/auction/README.md
+git commit --signoff -S -m "Document the rewrite_clicks creative setting"
+```
+
+---
+
+### Task 7: Full verification
+
+- [ ] **Step 1: Run every CI gate**
+
+```bash
+cargo fmt --all -- --check
+cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasm && cargo clippy-cli && cargo clippy-codegen
+cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
+./scripts/test-cli.sh
+cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity
+(cd crates/trusted-server-js/lib && npx vitest run && npm run format)
+(cd docs && npm run format)
+```
+
+Expected: every command exits 0. The validation run's results are in the spec handoff.
+
+- [ ] **Step 2: Map completion criteria to tests**
+
+| Criterion                               | Test                                                                                                                                                                                |
+| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Unset output unchanged                  | `unset_rewrite_clicks_preserves_existing_output`, existing rewrite tests, unset rows of `proxied_html_click_wrapping_follows_explicit_rewrite_clicks`                               |
+| Four-combination matrix on both paths   | `asset_and_click_switches_combine_on_auction_and_inline_paths`                                                                                                                      |
+| Clicks off: raw href, no `data-tsclick` | matrix, `build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off`                                                                                                                |
+| Clicks on, assets off                   | matrix, `convert_to_openrtb_response_wraps_clicks_without_rewriting_assets`                                                                                                         |
+| `exclude_domains` applies to anchors    | matrix (excluded anchor), `rewrite_click_urls_excludes_blacklisted_domains`, `exclude_domains_match_case_insensitively_in_the_rewrite_pass`                                         |
+| Proxied HTML rule                       | `proxied_html_click_wrapping_follows_explicit_rewrite_clicks`                                                                                                                       |
+| Default omitted from the blob           | `default_rewrite_clicks_is_unset_and_not_serialized`, `legacy_blob_without_rewrite_clicks_follows_rewrite_creatives`, `rewrite_clicks_environment_override_is_ignored_without_leaf` |
+| Docs and CHANGELOG                      | Task 6                                                                                                                                                                              |
+
+- [ ] **Step 3: Hand off**
+
+Open the PR through the `pr-creator` agent. Link the spec, state whether this PR introduced the shared step or rebased onto #1231's, and paste the rollout and rollback order from the spec.
diff --git a/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md b/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md
new file mode 100644
index 000000000..97bed4dfc
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md
@@ -0,0 +1,511 @@
+# Creative click rewriting switch
+
+**Issue:** [IABTechLab/trusted-server#1234](https://github.com/IABTechLab/trusted-server/issues/1234)
+
+**Related:** [IABTechLab/trusted-server#1231](https://github.com/IABTechLab/trusted-server/issues/1231) (asset host allowlist)
+
+**Date:** 2026-10-07
+
+**Status:** Draft
+
+Line numbers below are at `main` 7a0ecb4c.
+
+## Problem
+
+Click wrapping is not a feature of its own. The creative rewrite pass turns every eligible `` and `` into a signed `/first-party/click` redirect. It does this in the same lol_html pass, behind the same switch (`[auction] rewrite_creatives`) and through the same per-URL check (`to_abs`) that it uses to proxy assets.
+
+As a result an operator cannot:
+
+- keep first-party click tracking while leaving creative assets direct; or
+- proxy creative assets while leaving advertiser landing links untouched.
+
+## Goals
+
+- Add a switch that turns click wrapping on or off without affecting asset rewriting.
+- Narrow `rewrite_creatives` to asset rewriting. Keep its name so existing configs still parse.
+- Leave rewritten output unchanged for every existing config. In particular, upgrading must not turn click wrapping on for operators who have `rewrite_creatives = false`.
+- Keep `[rewrite] exclude_domains` applying to anchors.
+- Define the `to_abs` split that this issue and #1231 share, so either implementation can land first.
+- Keep the config blob compatible with older binaries while the new setting is unset.
+
+## Non-goals
+
+- The asset host allowlist (#1231). It never applies to anchors.
+- Unregistering `/first-party/click` or `/first-party/proxy-rebuild` when clicks are off. Creatives that were already served can still hold signed click URLs.
+- Adding a click guard to the inline SSAT/page-bids path.
+- Mapping any server setting onto the client `TsCreativeConfig` flags.
+- Changing asset rewriting inside HTML or CSS fetched through `/first-party/proxy`. That path keeps proxying assets regardless of `rewrite_creatives`.
+- Domain checks at `/first-party/click` and `/first-party/proxy-rebuild`. Those endpoints redirect any validly signed target by design (`proxy.rs:1896-1904`).
+
+## Current behavior
+
+### One pass, one switch
+
+`process_auction_creative_with_rewriter` (`crates/trusted-server-core/src/creative.rs:1008-1035`) enforces the size cap, optionally sanitizes, and then runs the rewrite pass only when `settings.auction.rewrite_creatives` is true (`creative.rs:1031`). With rewriting off, the markup ships exactly as sanitization left it.
+
+The pass itself, `rewrite_creative_html_impl` (`creative.rs:1125`), registers every handler unconditionally:
+
+| Handler                          | Location                             | Effect                                                                     |
+| -------------------------------- | ------------------------------------ | -------------------------------------------------------------------------- |
+| `` removal                 | `creative.rs:1163`                   | Removes bidder `` so root-relative first-party URLs cannot rebase.   |
+| `` TSJS injection          | `creative.rs:1168`                   | Prepends the unified bundle once when `inject_tsjs` is set.                |
+| Asset handlers                   | `creative.rs:1179-1281`, `1293-1339` | img, script, link, media, object/embed, input, SVG, iframe, srcset, CSS.   |
+| Anchor handler                   | `creative.rs:1283`                   | `a[href], area[href]` → `build_click_url`, sets `href` and `data-tsclick`. |
+| Body-less fragment TSJS fallback | `creative.rs:1391`                   | Prepends the bundle when no `` token was seen.                       |
+
+Every per-URL decision for assets and anchors goes through `to_abs` (`creative.rs:54-75`). It trims, maps `//host` to `https://host`, rejects anything that is not an `http://` or `https://` prefix, and returns `None` when `settings.rewrite.is_excluded` matches (`creative.rs:70`). `None` leaves the attribute untouched, so an excluded anchor keeps its raw `href` and gets no `data-tsclick`.
+
+`to_abs` returns the trimmed input string, not a parsed URL. `build_signed_url_for` (`creative.rs:554`) parses it again with `url::Url` and, if parsing fails, returns the clear URL unchanged (`creative.rs:560-562`).
+
+`Rewrite::is_excluded` (`crates/trusted-server-core/src/settings.rs:645-670`) parses the URL and compares the lowercased host against raw config entries, so an entry written in uppercase never matches. `proxy::is_host_allowed` (`crates/trusted-server-core/src/proxy.rs:1242`) already lowercases both sides and uses the same exact or `*.example.com` rule.
+
+### Three entry points
+
+| Entry point                           | Caller                                                                                                                                        | Gated by `rewrite_creatives` | URL form                  | TSJS                                                   |
+| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ------------------------- | ------------------------------------------------------ |
+| `POST /auction` `adm`                 | `auction/formats.rs:398` → `process_auction_creative` (`creative.rs:985`)                                                                     | Yes                          | Root-relative             | Injected (`rewrite_creative_html`, `creative.rs:1051`) |
+| SSAT and page-bids inline `adm`       | `publisher.rs:5835` → `process_inline_auction_creative` (`creative.rs:998`)                                                                   | Yes                          | Absolute on `base_origin` | Not injected (`creative.rs:1085`)                      |
+| HTML fetched via `/first-party/proxy` | `finalize_proxied_response` (`proxy.rs:643-678`) → `CreativeHtmlProcessor` (`creative.rs:1409`) → `rewrite_proxied_html` (`creative.rs:1062`) | No                           | Root-relative             | Injected                                               |
+
+The proxy path ignoring `rewrite_creatives` is pinned by `auction_rewrite_setting_does_not_change_proxied_html_or_css_rewriting` (`proxy.rs:3582`).
+
+### Runtime pieces
+
+- `/first-party/click` (`handle_first_party_click`, `proxy.rs:1545`) validates the signature, appends `ts-ec`, and returns a `302`.
+- `/first-party/proxy-rebuild` (`handle_first_party_proxy_rebuild`, `proxy.rs:1812`) re-signs a click after creative script added or removed query parameters.
+- Both are routed in every adapter and gated by no setting.
+- The injected script is the unified bundle (`tsjs::tsjs_unified_script_tag`, `tsjs.rs:66`), which is core plus immediate modules (`publisher.rs:544-549`). `creative` is always an immediate module (`integrations/registry.rs:1216`).
+- The creative runtime (`crates/trusted-server-js/lib/src/integrations/creative/index.ts:13-16`) installs the click guard when `TsCreativeConfig.clickGuard` is true (the default) and the render guard when `renderGuard` is true (default false). `TsCreativeConfig` is defined in `crates/trusted-server-js/lib/src/shared/globals.ts:10-15`. No server setting maps to either flag.
+- The click guard (`crates/trusted-server-js/lib/src/integrations/creative/click.ts`) acts only on anchors that carry a server-set `data-tsclick`. `handleGuardedClick` returns early without one (`click.ts:433-434`), and the mutation monitor scans only `a[data-tsclick], area[data-tsclick]` (`click.ts:467`).
+
+### Config plumbing
+
+`AuctionConfig` uses `deny_unknown_fields` (`crates/trusted-server-core/src/auction_config_types.rs:16-17`). `rewrite_creatives` is a `bool` with default `true` and `skip_serializing_if = "is_default_rewrite_creatives"` (`auction_config_types.rs:46-50`, `141-148`), so the default never reaches the blob. The tests are `default_rewrite_creatives_is_not_serialized` (`auction_config_types.rs:233`), `legacy_blob_without_rewrite_creatives_preserves_rewriting` (`config_payload.rs:715`) and `disabled_rewrite_creatives_survives_blob_round_trip` (`config_payload.rs:739`).
+
+The EdgeZero typed loader applies `TRUSTED_SERVER__...` overlays at `ts config` time. It only replaces scalar leaves already present in the TOML, and the existing TOML type drives coercion (see `docs/guide/cli.md:118-124` and the CLI tests in `crates/trusted-server-cli/tests/config_env_overlay.rs:123-342`). `trusted-server.example.toml:257-268` keeps `rewrite_creatives = true` uncommented for that reason.
+
+## Design
+
+### Overview
+
+```mermaid
+flowchart TD
+    A[Creative markup] --> B{assets or clicks on?}
+    B -->|Neither| P[Pass through unchanged]
+    B -->|Either| R[Rewrite pass]
+    R --> BASE[Remove base]
+    R --> TSJS[Inject TSJS when the path injects]
+    R -->|assets on| AS[Asset handlers: normalize, should_proxy_asset, /first-party/proxy]
+    R -->|clicks on| CL[Anchor handler: normalize, should_wrap_click, /first-party/click]
+```
+
+### Setting
+
+Add one field to `AuctionConfig`, next to `rewrite_creatives`:
+
+```rust
+/// Wrap creative click-through links (``, ``) in signed
+/// `/first-party/click` redirects.
+///
+/// Unset keeps each path's existing behavior: auction creatives follow
+/// [`Self::rewrite_creatives`], and HTML fetched through `/first-party/proxy`
+/// keeps wrapping. An explicit value applies to every path. Unset is omitted
+/// from serialized config blobs, so older binaries keep loading them.
+#[serde(default, skip_serializing_if = "Option::is_none")]
+pub rewrite_clicks: Option,
+```
+
+`AuctionConfig::default()` sets it to `None`. Two methods resolve it, so no call site reads the raw field:
+
+```rust
+impl AuctionConfig {
+    /// Whether auction creatives (`/auction` and inline SSAT) wrap clicks.
+    #[must_use]
+    pub fn rewrites_auction_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(self.rewrite_creatives)
+    }
+
+    /// Whether HTML fetched through `/first-party/proxy` wraps clicks.
+    #[must_use]
+    pub fn rewrites_proxied_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(true)
+    }
+}
+```
+
+Resolution:
+
+| `rewrite_creatives` | `rewrite_clicks` | Auction assets | Auction clicks | Proxied HTML assets | Proxied HTML clicks |
+| ------------------- | ---------------- | -------------- | -------------- | ------------------- | ------------------- |
+| `true` (default)    | unset (default)  | on             | on             | on                  | on                  |
+| `false`             | unset            | off            | off            | on                  | on                  |
+| `true`              | `false`          | on             | off            | on                  | off                 |
+| `false`             | `true`           | off            | on             | on                  | on                  |
+| `true`              | `true`           | on             | on             | on                  | on                  |
+| `false`             | `false`          | off            | off            | on                  | off                 |
+
+The first two rows are today's behavior, byte for byte.
+
+`rewrite_creatives` keeps its name and type. Its doc comment, the example TOML and the guides change to say it governs asset rewriting only, plus the shared `` removal and TSJS injection described below.
+
+### URL normalization and policy (shared with #1231)
+
+This section is the coordination contract. The #1231 spec describes the same split, and the shared step's code is identical in both implementation PRs. The canonical code is in the implementation plan's "Shared step" section.
+
+`to_abs` is replaced by a pure normalizer in `creative.rs` with the same crate visibility `to_abs` has today:
+
+```rust
+pub(super) fn normalize_creative_url(url: &str) -> Option {
+    let trimmed = url.trim();
+    let is_http = trimmed
+        .get(..7)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("http://"));
+    let is_https = trimmed
+        .get(..8)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("https://"));
+    if trimmed.starts_with("//") {
+        url::Url::parse(&format!("https:{trimmed}")).ok()
+    } else if is_http || is_https {
+        url::Url::parse(trimmed).ok()
+    } else {
+        None
+    }
+}
+```
+
+It trims the input, maps a protocol-relative `//host/...` to `https://host/...`, and accepts only a case-insensitive `http://` or `https://` prefix. It parses absolute input in place without allocating, and returns `None` for empty, relative, non-http(s) or unparseable input. It needs no separate hostless check, because `url` rejects an `http(s)` URL without a host. It makes no policy decision and reads no settings.
+
+Policy lives on `Rewrite` in `settings.rs`:
+
+```rust
+impl Rewrite {
+    #[must_use]
+    pub fn should_proxy_asset(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
+
+    #[must_use]
+    pub fn should_wrap_click(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
+
+    fn is_excluded_host(&self, host: &str) -> bool {
+        self.exclude_domains
+            .iter()
+            .any(|pattern| is_host_allowed(host, pattern))
+    }
+
+    fn normalize(&mut self) { /* trim, lowercase, drop "" and "*" with a warning */ }
+}
+```
+
+- `should_proxy_asset`: not excluded. #1231 adds its include clause ("include list empty or `host` included") through `proxy::is_host_permitted`. Only #1231 makes `is_host_permitted` `pub(crate)` and imports it, because this issue doesn't use it and an unused import fails clippy.
+- `should_wrap_click`: not excluded. The #1231 include list never applies here.
+- The shared matcher is the existing case-insensitive `proxy::is_host_allowed` (`proxy.rs:1242`): an exact host, or `*.example.com` matching `example.com` and any subdomain on a dot boundary. `host` is `Url::host_str()` from the normalized URL.
+- `Rewrite::normalize` trims and lowercases `exclude_domains`, and drops empty and bare `"*"` entries with a warning. It runs from `Settings::normalize_deserialized`, right after `self.proxy.normalize()`, so it covers TOML and config-blob loads alike. This mirrors `Proxy::normalize` for `allowed_domains`.
+- `Rewrite::is_excluded` and its stale `#[allow(dead_code)]` are removed.
+
+Call sites in `creative.rs` use two private helpers:
+
+```rust
+fn asset_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_proxy_asset(host))
+    })
+}
+
+fn click_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_wrap_click(host))
+    })
+}
+```
+
+- `proxy_if_abs`, `rewrite_srcset` and `CssUrlRewriter::rewrite` call `asset_target`. `proxied_attr_value` follows through `proxy_if_abs`.
+- The anchor handler calls `click_target`.
+- `build_proxy_url` and `build_click_url` keep their `&str` parameter and receive `url.as_str()`. `build_signed_url_for` already re-parses its input (`creative.rs:560`), and re-parsing a serialized `Url` is the identity, so the signed output is unchanged.
+- `/first-party/sign` (`proxy.rs:1672-1702`) handles the target in this order:
+  1. It keeps its own request-scheme handling for `//` input, building the string into a `let protocol_relative;` binding, and borrows absolute input as-is.
+  2. It calls `normalize_creative_url`, binding the result as `target`. `None` becomes `unsupported url`.
+  3. It keeps the `missing host` guard.
+  4. It calls `should_proxy_asset`. A decline logs `log::debug!("sign request for `{host}` declined by rewrite policy")` and returns `unsupported url`.
+  5. It runs the existing `proxy.allowed_domains` check unchanged.
+
+  Malformed input that used to return `invalid url`, and a non-http scheme that used to return `unsupported scheme`, now both return `unsupported url`. All three are `TrustedServerError::Proxy`, which maps to `502`, so client-visible status is unchanged.
+
+These edge cases change, all in the safe direction:
+
+- Uppercase `exclude_domains` entries start matching, because the shared matcher is case-insensitive and entries are lowercased at load.
+- Empty and bare `"*"` exclude entries, which never matched a host, are dropped with a warning.
+- An `http(s)`-prefixed value that `url::Url` cannot parse (for example `https://exa mple.example/x`) is left byte-for-byte untouched. Before, `to_abs` accepted it and `build_signed_url_for` wrote the raw string back re-quoted, so `` became ``, and an anchor also gained `data-tsclick="https://exa mple.example/x"`. Red-phase runs against `main` reproduced both.
+
+These go in `CHANGELOG.md` under Unreleased › Fixed, in whichever PR lands the shared step.
+
+### Rewrite pass
+
+`rewrite_creative_html_impl` takes one more parameter, which stays within the seven-argument limit:
+
+```rust
+/// Which rewrite features one creative pass applies.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct CreativeFeatures {
+    assets: bool,
+    clicks: bool,
+}
+
+impl CreativeFeatures {
+    const ALL: Self = Self { assets: true, clicks: true };
+    fn any(self) -> bool { self.assets || self.clicks }
+}
+
+fn rewrite_creative_html_impl(
+    settings: &Settings,
+    markup: &str,
+    base_origin: &str,
+    inject_tsjs: bool,
+    max_output_size: usize,
+    features: CreativeFeatures,
+) -> String
+```
+
+`CreativeFeatures` is its own type because `process_auction_creative_with_rewriter` resolves it once from settings and hands it to the per-path rewriter closure. Two positional `bool`s would be easy to swap. A full options struct for all five per-call arguments was considered and dropped, because it grows the diff without simplifying any caller.
+
+The handler vector is built before `HtmlRewriter::new`:
+
+```rust
+let mut element_content_handlers = vec![/*  removal,  TSJS */];
+if features.assets {
+    element_content_handlers.extend([/* asset handlers, unchanged */]);
+}
+if features.clicks {
+    element_content_handlers.push(/* anchor handler */);
+}
+```
+
+- `` removal and the `` TSJS handler are always registered. The pass only runs when at least one switch is on.
+- Asset handlers (img, script, link, media, object, embed, input, SVG, iframe, `[style]`, `
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

+

+Offer
+Area
+"#;
+
+    fn encoded_tsurl(url: &str) -> String {
+        format!(
+            "tsurl={}",
+            url::form_urlencoded::byte_serialize(url.as_bytes()).collect::()
+        )
+    }
+
+    fn allowlist_settings() -> crate::settings::Settings {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings.rewrite.include_domains = vec!["*.example.com".to_owned()];
+        settings
+    }
+
+    fn assert_allowlist_applied(label: &str, out: &str) {
+        for listed in ALLOWLIST_LISTED {
+            assert!(
+                out.contains(&encoded_tsurl(listed)),
+                "{label}: should proxy listed URL `{listed}`: {out}"
+            );
+        }
+        for off_list in ALLOWLIST_OFF_LIST {
+            assert!(
+                !out.contains(&encoded_tsurl(off_list)),
+                "{label}: should not proxy off-list URL `{off_list}`: {out}"
+            );
+            assert!(
+                out.contains(off_list),
+                "{label}: should keep off-list URL `{off_list}` raw: {out}"
+            );
+        }
+        for landing in [
+            "https://landing.example.net/offer",
+            "https://landing.example.org/area",
+        ] {
+            assert!(
+                out.contains(&format!("/first-party/click?{}", encoded_tsurl(landing))),
+                "{label}: should wrap click `{landing}` regardless of include_domains: {out}"
+            );
+        }
+        assert_eq!(
+            out.matches("data-tsclick=").count(),
+            2,
+            "{label}: should mark both click-through links"
+        );
+    }
+
+    #[test]
+    fn include_domains_limit_asset_rewriting_on_every_html_path() {
+        let settings = allowlist_settings();
+
+        assert_allowlist_applied(
+            "auction",
+            &rewrite_creative_html(&settings, ALLOWLIST_CREATIVE),
+        );
+        assert_allowlist_applied(
+            "inline",
+            &rewrite_inline_creative_html(&settings, "https://www.example.com", ALLOWLIST_CREATIVE),
+        );
+        assert_allowlist_applied(
+            "proxied",
+            &rewrite_proxied_html(&settings, ALLOWLIST_CREATIVE),
+        );
+    }
+
+    #[test]
+    fn include_domains_limit_css_body_rewriting() {
+        let settings = allowlist_settings();
+        let css = r#"@import "https://assets.example.com/i.css";
+@import "https://cdn.example.net/i.css";
+.a { background: url(https://assets.example.com/bg.png); }
+.b { background: url(https://cdn.example.net/bg.png); }"#;
+
+        let out = rewrite_css_body(&settings, css).expect("should rewrite CSS body");
+
+        for listed in [
+            "https://assets.example.com/i.css",
+            "https://assets.example.com/bg.png",
+        ] {
+            assert!(
+                out.contains(&encoded_tsurl(listed)),
+                "should proxy listed CSS URL `{listed}`: {out}"
+            );
+        }
+        for off_list in [
+            "https://cdn.example.net/i.css",
+            "https://cdn.example.net/bg.png",
+        ] {
+            assert!(
+                out.contains(off_list) && !out.contains(&encoded_tsurl(off_list)),
+                "should keep off-list CSS URL `{off_list}` raw: {out}"
+            );
+        }
+    }
+```
+
+```rust
+    #[test]
+    fn host_in_both_lists_is_left_alone() {
+        let mut settings = allowlist_settings();
+        settings.rewrite.exclude_domains = vec!["assets.example.com".to_owned()];
+        let html = r#"x"#;
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(r#""#),
+            "should leave an asset on a host in both lists raw: {out}"
+        );
+        assert!(
+            out.contains(r#""#)
+                && !out.contains("data-tsclick"),
+            "should leave an excluded click-through link raw: {out}"
+        );
+    }
+```
+
+In `proxy.rs` tests:
+
+```rust
+    #[test]
+    fn proxy_sign_applies_rewrite_include_domains_for_get_and_post() {
+        struct Case {
+            name: &'static str,
+            include_domains: &'static [&'static str],
+            exclude_domains: &'static [&'static str],
+            allowed_domains: &'static [&'static str],
+            target: &'static str,
+            expected: StatusCode,
+        }
+
+        let cases = [
+            Case {
+                name: "listed absolute",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::OK,
+            },
+            Case {
+                name: "listed protocol-relative",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "//img.example.com/a.png",
+                expected: StatusCode::OK,
+            },
+            Case {
+                name: "off-list absolute",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "off-list protocol-relative",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "//img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "host in both lists",
+                include_domains: &["*.example.com"],
+                exclude_domains: &["img.example.com"],
+                allowed_domains: &[],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "off include list and off proxy allowlist",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &["*.example.org"],
+                target: "https://img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "listed but off proxy allowlist",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &["*.example.org"],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::FORBIDDEN,
+            },
+            Case {
+                name: "mixed-case host",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://IMG.Example.COM/a.png",
+                expected: StatusCode::OK,
+            },
+        ];
+
+        let to_owned = |values: &[&str]| -> Vec {
+            values.iter().map(|value| (*value).to_owned()).collect()
+        };
+
+        futures::executor::block_on(async {
+            for case in &cases {
+                for method in [&Method::GET, &Method::POST] {
+                    let label = format!("{} {}", method.as_str(), case.name);
+                    let mut settings = create_test_settings();
+                    settings.rewrite.include_domains = to_owned(case.include_domains);
+                    settings.rewrite.exclude_domains = to_owned(case.exclude_domains);
+                    settings.proxy.allowed_domains = to_owned(case.allowed_domains);
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example.com/first-party/sign",
+                        case.target,
+                    );
+
+                    let status =
+                        match handle_first_party_proxy_sign(&settings, &noop_services(), req).await
+                        {
+                            Ok(response) => response.status(),
+                            Err(error) => error.current_context().status_code(),
+                        };
+
+                    assert_eq!(
+                        status, case.expected,
+                        "{label} should return {}",
+                        case.expected
+                    );
+                }
+            }
+        });
+    }
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run each separately, because a wasm panic stops the binary:
+
+```bash
+cargo test-fastly -- rewrite_should_proxy_asset_honors --nocapture
+cargo test-fastly -- include_domains_limit_asset --nocapture
+cargo test-fastly -- include_domains_limit_css --nocapture
+cargo test-fastly -- proxy_sign_applies_rewrite_include_domains --nocapture
+```
+
+Expected failures:
+
+- `should_proxy_asset(`evil-example.com`) should be false`
+- `auction: should not proxy off-list URL `https://cdn.example.net/a.css``
+- `should keep off-list CSS URL `https://cdn.example.net/i.css` raw`
+- `GET off-list absolute should return 502 Bad Gateway`
+
+`host_in_both_lists_is_left_alone` already passes here: `exclude_domains` wins without the include clause. It guards precedence after Step 3.
+
+- [ ] **Step 3: Implement the include clause**
+
+In `proxy.rs:1228`, widen `is_host_permitted`'s visibility only. This belongs to the include clause, not the shared step:
+
+```rust
+pub(crate) fn is_host_permitted>(allowed_domains: &[S], host: &str) -> bool {
+```
+
+In `settings.rs`, widen the import:
+
+```rust
+use crate::proxy::{is_host_allowed, is_host_permitted};
+```
+
+Replace `should_proxy_asset` and its doc comment:
+
+```rust
+    /// Returns `true` when an asset URL on `host` should be rewritten to
+    /// `/first-party/proxy`.
+    ///
+    /// The host must not match [`Self::exclude_domains`], and must match
+    /// [`Self::include_domains`] when that list is non-empty. Matching is
+    /// case-insensitive; see [`is_host_allowed`] for the pattern rules.
+    #[must_use]
+    pub fn should_proxy_asset(&self, host: &str) -> bool {
+        !self.is_excluded_host(host) && is_host_permitted(&self.include_domains, host)
+    }
+```
+
+`is_host_permitted` returns `true` for an empty list, which is the "include list empty" half of the rule.
+
+In `proxy.rs`, update the decline log in `handle_first_party_proxy_sign`:
+
+```rust
+        log::debug!(
+            "sign request for `{host}` declined: host excluded or not in rewrite.include_domains"
+        );
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `cargo test-fastly -- rewrite_should_proxy_asset_honors include_domains_limit host_in_both_lists proxy_sign_`
+
+Expected: `12 passed`.
+
+- [ ] **Step 5: Run the full core suite and lint**
+
+Run: `cargo fmt --all && cargo test-fastly && cargo clippy-fastly`
+
+Expected: PASS, no warnings.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add crates/trusted-server-core/src/settings.rs crates/trusted-server-core/src/creative.rs crates/trusted-server-core/src/proxy.rs
+git commit --signoff -S -m "Limit creative asset rewriting to rewrite include_domains" -m "Apply the include list to every asset handler and to first-party sign. Click-through links and fetch-time proxy checks are unchanged."
+```
+
+---
+
+### Task 5: Rewrite `` once on the inline path
+
+On ``, both the `link[href]` handler and the standalone `[imagesrcset]` handler rewrite `imagesrcset`. On the root-relative paths (`/auction`, proxied HTML) the second pass sees `/first-party/proxy?...`, a relative URL, and leaves it alone. On the inline path the first pass emits absolute `https:///first-party/proxy?...` URLs, so the second pass proxies them again, producing duplicated `tsurl` and `tstoken` parameters. The fix removes the `imagesrcset` handling from `link[href]`. `[imagesrcset]` already rewrites it once on every element.
+
+**Files:**
+
+- Modify: `crates/trusted-server-core/src/creative.rs` (`link[href]` handler in `rewrite_creative_html_impl`, `:1220-1242`)
+- Test: `creative.rs` tests, before `host_in_both_lists_is_left_alone`
+
+**Interfaces:** none change.
+
+- [ ] **Step 1: Write the failing test and the root-relative guard**
+
+```rust
+    // `imagesrcset` is rewritten only by the `[imagesrcset]` handler. When the
+    // `link[href]` handler also rewrote it, the inline path's absolute proxy
+    // URLs were proxied a second time.
+    #[test]
+    fn inline_link_imagesrcset_is_proxied_once() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = r#""#;
+
+        let out = rewrite_inline_creative_html(&settings, "https://publisher.example.com", html);
+
+        let imagesrcset = out
+            .split("imagesrcset=\"")
+            .nth(1)
+            .and_then(|rest| rest.split('"').next())
+            .expect("should keep an imagesrcset attribute");
+        assert_eq!(
+            imagesrcset.matches("tsurl=").count(),
+            1,
+            "should proxy the imagesrcset candidate exactly once: {imagesrcset}"
+        );
+    }
+
+    #[test]
+    fn root_relative_link_imagesrcset_is_proxied_once() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = r#""#;
+
+        for (label, out) in [
+            ("auction", rewrite_creative_html(&settings, html)),
+            ("proxied", rewrite_proxied_html(&settings, html)),
+        ] {
+            let imagesrcset = out
+                .split("imagesrcset=\"")
+                .nth(1)
+                .and_then(|rest| rest.split('"').next())
+                .expect("should keep an imagesrcset attribute");
+            assert_eq!(
+                imagesrcset.matches("tsurl=").count(),
+                1,
+                "{label}: should proxy the imagesrcset candidate exactly once: {imagesrcset}"
+            );
+        }
+    }
+```
+
+- [ ] **Step 2: Run the tests to verify the inline one fails**
+
+Run each alone (a wasm panic aborts the whole test binary):
+
+```bash
+cargo test-fastly -p trusted-server-core --lib -- --exact creative::tests::inline_link_imagesrcset_is_proxied_once --nocapture
+cargo test-fastly -p trusted-server-core --lib -- --exact creative::tests::root_relative_link_imagesrcset_is_proxied_once
+```
+
+Expected:
+
+- The inline test panics with "should proxy the imagesrcset candidate exactly once: https://publisher.example.com/first-party/proxy?tsurl=https%3A%2F%2Fpublisher.example.com%2Ffirst-party%2Fproxy&tsurl=...".
+- The root-relative guard passes. Those paths were never affected.
+
+- [ ] **Step 3: Implement**
+
+In the `link[href]` handler, delete the `imagesrcset` block and keep the `href` handling:
+
+```rust
+                // Stylesheets and preloads
+                element!("link[href]", |el| {
+                    let rel = el
+                        .get_attribute("rel")
+                        .unwrap_or_default()
+                        .to_ascii_lowercase();
+                    if rel.contains("stylesheet")
+                        || rel.contains("preload")
+                        || rel.contains("prefetch")
+                    {
+                        // `imagesrcset` is left to the `[imagesrcset]` handler, which
+                        // also runs on this element; rewriting it here too would
+                        // proxy the inline path's absolute proxy URLs a second time.
+                        if let Some(p) =
+                            proxied_attr_value(settings, el.get_attribute("href"), base_origin)
+                        {
+                            let _ = el.set_attribute("href", &p);
+                        }
+                    }
+                    Ok(())
+                }),
+```
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: `cargo test-fastly -p trusted-server-core --lib -- imagesrcset include_domains_limit`
+
+Expected: `9 passed`. This includes both new tests, `include_domains_limit_asset_rewriting_on_every_html_path` (whose inline case uses the realistic `https://www.example.com` origin), and the existing `imagesrcset` rewrite and sanitizer tests.
+
+- [ ] **Step 5: Run the full core suite and lint**
+
+Run: `cargo fmt --all && cargo test-fastly && cargo clippy-fastly`
+
+Expected: PASS, no warnings.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add crates/trusted-server-core/src/creative.rs
+git commit --signoff -S -m "Rewrite link imagesrcset once on the inline creative path"
+```
+
+---
+
+### Task 6: Operator documentation, example config and changelog
+
+**Files:**
+
+- Modify: `trusted-server.example.toml:127-129`
+- Modify: `docs/guide/configuration.md:985-1068`
+- Modify: `docs/guide/creative-processing.md:637-700`
+- Modify: `docs/guide/first-party-proxy.md:173`, `:509-521`
+- Modify: `docs/guide/api-reference.md:565-570`
+- Modify: `CHANGELOG.md` (`## [Unreleased]`: `### Fixed` at `:34`, `### Added` at `:41`; both already exist)
+
+- [ ] **Step 1: `trusted-server.example.toml`**
+
+Replace lines 127-129 with:
+
+```toml
+# [rewrite]
+# Hosts never rewritten: assets stay direct and click-through links are not
+# wrapped. Supports "*.example.com" wildcards (matches the apex too).
+# exclude_domains = ["*.cdn.example.com"]
+# When non-empty, only asset URLs on these hosts are proxied; other assets load
+# directly. Click-through links are not affected. Set this in TOML; environment
+# overrides cannot replace arrays.
+# include_domains = ["assets.example.com", "*.img.example.com"]
+```
+
+- [ ] **Step 2: `docs/guide/configuration.md`**
+
+- Section intro (`:989`): "Control which hosts first-party rewriting covers."
+- `[rewrite]` table (`:993-995`):
+
+```markdown
+| Field             | Type          | Required         | Description                                                                                         |
+| ----------------- | ------------- | ---------------- | --------------------------------------------------------------------------------------------------- |
+| `exclude_domains` | Array[String] | No (default: []) | Hosts never rewritten, for assets and click-through links                                           |
+| `include_domains` | Array[String] | No (default: []) | When non-empty, the only hosts whose asset URLs are proxied. Does not apply to click-through links. |
+```
+
+- Add `include_domains = ["assets.example.com", "*.img.example.com"]` to the example and change the Environment Override text to "EdgeZero v0.0.4 cannot replace these arrays or address their elements by index. Edit `exclude_domains` and `include_domains` in TOML, then validate and push the file again."
+- After the exact-pattern list, add:
+
+```markdown
+**Normalization and precedence**:
+
+- Both lists are trimmed and lowercased at load, so matching is case-insensitive.
+- `exclude_domains` wins when a host matches both lists.
+- `include_domains` rejects empty entries, a bare `*`, and any `*` other than a
+  leading `*.`. Empty and bare `*` entries in `exclude_domains` are dropped with
+  a warning, because they never match a host.
+
+::: tip
+When both `include_domains` and `proxy.allowed_domains` are set, keep every
+`include_domains` host inside `proxy.allowed_domains`. Otherwise a listed asset
+is rewritten to a proxy URL that then fails with `403`.
+:::
+```
+
+- [ ] **Step 3: `docs/guide/creative-processing.md`**
+
+- Wildcard example (`:656-663`): replace `❌ cdn.example.com (no subdomain)` with `✅ cdn.example.com (base domain)`.
+- Under `### Pattern Matching`, add "Matching is case-insensitive. Entries are trimmed and lowercased at load." and change the wildcard lead-in to "`*.` matches the base domain and any subdomain".
+- After the Exclude Domains use cases, add:
+
+```markdown
+## Include Domains
+
+Limit asset rewriting to hosts you list:
+
+    [rewrite]
+    include_domains = ["assets.example.com", "*.img.example.com"]
+
+- When `include_domains` is empty (the default), every eligible asset URL is
+  proxied.
+- When it is non-empty, only asset URLs whose host matches an entry are
+  rewritten to `/first-party/proxy`. Other assets keep their original URL, load
+  directly, and do not receive the EC ID that proxied fetches append.
+- `exclude_domains` wins when a host matches both lists.
+- Click-through links (``, ``) are not affected; they are
+  still wrapped in `/first-party/click` unless excluded.
+- HTML and CSS fetched through `/first-party/proxy` go through the same rewrite
+  pass and follow the same list.
+- `/first-party/sign` declines off-list hosts with a non-`403` error, so the
+  creative runtime loads them directly.
+
+Patterns use the same syntax as `exclude_domains`.
+```
+
+(In the guide the TOML is a fenced `toml` block.)
+
+- [ ] **Step 4: `docs/guide/first-party-proxy.md`**
+
+- Rename `### URL Rewrite Exclusions` (`:509`) to `### URL Rewrite Host Lists`, and after "URLs matching these patterns will NOT be rewritten to `/first-party/proxy`." add a fenced `toml` example with `include_domains = ["assets.example.com", "*.img.example.com"]` followed by: "When `include_domains` is non-empty, only asset URLs on matching hosts are rewritten. `exclude_domains` wins when a host matches both lists, and click-through links ignore `include_domains`."
+- Before "A valid host outside a non-empty allowlist returns `403 Forbidden`." (`:173`), add the paragraph: "A host excluded by `rewrite.exclude_domains`, or outside a non-empty `rewrite.include_domains`, returns `502`. The creative runtime treats it as a non-policy failure and loads the URL directly. These checks run before `proxy.allowed_domains`."
+
+- [ ] **Step 5: `docs/guide/api-reference.md`**
+
+In `/first-party/sign` Error Responses, after the `413` bullet:
+
+```markdown
+- `502 Bad Gateway`: The target host matches `rewrite.exclude_domains` or is outside a non-empty `rewrite.include_domains`
+```
+
+- [ ] **Step 6: `CHANGELOG.md`**
+
+Append to the existing `### Fixed` list under `## [Unreleased]`:
+
+```markdown
+- Inline SSAT/page-bids creatives no longer proxy `` candidates twice. The `imagesrcset` attribute is now rewritten once, by the same handler on every creative path; previously the inline path's absolute proxy URLs were wrapped in a second `/first-party/proxy` URL.
+- `rewrite.exclude_domains` matching is now case-insensitive and ignores surrounding whitespace. Entries written with uppercase letters previously never matched and now take effect, so those hosts stop being proxied and click-wrapped. Audit `exclude_domains` for mixed-case entries before upgrading.
+```
+
+Insert at the top of the existing `### Added` list under `## [Unreleased]`:
+
+```markdown
+- `[rewrite] include_domains` limits creative asset rewriting to listed hosts. When non-empty, only asset URLs on matching hosts are proxied through `/first-party/proxy`, so only those hosts receive the EC ID on proxied fetches. Other assets keep their original URL, and `/first-party/sign` returns `502` for them so the creative runtime loads them directly. Click-through links are not affected, and `exclude_domains` still wins.
+```
+
+- [ ] **Step 7: Format and check**
+
+Run: `cd docs && npm ci && cd .. && docs/node_modules/.bin/prettier --config docs/.prettierrc --write docs/guide/configuration.md docs/guide/creative-processing.md docs/guide/first-party-proxy.md docs/guide/api-reference.md CHANGELOG.md`
+
+Expected: only `configuration.md` changes (table column alignment); the rest report `(unchanged)`.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add docs/guide/configuration.md docs/guide/creative-processing.md docs/guide/first-party-proxy.md docs/guide/api-reference.md trusted-server.example.toml CHANGELOG.md
+git commit --signoff -S -m "Document rewrite include_domains and case-insensitive exclusions"
+```
+
+---
+
+### Task 7: Full CI gate
+
+**Files:** none.
+
+- [ ] **Step 1: Format and lint**
+
+```bash
+cargo fmt --all -- --check
+cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm \
+  && cargo clippy-spin-native && cargo clippy-spin-wasm && cargo clippy-cli && cargo clippy-codegen
+```
+
+Expected: all pass.
+
+- [ ] **Step 2: Tests**
+
+```bash
+cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
+cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity
+./scripts/test-cli.sh
+```
+
+Expected on `7a0ecb4c` plus this change: `test-fastly` 3155 passed, 10 ignored (core 2908 passed, 6 ignored); `test-fastly-reuse` 219 passed; `test-axum` 43 passed; `test-cloudflare` 53 passed; `test-spin` 87 passed; parity 17 passed; CLI 733 passed.
+
+- [ ] **Step 3: JS and docs**
+
+```bash
+cd crates/trusted-server-js/lib && npx vitest run && npm run format && cd -
+cd docs && npm run format && cd -
+docs/node_modules/.bin/prettier --config docs/.prettierrc --check "*.md" ".claude/**/*.md" ".github/**/*.md" "crates/**/*.md" "scripts/**/*.md" "tinybird/**/*.md"
+```
+
+Expected: all pass. JS is unchanged.
+
+- [ ] **Step 4: Spec conformance**
+
+| Criterion                                                       | Evidence                                                                                               |
+| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| Empty list leaves existing rewrite assertions unchanged         | Task 2 Step 6 and Task 4 Step 5: full suite, no assertion edits                                        |
+| Listed proxied, off-list raw, every handler, three HTML paths   | `include_domains_limit_asset_rewriting_on_every_html_path`, `include_domains_limit_css_body_rewriting` |
+| Host in both lists left alone                                   | `host_in_both_lists_is_left_alone`, sign case "host in both lists"                                     |
+| Case-insensitive; `*.example.com` matches apex                  | Task 1 and Task 4 policy tests, `rewrite_exclude_domains_match_mixed_case_entries_from_toml`           |
+| Sign returns `502` off-list for absolute and `//`, GET and POST | `proxy_sign_applies_rewrite_include_domains_for_get_and_post`                                          |
+| Anchors wrapped regardless of allowlist                         | `assert_allowlist_applied` click assertions                                                            |
+| Default omitted from serialized blob                            | `legacy_blob_without_rewrite_include_domains_loads_with_empty_list`                                    |
+| Docs, example TOML, CHANGELOG                                   | Task 6                                                                                                 |
+
+- [ ] **Step 5: Open the PR**
+
+Use the `pr-creator` agent. Link the spec and #1231. State whether this PR introduced the shared step (Tasks 1-2) or rebased onto #1234's.
+
+## Notes
+
+- **Unparseable absolute URLs.** Before this change an absolute-looking value that `url::Url::parse` rejects had its attribute rewritten to its own value, re-quoted with double quotes. After, it is left untouched. `unparseable_absolute_url_is_left_byte_identical` and `unparseable_absolute_click_url_is_left_byte_identical` pin the new behavior. Against the pre-change code both were observed failing: `` came back re-quoted, and the anchor also gained `data-tsclick`.
diff --git a/docs/superpowers/specs/2026-10-07-1231-creative-asset-host-allowlist-design.md b/docs/superpowers/specs/2026-10-07-1231-creative-asset-host-allowlist-design.md
new file mode 100644
index 000000000..65aa552bc
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-07-1231-creative-asset-host-allowlist-design.md
@@ -0,0 +1,409 @@
+# Creative asset host allowlist
+
+**Issue:** [IABTechLab/trusted-server#1231](https://github.com/IABTechLab/trusted-server/issues/1231)
+
+**Date:** 2026-10-07
+
+**Status:** Draft (design validated by an exploratory implementation; see the plan, `docs/superpowers/plans/2026-10-07-1231-creative-asset-host-allowlist.md`)
+
+**Related:** [#1234](https://github.com/IABTechLab/trusted-server/issues/1234) (separate click-rewriting switch). Both issues replace `to_abs`; see [Coordination with #1234](#coordination-with-1234).
+
+Line numbers below are against `main` at `7a0ecb4c`.
+
+## Problem
+
+The creative rewrite pass proxies every absolute HTTP(S) asset URL in a creative through `/first-party/proxy` unless its host matches `[rewrite] exclude_domains`. An operator cannot say "proxy only these hosts".
+
+That matters for more than egress control. A proxied fetch appends the visitor's EC ID to the upstream URL (`append_ec_id`, `crates/trusted-server-core/src/proxy.rs:1204`, called at `proxy.rs:793`), so today every third-party asset host that appears in a winning creative receives `ts-ec`. A deny list cannot bound that set, because creatives name new hosts all the time.
+
+`proxy.allowed_domains` does not solve it. It is the fetch-time SSRF control, shared with the Prebid external bundle fetch. With it set, the rewriter still turns off-list URLs into `/first-party/proxy` links, and the fetch then fails with `403`. The asset breaks instead of loading directly.
+
+A second, smaller defect sits in the same code: `Rewrite::is_excluded` compares raw config entries against a host the `url` crate has already lowercased, so an entry written with an uppercase letter never matches.
+
+## Goals
+
+- Add an asset host allowlist, `[rewrite] include_domains`. Empty (the default) keeps today's behavior.
+- When non-empty, rewrite only asset URLs whose host matches the list. Every other asset URL keeps its original bytes.
+- `exclude_domains` wins when a host matches both lists.
+- Apply the allowlist to every asset handler in the pass, on all three entry points, including `srcset` candidates and CSS `url()`, `image-set()` strings and `@import`.
+- Never apply the allowlist to anchors (``, ``).
+- Apply the same policy at `/first-party/sign` for absolute and protocol-relative input, returning the same non-`403` error excluded hosts already get.
+- Use one case-insensitive host matcher for both rewrite lists, which also fixes `exclude_domains` case sensitivity.
+- Keep older binaries able to read config blobs written with the default.
+- Fix the inline path's double rewrite of ``, so each candidate is proxied exactly once on every path.
+
+## Non-goals
+
+- Click rewriting policy or a click switch. That is #1234.
+- Domain checks at `/first-party/proxy`, `/first-party/click` or `/first-party/proxy-rebuild`. Fetch-time SSRF stays with `proxy.allowed_domains`, and the click endpoints redirect any validly signed target by design (`proxy.rs:1898-1907`).
+- Narrowing EC forwarding on click redirects (`handle_first_party_click`, `proxy.rs:1545`) or on proxy redirect hops. The allowlist only decides which asset URLs become proxy URLs.
+- Cross-validating `include_domains` against `proxy.allowed_domains`. The docs recommend keeping the allowlist inside the proxy allowlist; a load-time warning can be a follow-up.
+- Client-side changes. The render guard asks `/first-party/sign` and follows its answer.
+- Environment-overlay support for the list. EdgeZero overlays only replace scalar leaves that already exist in the TOML, so the list is set in `trusted-server.toml` and published with `ts config push`, like `exclude_domains`.
+
+## Current behavior
+
+### The rewrite pass
+
+`rewrite_creative_html_impl` (`crates/trusted-server-core/src/creative.rs:1125`) is one lol_html pass. Its asset handlers live at `creative.rs:1180-1339`:
+
+| Handler                                                            | Lines     | Per-URL helper                         |
+| ------------------------------------------------------------------ | --------- | -------------------------------------- |
+| `img` `src`, `data-src`                                            | 1180-1192 | `proxy_if_abs`                         |
+| `script[src]`                                                      | 1194-1201 | `proxied_attr_value` → `proxy_if_abs`  |
+| `link[href]` (stylesheet, preload, prefetch) and its `imagesrcset` | 1203-1225 | `proxied_attr_value`, `rewrite_srcset` |
+| `video`, `audio`, `source` `src`                                   | 1227-1234 | `proxied_attr_value`                   |
+| `object[data]`, `embed[src]`                                       | 1236-1251 | `proxied_attr_value`                   |
+| `input[type=image]`                                                | 1253-1267 | `proxied_attr_value`                   |
+| SVG `image`/`use` `href`, `xlink:href`                             | 1269-1281 | `proxied_attr_value`                   |
+| `[style]` attribute                                                | 1294-1302 | `CssUrlRewriter::rewrite`              |
+| `
AdExcludedMail"#;
-    const MATRIX_FIXTURE_FRAGMENT: &str = r#"AdExcluded"#;
+    const MATRIX_FIXTURE_FRAGMENT: &str = r#"
AdExcludedMail"#;
     const INLINE_ORIGIN: &str = "https://www.example.com";
 
     #[derive(Debug, Clone, Copy)]
@@ -1001,6 +1001,8 @@ At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
 
 - [ ] **Step 2: Matrix and unset-pin tests**
 
+`MATRIX_DEFAULT_AUCTION_OUTPUT` and `MATRIX_DEFAULT_INLINE_OUTPUT` are the exact output of `MATRIX_FIXTURE_WITH_BODY` on `main`, before this change. To capture them, add a throwaway test to a scratch worktree of `main` that builds the same settings without `rewrite_clicks` (`create_test_settings()`, `sanitize_creatives = false`, `rewrite_creatives = true`, `exclude_domains = ["excluded.example.com"]`). Have it print `process_auction_creative` and `process_inline_auction_creative(.., "https://www.example.com", ..)` twice each, run it with `--nocapture`, and confirm the two runs match. Then remove the scratch worktree. Signatures are deterministic for the fixed test settings, and the injected script tag carries no hash.
+
 ```rust
     #[test]
     fn asset_and_click_switches_combine_on_auction_and_inline_paths() {
@@ -1012,8 +1014,7 @@ At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
                 for (assets, clicks) in [(true, true), (true, false), (false, true), (false, false)]
                 {
                     let settings = matrix_settings(assets, Some(clicks));
-                    let label =
-                        format!("{path:?} assets={assets} clicks={clicks} body={has_body}");
+                    let label = format!("{path:?} assets={assets} clicks={clicks} body={has_body}");
                     let prefix = match path {
                         CreativePath::Auction => "",
                         CreativePath::Inline => INLINE_ORIGIN,
@@ -1024,7 +1025,7 @@ At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
                     let expected_proxied = match (assets, has_body) {
                         (false, _) => 0,
                         (true, true) => 4,
-                        (true, false) => 1,
+                        (true, false) => 3,
                     };
                     assert_eq!(
                         out.matches("/first-party/proxy?tsurl=").count(),
@@ -1042,12 +1043,9 @@ At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
                         "{label}: image raw only when assets are off: {out}"
                     );
 
-                    // Each wrapped anchor carries the click URL in href and data-tsclick.
-                    let expected_click_urls = match (clicks, has_body) {
-                        (false, _) => 0,
-                        (true, true) => 4,
-                        (true, false) => 2,
-                    };
+                    // Each wrapped `` and `` carries the click URL in
+                    // href and data-tsclick.
+                    let expected_click_urls = if clicks { 4 } else { 0 };
                     assert_eq!(
                         out.matches("/first-party/click?tsurl=").count(),
                         expected_click_urls,
@@ -1072,12 +1070,10 @@ At the top of `creative.rs` `mod tests`, after the `use super::{...}` block:
                         out.contains(r#"href="https://excluded.example.com/page""#),
                         "{label}: excluded anchor always stays raw: {out}"
                     );
-                    if has_body {
-                        assert!(
-                            out.contains(r#"href="mailto:ads@example.com""#),
-                            "{label}: mailto anchor always stays raw: {out}"
-                        );
-                    }
+                    assert!(
+                        out.contains(r#"href="mailto:ads@example.com""#),
+                        "{label}: mailto anchor always stays raw: {out}"
+                    );
 
                     assert_eq!(
                         out.contains("
AdExcludedMail"#;
+    const MATRIX_DEFAULT_INLINE_OUTPUT: &str = r#"
AdExcludedMail"#;
+
+    #[test]
+    fn default_settings_rewrite_matrix_fixture_byte_for_byte_as_before() {
+        let settings = matrix_settings(true, None);
+
+        for (path, expected) in [
+            (CreativePath::Auction, MATRIX_DEFAULT_AUCTION_OUTPUT),
+            (CreativePath::Inline, MATRIX_DEFAULT_INLINE_OUTPUT),
+        ] {
+            assert_eq!(
+                process_for_path(&settings, path, MATRIX_FIXTURE_WITH_BODY),
+                expected,
+                "{path:?}: default settings should match the pre-rewrite_clicks output exactly"
+            );
+        }
+    }
 ```
 
 - [ ] **Step 3: End-to-end tests**
@@ -1215,7 +1232,7 @@ Run each test alone, as described in the environment notes. Expected:
 - `asset_and_click_switches_combine_on_auction_and_inline_paths` panics with "Auction assets=true clicks=false body=true: click URL count".
 - `convert_to_openrtb_response_wraps_clicks_without_rewriting_assets` panics with "should wrap the landing link". Nothing was rewritten.
 - `build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off` panics with "should leave the landing link raw". The anchor was wrapped.
-- `unset_rewrite_clicks_preserves_existing_output` **passes**. It pins today's behavior.
+- `unset_rewrite_clicks_preserves_existing_output` and `default_settings_rewrite_matrix_fixture_byte_for_byte_as_before` **pass**. They pin today's behavior.
 
 - [ ] **Step 5: Add `CreativeFeatures`**
 
@@ -1346,12 +1363,18 @@ Public wrappers: pass `CreativeFeatures::ALL` as the sixth argument in `rewrite_
 - `rewrite_inline_creative_html`: "its click guard is unnecessary for click URLs that are already absolute here".
 - `auction/endpoints.rs` response doc: "Creative HTML is inlined in each bid's `adm` field after optional sanitization ([`auction.sanitize_creatives`][...]). First-party asset rewriting ([`auction.rewrite_creatives`][...]) and click wrapping ([`auction.rewrite_clicks`][...]) are enabled by default. Bidder `` removal and creative TSJS injection run when either is on." This replaces the stale "mandatory server-side sanitization".
 - `auction/formats.rs`, on `convert_to_openrtb_response`: name `rewrite_clicks` next to `rewrite_creatives` and add the intra-doc link target `/// [`AuctionConfig::rewrite_clicks`]: crate::auction_config_types::AuctionConfig::rewrite_clicks`. Add `clicks {}` to the "Processed creative" debug log, fed by `settings.auction.rewrites_auction_clicks()`.
+- `auction/formats.rs`, the comment above `serialize_renderer` in `convert_to_openrtb_response`: "sanitization is opt-in, asset rewriting and click wrapping are on by default, and with all three disabled the creative ships exactly as the bidder returned it."
+- `creative.rs` module docs:
+  - add the goal "Route click-through links through a signed first-party click redirect.", and call the proxied list "asset URLs";
+  - add key behaviors for ``/`` wrapping with `data-tsclick`, for `exclude_domains` applying to assets and links, and for `` removal;
+  - add a "Switches" list. Auction creatives rewrite assets on `rewrite_creatives` and wrap links on `rewrites_auction_clicks`, and an unset `rewrite_clicks` follows `rewrite_creatives`. `` removal and TSJS run when either is on, and with both off the pass is skipped. Proxied HTML always rewrites assets and wraps links unless `rewrite_clicks` is explicitly `false`. The public wrappers rewrite both.
+  - Link the `AuctionConfig` items with reference definitions at the end of the module docs, because `creative.rs` does not import `AuctionConfig`.
 
 - [ ] **Step 9: Verify green**
 
 Run: `cargo fmt --all && cargo test-fastly -p trusted-server-core --lib -- creative:: auction::formats publisher::`
 
-Expected: `551 passed; 0 failed`, including every pre-existing rewrite test.
+Expected: `554 passed; 0 failed`, including every pre-existing rewrite test.
 
 Run: `cargo clippy-fastly`
 
@@ -1380,11 +1403,12 @@ Before `html_response_rewrite_preserves_non_standard_port`:
 ```rust
     #[test]
     fn proxied_html_click_wrapping_follows_explicit_rewrite_clicks() {
-        let html = r#""#;
+        let html = r#"Excluded"#;
         // (rewrite_creatives, rewrite_clicks, expect wrapped clicks)
         let cases = [
             (true, None, true),
             (false, None, true),
+            (true, Some(true), true),
             (true, Some(false), false),
             (false, Some(false), false),
             (false, Some(true), true),
@@ -1394,9 +1418,9 @@ Before `html_response_rewrite_preserves_non_standard_port`:
             let mut settings = create_test_settings();
             settings.auction.rewrite_creatives = rewrite_creatives;
             settings.auction.rewrite_clicks = rewrite_clicks;
-            let label = format!(
-                "rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
-            );
+            settings.rewrite.exclude_domains = vec!["excluded.example.com".to_owned()];
+            let label =
+                format!("rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}");
             let req = build_http_request(Method::GET, "https://edge.example.com/first-party/proxy");
             let mut response = build_http_response(StatusCode::OK, EdgeBody::from(html));
             response.headers_mut().insert(
@@ -1418,16 +1442,36 @@ Before `html_response_rewrite_preserves_non_standard_port`:
                 body.contains("/first-party/proxy?tsurl="),
                 "{label}: proxied HTML always proxies assets: {body}"
             );
+            // A wrapped landing link carries the click URL in href and data-tsclick.
             assert_eq!(
-                body.contains("/first-party/click?tsurl="),
-                expect_clicks,
+                body.matches("/first-party/click?tsurl=").count(),
+                if expect_clicks { 2 } else { 0 },
                 "{label}: click wrapping in proxied HTML: {body}"
             );
             assert_eq!(
-                body.contains("data-tsclick"),
-                expect_clicks,
+                body.matches("data-tsclick").count(),
+                usize::from(expect_clicks),
                 "{label}: data-tsclick in proxied HTML: {body}"
             );
+            if !expect_clicks {
+                let landing_href = body
+                    .split("Excluded"#),
+                "{label}: excluded link always stays raw: {body}"
+            );
+            assert!(
+                !body.contains(": {body}"
+            );
             assert!(
                 body.contains("/static/tsjs="),
                 "{label}: proxied HTML always receives the runtime: {body}"
@@ -1453,6 +1497,18 @@ In `rewrite_proxied_html`, replace `CreativeFeatures::ALL` with:
 
 Append this paragraph to its doc comment: "Asset rewriting here is unconditional. Click wrapping follows an explicit `[auction] rewrite_clicks`; when that is unset, links keep being wrapped regardless of `rewrite_creatives`."
 
+Fix the `CreativeHtmlProcessor` doc comment, which names `rewrite_creative_html` although the processor calls `rewrite_proxied_html`:
+
+```rust
+/// Stream processor for HTML fetched through `/first-party/proxy` that rewrites
+/// asset and click-through URLs to first-party endpoints.
+///
+/// This processor buffers input chunks and processes the complete HTML document
+/// when the stream ends, using [`rewrite_proxied_html`] internally. Asset URLs are
+/// always proxied; links are wrapped unless `[auction] rewrite_clicks` is
+/// explicitly `false`.
+```
+
 - [ ] **Step 4: Verify green**
 
 Run: `cargo test-fastly -p trusted-server-core --lib -- proxy::tests::proxied_html proxy::tests::auction_rewrite_setting_does_not_change`
@@ -1587,6 +1643,17 @@ rewrite_creatives = true
 # rewrite_clicks = true
 ```
 
+In the `sanitize_creatives` comment just below, replace the sentence that starts "Note that with `rewrite_creatives = true`" so it names both switches:
+
+```toml
+# (executable markup preserved). Note that with `rewrite_creatives` or
+# `rewrite_clicks` on, the adm is still not untouched: eligible asset URLs
+# and/or links are rewritten, bidder `` elements are removed, and the
+# creative TSJS runtime is injected. Enable it whenever creatives can render
+# in a context that shares the publisher origin (its primary defence there);
+# leave it off when creatives render in a
+```
+
 - [ ] **Step 4: Commit**
 
 ```bash
@@ -1625,18 +1692,24 @@ git commit --signoff -S -m "Document rewrite_clicks in the example config and pi
   - an explicit `rewrite_clicks` applies to links in proxied HTML, and unset keeps wrapping them.
 - Warning block:
   - add "any explicit `rewrite_clicks`" to the serialized values;
-  - rollback note: "Older binaries cannot express clicks off with assets on; rolling back from that configuration re-enables click wrapping.";
+  - rollback note: "Older binaries tie click wrapping to `rewrite_creatives` in both directions: rolling back from `rewrite_creatives = true` with `rewrite_clicks = false` turns click wrapping back on, and rolling back from `rewrite_creatives = false` with `rewrite_clicks = true` turns it off.";
   - overlay note: "`rewrite_clicks` is unset by default and so cannot appear as a TOML leaf until you set it; add `rewrite_clicks = true` or `false` under `[auction]` before relying on `TRUSTED_SERVER__AUCTION__REWRITE_CLICKS`."
 
 - [ ] **Step 2: `creative-processing.md`**
 
 - Processing Triggers, item 1, append: "Click-through links are wrapped according to `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset."
 - Item 2, append: "An explicitly set `[auction].rewrite_clicks` also applies to links in proxied HTML."
+- "Auction Rewrite Control" intro: "Three auction settings control the processing applied to winning-bid `adm` ... Sanitization is opt-in (default `false`) and asset rewriting is enabled by default. Click wrapping is controlled by `rewrite_clicks`, which follows `rewrite_creatives` when unset; see [Assets and clicks](#assets-and-clicks)." Add `# rewrite_clicks unset: follows rewrite_creatives` to its TOML snippet.
+- Its `sanitize_creatives` × `rewrite_creatives` table: asset URLs follow `rewrite_creatives` and links follow `rewrite_clicks`.
+  - `false`/`false`: "Asset URLs stay direct. With `rewrite_clicks` unset or `false`, deliver the creative exactly as the bidder returned it (subject to the size cap); with `rewrite_clicks = true`, wrap links as described in [Assets and clicks](#assets-and-clicks)."
+  - `true`/`false`: "...then deliver without asset rewriting. Sanitizer-accepted external resource and inline CSS URLs remain direct; links follow `rewrite_clicks` and stay direct when it is unset or `false`."
+  - `false`/`true`: "Rewrite eligible resource/CSS URLs ... removing any bidder `` element. Links are wrapped unless `rewrite_clicks = false`. Executable markup is preserved."
 - After the sanitization warning in "Auction Rewrite Control", add an `### Assets and clicks` subsection containing:
   - the TOML snippet (`rewrite_creatives = true`, `# rewrite_clicks = false`);
   - the four-row table (assets × clicks → asset URLs, links, ``, TSJS);
   - "SSAT/page-bids follows the same table with absolute URLs and never injects TSJS. `exclude_domains` applies to both assets and links.";
-  - the `clickGuard` versus `rewrite_clicks` paragraph.
+  - the `clickGuard` versus `rewrite_clicks` paragraph;
+  - a `renderGuard` note: "With `rewrite_creatives = false` and `rewrite_clicks = true`, `POST /auction` still injects TSJS. A creative that turns on `tsCreativeConfig.renderGuard` can therefore still send assets inserted by its own script through `/first-party/sign` and `/first-party/proxy`, even though the server left the markup's asset URLs direct."
 - "Anchors (Click Tracking)": add `**Controlled by**: [auction].rewrite_clicks (follows rewrite_creatives when unset)`, switch the example to `advertiser.example.com`, and show the `data-tsclick` attribute.
 
 - [ ] **Step 3: Remaining guides**
@@ -1653,7 +1726,13 @@ git commit --signoff -S -m "Document rewrite_clicks in the example config and pi
 Under `### Added`, as the first entry:
 
 ```markdown
-- Added `[auction].rewrite_clicks` to control creative click-through wrapping (``/`` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `` removal and creative TSJS injection run when either setting is on. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary; older binaries cannot express clicks off with assets on.
+- Added `[auction].rewrite_clicks` to control creative click-through wrapping (``/`` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `` removal and creative TSJS injection run when either setting is on. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary. Older binaries tie click wrapping to `rewrite_creatives` in both directions, so rolling back turns clicks back on for `rewrite_creatives = true` with `rewrite_clicks = false`, and off for `rewrite_creatives = false` with `rewrite_clicks = true`.
+```
+
+In the existing Unreleased `rewrite_creatives`/`sanitize_creatives` entry, replace "(proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only)." with:
+
+```markdown
+(asset URL conversion to `/first-party/proxy`, bidder `` removal; creative TSJS injection on `POST /auction` only). Click-through wrapping is controlled by `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset.
 ```
 
 - [ ] **Step 5: Format**
@@ -1694,16 +1773,16 @@ Expected: every command exits 0. The validation run's results are in the spec ha
 
 - [ ] **Step 2: Map completion criteria to tests**
 
-| Criterion                               | Test                                                                                                                                                                                |
-| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Unset output unchanged                  | `unset_rewrite_clicks_preserves_existing_output`, existing rewrite tests, unset rows of `proxied_html_click_wrapping_follows_explicit_rewrite_clicks`                               |
-| Four-combination matrix on both paths   | `asset_and_click_switches_combine_on_auction_and_inline_paths`                                                                                                                      |
-| Clicks off: raw href, no `data-tsclick` | matrix, `build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off`                                                                                                                |
-| Clicks on, assets off                   | matrix, `convert_to_openrtb_response_wraps_clicks_without_rewriting_assets`                                                                                                         |
-| `exclude_domains` applies to anchors    | matrix (excluded anchor), `rewrite_click_urls_excludes_blacklisted_domains`, `exclude_domains_match_case_insensitively_in_the_rewrite_pass`                                         |
-| Proxied HTML rule                       | `proxied_html_click_wrapping_follows_explicit_rewrite_clicks`                                                                                                                       |
-| Default omitted from the blob           | `default_rewrite_clicks_is_unset_and_not_serialized`, `legacy_blob_without_rewrite_clicks_follows_rewrite_creatives`, `rewrite_clicks_environment_override_is_ignored_without_leaf` |
-| Docs and CHANGELOG                      | Task 6                                                                                                                                                                              |
+| Criterion                               | Test                                                                                                                                                                                                                     |
+| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Unset output unchanged                  | `default_settings_rewrite_matrix_fixture_byte_for_byte_as_before`, `unset_rewrite_clicks_preserves_existing_output`, existing rewrite tests, unset rows of `proxied_html_click_wrapping_follows_explicit_rewrite_clicks` |
+| Four-combination matrix on both paths   | `asset_and_click_switches_combine_on_auction_and_inline_paths`                                                                                                                                                           |
+| Clicks off: raw href, no `data-tsclick` | matrix, `build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off`                                                                                                                                                     |
+| Clicks on, assets off                   | matrix, `convert_to_openrtb_response_wraps_clicks_without_rewriting_assets`                                                                                                                                              |
+| `exclude_domains` applies to anchors    | matrix (excluded anchor), `rewrite_click_urls_excludes_blacklisted_domains`, `exclude_domains_match_case_insensitively_in_the_rewrite_pass`                                                                              |
+| Proxied HTML rule                       | `proxied_html_click_wrapping_follows_explicit_rewrite_clicks`                                                                                                                                                            |
+| Default omitted from the blob           | `default_rewrite_clicks_is_unset_and_not_serialized`, `legacy_blob_without_rewrite_clicks_follows_rewrite_creatives`, `rewrite_clicks_environment_override_is_ignored_without_leaf`                                      |
+| Docs and CHANGELOG                      | Task 6                                                                                                                                                                                                                   |
 
 - [ ] **Step 3: Hand off**
 
diff --git a/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md b/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md
index 97bed4dfc..99a1b18ea 100644
--- a/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md
+++ b/docs/superpowers/specs/2026-10-07-1234-creative-click-rewrite-switch-design.md
@@ -233,15 +233,17 @@ fn click_target(settings: &Settings, raw: &str) -> Option {
   4. It calls `should_proxy_asset`. A decline logs `log::debug!("sign request for `{host}` declined by rewrite policy")` and returns `unsupported url`.
   5. It runs the existing `proxy.allowed_domains` check unchanged.
 
-  Malformed input that used to return `invalid url`, and a non-http scheme that used to return `unsupported scheme`, now both return `unsupported url`. All three are `TrustedServerError::Proxy`, which maps to `502`, so client-visible status is unchanged.
+  Malformed input that used to return `invalid url`, and a non-http scheme that used to return `unsupported scheme`, now both return `unsupported url`. All three are `TrustedServerError::Proxy`, which maps to `502`, so those statuses are unchanged.
+
+  One client-visible status does change. A host matched only by a mixed-case `exclude_domains` entry used to pass the exclusion check, so sign returned `200`, or `403` when the host was also outside `proxy.allowed_domains`. It now returns `502`, like every other excluded host.
 
 These edge cases change, all in the safe direction:
 
-- Uppercase `exclude_domains` entries start matching, because the shared matcher is case-insensitive and entries are lowercased at load.
+- Uppercase `exclude_domains` entries start matching, because the shared matcher is case-insensitive and entries are lowercased at load. `/first-party/sign` returns `502` for those hosts, where it used to return `200`, or `403` for hosts outside `proxy.allowed_domains`.
 - Empty and bare `"*"` exclude entries, which never matched a host, are dropped with a warning.
 - An `http(s)`-prefixed value that `url::Url` cannot parse (for example `https://exa mple.example/x`) is left byte-for-byte untouched. Before, `to_abs` accepted it and `build_signed_url_for` wrote the raw string back re-quoted, so `` became ``, and an anchor also gained `data-tsclick="https://exa mple.example/x"`. Red-phase runs against `main` reproduced both.
 
-These go in `CHANGELOG.md` under Unreleased › Fixed, in whichever PR lands the shared step.
+These go in `CHANGELOG.md` under Unreleased › Fixed, as one entry, in whichever PR lands the shared step. The exact text is in the plan's "Shared step" section.
 
 ### Rewrite pass
 
@@ -339,6 +341,7 @@ Both run whenever the pass runs, which is when either switch is on.
 
 - `` removal protects root-relative `/first-party/proxy` URLs, root-relative `/first-party/click` URLs and the root-relative `/static/tsjs=` script alike, so it is needed whichever feature is on. On the inline path URLs are absolute, but removal is kept for parity, as `inline_rewrite_strips_base_elements` (`creative.rs`) pins today.
 - TSJS injection on `/auction` with clicks on delivers the click guard, which is required.
+- TSJS injection on `/auction` with only clicks on also gives a creative that sets `tsCreativeConfig.renderGuard` a way to send assets inserted by its own script through `/first-party/sign` and `/first-party/proxy`, although the server left the markup's asset URLs direct. `creative-processing.md` says so.
 - TSJS injection on `/auction` with only assets on is usually inert. The click guard has no `data-tsclick` anchor to act on, and the render guard defaults to off (`renderGuard: false`). The reason to keep it is that toggling `rewrite_clicks` then changes only anchors and nothing else in the output. Assets-only output is today's default output with the anchors left raw, which keeps the four-combination matrix easy to reason about and test.
 - The inline path never injects TSJS, as today.
 - With both switches off, nothing runs: no `` removal, no TSJS, byte-for-byte pass-through.
@@ -410,7 +413,7 @@ The shared test set is listed in the plan's "Shared step" section and is identic
 
 One table-driven test (`asset_and_click_switches_combine_on_auction_and_inline_paths`) covers both entry points, `process_auction_creative` (`/auction`) and `process_inline_auction_creative` (inline), over `(rewrite_creatives, rewrite_clicks)` in `{(true, Some(true)), (true, Some(false)), (false, Some(true)), (false, Some(false))}`. Each case names its combination and path in every assertion message.
 
-Fixture: a ``, ``, `srcset`, an inline `style` with `url()`, a `
AdExcludedMail"#;
+    const MATRIX_FIXTURE_FRAGMENT: &str = r#"
AdExcludedMail"#;
+    const INLINE_ORIGIN: &str = "https://www.example.com";
+
+    #[derive(Debug, Clone, Copy)]
+    enum CreativePath {
+        Auction,
+        Inline,
+    }
+
+    fn process_for_path(settings: &Settings, path: CreativePath, html: &str) -> String {
+        match path {
+            CreativePath::Auction => process_auction_creative(settings, html),
+            CreativePath::Inline => {
+                super::process_inline_auction_creative(settings, INLINE_ORIGIN, html)
+            }
+        }
+    }
+
+    fn matrix_settings(assets: bool, clicks: Option) -> Settings {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings.auction.sanitize_creatives = false;
+        settings.auction.rewrite_creatives = assets;
+        settings.auction.rewrite_clicks = clicks;
+        settings.rewrite.exclude_domains = vec!["excluded.example.com".to_owned()];
+        settings
+    }
+
     fn rewrite_srcset_attr(attr_name: &str, attr_value: &str) -> String {
         let settings = crate::test_support::tests::create_test_settings();
         let html = format!(r#""#);
@@ -1713,6 +1834,141 @@ mod tests {
         );
     }
 
+    #[test]
+    fn asset_and_click_switches_combine_on_auction_and_inline_paths() {
+        for path in [CreativePath::Auction, CreativePath::Inline] {
+            for (fixture, has_body) in [
+                (MATRIX_FIXTURE_WITH_BODY, true),
+                (MATRIX_FIXTURE_FRAGMENT, false),
+            ] {
+                for (assets, clicks) in [(true, true), (true, false), (false, true), (false, false)]
+                {
+                    let settings = matrix_settings(assets, Some(clicks));
+                    let label = format!("{path:?} assets={assets} clicks={clicks} body={has_body}");
+                    let prefix = match path {
+                        CreativePath::Auction => "",
+                        CreativePath::Inline => INLINE_ORIGIN,
+                    };
+
+                    let out = process_for_path(&settings, path, fixture);
+
+                    let expected_proxied = match (assets, has_body) {
+                        (false, _) => 0,
+                        (true, true) => 4,
+                        (true, false) => 3,
+                    };
+                    assert_eq!(
+                        out.matches("/first-party/proxy?tsurl=").count(),
+                        expected_proxied,
+                        "{label}: proxied asset URL count: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(&format!("src=\"{prefix}/first-party/proxy?tsurl=")),
+                        assets,
+                        "{label}: image proxied only when assets are on: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(r#"src="https://cdn.example.com/ad.png""#),
+                        !assets,
+                        "{label}: image raw only when assets are off: {out}"
+                    );
+
+                    // Each wrapped `` and `` carries the click URL in
+                    // href and data-tsclick.
+                    let expected_click_urls = if clicks { 4 } else { 0 };
+                    assert_eq!(
+                        out.matches("/first-party/click?tsurl=").count(),
+                        expected_click_urls,
+                        "{label}: click URL count: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(&format!("href=\"{prefix}/first-party/click?tsurl=")),
+                        clicks,
+                        "{label}: anchor wrapped only when clicks are on: {out}"
+                    );
+                    assert_eq!(
+                        out.contains(r#"href="https://landing.example.com/page""#),
+                        !clicks,
+                        "{label}: anchor raw only when clicks are off: {out}"
+                    );
+                    assert_eq!(
+                        out.contains("data-tsclick"),
+                        clicks,
+                        "{label}: data-tsclick only when clicks are on: {out}"
+                    );
+                    assert!(
+                        out.contains(r#"href="https://excluded.example.com/page""#),
+                        "{label}: excluded anchor always stays raw: {out}"
+                    );
+                    assert!(
+                        out.contains(r#"href="mailto:ads@example.com""#),
+                        "{label}: mailto anchor always stays raw: {out}"
+                    );
+
+                    assert_eq!(
+                        out.contains(" removed whenever either switch is on: {out}"
+                    );
+                    let expected_tsjs =
+                        usize::from(matches!(path, CreativePath::Auction) && (assets || clicks));
+                    assert_eq!(
+                        out.matches("/static/tsjs=").count(),
+                        expected_tsjs,
+                        "{label}: TSJS injected once on /auction when either switch is on: {out}"
+                    );
+                    if !assets && !clicks {
+                        assert_eq!(
+                            out, fixture,
+                            "{label}: should pass through byte for byte with both switches off"
+                        );
+                    }
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn unset_rewrite_clicks_preserves_existing_output() {
+        for path in [CreativePath::Auction, CreativePath::Inline] {
+            let unset = matrix_settings(true, None);
+            let explicit = matrix_settings(true, Some(true));
+            assert_eq!(
+                process_for_path(&unset, path, MATRIX_FIXTURE_WITH_BODY),
+                process_for_path(&explicit, path, MATRIX_FIXTURE_WITH_BODY),
+                "{path:?}: unset rewrite_clicks with rewrite_creatives = true should match full rewriting"
+            );
+
+            let disabled = matrix_settings(false, None);
+            assert_eq!(
+                process_for_path(&disabled, path, MATRIX_FIXTURE_WITH_BODY),
+                MATRIX_FIXTURE_WITH_BODY,
+                "{path:?}: unset rewrite_clicks with rewrite_creatives = false should pass through"
+            );
+        }
+    }
+
+    // Captured from `main` before `rewrite_clicks` existed: default settings must
+    // keep producing these bytes on both auction paths.
+    const MATRIX_DEFAULT_AUCTION_OUTPUT: &str = r#"
AdExcludedMail"#;
+    const MATRIX_DEFAULT_INLINE_OUTPUT: &str = r#"
AdExcludedMail"#;
+
+    #[test]
+    fn default_settings_rewrite_matrix_fixture_byte_for_byte_as_before() {
+        let settings = matrix_settings(true, None);
+
+        for (path, expected) in [
+            (CreativePath::Auction, MATRIX_DEFAULT_AUCTION_OUTPUT),
+            (CreativePath::Inline, MATRIX_DEFAULT_INLINE_OUTPUT),
+        ] {
+            assert_eq!(
+                process_for_path(&settings, path, MATRIX_FIXTURE_WITH_BODY),
+                expected,
+                "{path:?}: default settings should match the pre-rewrite_clicks output exactly"
+            );
+        }
+    }
+
     fn normalized(raw: &str) -> Option {
         normalize_creative_url(raw).map(|url| url.as_str().to_owned())
     }
diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs
index 61bc7efe5..254118f6f 100644
--- a/crates/trusted-server-core/src/publisher.rs
+++ b/crates/trusted-server-core/src/publisher.rs
@@ -22668,6 +22668,50 @@ mod tests {
             );
         }
 
+        #[test]
+        fn build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off() {
+            let mut settings = test_settings();
+            settings.auction.rewrite_creatives = true;
+            settings.auction.rewrite_clicks = Some(false);
+            settings.publisher.domain = "example.com".to_string();
+
+            let mut winning_bids = HashMap::new();
+            let mut bid = make_bid(
+                "atf_sidebar_ad",
+                1.50,
+                "examplessp",
+                "abc123",
+                "https://ssp.example.com/win",
+                "https://ssp.example.com/bill",
+            );
+            bid.creative = Some(
+                ""
+                    .to_string(),
+            );
+            winning_bids.insert("atf_sidebar_ad".to_string(), bid);
+
+            let map = build_bid_map(&winning_bids, PriceGranularity::Dense, &settings, "", false);
+            let adm = map
+                .get("atf_sidebar_ad")
+                .and_then(|v| v.as_object())
+                .and_then(|o| o.get("adm"))
+                .and_then(|v| v.as_str())
+                .expect("should include a rewritten adm");
+
+            assert!(
+                adm.contains("https://example.com/first-party/proxy?tsurl="),
+                "should still proxy the image with an absolute URL, got: {adm}"
+            );
+            assert!(
+                adm.contains("href=\"https://landing.example.com/page\""),
+                "should leave the landing link raw, got: {adm}"
+            );
+            assert!(
+                !adm.contains("data-tsclick") && !adm.contains("/first-party/click"),
+                "should not wrap the landing link, got: {adm}"
+            );
+        }
+
         #[test]
         fn build_bid_map_uses_request_origin_for_inline_urls() {
             // The inline adm's absolute first-party URLs must resolve against the

From 7c7e202d1ad06c96bd2b728df2e28870bcf49345 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:25 +0530
Subject: [PATCH 145/230] Apply an explicit rewrite_clicks to HTML fetched
 through the first-party proxy

Signed-off-by: dhruv8sh 
---
 crates/trusted-server-core/src/creative.rs | 16 ++++-
 crates/trusted-server-core/src/proxy.rs    | 78 ++++++++++++++++++++++
 2 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 1fcb9b84c..074e5f35d 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -1125,6 +1125,10 @@ pub fn rewrite_creative_html(settings: &Settings, markup: &str) -> String {
 /// own [`MAX_REWRITABLE_BODY_SIZE`] rather than the per-creative auction cap:
 /// a proxied document is a whole page, not an `adm`, and legitimately exceeds
 /// 1 MiB. The creative runtime is still injected so click mediation survives.
+///
+/// Asset rewriting here is unconditional. Click wrapping follows an explicit
+/// `[auction] rewrite_clicks`; when that is unset, links keep being wrapped
+/// regardless of `rewrite_creatives`.
 #[must_use]
 pub fn rewrite_proxied_html(settings: &Settings, markup: &str) -> String {
     rewrite_creative_html_impl(
@@ -1133,7 +1137,10 @@ pub fn rewrite_proxied_html(settings: &Settings, markup: &str) -> String {
         "",
         true,
         MAX_REWRITABLE_BODY_SIZE,
-        CreativeFeatures::ALL,
+        CreativeFeatures {
+            assets: true,
+            clicks: settings.auction.rewrites_proxied_clicks(),
+        },
     )
 }
 
@@ -1510,10 +1517,13 @@ fn rewrite_creative_html_impl(
     rewritten
 }
 
-/// Stream processor for creative HTML that rewrites URLs to first-party proxy.
+/// Stream processor for HTML fetched through `/first-party/proxy` that rewrites
+/// asset and click-through URLs to first-party endpoints.
 ///
 /// This processor buffers input chunks and processes the complete HTML document
-/// when the stream ends, using `rewrite_creative_html` internally.
+/// when the stream ends, using [`rewrite_proxied_html`] internally. Asset URLs are
+/// always proxied; links are wrapped unless `[auction] rewrite_clicks` is
+/// explicitly `false`.
 pub struct CreativeHtmlProcessor<'a> {
     settings: &'a Settings,
     buffer: Vec,
diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs
index 6d1805c25..1b9139753 100644
--- a/crates/trusted-server-core/src/proxy.rs
+++ b/crates/trusted-server-core/src/proxy.rs
@@ -3655,6 +3655,84 @@ mod tests {
         );
     }
 
+    #[test]
+    fn proxied_html_click_wrapping_follows_explicit_rewrite_clicks() {
+        let html = r#"Excluded"#;
+        // (rewrite_creatives, rewrite_clicks, expect wrapped clicks)
+        let cases = [
+            (true, None, true),
+            (false, None, true),
+            (true, Some(true), true),
+            (true, Some(false), false),
+            (false, Some(false), false),
+            (false, Some(true), true),
+        ];
+
+        for (rewrite_creatives, rewrite_clicks, expect_clicks) in cases {
+            let mut settings = create_test_settings();
+            settings.auction.rewrite_creatives = rewrite_creatives;
+            settings.auction.rewrite_clicks = rewrite_clicks;
+            settings.rewrite.exclude_domains = vec!["excluded.example.com".to_owned()];
+            let label =
+                format!("rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}");
+            let req = build_http_request(Method::GET, "https://edge.example.com/first-party/proxy");
+            let mut response = build_http_response(StatusCode::OK, EdgeBody::from(html));
+            response.headers_mut().insert(
+                header::CONTENT_TYPE,
+                HeaderValue::from_static("text/html; charset=utf-8"),
+            );
+
+            let body = response_body_string(
+                finalize(
+                    &settings,
+                    &req,
+                    "https://cdn.example.com/creative.html",
+                    response,
+                )
+                .expect("should finalize proxied HTML"),
+            );
+
+            assert!(
+                body.contains("/first-party/proxy?tsurl="),
+                "{label}: proxied HTML always proxies assets: {body}"
+            );
+            // A wrapped landing link carries the click URL in href and data-tsclick.
+            assert_eq!(
+                body.matches("/first-party/click?tsurl=").count(),
+                if expect_clicks { 2 } else { 0 },
+                "{label}: click wrapping in proxied HTML: {body}"
+            );
+            assert_eq!(
+                body.matches("data-tsclick").count(),
+                usize::from(expect_clicks),
+                "{label}: data-tsclick in proxied HTML: {body}"
+            );
+            if !expect_clicks {
+                let landing_href = body
+                    .split("Excluded"#),
+                "{label}: excluded link always stays raw: {body}"
+            );
+            assert!(
+                !body.contains(": {body}"
+            );
+            assert!(
+                body.contains("/static/tsjs="),
+                "{label}: proxied HTML always receives the runtime: {body}"
+            );
+        }
+    }
+
     #[test]
     fn html_response_rewrite_preserves_non_standard_port() {
         // Verify that HTML rewriting preserves non-standard ports in sub-resource URLs.

From 6b82d6de78fdb9fc986b5304f4073e311b3207c2 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:44 +0530
Subject: [PATCH 146/230] Document rewrite_clicks in the example config and pin
 its overlay behavior

Signed-off-by: dhruv8sh 
---
 .../tests/config_env_overlay.rs               | 71 +++++++++++++++++++
 trusted-server.example.toml                   | 29 +++++---
 2 files changed, 91 insertions(+), 9 deletions(-)

diff --git a/crates/trusted-server-cli/tests/config_env_overlay.rs b/crates/trusted-server-cli/tests/config_env_overlay.rs
index d81b0366a..06cc3c058 100644
--- a/crates/trusted-server-cli/tests/config_env_overlay.rs
+++ b/crates/trusted-server-cli/tests/config_env_overlay.rs
@@ -29,6 +29,7 @@ ids = ["trusted_server_secrets"]
 "#;
 const REWRITE_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CREATIVES";
 const SANITIZE_ENV: &str = "TRUSTED_SERVER__AUCTION__SANITIZE_CREATIVES";
+const CLICKS_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CLICKS";
 const GAM_ATTRIBUTION_ENV: &str = "TRUSTED_SERVER__INTEGRATIONS__GPT__GAM_ATTRIBUTION_ENABLED";
 const AD_TEMPLATES_ENABLED_ENV: &str = "TRUSTED_SERVER__CREATIVE_OPPORTUNITIES__ENABLED";
 const PROVIDER_ENDPOINT_ENV: &str = "TRUSTED_SERVER__AUCTION__PROVIDERS__PBS-MAIN__ENDPOINT";
@@ -82,6 +83,76 @@ fn validate_with_overlay(project: &MigratedProject, raw_value: &str) -> Output {
         .expect("should run ts config validate")
 }
 
+fn pushed_auction_with_env(
+    project: &MigratedProject,
+    key: &str,
+    raw_value: &str,
+) -> serde_json::Value {
+    let output = Command::new(env!("CARGO_BIN_EXE_ts"))
+        .args(["config", "push", "--adapter", "axum", "--manifest"])
+        .arg(&project.manifest_path)
+        .arg("--app-config")
+        .arg(&project.config_path)
+        .args(["--yes", "--no-diff"])
+        .current_dir(project.directory.path())
+        .env(key, raw_value)
+        .output()
+        .expect("should run ts config push");
+    assert!(
+        output.status.success(),
+        "config push should succeed: {}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+
+    let local_store_path = project
+        .directory
+        .path()
+        .join(".edgezero/local-config-trusted_server_config.json");
+    let local_store: serde_json::Value = serde_json::from_str(
+        &fs::read_to_string(local_store_path).expect("should read pushed local config"),
+    )
+    .expect("should parse local config store");
+    let envelope_json = local_store
+        .as_object()
+        .and_then(|entries| entries.values().next())
+        .and_then(serde_json::Value::as_str)
+        .expect("should contain a blob envelope");
+    let envelope: serde_json::Value =
+        serde_json::from_str(envelope_json).expect("should parse blob envelope");
+    envelope["data"]["auction"].clone()
+}
+
+#[test]
+fn rewrite_clicks_environment_override_applies_when_leaf_present() {
+    let project = migrated_project();
+    let mut document = fs::read_to_string(&project.config_path)
+        .expect("should read migrated config")
+        .parse::()
+        .expect("should parse migrated config");
+    document["auction"]["rewrite_clicks"] = value(true);
+    fs::write(&project.config_path, document.to_string()).expect("should write config");
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert_eq!(
+        auction["rewrite_clicks"],
+        serde_json::Value::Bool(false),
+        "pushed config should contain the rewrite_clicks environment override"
+    );
+}
+
+#[test]
+fn rewrite_clicks_environment_override_is_ignored_without_leaf() {
+    let project = migrated_project();
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert!(
+        auction.get("rewrite_clicks").is_none(),
+        "an override for a missing leaf should be ignored and the unset default omitted"
+    );
+}
+
 #[test]
 fn config_validate_explains_legacy_provider_list_migration() {
     let project = migrated_project();
diff --git a/trusted-server.example.toml b/trusted-server.example.toml
index e841515ae..d4789570c 100644
--- a/trusted-server.example.toml
+++ b/trusted-server.example.toml
@@ -261,18 +261,29 @@ proxy = "enabled"
 # Keep disabled until provider endpoints, routes, and profile values below are
 # replaced with deployment-specific settings.
 enabled = false
-# Rewrite winning-bid creative HTML to first-party endpoints (default true). Set
-# false to skip proxy/click-URL conversion and creative TSJS injection.
-# Sanitization is controlled separately by `sanitize_creatives` below. Restore
-# true before rolling back to an older binary that rejects unknown fields.
+# Rewrite winning-bid creative asset URLs (images, scripts, styles, media,
+# iframes, CSS url()) to first-party /first-party/proxy endpoints (default
+# true). Set false to leave asset URLs direct. Bidder  removal and
+# creative TSJS injection run when this or rewrite_clicks is on. Sanitization
+# is controlled separately by `sanitize_creatives` below. Restore true before
+# rolling back to an older binary that rejects unknown fields.
 rewrite_creatives = true
+# Wrap creative click-through links (, ) in signed
+# /first-party/click redirects. Unset (the default) follows rewrite_creatives
+# for auction creatives and keeps wrapping links in HTML fetched through
+# /first-party/proxy; a set value applies to both. Leave it commented out to
+# keep that behavior: any explicit value is stored in the pushed config, and
+# binaries older than this setting reject it. Uncomment it before relying on
+# TRUSTED_SERVER__AUCTION__REWRITE_CLICKS, which cannot create a missing leaf.
+# rewrite_clicks = true
 # Strip executable markup (script/object/embed/form/...) from winning-bid adm,
 # removing those elements together with their inner content. Defaults to false
-# (executable markup preserved). Note that with `rewrite_creatives = true` the
-# adm is still not untouched: eligible URLs are rewritten, bidder ``
-# elements are removed, and the creative TSJS runtime is injected. Enable it
-# whenever creatives can render in a context that shares the publisher origin
-# (its primary defence there); leave it off when creatives render in a
+# (executable markup preserved). Note that with `rewrite_creatives` or
+# `rewrite_clicks` on, the adm is still not untouched: eligible asset URLs
+# and/or links are rewritten, bidder `` elements are removed, and the
+# creative TSJS runtime is injected. Enable it whenever creatives can render
+# in a context that shares the publisher origin (its primary defence there);
+# leave it off when creatives render in a
 # foreign-origin frame (e.g. the Prebid Universal Creative inside the ad
 # server's iframe), since it removes script-based creatives entirely and would
 # blank slots on a script-heavy demand stack.

From 3c1012dad635891059bb765b8bd1b945f09b5fc4 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:44 +0530
Subject: [PATCH 147/230] Document the rewrite_clicks creative setting

Signed-off-by: dhruv8sh 
---
 CHANGELOG.md                                  |  3 +-
 .../trusted-server-core/src/auction/README.md |  6 +-
 docs/guide/api-reference.md                   |  5 +-
 docs/guide/auction-orchestration.md           | 11 ++--
 docs/guide/configuration.md                   | 57 +++++++++-------
 docs/guide/creative-processing.md             | 66 +++++++++++++++----
 6 files changed, 103 insertions(+), 45 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index c370373ff..ce3904fea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,7 +41,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
+- Added `[auction].rewrite_clicks` to control creative click-through wrapping (``/`` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `` removal and creative TSJS injection run when either setting is on. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary. Older binaries tie click wrapping to `rewrite_creatives` in both directions, so rolling back turns clicks back on for `rewrite_creatives = true` with `rewrite_clicks = false`, and off for `rewrite_creatives = false` with `rewrite_clicks = true`.
+- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (asset URL conversion to `/first-party/proxy`, bidder `` removal; creative TSJS injection on `POST /auction` only). Click-through wrapping is controlled by `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset. Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.
 - Added typed APS renderer transport for direct auctions and GAM/Prebid Universal Creative, using a minimized one-bid envelope, a fragment-bound nonce, and an opaque sandboxed renderer endpoint.
diff --git a/crates/trusted-server-core/src/auction/README.md b/crates/trusted-server-core/src/auction/README.md
index 8a745eabc..7b0ae585b 100644
--- a/crates/trusted-server-core/src/auction/README.md
+++ b/crates/trusted-server-core/src/auction/README.md
@@ -108,8 +108,10 @@ as `pbs-main`.
 response.
 
 - `sanitize_creatives = true` strips executable markup. It is opt-in.
-- `rewrite_creatives = true` rewrites eligible URLs through first-party routes
-  and removes bidder `` elements. It is enabled by default.
+- `rewrite_creatives = true` rewrites eligible asset URLs through first-party
+  routes and removes bidder `` elements. It is enabled by default.
+- `rewrite_clicks` wraps creative links in `/first-party/click`. Unset, it
+  follows `rewrite_creatives`.
 - The publisher inline delivery path uses absolute first-party URLs without
   injecting the direct endpoint's creative runtime.
 - Creatives over the configured hard cap are rejected.
diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md
index e444ec147..bc95ff9ac 100644
--- a/docs/guide/api-reference.md
+++ b/docs/guide/api-reference.md
@@ -168,8 +168,9 @@ curl -i "https://edge.example.com/_ts/clear-tester"
 
 Browser and programmatic auction endpoint. It accepts the Trusted Server ad-unit
 request shape and returns an OpenRTB response. Creative markup follows the
-independent `[auction].sanitize_creatives` and `[auction].rewrite_creatives`
-settings; sanitization is opt-in and rewriting is enabled by default.
+independent `[auction].sanitize_creatives`, `[auction].rewrite_creatives` and
+`[auction].rewrite_clicks` settings; sanitization is opt-in, asset rewriting is
+enabled by default, and click wrapping follows `rewrite_creatives` unless set.
 
 Configured provider IDs appear in response metadata and provider responses.
 Consumers that previously matched the literal provider name `prebid` must use
diff --git a/docs/guide/auction-orchestration.md b/docs/guide/auction-orchestration.md
index edbc38471..46347c80e 100644
--- a/docs/guide/auction-orchestration.md
+++ b/docs/guide/auction-orchestration.md
@@ -161,7 +161,7 @@ sequenceDiagram
       Note right of Client: Fragment-bound nonce and one-time acknowledgement
No allow-same-origin on the outer frame
     else Ordinary creative
       Client->>Client: Inject winning creative
Render iframe
Load creative resources
-      Note right of Client: Default: first-party proxy/click URLs
rewrite_creatives=false: accepted external URLs remain direct
+      Note right of Client: Default: first-party proxy/click URLs
rewrite_creatives=false: asset URLs direct; clicks follow rewrite_clicks
     end
     deactivate Client
   end
@@ -195,7 +195,7 @@ AuctionOrchestrator.run_auction()
 Convert OrchestrationResult → OpenRTB 2.x Response
   │
   ├─[sanitize_creatives=true] Strip executable markup
-  ├─[rewrite_creatives=true] Rewrite URLs and inject creative TSJS
+  ├─[rewrite_creatives or rewrite_clicks] Rewrite assets and/or links, inject creative TSJS
   ├─ Add ext.orchestrator metadata
   └─ Set consent and optional EID response headers
 ```
@@ -638,8 +638,11 @@ HTML rewriter (`lol_html`) that converts eligible external resource and click
 URLs to signed first-party paths, adds `data-tsclick`, rewrites inline CSS
 `url(...)` values, removes bidder-supplied `` elements, and injects the
 unified creative TSJS runtime exactly once, whether or not the bidder supplied a
-`` element. In every mode, a creative
-larger than the 1 MiB per-creative cap is rejected and its `adm` is dropped.
+`` element. `rewrite_clicks` (unset by default, following
+`rewrite_creatives`) controls click-through link wrapping separately; see
+[Creative Processing](/guide/creative-processing#assets-and-clicks). In every
+mode, a creative larger than the 1 MiB per-creative cap is rejected and its
+`adm` is dropped.
 
 ```toml
 [auction]
diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md
index f7220d58c..908239d2b 100644
--- a/docs/guide/configuration.md
+++ b/docs/guide/configuration.md
@@ -1927,15 +1927,16 @@ provider or bidder route.
 
 ### `[auction]`
 
-| Field                  | Type    | Default            | Description                                                    |
-| ---------------------- | ------- | ------------------ | -------------------------------------------------------------- |
-| `enabled`              | Boolean | `false`            | Enable the auction orchestrator                                |
-| `sanitize_creatives`   | Boolean | `false`            | Strip executable markup from winning-bid `adm` before delivery |
-| `rewrite_creatives`    | Boolean | `true`             | Rewrite winning-bid `adm` through first-party endpoints        |
-| `timeout_ms`           | Integer | `2000`             | Logical auction budget in milliseconds                         |
-| `mediator`             | String  | `None`             | Optional separate `adserver_mock` mediator                     |
-| `creative_store`       | String  | `"creative_store"` | Deprecated; creatives are delivered inline                     |
-| `allowed_context_keys` | Array   | `[]`               | Request context keys admitted into the auction                 |
+| Field                  | Type    | Default            | Description                                                                    |
+| ---------------------- | ------- | ------------------ | ------------------------------------------------------------------------------ |
+| `enabled`              | Boolean | `false`            | Enable the auction orchestrator                                                |
+| `sanitize_creatives`   | Boolean | `false`            | Strip executable markup from winning-bid `adm` before delivery                 |
+| `rewrite_creatives`    | Boolean | `true`             | Rewrite winning-bid asset URLs through first-party endpoints                   |
+| `rewrite_clicks`       | Boolean | unset              | Wrap creative links in `/first-party/click`; unset follows `rewrite_creatives` |
+| `timeout_ms`           | Integer | `2000`             | Logical auction budget in milliseconds                                         |
+| `mediator`             | String  | `None`             | Optional separate `adserver_mock` mediator                                     |
+| `creative_store`       | String  | `"creative_store"` | Deprecated; creatives are delivered inline                                     |
+| `allowed_context_keys` | Array   | `[]`               | Request context keys admitted into the auction                                 |
 
 Creative markup delivered by `POST /auction` and the publisher SSAT/page-bids
 path is processed by two independent passes. With `sanitize_creatives = true`
@@ -1943,23 +1944,29 @@ path is processed by two independent passes. With `sanitize_creatives = true`
 and event handlers) is stripped together with its inner content. This blanks
 script-based creatives, so enable it only when creatives render in a context
 that shares the publisher's origin. With `rewrite_creatives = true` (the
-default), eligible absolute or protocol-relative resource and click URLs not
-excluded by rewrite configuration are converted to signed first-party
-endpoints, and any bidder-supplied `` element is removed. The
-`POST /auction` path emits root-relative endpoints and injects the creative TSJS
-runtime exactly once, whether or not the bidder supplied a ``, since bare
+default), eligible absolute or protocol-relative asset URLs not excluded by
+rewrite configuration are converted to signed `/first-party/proxy` endpoints.
+`rewrite_clicks` controls click-through links (``, ``)
+separately: when it is unset, it follows `rewrite_creatives`, so existing
+configs keep their current behavior. Any bidder-supplied `` element is
+removed whenever either setting is on. The `POST /auction` path emits
+root-relative endpoints and, when either setting is on, injects the creative
+TSJS runtime exactly once, whether or not the bidder supplied a ``, since bare
 fragments are the common `adm` shape. The foreign-origin SSAT renderer emits
-absolute endpoints and does not inject that bundle. With both disabled, `adm`
-ships exactly as the bidder returned it, except that a creative larger than the
+absolute endpoints and does not inject that bundle. With all three disabled,
+`adm` ships exactly as the bidder returned it, except that a creative larger than the
 1 MiB per-creative cap is rejected in every mode and its `adm` is dropped.
-Accepted external URLs are not host allowlisted by the sanitizer. Neither
-setting affects HTML or CSS fetched through `/first-party/proxy`. See
+Accepted external URLs are not host allowlisted by the sanitizer.
+`rewrite_creatives` and `sanitize_creatives` do not affect HTML or CSS fetched
+through `/first-party/proxy`. An explicitly set `rewrite_clicks` does apply to
+links in proxied HTML; unset, proxied HTML keeps wrapping links. See
 [Creative Processing](/guide/creative-processing#auction-rewrite-control).
 
 ::: warning Existing configs, upgrade sequencing, and rollback
 Default values are omitted from stored JSON; non-default values
-(`sanitize_creatives = true`, `rewrite_creatives = false`) are serialized, and
-older `AuctionConfig` schemas reject unknown fields.
+(`sanitize_creatives = true`, `rewrite_creatives = false`, and any explicit
+`rewrite_clicks`) are serialized, and older `AuctionConfig` schemas reject
+unknown fields.
 
 **Upgrading:** binaries that predate `sanitize_creatives` reject a blob that
 carries it, so in a rolling deployment upgrade the binary **first**, then push
@@ -1973,14 +1980,20 @@ on new code, while an explicit `true` fails startup on old code.
 **Rolling back:** before reverting to a binary that does not know a field,
 remove that field's non-default value (and any environment override), run
 `ts config validate`, push the resulting default-compatible blob, and only then
-roll back the binary.
+roll back the binary. Older binaries tie click wrapping to `rewrite_creatives`
+in both directions: rolling back from `rewrite_creatives = true` with
+`rewrite_clicks = false` turns click wrapping back on, and rolling back from
+`rewrite_creatives = false` with `rewrite_clicks = true` turns it off.
 
 **Environment overlays:** The pinned EdgeZero loader cannot create missing TOML
 leaves. Existing configs must add **both** leaves under `[auction]`
 (`rewrite_creatives` and `sanitize_creatives`) before
 `TRUSTED_SERVER__AUCTION__REWRITE_CREATIVES` /
 `TRUSTED_SERVER__AUCTION__SANITIZE_CREATIVES` can take effect. An override for a
-missing leaf is silently ignored.
+missing leaf is silently ignored. `rewrite_clicks` is unset by default and so
+cannot appear as a TOML leaf until you set it; add `rewrite_clicks = true` or
+`false` under `[auction]` before relying on
+`TRUSTED_SERVER__AUCTION__REWRITE_CLICKS`.
 :::
 
 ### Provider map
diff --git a/docs/guide/creative-processing.md b/docs/guide/creative-processing.md
index 10dcb62d5..823ae1260 100644
--- a/docs/guide/creative-processing.md
+++ b/docs/guide/creative-processing.md
@@ -47,8 +47,8 @@ publisher's first-party domain. This provides:
 
 The creative rewriters are invoked by independent delivery paths:
 
-1. **Auction `adm`**: Winning-bid HTML returned by `POST /auction` is optionally sanitized (`[auction].sanitize_creatives`, opt-in, default `false`) and rewritten (`[auction].rewrite_creatives`, default `true`).
-2. **First-party proxy**: Non-streaming `text/html` and `text/css` responses fetched through `/first-party/proxy` are rewritten independently of the auction setting.
+1. **Auction `adm`**: Winning-bid HTML returned by `POST /auction` is optionally sanitized (`[auction].sanitize_creatives`, opt-in, default `false`) and rewritten (`[auction].rewrite_creatives`, default `true`). Click-through links are wrapped according to `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset.
+2. **First-party proxy**: Non-streaming `text/html` and `text/css` responses fetched through `/first-party/proxy` are rewritten independently of the auction setting. An explicitly set `[auction].rewrite_clicks` also applies to links in proxied HTML.
 3. **Integration processing**: Publisher HTML integrations use their own registration and configuration controls.
 
 ::: info Streaming Mode
@@ -57,26 +57,28 @@ When `with_streaming()` is enabled in `ProxyRequestConfig`, proxied HTML/CSS pro
 
 ## Auction Rewrite Control
 
-Two independent auction settings control the processing applied to winning-bid
-`adm` returned by `POST /auction` and delivered through the publisher
-SSAT/page-bids path. Sanitization is opt-in (default `false`); rewriting is
-enabled by default:
+Three auction settings control the processing applied to winning-bid `adm`
+returned by `POST /auction` and delivered through the publisher SSAT/page-bids
+path. Sanitization is opt-in (default `false`) and asset rewriting is enabled
+by default. Click wrapping is controlled by `rewrite_clicks`, which follows
+`rewrite_creatives` when unset; see [Assets and clicks](#assets-and-clicks).
 
 ```toml
 [auction]
 sanitize_creatives = false
 rewrite_creatives = true
+# rewrite_clicks unset: follows rewrite_creatives
 ```
 
 Regardless of mode, a creative larger than the 1 MiB per-creative cap is
 rejected and its `adm` is dropped.
 
-| `sanitize_creatives` | `rewrite_creatives` | Auction winning-bid `adm` behavior                                                                                                                                                                                                                                         |
-| -------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `false` (default)    | `false`             | Deliver the creative exactly as the bidder returned it (subject to the size cap).                                                                                                                                                                                          |
-| `true`               | `false`             | Strip executable markup (`script`/`object`/`embed`/`form`, event handlers) with its inner content, then deliver without rewriting. Sanitizer-accepted external resource, click, and inline CSS URLs remain direct.                                                         |
-| `false`              | `true` (default)    | Rewrite eligible resource/CSS and click URLs in the raw bidder markup to signed first-party endpoints, removing any bidder `` element. Executable markup is preserved.                                                                                               |
-| `true`               | `true`              | Sanitize first, then rewrite. `POST /auction` emits root-relative endpoints and injects creative TSJS exactly once, whether or not the bidder supplied a ``; SSAT/page-bids emits absolute endpoints for its foreign-origin renderer and does not inject the bundle. |
+| `sanitize_creatives` | `rewrite_creatives` | Auction winning-bid `adm` behavior                                                                                                                                                                                                                                                          |
+| -------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `false` (default)    | `false`             | Asset URLs stay direct. With `rewrite_clicks` unset or `false`, deliver the creative exactly as the bidder returned it (subject to the size cap); with `rewrite_clicks = true`, wrap links as described in [Assets and clicks](#assets-and-clicks).                                         |
+| `true`               | `false`             | Strip executable markup (`script`/`object`/`embed`/`form`, event handlers) with its inner content, then deliver without asset rewriting. Sanitizer-accepted external resource and inline CSS URLs remain direct; links follow `rewrite_clicks` and stay direct when it is unset or `false`. |
+| `false`              | `true` (default)    | Rewrite eligible resource/CSS URLs in the raw bidder markup to signed first-party endpoints, removing any bidder `` element. Links are wrapped unless `rewrite_clicks = false`. Executable markup is preserved.                                                                       |
+| `true`               | `true`              | Sanitize first, then rewrite. `POST /auction` emits root-relative endpoints and injects creative TSJS exactly once, whether or not the bidder supplied a ``; SSAT/page-bids emits absolute endpoints for its foreign-origin renderer and does not inject the bundle.                  |
 
 ::: warning Sanitization blanks script-based creatives
 Sanitization removes `script`/`object`/`embed`/`form` and similar elements
@@ -90,6 +92,39 @@ accepted HTTP(S) URLs may cause the browser to contact external creative hosts
 directly.
 :::
 
+### Assets and clicks
+
+`rewrite_creatives` governs asset URLs. `rewrite_clicks` governs click-through
+links and follows `rewrite_creatives` when unset:
+
+```toml
+[auction]
+rewrite_creatives = true
+# Unset: follows rewrite_creatives. Set false to keep landing links direct.
+# rewrite_clicks = false
+```
+
+| Assets (`rewrite_creatives`) | Clicks (`rewrite_clicks`) | Asset URLs           | Links                                 | `` | TSJS on `POST /auction` |
+| ---------------------------- | ------------------------- | -------------------- | ------------------------------------- | -------- | ----------------------- |
+| `true`                       | `true` or unset           | `/first-party/proxy` | `/first-party/click` + `data-tsclick` | removed  | injected                |
+| `true`                       | `false`                   | `/first-party/proxy` | direct                                | removed  | injected                |
+| `false`                      | `true`                    | direct               | `/first-party/click` + `data-tsclick` | removed  | injected                |
+| `false`                      | `false` or unset          | direct               | direct                                | kept     | not injected            |
+
+SSAT/page-bids follows the same table with absolute URLs and never injects TSJS.
+`exclude_domains` applies to both assets and links.
+
+The client-side `tsCreativeConfig.clickGuard` flag is separate. `rewrite_clicks`
+decides whether the server emits signed click URLs; `clickGuard` decides whether
+the creative runtime re-signs them after creative script changes their query
+parameters. With `rewrite_clicks` off there is nothing for the guard to act on.
+
+With `rewrite_creatives = false` and `rewrite_clicks = true`, `POST /auction`
+still injects TSJS. A creative that turns on `tsCreativeConfig.renderGuard` can
+therefore still send assets inserted by its own script through
+`/first-party/sign` and `/first-party/proxy`, even though the server left the
+markup's asset URLs direct.
+
 ::: info Runtime protections inside the sandboxed creative iframe
 Creatives rendered by Trusted Server's own path run in a sandboxed iframe
 **without** `allow-same-origin`, i.e. an opaque origin. The injected creative
@@ -307,15 +342,18 @@ If the iframe content itself contains HTML, it will be processed recursively. Ea
 
 **Rewrite Mode**: Uses `/first-party/click` for direct redirects
 
+**Controlled by**: `[auction].rewrite_clicks` (follows `rewrite_creatives` when unset)
+
 **Example**:
 
 ```html
 
-Buy Now
+Buy Now
 
 
 Buy Now
 ```

From d8f043909b4907fd514ea6a34a1c8344c1ce1893 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:02:38 +0530
Subject: [PATCH 148/230] Split creative URL normalization from rewrite host
 policy

Replace to_abs with normalize_creative_url and move exclusion policy to Rewrite::should_proxy_asset and Rewrite::should_wrap_click, sharing the case-insensitive proxy host matcher. Normalize exclude_domains at load. This is the shared step for #1231 and #1234.

Signed-off-by: dhruv8sh 
---
 CHANGELOG.md                               |   1 +
 crates/trusted-server-core/src/creative.rs | 271 +++++++++++----------
 crates/trusted-server-core/src/proxy.rs    | 108 +++++---
 crates/trusted-server-core/src/settings.rs | 150 +++++++++---
 4 files changed, 331 insertions(+), 199 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 101087b8d..c370373ff 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `rewrite.exclude_domains` matching is now case-insensitive and ignores surrounding whitespace. Entries written with uppercase letters previously never matched and now take effect, so those hosts stop being proxied and click-wrapped, and `/first-party/sign` now rejects them with `502`; it used to sign them, or return `403` when the host was also outside `proxy.allowed_domains`. Empty and bare `"*"` `exclude_domains` entries, which never matched a host, are dropped at load with a warning. Absolute `http(s)` creative URLs that cannot be parsed (for example, a host containing a space) are now left untouched; previously the attribute was re-quoted and a link also gained `data-tsclick`. Audit `exclude_domains` for mixed-case entries before upgrading.
 - `[auction].allowed_context_keys` now serializes in sorted, deduplicated order, so ESI template-cache fingerprints and `ts config diff`/`push` envelope hashes are stable across loads. Template fingerprints also sort object keys independently of `serde_json/preserve_order`. Existing envelopes may show a one-time allowlist reorder after upgrading; push once to settle it. The updated fingerprint format causes one template-cache miss per cached page after deployment.
 - TSJS-generated envelopes now send `trustedServer.params.storedRequest: false`, preventing accidental PBS stored lookups without suppressing eligible non-PBS demand. PBS filters unusable impressions after overrides; explicit `true` and omission in valid envelopes retain inline-first stored fallback. A malformed envelope disables stored fallback for the entire slot, including independent direct demand left unusable after overrides. Publisher intent survives repeated and refresh auctions. Deploy compatible server admission everywhere before serving the new JS, and retain it during rollback while cached clients remain. See the Prebid deployment guide.
 - Protocol-relative creative URLs now honor `rewrite.exclude_domains`, so excluded creative assets stay direct and excluded absolute or protocol-relative URLs submitted to `/first-party/sign` are rejected.
diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 578d86d68..8352c77c0 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -22,9 +22,10 @@
 //!   `blob:`, `about:`.
 //!
 //! Notable helpers:
-//! - `to_abs(&Settings, &str) -> Option`: Normalizes a string to an absolute URL if
-//!   it is already absolute or protocol-relative; returns `None` otherwise or for
-//!   non-network schemes.
+//! - `normalize_creative_url(&str) -> Option`: Normalizes an absolute or
+//!   protocol-relative http(s) URL; returns `None` for relative input,
+//!   non-network schemes and unparseable values. Host policy lives on
+//!   [`crate::settings::Rewrite`].
 //! - `rewrite_srcset(&Settings, &str) -> String`: Rewrites `srcset`/`imagesrcset`
 //!   values, proxying absolute candidates and preserving descriptors (`1x`,
 //!   `1.5x`, `100w`).
@@ -49,29 +50,45 @@ use std::io;
 /// Responses larger than this will be rejected to prevent memory exhaustion.
 const MAX_REWRITABLE_BODY_SIZE: usize = 10 * 1024 * 1024; // 10 MB
 
-// Helper: normalize to absolute URL if http/https or protocol-relative. Otherwise None.
-// Checks against the rewrite blacklist to exclude configured domains/patterns from proxying.
-pub(super) fn to_abs(settings: &Settings, u: &str) -> Option {
-    let t = u.trim();
-    if t.is_empty() {
-        return None;
-    }
-
-    let lower = t.to_ascii_lowercase();
-    let absolute = if t.starts_with("//") {
-        format!("https:{t}")
-    } else if lower.starts_with("http://") || lower.starts_with("https://") {
-        t.to_owned()
+/// Normalizes a creative URL to an absolute HTTP(S) [`url::Url`].
+///
+/// Trims surrounding whitespace, resolves a protocol-relative `//host/...`
+/// against `https:`, and accepts only `http://` and `https://` input (ASCII
+/// case-insensitive). Returns `None` for empty, relative, non-network-scheme or
+/// unparseable input. Applies no host policy; see
+/// [`crate::settings::Rewrite::should_proxy_asset`] and
+/// [`crate::settings::Rewrite::should_wrap_click`].
+pub(super) fn normalize_creative_url(url: &str) -> Option {
+    let trimmed = url.trim();
+    let is_http = trimmed
+        .get(..7)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("http://"));
+    let is_https = trimmed
+        .get(..8)
+        .is_some_and(|prefix| prefix.eq_ignore_ascii_case("https://"));
+    if trimmed.starts_with("//") {
+        url::Url::parse(&format!("https:{trimmed}")).ok()
+    } else if is_http || is_https {
+        url::Url::parse(trimmed).ok()
     } else {
-        return None;
-    };
-
-    // Match exclusions against the same absolute URL used for rewriting.
-    if settings.rewrite.is_excluded(&absolute) {
-        return None;
+        None
     }
+}
 
-    Some(absolute)
+/// Normalizes `raw` and returns it when asset policy allows proxying its host.
+fn asset_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_proxy_asset(host))
+    })
+}
+
+/// Normalizes `raw` and returns it when click policy allows wrapping its host.
+fn click_target(settings: &Settings, raw: &str) -> Option {
+    normalize_creative_url(raw).filter(|url| {
+        url.host_str()
+            .is_some_and(|host| settings.rewrite.should_wrap_click(host))
+    })
 }
 
 /// Maximum number of nested parser scopes [`rewrite_style_urls`] will enter.
@@ -399,14 +416,14 @@ impl CssUrlRewriter<'_> {
         parser: &cssparser::Parser<'_, '_>,
         shape: UrlShape,
     ) {
-        let Some(absolute) = to_abs(self.settings, value) else {
+        let Some(target) = asset_target(self.settings, value) else {
             return;
         };
         let token_end = parser.position().byte_index();
         let result = (|| -> fmt::Result {
             self.out
                 .write_str(&self.style[self.write_pos..token_start])?;
-            let proxied = build_proxy_url(self.settings, &absolute, self.base_origin);
+            let proxied = build_proxy_url(self.settings, target.as_str(), self.base_origin);
             if let UrlShape::Function(name) = shape {
                 self.out.write_str(name)?;
                 self.out.write_char('(')?;
@@ -637,7 +654,7 @@ pub(super) fn build_click_url(settings: &Settings, clear_url: &str, base_origin:
 
 #[inline]
 pub(super) fn proxy_if_abs(settings: &Settings, val: &str, base_origin: &str) -> Option {
-    to_abs(settings, val).map(|abs| build_proxy_url(settings, &abs, base_origin))
+    asset_target(settings, val).map(|url| build_proxy_url(settings, url.as_str(), base_origin))
 }
 
 /// Split a srcset/imagesrcset attribute into candidate strings.
@@ -723,8 +740,8 @@ pub(super) fn rewrite_srcset(settings: &Settings, srcset: &str, base_origin: &st
         let mut parts = it.split_whitespace();
         let url = parts.next().unwrap_or("");
         let descriptor = parts.collect::>().join(" ");
-        let rewritten = if let Some(abs) = to_abs(settings, url) {
-            build_proxy_url(settings, &abs, base_origin)
+        let rewritten = if let Some(target) = asset_target(settings, url) {
+            build_proxy_url(settings, target.as_str(), base_origin)
         } else {
             url.to_owned()
         };
@@ -1282,9 +1299,9 @@ fn rewrite_creative_html_impl(
                 // Click-through links
                 element!("a[href], area[href]", |el| {
                     if let Some(href) = el.get_attribute("href")
-                        && let Some(abs) = to_abs(settings, &href)
+                        && let Some(target) = click_target(settings, &href)
                     {
-                        let click = build_click_url(settings, &abs, base_origin);
+                        let click = build_click_url(settings, target.as_str(), base_origin);
                         let _ = el.set_attribute("href", &click);
                         let _ = el.set_attribute("data-tsclick", &click);
                     }
@@ -1495,9 +1512,9 @@ impl StreamProcessor for CreativeCssProcessor<'_> {
 #[cfg(test)]
 mod tests {
     use super::{
-        CreativeCssProcessor, StreamProcessor as _, process_auction_creative,
-        rewrite_creative_html, rewrite_inline_creative_html, rewrite_srcset, rewrite_style_urls,
-        sanitize_creative_html, to_abs,
+        CreativeCssProcessor, StreamProcessor as _, normalize_creative_url,
+        process_auction_creative, proxy_if_abs, rewrite_creative_html,
+        rewrite_inline_creative_html, rewrite_srcset, rewrite_style_urls, sanitize_creative_html,
     };
 
     fn rewrite_srcset_attr(attr_name: &str, attr_value: &str) -> String {
@@ -1696,44 +1713,42 @@ mod tests {
         );
     }
 
-    #[test]
-    fn to_abs_conversions() {
-        let settings = crate::test_support::tests::create_test_settings();
-        assert_eq!(
-            to_abs(&settings, "//cdn.example/x"),
-            Some("https://cdn.example/x".to_owned())
-        );
-        assert_eq!(
-            to_abs(&settings, "HTTPS://cdn.example/x"),
-            Some("HTTPS://cdn.example/x".to_owned())
-        );
-        assert_eq!(
-            to_abs(&settings, "http://cdn.example/x"),
-            Some("http://cdn.example/x".to_owned())
-        );
-        assert_eq!(to_abs(&settings, "/local/x"), None);
-        assert_eq!(
-            to_abs(&settings, "   //cdn.example/y  "),
-            Some("https://cdn.example/y".to_owned())
-        );
-        assert_eq!(to_abs(&settings, "data:image/png;base64,abcd"), None);
-        assert_eq!(to_abs(&settings, "javascript:alert(1)"), None);
-        assert_eq!(to_abs(&settings, "mailto:test@example.com"), None);
+    fn normalized(raw: &str) -> Option {
+        normalize_creative_url(raw).map(|url| url.as_str().to_owned())
     }
 
     #[test]
-    fn to_abs_preserves_port_in_protocol_relative() {
-        let settings = crate::test_support::tests::create_test_settings();
-        assert_eq!(
-            to_abs(&settings, "//cdn.example.com:8080/asset.js"),
-            Some("https://cdn.example.com:8080/asset.js".to_owned()),
-            "should preserve port 8080 in protocol-relative URL"
-        );
-        assert_eq!(
-            to_abs(&settings, "//cdn.example.com:9443/img.png"),
-            Some("https://cdn.example.com:9443/img.png".to_owned()),
-            "should preserve port 9443 in protocol-relative URL"
-        );
+    fn normalize_creative_url_conversions() {
+        for (raw, expected) in [
+            ("//cdn.example/x", Some("https://cdn.example/x")),
+            ("HTTPS://cdn.example/x", Some("https://cdn.example/x")),
+            ("http://cdn.example/x", Some("http://cdn.example/x")),
+            ("   //cdn.example/y  ", Some("https://cdn.example/y")),
+            ("   https://cdn.example/a   ", Some("https://cdn.example/a")),
+            (
+                "//cdn.example.com:8080/asset.js",
+                Some("https://cdn.example.com:8080/asset.js"),
+            ),
+            (
+                "//cdn.example.com:9443/img.png",
+                Some("https://cdn.example.com:9443/img.png"),
+            ),
+            ("/local/x", None),
+            ("", None),
+            ("data:image/png;base64,abcd", None),
+            ("javascript:alert(1)", None),
+            ("mailto:test@example.com", None),
+            ("blob:xyz", None),
+            ("tel:+123", None),
+            ("about:blank", None),
+            ("https://exa mple.example/x", None),
+        ] {
+            assert_eq!(
+                normalized(raw).as_deref(),
+                expected,
+                "should normalize `{raw}` to {expected:?}"
+            );
+        }
     }
 
     #[test]
@@ -3416,18 +3431,6 @@ b{background:url(\"https://cdn.example/c.png\")}";
         }
     }
 
-    #[test]
-    fn to_abs_additional_cases() {
-        let settings = crate::test_support::tests::create_test_settings();
-        assert_eq!(
-            to_abs(&settings, "   https://cdn.example/a   "),
-            Some("https://cdn.example/a".to_owned())
-        );
-        assert_eq!(to_abs(&settings, "blob:xyz"), None);
-        assert_eq!(to_abs(&settings, "tel:+123"), None);
-        assert_eq!(to_abs(&settings, "about:blank"), None);
-    }
-
     #[test]
     fn rewrites_lazy_img_data_src_and_data_srcset() {
         let settings = crate::test_support::tests::create_test_settings();
@@ -3443,62 +3446,86 @@ b{background:url(\"https://cdn.example/c.png\")}";
     }
 
     #[test]
-    fn to_abs_respects_exclude_domains() {
+    fn proxy_if_abs_respects_exclude_domains() {
         let mut settings = crate::test_support::tests::create_test_settings();
-        settings.rewrite.exclude_domains = vec!["trusted-cdn.example.com".to_owned()];
+        settings.rewrite.exclude_domains = vec![
+            "trusted-cdn.example.com".to_owned(),
+            "*.example.org".to_owned(),
+        ];
 
-        // Excluded domain should return None (not proxied)
-        assert_eq!(
-            to_abs(&settings, "https://trusted-cdn.example.com/lib.js"),
-            None
-        );
+        for excluded in [
+            "https://trusted-cdn.example.com/lib.js",
+            "//trusted-cdn.example.com/lib.js",
+            "https://example.org/cdn.js",
+            "//cdnjs.example.org/lib.js",
+        ] {
+            assert_eq!(
+                proxy_if_abs(&settings, excluded, ""),
+                None,
+                "should leave excluded URL `{excluded}` unproxied"
+            );
+        }
+        for proxied in [
+            "https://other-cdn.example.com/lib.js",
+            "//other-cdn.example.com/lib.js",
+            "https://notexample.org/lib.js",
+        ] {
+            assert!(
+                proxy_if_abs(&settings, proxied, "")
+                    .is_some_and(|url| url.starts_with("/first-party/proxy?tsurl=")),
+                "should proxy non-excluded URL `{proxied}`"
+            );
+        }
+    }
 
-        assert_eq!(
-            to_abs(&settings, "//trusted-cdn.example.com/lib.js"),
-            None,
-            "should exclude a protocol-relative URL by exact domain"
+    #[test]
+    fn unparseable_absolute_url_is_left_byte_identical() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = "";
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(html),
+            "should leave an unparseable absolute URL untouched, including its quoting: {out}"
         );
+    }
 
-        // Non-excluded domain should return Some
-        assert_eq!(
-            to_abs(&settings, "https://other-cdn.example.com/lib.js"),
-            Some("https://other-cdn.example.com/lib.js".to_owned())
+    #[test]
+    fn unparseable_absolute_click_url_is_left_byte_identical() {
+        let settings = crate::test_support::tests::create_test_settings();
+        let html = "x";
+
+        let out = rewrite_creative_html(&settings, html);
+
+        assert!(
+            out.contains(html),
+            "should leave an unparseable click URL untouched, including its quoting: {out}"
         );
-        assert_eq!(
-            to_abs(&settings, "//other-cdn.example.com/lib.js"),
-            Some("https://other-cdn.example.com/lib.js".to_owned()),
-            "should normalize a non-excluded protocol-relative URL"
+        assert!(
+            !out.contains("data-tsclick"),
+            "should not mark an unparseable link for the click guard: {out}"
         );
     }
 
     #[test]
-    fn to_abs_respects_wildcard_domains() {
+    fn exclude_domains_match_case_insensitively_in_the_rewrite_pass() {
         let mut settings = crate::test_support::tests::create_test_settings();
-        settings.rewrite.exclude_domains = vec!["*.cloudflare.com".to_owned()];
+        settings
+            .rewrite
+            .exclude_domains
+            .extend(["Landing.Example.com", "CDN.example.com"].map(str::to_owned));
+        let html = r#"x"#;
 
-        // Should exclude base domain
-        assert_eq!(to_abs(&settings, "https://cloudflare.com/cdn.js"), None);
+        let out = rewrite_creative_html(&settings, html);
 
-        // Should exclude subdomain
-        assert_eq!(
-            to_abs(&settings, "https://cdnjs.cloudflare.com/lib.js"),
-            None
-        );
-        assert_eq!(
-            to_abs(&settings, "//cloudflare.com/cdn.js"),
-            None,
-            "should exclude a protocol-relative wildcard base domain"
-        );
-        assert_eq!(
-            to_abs(&settings, "//cdnjs.cloudflare.com/lib.js"),
-            None,
-            "should exclude a protocol-relative wildcard subdomain"
+        assert!(
+            out.contains(r#""#),
+            "should leave a link raw when its host matches a mixed-case entry: {out}"
         );
-
-        // Should not exclude different domain
-        assert_eq!(
-            to_abs(&settings, "https://notcloudflare.com/lib.js"),
-            Some("https://notcloudflare.com/lib.js".to_owned())
+        assert!(
+            out.contains(r#""#),
+            "should leave an asset raw when its host matches a mixed-case entry: {out}"
         );
     }
 
diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs
index 5d659cdbf..6d1805c25 100644
--- a/crates/trusted-server-core/src/proxy.rs
+++ b/crates/trusted-server-core/src/proxy.rs
@@ -1670,37 +1670,31 @@ pub async fn handle_first_party_proxy_sign(
     };
 
     let trimmed = payload.url.trim();
-    let abs = if trimmed.starts_with("//") {
-        format!("{}:{}", request_scheme, trimmed)
+    let protocol_relative;
+    // A protocol-relative target inherits the signing request's scheme before
+    // normalization, so `//` and absolute input reach the same policy check.
+    let candidate = if trimmed.starts_with("//") {
+        protocol_relative = format!("{request_scheme}:{trimmed}");
+        protocol_relative.as_str()
     } else {
-        crate::creative::to_abs(settings, trimmed).ok_or_else(|| {
-            Report::new(TrustedServerError::Proxy {
-                message: "unsupported url".to_string(),
-            })
-        })?
+        trimmed
     };
-
-    if settings.rewrite.is_excluded(&abs) {
-        return Err(Report::new(TrustedServerError::Proxy {
+    let target = crate::creative::normalize_creative_url(candidate).ok_or_else(|| {
+        Report::new(TrustedServerError::Proxy {
             message: "unsupported url".to_string(),
-        }));
-    }
-
-    let parsed = url::Url::parse(&abs).change_context(TrustedServerError::Proxy {
-        message: "invalid url".to_string(),
+        })
     })?;
-    let scheme = parsed.scheme();
-    if scheme != "http" && scheme != "https" {
-        return Err(Report::new(TrustedServerError::Proxy {
-            message: "unsupported scheme".to_string(),
-        }));
-    }
-
-    let host = parsed.host_str().ok_or_else(|| {
+    let host = target.host_str().ok_or_else(|| {
         Report::new(TrustedServerError::Proxy {
             message: "missing host".to_string(),
         })
     })?;
+    if !settings.rewrite.should_proxy_asset(host) {
+        log::debug!("sign request for `{host}` declined by rewrite policy");
+        return Err(Report::new(TrustedServerError::Proxy {
+            message: "unsupported url".to_string(),
+        }));
+    }
     if !is_host_permitted(&settings.proxy.allowed_domains, host) {
         log::warn!(
             "sign request for `{}` blocked: host not in proxy.allowed_domains",
@@ -1720,10 +1714,10 @@ pub async fn handle_first_party_proxy_sign(
         .to_string();
     let extras = vec![(String::from("tsexp"), tsexp)];
 
-    let mut base = parsed.clone();
+    let mut base = target.clone();
     base.set_query(None);
     base.set_fragment(None);
-    let proxied = crate::creative::build_proxy_url_with_extras(settings, &abs, &extras);
+    let proxied = crate::creative::build_proxy_url_with_extras(settings, target.as_str(), &extras);
 
     let resp = ProxySignResp {
         href: proxied,
@@ -2695,20 +2689,58 @@ mod tests {
             let mut settings = create_test_settings();
             settings.rewrite.exclude_domains = vec!["cdn.example".to_owned()];
 
-            for url in ["https://cdn.example/asset.js", "//cdn.example/asset.js"] {
-                let body = serde_json::json!({ "url": url });
-                let req =
-                    build_http_post_json_request("https://edge.example/first-party/sign", &body);
-                let err: Report =
-                    handle_first_party_proxy_sign(&settings, &noop_services(), req)
-                        .await
-                        .expect_err("should reject excluded URL");
+            for method in [&Method::GET, &Method::POST] {
+                for url in [
+                    "https://cdn.example/asset.js",
+                    "//cdn.example/asset.js",
+                    "https://CDN.Example/asset.js",
+                ] {
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example/first-party/sign",
+                        url,
+                    );
+                    let err: Report =
+                        handle_first_party_proxy_sign(&settings, &noop_services(), req)
+                            .await
+                            .expect_err("should reject excluded URL");
+
+                    assert_eq!(
+                        err.current_context().status_code(),
+                        StatusCode::BAD_GATEWAY,
+                        "{} should reject excluded URL `{url}` as unsupported",
+                        method.as_str()
+                    );
+                }
+            }
+        });
+    }
 
-                assert_eq!(
-                    err.current_context().status_code(),
-                    StatusCode::BAD_GATEWAY,
-                    "should reject excluded URL `{url}` as unsupported"
-                );
+    #[test]
+    fn proxy_sign_rejects_excluded_urls_case_insensitively() {
+        futures::executor::block_on(async {
+            let mut settings = create_test_settings();
+            settings.rewrite.exclude_domains = vec!["CDN.Example".to_owned()];
+
+            for method in [&Method::GET, &Method::POST] {
+                for url in ["https://cdn.example/asset.js", "//cdn.example/asset.js"] {
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example/first-party/sign",
+                        url,
+                    );
+                    let err: Report =
+                        handle_first_party_proxy_sign(&settings, &noop_services(), req)
+                            .await
+                            .expect_err("should reject a host excluded by a mixed-case entry");
+
+                    assert_eq!(
+                        err.current_context().status_code(),
+                        StatusCode::BAD_GATEWAY,
+                        "{} should reject `{url}` excluded by a mixed-case entry as unsupported",
+                        method.as_str()
+                    );
+                }
             }
         });
     }
diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs
index bf4ff7999..f2207fde1 100644
--- a/crates/trusted-server-core/src/settings.rs
+++ b/crates/trusted-server-core/src/settings.rs
@@ -23,6 +23,7 @@ use crate::creative_opportunities::CreativeOpportunitiesConfig;
 use crate::error::TrustedServerError;
 use crate::host_header::validate_host_header_override_value;
 use crate::platform::PlatformImageOptimizerRegion;
+use crate::proxy::is_host_allowed;
 use crate::redacted::Redacted;
 
 #[cfg(test)]
@@ -643,30 +644,48 @@ pub struct Rewrite {
 }
 
 impl Rewrite {
-    /// Checks if a URL should be excluded from rewriting based on domain matching
-    #[allow(dead_code)]
+    /// Returns `true` when an asset URL on `host` should be rewritten to
+    /// `/first-party/proxy`.
+    ///
+    /// The host must not match [`Self::exclude_domains`]. Matching is
+    /// case-insensitive; see [`is_host_allowed`] for the pattern rules.
     #[must_use]
-    pub fn is_excluded(&self, url: &str) -> bool {
-        // Parse URL to extract host
-        let Ok(parsed) = url::Url::parse(url) else {
-            return false;
-        };
+    pub fn should_proxy_asset(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
 
-        let host = parsed.host_str().unwrap_or("");
+    /// Returns `true` when a click-through URL on `host` should be wrapped in
+    /// `/first-party/click`.
+    ///
+    /// Only [`Self::exclude_domains`] applies to click-through links.
+    #[must_use]
+    pub fn should_wrap_click(&self, host: &str) -> bool {
+        !self.is_excluded_host(host)
+    }
 
-        // Check exact domain matches (with wildcard support)
-        for domain in &self.exclude_domains {
-            if let Some(suffix) = domain.strip_prefix("*.") {
-                // Wildcard: *.example.com matches both example.com and sub.example.com
-                if host == suffix || host.ends_with(&format!(".{}", suffix)) {
-                    return true;
-                }
-            } else if host == domain {
-                return true;
-            }
-        }
+    fn is_excluded_host(&self, host: &str) -> bool {
+        self.exclude_domains
+            .iter()
+            .any(|pattern| is_host_allowed(host, pattern))
+    }
 
-        false
+    /// Trims and lowercases host patterns in place.
+    ///
+    /// Empty and bare `*` entries in `exclude_domains` are dropped with a
+    /// warning: neither can match a host, so dropping them changes no behavior.
+    fn normalize(&mut self) {
+        let before = self.exclude_domains.len();
+        self.exclude_domains = self
+            .exclude_domains
+            .iter()
+            .map(|pattern| pattern.trim().to_ascii_lowercase())
+            .filter(|pattern| !pattern.is_empty() && pattern != "*")
+            .collect();
+        if self.exclude_domains.len() < before {
+            log::warn!(
+                "rewrite.exclude_domains: removed empty or bare \"*\" entries, which never match a host"
+            );
+        }
     }
 }
 
@@ -2988,6 +3007,7 @@ impl Settings {
     pub(crate) fn normalize_deserialized(&mut self) {
         self.cache.normalize();
         self.proxy.normalize();
+        self.rewrite.normalize();
         self.image_optimizer.normalize();
         self.debug.auction_html_comment_options.normalize();
         self.tinybird.normalize();
@@ -6520,29 +6540,81 @@ source_domain = "partner.example.com"
     }
 
     #[test]
-    fn test_rewrite_is_excluded() {
-        let rewrite = Rewrite {
-            exclude_domains: vec!["cdn.example.com".to_string(), "*.example2.com".to_string()],
-        };
+    fn rewrite_policy_matches_exclude_patterns_case_insensitively() {
+        let mut rewrite = Rewrite::default();
+        rewrite
+            .exclude_domains
+            .extend(["cdn.example.com", "*.example.org"].map(str::to_owned));
+
+        for (host, expected) in [
+            ("cdn.example.com", false),
+            ("CDN.EXAMPLE.COM", false),
+            ("example.org", false),
+            ("a.b.example.org", false),
+            ("evil-example.org", true),
+            ("sub.cdn.example.com", true),
+            ("other.example.com", true),
+        ] {
+            assert_eq!(
+                rewrite.should_proxy_asset(host),
+                expected,
+                "should_proxy_asset(`{host}`) should be {expected}"
+            );
+            assert_eq!(
+                rewrite.should_wrap_click(host),
+                expected,
+                "should_wrap_click(`{host}`) should be {expected}"
+            );
+        }
+    }
+
+    #[test]
+    fn rewrite_normalize_trims_lowercases_and_drops_inert_exclude_entries() {
+        let mut rewrite = Rewrite::default();
+        rewrite
+            .exclude_domains
+            .extend(["  CDN.Example.com ", "", "*", "*.Example.ORG"].map(str::to_owned));
+
+        rewrite.normalize();
+
+        assert_eq!(
+            rewrite.exclude_domains,
+            vec!["cdn.example.com".to_owned(), "*.example.org".to_owned()],
+            "should trim, lowercase, and drop empty and bare `*` entries"
+        );
+    }
+
+    #[test]
+    fn rewrite_exclude_domains_match_mixed_case_entries_from_toml() {
+        let toml_str = crate_test_settings_str()
+            + r#"
+            [rewrite]
+            exclude_domains = ["CDN.Example.com"]
+            "#;
 
-        // Exact domain match
-        assert!(rewrite.is_excluded("http://cdn.example.com/image.png"));
+        let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
 
-        // Wildcard match - base domain
-        assert!(rewrite.is_excluded("https://example2.com/cdn.js"));
-        // Wildcard match - subdomains
-        assert!(rewrite.is_excluded("https://cdnjs.example2.com/lib.js"));
-        assert!(rewrite.is_excluded("https://sub.domain.example2.com/asset.js"));
+        assert!(
+            !settings.rewrite.should_proxy_asset("cdn.example.com"),
+            "should exclude a host whose config entry was written in mixed case"
+        );
+    }
 
-        // Should NOT match
-        assert!(!rewrite.is_excluded("https://other.example.com/asset.js"));
-        assert!(!rewrite.is_excluded("https://sub.cdn.example.com/asset.js"));
-        assert!(!rewrite.is_excluded("https://example2.com.fake.com/asset.js"));
-        assert!(!rewrite.is_excluded("https://notexample.com/asset.js"));
+    #[test]
+    fn settings_load_normalizes_rewrite_exclude_domains() {
+        let toml_str = crate_test_settings_str()
+            + r#"
+            [rewrite]
+            exclude_domains = ["  CDN.Example.com ", "*", ""]
+            "#;
+
+        let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
 
-        // Invalid URLs should not crash and should return false
-        assert!(!rewrite.is_excluded("not a url"));
-        assert!(!rewrite.is_excluded(""));
+        assert_eq!(
+            settings.rewrite.exclude_domains,
+            vec!["cdn.example.com".to_owned()],
+            "should trim, lowercase, and drop inert entries when settings load"
+        );
     }
 
     #[test]

From c9c79b54f4a0de8dabbd5d45d13f46afa3d8bc4f Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:03:06 +0530
Subject: [PATCH 149/230] Add an auction rewrite_clicks setting that follows
 rewrite_creatives when unset

Signed-off-by: dhruv8sh 
---
 .../src/auction/orchestrator.rs               |   1 +
 .../src/auction_config_types.rs               | 107 +++++++++++++++++-
 .../trusted-server-core/src/config_payload.rs |  44 +++++++
 crates/trusted-server-core/src/settings.rs    |  26 +++++
 4 files changed, 176 insertions(+), 2 deletions(-)

diff --git a/crates/trusted-server-core/src/auction/orchestrator.rs b/crates/trusted-server-core/src/auction/orchestrator.rs
index 204201e60..3d2044e32 100644
--- a/crates/trusted-server-core/src/auction/orchestrator.rs
+++ b/crates/trusted-server-core/src/auction/orchestrator.rs
@@ -4696,6 +4696,7 @@ mod tests {
                 enabled: true,
                 sanitize_creatives: true,
                 rewrite_creatives: true,
+                rewrite_clicks: None,
                 providers: AuctionConfig::legacy_provider_map(&[]),
                 bidders: Default::default(),
                 mediator: None,
diff --git a/crates/trusted-server-core/src/auction_config_types.rs b/crates/trusted-server-core/src/auction_config_types.rs
index 56171e1ea..42aab07ef 100644
--- a/crates/trusted-server-core/src/auction_config_types.rs
+++ b/crates/trusted-server-core/src/auction_config_types.rs
@@ -36,8 +36,13 @@ pub struct AuctionConfig {
     )]
     pub sanitize_creatives: bool,
 
-    /// Rewrite winning-bid creative HTML to first-party endpoints (applied
-    /// after sanitization when [`Self::sanitize_creatives`] is enabled).
+    /// Rewrite winning-bid creative asset URLs (images, scripts, styles,
+    /// media, iframes, CSS `url()`) to first-party `/first-party/proxy`
+    /// endpoints, applied after sanitization when
+    /// [`Self::sanitize_creatives`] is enabled.
+    ///
+    /// Bidder `` removal and creative TSJS injection run whenever this
+    /// or click rewriting ([`Self::rewrites_auction_clicks`]) is on.
     ///
     /// The default stays omitted from serialized config blobs to avoid adding
     /// this field when it has no effect. Any rollback across schema versions
@@ -49,6 +54,18 @@ pub struct AuctionConfig {
     )]
     pub rewrite_creatives: bool,
 
+    /// Wrap creative click-through links (``, ``) in signed
+    /// `/first-party/click` redirects.
+    ///
+    /// Unset keeps each path's existing behavior: auction creatives follow
+    /// [`Self::rewrite_creatives`], and HTML fetched through
+    /// `/first-party/proxy` keeps wrapping. An explicit value applies to every
+    /// path. Unset is omitted from serialized config blobs, so older binaries
+    /// keep loading them; any explicit value is serialized and rejected by
+    /// binaries that predate this field.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub rewrite_clicks: Option,
+
     /// Operator-defined bidder-provider instances, keyed by provider ID.
     #[serde(default, deserialize_with = "deserialize_provider_map")]
     pub providers: BTreeMap,
@@ -87,6 +104,7 @@ impl Default for AuctionConfig {
             enabled: false,
             sanitize_creatives: default_sanitize_creatives(),
             rewrite_creatives: default_rewrite_creatives(),
+            rewrite_clicks: None,
             providers: BTreeMap::new(),
             bidders: BTreeMap::new(),
             mediator: None,
@@ -160,6 +178,25 @@ fn default_allowed_context_keys() -> BTreeSet {
 }
 
 impl AuctionConfig {
+    /// Whether auction creatives (`POST /auction` and inline SSAT/page-bids)
+    /// wrap click-through links.
+    ///
+    /// Unset [`Self::rewrite_clicks`] follows [`Self::rewrite_creatives`].
+    #[must_use]
+    pub fn rewrites_auction_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(self.rewrite_creatives)
+    }
+
+    /// Whether HTML fetched through `/first-party/proxy` wraps click-through
+    /// links.
+    ///
+    /// Unset [`Self::rewrite_clicks`] keeps wrapping, as before the setting
+    /// existed.
+    #[must_use]
+    pub fn rewrites_proxied_clicks(&self) -> bool {
+        self.rewrite_clicks.unwrap_or(true)
+    }
+
     #[cfg(test)]
     pub(crate) fn legacy_provider_map(names: &[&str]) -> BTreeMap {
         names
@@ -255,6 +292,72 @@ mod tests {
         );
     }
 
+    #[test]
+    fn default_rewrite_clicks_is_unset_and_not_serialized() {
+        let config = AuctionConfig::default();
+
+        assert_eq!(
+            config.rewrite_clicks, None,
+            "should leave click rewriting unset by default"
+        );
+        let serialized = serde_json::to_value(&config).expect("should serialize defaults");
+        assert!(
+            serialized.get("rewrite_clicks").is_none(),
+            "should omit the unset click setting from serialized config"
+        );
+    }
+
+    #[test]
+    fn explicit_rewrite_clicks_is_serialized() {
+        for value in [true, false] {
+            let config = AuctionConfig {
+                rewrite_clicks: Some(value),
+                ..AuctionConfig::default()
+            };
+
+            let serialized =
+                serde_json::to_value(config).expect("should serialize explicit click setting");
+
+            assert_eq!(
+                serialized.get("rewrite_clicks"),
+                Some(&serde_json::Value::Bool(value)),
+                "should serialize explicit rewrite_clicks = {value}"
+            );
+        }
+    }
+
+    #[test]
+    fn click_rewriting_resolves_per_entry_point() {
+        // (rewrite_creatives, rewrite_clicks, auction clicks, proxied clicks)
+        let cases = [
+            (true, None, true, true),
+            (false, None, false, true),
+            (true, Some(false), false, false),
+            (false, Some(true), true, true),
+            (true, Some(true), true, true),
+            (false, Some(false), false, false),
+        ];
+
+        for (rewrite_creatives, rewrite_clicks, auction, proxied) in cases {
+            let config = AuctionConfig {
+                rewrite_creatives,
+                rewrite_clicks,
+                ..AuctionConfig::default()
+            };
+
+            assert_eq!(
+                config.rewrites_auction_clicks(),
+                auction,
+                "auction clicks for rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
+            );
+            assert_eq!(
+                config.rewrites_proxied_clicks(),
+                proxied,
+                "proxied clicks for rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}"
+            );
+        }
+    }
+
     #[test]
     fn default_sanitize_creatives_is_not_serialized() {
         let serialized =
diff --git a/crates/trusted-server-core/src/config_payload.rs b/crates/trusted-server-core/src/config_payload.rs
index c9690b01f..eee5d7ee8 100644
--- a/crates/trusted-server-core/src/config_payload.rs
+++ b/crates/trusted-server-core/src/config_payload.rs
@@ -749,6 +749,50 @@ mod tests {
         );
     }
 
+    #[test]
+    fn legacy_blob_without_rewrite_clicks_follows_rewrite_creatives() {
+        for rewrite_creatives in [true, false] {
+            let mut original = test_settings();
+            original.auction.rewrite_creatives = rewrite_creatives;
+            let data = serde_json::to_value(&original).expect("should serialize settings to JSON");
+            assert!(
+                data["auction"].get("rewrite_clicks").is_none(),
+                "should omit unset rewrite_clicks from the payload"
+            );
+
+            let reconstructed = load_settings(&envelope_json(&original))
+                .expect("should reconstruct settings without rewrite_clicks");
+
+            assert_eq!(
+                reconstructed.auction.rewrite_clicks, None,
+                "should load a blob without rewrite_clicks as unset"
+            );
+            assert_eq!(
+                reconstructed.auction.rewrites_auction_clicks(),
+                rewrite_creatives,
+                "should follow rewrite_creatives = {rewrite_creatives} when unset"
+            );
+        }
+    }
+
+    #[test]
+    fn explicit_rewrite_clicks_survives_blob_round_trip() {
+        for (rewrite_creatives, rewrite_clicks) in [(true, false), (false, true)] {
+            let mut original = test_settings();
+            original.auction.rewrite_creatives = rewrite_creatives;
+            original.auction.rewrite_clicks = Some(rewrite_clicks);
+
+            let reconstructed = load_settings(&envelope_json(&original))
+                .expect("should reconstruct explicit rewrite_clicks");
+
+            assert_eq!(
+                reconstructed.auction.rewrite_clicks,
+                Some(rewrite_clicks),
+                "should preserve rewrite_clicks = {rewrite_clicks} with rewrite_creatives = {rewrite_creatives}"
+            );
+        }
+    }
+
     #[test]
     fn strings_that_look_like_json_scalars_round_trip_as_strings() {
         let mut original = test_settings();
diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs
index f2207fde1..0d3cad1c9 100644
--- a/crates/trusted-server-core/src/settings.rs
+++ b/crates/trusted-server-core/src/settings.rs
@@ -6635,6 +6635,32 @@ source_domain = "partner.example.com"
             !settings.auction.sanitize_creatives,
             "creative sanitization is opt-in when the setting is omitted"
         );
+        assert_eq!(
+            settings.auction.rewrite_clicks, None,
+            "click rewriting stays unset when the setting is omitted"
+        );
+    }
+
+    #[test]
+    fn test_auction_rewrite_clicks_accepts_explicit_values() {
+        for value in [true, false] {
+            let toml_str = crate_test_settings_str()
+                + &format!(
+                    r#"
+            [auction]
+            enabled = true
+            rewrite_clicks = {value}
+            "#
+                );
+
+            let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
+
+            assert_eq!(
+                settings.auction.rewrite_clicks,
+                Some(value),
+                "should parse explicit rewrite_clicks = {value}"
+            );
+        }
     }
 
     #[test]

From 5a91c9224c93648fca979e600b5f57eb331974fc Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:06 +0530
Subject: [PATCH 150/230] Gate creative asset and click rewriting on separate
 switches

The rewrite pass registers asset handlers only when rewrite_creatives is on and the anchor handler only when click rewriting resolves on. Base removal and TSJS injection run whenever either is on.

Signed-off-by: dhruv8sh 
---
 .../src/auction/endpoints.rs                  |  12 +-
 .../src/auction/formats.rs                    |  51 +-
 crates/trusted-server-core/src/creative.rs    | 654 ++++++++++++------
 crates/trusted-server-core/src/publisher.rs   |  44 ++
 4 files changed, 550 insertions(+), 211 deletions(-)

diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs
index ab3585e3d..10600e945 100644
--- a/crates/trusted-server-core/src/auction/endpoints.rs
+++ b/crates/trusted-server-core/src/auction/endpoints.rs
@@ -84,10 +84,14 @@ const MAX_AUCTION_BODY_SIZE: usize = 256 * 1024;
 /// ## Response
 ///
 /// Returns an `OpenRTB 2.x` response. Creative HTML is inlined in each bid's
-/// `adm` field after mandatory server-side sanitization. First-party resource
-/// and click URL rewriting plus creative TSJS injection are enabled by default;
-/// setting [`auction.rewrite_creatives`][`crate::auction_config_types::AuctionConfig::rewrite_creatives`]
-/// to `false` skips only that rewrite pass.
+/// `adm` field after optional sanitization
+/// ([`auction.sanitize_creatives`][`crate::auction_config_types::AuctionConfig::sanitize_creatives`]).
+/// First-party asset rewriting
+/// ([`auction.rewrite_creatives`][`crate::auction_config_types::AuctionConfig::rewrite_creatives`])
+/// and click wrapping
+/// ([`auction.rewrite_clicks`][`crate::auction_config_types::AuctionConfig::rewrite_clicks`])
+/// are enabled by default. Bidder `` removal and creative TSJS injection
+/// run when either is on.
 ///
 /// ## Scroll, refresh, and SPA navigation
 ///
diff --git a/crates/trusted-server-core/src/auction/formats.rs b/crates/trusted-server-core/src/auction/formats.rs
index a0df63d94..956e81e32 100644
--- a/crates/trusted-server-core/src/auction/formats.rs
+++ b/crates/trusted-server-core/src/auction/formats.rs
@@ -316,14 +316,16 @@ pub(crate) struct OpenRtbResponseConversion {
 ///
 /// Creative HTML in the `adm` field is optionally sanitized and optionally
 /// rewritten according to the auction configuration
-/// ([`AuctionConfig::sanitize_creatives`], opt-in, and
-/// [`AuctionConfig::rewrite_creatives`], default-on); with both disabled the
-/// creative ships exactly as the bidder returned it, subject to the 1 MiB
-/// per-creative cap. Typed renderers are serialized in the response extension
+/// ([`AuctionConfig::sanitize_creatives`], opt-in;
+/// [`AuctionConfig::rewrite_creatives`] for assets, default-on; and
+/// [`AuctionConfig::rewrite_clicks`] for links, following `rewrite_creatives`
+/// when unset); with all disabled the creative ships exactly as the bidder
+/// returned it, subject to the 1 MiB per-creative cap. Typed renderers are serialized in the response extension
 /// instead of entering that pipeline at all.
 ///
 /// [`AuctionConfig::sanitize_creatives`]: crate::auction_config_types::AuctionConfig::sanitize_creatives
 /// [`AuctionConfig::rewrite_creatives`]: crate::auction_config_types::AuctionConfig::rewrite_creatives
+/// [`AuctionConfig::rewrite_clicks`]: crate::auction_config_types::AuctionConfig::rewrite_clicks
 ///
 /// # Errors
 ///
@@ -373,9 +375,10 @@ pub(crate) fn convert_to_openrtb_response_with_report(
         let height = to_openrtb_i32(bid.height, "height", &bid_context);
 
         // Ordinary markup goes through the configured creative processing:
-        // sanitization is opt-in, rewriting is on by default, and with both
-        // disabled the creative ships exactly as the bidder returned it. A typed
-        // renderer is serialized separately and never enters that pipeline.
+        // sanitization is opt-in, asset rewriting and click wrapping are on by
+        // default, and with all three disabled the creative ships exactly as
+        // the bidder returned it. A typed renderer is serialized separately and
+        // never enters that pipeline.
         let serialize_renderer = |renderer: &BidRenderer| {
             (BidExt {
                 trusted_server: BidTrustedServerExt { renderer },
@@ -398,12 +401,13 @@ pub(crate) fn convert_to_openrtb_response_with_report(
             let processed = creative::process_auction_creative(settings, raw_creative);
 
             log::debug!(
-                "Processed creative for auction {} slot {} bidder {} (sanitize {}, rewrite {}, raw {} bytes, output {} bytes)",
+                "Processed creative for auction {} slot {} bidder {} (sanitize {}, rewrite {}, clicks {}, raw {} bytes, output {} bytes)",
                 auction_request.id,
                 slot_id,
                 bid.bidder,
                 settings.auction.sanitize_creatives,
                 rewrite_creatives,
+                settings.auction.rewrites_auction_clicks(),
                 raw_creative.len(),
                 processed.len()
             );
@@ -1413,6 +1417,37 @@ mod tests {
         );
     }
 
+    #[test]
+    fn convert_to_openrtb_response_wraps_clicks_without_rewriting_assets() {
+        let mut settings = make_settings();
+        settings.auction.sanitize_creatives = false;
+        settings.auction.rewrite_creatives = false;
+        settings.auction.rewrite_clicks = Some(true);
+        let auction_request = make_auction_request();
+        let result = make_result(make_complete_creative_bid());
+
+        let response = convert_to_openrtb_response(&result, &settings, &auction_request, false)
+            .expect("should convert creative with click rewriting only");
+        let adm = response_adm(response);
+
+        assert!(
+            adm.contains("/first-party/click?tsurl=") && adm.contains("data-tsclick"),
+            "should wrap the landing link: {adm}"
+        );
+        assert!(
+            !adm.contains("/first-party/proxy?tsurl="),
+            "should not proxy any asset: {adm}"
+        );
+        assert!(
+            adm.contains(r#"src="https://cdn.example.com/ad.png""#),
+            "should keep the image URL direct: {adm}"
+        );
+        assert!(
+            adm.contains("tsjs-unified.min.js"),
+            "should inject the creative runtime for the click guard: {adm}"
+        );
+    }
+
     #[test]
     fn sanitize_creatives_defaults_to_disabled() {
         let config = crate::auction_config_types::AuctionConfig::default();
diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 8352c77c0..1fcb9b84c 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -3,10 +3,11 @@
 //! Goals:
 //! - Normalize external asset fetches in ad creatives (HTML/CSS) to a single
 //!   first-party proxy endpoint so the publisher can control egress.
+//! - Route click-through links through a signed first-party click redirect.
 //! - Leave relative URLs and non-network schemes untouched.
 //!
 //! Key behaviors:
-//! - Absolute and protocol-relative URLs (http/https or `//`) are proxied to
+//! - Absolute and protocol-relative asset URLs (http/https or `//`) are proxied to
 //!   `/first-party/proxy?tsurl=&&tstoken=` across these locations:
 //!   - ``, `data-src`, `[srcset]`, `[imagesrcset]`
 //!   - `
AdExcludedMail"#;
+    const MATRIX_DEFAULT_INLINE_OUTPUT: &str = r#"
AdExcludedMail"#;
+
+    #[test]
+    fn default_settings_rewrite_matrix_fixture_byte_for_byte_as_before() {
+        let settings = matrix_settings(true, None);
+
+        for (path, expected) in [
+            (CreativePath::Auction, MATRIX_DEFAULT_AUCTION_OUTPUT),
+            (CreativePath::Inline, MATRIX_DEFAULT_INLINE_OUTPUT),
+        ] {
+            assert_eq!(
+                process_for_path(&settings, path, MATRIX_FIXTURE_WITH_BODY),
+                expected,
+                "{path:?}: default settings should match the pre-rewrite_clicks output exactly"
+            );
+        }
+    }
+
     fn normalized(raw: &str) -> Option {
         normalize_creative_url(raw).map(|url| url.as_str().to_owned())
     }
diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs
index 61bc7efe5..254118f6f 100644
--- a/crates/trusted-server-core/src/publisher.rs
+++ b/crates/trusted-server-core/src/publisher.rs
@@ -22668,6 +22668,50 @@ mod tests {
             );
         }
 
+        #[test]
+        fn build_bid_map_keeps_inline_anchors_raw_when_clicks_are_off() {
+            let mut settings = test_settings();
+            settings.auction.rewrite_creatives = true;
+            settings.auction.rewrite_clicks = Some(false);
+            settings.publisher.domain = "example.com".to_string();
+
+            let mut winning_bids = HashMap::new();
+            let mut bid = make_bid(
+                "atf_sidebar_ad",
+                1.50,
+                "examplessp",
+                "abc123",
+                "https://ssp.example.com/win",
+                "https://ssp.example.com/bill",
+            );
+            bid.creative = Some(
+                ""
+                    .to_string(),
+            );
+            winning_bids.insert("atf_sidebar_ad".to_string(), bid);
+
+            let map = build_bid_map(&winning_bids, PriceGranularity::Dense, &settings, "", false);
+            let adm = map
+                .get("atf_sidebar_ad")
+                .and_then(|v| v.as_object())
+                .and_then(|o| o.get("adm"))
+                .and_then(|v| v.as_str())
+                .expect("should include a rewritten adm");
+
+            assert!(
+                adm.contains("https://example.com/first-party/proxy?tsurl="),
+                "should still proxy the image with an absolute URL, got: {adm}"
+            );
+            assert!(
+                adm.contains("href=\"https://landing.example.com/page\""),
+                "should leave the landing link raw, got: {adm}"
+            );
+            assert!(
+                !adm.contains("data-tsclick") && !adm.contains("/first-party/click"),
+                "should not wrap the landing link, got: {adm}"
+            );
+        }
+
         #[test]
         fn build_bid_map_uses_request_origin_for_inline_urls() {
             // The inline adm's absolute first-party URLs must resolve against the

From 35dd1d197ff6a1d9dad99d709ca996b77d0277b0 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:25 +0530
Subject: [PATCH 151/230] Apply an explicit rewrite_clicks to HTML fetched
 through the first-party proxy

Signed-off-by: dhruv8sh 
---
 crates/trusted-server-core/src/creative.rs | 16 ++++-
 crates/trusted-server-core/src/proxy.rs    | 78 ++++++++++++++++++++++
 2 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 1fcb9b84c..074e5f35d 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -1125,6 +1125,10 @@ pub fn rewrite_creative_html(settings: &Settings, markup: &str) -> String {
 /// own [`MAX_REWRITABLE_BODY_SIZE`] rather than the per-creative auction cap:
 /// a proxied document is a whole page, not an `adm`, and legitimately exceeds
 /// 1 MiB. The creative runtime is still injected so click mediation survives.
+///
+/// Asset rewriting here is unconditional. Click wrapping follows an explicit
+/// `[auction] rewrite_clicks`; when that is unset, links keep being wrapped
+/// regardless of `rewrite_creatives`.
 #[must_use]
 pub fn rewrite_proxied_html(settings: &Settings, markup: &str) -> String {
     rewrite_creative_html_impl(
@@ -1133,7 +1137,10 @@ pub fn rewrite_proxied_html(settings: &Settings, markup: &str) -> String {
         "",
         true,
         MAX_REWRITABLE_BODY_SIZE,
-        CreativeFeatures::ALL,
+        CreativeFeatures {
+            assets: true,
+            clicks: settings.auction.rewrites_proxied_clicks(),
+        },
     )
 }
 
@@ -1510,10 +1517,13 @@ fn rewrite_creative_html_impl(
     rewritten
 }
 
-/// Stream processor for creative HTML that rewrites URLs to first-party proxy.
+/// Stream processor for HTML fetched through `/first-party/proxy` that rewrites
+/// asset and click-through URLs to first-party endpoints.
 ///
 /// This processor buffers input chunks and processes the complete HTML document
-/// when the stream ends, using `rewrite_creative_html` internally.
+/// when the stream ends, using [`rewrite_proxied_html`] internally. Asset URLs are
+/// always proxied; links are wrapped unless `[auction] rewrite_clicks` is
+/// explicitly `false`.
 pub struct CreativeHtmlProcessor<'a> {
     settings: &'a Settings,
     buffer: Vec,
diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs
index 6d1805c25..1b9139753 100644
--- a/crates/trusted-server-core/src/proxy.rs
+++ b/crates/trusted-server-core/src/proxy.rs
@@ -3655,6 +3655,84 @@ mod tests {
         );
     }
 
+    #[test]
+    fn proxied_html_click_wrapping_follows_explicit_rewrite_clicks() {
+        let html = r#"Excluded"#;
+        // (rewrite_creatives, rewrite_clicks, expect wrapped clicks)
+        let cases = [
+            (true, None, true),
+            (false, None, true),
+            (true, Some(true), true),
+            (true, Some(false), false),
+            (false, Some(false), false),
+            (false, Some(true), true),
+        ];
+
+        for (rewrite_creatives, rewrite_clicks, expect_clicks) in cases {
+            let mut settings = create_test_settings();
+            settings.auction.rewrite_creatives = rewrite_creatives;
+            settings.auction.rewrite_clicks = rewrite_clicks;
+            settings.rewrite.exclude_domains = vec!["excluded.example.com".to_owned()];
+            let label =
+                format!("rewrite_creatives={rewrite_creatives} rewrite_clicks={rewrite_clicks:?}");
+            let req = build_http_request(Method::GET, "https://edge.example.com/first-party/proxy");
+            let mut response = build_http_response(StatusCode::OK, EdgeBody::from(html));
+            response.headers_mut().insert(
+                header::CONTENT_TYPE,
+                HeaderValue::from_static("text/html; charset=utf-8"),
+            );
+
+            let body = response_body_string(
+                finalize(
+                    &settings,
+                    &req,
+                    "https://cdn.example.com/creative.html",
+                    response,
+                )
+                .expect("should finalize proxied HTML"),
+            );
+
+            assert!(
+                body.contains("/first-party/proxy?tsurl="),
+                "{label}: proxied HTML always proxies assets: {body}"
+            );
+            // A wrapped landing link carries the click URL in href and data-tsclick.
+            assert_eq!(
+                body.matches("/first-party/click?tsurl=").count(),
+                if expect_clicks { 2 } else { 0 },
+                "{label}: click wrapping in proxied HTML: {body}"
+            );
+            assert_eq!(
+                body.matches("data-tsclick").count(),
+                usize::from(expect_clicks),
+                "{label}: data-tsclick in proxied HTML: {body}"
+            );
+            if !expect_clicks {
+                let landing_href = body
+                    .split("Excluded"#),
+                "{label}: excluded link always stays raw: {body}"
+            );
+            assert!(
+                !body.contains(": {body}"
+            );
+            assert!(
+                body.contains("/static/tsjs="),
+                "{label}: proxied HTML always receives the runtime: {body}"
+            );
+        }
+    }
+
     #[test]
     fn html_response_rewrite_preserves_non_standard_port() {
         // Verify that HTML rewriting preserves non-standard ports in sub-resource URLs.

From 0396aba1f48471f61a56ffca6626556fc5055b87 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:44 +0530
Subject: [PATCH 152/230] Document rewrite_clicks in the example config and pin
 its overlay behavior

Signed-off-by: dhruv8sh 
---
 .../tests/config_env_overlay.rs               | 71 +++++++++++++++++++
 trusted-server.example.toml                   | 29 +++++---
 2 files changed, 91 insertions(+), 9 deletions(-)

diff --git a/crates/trusted-server-cli/tests/config_env_overlay.rs b/crates/trusted-server-cli/tests/config_env_overlay.rs
index d81b0366a..06cc3c058 100644
--- a/crates/trusted-server-cli/tests/config_env_overlay.rs
+++ b/crates/trusted-server-cli/tests/config_env_overlay.rs
@@ -29,6 +29,7 @@ ids = ["trusted_server_secrets"]
 "#;
 const REWRITE_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CREATIVES";
 const SANITIZE_ENV: &str = "TRUSTED_SERVER__AUCTION__SANITIZE_CREATIVES";
+const CLICKS_ENV: &str = "TRUSTED_SERVER__AUCTION__REWRITE_CLICKS";
 const GAM_ATTRIBUTION_ENV: &str = "TRUSTED_SERVER__INTEGRATIONS__GPT__GAM_ATTRIBUTION_ENABLED";
 const AD_TEMPLATES_ENABLED_ENV: &str = "TRUSTED_SERVER__CREATIVE_OPPORTUNITIES__ENABLED";
 const PROVIDER_ENDPOINT_ENV: &str = "TRUSTED_SERVER__AUCTION__PROVIDERS__PBS-MAIN__ENDPOINT";
@@ -82,6 +83,76 @@ fn validate_with_overlay(project: &MigratedProject, raw_value: &str) -> Output {
         .expect("should run ts config validate")
 }
 
+fn pushed_auction_with_env(
+    project: &MigratedProject,
+    key: &str,
+    raw_value: &str,
+) -> serde_json::Value {
+    let output = Command::new(env!("CARGO_BIN_EXE_ts"))
+        .args(["config", "push", "--adapter", "axum", "--manifest"])
+        .arg(&project.manifest_path)
+        .arg("--app-config")
+        .arg(&project.config_path)
+        .args(["--yes", "--no-diff"])
+        .current_dir(project.directory.path())
+        .env(key, raw_value)
+        .output()
+        .expect("should run ts config push");
+    assert!(
+        output.status.success(),
+        "config push should succeed: {}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+
+    let local_store_path = project
+        .directory
+        .path()
+        .join(".edgezero/local-config-trusted_server_config.json");
+    let local_store: serde_json::Value = serde_json::from_str(
+        &fs::read_to_string(local_store_path).expect("should read pushed local config"),
+    )
+    .expect("should parse local config store");
+    let envelope_json = local_store
+        .as_object()
+        .and_then(|entries| entries.values().next())
+        .and_then(serde_json::Value::as_str)
+        .expect("should contain a blob envelope");
+    let envelope: serde_json::Value =
+        serde_json::from_str(envelope_json).expect("should parse blob envelope");
+    envelope["data"]["auction"].clone()
+}
+
+#[test]
+fn rewrite_clicks_environment_override_applies_when_leaf_present() {
+    let project = migrated_project();
+    let mut document = fs::read_to_string(&project.config_path)
+        .expect("should read migrated config")
+        .parse::()
+        .expect("should parse migrated config");
+    document["auction"]["rewrite_clicks"] = value(true);
+    fs::write(&project.config_path, document.to_string()).expect("should write config");
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert_eq!(
+        auction["rewrite_clicks"],
+        serde_json::Value::Bool(false),
+        "pushed config should contain the rewrite_clicks environment override"
+    );
+}
+
+#[test]
+fn rewrite_clicks_environment_override_is_ignored_without_leaf() {
+    let project = migrated_project();
+
+    let auction = pushed_auction_with_env(&project, CLICKS_ENV, "false");
+
+    assert!(
+        auction.get("rewrite_clicks").is_none(),
+        "an override for a missing leaf should be ignored and the unset default omitted"
+    );
+}
+
 #[test]
 fn config_validate_explains_legacy_provider_list_migration() {
     let project = migrated_project();
diff --git a/trusted-server.example.toml b/trusted-server.example.toml
index e841515ae..d4789570c 100644
--- a/trusted-server.example.toml
+++ b/trusted-server.example.toml
@@ -261,18 +261,29 @@ proxy = "enabled"
 # Keep disabled until provider endpoints, routes, and profile values below are
 # replaced with deployment-specific settings.
 enabled = false
-# Rewrite winning-bid creative HTML to first-party endpoints (default true). Set
-# false to skip proxy/click-URL conversion and creative TSJS injection.
-# Sanitization is controlled separately by `sanitize_creatives` below. Restore
-# true before rolling back to an older binary that rejects unknown fields.
+# Rewrite winning-bid creative asset URLs (images, scripts, styles, media,
+# iframes, CSS url()) to first-party /first-party/proxy endpoints (default
+# true). Set false to leave asset URLs direct. Bidder  removal and
+# creative TSJS injection run when this or rewrite_clicks is on. Sanitization
+# is controlled separately by `sanitize_creatives` below. Restore true before
+# rolling back to an older binary that rejects unknown fields.
 rewrite_creatives = true
+# Wrap creative click-through links (, ) in signed
+# /first-party/click redirects. Unset (the default) follows rewrite_creatives
+# for auction creatives and keeps wrapping links in HTML fetched through
+# /first-party/proxy; a set value applies to both. Leave it commented out to
+# keep that behavior: any explicit value is stored in the pushed config, and
+# binaries older than this setting reject it. Uncomment it before relying on
+# TRUSTED_SERVER__AUCTION__REWRITE_CLICKS, which cannot create a missing leaf.
+# rewrite_clicks = true
 # Strip executable markup (script/object/embed/form/...) from winning-bid adm,
 # removing those elements together with their inner content. Defaults to false
-# (executable markup preserved). Note that with `rewrite_creatives = true` the
-# adm is still not untouched: eligible URLs are rewritten, bidder ``
-# elements are removed, and the creative TSJS runtime is injected. Enable it
-# whenever creatives can render in a context that shares the publisher origin
-# (its primary defence there); leave it off when creatives render in a
+# (executable markup preserved). Note that with `rewrite_creatives` or
+# `rewrite_clicks` on, the adm is still not untouched: eligible asset URLs
+# and/or links are rewritten, bidder `` elements are removed, and the
+# creative TSJS runtime is injected. Enable it whenever creatives can render
+# in a context that shares the publisher origin (its primary defence there);
+# leave it off when creatives render in a
 # foreign-origin frame (e.g. the Prebid Universal Creative inside the ad
 # server's iframe), since it removes script-based creatives entirely and would
 # blank slots on a script-heavy demand stack.

From ee5715f11049ce26257f10e51ab4c88b1fba07c3 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:04:44 +0530
Subject: [PATCH 153/230] Document the rewrite_clicks creative setting

Signed-off-by: dhruv8sh 
---
 CHANGELOG.md                                  |  3 +-
 .../trusted-server-core/src/auction/README.md |  6 +-
 docs/guide/api-reference.md                   |  5 +-
 docs/guide/auction-orchestration.md           | 11 ++--
 docs/guide/configuration.md                   | 57 +++++++++-------
 docs/guide/creative-processing.md             | 66 +++++++++++++++----
 6 files changed, 103 insertions(+), 45 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index c370373ff..ce3904fea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,7 +41,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
+- Added `[auction].rewrite_clicks` to control creative click-through wrapping (``/`` → signed `/first-party/click`) independently of asset rewriting. Unset (the default) follows `rewrite_creatives` for `POST /auction` and SSAT/page-bids and keeps wrapping links in HTML fetched through `/first-party/proxy`, so existing configs behave as before; an explicit value applies to every path. `rewrite_creatives` now governs asset URLs only; bidder `` removal and creative TSJS injection run when either setting is on. Upgrading: deploy the binary first, then push a config that sets `rewrite_clicks`. Rolling back: remove any explicit `rewrite_clicks` (and its environment override), push the resulting config, then roll back the binary. Older binaries tie click wrapping to `rewrite_creatives` in both directions, so rolling back turns clicks back on for `rewrite_creatives = true` with `rewrite_clicks = false`, and off for `rewrite_creatives = false` with `rewrite_clicks = true`.
+- Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (asset URL conversion to `/first-party/proxy`, bidder `` removal; creative TSJS injection on `POST /auction` only). Click-through wrapping is controlled by `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset. Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery.
 - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary.
 - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape.
 - Added typed APS renderer transport for direct auctions and GAM/Prebid Universal Creative, using a minimized one-bid envelope, a fragment-bound nonce, and an opaque sandboxed renderer endpoint.
diff --git a/crates/trusted-server-core/src/auction/README.md b/crates/trusted-server-core/src/auction/README.md
index 8a745eabc..7b0ae585b 100644
--- a/crates/trusted-server-core/src/auction/README.md
+++ b/crates/trusted-server-core/src/auction/README.md
@@ -108,8 +108,10 @@ as `pbs-main`.
 response.
 
 - `sanitize_creatives = true` strips executable markup. It is opt-in.
-- `rewrite_creatives = true` rewrites eligible URLs through first-party routes
-  and removes bidder `` elements. It is enabled by default.
+- `rewrite_creatives = true` rewrites eligible asset URLs through first-party
+  routes and removes bidder `` elements. It is enabled by default.
+- `rewrite_clicks` wraps creative links in `/first-party/click`. Unset, it
+  follows `rewrite_creatives`.
 - The publisher inline delivery path uses absolute first-party URLs without
   injecting the direct endpoint's creative runtime.
 - Creatives over the configured hard cap are rejected.
diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md
index e444ec147..bc95ff9ac 100644
--- a/docs/guide/api-reference.md
+++ b/docs/guide/api-reference.md
@@ -168,8 +168,9 @@ curl -i "https://edge.example.com/_ts/clear-tester"
 
 Browser and programmatic auction endpoint. It accepts the Trusted Server ad-unit
 request shape and returns an OpenRTB response. Creative markup follows the
-independent `[auction].sanitize_creatives` and `[auction].rewrite_creatives`
-settings; sanitization is opt-in and rewriting is enabled by default.
+independent `[auction].sanitize_creatives`, `[auction].rewrite_creatives` and
+`[auction].rewrite_clicks` settings; sanitization is opt-in, asset rewriting is
+enabled by default, and click wrapping follows `rewrite_creatives` unless set.
 
 Configured provider IDs appear in response metadata and provider responses.
 Consumers that previously matched the literal provider name `prebid` must use
diff --git a/docs/guide/auction-orchestration.md b/docs/guide/auction-orchestration.md
index edbc38471..46347c80e 100644
--- a/docs/guide/auction-orchestration.md
+++ b/docs/guide/auction-orchestration.md
@@ -161,7 +161,7 @@ sequenceDiagram
       Note right of Client: Fragment-bound nonce and one-time acknowledgement
No allow-same-origin on the outer frame
     else Ordinary creative
       Client->>Client: Inject winning creative
Render iframe
Load creative resources
-      Note right of Client: Default: first-party proxy/click URLs
rewrite_creatives=false: accepted external URLs remain direct
+      Note right of Client: Default: first-party proxy/click URLs
rewrite_creatives=false: asset URLs direct; clicks follow rewrite_clicks
     end
     deactivate Client
   end
@@ -195,7 +195,7 @@ AuctionOrchestrator.run_auction()
 Convert OrchestrationResult → OpenRTB 2.x Response
   │
   ├─[sanitize_creatives=true] Strip executable markup
-  ├─[rewrite_creatives=true] Rewrite URLs and inject creative TSJS
+  ├─[rewrite_creatives or rewrite_clicks] Rewrite assets and/or links, inject creative TSJS
   ├─ Add ext.orchestrator metadata
   └─ Set consent and optional EID response headers
 ```
@@ -638,8 +638,11 @@ HTML rewriter (`lol_html`) that converts eligible external resource and click
 URLs to signed first-party paths, adds `data-tsclick`, rewrites inline CSS
 `url(...)` values, removes bidder-supplied `` elements, and injects the
 unified creative TSJS runtime exactly once, whether or not the bidder supplied a
-`` element. In every mode, a creative
-larger than the 1 MiB per-creative cap is rejected and its `adm` is dropped.
+`` element. `rewrite_clicks` (unset by default, following
+`rewrite_creatives`) controls click-through link wrapping separately; see
+[Creative Processing](/guide/creative-processing#assets-and-clicks). In every
+mode, a creative larger than the 1 MiB per-creative cap is rejected and its
+`adm` is dropped.
 
 ```toml
 [auction]
diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md
index f7220d58c..908239d2b 100644
--- a/docs/guide/configuration.md
+++ b/docs/guide/configuration.md
@@ -1927,15 +1927,16 @@ provider or bidder route.
 
 ### `[auction]`
 
-| Field                  | Type    | Default            | Description                                                    |
-| ---------------------- | ------- | ------------------ | -------------------------------------------------------------- |
-| `enabled`              | Boolean | `false`            | Enable the auction orchestrator                                |
-| `sanitize_creatives`   | Boolean | `false`            | Strip executable markup from winning-bid `adm` before delivery |
-| `rewrite_creatives`    | Boolean | `true`             | Rewrite winning-bid `adm` through first-party endpoints        |
-| `timeout_ms`           | Integer | `2000`             | Logical auction budget in milliseconds                         |
-| `mediator`             | String  | `None`             | Optional separate `adserver_mock` mediator                     |
-| `creative_store`       | String  | `"creative_store"` | Deprecated; creatives are delivered inline                     |
-| `allowed_context_keys` | Array   | `[]`               | Request context keys admitted into the auction                 |
+| Field                  | Type    | Default            | Description                                                                    |
+| ---------------------- | ------- | ------------------ | ------------------------------------------------------------------------------ |
+| `enabled`              | Boolean | `false`            | Enable the auction orchestrator                                                |
+| `sanitize_creatives`   | Boolean | `false`            | Strip executable markup from winning-bid `adm` before delivery                 |
+| `rewrite_creatives`    | Boolean | `true`             | Rewrite winning-bid asset URLs through first-party endpoints                   |
+| `rewrite_clicks`       | Boolean | unset              | Wrap creative links in `/first-party/click`; unset follows `rewrite_creatives` |
+| `timeout_ms`           | Integer | `2000`             | Logical auction budget in milliseconds                                         |
+| `mediator`             | String  | `None`             | Optional separate `adserver_mock` mediator                                     |
+| `creative_store`       | String  | `"creative_store"` | Deprecated; creatives are delivered inline                                     |
+| `allowed_context_keys` | Array   | `[]`               | Request context keys admitted into the auction                                 |
 
 Creative markup delivered by `POST /auction` and the publisher SSAT/page-bids
 path is processed by two independent passes. With `sanitize_creatives = true`
@@ -1943,23 +1944,29 @@ path is processed by two independent passes. With `sanitize_creatives = true`
 and event handlers) is stripped together with its inner content. This blanks
 script-based creatives, so enable it only when creatives render in a context
 that shares the publisher's origin. With `rewrite_creatives = true` (the
-default), eligible absolute or protocol-relative resource and click URLs not
-excluded by rewrite configuration are converted to signed first-party
-endpoints, and any bidder-supplied `` element is removed. The
-`POST /auction` path emits root-relative endpoints and injects the creative TSJS
-runtime exactly once, whether or not the bidder supplied a ``, since bare
+default), eligible absolute or protocol-relative asset URLs not excluded by
+rewrite configuration are converted to signed `/first-party/proxy` endpoints.
+`rewrite_clicks` controls click-through links (``, ``)
+separately: when it is unset, it follows `rewrite_creatives`, so existing
+configs keep their current behavior. Any bidder-supplied `` element is
+removed whenever either setting is on. The `POST /auction` path emits
+root-relative endpoints and, when either setting is on, injects the creative
+TSJS runtime exactly once, whether or not the bidder supplied a ``, since bare
 fragments are the common `adm` shape. The foreign-origin SSAT renderer emits
-absolute endpoints and does not inject that bundle. With both disabled, `adm`
-ships exactly as the bidder returned it, except that a creative larger than the
+absolute endpoints and does not inject that bundle. With all three disabled,
+`adm` ships exactly as the bidder returned it, except that a creative larger than the
 1 MiB per-creative cap is rejected in every mode and its `adm` is dropped.
-Accepted external URLs are not host allowlisted by the sanitizer. Neither
-setting affects HTML or CSS fetched through `/first-party/proxy`. See
+Accepted external URLs are not host allowlisted by the sanitizer.
+`rewrite_creatives` and `sanitize_creatives` do not affect HTML or CSS fetched
+through `/first-party/proxy`. An explicitly set `rewrite_clicks` does apply to
+links in proxied HTML; unset, proxied HTML keeps wrapping links. See
 [Creative Processing](/guide/creative-processing#auction-rewrite-control).
 
 ::: warning Existing configs, upgrade sequencing, and rollback
 Default values are omitted from stored JSON; non-default values
-(`sanitize_creatives = true`, `rewrite_creatives = false`) are serialized, and
-older `AuctionConfig` schemas reject unknown fields.
+(`sanitize_creatives = true`, `rewrite_creatives = false`, and any explicit
+`rewrite_clicks`) are serialized, and older `AuctionConfig` schemas reject
+unknown fields.
 
 **Upgrading:** binaries that predate `sanitize_creatives` reject a blob that
 carries it, so in a rolling deployment upgrade the binary **first**, then push
@@ -1973,14 +1980,20 @@ on new code, while an explicit `true` fails startup on old code.
 **Rolling back:** before reverting to a binary that does not know a field,
 remove that field's non-default value (and any environment override), run
 `ts config validate`, push the resulting default-compatible blob, and only then
-roll back the binary.
+roll back the binary. Older binaries tie click wrapping to `rewrite_creatives`
+in both directions: rolling back from `rewrite_creatives = true` with
+`rewrite_clicks = false` turns click wrapping back on, and rolling back from
+`rewrite_creatives = false` with `rewrite_clicks = true` turns it off.
 
 **Environment overlays:** The pinned EdgeZero loader cannot create missing TOML
 leaves. Existing configs must add **both** leaves under `[auction]`
 (`rewrite_creatives` and `sanitize_creatives`) before
 `TRUSTED_SERVER__AUCTION__REWRITE_CREATIVES` /
 `TRUSTED_SERVER__AUCTION__SANITIZE_CREATIVES` can take effect. An override for a
-missing leaf is silently ignored.
+missing leaf is silently ignored. `rewrite_clicks` is unset by default and so
+cannot appear as a TOML leaf until you set it; add `rewrite_clicks = true` or
+`false` under `[auction]` before relying on
+`TRUSTED_SERVER__AUCTION__REWRITE_CLICKS`.
 :::
 
 ### Provider map
diff --git a/docs/guide/creative-processing.md b/docs/guide/creative-processing.md
index 10dcb62d5..823ae1260 100644
--- a/docs/guide/creative-processing.md
+++ b/docs/guide/creative-processing.md
@@ -47,8 +47,8 @@ publisher's first-party domain. This provides:
 
 The creative rewriters are invoked by independent delivery paths:
 
-1. **Auction `adm`**: Winning-bid HTML returned by `POST /auction` is optionally sanitized (`[auction].sanitize_creatives`, opt-in, default `false`) and rewritten (`[auction].rewrite_creatives`, default `true`).
-2. **First-party proxy**: Non-streaming `text/html` and `text/css` responses fetched through `/first-party/proxy` are rewritten independently of the auction setting.
+1. **Auction `adm`**: Winning-bid HTML returned by `POST /auction` is optionally sanitized (`[auction].sanitize_creatives`, opt-in, default `false`) and rewritten (`[auction].rewrite_creatives`, default `true`). Click-through links are wrapped according to `[auction].rewrite_clicks`, which follows `rewrite_creatives` when unset.
+2. **First-party proxy**: Non-streaming `text/html` and `text/css` responses fetched through `/first-party/proxy` are rewritten independently of the auction setting. An explicitly set `[auction].rewrite_clicks` also applies to links in proxied HTML.
 3. **Integration processing**: Publisher HTML integrations use their own registration and configuration controls.
 
 ::: info Streaming Mode
@@ -57,26 +57,28 @@ When `with_streaming()` is enabled in `ProxyRequestConfig`, proxied HTML/CSS pro
 
 ## Auction Rewrite Control
 
-Two independent auction settings control the processing applied to winning-bid
-`adm` returned by `POST /auction` and delivered through the publisher
-SSAT/page-bids path. Sanitization is opt-in (default `false`); rewriting is
-enabled by default:
+Three auction settings control the processing applied to winning-bid `adm`
+returned by `POST /auction` and delivered through the publisher SSAT/page-bids
+path. Sanitization is opt-in (default `false`) and asset rewriting is enabled
+by default. Click wrapping is controlled by `rewrite_clicks`, which follows
+`rewrite_creatives` when unset; see [Assets and clicks](#assets-and-clicks).
 
 ```toml
 [auction]
 sanitize_creatives = false
 rewrite_creatives = true
+# rewrite_clicks unset: follows rewrite_creatives
 ```
 
 Regardless of mode, a creative larger than the 1 MiB per-creative cap is
 rejected and its `adm` is dropped.
 
-| `sanitize_creatives` | `rewrite_creatives` | Auction winning-bid `adm` behavior                                                                                                                                                                                                                                         |
-| -------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `false` (default)    | `false`             | Deliver the creative exactly as the bidder returned it (subject to the size cap).                                                                                                                                                                                          |
-| `true`               | `false`             | Strip executable markup (`script`/`object`/`embed`/`form`, event handlers) with its inner content, then deliver without rewriting. Sanitizer-accepted external resource, click, and inline CSS URLs remain direct.                                                         |
-| `false`              | `true` (default)    | Rewrite eligible resource/CSS and click URLs in the raw bidder markup to signed first-party endpoints, removing any bidder `` element. Executable markup is preserved.                                                                                               |
-| `true`               | `true`              | Sanitize first, then rewrite. `POST /auction` emits root-relative endpoints and injects creative TSJS exactly once, whether or not the bidder supplied a ``; SSAT/page-bids emits absolute endpoints for its foreign-origin renderer and does not inject the bundle. |
+| `sanitize_creatives` | `rewrite_creatives` | Auction winning-bid `adm` behavior                                                                                                                                                                                                                                                          |
+| -------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `false` (default)    | `false`             | Asset URLs stay direct. With `rewrite_clicks` unset or `false`, deliver the creative exactly as the bidder returned it (subject to the size cap); with `rewrite_clicks = true`, wrap links as described in [Assets and clicks](#assets-and-clicks).                                         |
+| `true`               | `false`             | Strip executable markup (`script`/`object`/`embed`/`form`, event handlers) with its inner content, then deliver without asset rewriting. Sanitizer-accepted external resource and inline CSS URLs remain direct; links follow `rewrite_clicks` and stay direct when it is unset or `false`. |
+| `false`              | `true` (default)    | Rewrite eligible resource/CSS URLs in the raw bidder markup to signed first-party endpoints, removing any bidder `` element. Links are wrapped unless `rewrite_clicks = false`. Executable markup is preserved.                                                                       |
+| `true`               | `true`              | Sanitize first, then rewrite. `POST /auction` emits root-relative endpoints and injects creative TSJS exactly once, whether or not the bidder supplied a ``; SSAT/page-bids emits absolute endpoints for its foreign-origin renderer and does not inject the bundle.                  |
 
 ::: warning Sanitization blanks script-based creatives
 Sanitization removes `script`/`object`/`embed`/`form` and similar elements
@@ -90,6 +92,39 @@ accepted HTTP(S) URLs may cause the browser to contact external creative hosts
 directly.
 :::
 
+### Assets and clicks
+
+`rewrite_creatives` governs asset URLs. `rewrite_clicks` governs click-through
+links and follows `rewrite_creatives` when unset:
+
+```toml
+[auction]
+rewrite_creatives = true
+# Unset: follows rewrite_creatives. Set false to keep landing links direct.
+# rewrite_clicks = false
+```
+
+| Assets (`rewrite_creatives`) | Clicks (`rewrite_clicks`) | Asset URLs           | Links                                 | `` | TSJS on `POST /auction` |
+| ---------------------------- | ------------------------- | -------------------- | ------------------------------------- | -------- | ----------------------- |
+| `true`                       | `true` or unset           | `/first-party/proxy` | `/first-party/click` + `data-tsclick` | removed  | injected                |
+| `true`                       | `false`                   | `/first-party/proxy` | direct                                | removed  | injected                |
+| `false`                      | `true`                    | direct               | `/first-party/click` + `data-tsclick` | removed  | injected                |
+| `false`                      | `false` or unset          | direct               | direct                                | kept     | not injected            |
+
+SSAT/page-bids follows the same table with absolute URLs and never injects TSJS.
+`exclude_domains` applies to both assets and links.
+
+The client-side `tsCreativeConfig.clickGuard` flag is separate. `rewrite_clicks`
+decides whether the server emits signed click URLs; `clickGuard` decides whether
+the creative runtime re-signs them after creative script changes their query
+parameters. With `rewrite_clicks` off there is nothing for the guard to act on.
+
+With `rewrite_creatives = false` and `rewrite_clicks = true`, `POST /auction`
+still injects TSJS. A creative that turns on `tsCreativeConfig.renderGuard` can
+therefore still send assets inserted by its own script through
+`/first-party/sign` and `/first-party/proxy`, even though the server left the
+markup's asset URLs direct.
+
 ::: info Runtime protections inside the sandboxed creative iframe
 Creatives rendered by Trusted Server's own path run in a sandboxed iframe
 **without** `allow-same-origin`, i.e. an opaque origin. The injected creative
@@ -307,15 +342,18 @@ If the iframe content itself contains HTML, it will be processed recursively. Ea
 
 **Rewrite Mode**: Uses `/first-party/click` for direct redirects
 
+**Controlled by**: `[auction].rewrite_clicks` (follows `rewrite_creatives` when unset)
+
 **Example**:
 
 ```html
 
-Buy Now
+Buy Now
 
 
 Buy Now
 ```

From 48d59b8daa6ea703420c15246dc6553936c59ab1 Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:08:47 +0530
Subject: [PATCH 154/230] Add rewrite include_domains setting with validation

Add the [rewrite] include_domains field with load-time normalization, strict entry validation and blob round-trip tests. Enforcement arrives in the next commit, which applies the list to asset rewriting and first-party sign.

Signed-off-by: dhruv8sh 
---
 .../trusted-server-core/src/config_payload.rs |  37 ++++
 crates/trusted-server-core/src/settings.rs    | 206 +++++++++++++++++-
 2 files changed, 241 insertions(+), 2 deletions(-)

diff --git a/crates/trusted-server-core/src/config_payload.rs b/crates/trusted-server-core/src/config_payload.rs
index eee5d7ee8..f818571c6 100644
--- a/crates/trusted-server-core/src/config_payload.rs
+++ b/crates/trusted-server-core/src/config_payload.rs
@@ -735,6 +735,43 @@ mod tests {
         );
     }
 
+    #[test]
+    fn legacy_blob_without_rewrite_include_domains_loads_with_empty_list() {
+        let data =
+            serde_json::to_value(test_settings()).expect("should serialize settings to JSON");
+        let rewrite = data
+            .get("rewrite")
+            .and_then(serde_json::Value::as_object)
+            .expect("should serialize rewrite settings as an object");
+        assert!(
+            !rewrite.contains_key("include_domains"),
+            "should omit the default include list from the payload"
+        );
+
+        let reconstructed = load_settings(&envelope_json(&test_settings()))
+            .expect("should reconstruct settings without include_domains");
+
+        assert!(
+            reconstructed.rewrite.include_domains.is_empty(),
+            "should load an empty include list from a blob without the key"
+        );
+    }
+
+    #[test]
+    fn rewrite_include_domains_survive_blob_round_trip() {
+        let mut original = test_settings();
+        original.rewrite.include_domains = vec!["*.cdn.example.com".to_owned()];
+
+        let reconstructed = load_settings(&envelope_json(&original))
+            .expect("should reconstruct settings with include_domains");
+
+        assert_eq!(
+            reconstructed.rewrite.include_domains,
+            vec!["*.cdn.example.com".to_owned()],
+            "should preserve the configured include list"
+        );
+    }
+
     #[test]
     fn disabled_rewrite_creatives_survives_blob_round_trip() {
         let mut original = test_settings();
diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs
index 0d3cad1c9..c4d0321d2 100644
--- a/crates/trusted-server-core/src/settings.rs
+++ b/crates/trusted-server-core/src/settings.rs
@@ -637,10 +637,23 @@ impl Ec {
 #[derive(Debug, Default, Clone, Deserialize, Serialize, Validate)]
 #[serde(deny_unknown_fields)]
 pub struct Rewrite {
-    /// List of domains to exclude from rewriting. Supports wildcards (e.g., "*.example.com").
-    /// URLs from these domains will not be proxied through first-party endpoints.
+    /// Hosts never rewritten, for assets and click-through links.
+    ///
+    /// Each entry is an exact host or `*.example.com`, which matches the apex
+    /// and any subdomain. Matching is case-insensitive. Wins over
+    /// [`Self::include_domains`].
     #[serde(default)]
     pub exclude_domains: Vec,
+    /// When non-empty, the only hosts whose asset URLs are rewritten to
+    /// `/first-party/proxy`.
+    ///
+    /// Same pattern syntax as [`Self::exclude_domains`]. Empty (the default)
+    /// proxies every eligible asset URL. Never applies to click-through links.
+    /// The default is omitted from serialized config so older binaries, which
+    /// reject unknown fields, can still read it.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    #[validate(custom(function = validate_include_domains))]
+    pub include_domains: Vec,
 }
 
 impl Rewrite {
@@ -686,6 +699,11 @@ impl Rewrite {
                 "rewrite.exclude_domains: removed empty or bare \"*\" entries, which never match a host"
             );
         }
+        self.include_domains = self
+            .include_domains
+            .iter()
+            .map(|pattern| pattern.trim().to_ascii_lowercase())
+            .collect();
     }
 }
 
@@ -3401,6 +3419,89 @@ impl Settings {
     }
 }
 
+/// Rejects `rewrite.include_domains` entries that can never equal a URL host.
+///
+/// Runs after [`Rewrite::normalize`], so entries are already trimmed and
+/// lowercased. Entries are matched against [`Url::host_str`], so each one must
+/// be:
+///
+/// - a DNS name, optionally prefixed with `*.`: one or more `.`-separated,
+///   non-empty labels of ASCII `a-z`, `0-9` and `-`;
+/// - an IPv4 address in dotted-quad form, such as `192.0.2.1`; or
+/// - an IPv6 address in brackets and compressed form, such as `[2001:db8::1]`.
+///
+/// A scheme, port, path, space, leading or trailing `.`, empty label, wildcard
+/// IP address or unbracketed IPv6 address is rejected with
+/// `invalid_rewrite_include_domain`. A non-ASCII (IDN) name is rejected with
+/// `non_ascii_rewrite_include_domain`; write it in punycode (`xn--`) form.
+fn validate_include_domains(patterns: &[String]) -> Result<(), ValidationError> {
+    for pattern in patterns {
+        if !pattern.is_ascii() {
+            let mut err = ValidationError::new("non_ascii_rewrite_include_domain");
+            err.add_param("value".into(), pattern);
+            err.message = Some(
+                "rewrite.include_domains entries must be ASCII; write internationalized domain names in punycode (xn--) form"
+                    .into(),
+            );
+            return Err(err);
+        }
+        if let Err(reason) = check_include_domain(pattern) {
+            let mut err = ValidationError::new("invalid_rewrite_include_domain");
+            err.add_param("value".into(), pattern);
+            err.add_param("reason".into(), &reason);
+            err.message = Some(
+                "rewrite.include_domains entries must be a host name, an optional leading `*.`, a dotted-quad IPv4 address or a bracketed IPv6 address"
+                    .into(),
+            );
+            return Err(err);
+        }
+    }
+    Ok(())
+}
+
+/// Checks one normalized ASCII `include_domains` entry, returning why it can
+/// never equal a URL host.
+fn check_include_domain(pattern: &str) -> Result<(), &'static str> {
+    if let Some(bracketed) = pattern.strip_prefix('[') {
+        let address = bracketed
+            .strip_suffix(']')
+            .and_then(|inner| inner.parse::().ok())
+            .ok_or("not a bracketed IPv6 address")?;
+        if format!("[{address}]") != pattern {
+            return Err("IPv6 address not in compressed form");
+        }
+        return Ok(());
+    }
+    if pattern.parse::().is_ok() {
+        return Err("IPv6 address without brackets");
+    }
+
+    let name = pattern.strip_prefix("*.").unwrap_or(pattern);
+    if name.is_empty() {
+        return Err("no host");
+    }
+    if !name.bytes().all(|byte| {
+        byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'-' | b'.')
+    }) {
+        return Err("character outside a-z, 0-9, `-` and `.`");
+    }
+    if name.split('.').any(str::is_empty) {
+        return Err("leading or trailing `.` or empty label");
+    }
+
+    // A URL host whose last label is numeric is parsed as an IPv4 address, so
+    // such an entry only matches as an exact dotted-quad address.
+    let last_label = name.rsplit('.').next().unwrap_or(name);
+    let numeric_last_label =
+        last_label.bytes().all(|byte| byte.is_ascii_digit()) || last_label.starts_with("0x");
+    if numeric_last_label
+        && (name.len() != pattern.len() || name.parse::().is_err())
+    {
+        return Err("numeric host that is not a dotted-quad IPv4 address");
+    }
+    Ok(())
+}
+
 fn validate_publisher_domain(value: &str) -> Result<(), ValidationError> {
     if value.trim() != value || value.is_empty() || value.len() > 253 {
         return Err(ValidationError::new("invalid_publisher_domain"));
@@ -6617,6 +6718,107 @@ source_domain = "partner.example.com"
         );
     }
 
+    #[test]
+    fn rewrite_include_domains_are_normalized_from_toml() {
+        let toml_str = crate_test_settings_str()
+            + r#"
+            [rewrite]
+            include_domains = ["*.CDN.Example.com", "  img.example.net "]
+            "#;
+
+        let settings = Settings::from_toml(&toml_str).expect("should parse valid TOML");
+
+        assert_eq!(
+            settings.rewrite.include_domains,
+            vec!["*.cdn.example.com".to_owned(), "img.example.net".to_owned()],
+            "should trim and lowercase include_domains entries"
+        );
+    }
+
+    fn include_domains_toml(entry: &str) -> String {
+        crate_test_settings_str() + &format!("\n[rewrite]\ninclude_domains = [\"{entry}\"]\n")
+    }
+
+    #[test]
+    fn rewrite_include_domains_reject_malformed_entries() {
+        for entry in [
+            "",
+            "   ",
+            "*",
+            "*.",
+            "cdn.*.example.com",
+            "**.example.com",
+            "https://cdn.example.com",
+            "cdn.example.com:443",
+            "cdn.example.com/x",
+            ".example.com",
+            "example.com.",
+            "cdn..example.com",
+            "cdn example.com",
+            "cdn_assets.example.com",
+            "::1",
+            "[::1",
+            "[2001:0db8::1]",
+            "*.192.0.2.1",
+            "192.0.2",
+            "256.0.2.1",
+        ] {
+            let result = Settings::from_toml(&include_domains_toml(entry));
+
+            let err = result.expect_err(&format!("should reject include_domains entry `{entry}`"));
+            assert!(
+                format!("{err:?}").contains("invalid_rewrite_include_domain"),
+                "should report `{entry}` as an invalid include_domains entry: {err:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn rewrite_include_domains_reject_non_ascii_entries_with_a_punycode_hint() {
+        for entry in ["bücher.example", "*.bücher.example"] {
+            let result = Settings::from_toml(&include_domains_toml(entry));
+
+            let err = result.expect_err(&format!("should reject non-ASCII entry `{entry}`"));
+            assert!(
+                format!("{err:?}").contains("non_ascii_rewrite_include_domain"),
+                "should report `{entry}` as non-ASCII: {err:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn rewrite_include_domains_accept_hosts_and_ip_literals() {
+        for entry in [
+            "cdn.example.com",
+            "*.example.com",
+            "xn--bcher-kva.example",
+            "192.0.2.1",
+            "[2001:db8::1]",
+            "[::1]",
+        ] {
+            let settings =
+                Settings::from_toml(&include_domains_toml(entry)).unwrap_or_else(|err| {
+                    panic!("should accept include_domains entry `{entry}`: {err:?}")
+                });
+
+            assert_eq!(
+                settings.rewrite.include_domains,
+                vec![entry.to_owned()],
+                "should keep accepted entry `{entry}` as written"
+            );
+        }
+    }
+
+    #[test]
+    fn rewrite_include_domains_default_to_empty() {
+        let settings = create_test_settings();
+
+        assert!(
+            settings.rewrite.include_domains.is_empty(),
+            "should default include_domains to an empty list"
+        );
+    }
+
     #[test]
     fn test_auction_creative_processing_defaults_when_omitted() {
         let toml_str = crate_test_settings_str()

From ef8ad67f9a4e6161c9fcfd950d0a44e4e166bb1d Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:09:34 +0530
Subject: [PATCH 155/230] Limit creative asset rewriting to rewrite
 include_domains

Apply the include list to every asset handler and to first-party sign. Click-through links and fetch-time proxy checks are unchanged.

Signed-off-by: dhruv8sh 
---
 crates/trusted-server-core/src/creative.rs | 215 ++++++++++++++++++++-
 crates/trusted-server-core/src/proxy.rs    | 119 +++++++++++-
 crates/trusted-server-core/src/settings.rs |  44 ++++-
 3 files changed, 371 insertions(+), 7 deletions(-)

diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 074e5f35d..14f3bbd5a 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -24,6 +24,8 @@
 //!   click guard.
 //! - Hosts matching `[rewrite] exclude_domains` are left untouched, for assets
 //!   and links alike.
+//! - When `[rewrite] include_domains` is non-empty, only asset URLs on listed
+//!   hosts are proxied. Links ignore it.
 //! - Bidder-supplied `` elements are removed so root-relative first-party
 //!   URLs cannot be rebased.
 //! - Relative URLs (e.g., `/path`, `../path`, `local/file`) remain unchanged.
@@ -1614,8 +1616,9 @@ impl StreamProcessor for CreativeCssProcessor<'_> {
 mod tests {
     use super::{
         CreativeCssProcessor, StreamProcessor as _, normalize_creative_url,
-        process_auction_creative, proxy_if_abs, rewrite_creative_html,
-        rewrite_inline_creative_html, rewrite_srcset, rewrite_style_urls, sanitize_creative_html,
+        process_auction_creative, proxy_if_abs, rewrite_creative_html, rewrite_css_body,
+        rewrite_inline_creative_html, rewrite_proxied_html, rewrite_srcset, rewrite_style_urls,
+        sanitize_creative_html,
     };
 
     use crate::settings::Settings;
@@ -3795,6 +3798,214 @@ b{background:url(\"https://cdn.example/c.png\")}";
         );
     }
 
+    const ALLOWLIST_LISTED: &[&str] = &[
+        "https://assets.example.com/a.css",
+        "https://assets.example.com/p1.png",
+        "https://assets.example.com/p1-2x.png",
+        "https://assets.example.com/a.js",
+        "https://assets.example.com/i.css",
+        "https://assets.example.com/bg.png",
+        "https://assets.example.com/set.png",
+        "https://assets.example.com/img.png",
+        "https://assets.example.com/lazy.png",
+        "https://assets.example.com/s1.png",
+        "https://assets.example.com/v.mp4",
+        "https://assets.example.com/o.swf",
+        "https://assets.example.com/e.swf",
+        "https://assets.example.com/btn.png",
+        "https://assets.example.com/icon.svg",
+        "https://assets.example.com/sprite.svg",
+        "https://assets.example.com/frame.html",
+        "https://assets.example.com/inline.png",
+    ];
+
+    const ALLOWLIST_OFF_LIST: &[&str] = &[
+        "https://cdn.example.net/a.css",
+        "https://cdn.example.net/p2.png",
+        "https://cdn.example.net/a.js",
+        "https://cdn.example.net/i.css",
+        "https://cdn.example.net/bg.png",
+        "https://cdn.example.net/set.png",
+        "https://cdn.example.net/img.png",
+        "https://cdn.example.net/lazy.png",
+        "https://cdn.example.net/s2.png",
+        "https://cdn.example.net/v.mp4",
+        "https://cdn.example.net/o.swf",
+        "https://cdn.example.net/e.swf",
+        "https://cdn.example.net/btn.png",
+        "https://cdn.example.net/icon.svg",
+        "https://cdn.example.net/sprite.svg",
+        "https://cdn.example.net/frame.html",
+        "https://cdn.example.net/inline.png",
+    ];
+
+    const ALLOWLIST_CREATIVE: &str = r#"
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

+

+Offer
+Area
+"#;
+
+    fn encoded_tsurl(url: &str) -> String {
+        format!(
+            "tsurl={}",
+            url::form_urlencoded::byte_serialize(url.as_bytes()).collect::()
+        )
+    }
+
+    fn allowlist_settings() -> crate::settings::Settings {
+        let mut settings = crate::test_support::tests::create_test_settings();
+        settings.rewrite.include_domains = vec!["*.example.com".to_owned()];
+        settings
+    }
+
+    fn assert_allowlist_applied(label: &str, out: &str) {
+        for listed in ALLOWLIST_LISTED {
+            assert!(
+                out.contains(&encoded_tsurl(listed)),
+                "{label}: should proxy listed URL `{listed}`: {out}"
+            );
+        }
+        for off_list in ALLOWLIST_OFF_LIST {
+            assert!(
+                !out.contains(&encoded_tsurl(off_list)),
+                "{label}: should not proxy off-list URL `{off_list}`: {out}"
+            );
+            assert!(
+                out.contains(off_list),
+                "{label}: should keep off-list URL `{off_list}` raw: {out}"
+            );
+        }
+        for landing in [
+            "https://landing.example.net/offer",
+            "https://landing.example.org/area",
+        ] {
+            assert!(
+                out.contains(&format!("/first-party/click?{}", encoded_tsurl(landing))),
+                "{label}: should wrap click `{landing}` regardless of include_domains: {out}"
+            );
+        }
+        assert_eq!(
+            out.matches("data-tsclick=").count(),
+            2,
+            "{label}: should mark both click-through links"
+        );
+    }
+
+    #[test]
+    fn include_domains_limit_asset_rewriting_on_every_html_path() {
+        let settings = allowlist_settings();
+
+        assert_allowlist_applied(
+            "auction",
+            &rewrite_creative_html(&settings, ALLOWLIST_CREATIVE),
+        );
+        assert_allowlist_applied(
+            "inline",
+            &rewrite_inline_creative_html(&settings, "https://www.example.com", ALLOWLIST_CREATIVE),
+        );
+        assert_allowlist_applied(
+            "proxied",
+            &rewrite_proxied_html(&settings, ALLOWLIST_CREATIVE),
+        );
+    }
+
+    #[test]
+    fn include_domains_limit_css_body_rewriting() {
+        let settings = allowlist_settings();
+        let css = r#"@import "https://assets.example.com/i.css";
+@import "https://cdn.example.net/i.css";
+.a { background: url(https://assets.example.com/bg.png); }
+.b { background: url(https://cdn.example.net/bg.png); }"#;
+
+        let out = rewrite_css_body(&settings, css).expect("should rewrite CSS body");
+
+        for listed in [
+            "https://assets.example.com/i.css",
+            "https://assets.example.com/bg.png",
+        ] {
+            assert!(
+                out.contains(&encoded_tsurl(listed)),
+                "should proxy listed CSS URL `{listed}`: {out}"
+            );
+        }
+        for off_list in [
+            "https://cdn.example.net/i.css",
+            "https://cdn.example.net/bg.png",
+        ] {
+            assert!(
+                out.contains(off_list) && !out.contains(&encoded_tsurl(off_list)),
+                "should keep off-list CSS URL `{off_list}` raw: {out}"
+            );
+        }
+    }
+
+    #[test]
+    fn host_in_both_lists_is_left_alone() {
+        let mut settings = allowlist_settings();
+        settings.rewrite.exclude_domains = vec!["assets.example.com".to_owned()];
+        let html = ALLOWLIST_CREATIVE.replace(
+            "",
+            r#"Excluded"#,
+        );
+
+        for (label, out) in [
+            ("auction", rewrite_creative_html(&settings, &html)),
+            (
+                "inline",
+                rewrite_inline_creative_html(&settings, "https://www.example.com", &html),
+            ),
+            ("proxied", rewrite_proxied_html(&settings, &html)),
+        ] {
+            assert!(
+                !out.contains("tsurl=https%3A%2F%2Fassets.example.com"),
+                "{label}: should not rewrite any URL on a host in both lists: {out}"
+            );
+            for listed in ALLOWLIST_LISTED {
+                assert!(
+                    out.contains(listed),
+                    "{label}: should keep `{listed}` raw when its host is also excluded: {out}"
+                );
+            }
+            assert!(
+                out.contains(r#"Excluded"#),
+                "{label}: should leave an excluded click-through link raw: {out}"
+            );
+            assert_eq!(
+                out.matches("data-tsclick=").count(),
+                2,
+                "{label}: should wrap only the two links on hosts that are not excluded: {out}"
+            );
+        }
+    }
+
     #[test]
     fn rewrite_html_excludes_blacklisted_domains() {
         let mut settings = crate::test_support::tests::create_test_settings();
diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs
index 1b9139753..7f6f1364e 100644
--- a/crates/trusted-server-core/src/proxy.rs
+++ b/crates/trusted-server-core/src/proxy.rs
@@ -1225,7 +1225,7 @@ fn append_ec_id(req: &Request, target_url_parsed: &mut url::Url) {
 ///
 /// When `allowed_domains` is empty every host is permitted (open mode).
 /// When non-empty the host must match at least one pattern via [`is_host_allowed`].
-fn is_host_permitted>(allowed_domains: &[S], host: &str) -> bool {
+pub(crate) fn is_host_permitted>(allowed_domains: &[S], host: &str) -> bool {
     allowed_domains.is_empty()
         || allowed_domains
             .iter()
@@ -1690,7 +1690,9 @@ pub async fn handle_first_party_proxy_sign(
         })
     })?;
     if !settings.rewrite.should_proxy_asset(host) {
-        log::debug!("sign request for `{host}` declined by rewrite policy");
+        log::debug!(
+            "sign request for `{host}` declined: host excluded or not in rewrite.include_domains"
+        );
         return Err(Report::new(TrustedServerError::Proxy {
             message: "unsupported url".to_string(),
         }));
@@ -2745,6 +2747,119 @@ mod tests {
         });
     }
 
+    #[test]
+    fn proxy_sign_applies_rewrite_include_domains_for_get_and_post() {
+        struct Case {
+            name: &'static str,
+            include_domains: &'static [&'static str],
+            exclude_domains: &'static [&'static str],
+            allowed_domains: &'static [&'static str],
+            target: &'static str,
+            expected: StatusCode,
+        }
+
+        let cases = [
+            Case {
+                name: "listed absolute",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::OK,
+            },
+            Case {
+                name: "listed protocol-relative",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "//img.example.com/a.png",
+                expected: StatusCode::OK,
+            },
+            Case {
+                name: "off-list absolute",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "off-list protocol-relative",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "//img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "host in both lists",
+                include_domains: &["*.example.com"],
+                exclude_domains: &["img.example.com"],
+                allowed_domains: &[],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "off include list and off proxy allowlist",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &["*.example.org"],
+                target: "https://img.example.net/a.png",
+                expected: StatusCode::BAD_GATEWAY,
+            },
+            Case {
+                name: "listed but off proxy allowlist",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &["*.example.org"],
+                target: "https://img.example.com/a.png",
+                expected: StatusCode::FORBIDDEN,
+            },
+            Case {
+                name: "mixed-case host",
+                include_domains: &["*.example.com"],
+                exclude_domains: &[],
+                allowed_domains: &[],
+                target: "https://IMG.Example.COM/a.png",
+                expected: StatusCode::OK,
+            },
+        ];
+
+        let to_owned = |values: &[&str]| -> Vec {
+            values.iter().map(|value| (*value).to_owned()).collect()
+        };
+
+        futures::executor::block_on(async {
+            for case in &cases {
+                for method in [&Method::GET, &Method::POST] {
+                    let label = format!("{} {}", method.as_str(), case.name);
+                    let mut settings = create_test_settings();
+                    settings.rewrite.include_domains = to_owned(case.include_domains);
+                    settings.rewrite.exclude_domains = to_owned(case.exclude_domains);
+                    settings.proxy.allowed_domains = to_owned(case.allowed_domains);
+                    let req = build_proxy_sign_request(
+                        method,
+                        "https://edge.example.com/first-party/sign",
+                        case.target,
+                    );
+
+                    let status =
+                        match handle_first_party_proxy_sign(&settings, &noop_services(), req).await
+                        {
+                            Ok(response) => response.status(),
+                            Err(error) => error.current_context().status_code(),
+                        };
+
+                    assert_eq!(
+                        status, case.expected,
+                        "{label} should return {}",
+                        case.expected
+                    );
+                }
+            }
+        });
+    }
+
     #[test]
     fn proxy_sign_inherits_the_request_scheme_for_protocol_relative_urls() {
         // A protocol-relative target adopts the scheme the visitor is on. It
diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs
index c4d0321d2..4f478e575 100644
--- a/crates/trusted-server-core/src/settings.rs
+++ b/crates/trusted-server-core/src/settings.rs
@@ -23,7 +23,7 @@ use crate::creative_opportunities::CreativeOpportunitiesConfig;
 use crate::error::TrustedServerError;
 use crate::host_header::validate_host_header_override_value;
 use crate::platform::PlatformImageOptimizerRegion;
-use crate::proxy::is_host_allowed;
+use crate::proxy::{is_host_allowed, is_host_permitted};
 use crate::redacted::Redacted;
 
 #[cfg(test)]
@@ -660,11 +660,12 @@ impl Rewrite {
     /// Returns `true` when an asset URL on `host` should be rewritten to
     /// `/first-party/proxy`.
     ///
-    /// The host must not match [`Self::exclude_domains`]. Matching is
+    /// The host must not match [`Self::exclude_domains`], and must match
+    /// [`Self::include_domains`] when that list is non-empty. Matching is
     /// case-insensitive; see [`is_host_allowed`] for the pattern rules.
     #[must_use]
     pub fn should_proxy_asset(&self, host: &str) -> bool {
-        !self.is_excluded_host(host)
+        !self.is_excluded_host(host) && is_host_permitted(&self.include_domains, host)
     }
 
     /// Returns `true` when a click-through URL on `host` should be wrapped in
@@ -6718,6 +6719,43 @@ source_domain = "partner.example.com"
         );
     }
 
+    #[test]
+    fn rewrite_should_proxy_asset_honors_include_list() {
+        let mut rewrite = Rewrite::default();
+        rewrite
+            .include_domains
+            .extend(["*.example.com", "img.example.net"].map(str::to_owned));
+        rewrite
+            .exclude_domains
+            .push("blocked.example.com".to_owned());
+
+        for (host, expected) in [
+            ("example.com", true),
+            ("a.b.example.com", true),
+            ("img.example.net", true),
+            ("IMG.EXAMPLE.NET", true),
+            ("evil-example.com", false),
+            ("cdn.example.org", false),
+            ("blocked.example.com", false),
+        ] {
+            assert_eq!(
+                rewrite.should_proxy_asset(host),
+                expected,
+                "should_proxy_asset(`{host}`) should be {expected}"
+            );
+        }
+        for host in ["cdn.example.org", "evil-example.com", "example.com"] {
+            assert!(
+                rewrite.should_wrap_click(host),
+                "should wrap click for `{host}` regardless of include_domains"
+            );
+        }
+        assert!(
+            !rewrite.should_wrap_click("blocked.example.com"),
+            "should not wrap a click to an excluded host"
+        );
+    }
+
     #[test]
     fn rewrite_include_domains_are_normalized_from_toml() {
         let toml_str = crate_test_settings_str()

From 9d9fb273320e094e127bba0d4e0f0fd21a10537d Mon Sep 17 00:00:00 2001
From: dhruv8sh 
Date: Wed, 7 Oct 2026 15:09:54 +0530
Subject: [PATCH 156/230] Rewrite link imagesrcset once on the inline creative
 path

Signed-off-by: dhruv8sh 
---
 crates/trusted-server-core/src/creative.rs | 55 +++++++++++++++++++---
 1 file changed, 48 insertions(+), 7 deletions(-)

diff --git a/crates/trusted-server-core/src/creative.rs b/crates/trusted-server-core/src/creative.rs
index 14f3bbd5a..e468fbe28 100644
--- a/crates/trusted-server-core/src/creative.rs
+++ b/crates/trusted-server-core/src/creative.rs
@@ -16,7 +16,7 @@
 //!   - ``
 //!   - SVG: ``, ``
 //!   - `