Skip to content

Preserve AWF ranked choices and request/failure metadata in unified sessions #69848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SivaKesava1 merged 4 commits into from
Oct 18, 2026
+410 −8
Merged

Preserve AWF ranked choices and request/failure metadata in unified sessions #69848

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1a7f37e
Initial plan
Copilot Oct 18, 2026
d5581f9
Preserve AWF ranked choices and request/failure metadata in unified s…
Copilot Oct 18, 2026
8f82558
Merge remote-tracking branch 'origin/main' into copilot/g-2-g-11-pres…
Copilot Oct 18, 2026
70c457c
Merge branch 'main' into copilot/g-2-g-11-preserve-candidate-rankings
SivaKesava1 Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions actions/setup/js/types/unified_session.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,26 @@ export interface ModelRoutingData {
deviations?: JsonValue;
requestedModel?: JsonValue;
requestedEffort?: JsonValue;
/** AWF route choices in rank order; ranks are one-based array positions. */
rankedChoices?: ModelRoutingRankedChoice[];
eligibleChoices?: JsonValue;
catalogueOverlap?: JsonValue;
selectedEndpoint?: JsonValue;
requestedEndpoint?: JsonValue;
upstreamEndpoint?: JsonValue;
/** Request-telemetry dimensions AWF did not observe, such as `model` and `effort`; not unavailable models. */
unavailable?: string[];
/** Observed HTTP status, independent of `outcome`. */
status?: JsonValue;
/** AWF failure phase, such as `primary` or `bootstrap`; not the collection provenance phase. */
phase?: JsonValue;
}

export interface ModelRoutingRankedChoice {
rank: number;
id?: string;
model?: string;
effort?: string | null;
}

export interface ModelEndpointMismatchData {
Expand Down
39 changes: 39 additions & 0 deletions actions/setup/js/unified_session_payload.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,15 @@ const EVENT_FIELDS = {
requestedModel: ["requested_model", "requestedModel"],
requestedEffort: ["requested_effort", "requestedEffort"],
outcome: ["outcome"],
rankedChoices: ["ranked_choices", "rankedChoices"],
eligibleChoices: ["eligible_choices", "eligibleChoices"],
catalogueOverlap: ["catalogue_overlap", "catalogueOverlap"],
selectedEndpoint: ["selected_endpoint", "selectedEndpoint"],
requestedEndpoint: ["requested_endpoint", "requestedEndpoint"],
upstreamEndpoint: ["upstream_endpoint", "upstreamEndpoint"],
unavailable: ["unavailable"],
status: ["status"],
phase: ["phase"],
},
"safe_output.request": SAFE_OUTPUT_FIELDS,
"safe_output.result": SAFE_OUTPUT_FIELDS,
Expand Down Expand Up @@ -526,6 +535,24 @@ function normalizeModelInventoryData(value) {
return { endpoints, ...(modelsFetchComplete !== undefined ? { modelsFetchComplete } : {}) };
}

/**
* AWF persists route choices in rank order; keep only the position, optional ID, model, and supplied effort.
* @param {any} choices
* @returns {import("./types/unified_session").ModelRoutingRankedChoice[] | undefined}
*/
function normalizeRankedChoices(choices) {

@github-actions github-actions Bot Oct 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] normalizeRankedChoices returns undefined for any non-array input (string, number, object), deleting rankedChoices entirely per the caller at line 576. This fallback path is untested; all current test inputs supply either a valid array or omit the field.

Suggested test

Add cases where ranked_choices is a string, a number, or a plain object, and assert the projected data has no rankedChoices property.

Also worth a quick case for a non-string id (e.g. id: 123) on a candidate entry, to confirm it is silently dropped rather than coerced; the current typeof choice.id === string check suggests this is intended but it is not currently verified.

@copilot please address this.

if (!Array.isArray(choices)) return undefined;
return choices.map((choice, index) => {
Comment thread
SivaKesava1 marked this conversation as resolved.
/** @type {import("./types/unified_session").ModelRoutingRankedChoice} */
const entry = { rank: index + 1 };
if (choice === null || typeof choice !== "object" || Array.isArray(choice)) return entry;
if (typeof choice.id === "string") entry.id = choice.id;
if (typeof choice.model === "string") entry.model = choice.model;
if (typeof choice.effort === "string" || choice.effort === null) entry.effort = choice.effort;
return entry;
});
}

/**
* Compact known payloads without truncating text or changing native parser traces.
* Unknown extension data remains opaque for forward compatibility.
Expand All @@ -543,6 +570,18 @@ function normalizeUnifiedSessionEvent(event, phase) {
if (inventory === undefined) throw new Error(`${ERR_VALIDATION}: Model inventory snapshot must contain valid endpoint model collections`);
Object.assign(data, inventory);
}
if (event.type === "firewall.model_routing") {
if (Object.hasOwn(data, "rankedChoices")) {
const rankedChoices = normalizeRankedChoices(data.rankedChoices);
if (rankedChoices === undefined) delete data.rankedChoices;
else data.rankedChoices = rankedChoices;
}
// AWF lists unobserved request-telemetry dimensions here, not unavailable models.
if (Object.hasOwn(data, "unavailable") && !(Array.isArray(data.unavailable) && data.unavailable.every(value => typeof value === "string"))) delete data.unavailable;

@github-actions github-actions Bot Oct 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The guard that drops unavailable unless it is an array of strings has no negative-case test. Current coverage only supplies a valid array or omits the field entirely.

💡 Suggested tests 
for (const bad of ["not-an-array", { nested: true }, ["model", 42]]) {
  const projected = project(routingEvent({ ...SMOKE_REQUEST, unavailable: bad }));
  expect(projected.data).not.toHaveProperty("unavailable");
}

The mixed-type array case (["model", 42]) is the most likely regression target since .every(value => typeof value === "string") is easy to accidentally weaken (e.g. to .some) without a failing test catching it.

@copilot please address this.

for (const key of ["eligibleChoices", "catalogueOverlap", "selectedEndpoint", "requestedEndpoint", "upstreamEndpoint", "status", "phase"]) {
if (data[key] !== null && typeof data[key] === "object") delete data[key];
Comment on lines +580 to +582

@github-actions github-actions Bot Oct 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The malformed-scalar-drop loop (dropping eligibleChoices, catalogueOverlap, selectedEndpoint, requestedEndpoint, upstreamEndpoint, status, phase when they are objects) is explicitly documented in the PR description ("Malformed scalars dropped"), but no test in unified_session_payload.test.cjs exercises it for any of these 7 fields.

💡 Suggested test 
it.each(["eligibleChoices", "catalogueOverlap", "selectedEndpoint", "requestedEndpoint", "upstreamEndpoint", "status", "phase"])("drops %s when AWF supplies an object instead of a scalar", key => {
  const snakeKey = SOURCE_FIELDS[key][0];
  const record = { ...SMOKE_SELECTION, [snakeKey]: { nested: true } };
  const projected = project(routingEvent(record));
  expect(projected.data).not.toHaveProperty(key);
});

Without this, a future refactor of the drop loop (e.g. an off-by-one in the field list, or an accidental removal) would ship silently, since nothing currently fails if the loop is deleted entirely.

@copilot please address this.

}
}
if (event.type === "workflow.info") {
const awContext = Object.hasOwn(source, "awContext") ? source.awContext : source.context;
if (awContext && typeof awContext === "object" && !Array.isArray(awContext)) {
Expand Down
Loading
Loading