Skip to content

Structure generated issues with a normalized report AST - #69756

Merged
pelikhan merged 6 commits into
mainfrom
pelikhan-failure-issue-presentation
Oct 17, 2026
Merged

pelikhan merged 6 commits into
mainfrom
pelikhan-failure-issue-presentation

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Why

Automated [aw] failure issues buried actionable causes beneath diagnostics and repeated debugging instructions. Issue creators also assembled Markdown independently, making structure, escaping, and hidden metadata inconsistent.

Approach

  • Introduce a typed report AST, normalizer, and GitHub Markdown renderer. Render the complete failure report, including detection warnings, attribution, expiration, and provenance, from one document.
  • Give failure issues consistent [aw] Workflow: reason titles and What happened, Next step, and Failure details sections. Keep classified causes visible, collapse secondary runtime evidence, and provide one copyable debugging prompt.
  • Route JavaScript GitHub REST and GraphQL issue creation through a shared publication boundary. Normalize Linear and Jira issue content while preserving their provider-specific output formats and limits.
  • Add semantic nodes for explicitly enabled or disabled closing references, JSON structured data, hidden XML/JSON metadata, and allowed mentions. Validate and escape values without breaking expiration, grouping, deduplication, or managed-section readers.
  • Add regression coverage that prevents production REST issue creators from bypassing the AST publisher.

Compatibility

The Markdown importer intentionally supports the runtime template dialect rather than all CommonMark syntax. Unsupported HTML is rendered literally. Existing authentication, retry behavior, mention budgets, and deliberately escaped closing references are retained. This changes future generated reports; existing issues are not rewritten.

Validation

  • 2,367 tests across 38 issue-creation, AST, and compatibility suites passed, including reruns of Git-fixture tests with TMPDIR=/tmp and a longer timeout for macOS.
  • Eight before/after body comparisons passed against the pre-AST failure template and legacy publication inputs.
  • Managed issue-clustering tests: 30 passed.
  • Formatting, linting, type checking, and make agent-report-progress passed.

Full generated-body previews for the original missing-tools, unknown-cause, and missing-model-pricing failures are provided in a separate PR comment.

pelikhan and others added 2 commits October 17, 2026 09:42
Render failure summaries, diagnostics, attribution, expiration, and machine-readable markers through a single escaping and normalization pipeline.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add semantic closing references, structured data, and safe hidden metadata nodes. Route REST, GraphQL, and provider issue creation through shared report normalization with compatibility regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

pelikhan commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Generated failure issue previews

These previews render the three original failure examples through the new report AST. Original run links, diagnostics, footer attribution, expiration, and failure-match metadata are retained; the generic diagnostic fallback uses the updated wording. The source issues have not been modified.

Missing required tools: Smoke Codex

Source: #69737

New title: [aw] Smoke Codex: is missing required tools

What happened

Smoke Codex is missing required tools.

View run #39046125927 · Workflow source · Previous failures

Branch: copilot/bump-mcp-gateway-v0435

Next step

Review the failure details below for a targeted fix, or assign this issue to a coding agent to investigate.

Failure details

[!WARNING]
Missing Tools Reported: The agent reported missing tools during execution.

Missing Tools:

  • GitHub MCP and web-fetch MCP: Neither MCP interface is available in this engine. The permitted gh read returned exactly two merged PR titles and numbers; web-fetch was not substituted.
  • report_progress: Repository instructions require report_progress for changed files, but it is not exposed among the enabled safe-output tools. The required make build syncs generated action-pin data; direct authenticated commits/pushes are not available or permitted.
Copy a debugging prompt

Paste this into your coding agent:

Debug the agentic workflow "smoke-codex" failure.
Follow https://raw.githubusercontent.com/github/gh-aw/main/debug.md
Failed run: https://github.com/github/gh-aw/actions/runs/39046125927

If the repository includes the agentic-workflows skill, load it before investigating.

Manage failure notifications

To stop creating failure issues for this workflow, set this in its frontmatter and recompile:

safe-outputs:
  report-failure-as-issue: false

This only disables issue reporting; it does not fix the failure.

Generated from Smoke Codex · codex · 10.5 AIC · ◷

  • expires on Oct 18, 2026, 3:56 AM UTC
Unknown failure cause: Smoke Gemini

Source: #69732

New title: [aw] Smoke Gemini: failed (cause unknown)

What happened

Smoke Gemini failed (cause unknown).

View run #39046126074 · Workflow source · Previous failures

Branch: copilot/bump-mcp-gateway-v0435

Pull Request: #69581

Next step

Review the failure details below for a targeted fix, or assign this issue to a coding agent to investigate.

Failure details

Runtime evidence

[!WARNING]
Task Could Not Be Completed: The workflow recorded a report_incomplete signal: the task was not completed or completion could not be confirmed.

Error details:
Failure classification: unknown
Driver exit code: 41. The engine driver exited before a terminal safe output was recorded.
Last tool call status: not recorded
Failed tool calls observed: 0
Token budget state: unknown (no exhaustion signal recorded)
Step budget state (LLM invocations): unknown (no exhaustion signal recorded)
Safe-output tools: available at gateway snapshot (registered 7 tools)
Safe-output CLI calls observed: unknown (not recorded)
Safe-output MCP calls observed: 0
Safe-output persistence: not recorded
Agent finished without emitting a terminal safe output; task completion could not be confirmed.
Last engine error type: unknown
Retry attempts observed: 0

A comment or other safe output does not confirm that the requested task was completed. Only inferred runtime status and error codes are reported; agent text is excluded.

Failure Diagnostics

Agent job: agent
Agent job conclusion: failure
Failing step: Execute Gemini CLI

No additional engine error was captured. Check the reported failure signals and linked run for details.

Only runtime metadata is shown; agent-generated text is excluded.

Copy a debugging prompt

Paste this into your coding agent:

Debug the agentic workflow "smoke-gemini" failure.
Follow https://raw.githubusercontent.com/github/gh-aw/main/debug.md
Failed run: https://github.com/github/gh-aw/actions/runs/39046126074

If the repository includes the agentic-workflows skill, load it before investigating.

Manage failure notifications

To stop creating failure issues for this workflow, set this in its frontmatter and recompile:

safe-outputs:
  report-failure-as-issue: false

This only disables issue reporting; it does not fix the failure.

Generated from Smoke Gemini · ◷

  • expires on Oct 18, 2026, 3:54 AM UTC
Missing model pricing: Smoke Copilot - AOAI (Entra)

Source: #69743

New title: [aw] Smoke Copilot - AOAI (Entra): has no AI credits pricing for model (o4-mini-aw)

What happened

Smoke Copilot - AOAI (Entra) has no AI credits pricing for model (o4-mini-aw).

View run #39046722029 · Workflow source · Previous failures

Branch: copilot/bump-mcp-gateway-v0435

Next step

Review the failure details below for a targeted fix, or assign this issue to a coding agent to investigate.

Failure details

[!WARNING]
Model has no AI credits pricing: The agent failed because model o4-mini-aw is not in the built-in pricing table and no default fallback pricing is configured. The AWF API proxy rejected every inference request with HTTP 400.

This is a configuration issue — retrying will not help. The model must have pricing before the workflow can run.

How to fix this

Option 1 — Add pricing in the workflow frontmatter:

models:
  providers:
    github-copilot:
      models:
        'o4-mini-aw':
          cost:
            # Placeholder values — replace with actual pricing for this model
            input: "0e0"      # $0.00 per million input tokens
            output: "0e0"     # $0.00 per million output tokens
            cache_read: "0e0"  # $0.00 per million cache-read tokens
            cache_write: "0e0" # $0.00 per million cache-write tokens

Use the provider key matching your engine: github-copilot (Copilot), anthropic (Claude), openai (Codex), or google (Gemini). Only input and output are required; the rest default to zero (or output for reasoning).

Option 2 — Map the model to a known model using the models field:

If o4-mini-aw is an alias for a model already in the built-in pricing table, use the models frontmatter field to provide the mapping:

models:
  'o4-mini-aw':
    - claude-sonnet-4-5

Option 3 — Switch to a model already in the built-in pricing table:

Replace o4-mini-aw in the workflow frontmatter with a model name that the AWF pricing system recognizes (e.g. claude-sonnet-4-5, gpt-4.1, gemini-2.0-flash).

Runtime evidence

[!WARNING]
Task Could Not Be Completed: The workflow recorded a report_incomplete signal: the task was not completed or completion could not be confirmed.

Error details:
Failure classification: request_rejection
Driver exit code: 1. The engine driver exited before a terminal safe output was recorded.
Last tool call status: not recorded
Failed tool calls observed: 0
Token budget state: unknown (no exhaustion signal recorded)
Step budget state (LLM invocations): unknown (no exhaustion signal recorded)
Safe-output tools: available at gateway snapshot (registered 18 tools)
Safe-output CLI calls observed: unknown (not recorded)
Safe-output MCP calls observed: 0
Safe-output persistence: not recorded
Engine error: Model has no AI credits pricing (missing_model_pricing_error)
Agent finished without emitting a terminal safe output; task completion could not be confirmed.
Engine HTTP error codes: 400
Last engine error type: missing_model_pricing_error
Retry attempts observed: 0

A comment or other safe output does not confirm that the requested task was completed. Only inferred runtime status and error codes are reported; agent text is excluded.

Failure Diagnostics

Agent job: agent
Agent job conclusion: failure
Failing step: Execute GitHub Copilot CLI

No additional engine error was captured. Check the reported failure signals and linked run for details.

Only runtime metadata is shown; agent-generated text is excluded.

Copy a debugging prompt

Paste this into your coding agent:

Debug the agentic workflow "smoke-copilot-aoai-entra" failure.
Follow https://raw.githubusercontent.com/github/gh-aw/main/debug.md
Failed run: https://github.com/github/gh-aw/actions/runs/39046722029

If the repository includes the agentic-workflows skill, load it before investigating.

Manage failure notifications

To stop creating failure issues for this workflow, set this in its frontmatter and recompile:

safe-outputs:
  report-failure-as-issue: false

This only disables issue reporting; it does not fix the failure.

Generated from Smoke Copilot - AOAI (Entra) · ◷

  • expires on Oct 18, 2026, 3:58 AM UTC

Render report sections as H3 and keep imported diagnostic headings at H4 or below. Cover the heading hierarchy with regression assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review October 17, 2026 17:28
Copilot AI max review requested due to automatic review settings October 17, 2026 17:28
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #69756

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · 14 AIC · ⌖ 15.1 AIC · ◇ 6.62 AIC · ⊞ 12.6K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · 11.1 AIC · ⌖ 36.2 AIC · ◇ 14.3 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L82-90: yagni: full GitHub Report AST constructed only to re-extract a title and literal ADF description. Call normalizeReportTitle/raw text normalization directly before requiredString; Jira never renders or consumes report nodes.

net: -7 lines possible.

Generated by ✂️ Ponytail Reviewer for #69756 · codex · gpt56 · 22.8 AIC · ⌖ 5.35 AIC · ⊞ 14.6K
Comment /ponytail to run again

Comment thread actions/setup/js/jira_handlers.cjs Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /improve-codebase-architecture and /codebase-design (pr-triage: architecture_change). This is a well-executed deepening refactor — one minor cleanup item noted inline, otherwise approving.

📋 Key Themes & Highlights

Key Themes

  • Dead code after refactor: generateFailureMatchMarker in handle_agent_failure.cjs is superseded by buildFailureMatchMetadata + the new buildFailureReportFooter boundary and is no longer called, exported, or tested. Flagged inline for removal.

Positive Highlights

  • ✅ Deep module design: report_ast.cjs exposes a narrow surface (normalizeReport, renderReport, a few helpers) while internally handling escaping, HTTPS validation, depth limits, and redaction — exactly the "simple interface, rich behavior" shape /improve-codebase-architecture looks for.
  • ✅ Single publication boundary: issue_report.cjs consolidates GitHub REST/GraphQL/Linear/Jira issue creation through renderIssueParameters/createReportIssue. Verified no production creator calls client.rest.issues.create directly outside this boundary (only work-queue test-check fixtures do).
  • ✅ Explicit opt-in metadata parsing: parseReportMarkdown's allowMetadata flag cleanly separates trusted footer/import parsing from diagnostic-fragment parsing, preventing diagnostic text from forging XML/JSON markers — a solid answer to the injection risk this refactor set out to close.
  • ✅ Consistent vocabulary: new AST node names (xml_marker, json_marker, closing_reference, structured_data) are used uniformly across report_ast.cjs, report_markdown.cjs, and report_footer.cjs, and are documented clearly in the updated README — satisfies /grill-with-docs concerns about new concepts needing shared vocabulary.
  • ✅ Strong test coverage: report_ast.test.cjs and issue_report.test.cjs cover normalization idempotency, escaping/redaction, structured-data edge cases (getters, sparse arrays, cycles), and the publication-boundary guarantee.

Scope note

The pre-fetched diff (3000 lines) truncated partway through report_markdown.cjs; I read the full file directly from disk to confirm behavior, so no gaps in this review.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 95.5 AIC · ⌖ 17.1 AIC · ⊞ 10.1K
Comment /matt to run again

Comment thread actions/setup/js/handle_agent_failure.cjs Outdated
Reuse shared report title and text normalization directly instead of constructing temporary report nodes. Preserve literal ADF behavior and add staged/live normalization, footer, and provider-limit regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the Test Quality Sentinel result: reported_no_action contains no requested test change. Validation relies on the tests actually executed for this branch, not on this status as proof of completion or publication.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the PR Code Quality Reviewer notice: this is an in-progress notification without a substantive code-quality finding, so no code change is needed. It is not being treated as review completion or approval.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the generated previews: all three examples already show the H3 report sections, H4 diagnostics, and retained metadata. This informational preview requests no further code change.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the Matt Pocock Skills Reviewer notice: this is an in-progress notification without an actionable finding, so no code change is needed. It is not being treated as a completed review or approval.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the Ponytail Reviewer completion notice: the substantive Jira simplification is addressed in its inline review thread. This completion notice itself requests no additional change.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to the Design Decision Gate result: the notification contains no ADR draft or requested design change, so no change is needed. reported_no_action alone is not evidence that every gate requirement was satisfied or publication succeeded.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

No merge-blocking issue in the changed lines stood up to static review.

Notes

I reviewed the prefetched diff plus the key new report AST and markdown importer paths. I did not run code or tests in this review pass, and I did not find a changed-line correctness or security regression with enough confidence to block merge.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 384.4 AIC · ⌖ 5.74 AIC · ⊞ 12.7K
Comment /review to run again

Represent imported Markdown soft breaks explicitly while keeping scalar metadata normalized. Restore the original multiline issue-body assertion and cover deterministic rendering and scalar injection boundaries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved security and content-compatibility regressions remain in the shared publication path.

11 open issues
What changed in this PR

Centralizes gh-aw issue publication around a structured report model and makes workflow failures easier to act on.

Changes:

  • Adds report nodes, normalization, Markdown importing, and rendering.
  • Restructures failure reports with consistent titles, diagnostics, and footers.
  • Migrates issue publishers and adds compatibility coverage.
File Description
actions/​setup/​README.md Documents structured reporting.
actions/​setup/​md/​agent_failure_issue.md Reorganizes failure-report sections.
actions/​setup/​js/​work_queue_issues_checks.cjs Updates creation fixtures.
actions/​setup/​js/​work_queue_issues_acceptance_checks.cjs Updates retry fixtures.
actions/​setup/​js/​work_queue_issue_api.cjs Normalizes GraphQL issue creation.
actions/​setup/​js/​sanitize_content.cjs Exports mention-alias expansion.
actions/​setup/​js/​run_validate_workflows.cjs Uses shared issue publication.
actions/​setup/​js/​run_operation_update_upgrade.cjs Migrates upgrade notifications.
actions/​setup/​js/​run_activity_report.cjs Migrates activity reports.
actions/​setup/​js/​report_markdown.cjs Imports supported Markdown into nodes.
actions/​setup/​js/​report_footer.test.cjs Tests footer compatibility.
actions/​setup/​js/​report_footer.cjs Builds structured failure footers.
actions/​setup/​js/​report_failed_jobs.test.cjs Updates fence assertions.
actions/​setup/​js/​report_failed_jobs.cjs Migrates failed-job reports.
actions/​setup/​js/​report_ast.test.cjs Tests normalization and rendering.
actions/​setup/​js/​report_ast.cjs Defines and renders report nodes.
actions/​setup/​js/​push_to_pull_request_branch.cjs Migrates fallback issues.
actions/​setup/​js/​missing_issue_helpers.cjs Migrates missing-information issues.
actions/​setup/​js/​messages_footer.cjs Exposes footer templates and metadata.
actions/​setup/​js/​linear_safe_outputs.test.cjs Tests normalized Linear content.
actions/​setup/​js/​linear_create_issue.cjs Renders structured Linear descriptions.
actions/​setup/​js/​jira_handlers.test.cjs Tests normalized Jira content.
actions/​setup/​js/​jira_handlers.cjs Normalizes literal Jira fields.
actions/​setup/​js/​issue_report.test.cjs Tests publication and migration coverage.
actions/​setup/​js/​issue_report.cjs Adds shared publication boundary.
actions/​setup/​js/​index.test.cjs Tests staged/live output parity.
actions/​setup/​js/​handle_noop_message.cjs Migrates tracking-issue creation.
actions/​setup/​js/​handle_detection_runs.cjs Migrates detection tracking issues.
actions/​setup/​js/​handle_create_pr_error.cjs Migrates error-report issues.
actions/​setup/​js/​handle_agent_failure.test.cjs Updates failure-report expectations.
actions/​setup/​js/​handle_agent_failure.cjs Builds structured failure reports.
actions/​setup/​js/​failure_report.test.cjs Tests report layout and escaping.
actions/​setup/​js/​failure_report.cjs Builds reports from failure templates.
actions/​setup/​js/​create_steering_issue.cjs Migrates steering issues.
actions/​setup/​js/​create_pull_request.test.cjs Updates rendered-body assertions.
actions/​setup/​js/​create_pull_request.cjs Migrates PR fallback issues.
actions/​setup/​js/​create_issue.test.cjs Tests rendering and mention compatibility.
actions/​setup/​js/​create_issue.cjs Integrates structured issue publication.
actions/​setup/​js/​create_forecast_issue.cjs Migrates forecast issues.
actions/​setup/​js/​check_workflow_recompile_needed.cjs Migrates recompile notifications.
actions/​setup/​js/​check_version_updates.cjs Migrates version-block notifications.
.github/​scripts/​test_aw_issue_clustering_publish.cjs Updates clustering publication coverage.

Review effort: Max

Comment thread actions/setup/js/report_ast.cjs Outdated
Comment thread actions/setup/js/report_ast.cjs Outdated
Comment thread actions/setup/js/report_markdown.cjs
Comment thread actions/setup/js/jira_handlers.cjs Outdated
Comment thread actions/setup/js/report_ast.cjs
Comment thread actions/setup/js/report_ast.cjs Outdated
Comment thread actions/setup/js/report_footer.cjs
Comment thread actions/setup/js/report_markdown.cjs Outdated
Comment thread actions/setup/js/report_markdown.cjs Outdated
Comment thread actions/setup/js/report_markdown.cjs Outdated
Canonicalize content before URL policy and redact complete fragments before tokenization. Preserve JSON numbers, multilingual Jira text, hard breaks, workflow identifiers, images, complete URLs, and custom footer placeholders. Remove the obsolete failure-marker helper and add regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan merged commit 40b678c into main Oct 17, 2026
100 checks passed
@pelikhan
pelikhan deleted the pelikhan-failure-issue-presentation branch October 17, 2026 19:19
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.92.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants