Skip to content

fix(cli): make recursive scans fail closed - #576

Open
chrisknvidia wants to merge 5 commits into
NVIDIA:mainfrom
chrisknvidia:fix/christopherk/recursive-fail-closed-reporting
Open

chrisknvidia wants to merge 5 commits into
NVIDIA:mainfrom
chrisknvidia:fix/christopherk/recursive-fail-closed-reporting

Conversation

@chrisknvidia

Copy link
Copy Markdown
Contributor

Summary

  • bound analyzer postprocessing by shared runtime and output budgets
  • preserve sanitized root and child failures through recursive ledger/report caps
  • keep traversal exhaustion separate from execution failure and count failed target attempts against the shared target budget
  • produce deterministic incomplete JSON, SARIF, Markdown, and terminal output without contaminating machine-readable stdout

Review context

This is the independent scan-budget, completeness, CLI, and reporting split requested in the review of #497. It intentionally excludes shell-truthiness logic, TM1 window reconciliation, and Python source classification.

Validation

  • exact review regressions verified red then green for failed-attempt budgeting, root/child fatal retention, and post-scan serialization accounting
  • focused CLI, runner, ledger, report, batch-report, and SARIF suite: 500 passed
  • full standard suite on the final head: 5,495 passed, 14 skipped, 39 deselected, 4 expected xfails
  • real recursive five-skill scans passed for JSON, SARIF, Markdown, and terminal; machine-readable stdout remained parseable
  • Ruff lint/format and diff checks passed

Split from #497 for focused architectural and security review.

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 10362c948e69ffd8d8df225e7921bb373b04cf93. Requesting changes because ledger caps can still discard a distinct fatal fact.

With two completed events plus distinct READ_ERROR and ANALYZER_RUNTIME_ERROR failures under a three-record cap, the input has two distinct failure work IDs, but the retained/public ledger is only READ_ERROR, one completed event, and OUTPUT_LIMIT. Completeness stays fail-closed, yet analyzer_runtime_error and its path disappear even though the cap can hold both failures plus the sentinel.

The root and child paths snapshot only next(...) failed event before truncation (cli.py around lines 1203 and 1841), so _ensure_required_failure_event never receives the other fatal facts. This contradicts the PR failure-retention contract. Please collect and prioritize every distinct pre-cap FAILED event and add a multi-failure cap regression.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed exact head 10362c948e69ffd8d8df225e7921bb373b04cf93 and the full recursive-scan/reporting diff.

The existing blocking review remains valid: root and child paths snapshot only the first FAILED event before bounded merging, so a cap that can retain two distinct failures plus its sentinel may nevertheless discard one fatal reason and path. Completeness remains fail-closed, but the public failure-retention contract is violated. Please collect and prioritize every distinct pre-cap FAILED work item and add the multi-failure cap regression described in the existing review.

No duplicate inline comment was added.

…rsive-fail-closed-reporting

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
@chrisknvidia

chrisknvidia commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

@rng1995 I addressed the ledger-retention review finding in fb86cf7: recursive scans now retain each distinct fatal root or child failure before applying the output cap, with tight-cap regression coverage. PTAL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants