Install ThreatCrush from npm; drop the PRD's non-existent hosts - #5
Merged
Conversation
The PRD's distribution URLs do not exist. Neither cli.threatcrush.com nor api.sh1pt.com resolves in public DNS, so every run so far took the failure path and scanned nothing. ThreatCrush actually ships as an npm package, @profullstack/threatcrush (bin: threatcrush), which is what https://threatcrush.com/install.sh installs after bootstrapping mise and Node. In CI that bootstrap is redundant, so the workflow now runs npm install -g directly and keeps the official installer as a fallback. Pinned to Node 20: the CLI depends on better-sqlite3, and 20 is the newest runtime with reliable prebuilds. On Node 24 the install falls through to a node-gyp source build, which is how this was diagnosed locally. Also adds a 'Record the CLI interface' step. The scan flags in this workflow came from the PRD and have never been checked against a real binary, so the run log now prints --help for both the CLI and its scan subcommand. Correct the scan step from that evidence rather than from the PRD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush scan results9 finding(s) from
Detection coverage — FAIL
Findings
Every file under Posted by |
Run 30686062988 installed the scanner successfully, then the scan died with
'error: unknown option --format' — and the job still went green. Two separate
faults, both fixed here.
1. The scan invocation was wrong. Confirmed from the published bundle:
.command("scan").argument("[path]", "Path to scan", ".")
No options exist. --format/--output/--config/--fail-on/--verbose are all
inventions of the PRD, and scan takes a PATH, not a pull request URL. That
settles PRD Open Question 1: there is no native PR-level scanning. Diff-only
scoping now happens in the workflow, which scans just the changed files
under vulns/ on a pull request.
2. The failsafe only checked whether the CLI installed, so a scan that ran and
failed still passed. It now requires that the scan produced usable SARIF.
A green check must mean the corpus was actually scanned.
Because the CLI cannot emit SARIF, scripts/threatcrush-to-sarif.py translates
its output. It fails closed: if the format is unrecognised it exits non-zero
and dumps the first 40 lines rather than emitting empty SARIF, since '0
findings' is indistinguishable from a clean scan and would be a silent false
negative on a repository that exists to contain findings. Handles JSON output
too, in case the CLI gains it.
Verified across six paths: text, JSON, clean-scan, unparseable, missing input,
and end-to-end scoring of converted SARIF through validate-coverage.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 30687082490 finally produced real scanner output: 9 findings (4 critical, 4 high, 1 medium), all secrets. Two defects showed up in how they were mapped. 1. Path mapping. The CLI reports paths relative to the directory it was given, so scanning 'vulns' yields 'secrets/aws-credentials-hardcoded.env'. Those never matched the catalog's repository-relative paths, so all 9 findings scored as 'outside corpus' and the true positive rate stayed at 0% despite the scan working. --path-prefix restores the prefix. 2. Output parsing. The output is a multi-line block per finding — a severity and title line, then 'File:', then 'Info:' — not one finding per line. Severity is bare for CRITICAL and bracketed for the rest. 'Code:' lines are skipped: they are redacted excerpts, and matching them double-counted every finding. Whole-file findings report line :0, which SARIF rejects; clamped. The scan step now always scans the whole corpus rather than a changed-file list, so the prefix cannot vary between invocations. The corpus is 31 files and scans in seconds, so diff-only scoping bought nothing but ambiguity. Real output is committed as tests/fixtures/threatcrush-scan-output.txt so the parser has a regression test against actual CLI behaviour rather than a guess. Measured result: 15.58% TPR (12/77), 0 false positives, 0 unattributed. That is a genuine reading — ThreatCrush is a secrets scanner, and it catches the credential fixtures while not attempting the code-level vulnerabilities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 1, 2026
The PRD's distribution hosts do not resolve and its scan flags do not exist. ThreatCrush ships as @profullstack/threatcrush, and scan takes a path and no options. Adds a text-to-SARIF converter that fails closed, and stops the job reporting success when the scan produced nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cli.threatcrush.comandapi.sh1pt.comdon't resolve — every run so far scanned nothing.ThreatCrush actually ships as
@profullstack/threatcrushon npm (bin:threatcrush), which is whatthreatcrush.com/install.shinstalls after bootstrapping mise + Node. In CI that bootstrap is redundant, so this runsnpm install -gdirectly, with the official installer as fallback.Pinned to Node 20 — the CLI needs
better-sqlite3, and 20 is the newest runtime with reliable prebuilds. On Node 24 it falls through to a node-gyp source build (that's how I found it).Also adds a Record the CLI interface step: the scan flags came from the PRD and were never verified against a real binary, so the log now prints
--helpfor the CLI and itsscansubcommand. Correct the scan step from that evidence.🤖 Generated with Claude Code