Install ThreatCrush from npm; drop the PRD's non-existent hosts

.github/workflows/threatcrush-scan.yml

@@ -32,6 +32,7 @@ concurrency:
   cancel-in-progress: true
 
 env:
+  THREATCRUSH_PACKAGE: '@profullstack/threatcrush'
   THREATCRUSH_VERSION: latest
   SARIF_FILE: threatcrush-results.sarif
 
@@ -59,6 +60,14 @@ jobs:
         with:
           python-version: '3.11'
 
+      # Node 20, not latest. The CLI depends on better-sqlite3, a native module;
+      # 20 is the newest runtime with reliable prebuilt binaries, so the install
+      # does not fall back to a node-gyp source build.
+      - name: Set up Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+
       - name: Validate test-case submissions
         id: lint
         run: bash scripts/validate-test-case.sh
@@ -69,46 +78,74 @@ jobs:
           set -uo pipefail
           echo "scanner_available=false" >> "$GITHUB_OUTPUT"
 
-          # Preferred source: the sh1pt artifact registry (PRD §10.1).
-          if [ -n "${SH1PT_TOKEN:-}" ] && curl -fsSL --max-time 60 \
-                -H "Authorization: Bearer ${SH1PT_TOKEN}" \
-                "https://api.sh1pt.com/v1/scanners/threatcrush/${THREATCRUSH_VERSION}" \
-                -o threatcrush-cli.tar.gz; then
-            tar -xzf threatcrush-cli.tar.gz
-            echo "$PWD" >> "$GITHUB_PATH"
+          # ThreatCrush ships as an npm package. https://threatcrush.com/install.sh
+          # bootstraps mise + node before running the same `npm i -g`, which is
+          # redundant here because setup-node already provided a runtime.
+          if npm install -g "${THREATCRUSH_PACKAGE}@${THREATCRUSH_VERSION}"; then
             echo "scanner_available=true" >> "$GITHUB_OUTPUT"
-            echo "installed ThreatCrush from sh1pt"
+            echo "installed ${THREATCRUSH_PACKAGE}@${THREATCRUSH_VERSION} from npm"
 
-          # Fallback: the public install script (PRD §7.1).
-          elif curl -fsSL --max-time 60 https://cli.threatcrush.com/install.sh -o install.sh; then
-            bash install.sh
+          # Fallback: the official installer, in case the package layout changes.
+          elif curl -fsSL --max-time 120 https://threatcrush.com/install.sh | sh; then
             echo "scanner_available=true" >> "$GITHUB_OUTPUT"
-            echo "installed ThreatCrush from cli.threatcrush.com"
+            echo "installed ThreatCrush via threatcrush.com/install.sh"
 
           else
-            echo "::warning title=ThreatCrush unavailable::Could not fetch the CLI from \
-            api.sh1pt.com or cli.threatcrush.com. The scan step will be skipped and this \
-            run will report zero findings. This is an infrastructure problem, not a \
-            detection result."
+            echo "::warning title=ThreatCrush unavailable::Could not install \
+            ${THREATCRUSH_PACKAGE} from npm or threatcrush.com/install.sh. The scan step \
+            will be skipped and this run will report zero findings. This is an \
+            infrastructure problem, not a detection result."
           fi
 
           command -v threatcrush >/dev/null 2>&1 && threatcrush --version || true
-        env:
-          SH1PT_TOKEN: ${{ secrets.SH1PT_TOKEN }}
 
+      # The PRD assumed this CLI's flags; they have never been verified against a
+      # real binary. Print the interface so the log is the source of truth, and
+      # the scan step below can be corrected from evidence rather than guesswork.
+      - name: Record the CLI interface
+        if: steps.install.outputs.scanner_available == 'true'
+        continue-on-error: true
+        run: |
+          echo "::group::threatcrush --help"; threatcrush --help || true; echo "::endgroup::"
+          echo "::group::threatcrush scan --help"; threatcrush scan --help || true; echo "::endgroup::"
+
+      # The real interface, confirmed from the CLI bundle:
+      #
+      #   .command("scan").argument("[path]", "Path to scan", ".")
+      #
+      # No options at all. The PRD's --format/--output/--config/--fail-on do not
+      # exist, and it scans a PATH, not a pull request URL — which answers PRD
+      # Open Question 1: no, there is no native PR-level scanning.
+      #
+      # Diff-only scoping is therefore done here rather than by the scanner: on a
+      # pull request, scan just the changed files.
       - name: Run ThreatCrush Scan
         id: scan
         if: steps.install.outputs.scanner_available == 'true'
         run: |
-          PR_URL="${{ github.event.pull_request.html_url }}"
-          echo "Scanning: ${PR_URL:-$GITHUB_REF}"
-
-          threatcrush scan "${PR_URL:-.}" \
-            --format sarif \
-            --output "$SARIF_FILE" \
-            --config .threatcrush.yml \
-            --fail-on critical,high \
-            --verbose
+          set -uo pipefail
+
+          # Always scan the whole corpus, never a per-file list. The CLI reports
+          # paths relative to the directory it was given, so a fixed root keeps
+          # --path-prefix correct; a changed-files list would make the prefix
+          # vary per invocation. The corpus is 31 files and scans in seconds, so
+          # diff-only scoping buys nothing here.
+          SCAN_ROOT=vulns
+          echo "Scanning: $SCAN_ROOT"
+
+          threatcrush scan "$SCAN_ROOT" > threatcrush-output.txt 2>&1
+          echo "scan_exit=$?" >> "$GITHUB_OUTPUT"
+          echo "::group::raw scanner output"; cat threatcrush-output.txt; echo "::endgroup::"
+
+          # The CLI cannot emit SARIF, so its text output is converted here.
+          if python3 scripts/threatcrush-to-sarif.py \
+               --input threatcrush-output.txt \
+               --output "$SARIF_FILE" \
+               --path-prefix "$SCAN_ROOT"; then
+            echo "sarif_produced=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "sarif_produced=false" >> "$GITHUB_OUTPUT"
+          fi
         continue-on-error: true
         env:
           THREATCRUSH_API_KEY: ${{ secrets.THREATCRUSH_API_KEY }}
@@ -178,16 +215,32 @@ jobs:
           name: threatcrush-scan-${{ github.event.pull_request.number || github.run_id }}
           path: |
             ${{ env.SARIF_FILE }}
+            threatcrush-output.txt
             coverage-report.json
             coverage-report.md
             scan-summary.md
           if-no-files-found: warn
           retention-days: 30
 
-      - name: Fail the job when the scanner could not run
-        if: always() && steps.install.outputs.scanner_available != 'true'
+      # A green check must mean "the corpus was actually scanned". Run 30686062988
+      # went green while scanning nothing: the install succeeded, the scan then
+      # died on an unknown flag, and this gate only looked at the install. It now
+      # checks that the scan ran AND produced real SARIF.
+      - name: Fail the job when nothing was actually scanned
+        if: always()
         run: |
-          echo "::error title=Scan did not run::The ThreatCrush CLI could not be \
-          installed, so this PR was never actually scanned. Fix the scanner source \
-          before trusting a green check on this workflow."
-          exit 1
+          FAILED=0
+
+          if [ "${{ steps.install.outputs.scanner_available }}" != "true" ]; then
+            echo "::error title=Scanner not installed::The ThreatCrush CLI could not be \
+            installed, so nothing was scanned."
+            FAILED=1
+          elif [ "${{ steps.scan.outputs.sarif_produced }}" != "true" ]; then
+            echo "::error title=Scan produced no results::The CLI installed but the scan \
+            did not yield usable SARIF. The findings below are empty because the scan \
+            failed, not because the corpus is clean. See the 'raw scanner output' group."
+            FAILED=1
+          fi
+
+          [ "$FAILED" = "1" ] && exit 1
+          echo "corpus was scanned and SARIF was produced"

docs/SCANNER_INTEGRATION.md

@@ -214,22 +214,52 @@ checks out untrusted code.
 
 ---
 
-## sh1pt.com artifact integration
+## Installing the scanner
 
-The workflow prefers the sh1pt artifact registry (PRD §10.1) and falls back to
-the public installer:
+ThreatCrush is distributed as an npm package, not a standalone binary:
+
+```
+@profullstack/threatcrush   →  bin: threatcrush
+```
+
+The workflow installs it directly:
 
 ```yaml
-curl -fsSL -H "Authorization: Bearer ${SH1PT_TOKEN}" \
-  "https://api.sh1pt.com/v1/scanners/threatcrush/${THREATCRUSH_VERSION}" \
-  -o threatcrush-cli.tar.gz
+npm install -g "@profullstack/threatcrush@latest"
 ```
 
-Set `SH1PT_TOKEN` and `THREATCRUSH_API_KEY` as repository secrets.
+and falls back to the official installer, `https://threatcrush.com/install.sh`,
+if the package layout ever changes. That script does the same `npm i -g` after
+bootstrapping mise and Node, which is redundant in CI where `setup-node` has
+already provided a runtime.
+
+**Node 20, deliberately.** The CLI depends on `better-sqlite3`, a native module.
+Node 20 is the newest runtime with reliable prebuilt binaries for it; on Node 24
+the install falls through to a `node-gyp` source build, which fails without a
+full toolchain. If you bump the Node version, verify the install still succeeds
+before trusting a run.
+
+Set `THREATCRUSH_API_KEY` as a repository secret if the scanner needs one.
+
+### The PRD's URLs do not exist
+
+The PRD (§7.1, §10.1) specified `cli.threatcrush.com/install.sh` and
+`api.sh1pt.com/v1/scanners/…`. Neither hostname resolves in public DNS — both
+were removed from the workflow on 2026-08-01 and replaced with the npm install
+above. `threatcrush.com` and `sh1pt.com` themselves do resolve; only those
+subdomains are absent.
+
+### The CLI's flags are unverified
+
+The scan invocation in the workflow —
+
+```
+threatcrush scan <target> --format sarif --output … --config … --fail-on critical,high
+```
 
-> **Status check.** As of 2026-08-01 neither `cli.threatcrush.com` nor
-> `api.sh1pt.com` resolves in public DNS (`threatcrush.com` and `sh1pt.com`
-> themselves do). Until those hosts exist, every run takes the fallback path:
-> the workflow writes an empty SARIF, reports "scanner did not run", and fails
-> the job at the final step so a green check never overstates what happened.
-> Confirm the real CLI distribution URL and update the install step.
+— comes from the PRD, and has never been checked against a real binary. The
+workflow therefore includes a **Record the CLI interface** step that prints
+`threatcrush --help` and `threatcrush scan --help` into the run log. Read that
+log and correct the scan step from the evidence before relying on any coverage
+number. If the flags differ, the scan step fails soft, so the run still reports
+"scanner did not run" rather than a misleading zero-findings result.