Security hardening: rotate local-only HF tokens, enable secret scanning, fix credential file permissions

Part of #108 · Phase 0 · label: security

Bundles the concrete, low-effort security fixes surfaced by the 2026-08-17 audit. The deeper architectural security work (pickle trust model, host-key verification) is tracked separately — see the Phase 3 security issue.

1. Rotate two real HuggingFace tokens (local-only exposure)

Two genuine HF tokens (hf_Fbf..., hf_hSV...) exist in local git objects:

Event	Commit	Date
Introduced in `notes/huggingface_validation_fix_2025-06-29.md`	`d30acd2`	2025-06-29 10:02
Removed	`9c3cdfc`	2025-06-29 10:08

They never reached GitHub — verified three ways: git merge-base --is-ancestor d30acd2 origin/master -> false; gh api repos/ContextLab/clustrix/commits/d30acd2 -> 422 No commit found for SHA; the rewritten master at f0278e8 shows hf_XXXX... redaction. The only ref retaining them is refs/original/refs/heads/master, a local git filter-branch backup.

Still rotate — they sat in a working tree for ~8 months.

Revoke both at https://huggingface.co/settings/tokens
Purge the local backup ref:
git update-ref -d refs/original/refs/heads/master && git reflog expire --expire=now --all && git gc --prune=now

2. Enable GitHub secret scanning + push protection

Currently disabled on this public repo: gh api repos/ContextLab/clustrix/secret-scanning/alerts -> 404 Secret scanning is disabled. This is exactly the control that would have blocked the 2025-06-29 commit at push time.

Settings -> Code security -> enable secret scanning and push protection

3. Stop generating scanner bait

Issue #107 was a false positive triggered by placeholders in the .env template:
credential_manager.py:355 (AKIAIOSFODNN7EXAMPLE, AWS's own doc placeholder), :356, :393 (hf_abcdefghij...).

Replace with non-matching forms (AKIA_YOUR_KEY_HERE, hf_YOUR_TOKEN_HERE) so scanners stop firing on this file

4. `.gitignore` does not ignore a bare `.env`

Only .env.local (line 61) and .env.validation (line 62) are covered. git check-ignore .env returns nothing. Mitigated in practice because the manager defaults to ~/.clustrix/.env, but a repo-root .env is currently committable.

Add .env and .env.* (with !.env.example if wanted)

5. Credentials written world-readable

clustrix/config.py:216-225 (save_to_file) and :308-323 (save_config) do asdict(self) -> yaml.dump/json.dump with no chmod. ClusterConfig.password (config.py:16) is a plain field, so an SSH password lands at default umask (0644).

Exclude password (and other secret fields) from serialization, or write to a 0600 temp file and os.replace into position
Test asserting file mode is 0600 and that no secret appears in the serialized output

6. GCP service-account JSON leaked to /tmp

clustrix/cli_credentials.py:502-506 writes the key to NamedTemporaryFile(delete=False), exports the path as GOOGLE_APPLICATION_CREDENTIALS, and never unlinks it.

try/finally or atexit cleanup; create the file 0600

7. Write-then-chmod TOCTOU

cli_credentials.py:634-639 and credential_manager.py:338-339 write secrets at default umask before calling chmod(0o600). secure_credentials.py:144-147 wraps cred_dir.chmod(0o700) in except Exception: pass, so a failed permission change is silent.

Create with restrictive mode via os.open(..., 0o600); never chmod after the fact
Remove the silent swallow

Explicitly out of scope / verified clean

No credential is ever logged or printed (auth_fallbacks.py:137,157, auth_manager.py:70,76 log variable names only). No sshpass, no op, no --password on argv, no echo <pw> | — nothing reaches a command line or ps. No shell=True in clustrix/. No verify=False anywhere. clustrix.yml, which contains a real host and NetID, is correctly gitignored and untracked.

Issue #111: Ignore .ccpm_backup/ - it carried a live PyPI token

Status update 2026-08-17. HF tokens confirmed rotated by @jeremyrmanning — thank you. But this issue is not complete: a second, previously-undetected credential turned up, and 6 of the original 7 items remain.

NEW: a live PyPI API token (please revoke)

GitHub push protection blocked the #112 branch push:

—— PyPI API Token ——
  commit: 6c918ca
  path: .ccpm_backup/.claude_20250828_140214/settings.local.json:30

Question	Answer
Ever public?	No — scanned every commit reachable from `origin/master`; absent from all
Where	`.ccpm_backup/`, an accidental 245-file / 1.2 MB backup of local `.claude/` settings
Exposure	11 local commits from `6c918ca` (2025-09-04) onward, plus on disk
Status	Blob stripped from those commits; `.ccpm_backup/` now gitignored (PR #129)

Revoke the PyPI token at https://pypi.org/manage/account/token/ — it sat in a commit and on disk for ~11 months

False alarm for the record: public commit d6b4b7b contains clustrix-pypi-test-v… / clustrix-pypi-valida…, which are 1Password item names, not tokens.

This is a gap in the original audit. The security track scanned the working tree and origin/master history, but not the unpushed commits, and PyPI tokens were not in its pattern set. Push protection caught what the audit missed — which is precisely the argument for item 2 below.

Remaining checklist

1. Rotate the two HF tokens — done
1b. Purge the local backup ref. Note refs/original/refs/heads/master now points at 4feda53 (the pre-rewrite state from the Reconcile 30 unpushed local commits; quality_gates.yml has never run #112 fix), having overwritten the older filter-branch backup. The pre-rewrite .git is also archived outside the repo. Purge when you are satisfied with PR Issue #112: Surface the test-coverage epic work for review (do not merge as-is) #128/Issue #109: Stop the test suite from provisioning billable AWS resources #129.
2. Enable secret scanning + push protection — still 404 Secret scanning is disabled. Interestingly, push protection is already active for some token types (it fired above), but the repo-level feature is off, so you cannot view or manage detected secrets. https://github.com/ContextLab/clustrix/settings/security_analysis
3. Scanner-bait placeholders still present verbatim: credential_manager.py:355 (AKIAIOSFODNN7EXAMPLE) and :393 (hf_abcdefghij…). These caused false-positive [Security Alert] Exposed API key(s) detected: AWS Access Key, HuggingFace Token #107.
4. .env still not gitignored — git check-ignore .env returns nothing
5. config.py:216-225,308-323 still serialize ClusterConfig.password with no chmod (world-readable at default umask)
6. cli_credentials.py:502-506 still leaves the GCP service-account JSON in /tmp unlinked
7. Write-then-chmod TOCTOU at cli_credentials.py:634-639, credential_manager.py:338-339; silent chmod failure at secure_credentials.py:144-147

So: 2 of 9 done. Items 2, 3 and 4 are quick and worth doing together.

Fix three fabricated-result bugs, close the RCE, and overhaul the doc…

Issue #111: gitignore .env and its siblings

Issue #111: create credential files at 0600 instead of narrowing them…

Issue #111/#116: fix the unguarded write-then-narrow, and invert both…

One secure writer, and config.py had drifted away from it

Issue #111: close the profile-store leak, make the write atomic, repl…

Issue #111: a widget save, and a credential lookup, stop leaking

Issue #111: scan pushable history for secrets, not just the working tree

Navigation Menu

Repository navigation

Security hardening: rotate local-only HF tokens, enable secret scanning, fix credential file permissions #111

1. Rotate two real HuggingFace tokens (local-only exposure)

2. Enable GitHub secret scanning + push protection

3. Stop generating scanner bait

4. `.gitignore` does not ignore a bare `.env`

5. Credentials written world-readable

6. GCP service-account JSON leaked to /tmp

7. Write-then-chmod TOCTOU

Explicitly out of scope / verified clean

NEW: a live PyPI API token (please revoke)

Remaining checklist

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Security hardening: rotate local-only HF tokens, enable secret scanning, fix credential file permissions #111

Description

1. Rotate two real HuggingFace tokens (local-only exposure)

2. Enable GitHub secret scanning + push protection

3. Stop generating scanner bait

4. .gitignore does not ignore a bare .env

5. Credentials written world-readable

6. GCP service-account JSON leaked to /tmp

7. Write-then-chmod TOCTOU

Explicitly out of scope / verified clean

Activity

jeremymanning commented on Aug 17, 2026

NEW: a live PyPI API token (please revoke)

Remaining checklist

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

4. `.gitignore` does not ignore a bare `.env`