Security: unauthenticated pickle of remote results (RCE) and disabled SSH host key verification #121

Part of #108 · Phase 3 · label: security · The most serious defect found in the audit.

Note: this is not what issue #107 reported. #107 was a false-positive scanner alert on documentation placeholders and has been closed. The issues below have no scanner signature — they are design decisions, which is why nothing flagged them.

1. Unauthenticated deserialization of remote data (remote -> local RCE)

Clustrix SFTP-downloads result.pkl / error.pkl from the remote job directory and pickle.loads it. Sites:

utils.py:175,179,181,183,184 · executor_core.py:172 · executor_cloud.py:358,387 · executor_kubernetes.py:169,178,179,184 · executor_scheduler_status.py:533,634

Implied trust model: the remote cluster is fully trusted. On a shared HPC filesystem, any user who can write to that job directory gets arbitrary code execution on the submitting user's laptop. Shared scratch directories on university clusters are frequently group-writable, so this is not hypothetical.

• Document the trust boundary explicitly and prominently — users must know that submitting a job grants the remote host code execution on their client
• Authenticate the payload: HMAC result.pkl with a per-job key generated client-side and passed to the job
• Reject unauthenticated payloads by default; any opt-out is explicit and loud
• Consider a restricted unpickler as defense in depth
• A test that a tampered result.pkl is rejected, not executed

2. SSH host key verification is disabled — 14 call sites

paramiko.AutoAddPolicy() is used unconditionally at:

executor_connections.py:38 · ssh_utils.py:101,146,348 · filesystem.py:240 · notebook_magic_ssh.py:159,200,446 · cli_credentials.py:391 · executor_cloud.py:294 · notebook_magic_widget.py:1513 · validation.py:29,84 · kubernetes/lambda_provisioner.py:393

Every password authentication is MITM-able. Chained with defect 1, an MITM also obtains local code execution — the attacker controls result.pkl.

Note the irony: ssh_utils.add_host_key is already exported from clustrix/__init__.py:12, so the machinery exists and is simply not used on the connection path.

• Default to RejectPolicy/WarningPolicy with a real known_hosts
• First-connection trust-on-first-use is an explicit, logged, opt-in decision
• AutoAddPolicy appears zero times, or exactly once behind a documented config flag defaulting to off

3. Silent failures mask security-relevant errors

40+ except Exception: -> pass/return None sites. Security-relevant examples:

• auth_methods.py:93-94 — except OSError: pass

• credential_manager.py:436,462,499,558,615 — swallowed auth failures

• executor_connections.py:73-75, auth_manager.py:223-225

• secure_credentials.py:144-147 — cred_dir.chmod(0o700) inside except Exception: pass, so a failed permission tightening is invisible

• config.py:206 — swallows all cloud-dependency install errors during __init__

• executor_kubernetes.py:304-308 — returns "completed" on any API exception

• Authentication and permission failures always surface; never pass

• A failed chmod on a credential directory is fatal, not ignored

4. Minor

• ssh_utils.py:393 — public key interpolated into a remote shell string with single-quote escaping only. Low impact (not a secret) but should be quoted properly.