[pjwerneck]

Summary

Each client now advertises an encryption policy (off, optional or required) in its version
file, and a pair encrypts when either side requires it. Keys and bundles are set up automatically,
so a client of an enclave that requires encryption needs no setting. The default stays optional.

Changes

• login(encryption=...) takes True (required), False (optional) or "off". The policy is the
  new encryption_policy field of SYFT_version.json.
• An optional client makes keys only when a peer requires encryption. The key file records a
  required policy, so later logins and syft-bg stay required. A key file without the field reads
  as required.
• An off and a required client conflict: each side skips the other with a warning. A peer whose
  version file has no policy runs an outdated syft and is skipped with a warning to upgrade.
• A pair that encrypts refuses plaintext: the owner archives such a message unread, the scientist
  skips it, both with a warning.
• A send that waits for the peer's policy or keys stays in memory until the next sync, with a
  warning. A scientist's send to a non-peer raises; the owner drops an outbox message to a
  non-peer with a warning.
• Files at rest are encrypted when the client is required. Files written before a switch stay
  plaintext and remain readable.
• A sync raises when another process made the key file required after this one started.
• The scientist sends one message per datasite owner. Before, all queued changes went to the owner
  of the last path.
• The enclave makes keys at boot and binds them to the attestation, also when optional. Clients
  follow the enclave's policy, so the demo scripts drop their encryption flag.

Note

Release note: clients skip an enclave image built before this change as outdated. Rebuild and
deploy the enclave image with, or before, this client release.

Testing

• New tests/unit/test_encryption_policy.py (57 tests, mock Drive) and enclave tests for the
  negotiated job flow and boot-time keys.
• Full suites pass: tests/unit 626, migrations 108, syft-enclave 298, syft-rds 151,
  syft-job 121, syft-datasets 51.
• Live run on real Drive (two test accounts): job round trips with the DO required, with the DS
  required, and with both optional; an off/required conflict; and a policy change while a
  process runs. All five passed, with the raw Drive files checked as encrypted or plaintext.

Asana task

• https://app.asana.com/0/1210542925864934/1218358857493588

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[rasswanth-s]

Minor Question , Great work @pjwerneck

[koenvanderveen]

I split the encryption policy out of VersionInfo into a new versioned PeerInfo object (d0f8827), which nests the VersionInfo and the policy and is now what we publish as SYFT_version.json, so VersionInfo stays about the syft version only and current() needs no policy. A version file from an older syft still loads, as PeerInfo with an OPTIONAL policy, so the migration fixture tests keep checking older formats. Methods that read or write the published file are renamed after PeerInfo, e.g. write_own_version → write_own_peer_info and load_peer_version → load_peer_info.

[koenvanderveen]

Pushed 383e81f, which keeps one encryption policy per SyftBox:

• One policy per SyftBox. It lives in <syftbox>/<email>/private/syft_settings.json. The first client to start writes it: the encryption it logged in with, or optional. Later clients, the notebook or syft-bg, log in with encryption=None (now the default) and take the policy on disk. Asking for another policy raises EncryptionPolicyMismatchError, and so does a sync() after the file changed under a running client. PeerStore.encryption_policy stays a plain EncryptionPolicy: resolve_encryption_policy() settles it before the store is built.
• To change the policy, delete the SyftBox and log in again. Files the owner keeps on Drive for itself are encrypted under the policy in force when they were written, so the policy can't be switched in place.
• syft-bg no longer stores a policy. It's gone from SyftBgConfig, SyncConfig and AutoApproveConfig, and from both orchestrators. syft-bg init --encryption ... is optional and writes the SyftBox file, and the interactive init only asks on a new SyftBox.
• Enclave keys: the runner writes its keys back after the fresh_state wipe, because the attestation server signs with the key file. delete_syftbox() no longer does this for every client.
• Renames: connection_for_version_read() is now connection_for_peer_file_read(), the Drive file-id helpers are now _get_own_peer_info_file_id() / _get_peer_info_file_id_of(), and the docstrings and warnings say "peer info". The file is still SYFT_version.json.
• Docs and tests: docs/connections.md is updated, and there are new tests in test_encryption_policy.py, syft_bg/test_init.py and test_runner.py.

[koenvanderveen]

Pushed 8c21942, which simplifies how per-peer state is kept. These are the decisions behind it:

• State about a peer lives on its Peer, in the PeerStore. PeerManager no longer keeps per-peer side dicts. _loaded_peer_versions, _peer_protocol_schemas, _states_at_start and _last_known_policies repeated what Peer.info already holds, or were a flag per peer.
• One rule for encryption: a pair encrypts when one side requires it. That rule is PeerStore.pair_encrypts(peer), a plain yes/no in which a peer with an unknown policy counts as not requiring it. The check that raises is kept apart, in PeerStore._peer_for_message(email), and only sending and reading call it. This replaces pair_uses_encryption, peer_policy, policy_of, _raise_unless_policy_known and peer_uses_encryption.
• A client that requires encryption doesn't wait for the peer's policy. The pair encrypts either way. It still waits for the peer's public key.
• use_encryption is renamed to requires_encryption, which says what it means.
• Peer schemas are looked up, not copied. PeerStore.protocol_schema(email, protocol_name) reads them from the peer's info. JobStorage and DatasetStorage take a peer_schema lookup function instead of a shared dict that has to be updated in place, and the router asks the peer store directly. Tests that stand in for a peer on an older release use the advertised_schemas fixture.
• A peer keeps its last info. load_peers() still replaces the peer list, but set_peers() keeps each peer's info and state_at_start, and a read that finds no info file keeps the last one. This keeps a peer that deleted its SyftBox readable, so a DS still applies the delete messages it sent.
• The warning about leftover connections stays. It is now Peer.state_at_start, which is never saved, instead of a dict on PeerManager.