Skip to content

encryption.md

Latest commit

 

History

History
145 lines (119 loc) · 6.62 KB

encryption.md

File metadata and controls

145 lines (119 loc) · 6.62 KB
Raw

Encryption (BYOK) — reference

Envelope encryption for data at rest with per-tenant, customer-controlled keys (Step 6.7). 6.7a ships the library + a local KMS; cloud KMS providers + storage wiring are 6.7b.

Overview

  • KeyManager (rag_core.spi.key_manager) — SPI over opaque bytes: encrypt(ctx, plaintext) -> bytes, decrypt(ctx, ciphertext) -> bytes, health(). The ciphertext is a self-describing envelope; treat it as opaque.
  • NoopKeyManager (rag_core.spi.noop) — passthrough, no encryption (test/dev wiring only).
  • EnvelopeKeyManager (rag_backends.kms) — client-side envelope base: a fresh AES-256-GCM DEK per payload (with ctx.tenant_id as AAD); subclasses implement _wrap_dek / _unwrap_dek.
  • LocalKeyManager (rag_backends.kms) — wraps the DEK with an in-process per-tenant KEK (dev / tests / air-gapped). Cloud providers (AWS/GCP/Azure/Vault) are 6.7b.
  • EncryptingStorage (rag_core.encrypting_storage) — a Storage decorator that encrypts on put / decrypts on get (crypto-free; uses the KeyManager).
  • Errors (rag_core.errors): EncryptionError (malformed / tampered / wrong tenant) and KeyUnavailableError (sealing — the KEK is revoked / unreachable).

Usage

import os
from rag_backends import LocalKeyManager
from rag_core.encrypting_storage import EncryptingStorage
from rag_core.spi.noop import NoopStorage

km = LocalKeyManager(keks={"acme": os.urandom(32)})          # per-tenant KEKs
ciphertext = await km.encrypt(ctx, b"secret chunk text")     # ctx.tenant_id = "acme"
assert await km.decrypt(ctx, ciphertext) == b"secret chunk text"

# Encrypt blobs at rest behind any Storage backend:
store = EncryptingStorage(NoopStorage(), km)
await store.put(ctx, "doc/1", b"secret")   # underlying store holds ciphertext
await store.get(ctx, "doc/1")              # → b"secret"

Guarantees

Property How
Confidentiality AES-256-GCM with a per-payload DEK; ciphertext ≠ plaintext
Tamper-evidence GCM auth tag → EncryptionError on any bit flip
Per-tenant isolation KEK per tenant + tenant_id bound as AAD (a blob can't be read under another tenant, even with a shared KEK)
Sealing No/revoked KEK for a tenant → KeyUnavailableError; that tenant's data is unreadable, others unaffected

CLI

ragctl kms --tenant acme --text "secret"   # round-trip + isolation + sealing demo

Configuration + providers (Step 6.7b)

Select a provider from rag.yaml; the gateway builds it (build_key_manager_from_config) and exposes it on app.state.key_manager.

kms:
  enabled: true
  provider: aws            # noop | local | aws  (gcp / azure / vault → 6.7c)
  default_key_id: ""       # fallback key reference for tenants without their own
  region: us-east-1        # AWS
  local_key: ${LOCAL_KEK}  # hex 32-byte default KEK for provider=local
tenants:
  - id: acme
    name: Acme
    kms_key_id: arn:aws:kms:us-east-1:111122223333:key/abcd…   # aws CMK
  - id: beta
    name: Beta
    kms_key_id: ${BETA_KEK_HEX}   # provider=local → hex 32-byte KEK
provider Class Extra Per-tenant kms_key_id
local LocalKeyManager — hex 32-byte KEK
aws AwsKmsKeyManager — (aioboto3 base dep) KMS key ARN
gcp GcpKmsKeyManager [kms-gcp] CryptoKey resource name
azure AzureKeyVaultKeyManager [kms-azure] Key Vault key URL
vault VaultKeyManager [kms-vault] Transit key name

Each cloud provider subclasses EnvelopeKeyManager and only wraps/unwraps the DEK via its KMS API (AWS Encrypt/Decrypt; GCP encrypt/decrypt; Azure wrap_key/unwrap_key; Vault Transit encrypt_data/decrypt_data). Connection uses each SDK's standard credential discovery (AWS chain, GCP ADC, Azure DefaultAzureCredential, Vault VAULT_ADDR/VAULT_TOKEN); kms.vault_mount sets the Vault Transit mount. Any KMS failure (revoked / denied / unreachable) or a missing key → KeyUnavailableError (sealing). A tenant with neither its own kms_key_id nor default_key_id is sealed. The SDKs are lazy-imported, so selecting a provider without its extra raises a clear ImportError.

Key rotation (Step 6.7d)

RotatingKeyManager (rag_core.rotating_key_manager) wraps a current KeyManager plus retired keys for zero-downtime KEK rotation:

from datetime import UTC, datetime, timedelta
from rag_core.rotating_key_manager import RetiredKey, RotatingKeyManager

km = RotatingKeyManager(
    current=new_key_manager,
    retired=[RetiredKey(old_key_manager, expires_at=datetime.now(UTC) + timedelta(days=30))],
)
new_ct = await km.encrypt(ctx, data)          # always the current key
plain = await km.decrypt(ctx, any_ct)          # current, then non-expired retired keys
migrated = await km.rewrap(ctx, old_ct)        # re-encrypt an old blob under current
  • encrypt uses the current key; decrypt tries the current key then each non-expired retired key. The try-all is safe (AES-GCM authenticates the DEK — a wrong KEK can't yield a valid key, so it never returns wrong plaintext).
  • An expired retired key is skipped, so old data not yet re-wrapped is sealed (raises EncryptionError) — "retain decrypt-only until expiry".
  • rewrap is the background-migration primitive: decrypt with whatever key still works, re-encrypt under the current key; once all blobs are migrated the retired key can be dropped. ragctl kms --rotate demos the full flow.

It is crypto-free (it orchestrates the KeyManager SPI), so it composes with the local + every cloud provider. Config-driven rotation + the storage-side re-encryption job land with the EncryptingStorage ingest wiring.

Internals

  • Envelope wire format: b"RAGK" | version | u32(len(wrapped_dek)) | wrapped_dek | data_nonce(12) | ciphertext — opaque to callers, parsed only by EnvelopeKeyManager.
  • Vectors stay plaintext. Encryption targets chunk content / blobs at rest, not embedding vectors (ANN search needs plaintext vectors). See ADR-0039.
  • cryptography provides AES-GCM (in rag-backends); rag-core stays crypto-free (SPI + decorator + noop only).

Extension points

Add a cloud KMS provider (6.7b): subclass EnvelopeKeyManager and implement _wrap_dek / _unwrap_dek by calling the KMS Encrypt/Decrypt API for the tenant's key (raise KeyUnavailableError when the key is denied / unreachable). The DEK generation, AES-GCM, AAD, and envelope framing are inherited — a provider is just the KEK wrap/unwrap.