Python: [Bug]: SecretString cannot be pickled, so credential-bearing workflow checkpoints cannot be saved or restored #9514

Observed Behavior

Since #8127 made SecretString an immutable, slots-based, non-str wrapper, it can no longer be
pickled — and the workflow checkpoint codec pickles every non-JSON-native value. So a workflow whose
executor/session state holds a SecretString cannot be durably checkpointed:

• FileCheckpointStorage.save() is refused with WorkflowCheckpointException: Failed to decode pickled checkpoint data: SecretString is immutable. (the save-time restore check added in Python: FileCheckpointStorage save/load symmetry (#8181) #8214).
• CosmosCheckpointStorage / Foundry-hosting state store share the codec but don't validate at
  save time yet, so save() succeeds and load()/resume later fails (the asymmetry of Python: [Bug]: CosmosCheckpointStorage can save checkpoints that its load() cannot restore #8874/Python: Refuse to save Cosmos DB checkpoints that cannot be restored #8875).
• InMemoryCheckpointStorage is unaffected (it deepcopys; SecretString.__deepcopy__ returns
  self and never pickles) — so the problem is invisible in dev/test and only bites durable runs.

Mechanism: SecretString has __slots__ = ("_value",) and a mutation-rejecting __setattr__
(raise AttributeError("SecretString is immutable.")) but no __getstate__/__setstate__/
__reduce__. Pickle protocols 0–1 fail at dumps; protocols ≥2 succeed at dumps but fail at
loads, because the BUILD step calls setattr(obj, "_value", …) and hits the immutability guard.

The obvious workaround — unwrap the credential to a plain str so the save succeeds — writes the raw
secret into the checkpoint in cleartext (file mode 0o666 & ~umask).

Expected Behavior

A supported, documented outcome for a credential in durable workflow state — one of:
(a) SecretString round-trips through checkpoints (if persisting secrets is intended, accepting that
the secret is written to storage); (b) the checkpoint path rejects it early with a precise error,
as the session-store path already does (_validate_durable_state_value raises a clear TypeError
for a SecretString in durable session state); or (c) secrets are excluded from persisted state and
re-injected from configuration on restore. The current behavior (encode succeeds, then an opaque
pickle BUILD failure — or, on Cosmos, failure only at restore) is none of these.

Steps to Reproduce

1. Check out current main (ebcf4db7f9462ef6c377761e0b3159c4c5a73027).
2. Put a SecretString in checkpointed executor/session state and call FileCheckpointStorage.save().
3. The save is refused. (Replacing it with a plain str lets the save succeed and writes the secret
   to the checkpoint file in cleartext.)

Minimal Reproduction

# (1) Core defect — bare pickle, no framework:
import pickle
from agent_framework import SecretString
pickle.loads(pickle.dumps(SecretString("sk-example-123")))
# -> AttributeError: SecretString is immutable.

# (2) Framework impact — FileCheckpointStorage.save() is refused:
import asyncio, tempfile
from agent_framework import FileCheckpointStorage, WorkflowCheckpoint, SecretString
from agent_framework.exceptions import WorkflowCheckpointException

async def main():
    with tempfile.TemporaryDirectory() as d:
        storage = FileCheckpointStorage(d)
        cp = WorkflowCheckpoint(
            workflow_name="wf", graph_signature_hash="h",
            state={"_executor_state": {"a": {"api_key": SecretString("sk-example-123")}}},
        )
        try:
            await storage.save(cp); print("saved (unexpected)")
        except WorkflowCheckpointException as e:
            print(f"save refused: {e}")

asyncio.run(main())
# -> save refused: Failed to decode pickled checkpoint data: SecretString is immutable.

End-to-end (a real WorkflowBuilder graph): an Executor returning {"api_key": SecretString(...)}
from on_checkpoint_save() logs Failed to create checkpoint at iteration N: Failed to decode pickled checkpoint data: SecretString is immutable. at every superstep and persists zero
checkpoints; the run itself does not abort (the failure is logged, not raised — _runner.py).

Error Messages and Stack Traces

agent_framework.exceptions.WorkflowCheckpointException: Failed to decode pickled checkpoint data: SecretString is immutable.

Raised from _workflows/_checkpoint_encoding.py _base64_to_unpickle (underlying
AttributeError: SecretString is immutable. from SecretString.__setattr__ during pickle BUILD).
On FileCheckpointStorage it surfaces from save() during the #8214 save-time restore check, before
any file is written.

Package Versions

agent-framework-core 1.22.0 at commit ebcf4db7f9462ef6c377761e0b3159c4c5a73027 (current main).

Python Version

Reproduced on CPython 3.10.22, 3.12.3, and 3.13.13 (identical; only the protocol-0/1
dumps error-string wording varies by version). Not tested on 3.11/3.14/3.15; the mechanism is
Python-version-independent.

Operating System

Linux (x86_64).

Regression

Yes — introduced by #8127 (merged 2026-09-08), which changed class SecretString(str) to a
slots-based immutable wrapper. As a str subclass the prior SecretString pickled as a string, so
durable checkpointing of state containing it regressed. (As a str subclass it was also JSON-native
and previously written into checkpoints verbatim — a separate cleartext concern that predates this.)

Additional Context

• Not a duplicate of Python: [Bug]: Explicitly allowed ZoneInfo dates cannot be saved in file checkpoints #9440 (ZoneInfo). Different mechanism — verified empirically:

  probe	SecretString (this issue)	ZoneInfo (Python: [Bug]: Explicitly allowed ZoneInfo dates cannot be saved in file checkpoints #9440)
  plain pickle.loads(dumps)	AttributeError: immutable	OK
  decode, allowed_types=None	WorkflowCheckpointException: immutable	OK
  decode, restricted allowlist	WorkflowCheckpointException: immutable	blocked zoneinfo:ZoneInfo._unpickle

  SecretString fails even with no allowlist (a type-level immutability failure at BUILD);
  ZoneInfo round-trips fine unless the restricted unpickler is used. The Python: [Bug]: Explicitly allowed ZoneInfo dates cannot be saved in file checkpoints #9440 fix would not fix
  this and vice-versa. register_checkpoint_type(SecretString) also does not help (the type is
  already permitted; the failure is setattr, not find_class).

• Related but distinct: Python: [Bug]: CosmosCheckpointStorage can save checkpoints that its load() cannot restore #8874/Python: Refuse to save Cosmos DB checkpoints that cannot be restored #8875 (Cosmos save/load asymmetry) govern when this surfaces on
  Cosmos; Python: FileCheckpointStorage saves state it cannot restore: silent save/load asymmetry, get_latest invisibility, listing disagreement, and undocumented JSONDecodeError #8181/Python: FileCheckpointStorage save/load symmetry (#8181) #8214 are why it surfaces as a save-time refusal on File.

• Confidentiality note (secondary, local). Pickling a SecretString embeds its cleartext
  _value in the stored payload (masking is __str__/__repr__-only). The current File path never
  writes it (save refused), but the plain-str workaround — and a naive __getstate__/__setstate__
  fix — both place the secret on durable storage in recoverable form. No attacker or trust-boundary
  crossing is involved; flagging it so the fix direction doesn't institutionalize cleartext-at-rest.
  The file mode is 0o666 & ~umask (umask/dir-dependent), not a fixed world-readable mode.

• Fix direction is a design decision (persist-and-round-trip vs. reject-early vs.
  exclude-and-re-hydrate); happy to prepare a PR once maintainers indicate the preferred direction.

• AI assistance disclosure: AI-assisted analysis and drafting were used; I personally reproduced
  and verified the reported behavior against current source (bare pickle across protocols, the
  checkpoint codec, FileCheckpointStorage save/load, and an end-to-end WorkflowBuilder run on
  CPython 3.10/3.12/3.13), reviewed the submitted content, and accept responsibility for it.

Acknowledgements

• I searched existing issues and did not find a duplicate.
• I personally verified this behavior and the reproduction details are authentic.
• I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.