datasets: define the agent-trace schema and seed the first entries

[gautamgb]

Summary

Defines a data format for datasets/agentdataflow_toolexchange_traces/, ships the tooling to enforce it, and seeds the first entries so the schema is grounded in real traces rather than designed in the abstract. The folder's README currently carries a <!-- TODO: Define schema once initial data is contributed --> where its format should be.

Supports the Data Collection and Data Validation workstreams.

File
schema.json	JSON Schema draft 2020-12
validate.py	Schema, span graph, provenance, date, and secret-scan checks
build_index.py / index.csv	Same pattern as exploit_dataset
example.json	Worked template, validated alongside the real entries
entries/ (2)	Two traces of tool-contract change reaching an agent's data plane
README.md	Data Format section written; Scope and Sanitization sections unchanged

First: there is already a schema for this dataset

data_validation/schemas/agentdataflow_trace.schema.json exists and defines this dataset. I did not see it before starting, and you should have that fact up front rather than discover it in review.

Having found it, the situation looks less like a duplicate and more like something worth deciding deliberately.

The repo already has two schema systems, and they already disagree - before this PR. datasets/exploit_dataset/ ships its own schema.json with seven live entries validated against it. Those same entries fail data_validation/schemas/exploit.schema.json. Taking the merged entries/AML.T0010.json against the central schema:

6 errors
  'name' is a required property                    (local schema uses 'title')
  'source' is a required property                  (local schema uses 'source_urls')
  'AML.T0010' does not match '^EXP-\d{4}$'
  ['Ability to publish, modify, or impersonate an upstream component.'] is not of type 'string'
  ['supply_chain_compromise', 'integrity_compromise', ...] is not of type 'string'
  ['Newly added dependencies from unusual publishers.', ...] is not of type 'string'

The central data_validation/schemas/*.json describe an earlier design - sequential EXP-0001 identifiers, scalar impact, source as a single string - that the dataset-local schemas superseded in practice. The dataset-local one is the one with data behind it. That is the precedent this PR follows.

What I aligned. The category enum now uses the central schema's values, since that divergence was free to give up:

delegation_chain  ->  multi_agent_delegation
plugin_exchange   ->  plugin_data_exchange
memory_io         ->  memory_read_write

What I kept, and why. These are genuine disagreements rather than cosmetic ones:

Central stub	Here	Why
trace_id ^TRACE-\d{4}$	DSGAI-TRACE-<slug>	Sequential identifiers collide when contributors work in parallel. exploit_dataset abandoned EXP-\d{4} for the same reason.
trace_data, array of untyped objects	spans, typed	An untyped array cannot express actor, operation, data class, or where a finding lands, which is most of what makes a trace analyzable.
type: benign / adversarial	disposition, plus unintentional_failure	Most data-security failures have no adversary. Forcing them into benign or adversarial loses the majority case. Renamed off type because a property called type inside a JSON Schema reads badly.
security_observations: string	array of strings	One falsifiable claim per item, so they can be counted and checked.
sensitivity_annotations: top-level array	per-span data_classes + sensitivity	Sensitivity is a property of a step, not of a trace.

What I would like from you is a direction, not a decision in this PR. Three options:

1. Leave the drift. Dataset-local schemas are authoritative; the central ones remain historical stubs. Cheapest, and matches what already happens.
2. I open a follow-up replacing data_validation/schemas/agentdataflow_trace.schema.json with a pointer to the dataset-local one, and file an issue recording the same drift for exploit_dataset.
3. Reconcile properly: one schema per dataset, centrally held, dataset folders reference it. Much larger, and your call whether it earns its cost.

I would suggest 2, and I am happy to do the work.

The design decision worth your review: provenance tiers

This dataset has been open for contributions and has stayed empty, and I think the reason is structural rather than a lack of interest. The people holding real agent traces mostly cannot publish them - production telemetry contains exactly what the anonymization rule forbids. A dataset that accepts only production telemetry stays empty; a dataset that accepts anything fills with plausible fiction.

provenance.tier takes both without confusing them:

• observed_production - sanitized telemetry from a real deployment
• observed_lab - sandbox, test environment, CTF, red-team exercise
• derived_from_public_observation - the mechanism rendered as a trace, with its precondition carried as a citation in provenance.evidence
• hypothetical - illustrative only, excluded from any statistic

Every tier except hypothetical requires at least one evidence item, enforced in validate.py, and each item states narrowly which claim it backs. The two entries here are derived_from_public_observation: the agent sessions are synthetic, and each precondition cites a counted category in a public CC BY 4.0 contract-drift corpus (DOI 10.5281/zenodo.21449149), with the field and value named in the locator.

If you would rather this dataset accept only observed traces, say so and I will cut the derived tier - but I would expect the folder to stay empty, and I would rather make that argument now than after the format is set.

Two smaller choices, both easy to reverse

1. sanitization.attestation is const: true, so an entry cannot merge with the anonymization box quietly unticked, and validate.py scans for credential-shaped and PII-shaped values independently of the attestation. The scan covers the whole entry, not just span payloads: summary, notes, collection_method, and evidence locator are where a real hostname actually gets typed, because that is the prose someone writes while looking at a live trace. Related to The PR template asks contributors to certify a validation script that cannot fail #61 - this is a certification a validator actually enforces.

2. mitre_atlas_mapping is release-pinned: {atlas_release, techniques: [{id, name_at_release}]} rather than a flat list of identifiers. Correct eleven MITRE ATLAS technique citations #59 and ATLAS mappings needing a judgement call, and an offer of the verifier behind #59 #60 make the case better than I could - identifiers and names have both moved, and an unpinned pair rots silently. Both entries here deliberately omit ATLAS mappings rather than assert an unverified pair while that review is open. Happy to flatten this to match exploit_dataset if you would rather have cross-dataset consistency than the pin.

DSGAI mapping

• DSGAI-TRACE-mcp-declared-effect-flip-stale-autoapproval - DSGAI06, DSGAI05, DSGAI07 (LLM06:2025, LLM03:2025). A tool declares destructiveHint: false, an approval policy caches it into an auto-approve class, the tool later re-declares itself destructive with no change to the server's declared version, and the next call runs destructively with no human approval requested. Backed by 123 deduped incidents of that change class in the source corpus, 25 of them the guarantee-change-under-stable-version case this trace models.

• DSGAI-TRACE-mcp-output-schema-expansion-context-overshare - DSGAI15, DSGAI01, DSGAI14, DSGAI05 (LLM02:2025, LLM05:2025). A tool's output schema widens from a projection to the full record under a stable declared version; a schema-agnostic context append carries three unentitled fields into the model context, the user-facing answer, and the observability sink. Backed by 992 of 2,503 deduped safety-relevant incidents, the largest single class in the corpus, against an overall 62.4% version-silent share.

• example.json - DSGAI04, DSGAI06, DSGAI02. Whole-toolset replacement under a stable server identity, where a name-keyed binding and a name-scoped credential both survive a change to what the name refers to.

Schema-level coverage spans DSGAI01-21 via dsgai_mapping, validated against _shared/dsgai_taxonomy.json.

Worth flagging honestly: all three files are the same archetype - a cached client-side assumption outliving a supplier-side change. That is what one corpus gives you. A second wave should deliberately go elsewhere: a delegation chain, a memory_read_write trace, and at least one adversarial disposition.

Checklist

• Data is anonymized (no PII, credentials, or org names) - all three files are fully synthetic; payloads are typed placeholders by convention, and validate.py enforces it across the whole entry. No third party is named as the subject of a failure.
• Entries reference at least one DSGAI ID where applicable - required by schema, cross-checked against the shared taxonomy, and every span-level finding must map to a risk the trace declares.
• Validation scripts pass - data_validation/run_all_checks.py is currently a stub that prints "Validators not yet implemented", so I ran the dataset-local validator, matching the exploit_dataset pattern:

$ python validate.py
OK: 2 entries + example.json validated against schema.

$ python build_index.py
Wrote index.csv with 2 entries.

Each check was confirmed to fail on a deliberately broken copy. Twelve cases - leaked bearer token in a payload, RFC1918 address in a span summary, email in notes, home path in an evidence locator, non-ISO date_added, missing security_observations, off-enum category, missing evidence on a derived-tier entry, attestation: false, dangling parent_span_id, non-DOI citation typed as doi, and filename/trace_id mismatch - each produced exactly one error and a non-zero exit, with no false positives on the three real files.

• One entry per file for dataset contributions - entries/<trace_id>.json, filename agreement enforced by the validator.

---

Two questions before this is mergeable from my side.

Would you rather review the schema and the entries as separate PRs? Happy to split.

And LICENSE currently reads NOASSERTION while #44 proposes an Apache-2.0 code / CC BY-SA data split, still in draft. Which license should I take this contribution as being accepted under? Everything here is original work written for this repo - the corpus it cites stays where it is, under CC BY 4.0 - so I do not think anything is blocked either way, but I would rather have the answer on the record than assume.

For context on the last point: every validator in data_validation/validators/ is currently a pass with a # TODO - schema_validator, anonymization_scanner, dsgai_mapping_check, completeness_check. The validate.py here implements roughly what those four promise, so lifting it into data_validation/ would close #61's underlying complaint. Separate PR, if you want it.

[gautamgb]

Revised the description and force-pushed a corrected commit before anyone had to read the first one.

Self-review turned up three things: data_validation/schemas/agentdataflow_trace.schema.json already exists and I had missed it (now addressed head-on in the description, along with evidence that the same drift already affects exploit_dataset); the secret scan only walked span payloads and left the free-text fields unscanned; and "format": "date" was decorative, since jsonschema does not enforce format without a format checker, so a garbage date_added passed. All three are fixed and the negative suite is now twelve cases.