feat(ocsf): configurable schema version for SIEM backward compatibility

[zanetworker]

Summary

Add a gateway-configurable OCSF schema version target that downgrades JSONL output for SIEMs that only support older schema versions. AWS Security Lake requires v1.1.0, Splunk CIM Add-On targets v1.1-v1.3, CrowdStrike FDR uses v1.5.0. OpenShell's internal model stays at the current version; only the JSONL serialization is transformed.

Related Issue

Closes #2662

Approach

A downgrade filter in the JSONL layer runs after event serialization and before writing. Based on a field-by-field diff across v1.1.0, v1.3.0, and v1.7.0 (analysis), the filter is small:

1. Strip profile-gated fields: ai_model, container, observation_point_id
2. Remove unknown profiles from metadata.profiles: ai_operation, container
3. Rewrite metadata.version to the target version

No field renames needed. The core event structure (class UIDs, activity IDs, HTTP/network fields) is identical across all versions. The differences are all profile-gated additions.

The shorthand log output is unaffected (human-readable, not consumed by SIEMs).

Gateway Configuration

[openshell.gateway]
ocsf_schema_version = "1.1"

The setting flows to sandboxes via the settings bundle (same mechanism as ocsf_json_enabled). Empty or unset means no downgrade.

Changes

openshell-ocsf crate:

• format/downgrade.rs — downgrade_event() function that strips fields and profiles based on target version
• tracing_layers/jsonl_layer.rs — with_target_version() method; applies downgrade before writing

openshell-sandbox crate:

• main.rs — creates shared ocsf_schema_version mutex, passes to JSONL layer
• lib.rs — apply_ocsf_schema_version_setting() reads the gateway setting and updates the shared version

SIEM Compatibility Matrix

SIEM	OCSF Version	Downgrade Needed
AWS Security Lake	v1.1.0	Yes
Splunk CIM Add-On	v1.1-v1.3	Yes
CrowdStrike FDR	v1.5.0	Yes (strips v1.7+ fields)
Datadog Cloud SIEM	v1.5.0 (selectable)	Depends on config

Testing

• cargo test -p openshell-ocsf — 134 passed (7 new downgrade tests)
• cargo clippy --all-targets -- -D warnings — clean
• cargo check -p openshell-sandbox — compiles with settings wiring

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Unit tests added

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[zanetworker]

E2E Verification: Splunk CIM mapping with v1.1 downgraded events

Tested the full pipeline on a live cluster with Splunk Enterprise:

1. Gateway setting: ocsf_schema_version = "1.1"
2. Sandbox emits events internally at v1.7, JSONL layer downgrades to v1.1
3. Events sent to Splunk HEC with sourcetype=ocsf
4. OCSF-CIM Add-On maps events to standard CIM data models

Splunk CIM query: sourcetype=ocsf tag=network tag=communicate | table _time dest dvc vendor_product metadata.version

Result: 13 events mapped to CIM Network Traffic data model with metadata.version = 1.1. CIM fields (dest, dvc, vendor_product) populated automatically via the OCSF-CIM Add-On's eventtypes and field aliases. No custom dashboards or SPL required.

This proves the "plug OpenShell into your existing SIEM and it just works" story for customers running Splunk with the OCSF-CIM Add-On on v1.1 schemas.

[zanetworker]

Related: #2664 adds AI inference events via the ai_operation profile on ApiActivity [6003] and bumps the schema to v1.8.0. The downgrade filter in this PR strips those profile fields when targeting v1.1/v1.3 for older SIEMs. The two PRs are independent and can be merged in either order.

[johntmyers]

@zanetworker does this configuration need to be addressed in the docs?

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid implementation of accepted issue #2662; the author also supplied live Splunk v1.1 integration evidence.
Head SHA: 8fdaeece2ce7396ade50850d4de6765a01c01985
Base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Merge base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Patch ID: 99ebca72e7706c76032c9ed2d3289e7b3ea8930c
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Thanks @zanetworker, I checked the live Splunk verification and the note that #2664 is independent; those support the feature's validation. @johntmyers's question about documentation is also well-founded: the existing published OCSF export contract becomes inaccurate at this head.

Blocking findings:

• GATOR-8fdaeece-01: Unsupported and malformed target versions are accepted; see the inline finding.
• GATOR-8fdaeece-02 — Warning: Published OCSF export guidance must accurately document selectable output versions and the supported configuration workflow.
  • Prerequisite: An operator follows /observability/ocsf-json-export to configure or integrate JSON export.
  • Entry point → sink: Published OCSF export docs and openshell settings → schema-aware SIEM ingestion of the emitted JSONL.
  • Base → head: The page accurately promised complete v1.7.0 objects before schema selection existed; this head adds field-stripping, schema-selectable output while the page still promises every record is complete v1.7.0 and gives no selection workflow.
  • Impact: The primary integration page materially misstates the emitted contract and leaves operators unable to discover or correctly configure the feature.
  • Reproducer: Follow docs/observability/ocsf-json-export.mdx, then set ocsf_schema_version; downgraded records contradict its v1.7.0 guarantee, and the page has no 1.1/1.3 instructions.
  • PR ownership: The contradiction is created by this PR's new public setting; the relevant docs page already exists but is unchanged, so this blocker cannot be anchored to a docs diff line. The unchanged contract is at docs/observability/ocsf-json-export.mdx:42 and :164; .agents/skills/openshell-cli/SKILL.md:616 also documents the neighboring settings workflow but omits this key.
  • Requested change: Update the export page with global/per-sandbox commands, exact supported values, default/unset behavior, hot-reload timing, metadata rewriting, and omitted fields; update the CLI skill guidance; reference the new screenshot or remove it.

Carried findings:

• None

Non-blocking suggestions:

• None

Docs: Missing for a direct user-facing gateway setting; the current export page is materially false.

Next state: gator:in-review

[johntmyers]

gator-agent

Warning — GATOR-8fdaeece-01

Invariant: Only explicitly supported OCSF target versions may be configured; invalid values must be rejected before they affect emitted metadata or downgrade behavior.

Prerequisite: An operator with settings permission supplies an unsupported or malformed ocsf_schema_version.

Entry point → sink: openshell settings set, the TUI settings editor, or UpdateConfig gRPC → the sandbox JSONL downgrade layer and emitted metadata.version.

Base → head: The key previously did not exist and settings APIs rejected it as unknown. This line registers it with no allowed-value constraint, so any string is accepted; banana parses as 0.0.0 and is emitted as the metadata version, while 1.6 performs no transformation and leaves metadata at 1.7.0.

Impact: A typo or unsupported version is reported as successful but produces mislabeled or wrong-version telemetry that schema-aware SIEM ingestion can reject, violating issue #2662's accepted actionable-rejection requirement.

Reproducer: Set the global key to banana, enable JSON export, and emit an event; the command succeeds and output is stripped and labeled metadata.version: "banana". Set it to 1.6; the command succeeds while output remains labeled 1.7.0.

PR ownership: This PR introduces the setting and leaves it unrestricted even though the shared registry already supports allow-list validation.

Requested change: Define the supported versions as a shared constant, use it for allowed_string_values, and add registry/API tests showing supported values succeed while malformed, newer, and otherwise unsupported values fail with the allowed list.

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid implementation of accepted issue #2662; the author also supplied live Splunk v1.1 integration evidence.
Head SHA: da10094fda901b31d2ead35f8e6c243b921ac291
Base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Merge base SHA: dd2b4e3bc0688bdd59f90030f7c1d52511d6e354
Patch ID: c14374c353d8b1657f1c4dcdcf47a657a40a6bf9
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: 8fdaeece2ce7396ade50850d4de6765a01c01985
Review budget exhausted: no
Maintainer decision required: no

Thanks @zanetworker. I checked the validation and documentation update in your latest commit. The allow-list and tests resolve GATOR-8fdaeece-01, but the published configuration guidance still needs one consistency pass.

Blocking findings:

• GATOR-8fdaeece-02 — Warning (carried): Published OCSF export guidance must document only configuration paths and target versions that the runtime accepts.
  • Prerequisite: An operator follows the new OCSF export documentation to configure the downgrade through gateway.toml or for a v1.5 SIEM integration.
  • Entry point → sink: the documented gateway --config / OPENSHELL_GATEWAY_CONFIG path or openshell settings set → gateway TOML deserialization or RegisteredSetting::validate_string_value.
  • Base → head: The previous head omitted the setting. This head documents [openshell.gateway].ocsf_schema_version, but the gateway file schema has no such field and rejects it; it also recommends 1.5 for CrowdStrike and Datadog even though the new allow-list accepts only empty, 1.1, and 1.3.
  • Impact: Following the TOML example deterministically prevents gateway startup, while following either v1.5 recommendation produces a settings validation error and cannot configure the advertised integration.
  • Reproducer: Start the gateway with [openshell.gateway] containing ocsf_schema_version = "1.1"; parsing rejects the unknown field. Separately run openshell settings set --global --key ocsf_schema_version --value "1.5"; the new allow-list rejects it.
  • PR ownership: This PR introduces the setting, and the latest delta adds these incompatible examples while resolving the existing docs obligation.
  • Locations: docs/observability/ocsf-json-export.mdx:160, :181, :182, and :196; docs/reference/gateway-config.mdx:81.
  • Requested change: Remove the gateway.toml example and gateway-config reference unless that entry point is implemented. Remove or clearly mark the v1.5 workflows as unsupported, and state that selectable downgrade targets are 1.1 and 1.3 through global or per-sandbox settings.

Carried findings:

• GATOR-8fdaeece-02: still open as described above.

Docs: Present, but currently advertises unsupported configuration paths and values.

Next state: gator:in-review