Shared source AST standard (discovery + lineage types)

[matthewmoorcroft]

What this adds

This PR establishes the source-neutral discovery contract that ADF and Airflow can both map onto before Databricks IR conversion. It adds the shared source AST, shared lineage vocabulary, JSON round-trip helpers, and source-neutral lineage derivation primitives. It is additive infrastructure only: this PR does not wire either source, change convert, or change packaging.

Shared source AST schema

SourceGraph

One source workflow (ADF pipeline or Airflow DAG):

• name, source, optional description
• parameters and variables, each keyed to ParameterSpec {type, default}
• optional schedule: ScheduleSpec {kind, expression, timezone, extensions}
• optional default_policy: PolicySpec {timeout_seconds, max_retries, retry_interval_seconds, extensions}
• optional run_timeout_seconds for a whole workflow run
• tags, ordered top-level tasks
• optional lineage
• properties, extensions, and optional verbatim raw for source-specific detail

SourceNode

One source task, kept source-faithful:

• identity/classification: source_id, task_key, concept, source, optional name, optional verbatim native_type
• dependencies: dependencies[]
• execution semantics: optional node-level run_condition, optional policy
• data endpoints: data_reads[], data_writes[]
• source-specific properties and optional verbatim raw

The JSON form also carries a node_type discriminator so SourceNode, ContainerNode, and GapNode round-trip to the correct class.

ContainerNode

Extends SourceNode with branches: {label: SourceNode[]}. The branch labels remain source-owned: for example ADF IfCondition maps to true/false, ForEach and Until to body, Switch to case values plus default, and an Airflow task group can use group. Empty branches remain present instead of being flattened away.

SourceDependency

Each upstream edge is {upstream, conditions[], resolved}. conditions preserves per-edge source outcomes such as ADF Succeeded/Skipped; resolved: false records a missing upstream rather than dropping it.

GapNode is the lossless fallback: it defaults concept to gap, adds an optional reason, and retains the source payload in raw.

Shared lineage schema

• DataAsset {signature, identity?, asset_type?, properties} — signature is the always-present neutral descriptor; identity is a resolved physical identity or null, never guessed; asset_type is an open vocabulary that includes physical assets (table, file, volume) and logical/value hand-offs such as a dataset or XCom.
• ControlEdge {source_workflow, target_workflow, via_task_key, wait_for_completion?, resolved} — a cross-workflow invocation with unresolved callees retained.
• DataEdge {source_task_key, target_task_key, match_kind, match_key, identity?, asset_type?} — a proven producer/consumer hand-off, recording whether matching used strong identity or fallback signature.
• MotifAnnotation {motif_id, member_task_keys[], display_name?, databricks_replacement?, notes[]} — a neutral annotation shape for a detected multi-task motif.
• Lineage {control_edges[], data_edges[], motifs[]} — stable concrete lists, including when empty.

Neutrality choices

The schema distinguishes execution semantics that look similar but belong at different levels:

• SourceDependency.conditions is per edge; SourceNode.run_condition is a downstream-node aggregate rule such as an Airflow trigger rule.
• SourceNode.policy is task-specific; SourceGraph.default_policy is the cascading workflow default; SourceGraph.run_timeout_seconds is the timeout for the whole workflow run.
• DataAsset is deliberately best-effort and open. Discovery may record physical, logical, or value-level data, may leave identity unresolved, and must not invent a physical identifier.
• Shared typed fields cover only concepts genuinely common to sources. Source-specific data remains available through raw, properties, and graph-level extensions.

How the AST and lineage feed inventory.json

The end-to-end discovery path built on this foundation is:

1. A source mapper creates one SourceGraph per workflow and one node per source task, preserving dependencies, branches, raw payloads, strategy metadata, reads/writes, and workflow-call markers.
2. The source-neutral lineage walker traverses every ContainerNode branch. It derives control edges from workflow-call markers and data edges from data_writes -> data_reads, then attaches one Lineage block to the graph.
3. The source-agnostic inventory projector flattens the graph's nodes in source order into each pipeline's activities[], preserves dependency conditions/raw data additively, and serializes the graph's lineage beside those activities as the per-pipeline lineage block.
4. All sources therefore feed the same top-level inventory shape (source, source_dir, pipelines, summary) once their mapper targets this AST.

This PR supplies the shared models, serde, walkers, and edge-building core used by that path. The source mappers and inventory.json wiring land in the follow-on discovery PRs; #84 itself intentionally imports no flowx.sources.adf or flowx.sources.airflow modules.

Non-breaking scope

• No ADF/Airflow source wiring in this PR.
• No conversion or package behavior changes.
• Discovery AST and lineage serde preserve raw/properties/extensions and empty branch/list structure.
• Existing flows remain unchanged until a source opts into the shared contract.

Verification

The PR's unit coverage exercises model/serde round trips, container traversal, control-lineage derivation, two-tier data matching, unresolved edges, and source-neutral import guards. make test and make fmt were the acceptance commands for the branch.