Phase 1 stack #3: agentic insights / enrich

[matthewmoorcroft]

Part of the Phase 1 stack. Base: stack/adf-discovery. (Content of the former #82.)

What this adds

An optional, source-neutral, agent-authored insights layer over discovery. The deterministic inventory records what the source is; an external agent authors judgment; the library has no LLM — it validates the authored object and atomically merges it under one additive insights block.

Schema (insights block)

schema_version, inventory_sha256 (fingerprint binding), overview, system_recommendation, pipeline_insights[], pipeline_relationships[] — with recommended_patterns[] and per-pattern simplification_pattern.

Contract

enrich validates foreign keys / lineage refs / evidence / schema, stamps the library-owned fields, and atomically merges. Author→validate→merge; nothing else in the inventory changes.

GA/Preview release state — structured, validated, tiered disclosure

What. Two new optional fields on each recommended_patterns[] entry (per pipeline and system-wide):

• release_state — one of ga / public_preview / private_preview / beta / unknown.
• release_state_source — the doc URL / citation grounding the state.

Why. GA/Preview grounding used to be prose-only in the enrich skill (verify status, then write it into fit/conversion_notes). Nothing structured it, the validator never checked it, and the closed key allow-list would have rejected a status field. This makes it a real, validated, structured field that downstream routing can surface consistently.

How it works (validator rules, all violations aggregated — no fail-fast):

• release_state is enum-validated when present.
• Required-when: release_state is required whenever simplification_pattern == true — a distinctive capability must declare its verified release state.
• Citation-required: release_state_source (a non-empty string) is required when release_state ∈ {public_preview, private_preview, beta}; not required for ga / unknown.

Surfacing intent (neutral factual disclosure, not a warning — see #97 for the routing side):
ga and unknown are silent (unknown is treated exactly like ga); public_preview is disclosed as "Public Preview (production-ready)" (Public Preview is generally production-ready and supported per Databricks); private_preview / beta are stated as plain factual labels. No alarm/severity/warning framing.

Grounding rule kept as-is (independent of surfacing tone): verify GA/Preview status against current public Databricks docs before recommending (never "recommend now, verify later").

Private Preview is now disclose-and-decide, not a hard gate. The earlier draft treated a Private Preview feature as doNotSuggest unless the workspace had confirmed enrollment/entitlement. Because release state goes stale and enrollment is the customer's decision, the skill now discloses the verified release state (via the structured release_state / release_state_source fields), notes that Private Preview typically requires workspace enrollment/entitlement, and lets the customer decide whether to proceed — the tool no longer unilaterally refuses to suggest it (there is no doNotSuggest gate). The verify-before-recommend requirement and the neutral (non-warning) surfacing are unchanged.

No hardcoded GA/Preview status in the skill text or examples

Hardcoded GA/Preview statuses go stale — we confirmed via live docs that a real connector's release state had already moved (a "Gateway CDC = Private Preview" style claim can be wrong; SQL Server LFC is GA). So the enrich skill's illustrative JSON examples no longer pin any real named connector to a literal status value: the system_recommendation and pipeline_insights examples now use anonymized pattern names with a clearly-placeholder release_state that tells the author to verify live and not copy it, and the decision_driver example no longer hardcodes "GA". A grep of the enrich skill confirms no real-connector-to-status literal remains; everything routes through live verification + the structured disclosure fields. (The parallel Airflow-side grounding rule is tracked separately in #103 and is untouched here.)

Back-compat. The new fields are optional; insights authored before this field still validate — except the one intended new strictness: simplification_pattern == true now requires release_state.

This pull request and its description were written by Isaac.