feat: serve IDC v25 (idc-index 0.13.0)

[fedorov]

Moves the served data release from v24 to v25 and fixes a schema-discovery gap that v25's new column exposed.

Serving IDC v25

idc-index 0.12.5 → 0.13.0, pulling idc-index-data 24.2.2 → 25.0.0. That release is the data bump plus CI chores — no upstream API change. /v3/version and the MCP get_idc_version now report v25: 179 collections, 26 analysis results, 1,044,191 series, 99.9 TB, 242 DOIs (up from 237).

Schema drift across all 17 tables is two columns: analysis_results_index gains provenance, and prior_versions_index drops gcs_bucket_1 (unreferenced here).

Struct columns were undiscoverable

v25's provenance is a struct, and get_table_schema described it as the bare type RECORD — which names no field a caller could select. The same was already true of collections_index.sources, a RECORD[] whose elements nest two further structs (license, provenance). Writing SELECT provenance.data_contributor meant guessing field names, which is precisely what this server tells callers not to do.

The upstream schema JSON ships no fields for a RECORD, so those names exist nowhere but the Parquet footer. get_table_schema now reads them from there and advertises the real type:

provenance: STRUCT(data_contributor VARCHAR, source_data_provider VARCHAR,
                   deidentification_party VARCHAR, dicom_conversion_by VARCHAR)

Three constraints shaped the fix:

• Only RECORD columns are substituted. The rest keep their BigQuery-style names, which numeric_range_attributes() matches on.
• A repeated record's DuckDB type already ends in [], so the mode is not applied on top of it (no [][]).
• Specialized indices have no local Parquet until fetched, so they degrade to the un-expanded RECORD rather than failing discovery.

The runtime path needed no change — structs already serialized correctly, and dot access and UNNEST already worked.

provenance is deliberately left reachable via SQL only; it is not promoted into the typed AnalysisResult / CollectionDetail responses.

Testing

• 103 pass with the full specialized index set; 92 pass (10 skipped) on a bundled-only build, covering the degradation path.
• Five new tests in tests/test_struct_columns.py pin the expansion, the no-double-[] rule, selectability, serialization, and the missing-Parquet fallback.
• Both SQL examples added to the docs were run verbatim.
• ruff check / ruff format clean; bandit shows only pre-existing nosec notes.

Docs

Struct guidance sits beside the existing array guidance in docs/user-guide.md and is mirrored into the idc://guide MCP resource. INSTRUCTIONS is untouched — it stays lean. Changelog entry added under [Unreleased].

The version bump to 3.0.0b5 is not in this PR: the runbook requires it be its own commit, cut after this merges.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The production build ignores the lockfile, so the lower-bound dependency can silently serve a later IDC release.

3 open findings

• Pin idc-index exactly to version 0.13.0 · New
• Update cohort call and DOI counts from 237 to 242 · New
• Correct outdated round-trip count from 237 to 242 · New

What changed in this PR

Updates the service to IDC v25 and expands struct schemas using Parquet metadata.

Changes:

• Bumps idc-index data dependencies to v25.
• Exposes nested struct field types in schema discovery.
• Adds struct tests and updates user-facing documentation.

File	Description
pyproject.toml	Updates the index dependency.
uv.lock	Locks IDC v25 packages.
src/​idc_api/​core/​schema.py	Expands RECORD column types.
src/​idc_api/​core/​models.py	Updates version examples.
src/​idc_api/​core/​services/​citations.py	Updates DOI-count documentation.
src/​idc_api/​rest/​app.py	Updates REST examples to v25.
src/​idc_api/​mcp/​server.py	Adds struct-query guidance.
tests/​test_struct_columns.py	Tests struct discovery and serialization.
tests/​test_citations.py	Updates citation documentation.
docs/​user-guide.md	Documents struct queries and v25.
dev/​deployment.md	Updates release-pin example.
CHANGELOG.md	Records v25 and schema discovery changes.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[fedorov]

Opened #57 for the root cause behind the exact-pin discussion above.

The idc-index==0.13.0 pin in this PR fixes the one dependency where resolution drift is user-visible (it decides the IDC release advertised at /v3/version). The general problem is wider: Dockerfile:14-16 installs from pyproject.toml alone, so uv.lock constrains CI but not the image — meaning every other runtime dependency can still differ between the versions CI tested and audited and the versions that ship.

Out of scope here; tracked in #57.

_{Drafted with Claude Code.}

[Copilot]

🔵 Needs a closer look

The fallback test can pass without exercising fallback behavior, and schema adapter parity remains untested.

0 open findings

3 resolved since last review

• Pin idc-index exactly to version 0.13.0
• Correct outdated round-trip count from 237 to 242
• Update cohort call and DOI counts from 237 to 242

Previously missed (2)

In code that hasn't changed since last review

Add REST and MCP schema parity assertions

tests/​test_struct_columns.py:22

The new schema contract is asserted only through the core service. Repository guidance requires capability changes to retain core/REST/MCP parity; add an async parity assertion comparing this expanded schema with GET /v3/tables/analysis_results_index and the MCP get_table_schema result so adapter serialization regressions are covered.

Test does not verify missing-Parquet fallback behavior

tests/​test_struct_columns.py:74

This test does not actually pin the missing-Parquet fallback: it verifies that seg_index lacks a path but then describes clinical_index, and the final assertion explicitly accepts STRUCT(...), which is the non-fallback result. Use the same table throughout, force that table's Parquet lookup to be absent (clearing the schema caches), and assert the exact bare RECORD/RECORD[] result so a regression cannot pass.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[fedorov]

Both points from the second review pass are fixed in 3257d3a — they were fair.

The fallback test was close to vacuous. It asserted on seg_index, then described clinical_index, and its final or ... startswith("STRUCT(") accepted the non-fallback result, so nothing could make it fail. Replaced with three tests that can:

• one table throughout, asserting the exact bare RECORD[]
• direct _column_type assertions for the fallback rendering, including that an unrelated expansion is not borrowed for a column that has none
• the real degradation path forced on a table that otherwise expands — drop analysis_results_index's parquet_filepath, clear the lru_caches, assert RECORD (with a fixture clearing the caches on both sides, so a fake value cannot outlive the monkeypatch)

Verified by mutation rather than by eye: rendering an empty struct as "STRUCT()" fails all three. The old test passed that mutation, which is precisely the criticism.

Parity was genuinely missing. test_parity.py covered version, counts, citations and clinical, but not table schemas — the payload this PR changes. Added test_table_schema_parity_including_struct_columns: core == REST == MCP for both struct tables, plus an assertion that collections_index.sources keeps its embedded quoted identifier ("Access" VARCHAR) through JSON on both surfaces. That last one is the concrete serialization risk an expanded struct introduces, and it was untested.

Suite is 106 passing with all indices, 95 + 10 skipped bundled-only.

_{Drafted with Claude Code.}