Skip to content

feat(zarr): store the source array metadata document on read (provenance) - #11483

Open
d-v-b wants to merge 2 commits into
pydata:mainfrom
d-v-b:claude/zarr-metadata-fragment
Open

feat(zarr): store the source array metadata document on read (provenance)#11483
d-v-b wants to merge 2 commits into
pydata:mainfrom
d-v-b:claude/zarr-metadata-fragment

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds a new zarr_array_metadata key to the encoding dict when reading a Zarr array. zarr_array_metadata is a dict representation of a zarr v2 or zarr v3 array metadata document. including this document in the encoding dict means xarray gains a simple, complete representation of input zarr data that's much more robust than vague chunks and shards keys.

supporting this change are a few additions:

  • typeddict types that very generally model the structure of zarr array metadata for v2 and v3. These types are defined by the zarr spec, so they are not going to change any time soon. I think defining these types locally is fine, but if you want to import these types look at https://zarr-metadata.readthedocs.io/en/latest/, which is not stable yet but getting there.
  • routines that extract the zarr format version (2 or 3) and the metadata document itself from a zarr array. these are defined as separate functions to insulate xarray from changes to the zarr array API. bare property access on a zarr array is not as durable as a dedicated function.

this PR is stacked on top of #11479. the end goal is closing out #11431, by getting xarray to use the stored array metadata for serializaing zarr data instead of keys like shards and chunks.

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR. Tools: Claude code

@d-v-b @claude
refactor(zarr): centralize and split encoding validation on the write…
eab28a3 
… path

Define the encoding keys the Zarr backend accepts as module-level constants
(ZARR_V2_ENCODING_KEYS, ZARR_V3_ENCODING_KEYS, ZARR_READ_ONLY_ENCODING_KEYS)
and extract a pure _validate_zarr_variable_encoding helper from
extract_zarr_variable_encoding.

_create_new_array now assembles zarr create() arguments in its own dict
instead of mutating the variable's encoding, takes the dimension names
explicitly, and records them in one place for both zarr formats (native
dimension_names metadata for format 3, the hidden _ARRAY_DIMENSIONS
attribute for format 2). Remove unreachable "order" handling from array
creation, and correct the formatting of the conflicting-write_empty_chunks
error message.

Behavior is otherwise unchanged; tests are split one case per function and
new tests pin the constants and the conflict error.

Assisted-by: ClaudeCode:claude-fable-5
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added topic-backends topic-zarr Related to zarr storage library io labels Jul 30, 2026
@d-v-b d-v-b changed the title Claude/zarr metadata fragment feat(zarr): store the source array metadata document on read (provenance) Jul 30, 2026
@d-v-b

d-v-b commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

if this is merged, I would recommend considering a deprecation path for the current top-level encoding keys like shards, chunks, serializer, etc. All the information those keys convey is better conveyed by parsing the raw metadata document, and that process doesn't depend on zarr-python's API.

If such a deprecation path isn't an option, then I recommend at a minimum deriving the values for these keys from the metadata document. Either way, the end goal is to use the metadata document as the source of truth for the structure of the zarr array, not the zarr array object.

@d-v-b @claude
feat(zarr): store the source array metadata document on read
943f9b4 
When a variable is read from a Zarr store, keep the array's complete metadata
document on encoding["zarr_array_metadata"] for provenance and introspection.
The key is read-only: it is listed in ZARR_READ_ONLY_ENCODING_KEYS, dropped on
write, and never forwarded to zarr's array-creation call.

The document is emitted by private per-format helpers (_v2_array_metadata,
_v3_array_metadata) that read the metadata keys individually and return
locally-defined TypedDicts mirroring the key names of the zarr-metadata
package's ZarrV2ArrayMetadataJSON / ZarrV3ArrayMetadataJSON, with deliberately
wide value types. These helpers are the only places xarray touches
zarr-python's metadata API, so future changes to zarr-python's array class
shape are absorbed there and surface loudly.

Assisted-by: ClaudeCode:claude-fable-5
Co-authored-by: Claude <noreply@anthropic.com>
@d-v-b
d-v-b force-pushed the claude/zarr-metadata-fragment branch from 34e4e81 to 943f9b4 Compare July 30, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

io topic-backends topic-zarr Related to zarr storage library

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b