Skip to content

feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope - #426

Draft
d-v-b wants to merge 2 commits into
mainfrom
claude/zarr-python-scope-a042b7
Draft

d-v-b wants to merge 2 commits into
mainfrom
claude/zarr-python-scope-a042b7

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Add the ZARR_PYTHON_CONTEXT scope, which reads what zarr-python writes: CORE_AND_EXTENSIONS_CONTEXT plus zarr-python's own data types null_terminated_bytes (NumPy's S) and variable_length_bytes, which no specification registers. CORE_AND_EXTENSIONS_CONTEXT now also reads the fixed_length_utf32 data type (NumPy's U) and the vlen-utf8 and vlen-bytes codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.

Rename the v3 scopes CORE and CORE_AND_EXTENSIONS to CORE_CONTEXT and CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a ZarrV3Context.

Fixes #423

What the agent checked

The agent generated v3 metadata with zarr-python 3.4.2 for <U3, S4, str and VariableLengthBytes() arrays. All four now read with no problem and no unclaimed field in ZARR_PYTHON_CONTEXT. Before this change, CORE_AND_EXTENSIONS left the string array's vlen-utf8 codec unclaimed as well as the data types.

Written by zarr-python Registered in zarr-extensions Scope
fixed_length_utf32 yes CORE_AND_EXTENSIONS_CONTEXT
vlen-utf8, vlen-bytes codecs yes CORE_AND_EXTENSIONS_CONTEXT
null_terminated_bytes no (zarr-python warns the spelling may change) ZARR_PYTHON_CONTEXT
variable_length_bytes no (zarr-python warns the spelling may change) ZARR_PYTHON_CONTEXT

Rules the new definitions check:

  • length_bytes is a positive multiple of 4 for fixed_length_utf32, and at least 1 for null_terminated_bytes.
  • A fill value fits the type, and is valid base64 where zarr-python writes base64. The canonical null_terminated_bytes fill value drops trailing null bytes.
  • fixed_length_utf32 is multi-byte, so the bytes codec requires an endian.
  • The vlen-* codecs reject a data type of fixed size. This uses the data type's declared storage, so it does not tell string from bytes: a name check would refuse zarr-python's own variable_length_bytes + vlen-bytes pairing.

Not covered

  • zarr-python's raw_bytes (NumPy V): its type names would collide with the RawBytes* names that r* already exports at the top level.
  • The legacy structured alias that earlier zarr-python 3.x releases wrote.
  • numcodecs.* codecs.
  • The v2 scope CORE_V2 keeps its name.

just check passes on Python 3.11 and the newest supported version, including the strict docs build.

🤖 Generated with Claude Code

d-v-b added 2 commits October 29, 2026 20:02
…ONTEXT scope

Add definitions for the extensions zarr-python writes. zarr-extensions
registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes`
codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own
`null_terminated_bytes` and `variable_length_bytes`, which no
specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the
default scope stays spec-pure.

Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to
CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context.

Fixes #423

Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zarr-metadata: ship definitions for zarr-python's registered extension data types

1 participant