Skip to content

feat(array): add ArrayIO and AsyncArrayIO, the explicit-IO layer under Array and AsyncArray - #419

Draft
d-v-b wants to merge 8 commits into
mainfrom
claude/arrayio-lazy-metadata-48d8e5
Draft

feat(array): add ArrayIO and AsyncArrayIO, the explicit-IO layer under Array and AsyncArray#419
d-v-b wants to merge 8 commits into
mainfrom
claude/arrayio-lazy-metadata-48d8e5

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Added zarr.ArrayIO and zarr.AsyncArrayIO, an explicit-IO layer over Zarr arrays. Constructing one performs no IO against storage: storing metadata (store_metadata), storing or reading chunks (setitem / getitem), and their erasing inverses (erase_metadata, erase_chunk, erase_chunks) are explicit method calls that run in whatever order the caller chooses — so chunks can be written before metadata, as in the array API of zarrs. open auto-detects the Zarr format by default, as zarrs does, and verify reads the stored metadata and raises, naming each differing field, when it does not match the bound metadata.

The binding holds the metadata document as a zarr-metadata model — zarr-python now depends on zarr-metadata>=0.6.0, its first sibling-package dependency — reachable as ArrayIO.document. Document operations (store_metadata, erase_metadata, erase_chunk, erase_chunks, verify, equality) never resolve codec or data type names: an array whose codecs this environment lacks still opens for document work and refuses only when chunks are read or written. The strict model read refuses a stored document zarr can only read as repaired (AsyncArray.open remains the repairing reader), and instances built from resolved metadata — every AsyncArray — keep deriving their chunk grid, chunk keys and equality from the resolved form, because zarr-python writes some documents the strict model refuses (Zarr format 2 generic time units).

AsyncArray is now an array-like layer over this one: it holds a single wrapped AsyncArrayIO, exposes metadata / store_path / config / codec_pipeline as properties over it, and delegates its metadata and selection IO to it, with resize rebinding the wrapped state so the two layers cannot drift. AsyncArrayIO.to_array() returns an AsyncArray sharing the same binding. AsyncArrayIO defines value equality over the full binding, preserving AsyncArray's previous dataclass equality.

The normalization half of init_array is extracted into the pure build_array_metadata, which init_array and AsyncArrayIO.build share; init_array's behavior is unchanged.

Design notes:

  • Every write has an erasing inverse (store_metadata ↔ erase_metadata, chunk writes ↔ erase_chunk / erase_chunks), each succeeding when the target is already absent, copying zarrs.
  • store_metadata() writes unconditionally, like store_metadata in zarrs; guarded creation (raising when a node exists) remains create_array's job, and chunks of a previous array at the location are kept — erase_chunks is the broom.
  • The bound metadata is assumed, never verified against the store (TensorStore's assume_metadata hazard); this is documented on the class. verify is the explicit check.
  • The classes live in zarr.core.array and are exported from the zarr namespace, next to the classes built on them; the wrapped layer is reachable as Array.io / AsyncArray.io, the inverse of to_array().

Issues this layer addresses or advances: zarr-developers#4217 (the 2026-10-29 comment: avoiding implicit IO, writing chunks before metadata — the Array-over-engine layering that issue describes now exists at the array layer), zarr-developers#1435 (idempotent, non-deleting parallel creation), zarr-developers#4175 (build_array_metadata moves metadata creation off AsyncArray), zarr-developers#4399 / zarr-developers#1754 (storage-free declaration with caller-scheduled IO), zarr-developers#3017 (metadata IO as explicit free-standing operations), and zarr-developers#4027 / zarr-developers#3782 (the Array / AsyncArray relationship: the sync ArrayIO still routes through sync(), so the event-loop issues — zarr-developers#2878, zarr-developers#3487, zarr-developers#3859 — need a sync engine under zarr-developers#4217 before they close).

Stack

Base of the stack. #420 (Context scoping) stacks on this branch.

🤖 Generated with Claude Code

d-v-b added 4 commits October 29, 2026 11:53
@d-v-b
feat(experimental): add ArrayIO and AsyncArrayIO, an explicit-IO arra…
88ab631 
…y layer

Declaring an array today eagerly stores its metadata, so chunks cannot be
written first. ArrayIO and AsyncArrayIO bind an array metadata document to
a store location without performing any IO on construction; storing
metadata (store_metadata) and storing or reading chunks (setitem/getitem)
are explicit method calls that run in whatever order the caller chooses,
mirroring the array APIs of zarrs and zarrita.

The normalization half of init_array is extracted into the pure
build_array_metadata, which init_array and AsyncArrayIO.build share;
init_array's behavior is unchanged.

Refs zarr-developers#4217

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
docs: changelog fragment for #419
c15b44a 
Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
feat(experimental): give ArrayIO erasing inverses and a format-detect…
a51bdbe 
…ing open

Every write now has an inverse, copying zarrs: store_metadata ↔
erase_metadata, chunk writes ↔ erase_chunk / erase_chunks, each
succeeding when the target is already absent. open's zarr_format now
defaults to None (read format 3, fall back to format 2), matching
zarrs' open, instead of requiring format 3.

Adversarial review against zarrs and TensorStore also found contract
gaps now documented: the bound metadata is assumed, never verified
(TensorStore's assume_metadata hazard); store_metadata keeps any
previous array's chunks; a repaired document is stored before the
first chunk write; and the module docstring no longer cites zarrita,
whose create is eager.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
feat(experimental): let ArrayIO verify its bound metadata against the…
c618bbe 
… store

A writer that declared an array independently (the distributed
chunks-first pattern: one coordinator stores metadata, N workers build
the same declaration) can assert its assumption before writing chunks.
verify() reads the stored document, favoring format 3 when both exist,
and raises naming each differing field — the explicit counterpart of
TensorStore's spec-constraint validation on open.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b d-v-b mentioned this pull request Oct 29, 2026
Draft
d-v-b added 3 commits October 29, 2026 13:04
@d-v-b
refactor(array): make AsyncArray wrap AsyncArrayIO
102a1ad 
The explicit-IO layer is the lower layer, so it moves into
zarr.core.array as a self-contained class holding its own (metadata,
store path, config, codec pipeline, chunk grid) state, with
zarr.experimental.array_io re-exporting it. AsyncArray now holds one
field, the wrapped AsyncArrayIO, exposes metadata/store_path/config/
codec_pipeline/_chunk_grid as properties over it, and delegates its
metadata and selection IO to it; resize rebinds the wrapped state, so
the two layers cannot drift. AsyncArrayIO defines value equality over
the full binding, preserving AsyncArray's previous dataclass equality,
and to_array() returns an AsyncArray sharing the same binding. The
repair-before-chunk-write hook and the stored-document mark reset
become module functions used by the one remaining implementation.

Refs zarr-developers#4217

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
docs: note the AsyncArray delegation in the changelog fragment
ea0c01b 
Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
feat(array): make ArrayIO public and reachable from every array handle
63c6ed7 
ArrayIO and AsyncArrayIO move out of zarr.experimental into the zarr
namespace, next to Array and AsyncArray, which are built on them: the
wrapped layer is reachable as Array.io / AsyncArray.io, the inverse of
to_array(). The zarr.experimental.array_io shim is removed and the
layer is documented on the array API page.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b d-v-b changed the title feat(experimental): add ArrayIO and AsyncArrayIO, an explicit-IO array layer feat(array): add ArrayIO and AsyncArrayIO, the explicit-IO layer under Array and AsyncArray Oct 29, 2026
@d-v-b
feat(array): hold the ArrayIO document as a zarr-metadata model
adf48d2 
zarr-python now depends on zarr-metadata (>=0.6.0): AsyncArrayIO holds
one source — the document, a zarr_metadata model, when constructed by
open; the resolved metadata when constructed by build or by AsyncArray
— and derives the other form, the chunk grid, the chunk key encoding
and the codec pipeline as caches. Document operations (store_metadata,
erase_metadata, erase_chunk, erase_chunks, verify, equality) never
resolve codec or data type names, so an array whose codecs this
environment lacks still opens for document work and refuses only when
the data path is built; a document zarr can only read as repaired is
refused by the strict model read, with AsyncArray.open remaining the
repairing reader. Grid derivation goes through the same grid-metadata
objects ChunkGrid.from_metadata dispatches on, so both derivations lay
out chunks identically, and model JSON is normalized through JSON text
because the model returns read-only forms.

Resolved-seeded instances keep deriving their grid, chunk keys and
equality from the resolved metadata: zarr-python writes some documents
the strict model refuses (Zarr format 2 generic time units), and the
array API's behavior must not depend on the model accepting them.

Refs zarr-developers#4217

Assisted-by: ClaudeCode:claude-fable-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b