feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope

packages/zarr-metadata/README.md

@@ -41,7 +41,7 @@ closely model the content of the Zarr specifications, such as:
   `TypedDict` with its rules, and the scope that decides which definitions
   read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
   definition and scope of their format, such as `BYTES_CODEC` and
-  `CORE_AND_EXTENSIONS`.
+  `CORE_AND_EXTENSIONS_CONTEXT`.
 - **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
   `check` checks a JSON value against any `TypedDict` and reports every
   problem with its location.
@@ -104,7 +104,9 @@ of context-free consistency rules, e.g.:
 In a v3 document, validation also checks each extension point: the data
 type, the chunk grid, the chunk key encoding, each codec, and each storage
 transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
+a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
+`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
+specification registers.
 If the definition rejects a configuration, the reader rejects it too,
 and a configuration key the definition does not declare is reported as
 `unknown_key`. If no definition in the scope claims a name, the reader
@@ -232,13 +234,13 @@ each field:
 ```python test="true" session="readme"
 from zarr_metadata.model import read_array_metadata_v3
 from zarr_metadata.definition import canonical_of, with_problems
-from zarr_metadata.v3.definition import CORE
+from zarr_metadata.v3.definition import CORE_CONTEXT
 
 reading = read_array_metadata_v3(raw)
 beyond_core = [
     loc
     for loc, field in reading.fields()
-    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+    if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None
 ]
 simplest = {
     loc: canonical_of(field, problems)
@@ -283,15 +285,15 @@ scope claims is an `UnclaimedField` too, also when it sits inside another
 field, such as a codec inside a shard. A model's `unclaimed_fields` lists
 every field nothing in scope claims, at any depth, by its location, and
 leaves out the members that waive `must_understand`: a reader can open
-the node when it is empty. `CORE` holds
+the node when it is empty. `CORE_CONTEXT` holds
 `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
 `consolidated_metadata` as the envelope the spec fixes. A group's
 `consolidated()` reads the documents that member lists, in the group's
 own scope.
 
 A v3 model consists of its document and the scope it was read in.
 `ZarrV3ArrayMetadata(document, context=None)` reads the document in the
-scope (`CORE_AND_EXTENSIONS` when none is given) and raises
+scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
 `MetadataValidationError` with every problem, so an invalid model is never
 built. `to_json` returns the document as it was written, and
 `to_key_value` writes that same document to store bytes.
@@ -322,7 +324,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
 `read_consolidated_metadata_v3`, which returns problems instead of
 raising them. A document with a problem then fails only when it is read.
 Pass `context=group.context` to read the documents in the group's own
-scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
+scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
 
 To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
 pass it to the group's `update`. The group reads the listed models in its
@@ -332,7 +334,7 @@ models as entries. Such an entry is accepted when its claims refine into
 the model's scope, and refused at its path otherwise. To consolidate
 children that were read in several scopes, consolidate them in the
 `union` of those scopes. Every reader takes `context=None` for the
-default scope, `CORE_AND_EXTENSIONS`.
+default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
 
 Two models are equal when they represent the same document, even if each
 is spelled differently. Values the package interprets (each field, and the

packages/zarr-metadata/changes/426.feature.md

@@ -0,0 +1 @@
+Add the `ZARR_PYTHON_CONTEXT` scope, which reads what zarr-python writes: `CORE_AND_EXTENSIONS_CONTEXT` plus zarr-python's own data types `null_terminated_bytes` (NumPy's `S`) and `variable_length_bytes`, which no specification registers. `CORE_AND_EXTENSIONS_CONTEXT` now also reads the `fixed_length_utf32` data type (NumPy's `U`) and the `vlen-utf8` and `vlen-bytes` codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.

packages/zarr-metadata/changes/426.removal.md

@@ -0,0 +1 @@
+Rename the v3 scopes `CORE` and `CORE_AND_EXTENSIONS` to `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`, so that a scope's name says it is a `ZarrV3Context`. Replace each import and use of the old names with the new ones.

packages/zarr-metadata/docs/api/index.md

@@ -35,7 +35,7 @@ The package is organized to mirror the structure of the Zarr specifications:
   scope type `Context`. Its module docstring is the guide
 - [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds
   of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec
-  pipeline types, and the v3 scopes `CORE` and `CORE_AND_EXTENSIONS`
+  pipeline types, and the v3 scopes `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`
 
 The document types, models, and spec vocabulary — including the store keys —
 are re-exported at the top level, so

packages/zarr-metadata/docs/api/v3/codec.md

@@ -22,4 +22,8 @@ title: codec
 
 ::: zarr_metadata.v3.codec.transpose
 
+::: zarr_metadata.v3.codec.vlen_bytes
+
+::: zarr_metadata.v3.codec.vlen_utf8
+
 ::: zarr_metadata.v3.codec.zstd

packages/zarr-metadata/docs/api/v3/data_type.md

@@ -40,8 +40,14 @@ title: data_type
 
 ::: zarr_metadata.v3.data_type.string
 
+::: zarr_metadata.v3.data_type.fixed_length_utf32
+
 ::: zarr_metadata.v3.data_type.numpy_datetime64
 
 ::: zarr_metadata.v3.data_type.numpy_timedelta64
 
 ::: zarr_metadata.v3.data_type.struct
+
+::: zarr_metadata.v3.data_type.null_terminated_bytes
+
+::: zarr_metadata.v3.data_type.variable_length_bytes

packages/zarr-metadata/docs/index.md

@@ -39,7 +39,7 @@ closely model the content of the Zarr specifications, such as:
   `TypedDict` with its rules, and the scope that decides which definitions
   read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
   definition and scope of their format, such as `BYTES_CODEC` and
-  `CORE_AND_EXTENSIONS`.
+  `CORE_AND_EXTENSIONS_CONTEXT`.
 - **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)):
   `check` checks a JSON value against any `TypedDict` and reports every
   problem with its location.
@@ -102,7 +102,9 @@ of context-free consistency rules, e.g.:
 In a v3 document, validation also checks each extension point: the data
 type, the chunk grid, the chunk key encoding, each codec, and each storage
 transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
+a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
+`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
+specification registers.
 If the definition rejects a configuration, the reader rejects it too,
 and a configuration key the definition does not declare is reported as
 `unknown_key`. If no definition in the scope claims a name, the reader
@@ -234,13 +236,13 @@ each field:
 ```python test="true" session="index"
 from zarr_metadata.model import read_array_metadata_v3
 from zarr_metadata.definition import canonical_of, with_problems
-from zarr_metadata.v3.definition import CORE
+from zarr_metadata.v3.definition import CORE_CONTEXT
 
 reading = read_array_metadata_v3(raw)
 beyond_core = [
     loc
     for loc, field in reading.fields()
-    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+    if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None
 ]
 simplest = {
     loc: canonical_of(field, problems)
@@ -285,15 +287,15 @@ scope claims is an `UnclaimedField` too, also when it sits inside another
 field, such as a codec inside a shard. A model's `unclaimed_fields` lists
 every field nothing in scope claims, at any depth, by its location, and
 leaves out the members that waive `must_understand`: a reader can open
-the node when it is empty. `CORE` holds
+the node when it is empty. `CORE_CONTEXT` holds
 `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
 `consolidated_metadata` as the envelope the spec fixes. A group's
 `consolidated()` reads the documents that member lists, in the group's
 own scope.
 
 A v3 model consists of its document and the scope it was read in.
 `ZarrV3ArrayMetadata(document, context=None)` reads the document in the
-scope (`CORE_AND_EXTENSIONS` when none is given) and raises
+scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
 `MetadataValidationError` with every problem, so an invalid model is never
 built. `to_json` returns the document as it was written, and
 `to_key_value` writes that same document to store bytes.
@@ -324,7 +326,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
 `read_consolidated_metadata_v3`, which returns problems instead of
 raising them. A document with a problem then fails only when it is read.
 Pass `context=group.context` to read the documents in the group's own
-scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
+scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
 
 To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
 pass it to the group's `update`. The group reads the listed models in its
@@ -334,7 +336,7 @@ models as entries. Such an entry is accepted when its claims refine into
 the model's scope, and refused at its path otherwise. To consolidate
 children that were read in several scopes, consolidate them in the
 `union` of those scopes. Every reader takes `context=None` for the
-default scope, `CORE_AND_EXTENSIONS`.
+default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
 
 Two models are equal when they represent the same document, even if each
 is spelled differently. Values the package interprets (each field, and the

packages/zarr-metadata/docs/parsing-validation.md

@@ -161,7 +161,7 @@ This procedure yields one of two values:
 
 `zarr-metadata` does not run each stage as a separate function. Instead, a single read of the document collects the problems from every stage, so a caller gets the full list of problems at once.
 
-The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE` holds only the definitions from the Zarr V3 spec.
+The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS_CONTEXT`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE_CONTEXT` holds only the definitions from the Zarr V3 spec. `ZARR_PYTHON_CONTEXT` adds to `CORE_AND_EXTENSIONS_CONTEXT` the data types that zarr-python writes but no specification registers (`null_terminated_bytes` for NumPy's `S`, and `variable_length_bytes`), so a strict check of what zarr-python writes passes it.
 
 `read_array_metadata_v3` reads a JSON document and returns a "reading". `reading.problems` is a tuple of the problems, and `reading.metadata` is the validated model, or `None` if there are problems. Each problem has a `loc`, which is its position in the document as a tuple, a `kind`, like `missing_key` or `invalid_value`, and a human-readable `message`:
 
@@ -193,10 +193,10 @@ To read one field on its own, such as a codec, use `resolve` from `zarr_metadata
 
 ```python test="true" session="parsing-validation"
 from zarr_metadata.definition import resolve
-from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
 field, problems = resolve(
-    {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS
+    {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT
 )
 for problem in problems:
     print(problem.loc, problem.kind)
@@ -214,17 +214,17 @@ To make an unknown data type or codec known, add its definition to the scope's s
 
 ```python test="true" session="parsing-validation"
 from zarr_metadata.definition import EmptyConfiguration
-from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
 # The definition of the codec that was unknown: here, a codec named
 # `acme.codec` that takes no configuration.
 MY_CODEC = CodecDefinition(
     name="acme.codec", configuration=EmptyConfiguration, kind="bytes_bytes", size="dynamic"
 )
-context = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), MY_CODEC])
+context = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), MY_CODEC])
 reading = read_array_metadata_v3(raw, context=context)
 ```
 
-`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS.update(codecs={**CORE_AND_EXTENSIONS.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
+`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS_CONTEXT.update(codecs={**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
 
 [^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the field `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`.

packages/zarr-metadata/src/zarr_metadata/__init__.py

@@ -136,6 +136,16 @@
     TransposeCodecMetadata,
     TransposeCodecName,
 )
+from zarr_metadata.v3.codec.vlen_bytes import (
+    VLEN_BYTES_CODEC_NAME,
+    VlenBytesCodecMetadata,
+    VlenBytesCodecName,
+)
+from zarr_metadata.v3.codec.vlen_utf8 import (
+    VLEN_UTF8_CODEC_NAME,
+    VlenUtf8CodecMetadata,
+    VlenUtf8CodecName,
+)
 from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME, ZstdCodecMetadata, ZstdCodecName
 from zarr_metadata.v3.consolidated import (
     ZARR_V3_CONSOLIDATED_METADATA_KEY,
@@ -162,6 +172,11 @@
     Complex128DataTypeName,
     Complex128FillValue,
 )
+from zarr_metadata.v3.data_type.fixed_length_utf32 import (
+    FIXED_LENGTH_UTF32_DATA_TYPE_NAME,
+    FixedLengthUtf32DataTypeName,
+    FixedLengthUtf32FillValue,
+)
 from zarr_metadata.v3.data_type.float16 import (
     FLOAT16_DATA_TYPE_NAME,
     Float16DataTypeName,
@@ -265,6 +280,7 @@
     "DEFAULT_CHUNK_KEY_ENCODING_NAME",
     "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR",
     "ENDIANNESS",
+    "FIXED_LENGTH_UTF32_DATA_TYPE_NAME",
     "FLOAT16_DATA_TYPE_NAME",
     "FLOAT32_DATA_TYPE_NAME",
     "FLOAT64_DATA_TYPE_NAME",
@@ -291,6 +307,8 @@
     "UNSET",
     "V2_CHUNK_KEY_ENCODING_NAME",
     "V2_CHUNK_KEY_ENCODING_SEPARATOR",
+    "VLEN_BYTES_CODEC_NAME",
+    "VLEN_UTF8_CODEC_NAME",
     "ZARR_V2_ARRAY_DIMENSION_SEPARATOR",
     "ZARR_V2_ARRAY_METADATA_STORE_KEY",
     "ZARR_V2_ARRAY_ORDER",
@@ -325,6 +343,8 @@
     "DefaultChunkKeyEncodingName",
     "DefaultChunkKeyEncodingSeparator",
     "Endianness",
+    "FixedLengthUtf32DataTypeName",
+    "FixedLengthUtf32FillValue",
     "Float16DataTypeName",
     "Float16FillValue",
     "Float32DataTypeName",
@@ -378,6 +398,10 @@
     "V2ChunkKeyEncodingName",
     "V2ChunkKeyEncodingSeparator",
     "ValidationProblem",
+    "VlenBytesCodecMetadata",
+    "VlenBytesCodecName",
+    "VlenUtf8CodecMetadata",
+    "VlenUtf8CodecName",
     "ZarrV2ArrayDimensionSeparator",
     "ZarrV2ArrayMetadata",
     "ZarrV2ArrayMetadataJSON",

packages/zarr-metadata/src/zarr_metadata/definition/__init__.py

@@ -40,7 +40,7 @@
    configuration, checks the envelope of each nested field (an extra
    member, a `must_understand` of `false`), and then runs the rules, for
    one configuration: `GZIP_CODEC.read_configuration({"level": 12})`.
-3. `resolve(field, CodecDefinition, CORE_AND_EXTENSIONS)` reads a whole
+3. `resolve(field, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)` reads a whole
    field in a scope. It checks the envelope, maps the name to a
    definition, checks the configuration, and reads each nested field the
    same way. It returns the result of the reading and every problem. The
@@ -70,17 +70,17 @@
 
     from zarr_metadata.definition import resolve
     from zarr_metadata.v3.codec.gzip import GZIP_CODEC
-    from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+    from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
     resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}},
-                                 CodecDefinition, CORE_AND_EXTENSIONS)
+                                 CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)
     resolved                # RefusedField(..., definition=CodecDefinition(name='gzip'), ...)
     problems[0].loc         # ('configuration', 'level')
     problems[0].input       # 12
     dict(problems[0].ctx)   # {'ge': 0, 'le': 9}
 
     resolved, problems = resolve({"name": "gzip", "configuration": {"level": 5}},
-                                 CodecDefinition, CORE_AND_EXTENSIONS)
+                                 CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)
     resolved.definition is GZIP_CODEC        # True
     resolved.configuration                   # {'level': 5}
 
@@ -124,7 +124,7 @@
 
     from zarr_metadata import ValidationProblem
     from zarr_metadata.definition import Nested
-    from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+    from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
 
     class AcmeLz4Configuration(TypedDict, closed=True):
@@ -149,7 +149,7 @@ def acme_lz4_rules(
         size="dynamic",
         rules=acme_lz4_rules,
     )
-    CONTEXT = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), ACME_LZ4])
+    CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4])
 
 The TypedDict declares how keys it does not declare are handled. With
 `closed=True`, such a key is a problem, as above. With `extra_items=`,

packages/zarr-metadata/src/zarr_metadata/definition/_definition.py

@@ -1365,7 +1365,7 @@ def resolve(
     else raises `TypeError`.
     """
     if not _is_scope(context):
-        msg = f"a field is read in a scope, such as CORE_AND_EXTENSIONS, got {described(context)}"
+        msg = f"a field is read in a scope, such as CORE_AND_EXTENSIONS_CONTEXT, got {described(context)}"
         raise TypeError(msg)
     asked = as_kind(kind)
     _check_format(asked, context)

packages/zarr-metadata/src/zarr_metadata/json_schema.py

@@ -30,7 +30,7 @@
 
 **A whole `zarr.json`.** `node_metadata_json_schema_v3(context=None)`
 writes a whole v3 `zarr.json`, array or group, as the scope reads it
-(`CORE_AND_EXTENSIONS` when none is given). Its fields are written
+(`CORE_AND_EXTENSIONS_CONTEXT` when none is given). Its fields are written
 as the scope reads them. For each definition, the schema accepts its
 field (its name, its configuration as its TypedDict declares it,
 including bounds, a `must_understand` of `true`, and its bare name when