diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index 3c5513d1dd..12583e0c96 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -41,7 +41,7 @@ closely model the content of the Zarr specifications, such as: `TypedDict` with its rules, and the scope that decides which definitions read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS`. + `CORE_AND_EXTENSIONS_CONTEXT`. - **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)): `check` checks a JSON value against any `TypedDict` and reports every problem with its location. @@ -104,7 +104,9 @@ of context-free consistency rules, e.g.: In a v3 document, validation also checks each extension point: the data type, the chunk grid, the chunk key encoding, each codec, and each storage transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed. +a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. +`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no +specification registers. If the definition rejects a configuration, the reader rejects it too, and a configuration key the definition does not declare is reported as `unknown_key`. If no definition in the scope claims a name, the reader @@ -232,13 +234,13 @@ each field: ```python test="true" session="readme" from zarr_metadata.model import read_array_metadata_v3 from zarr_metadata.definition import canonical_of, with_problems -from zarr_metadata.v3.definition import CORE +from zarr_metadata.v3.definition import CORE_CONTEXT reading = read_array_metadata_v3(raw) beyond_core = [ loc for loc, field in reading.fields() - if field.name is not None and CORE.claimant(field.read_as, field.name) is None + if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None ] simplest = { loc: canonical_of(field, problems) @@ -283,7 +285,7 @@ scope claims is an `UnclaimedField` too, also when it sits inside another field, such as a codec inside a shard. A model's `unclaimed_fields` lists every field nothing in scope claims, at any depth, by its location, and leaves out the members that waive `must_understand`: a reader can open -the node when it is empty. `CORE` holds +the node when it is empty. `CORE_CONTEXT` holds `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's `consolidated_metadata` as the envelope the spec fixes. A group's `consolidated()` reads the documents that member lists, in the group's @@ -291,7 +293,7 @@ own scope. A v3 model consists of its document and the scope it was read in. `ZarrV3ArrayMetadata(document, context=None)` reads the document in the -scope (`CORE_AND_EXTENSIONS` when none is given) and raises +scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises `MetadataValidationError` with every problem, so an invalid model is never built. `to_json` returns the document as it was written, and `to_key_value` writes that same document to store bytes. @@ -322,7 +324,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with `read_consolidated_metadata_v3`, which returns problems instead of raising them. A document with a problem then fails only when it is read. Pass `context=group.context` to read the documents in the group's own -scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`. +scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`. To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and pass it to the group's `update`. The group reads the listed models in its @@ -332,7 +334,7 @@ models as entries. Such an entry is accepted when its claims refine into the model's scope, and refused at its path otherwise. To consolidate children that were read in several scopes, consolidate them in the `union` of those scopes. Every reader takes `context=None` for the -default scope, `CORE_AND_EXTENSIONS`. +default scope, `CORE_AND_EXTENSIONS_CONTEXT`. Two models are equal when they represent the same document, even if each is spelled differently. Values the package interprets (each field, and the diff --git a/packages/zarr-metadata/changes/426.feature.md b/packages/zarr-metadata/changes/426.feature.md new file mode 100644 index 0000000000..c11c8edd20 --- /dev/null +++ b/packages/zarr-metadata/changes/426.feature.md @@ -0,0 +1 @@ +Add the `ZARR_PYTHON_CONTEXT` scope, which reads what zarr-python writes: `CORE_AND_EXTENSIONS_CONTEXT` plus zarr-python's own data types `null_terminated_bytes` (NumPy's `S`) and `variable_length_bytes`, which no specification registers. `CORE_AND_EXTENSIONS_CONTEXT` now also reads the `fixed_length_utf32` data type (NumPy's `U`) and the `vlen-utf8` and `vlen-bytes` codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed. diff --git a/packages/zarr-metadata/changes/426.removal.md b/packages/zarr-metadata/changes/426.removal.md new file mode 100644 index 0000000000..ab932f69b3 --- /dev/null +++ b/packages/zarr-metadata/changes/426.removal.md @@ -0,0 +1 @@ +Rename the v3 scopes `CORE` and `CORE_AND_EXTENSIONS` to `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`, so that a scope's name says it is a `ZarrV3Context`. Replace each import and use of the old names with the new ones. diff --git a/packages/zarr-metadata/docs/api/index.md b/packages/zarr-metadata/docs/api/index.md index 4a23c5c2fb..50c17f2dfb 100644 --- a/packages/zarr-metadata/docs/api/index.md +++ b/packages/zarr-metadata/docs/api/index.md @@ -35,7 +35,7 @@ The package is organized to mirror the structure of the Zarr specifications: scope type `Context`. Its module docstring is the guide - [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec - pipeline types, and the v3 scopes `CORE` and `CORE_AND_EXTENSIONS` + pipeline types, and the v3 scopes `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT` The document types, models, and spec vocabulary — including the store keys — are re-exported at the top level, so diff --git a/packages/zarr-metadata/docs/api/v3/codec.md b/packages/zarr-metadata/docs/api/v3/codec.md index cb96d2c7d5..ae5171acd0 100644 --- a/packages/zarr-metadata/docs/api/v3/codec.md +++ b/packages/zarr-metadata/docs/api/v3/codec.md @@ -22,4 +22,8 @@ title: codec ::: zarr_metadata.v3.codec.transpose +::: zarr_metadata.v3.codec.vlen_bytes + +::: zarr_metadata.v3.codec.vlen_utf8 + ::: zarr_metadata.v3.codec.zstd diff --git a/packages/zarr-metadata/docs/api/v3/data_type.md b/packages/zarr-metadata/docs/api/v3/data_type.md index 3f8444625a..5c63a71939 100644 --- a/packages/zarr-metadata/docs/api/v3/data_type.md +++ b/packages/zarr-metadata/docs/api/v3/data_type.md @@ -40,8 +40,14 @@ title: data_type ::: zarr_metadata.v3.data_type.string +::: zarr_metadata.v3.data_type.fixed_length_utf32 + ::: zarr_metadata.v3.data_type.numpy_datetime64 ::: zarr_metadata.v3.data_type.numpy_timedelta64 ::: zarr_metadata.v3.data_type.struct + +::: zarr_metadata.v3.data_type.null_terminated_bytes + +::: zarr_metadata.v3.data_type.variable_length_bytes diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index b3936156d0..6b238b7017 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -39,7 +39,7 @@ closely model the content of the Zarr specifications, such as: `TypedDict` with its rules, and the scope that decides which definitions read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS`. + `CORE_AND_EXTENSIONS_CONTEXT`. - **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)): `check` checks a JSON value against any `TypedDict` and reports every problem with its location. @@ -102,7 +102,9 @@ of context-free consistency rules, e.g.: In a v3 document, validation also checks each extension point: the data type, the chunk grid, the chunk key encoding, each codec, and each storage transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed. +a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. +`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no +specification registers. If the definition rejects a configuration, the reader rejects it too, and a configuration key the definition does not declare is reported as `unknown_key`. If no definition in the scope claims a name, the reader @@ -234,13 +236,13 @@ each field: ```python test="true" session="index" from zarr_metadata.model import read_array_metadata_v3 from zarr_metadata.definition import canonical_of, with_problems -from zarr_metadata.v3.definition import CORE +from zarr_metadata.v3.definition import CORE_CONTEXT reading = read_array_metadata_v3(raw) beyond_core = [ loc for loc, field in reading.fields() - if field.name is not None and CORE.claimant(field.read_as, field.name) is None + if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None ] simplest = { loc: canonical_of(field, problems) @@ -285,7 +287,7 @@ scope claims is an `UnclaimedField` too, also when it sits inside another field, such as a codec inside a shard. A model's `unclaimed_fields` lists every field nothing in scope claims, at any depth, by its location, and leaves out the members that waive `must_understand`: a reader can open -the node when it is empty. `CORE` holds +the node when it is empty. `CORE_CONTEXT` holds `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's `consolidated_metadata` as the envelope the spec fixes. A group's `consolidated()` reads the documents that member lists, in the group's @@ -293,7 +295,7 @@ own scope. A v3 model consists of its document and the scope it was read in. `ZarrV3ArrayMetadata(document, context=None)` reads the document in the -scope (`CORE_AND_EXTENSIONS` when none is given) and raises +scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises `MetadataValidationError` with every problem, so an invalid model is never built. `to_json` returns the document as it was written, and `to_key_value` writes that same document to store bytes. @@ -324,7 +326,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with `read_consolidated_metadata_v3`, which returns problems instead of raising them. A document with a problem then fails only when it is read. Pass `context=group.context` to read the documents in the group's own -scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`. +scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`. To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and pass it to the group's `update`. The group reads the listed models in its @@ -334,7 +336,7 @@ models as entries. Such an entry is accepted when its claims refine into the model's scope, and refused at its path otherwise. To consolidate children that were read in several scopes, consolidate them in the `union` of those scopes. Every reader takes `context=None` for the -default scope, `CORE_AND_EXTENSIONS`. +default scope, `CORE_AND_EXTENSIONS_CONTEXT`. Two models are equal when they represent the same document, even if each is spelled differently. Values the package interprets (each field, and the diff --git a/packages/zarr-metadata/docs/parsing-validation.md b/packages/zarr-metadata/docs/parsing-validation.md index adfe8e51c2..01c8b0946b 100644 --- a/packages/zarr-metadata/docs/parsing-validation.md +++ b/packages/zarr-metadata/docs/parsing-validation.md @@ -161,7 +161,7 @@ This procedure yields one of two values: `zarr-metadata` does not run each stage as a separate function. Instead, a single read of the document collects the problems from every stage, so a caller gets the full list of problems at once. -The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE` holds only the definitions from the Zarr V3 spec. +The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS_CONTEXT`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE_CONTEXT` holds only the definitions from the Zarr V3 spec. `ZARR_PYTHON_CONTEXT` adds to `CORE_AND_EXTENSIONS_CONTEXT` the data types that zarr-python writes but no specification registers (`null_terminated_bytes` for NumPy's `S`, and `variable_length_bytes`), so a strict check of what zarr-python writes passes it. `read_array_metadata_v3` reads a JSON document and returns a "reading". `reading.problems` is a tuple of the problems, and `reading.metadata` is the validated model, or `None` if there are problems. Each problem has a `loc`, which is its position in the document as a tuple, a `kind`, like `missing_key` or `invalid_value`, and a human-readable `message`: @@ -193,10 +193,10 @@ To read one field on its own, such as a codec, use `resolve` from `zarr_metadata ```python test="true" session="parsing-validation" from zarr_metadata.definition import resolve -from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition field, problems = resolve( - {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS + {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT ) for problem in problems: print(problem.loc, problem.kind) @@ -214,17 +214,17 @@ To make an unknown data type or codec known, add its definition to the scope's s ```python test="true" session="parsing-validation" from zarr_metadata.definition import EmptyConfiguration -from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition # The definition of the codec that was unknown: here, a codec named # `acme.codec` that takes no configuration. MY_CODEC = CodecDefinition( name="acme.codec", configuration=EmptyConfiguration, kind="bytes_bytes", size="dynamic" ) -context = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), MY_CODEC]) +context = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), MY_CODEC]) reading = read_array_metadata_v3(raw, context=context) ``` -`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS.update(codecs={**CORE_AND_EXTENSIONS.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own. +`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS_CONTEXT.update(codecs={**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own. [^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the field `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`. diff --git a/packages/zarr-metadata/src/zarr_metadata/__init__.py b/packages/zarr-metadata/src/zarr_metadata/__init__.py index bc57d90fd0..5e8b1ffcba 100644 --- a/packages/zarr-metadata/src/zarr_metadata/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/__init__.py @@ -136,6 +136,16 @@ TransposeCodecMetadata, TransposeCodecName, ) +from zarr_metadata.v3.codec.vlen_bytes import ( + VLEN_BYTES_CODEC_NAME, + VlenBytesCodecMetadata, + VlenBytesCodecName, +) +from zarr_metadata.v3.codec.vlen_utf8 import ( + VLEN_UTF8_CODEC_NAME, + VlenUtf8CodecMetadata, + VlenUtf8CodecName, +) from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME, ZstdCodecMetadata, ZstdCodecName from zarr_metadata.v3.consolidated import ( ZARR_V3_CONSOLIDATED_METADATA_KEY, @@ -162,6 +172,11 @@ Complex128DataTypeName, Complex128FillValue, ) +from zarr_metadata.v3.data_type.fixed_length_utf32 import ( + FIXED_LENGTH_UTF32_DATA_TYPE_NAME, + FixedLengthUtf32DataTypeName, + FixedLengthUtf32FillValue, +) from zarr_metadata.v3.data_type.float16 import ( FLOAT16_DATA_TYPE_NAME, Float16DataTypeName, @@ -265,6 +280,7 @@ "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", "ENDIANNESS", + "FIXED_LENGTH_UTF32_DATA_TYPE_NAME", "FLOAT16_DATA_TYPE_NAME", "FLOAT32_DATA_TYPE_NAME", "FLOAT64_DATA_TYPE_NAME", @@ -291,6 +307,8 @@ "UNSET", "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", + "VLEN_BYTES_CODEC_NAME", + "VLEN_UTF8_CODEC_NAME", "ZARR_V2_ARRAY_DIMENSION_SEPARATOR", "ZARR_V2_ARRAY_METADATA_STORE_KEY", "ZARR_V2_ARRAY_ORDER", @@ -325,6 +343,8 @@ "DefaultChunkKeyEncodingName", "DefaultChunkKeyEncodingSeparator", "Endianness", + "FixedLengthUtf32DataTypeName", + "FixedLengthUtf32FillValue", "Float16DataTypeName", "Float16FillValue", "Float32DataTypeName", @@ -378,6 +398,10 @@ "V2ChunkKeyEncodingName", "V2ChunkKeyEncodingSeparator", "ValidationProblem", + "VlenBytesCodecMetadata", + "VlenBytesCodecName", + "VlenUtf8CodecMetadata", + "VlenUtf8CodecName", "ZarrV2ArrayDimensionSeparator", "ZarrV2ArrayMetadata", "ZarrV2ArrayMetadataJSON", diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py index 6dd9c86173..782807f4a8 100644 --- a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py @@ -40,7 +40,7 @@ configuration, checks the envelope of each nested field (an extra member, a `must_understand` of `false`), and then runs the rules, for one configuration: `GZIP_CODEC.read_configuration({"level": 12})`. -3. `resolve(field, CodecDefinition, CORE_AND_EXTENSIONS)` reads a whole +3. `resolve(field, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)` reads a whole field in a scope. It checks the envelope, maps the name to a definition, checks the configuration, and reads each nested field the same way. It returns the result of the reading and every problem. The @@ -70,17 +70,17 @@ from zarr_metadata.definition import resolve from zarr_metadata.v3.codec.gzip import GZIP_CODEC - from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition + from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}}, - CodecDefinition, CORE_AND_EXTENSIONS) + CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) resolved # RefusedField(..., definition=CodecDefinition(name='gzip'), ...) problems[0].loc # ('configuration', 'level') problems[0].input # 12 dict(problems[0].ctx) # {'ge': 0, 'le': 9} resolved, problems = resolve({"name": "gzip", "configuration": {"level": 5}}, - CodecDefinition, CORE_AND_EXTENSIONS) + CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) resolved.definition is GZIP_CODEC # True resolved.configuration # {'level': 5} @@ -124,7 +124,7 @@ from zarr_metadata import ValidationProblem from zarr_metadata.definition import Nested - from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition + from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition class AcmeLz4Configuration(TypedDict, closed=True): @@ -149,7 +149,7 @@ def acme_lz4_rules( size="dynamic", rules=acme_lz4_rules, ) - CONTEXT = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), ACME_LZ4]) + CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4]) The TypedDict declares how keys it does not declare are handled. With `closed=True`, such a key is a problem, as above. With `extra_items=`, diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py b/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py index 3e7ec67ef9..c6ce82f2a6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py +++ b/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py @@ -1365,7 +1365,7 @@ def resolve( else raises `TypeError`. """ if not _is_scope(context): - msg = f"a field is read in a scope, such as CORE_AND_EXTENSIONS, got {described(context)}" + msg = f"a field is read in a scope, such as CORE_AND_EXTENSIONS_CONTEXT, got {described(context)}" raise TypeError(msg) asked = as_kind(kind) _check_format(asked, context) diff --git a/packages/zarr-metadata/src/zarr_metadata/json_schema.py b/packages/zarr-metadata/src/zarr_metadata/json_schema.py index 9b7443a8f4..ea80ac0242 100644 --- a/packages/zarr-metadata/src/zarr_metadata/json_schema.py +++ b/packages/zarr-metadata/src/zarr_metadata/json_schema.py @@ -30,7 +30,7 @@ **A whole `zarr.json`.** `node_metadata_json_schema_v3(context=None)` writes a whole v3 `zarr.json`, array or group, as the scope reads it -(`CORE_AND_EXTENSIONS` when none is given). Its fields are written +(`CORE_AND_EXTENSIONS_CONTEXT` when none is given). Its fields are written as the scope reads them. For each definition, the schema accepts its field (its name, its configuration as its TypedDict declares it, including bounds, a `must_understand` of `true`, and its bare name when diff --git a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py index 98776c7c5d..ab6006ca2a 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py @@ -9,7 +9,7 @@ `read_consolidated_metadata_v2`. A reader checks a document's JSON structure. In a v3 document it also reads each extension point (codecs, chunk grids, data types, ...) through the definition that claims its -name in a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` +name in a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. A v3 reader checks the fill value against the data type the document names, the chunk grid against the shape, and the codecs as a pipeline, each codec against the chunk it receives. Each reader does one diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py index 53d12254bc..58fb7e30c5 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py @@ -62,7 +62,7 @@ # A runtime import: `get_type_hints` resolves the public signatures. from zarr_metadata.v3._context import ZarrV3Context # noqa: TC001 -from zarr_metadata.v3._core import CORE_AND_EXTENSIONS +from zarr_metadata.v3._core import CORE_AND_EXTENSIONS_CONTEXT from zarr_metadata.v3._definition import ( ArrayExtraFieldDefinition, Chunk, @@ -499,7 +499,7 @@ class ZarrV3ArrayMetadata(Keyed): node_type: Final[Literal["array"]] = "array" # noqa: PYI064 - as above def __init__(self, document: object, context: ZarrV3Context | None = None) -> None: - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) reading, members = read_array_v3(document, context) if members is None: raise MetadataValidationError(reading.problems) @@ -696,7 +696,7 @@ def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3ArrayMetad in `context`. The existing reading is reused when `context` reads every claim identically. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) if disagreements(context, claims_of(self._reading.fields())).agrees: return self._of(self._document, context, self._reading, self._members) return type(self)(self._document, context=context) @@ -713,7 +713,7 @@ def refined_in(self, context: ZarrV3Context | None = None) -> ZarrV3ArrayMetadat gain can expose a problem. `with_context` reads the document in any scope. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) found = disagreements(context, claims_of(self._reading.fields())) if len(found.conflicts) != 0: raise ScopeConflictError(located_conflicts(self._reading.fields(), found.conflicts)) @@ -811,7 +811,7 @@ def from_key_value( are not JSON, or the document is not valid. Raises `TypeError` for a scope of another format before any bytes are decoded. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) return cls(load_store_json(mapping, ZARR_V3_ARRAY_METADATA_STORE_KEY), context=context) @@ -888,7 +888,7 @@ def read_array_metadata_v3( spec's fields), iterate over the reading's `fields()`. A value that is not an object has no fields. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) reading, members = read_array_v3(value, context) if members is None: return reading diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py index daae22cdfe..63a66891a6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py @@ -79,7 +79,7 @@ # A runtime import: `get_type_hints` resolves the public signatures. from zarr_metadata.v3._context import ZarrV3Context # noqa: TC001 -from zarr_metadata.v3._core import CORE_AND_EXTENSIONS +from zarr_metadata.v3._core import CORE_AND_EXTENSIONS_CONTEXT from zarr_metadata.v3._definition import GroupExtraFieldDefinition from zarr_metadata.v3._hierarchy import hierarchy_problems, path_faults, said from zarr_metadata.v3.consolidated import ( @@ -168,7 +168,7 @@ class ZarrV3GroupMetadata(Keyed): the document as written, refined, and `context` is the scope. `attributes` and `consolidated_metadata` are views of the values the read refined. Each member the spec does not define is an extra field, - in `extra_fields`, as the scope read it. `CORE` reads a + in `extra_fields`, as the scope read it. `CORE_CONTEXT` reads a `consolidated_metadata` member as the envelope the core spec fixes, and no definition reads the documents the member lists: they follow a convention of the reference implementation, not the core spec. To @@ -185,7 +185,7 @@ class ZarrV3GroupMetadata(Keyed): node_type: Final[Literal["group"]] = "group" # noqa: PYI064 - as above def __init__(self, document: object, context: ZarrV3Context | None = None) -> None: - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) reading, members = read_group_v3(document, context) if members is None or len(reading.problems) != 0: raise MetadataValidationError(reading.problems) @@ -280,7 +280,7 @@ def attributes(self) -> Mapping[str, JSONValue]: def extra_fields( self, ) -> Mapping[str, AcceptedField[GroupExtraFieldDefinition[Any]] | UnclaimedField]: - """Each member the spec does not define, by key, as the scope read it: an `AcceptedField` when a definition in scope claims the key (as `CORE` claims `consolidated_metadata`), and an `UnclaimedField` otherwise.""" + """Each member the spec does not define, by key, as the scope read it: an `AcceptedField` when a definition in scope claims the key (as `CORE_CONTEXT` claims `consolidated_metadata`), and an `UnclaimedField` otherwise.""" return self._shown[3] @property @@ -364,7 +364,7 @@ def update(self, **members: Unpack[ZarrV3GroupMetadataUpdate]) -> ZarrV3GroupMet def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3GroupMetadata: """Return this document read in `context`, whatever that changes. Raises `MetadataValidationError` when the document has a problem in `context`. The existing reading is reused when `context` reads every claim identically.""" - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) if disagreements(context, claims_of(self._reading.fields())).agrees: return self._of(self._document, context, self._reading, self._members) return type(self)(self._document, context=context) @@ -381,7 +381,7 @@ def refined_in(self, context: ZarrV3Context | None = None) -> ZarrV3GroupMetadat in `context`, and `ZarrV3ConsolidatedMetadata.refined_in` checks that they read the same there. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) found = disagreements(context, claims_of(self._reading.fields())) if len(found.conflicts) != 0: raise ScopeConflictError(located_conflicts(self._reading.fields(), found.conflicts)) @@ -408,7 +408,7 @@ def create_default( **overrides: Unpack[ZarrV3GroupMetadataUpdate], ) -> ZarrV3GroupMetadata: """Return a group with no attributes, with any document members in `overrides` applied, read in `context`. `overrides` takes what `update` takes, including `UNSET` to drop a member and a `ZarrV3ConsolidatedMetadata` as `consolidated_metadata`. Raises `MetadataValidationError` when the resulting document has a problem.""" - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) changes = member_changes( overrides, ZarrV3GroupMetadataUpdate, GroupExtraFieldDefinition.spec_members ) @@ -425,7 +425,7 @@ def from_json( Raises `MetadataValidationError` with every problem the read finds. A `consolidated_metadata` of `null`, which zarr-python 3.0 and 3.1 wrote, is a value the document wrote that is not an object. In a - scope that claims the member, as `CORE` does, this is a problem, + scope that claims the member, as `CORE_CONTEXT` does, this is a problem, because the spec requires an object. Use `read_repaired_node_metadata_v3` to read such a store. A member the spec does not define is kept in `extra_fields`. @@ -442,7 +442,7 @@ def from_key_value( are not JSON, or the document is not valid. Raises `TypeError` for a scope of another format before any bytes are decoded. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) return cls(load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY), context=context) @@ -494,7 +494,7 @@ class ZarrV3ConsolidatedMetadata(Keyed): must_understand: Final = False def __init__(self, member: object, context: ZarrV3Context | None = None) -> None: - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) # The member is always under a group's key, even when read on its # own, so nesting depth is counted from there, as in the group. readings, members, document, problems = _read_consolidated_v3( @@ -560,7 +560,7 @@ def __reduce__(self) -> tuple[type[ZarrV3ConsolidatedMetadata], tuple[object, Za def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3ConsolidatedMetadata: """Return this member with every document read in `context`, whatever that changes. Raises `MetadataValidationError` when a document has a problem in `context`. The existing readings are reused when `context` reads every claim of every document identically.""" - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) if all( disagreements(context, claims_of(node.reading.fields())).agrees for node in self._metadata.values() @@ -576,7 +576,7 @@ def refined_in(self, context: ZarrV3Context | None = None) -> ZarrV3Consolidated document's path. Raises `MetadataValidationError` when a gain exposes a problem. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) conflicts = [ dataclasses.replace( conflict, loc=("metadata", path, *(() if conflict.loc is None else conflict.loc)) @@ -680,8 +680,8 @@ def __reduce__(self) -> str | tuple[object, ...]: def read_consolidated_metadata_v3( value: object, *, context: ZarrV3Context | None = None ) -> ZarrV3ConsolidatedMetadataReading: - """Read `value`, a group's `consolidated_metadata` member, in `context` (`CORE_AND_EXTENSIONS` when none is given), whatever it holds: every problem the read finds, and the model when there are none. This is the form of `ZarrV3ConsolidatedMetadata(value, context)` that returns problems instead of raising them.""" - context = scoped(context, CORE_AND_EXTENSIONS) + """Read `value`, a group's `consolidated_metadata` member, in `context` (`CORE_AND_EXTENSIONS_CONTEXT` when none is given), whatever it holds: every problem the read finds, and the model when there are none. This is the form of `ZarrV3ConsolidatedMetadata(value, context)` that returns problems instead of raising them.""" + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) readings, members, document, problems = _read_consolidated_v3( value, context, (ZARR_V3_CONSOLIDATED_METADATA_KEY,) ) @@ -736,7 +736,7 @@ def read_node_metadata_v3( read `node_type` from unvalidated JSON, and a document in another format is reported as not v3. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) node_type, problems = _node_type(value) if node_type == "array": return read_array_metadata_v3(value, context=context) @@ -760,7 +760,7 @@ def node_metadata_from_json_v3( `MetadataValidationError` with every problem `read_node_metadata_v3` finds, including a `node_type` that is neither array nor group. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) reading = read_node_metadata_v3(data, context=context) if reading.metadata is None: raise MetadataValidationError(reading.problems) @@ -775,7 +775,7 @@ def node_metadata_from_key_value_v3( Raises `MetadataValidationError` when the key is missing, its bytes are not JSON, or the document is not a valid array or group. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) # Array documents and group documents are both stored at `zarr.json`. document = load_store_json(mapping, ZARR_V3_GROUP_METADATA_STORE_KEY) return node_metadata_from_json_v3(document, context=context) @@ -852,13 +852,13 @@ def read_group_metadata_v3( reader must understand each one that does not declare `must_understand: false`; the model reports these as `must_understand_fields`. A `consolidated_metadata` member, if - present and claimed by the scope, as `CORE` claims it, is checked + present and claimed by the scope, as `CORE_CONTEXT` claims it, is checked against the envelope the core spec fixes. The documents it lists are not read; `read_consolidated_metadata_v3` reads them. This is the form of `ZarrV3GroupMetadata(value, context)` that returns problems instead of raising them. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) reading, members = read_group_v3(value, context) if members is None or members.document is None: return reading diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_repair.py b/packages/zarr-metadata/src/zarr_metadata/model/_repair.py index d16663b1fe..57e174f44c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_repair.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_repair.py @@ -47,7 +47,7 @@ from zarr_metadata.v2.array import ZARR_V2_ARRAY_METADATA_STORE_KEY from zarr_metadata.v2.definition import CORE_V2 from zarr_metadata.v2.group import ZARR_V2_GROUP_METADATA_STORE_KEY -from zarr_metadata.v3._core import CORE_AND_EXTENSIONS +from zarr_metadata.v3._core import CORE_AND_EXTENSIONS_CONTEXT from zarr_metadata.v3.consolidated import ZARR_V3_CONSOLIDATED_METADATA_KEY if TYPE_CHECKING: @@ -278,7 +278,7 @@ def read_repaired_node_metadata_v3( Content that no repair applies to is read as it is, and reported the same way `read_node_metadata_v3` reports it. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) repaired, repairs = repair_node_metadata_v3(value) return ZarrV3RepairedNodeMetadataReading( read_node_metadata_v3(repaired, context=context), repairs diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py index f369f2beec..8960abdad4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py +++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py @@ -20,7 +20,7 @@ `"zarr_metadata_context_v2"` item. This lets a model with fields of both formats give a scope for each format. - If no scope is given for a format, its field types read in the default - scope for that format, `CORE_AND_EXTENSIONS` or `CORE_V2`. + scope for that format, `CORE_AND_EXTENSIONS_CONTEXT` or `CORE_V2`. A mapping key that starts like these keys but is not one of them raises `TypeError`, so a misspelled key is not read as no scope. To fix a field's @@ -129,7 +129,7 @@ class ArrayManifest(BaseModel): from zarr_metadata.definition._definition import Context from zarr_metadata.definition._scope import ScopeConflictError from zarr_metadata.v2.definition import CORE_V2, ZarrV2Context -from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, ZarrV3Context +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, ZarrV3Context if TYPE_CHECKING: from collections.abc import Callable, Mapping @@ -321,7 +321,7 @@ def _context_of(validation_context: object, default: _C, key: str) -> _C | None: _model.ZarrV3ArrayMetadata, _model.ZarrV3ArrayMetadata.from_json, _model.ZarrV3ArrayMetadata.refined_in, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, CONTEXT_KEY, ), json_schema_input_type=_ZarrV3ArrayMetadataSchema, @@ -353,7 +353,7 @@ def _context_of(validation_context: object, default: _C, key: str) -> _C | None: _model.ZarrV3GroupMetadata, _model.ZarrV3GroupMetadata.from_json, _model.ZarrV3GroupMetadata.refined_in, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, CONTEXT_KEY, ), json_schema_input_type=_ZarrV3GroupMetadataSchema, @@ -385,7 +385,7 @@ def _context_of(validation_context: object, default: _C, key: str) -> _C | None: _model.ZarrV3ConsolidatedMetadata, _model.ZarrV3ConsolidatedMetadata.from_json, _model.ZarrV3ConsolidatedMetadata.refined_in, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, CONTEXT_KEY, ), json_schema_input_type=_ZarrV3ConsolidatedMetadataSchema, @@ -419,12 +419,12 @@ def _context_of(validation_context: object, default: _C, key: str) -> _C | None: _FIELD_PARTS: Final[Mapping[type[Any], tuple[ZarrV3Context | ZarrV2Context, object]]] = ( MappingProxyType( { - _model.ZarrV3ArrayMetadata: (CORE_AND_EXTENSIONS, _ZarrV3ArrayMetadataSchema), + _model.ZarrV3ArrayMetadata: (CORE_AND_EXTENSIONS_CONTEXT, _ZarrV3ArrayMetadataSchema), _model.ZarrV2ArrayMetadata: (CORE_V2, _ZarrV2ArrayMetadataSchema), - _model.ZarrV3GroupMetadata: (CORE_AND_EXTENSIONS, _ZarrV3GroupMetadataSchema), + _model.ZarrV3GroupMetadata: (CORE_AND_EXTENSIONS_CONTEXT, _ZarrV3GroupMetadataSchema), _model.ZarrV2GroupMetadata: (CORE_V2, _ZarrV2GroupMetadataSchema), _model.ZarrV3ConsolidatedMetadata: ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, _ZarrV3ConsolidatedMetadataSchema, ), _model.ZarrV2ConsolidatedMetadata: (CORE_V2, _ZarrV2ConsolidatedMetadataSchema), diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py index c9b1fc425b..22433ef144 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py @@ -8,8 +8,8 @@ `definition`. This module also exports every v3 definition (`BYTES_CODEC`, -`FLOAT64_DATA_TYPE`, `REGULAR_CHUNK_GRID`, ...), the scopes `CORE` and -`CORE_AND_EXTENSIONS`, and the scope class `ZarrV3Context`, so that one +`FLOAT64_DATA_TYPE`, `REGULAR_CHUNK_GRID`, ...), the scopes `CORE_CONTEXT`, +`CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT`, and the scope class `ZarrV3Context`, so that one import builds or updates a scope. Each is the same object as in its own submodule, where a definition's configuration types and constants are. """ @@ -29,12 +29,15 @@ from zarr_metadata.v3.codec.scale_offset import SCALE_OFFSET_CODEC from zarr_metadata.v3.codec.sharding_indexed import SHARDING_INDEXED_CODEC from zarr_metadata.v3.codec.transpose import TRANSPOSE_CODEC +from zarr_metadata.v3.codec.vlen_bytes import VLEN_BYTES_CODEC +from zarr_metadata.v3.codec.vlen_utf8 import VLEN_UTF8_CODEC from zarr_metadata.v3.codec.zstd import ZSTD_CODEC from zarr_metadata.v3.consolidated import CONSOLIDATED_METADATA_EXTRA_FIELD from zarr_metadata.v3.data_type.bool import BOOL_DATA_TYPE from zarr_metadata.v3.data_type.bytes import BYTES_DATA_TYPE from zarr_metadata.v3.data_type.complex64 import COMPLEX64_DATA_TYPE from zarr_metadata.v3.data_type.complex128 import COMPLEX128_DATA_TYPE +from zarr_metadata.v3.data_type.fixed_length_utf32 import FIXED_LENGTH_UTF32_DATA_TYPE from zarr_metadata.v3.data_type.float16 import FLOAT16_DATA_TYPE from zarr_metadata.v3.data_type.float32 import FLOAT32_DATA_TYPE from zarr_metadata.v3.data_type.float64 import FLOAT64_DATA_TYPE @@ -42,6 +45,7 @@ from zarr_metadata.v3.data_type.int16 import INT16_DATA_TYPE from zarr_metadata.v3.data_type.int32 import INT32_DATA_TYPE from zarr_metadata.v3.data_type.int64 import INT64_DATA_TYPE +from zarr_metadata.v3.data_type.null_terminated_bytes import NULL_TERMINATED_BYTES_DATA_TYPE from zarr_metadata.v3.data_type.numpy_datetime64 import NUMPY_DATETIME64_DATA_TYPE from zarr_metadata.v3.data_type.numpy_timedelta64 import NUMPY_TIMEDELTA64_DATA_TYPE from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE @@ -51,9 +55,11 @@ from zarr_metadata.v3.data_type.uint16 import UINT16_DATA_TYPE from zarr_metadata.v3.data_type.uint32 import UINT32_DATA_TYPE from zarr_metadata.v3.data_type.uint64 import UINT64_DATA_TYPE +from zarr_metadata.v3.data_type.variable_length_bytes import VARIABLE_LENGTH_BYTES_DATA_TYPE from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, + ZARR_PYTHON_CONTEXT, ZarrV3Context, ) @@ -66,10 +72,11 @@ "COMPLEX64_DATA_TYPE", "COMPLEX128_DATA_TYPE", "CONSOLIDATED_METADATA_EXTRA_FIELD", - "CORE", - "CORE_AND_EXTENSIONS", + "CORE_AND_EXTENSIONS_CONTEXT", + "CORE_CONTEXT", "CRC32C_CODEC", "DEFAULT_CHUNK_KEY_ENCODING", + "FIXED_LENGTH_UTF32_DATA_TYPE", "FLOAT16_DATA_TYPE", "FLOAT32_DATA_TYPE", "FLOAT64_DATA_TYPE", @@ -78,6 +85,7 @@ "INT16_DATA_TYPE", "INT32_DATA_TYPE", "INT64_DATA_TYPE", + "NULL_TERMINATED_BYTES_DATA_TYPE", "NUMPY_DATETIME64_DATA_TYPE", "NUMPY_TIMEDELTA64_DATA_TYPE", "RAW_BYTES_DATA_TYPE", @@ -93,6 +101,10 @@ "UINT32_DATA_TYPE", "UINT64_DATA_TYPE", "V2_CHUNK_KEY_ENCODING", + "VARIABLE_LENGTH_BYTES_DATA_TYPE", + "VLEN_BYTES_CODEC", + "VLEN_UTF8_CODEC", + "ZARR_PYTHON_CONTEXT", "ZSTD_CODEC", "NodeName", "NodePath", diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_core.py b/packages/zarr-metadata/src/zarr_metadata/v3/_core.py index 5a3f52f4e4..14c0a1dd78 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_core.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_core.py @@ -1,9 +1,11 @@ """The scopes this package provides for Zarr v3, because "is this valid?" has two useful answers. -`CORE` contains what the Zarr v3 specification itself defines, so a +`CORE_CONTEXT` contains what the Zarr v3 specification itself defines, so a document read in it uses nothing that an implementation could reject as -optional. `CORE_AND_EXTENSIONS` adds the extensions that `zarr-extensions` -registers and this package defines. A name in neither scope is not +optional. `CORE_AND_EXTENSIONS_CONTEXT` adds the extensions that `zarr-extensions` +registers and this package defines. `ZARR_PYTHON_CONTEXT` adds the data types that +zarr-python writes but no specification registers, whose spelling +zarr-python may still change. A name in none of these scopes is not rejected, because that is what keeps the format open. Instead it is read as an `UnclaimedField` and not checked. """ @@ -25,12 +27,15 @@ from zarr_metadata.v3.codec.scale_offset import SCALE_OFFSET_CODEC from zarr_metadata.v3.codec.sharding_indexed import SHARDING_INDEXED_CODEC from zarr_metadata.v3.codec.transpose import TRANSPOSE_CODEC +from zarr_metadata.v3.codec.vlen_bytes import VLEN_BYTES_CODEC +from zarr_metadata.v3.codec.vlen_utf8 import VLEN_UTF8_CODEC from zarr_metadata.v3.codec.zstd import ZSTD_CODEC from zarr_metadata.v3.consolidated import CONSOLIDATED_METADATA_EXTRA_FIELD from zarr_metadata.v3.data_type.bool import BOOL_DATA_TYPE from zarr_metadata.v3.data_type.bytes import BYTES_DATA_TYPE from zarr_metadata.v3.data_type.complex64 import COMPLEX64_DATA_TYPE from zarr_metadata.v3.data_type.complex128 import COMPLEX128_DATA_TYPE +from zarr_metadata.v3.data_type.fixed_length_utf32 import FIXED_LENGTH_UTF32_DATA_TYPE from zarr_metadata.v3.data_type.float16 import FLOAT16_DATA_TYPE from zarr_metadata.v3.data_type.float32 import FLOAT32_DATA_TYPE from zarr_metadata.v3.data_type.float64 import FLOAT64_DATA_TYPE @@ -38,6 +43,7 @@ from zarr_metadata.v3.data_type.int16 import INT16_DATA_TYPE from zarr_metadata.v3.data_type.int32 import INT32_DATA_TYPE from zarr_metadata.v3.data_type.int64 import INT64_DATA_TYPE +from zarr_metadata.v3.data_type.null_terminated_bytes import NULL_TERMINATED_BYTES_DATA_TYPE from zarr_metadata.v3.data_type.numpy_datetime64 import NUMPY_DATETIME64_DATA_TYPE from zarr_metadata.v3.data_type.numpy_timedelta64 import NUMPY_TIMEDELTA64_DATA_TYPE from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE @@ -47,8 +53,9 @@ from zarr_metadata.v3.data_type.uint16 import UINT16_DATA_TYPE from zarr_metadata.v3.data_type.uint32 import UINT32_DATA_TYPE from zarr_metadata.v3.data_type.uint64 import UINT64_DATA_TYPE +from zarr_metadata.v3.data_type.variable_length_bytes import VARIABLE_LENGTH_BYTES_DATA_TYPE -CORE: Final = ZarrV3Context( +CORE_CONTEXT: Final = ZarrV3Context( data_types=( BOOL_DATA_TYPE, INT8_DATA_TYPE, @@ -80,21 +87,44 @@ ) """A scope with only what the Zarr v3 specification defines, including the envelope of a group's `consolidated_metadata`, which the spec fixes.""" -CORE_AND_EXTENSIONS: Final = CORE.update( +CORE_AND_EXTENSIONS_CONTEXT: Final = CORE_CONTEXT.update( data_types=[ - *CORE.data_types.values(), + *CORE_CONTEXT.data_types.values(), BYTES_DATA_TYPE, STRING_DATA_TYPE, + FIXED_LENGTH_UTF32_DATA_TYPE, NUMPY_DATETIME64_DATA_TYPE, NUMPY_TIMEDELTA64_DATA_TYPE, STRUCT_DATA_TYPE, ], - chunk_grids=[*CORE.chunk_grids.values(), RECTILINEAR_CHUNK_GRID], - codecs=[*CORE.codecs.values(), CAST_VALUE_CODEC, SCALE_OFFSET_CODEC, ZSTD_CODEC], + chunk_grids=[*CORE_CONTEXT.chunk_grids.values(), RECTILINEAR_CHUNK_GRID], + codecs=[ + *CORE_CONTEXT.codecs.values(), + CAST_VALUE_CODEC, + SCALE_OFFSET_CODEC, + VLEN_BYTES_CODEC, + VLEN_UTF8_CODEC, + ZSTD_CODEC, + ], ) """A scope with what the specification defines, plus the extensions that `zarr-extensions` registers and this package defines.""" +ZARR_PYTHON_CONTEXT: Final = CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[ + *CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), + NULL_TERMINATED_BYTES_DATA_TYPE, + VARIABLE_LENGTH_BYTES_DATA_TYPE, + ], +) +"""A scope with `CORE_AND_EXTENSIONS_CONTEXT`, plus the data types that zarr-python writes but no specification registers: `null_terminated_bytes` and `variable_length_bytes`. + +zarr-python warns when it writes these that their spelling may change, so +they are kept out of `CORE_AND_EXTENSIONS_CONTEXT`. Read in this scope to check +strictly what zarr-python writes. +""" + __all__ = [ - "CORE", - "CORE_AND_EXTENSIONS", + "CORE_AND_EXTENSIONS_CONTEXT", + "CORE_CONTEXT", + "ZARR_PYTHON_CONTEXT", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py b/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py index 420266eec4..3640e92abd 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py @@ -15,7 +15,7 @@ ) from zarr_metadata.definition._envelope import EXTENSION_NAME_SCHEMA_PATTERN from zarr_metadata.v3._context import ZarrV3Context, ZarrV3Kind, is_v3_kind -from zarr_metadata.v3._core import CORE_AND_EXTENSIONS +from zarr_metadata.v3._core import CORE_AND_EXTENSIONS_CONTEXT from zarr_metadata.v3._definition import ( RAW_BYTES_NAME, RAW_BYTES_NAME_PATTERN, @@ -52,7 +52,7 @@ def kind_field(kind: type[Definition[Any]]) -> object: def field_json_schema_v3( kind: type[ZarrV3Kind], *, context: ZarrV3Context | None = None ) -> JSONSchema: - """The JSON Schema of one metadata field read as `kind` in `context` (`CORE_AND_EXTENSIONS` when none is given): everything `resolve` checks, except the rules. A scope of another format raises `TypeError`, as it does for every v3 reader. An extra field is written as the member alone, `{key: value}`. + """The JSON Schema of one metadata field read as `kind` in `context` (`CORE_AND_EXTENSIONS_CONTEXT` when none is given): everything `resolve` checks, except the rules. A scope of another format raises `TypeError`, as it does for every v3 reader. An extra field is written as the member alone, `{key: value}`. The schema accepts a field that one of the definitions in scope reads (its name, its configuration as the TypedDict declares it, an optional @@ -75,7 +75,7 @@ def field_json_schema_v3( "Zarr v3 fields only" ) raise TypeError(msg) - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) schemas = Schemas(field_schemas(context)) if issubclass(asked, (ArrayExtraFieldDefinition, GroupExtraFieldDefinition)): return schemas.document(_extra_field_schema(asked, context, schemas)) @@ -207,7 +207,7 @@ def node_metadata_json_schema_v3(*, context: ZarrV3Context | None = None) -> JSO definition declares, or a name that no definition claims. When the document names a data type in scope, the fill value must have the JSON shape that the data type's definition declares (for `int8`, an - integer in [-128, 127]). In a scope that claims it, as `CORE` does, + integer in [-128, 127]). In a scope that claims it, as `CORE_CONTEXT` does, a group's `consolidated_metadata` is the envelope the core spec fixes, with any JSON as the documents it lists, because the reader does not read them, and a `null` value, which zarr-python 3.0 and 3.1 @@ -226,7 +226,7 @@ def node_metadata_json_schema_v3(*, context: ZarrV3Context | None = None) -> JSO arrays as lists. A model's `to_json` returns tuples, and a Python validator does not accept tuples as arrays. """ - context = scoped(context, CORE_AND_EXTENSIONS) + context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT) schemas = Schemas(_documents(context)) array = schemas.of(ZarrV3ArrayMetadataJSON) group = schemas.of(ZarrV3GroupMetadataJSON) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py index 569a298d7b..8ff769eb49 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py @@ -2,7 +2,7 @@ Each codec defined by the spec or by zarr-extensions has its own submodule (`blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, -`sharding_indexed`, `transpose`, `zstd`). Import a codec's types and its +`sharding_indexed`, `transpose`, `vlen_bytes`, `vlen_utf8`, `zstd`). Import a codec's types and its definition from its submodule: `CodecMetadata`, the canonical type for the codec's permitted JSON shapes (object form plus, where the spec allows, a bare-string short-hand form), `CodecObject`, `_CODEC`, and, for a diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/_vlen.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/_vlen.py new file mode 100644 index 0000000000..335fb93832 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/_vlen.py @@ -0,0 +1,29 @@ +"""The rule that the variable-length codecs `vlen-utf8` and `vlen-bytes` share.""" + +from collections.abc import Iterator + +from zarr_metadata._json import ValidationProblem, shown +from zarr_metadata.definition._definition import EmptyConfiguration, Nested +from zarr_metadata.v3._definition import Chunk, storage_of + + +def vlen_chunk_rules( + configuration: EmptyConfiguration, nested: Nested, chunk: Chunk +) -> Iterator[ValidationProblem]: + """Check that the data type's values vary in size, the converse of the `bytes` codec's rule. + + Each value is written as its length followed by its bytes, so a data + type of fixed size needs a fixed-size codec such as `bytes`. The check + uses the storage that the data type declares, so it does not tell + `string` from `bytes`. If the storage is unknown, nothing is checked. + """ + if chunk.data_type is None: + return + storage = storage_of(chunk.data_type) + if storage is not None and storage != "variable_length": + yield ValidationProblem( + (), + f"expected a data type whose values vary in size, got {shown(chunk.data_type.name)}, " + "whose values have a fixed size", + "invalid_value", + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_bytes.py new file mode 100644 index 0000000000..554191948a --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_bytes.py @@ -0,0 +1,57 @@ +""" +`vlen-bytes` codec types (zarr-extensions). + +See https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/codecs/vlen-bytes/README.md + +The codec has no configuration fields, so the `configuration` key is +absent from the metadata. +""" + +from typing import Final, Literal, NotRequired + +from typing_extensions import TypedDict + +from zarr_metadata.definition._definition import EmptyConfiguration +from zarr_metadata.v3._definition import CodecDefinition +from zarr_metadata.v3.codec._vlen import vlen_chunk_rules + +VLEN_BYTES_CODEC_NAME: Final = "vlen-bytes" +"""The `name` field value of the `vlen-bytes` codec.""" + +VlenBytesCodecName = Literal["vlen-bytes"] +"""Literal type of the `name` field of the `vlen-bytes` codec.""" + + +class VlenBytesCodecObject(TypedDict, closed=True): + """`vlen-bytes` codec metadata in object form. The codec has no configuration fields.""" + + name: VlenBytesCodecName + configuration: NotRequired[EmptyConfiguration] + must_understand: NotRequired[bool] + + +VlenBytesCodecMetadata = VlenBytesCodecObject | VlenBytesCodecName +"""Permitted JSON shapes for `vlen-bytes` codec metadata: the object form, or the bare name, since the codec has no configuration.""" + + +VLEN_BYTES_CODEC: Final = CodecDefinition( + name=VLEN_BYTES_CODEC_NAME, + configuration=EmptyConfiguration, + kind="array_bytes", + size="dynamic", + chunk_rules=vlen_chunk_rules, +) +"""The `vlen-bytes` codec, which serializes variable-length byte strings. + +It is "only compatible with the `"bytes"` data type" +(https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/codecs/vlen-bytes/README.md?plain=1#L30). +The size of its output depends on the values, so it is dynamic. +""" + +__all__ = [ + "VLEN_BYTES_CODEC", + "VLEN_BYTES_CODEC_NAME", + "VlenBytesCodecMetadata", + "VlenBytesCodecName", + "VlenBytesCodecObject", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_utf8.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_utf8.py new file mode 100644 index 0000000000..383c11bb45 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/vlen_utf8.py @@ -0,0 +1,57 @@ +""" +`vlen-utf8` codec types (zarr-extensions). + +See https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/codecs/vlen-utf8/README.md + +The codec has no configuration fields, so the `configuration` key is +absent from the metadata. +""" + +from typing import Final, Literal, NotRequired + +from typing_extensions import TypedDict + +from zarr_metadata.definition._definition import EmptyConfiguration +from zarr_metadata.v3._definition import CodecDefinition +from zarr_metadata.v3.codec._vlen import vlen_chunk_rules + +VLEN_UTF8_CODEC_NAME: Final = "vlen-utf8" +"""The `name` field value of the `vlen-utf8` codec.""" + +VlenUtf8CodecName = Literal["vlen-utf8"] +"""Literal type of the `name` field of the `vlen-utf8` codec.""" + + +class VlenUtf8CodecObject(TypedDict, closed=True): + """`vlen-utf8` codec metadata in object form. The codec has no configuration fields.""" + + name: VlenUtf8CodecName + configuration: NotRequired[EmptyConfiguration] + must_understand: NotRequired[bool] + + +VlenUtf8CodecMetadata = VlenUtf8CodecObject | VlenUtf8CodecName +"""Permitted JSON shapes for `vlen-utf8` codec metadata: the object form, or the bare name, since the codec has no configuration.""" + + +VLEN_UTF8_CODEC: Final = CodecDefinition( + name=VLEN_UTF8_CODEC_NAME, + configuration=EmptyConfiguration, + kind="array_bytes", + size="dynamic", + chunk_rules=vlen_chunk_rules, +) +"""The `vlen-utf8` codec, which serializes variable-length UTF-8 strings. + +It is "only compatible with the `"string"` data type" +(https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/codecs/vlen-utf8/README.md?plain=1#L30). +The size of its output depends on the values, so it is dynamic. +""" + +__all__ = [ + "VLEN_UTF8_CODEC", + "VLEN_UTF8_CODEC_NAME", + "VlenUtf8CodecMetadata", + "VlenUtf8CodecName", + "VlenUtf8CodecObject", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/consolidated.py b/packages/zarr-metadata/src/zarr_metadata/v3/consolidated.py index a3aa01afb9..d96aff7146 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/consolidated.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/consolidated.py @@ -9,7 +9,7 @@ only its `{kind, must_understand, metadata}` envelope, which `ZarrV3ConsolidatedMetadataEnvelopeJSON` types with any JSON as the entries. `CONSOLIDATED_METADATA_EXTRA_FIELD` reads a group's member as -that envelope, and `CORE` holds it. `ZarrV3ConsolidatedMetadataJSON` types +that envelope, and `CORE_CONTEXT` holds it. `ZarrV3ConsolidatedMetadataJSON` types the entries as the reference implementation writes them. """ diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py index d1a193b2c4..8206968089 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py @@ -4,8 +4,10 @@ - Core primitives: `bool`, `int8`/`16`/`32`/`64`, `uint8`/`16`/`32`/`64`, `float16`/`32`/`64`, `complex64`/`128`, `raw` (for `r`) -- zarr-extensions: `bytes`, `string`, `numpy_datetime64`, `numpy_timedelta64`, - `struct` +- zarr-extensions: `bytes`, `string`, `fixed_length_utf32`, + `numpy_datetime64`, `numpy_timedelta64`, `struct` +- zarr-python, registered by no specification: `null_terminated_bytes`, + `variable_length_bytes` Import a data type's types and its definition from its submodule: @@ -13,7 +15,8 @@ (or, for named-config dtypes, the literal value of their `name` field) - `FillValue` -- the permitted JSON shape of the `fill_value` field - the envelope and configuration TypedDicts of named-config dtypes - (`numpy_datetime64`, `numpy_timedelta64`, `struct`), the branded + (`fixed_length_utf32`, `null_terminated_bytes`, `numpy_datetime64`, + `numpy_timedelta64`, `struct`), the branded `HexFloat` / `Base64Bytes` types, and their validator functions - `_DATA_TYPE`, the definition diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py index 07de456e22..22bce59f60 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py @@ -1,15 +1,30 @@ -"""A byte value, as used in the fill values of raw bits and of `bytes`. +"""A byte value, as used in the fill values of raw bits and of `bytes`, and the rules of a fill value written in base64. -Each is an integer in `[0, 255]` +Each byte value is an integer in `[0, 255]` (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/data-types/index.rst#L97-L99; the reference for the `bytes` type is https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/data-types/bytes/README.md?plain=1#L8). """ +from collections.abc import Iterator from typing import Annotated from annotated_types import Interval +from zarr_metadata._json import ValidationProblem, shown +from zarr_metadata._numpy import base64_bytes + ByteValue = Annotated[int, Interval(ge=0, le=255)] """One byte of a fill value: a JSON integer in `[0, 255]`.""" -__all__ = ["ByteValue"] + +def base64_problems(value: str) -> Iterator[ValidationProblem]: + """Check that a fill value written as a string is standard-alphabet base64.""" + try: + base64_bytes(value) + except ValueError: + yield ValidationProblem( + (), f"expected standard-alphabet base64, got {shown(value)}", "invalid_value" + ) + + +__all__ = ["ByteValue", "base64_problems"] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py index 1f6dc1b7f0..d5b81e8dc7 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py @@ -8,7 +8,7 @@ from collections.abc import Iterator from typing import Final, Literal -from zarr_metadata._json import ValidationProblem, shown +from zarr_metadata._json import ValidationProblem from zarr_metadata._numpy import Base64Bytes, base64_bytes from zarr_metadata.definition._definition import ( EmptyConfiguration, @@ -18,7 +18,7 @@ DataTypeDefinition, variable_length, ) -from zarr_metadata.v3.data_type._byte import ByteValue +from zarr_metadata.v3.data_type._byte import ByteValue, base64_problems BYTES_DATA_TYPE_NAME: Final = "bytes" """The `data_type` value for the variable-length `bytes` type.""" @@ -38,14 +38,8 @@ def _fill_value_rules( configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue ) -> Iterator[ValidationProblem]: """Check that a string fill value is standard-alphabet base64. A fill value written as byte values is not checked here.""" - if not isinstance(value, str): - return - try: - base64_bytes(value) - except ValueError: - yield ValidationProblem( - (), f"expected standard-alphabet base64, got {shown(value)}", "invalid_value" - ) + if isinstance(value, str): + yield from base64_problems(value) def _fill_value_canonical( diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py new file mode 100644 index 0000000000..d0cfa0a68e --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py @@ -0,0 +1,100 @@ +""" +Zarr `fixed_length_utf32` data type (fixed-length UTF-32 strings, zarr-extensions). + +See https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md +""" + +from collections.abc import Iterator +from typing import Final, Literal, NotRequired + +from typing_extensions import ReadOnly, TypedDict + +from zarr_metadata._json import ValidationProblem +from zarr_metadata.definition._definition import Nested +from zarr_metadata.v3._definition import DataTypeDefinition, multi_byte + +FIXED_LENGTH_UTF32_DATA_TYPE_NAME: Final = "fixed_length_utf32" +"""The `name` field value of the `fixed_length_utf32` data type.""" + +FixedLengthUtf32DataTypeName = Literal["fixed_length_utf32"] +"""Literal type of the `name` field of the `fixed_length_utf32` data type.""" + +_CODE_POINT_BYTES: Final = 4 + + +class FixedLengthUtf32Configuration(TypedDict, closed=True): + """ + Configuration for the `fixed_length_utf32` data type. + + Attributes + ---------- + length_bytes + The size in bytes of every value: a positive multiple of 4, since + each code point takes 4 bytes. + """ + + length_bytes: ReadOnly[int] + + +class FixedLengthUtf32(TypedDict, closed=True): + """`fixed_length_utf32` data type metadata.""" + + name: FixedLengthUtf32DataTypeName + configuration: FixedLengthUtf32Configuration + must_understand: NotRequired[bool] + + +FixedLengthUtf32FillValue = str +"""Permitted JSON shape of the `fill_value` field for `fixed_length_utf32`: a JSON string of at most `length_bytes / 4` code points.""" + + +def _rules( + configuration: FixedLengthUtf32Configuration, nested: Nested +) -> Iterator[ValidationProblem]: + """Check that `length_bytes` is a "positive integer, multiple of `4`" (https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md?plain=1#L41).""" + length_bytes = configuration["length_bytes"] + if length_bytes <= 0 or length_bytes % _CODE_POINT_BYTES != 0: + yield ValidationProblem( + ("length_bytes",), + f"expected a positive multiple of {_CODE_POINT_BYTES}, got {length_bytes}", + "invalid_value", + ) + + +def _fill_value_rules( + configuration: FixedLengthUtf32Configuration, nested: Nested, value: FixedLengthUtf32FillValue +) -> Iterator[ValidationProblem]: + """Check that the fill value fits: "a JSON string of at most `length_bytes / 4` code points" (https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md?plain=1#L124-L126).""" + capacity = configuration["length_bytes"] // _CODE_POINT_BYTES + if len(value) > capacity: + yield ValidationProblem( + (), + f"expected at most {capacity} code points, got {len(value)}", + "invalid_value", + ) + + +FIXED_LENGTH_UTF32_DATA_TYPE: Final = DataTypeDefinition( + name=FIXED_LENGTH_UTF32_DATA_TYPE_NAME, + configuration=FixedLengthUtf32Configuration, + rules=_rules, + fill_value=FixedLengthUtf32FillValue, + fill_value_rules=_fill_value_rules, + storage=multi_byte, +) +"""The `fixed_length_utf32` data type: NumPy's `U`. + +Each code point is a 4-byte UTF-32 code unit, so the `bytes` codec "MUST +be configured with an explicit `endian` setting" +(https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md?plain=1#L100-L103). +""" + + +__all__ = [ + "FIXED_LENGTH_UTF32_DATA_TYPE", + "FIXED_LENGTH_UTF32_DATA_TYPE_NAME", + "FixedLengthUtf32", + "FixedLengthUtf32Configuration", + "FixedLengthUtf32DataTypeName", + "FixedLengthUtf32FillValue", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py new file mode 100644 index 0000000000..18f9dee137 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py @@ -0,0 +1,117 @@ +""" +zarr-python's `null_terminated_bytes` data type (fixed-length, null-padded bytes). + +No specification registers this data type. zarr-python writes it for +NumPy's `S` data types, and warns that its spelling may change +(https://github.com/zarr-developers/zarr-python/blob/a2f2415c86f7d35212ae8c0d5dc5e4a22a6c6191/src/zarr/core/dtype/npy/bytes.py#L385-L390). +This module describes what zarr-python writes. Its definition is in the +`ZARR_PYTHON_CONTEXT` scope, not in `CORE_AND_EXTENSIONS_CONTEXT`. +""" + +import base64 +from collections.abc import Iterator +from typing import Final, Literal, NotRequired + +from typing_extensions import ReadOnly, TypedDict + +from zarr_metadata._json import ValidationProblem +from zarr_metadata._numpy import Base64Bytes +from zarr_metadata.definition._definition import Nested +from zarr_metadata.v3._definition import DataTypeDefinition, single_byte +from zarr_metadata.v3.data_type._byte import base64_problems + +NULL_TERMINATED_BYTES_DATA_TYPE_NAME: Final = "null_terminated_bytes" +"""The `name` field value of the `null_terminated_bytes` data type.""" + +NullTerminatedBytesDataTypeName = Literal["null_terminated_bytes"] +"""Literal type of the `name` field of the `null_terminated_bytes` data type.""" + + +class NullTerminatedBytesConfiguration(TypedDict, closed=True): + """ + Configuration for the `null_terminated_bytes` data type. + + Attributes + ---------- + length_bytes + The size in bytes of every value, at least 1. + """ + + length_bytes: ReadOnly[int] + + +class NullTerminatedBytes(TypedDict, closed=True): + """`null_terminated_bytes` data type metadata.""" + + name: NullTerminatedBytesDataTypeName + configuration: NullTerminatedBytesConfiguration + must_understand: NotRequired[bool] + + +NullTerminatedBytesFillValue = Base64Bytes +"""Permitted JSON shape of the `fill_value` field for `null_terminated_bytes`: the bytes, without their trailing null bytes, as a `Base64Bytes` string.""" + + +def _rules( + configuration: NullTerminatedBytesConfiguration, nested: Nested +) -> Iterator[ValidationProblem]: + """Check that `length_bytes` is positive. zarr-python refuses a length below 1, since such a type holds no data.""" + length_bytes = configuration["length_bytes"] + if length_bytes < 1: + yield ValidationProblem( + ("length_bytes",), f"expected a positive size, got {length_bytes}", "invalid_value" + ) + + +def _fill_value_rules( + configuration: NullTerminatedBytesConfiguration, + nested: Nested, + value: NullTerminatedBytesFillValue, +) -> Iterator[ValidationProblem]: + """Check that the fill value is base64, and that the bytes it encodes fit in `length_bytes`.""" + problems = list(base64_problems(value)) + if problems: + yield from problems + return + size = len(base64.b64decode(value)) + if size > configuration["length_bytes"]: + yield ValidationProblem( + (), + f"expected at most {configuration['length_bytes']} bytes, got {size}", + "invalid_value", + ) + + +def _fill_value_canonical( + configuration: NullTerminatedBytesConfiguration, + nested: Nested, + value: NullTerminatedBytesFillValue, +) -> str: + """Return the bytes without their trailing null bytes, encoded in base64 again. + + A value is padded with null bytes to `length_bytes`, so `"QQ=="` and + `"QQAA"` are the same value, as are `"QR=="` and `"QQ=="`. + """ + return base64.b64encode(base64.b64decode(value).rstrip(b"\x00")).decode("ascii") + + +NULL_TERMINATED_BYTES_DATA_TYPE: Final = DataTypeDefinition( + name=NULL_TERMINATED_BYTES_DATA_TYPE_NAME, + configuration=NullTerminatedBytesConfiguration, + rules=_rules, + fill_value=NullTerminatedBytesFillValue, + fill_value_rules=_fill_value_rules, + fill_value_canonical=_fill_value_canonical, + storage=single_byte, +) +"""zarr-python's `null_terminated_bytes` data type: NumPy's `S`. Each value is `length_bytes` single bytes.""" + + +__all__ = [ + "NULL_TERMINATED_BYTES_DATA_TYPE", + "NULL_TERMINATED_BYTES_DATA_TYPE_NAME", + "NullTerminatedBytes", + "NullTerminatedBytesConfiguration", + "NullTerminatedBytesDataTypeName", + "NullTerminatedBytesFillValue", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py new file mode 100644 index 0000000000..a4e862ffa3 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py @@ -0,0 +1,62 @@ +""" +zarr-python's `variable_length_bytes` data type (variable-length raw bytes). + +No specification registers this name. zarr-python writes it for arrays of +Python `bytes` objects, and warns that its spelling may change +(https://github.com/zarr-developers/zarr-python/blob/a2f2415c86f7d35212ae8c0d5dc5e4a22a6c6191/src/zarr/core/dtype/npy/bytes.py#L1141-L1143). +zarr-extensions registers the same values as `bytes`. This module describes +what zarr-python writes. Its definition is in the `ZARR_PYTHON_CONTEXT` scope, not +in `CORE_AND_EXTENSIONS_CONTEXT`. +""" + +import base64 +from collections.abc import Iterator +from typing import Final, Literal + +from zarr_metadata._json import ValidationProblem +from zarr_metadata._numpy import Base64Bytes +from zarr_metadata.definition._definition import EmptyConfiguration, Nested +from zarr_metadata.v3._definition import DataTypeDefinition, variable_length +from zarr_metadata.v3.data_type._byte import base64_problems + +VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME: Final = "variable_length_bytes" +"""The `data_type` value for the `variable_length_bytes` type.""" + +VariableLengthBytesDataTypeName = Literal["variable_length_bytes"] +"""Literal type of the `data_type` field for `variable_length_bytes`.""" + +VariableLengthBytesFillValue = Base64Bytes +"""Permitted JSON shape of the `fill_value` field for `variable_length_bytes`: a `Base64Bytes` string, which is what zarr-python writes and reads.""" + + +def _fill_value_rules( + configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue +) -> Iterator[ValidationProblem]: + """Check that the fill value is standard-alphabet base64.""" + yield from base64_problems(value) + + +def _fill_value_canonical( + configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue +) -> str: + """Return the bytes encoded in base64 again, so `"QR=="` becomes `"QQ=="`.""" + return base64.b64encode(base64.b64decode(value)).decode("ascii") + + +VARIABLE_LENGTH_BYTES_DATA_TYPE: Final = DataTypeDefinition( + name=VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME, + configuration=EmptyConfiguration, + fill_value=VariableLengthBytesFillValue, + fill_value_rules=_fill_value_rules, + fill_value_canonical=_fill_value_canonical, + storage=variable_length, +) +"""zarr-python's `variable_length_bytes` data type. It is written as a bare name and has no configuration, and each value is stored in as many bytes as it needs.""" + + +__all__ = [ + "VARIABLE_LENGTH_BYTES_DATA_TYPE", + "VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME", + "VariableLengthBytesDataTypeName", + "VariableLengthBytesFillValue", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/definition.py b/packages/zarr-metadata/src/zarr_metadata/v3/definition.py index 90cb1e466e..31c6b4893d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/definition.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/definition.py @@ -11,8 +11,8 @@ which both formats share. `ZarrV3Context` is the v3 scope, with one slot for each kind: `data_types`, `chunk_grids`, `chunk_key_encodings`, `codecs`, `storage_transformers`, `array_extra_fields` and -`group_extra_fields`. `CORE` and `CORE_AND_EXTENSIONS` are the v3 scopes -this package provides. +`group_extra_fields`. `CORE_CONTEXT`, `CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT` are +the v3 scopes this package provides. A scope reads whole documents as well as fields. `read_array_metadata_v3(document, context=CONTEXT)`, from @@ -102,7 +102,7 @@ ZarrV3StorageTransformerField, ) from zarr_metadata.v3._context import ZarrV3Context, ZarrV3ContextSlots, ZarrV3Kind -from zarr_metadata.v3._core import CORE, CORE_AND_EXTENSIONS +from zarr_metadata.v3._core import CORE_AND_EXTENSIONS_CONTEXT, CORE_CONTEXT, ZARR_PYTHON_CONTEXT from zarr_metadata.v3._definition import ( ArrayExtraFieldDefinition, Chunk, @@ -122,8 +122,9 @@ from zarr_metadata.v3._pipeline import Stage __all__ = [ - "CORE", - "CORE_AND_EXTENSIONS", + "CORE_AND_EXTENSIONS_CONTEXT", + "CORE_CONTEXT", + "ZARR_PYTHON_CONTEXT", "ArrayExtraFieldDefinition", "Chunk", "ChunkGridDefinition", diff --git a/packages/zarr-metadata/tests/model/test_array.py b/packages/zarr-metadata/tests/model/test_array.py index cd37495b27..ec6765d551 100644 --- a/packages/zarr-metadata/tests/model/test_array.py +++ b/packages/zarr-metadata/tests/model/test_array.py @@ -53,8 +53,8 @@ from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, ArrayExtraFieldDefinition, CodecDefinition, ) @@ -473,16 +473,16 @@ def test_update_can_add_an_extension_member() -> None: def test_update_reads_every_member_in_the_models_own_scope() -> None: """`update` reads the new document in the scope the model was read in, so a name that scope leaves unclaimed stays unclaimed. Use `with_context` to read it in another scope. - `zstd` is an extension, so `CORE` leaves it unclaimed and unchecked. + `zstd` is an extension, so `CORE_CONTEXT` leaves it unclaimed and unchecked. """ little: ZarrV3NamedConfigJSON = {"name": "bytes", "configuration": {"endian": "little"}} zstd: ZarrV3NamedConfigJSON = {"name": "zstd", "configuration": {"level": 3, "checksum": False}} - base = ZarrV3ArrayMetadata.create_default(context=CORE, codecs=(little, zstd)) + base = ZarrV3ArrayMetadata.create_default(context=CORE_CONTEXT, codecs=(little, zstd)) assert isinstance(base.codecs[1], UnclaimedField) kept = base.update(attributes={"k": 1}) assert kept.codecs[1] == base.codecs[1] - assert kept.context == CORE - given = base.with_context(CORE_AND_EXTENSIONS).update(codecs=(little, zstd)) + assert kept.context == CORE_CONTEXT + given = base.with_context(CORE_AND_EXTENSIONS_CONTEXT).update(codecs=(little, zstd)) assert isinstance(given.codecs[1], AcceptedField) @@ -689,7 +689,7 @@ def test_a_model_pickles_and_copies_with_its_definitions() -> None: {"name": "bytes", "configuration": {"endian": "little"}}, {"name": "gzip", "configuration": {"level": 1}}, ), - context=CORE, + context=CORE_CONTEXT, ) for again in (pickle.loads(pickle.dumps(model)), copy.copy(model), copy.deepcopy(model)): assert again == model @@ -996,14 +996,14 @@ def test_v3_parser_accepts_bare_string_data_type() -> None: @pytest.mark.parametrize("name", ["bytes", "acme.codec", "urn:example:codec"]) def test_metadata_field_accepts_a_name_as_the_spec_names_one(name: str) -> None: """The envelope check accepts a name in a form the spec allows for an extension name, whether or not a definition in scope claims it.""" - assert resolve({"name": name}, CodecDefinition, CORE_AND_EXTENSIONS)[1] == () + assert resolve({"name": name}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[1] == () @pytest.mark.parametrize("value", [0, 1, "false", None]) def test_metadata_field_must_understand_must_be_boolean(value: object) -> None: """must_understand is a JSON boolean, not a truthy scalar.""" _, problems = resolve( - {"name": "acme.x", "must_understand": value}, CodecDefinition, CORE_AND_EXTENSIONS + {"name": "acme.x", "must_understand": value}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT ) assert [(problem.loc, problem.kind) for problem in problems] == [ (("must_understand",), "invalid_type") @@ -1012,7 +1012,9 @@ def test_metadata_field_must_understand_must_be_boolean(value: object) -> None: def test_metadata_field_rejects_unknown_envelope_member() -> None: """Unknown envelope keys cannot be silently discarded during normalization.""" - _, problems = resolve({"name": "acme.x", "typo": 1}, CodecDefinition, CORE_AND_EXTENSIONS) + _, problems = resolve( + {"name": "acme.x", "typo": 1}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT + ) assert [(problem.loc, problem.kind) for problem in problems] == [(("typo",), "unknown_key")] @@ -1240,7 +1242,7 @@ def test_a_field_read_from_abstract_containers_writes_canonical_containers() -> """`resolve` reads a field given as abstract containers, and the field writes canonical containers at every nesting level.""" value = UserDict({"name": "example", "configuration": UserDict({"values": range(2)})}) - field, problems = resolve(value, CodecDefinition, CORE_AND_EXTENSIONS) + field, problems = resolve(value, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert isinstance(field, UnclaimedField) written = field.to_json() @@ -1282,7 +1284,7 @@ def test_resolve_reports_the_problems_of_a_metadata_field( case: Expect[object, frozenset[tuple[str | int, ...]]], ) -> None: """`resolve` reports the problems of a metadata-field value, read as a codec.""" - _, problems = resolve(case.input, CodecDefinition, CORE_AND_EXTENSIONS) + _, problems = resolve(case.input, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert (problems == ()) is (case.output == frozenset()) assert {p.loc for p in problems} >= case.output @@ -1952,11 +1954,11 @@ def test_a_field_s_configuration_member_is_counted_from_the_field_s_root() -> No for _ in range(JSON_DEPTH - 2): nested = {"x": nested} field = {"name": "acme.x", "configuration": {"y": nested}} - _, problems = resolve(field, CodecDefinition, CORE_AND_EXTENSIONS) + _, problems = resolve(field, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert [(len(p.loc), p.kind) for p in problems] == [(JSON_DEPTH, "invalid_value")] assert problems[0].loc[:2] == ("configuration", "y") shallower = {"name": "acme.x", "configuration": {"y": nested["x"]}} - assert resolve(shallower, CodecDefinition, CORE_AND_EXTENSIONS)[1] == () + assert resolve(shallower, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[1] == () def test_a_v2_dtype_of_nested_records_is_read_to_the_levels_a_reader_walks() -> None: @@ -2438,7 +2440,9 @@ def test_a_problem_shows_an_integer_too_long_to_write_by_its_size( document = {**ZarrV3ArrayMetadata.create_default().to_json(), "zarr_format": 10**5000} (problem,) = read_array_metadata_v3(document).problems assert problem.message == f"expected 3, got an integer of {(10**5000).bit_length()} bits" - (problem,) = resolve({"name": "gzip", 10**5000: 1}, CodecDefinition, CORE_AND_EXTENSIONS)[1] + (problem,) = resolve( + {"name": "gzip", 10**5000: 1}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT + )[1] assert problem.message == ( f"non-string key an integer of {(10**5000).bit_length()} bits in JSON object" ) @@ -2450,7 +2454,9 @@ def test_a_problem_shows_an_integer_too_long_to_write_by_its_size( assert ( problem.message == "expected 3, got a value of type list the interpreter will not write" ) - (problem,) = resolve({"name": "gzip", (10**5000,): 1}, CodecDefinition, CORE_AND_EXTENSIONS)[1] + (problem,) = resolve( + {"name": "gzip", (10**5000,): 1}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT + )[1] assert problem.message == ( "non-string key a value of type tuple the interpreter will not write in JSON object" ) diff --git a/packages/zarr-metadata/tests/model/test_consolidating.py b/packages/zarr-metadata/tests/model/test_consolidating.py index 850888d223..3c7f4e423e 100644 --- a/packages/zarr-metadata/tests/model/test_consolidating.py +++ b/packages/zarr-metadata/tests/model/test_consolidating.py @@ -23,8 +23,8 @@ from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, CodecDefinition, ZarrV3Context, ) @@ -41,7 +41,7 @@ MY_BYTES = CodecDefinition( name="bytes", configuration=EmptyConfiguration, kind="array_bytes", size="static" ) -PRIVATE = CORE.update(codecs={**CORE.codecs, "bytes": MY_BYTES}) +PRIVATE = CORE_CONTEXT.update(codecs={**CORE_CONTEXT.codecs, "bytes": MY_BYTES}) INLINE: dict[str, Any] = {"kind": "inline", "must_understand": False} @@ -64,11 +64,16 @@ def _group_document(**entries: object) -> dict[str, Any]: @pytest.mark.parametrize( ("context", "child", "beside", "gained"), [ - (CORE_AND_EXTENSIONS, ZarrV3ArrayMetadata(ARRAY), {}, False), - (CORE_AND_EXTENSIONS, ZarrV3ArrayMetadata(ARRAY, context=CORE), {}, False), - (CORE_AND_EXTENSIONS, ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE), {}, True), + (CORE_AND_EXTENSIONS_CONTEXT, ZarrV3ArrayMetadata(ARRAY), {}, False), + (CORE_AND_EXTENSIONS_CONTEXT, ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT), {}, False), ( - CORE, + CORE_AND_EXTENSIONS_CONTEXT, + ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE_CONTEXT), + {}, + True, + ), + ( + CORE_CONTEXT, ZarrV3GroupMetadata(_group_document(x=ARRAY), context=ZarrV3Context()), {"a/x": ZarrV3ArrayMetadata(ARRAY, context=ZarrV3Context())}, True, @@ -124,7 +129,7 @@ def test_error_a_model_that_conflicts_with_the_scope_is_refused_at_its_path() -> """A model read with a private `bytes`, given to consolidated metadata whose scope reads the core `bytes`, is a conflict. The problem is at the entry's path, names the kind and the name, and tells the two definitions apart. The model is not silently read again.""" child = ZarrV3ArrayMetadata(ARRAY, context=PRIVATE) with pytest.raises(MetadataValidationError) as raised: - _consolidated(CORE_AND_EXTENSIONS, a=child) + _consolidated(CORE_AND_EXTENSIONS_CONTEXT, a=child) (problem,) = raised.value.problems assert problem.loc == ("metadata", "a", "codecs", 0) assert problem.kind == "invalid_value" @@ -137,9 +142,9 @@ def test_error_a_model_that_conflicts_with_the_scope_is_refused_at_its_path() -> def test_error_a_model_that_would_lose_a_meaning_is_refused_at_its_path() -> None: """A model read in a scope that claims `zstd`, given to consolidated metadata whose scope leaves `zstd` unclaimed, would lose information. It is refused, the same way a conflict is.""" - child = ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE_AND_EXTENSIONS) + child = ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE_AND_EXTENSIONS_CONTEXT) with pytest.raises(MetadataValidationError) as raised: - _consolidated(CORE, a=child) + _consolidated(CORE_CONTEXT, a=child) (problem,) = raised.value.problems assert problem.loc == ("metadata", "a", "codecs", 1) assert problem.message.endswith("which that scope leaves unclaimed") @@ -147,9 +152,9 @@ def test_error_a_model_that_would_lose_a_meaning_is_refused_at_its_path() -> Non def test_error_a_gain_that_surfaces_a_problem_is_reported_at_the_problem() -> None: """A model whose scope left `zstd` unclaimed, given to consolidated metadata whose scope claims it, is read again in that scope. A level that the `zstd` definition refuses is reported as a problem at its location.""" - child = ZarrV3ArrayMetadata(WITH_LOOSE_ZSTD, context=CORE) + child = ZarrV3ArrayMetadata(WITH_LOOSE_ZSTD, context=CORE_CONTEXT) with pytest.raises(MetadataValidationError) as raised: - _consolidated(CORE_AND_EXTENSIONS, a=child) + _consolidated(CORE_AND_EXTENSIONS_CONTEXT, a=child) assert [problem.loc for problem in raised.value.problems] == [ ("metadata", "a", "codecs", 1, "configuration", "level") ] @@ -175,25 +180,25 @@ def test_error_the_reader_reports_a_conflicting_model_as_the_constructor_does() def test_a_group_writes_the_consolidated_metadata_it_is_given_in_its_own_scope() -> None: """A `ZarrV3ConsolidatedMetadata` given to `update` is written as its document. It does not bring its own scope into the group: the group's `context` stays its own, and reading the written member in that scope gives the models in it.""" - held = _consolidated(CORE_AND_EXTENSIONS, a=ZarrV3ArrayMetadata(ARRAY)) - group = ZarrV3GroupMetadata.create_default(context=CORE, attributes={"t": 1}) + held = _consolidated(CORE_AND_EXTENSIONS_CONTEXT, a=ZarrV3ArrayMetadata(ARRAY)) + group = ZarrV3GroupMetadata.create_default(context=CORE_CONTEXT, attributes={"t": 1}) updated = group.update(consolidated_metadata=held) - assert updated.context == CORE + assert updated.context == CORE_CONTEXT assert updated.attributes == {"t": 1} assert updated.consolidated_metadata == held.to_json() again = ZarrV3ConsolidatedMetadata(updated.consolidated_metadata, context=updated.context) - assert again.metadata["a"].context == CORE + assert again.metadata["a"].context == CORE_CONTEXT assert again == held def test_error_a_group_refuses_consolidated_metadata_that_conflicts_with_its_scope() -> None: """A `ZarrV3ConsolidatedMetadata` read in another scope is read again in the group's scope before the group writes it, as a listing of node models is. A model read with a private `bytes` conflicts with the core `bytes` the group reads, so `update` and `create_default` refuse it at the child's path.""" held = _consolidated(PRIVATE, a=ZarrV3ArrayMetadata(ARRAY, context=PRIVATE)) - group = ZarrV3GroupMetadata.create_default(context=CORE_AND_EXTENSIONS) + group = ZarrV3GroupMetadata.create_default(context=CORE_AND_EXTENSIONS_CONTEXT) for write in ( lambda: group.update(consolidated_metadata=held), lambda: ZarrV3GroupMetadata.create_default( - context=CORE_AND_EXTENSIONS, consolidated_metadata=held + context=CORE_AND_EXTENSIONS_CONTEXT, consolidated_metadata=held ), ): with pytest.raises(MetadataValidationError) as raised: @@ -205,9 +210,9 @@ def test_error_a_group_refuses_consolidated_metadata_that_conflicts_with_its_sco def test_error_a_group_refuses_consolidated_metadata_that_would_lose_a_meaning() -> None: """A listing read in a scope that claims `zstd`, given to a group whose scope leaves `zstd` unclaimed, would lose information when the group's scope reads it, so the group refuses it at the child's path.""" - held = _consolidated(CORE_AND_EXTENSIONS, a=ZarrV3ArrayMetadata(WITH_ZSTD)) + held = _consolidated(CORE_AND_EXTENSIONS_CONTEXT, a=ZarrV3ArrayMetadata(WITH_ZSTD)) with pytest.raises(MetadataValidationError) as raised: - ZarrV3GroupMetadata.create_default(context=CORE).update(consolidated_metadata=held) + ZarrV3GroupMetadata.create_default(context=CORE_CONTEXT).update(consolidated_metadata=held) assert [problem.loc for problem in raised.value.problems] == [ ("consolidated_metadata", "metadata", "a", "codecs", 1) ] @@ -217,7 +222,7 @@ def test_error_a_group_refuses_consolidated_metadata_that_would_lose_a_meaning() ("context", "found"), [ (ZarrV3Context(), []), - (CORE, [("metadata", "g", "consolidated_metadata", "garbage")]), + (CORE_CONTEXT, [("metadata", "g", "consolidated_metadata", "garbage")]), ], ids=["unclaimed", "claimed"], ) @@ -232,12 +237,12 @@ def test_a_listed_group_s_listing_is_read_only_where_the_scope_claims_it( def test_a_group_reads_its_listing_in_its_own_scope() -> None: """`consolidated()` reads the documents a group's `consolidated_metadata` lists in the group's own scope, so a child reads as the group's reader would read it, and returns None for a group that lists none.""" - group = ZarrV3GroupMetadata(_group_document(a=WITH_ZSTD), context=CORE) + group = ZarrV3GroupMetadata(_group_document(a=WITH_ZSTD), context=CORE_CONTEXT) listing = group.consolidated() assert listing is not None - assert listing.context == CORE - assert listing.metadata["a"].context == CORE - assert listing == ZarrV3ConsolidatedMetadata(group.consolidated_metadata, context=CORE) + assert listing.context == CORE_CONTEXT + assert listing.metadata["a"].context == CORE_CONTEXT + assert listing == ZarrV3ConsolidatedMetadata(group.consolidated_metadata, context=CORE_CONTEXT) assert ZarrV3GroupMetadata.create_default().consolidated() is None @@ -267,11 +272,11 @@ def test_error_a_group_s_listing_that_is_not_an_object(listing: object) -> None: def test_children_of_different_scopes_consolidate_in_their_union() -> None: """Children read in different scopes are consolidated in the union of their scopes. Each child refines into that scope, and the consolidated metadata reads in it.""" - a = ZarrV3ArrayMetadata(ARRAY, context=CORE) - b = ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE_AND_EXTENSIONS) + a = ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT) + b = ZarrV3ArrayMetadata(WITH_ZSTD, context=CORE_AND_EXTENSIONS_CONTEXT) context = a.context.union(b.context) held = _consolidated(context, a=a, b=b) - assert held.context == CORE_AND_EXTENSIONS + assert held.context == CORE_AND_EXTENSIONS_CONTEXT assert held.metadata["a"] == a assert held.metadata["b"] == b @@ -310,10 +315,10 @@ def test_error_a_data_type_conflict_names_the_kind_and_the_written_name() -> Non ) child = ZarrV3ArrayMetadata( {**ARRAY, "data_type": "r16", "fill_value": [0, 0], "codecs": ("bytes",)}, - context=CORE.update(data_types={**CORE.data_types, "r*": strict}), + context=CORE_CONTEXT.update(data_types={**CORE_CONTEXT.data_types, "r*": strict}), ) with pytest.raises(MetadataValidationError) as raised: - _consolidated(CORE, a=child) + _consolidated(CORE_CONTEXT, a=child) (problem,) = raised.value.problems assert problem.loc == ("metadata", "a", "data_type") assert problem.message == ( @@ -328,10 +333,10 @@ def test_error_a_conflict_between_definitions_alike_says_they_differ() -> None: strict = dataclasses.replace(GZIP_CODEC, rules=lambda configuration, nested: iter(())) child = ZarrV3ArrayMetadata( {**ARRAY, "codecs": ("bytes", {"name": "gzip", "configuration": {"level": 1}})}, - context=CORE.update(codecs={**CORE.codecs, "gzip": strict}), + context=CORE_CONTEXT.update(codecs={**CORE_CONTEXT.codecs, "gzip": strict}), ) with pytest.raises(MetadataValidationError) as raised: - _consolidated(CORE, a=child) + _consolidated(CORE_CONTEXT, a=child) (problem,) = raised.value.problems assert problem.message == ( "expected a document read in the scope of the consolidated metadata, got a model that " diff --git a/packages/zarr-metadata/tests/model/test_construction.py b/packages/zarr-metadata/tests/model/test_construction.py index 037266de42..0e7d804621 100644 --- a/packages/zarr-metadata/tests/model/test_construction.py +++ b/packages/zarr-metadata/tests/model/test_construction.py @@ -31,8 +31,8 @@ from zarr_metadata.v3.codec.gzip import GZIP_CODEC, GzipCodecConfiguration from zarr_metadata.v3.data_type.int8 import INT8_DATA_TYPE from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, CodecDefinition, ) @@ -195,8 +195,8 @@ def counted( yield from () counting = dataclasses.replace(INT8_DATA_TYPE, fill_value_rules=counted) - context = CORE_AND_EXTENSIONS.update( - data_types={**CORE_AND_EXTENSIONS.data_types, "int8": counting} + context = CORE_AND_EXTENSIONS_CONTEXT.update( + data_types={**CORE_AND_EXTENSIONS_CONTEXT.data_types, "int8": counting} ) model = ZarrV3ArrayMetadata.create_default(context=context, data_type="int8", fill_value=3) assert values == [3] @@ -324,7 +324,9 @@ def test_error_update_takes_fields_as_json() -> None: def test_error_update_takes_no_refused_field() -> None: """A field refused by its definition holds JSON its definition does not read, so it has no JSON for `update` to write: given in a member, it is refused with a `TypeError` that says to give the member as JSON.""" - refused, _ = resolve({"name": "gzip", "configuration": {"level": 99}}, CodecDefinition, CORE) + refused, _ = resolve( + {"name": "gzip", "configuration": {"level": 99}}, CodecDefinition, CORE_CONTEXT + ) assert isinstance(refused, RefusedField) with pytest.raises(TypeError, match="a refused field has no JSON"): ARRAY.update(codecs=(ARRAY.to_json()["codecs"][0], refused)) # pyright: ignore[reportArgumentType] @@ -456,9 +458,9 @@ def counting(configuration: GzipCodecConfiguration) -> GzipCodecConfiguration: asked.append(1) return configuration - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( codecs={ - **CORE_AND_EXTENSIONS.codecs, + **CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": dataclasses.replace(GZIP_CODEC, canonical=counting), } ) @@ -489,9 +491,9 @@ def test_error_a_canonical_that_raises_is_raised_when_the_model_is_first_compare def refuses(configuration: GzipCodecConfiguration) -> GzipCodecConfiguration: raise ValueError("no") - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( codecs={ - **CORE_AND_EXTENSIONS.codecs, + **CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": dataclasses.replace(GZIP_CODEC, canonical=refuses), } ) diff --git a/packages/zarr-metadata/tests/model/test_extension_points.py b/packages/zarr-metadata/tests/model/test_extension_points.py index 063338e160..38d738f620 100644 --- a/packages/zarr-metadata/tests/model/test_extension_points.py +++ b/packages/zarr-metadata/tests/model/test_extension_points.py @@ -2,7 +2,7 @@ The model checks a document's structure itself. The configuration of a codec, data type, chunk grid or chunk key encoding is checked by its -definition. The definition is found in a scope (`CORE_AND_EXTENSIONS` +definition. The definition is found in a scope (`CORE_AND_EXTENSIONS_CONTEXT` unless a caller passes another), and problems are reported at the field's location in the document. A name that no definition in the scope claims is not checked. @@ -25,8 +25,8 @@ ) from zarr_metadata.problems import MetadataValidationError from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, CodecDefinition, ) @@ -63,21 +63,24 @@ class LenientGzipConfiguration(TypedDict, closed=True): @pytest.mark.parametrize( ("document", "context"), [ - (_document(), CORE_AND_EXTENSIONS), + (_document(), CORE_AND_EXTENSIONS_CONTEXT), ( _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 5}}]), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, ), ( _document(codecs=[BYTES, {"name": "acme.lz9", "configuration": {"anything": 1}}]), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, ), - (_document(codecs=[BYTES, {"name": "zstd", "configuration": {"level": 99}}]), CORE), + (_document(codecs=[BYTES, {"name": "zstd", "configuration": {"level": 99}}]), CORE_CONTEXT), ( _document(codecs=[BYTES, {"name": "gzip", "configuration": {"level": 99}}]), - CORE.update(codecs={**CORE.codecs, "gzip": LENIENT_GZIP}), + CORE_CONTEXT.update(codecs={**CORE_CONTEXT.codecs, "gzip": LENIENT_GZIP}), + ), + ( + dict(ZarrV3ArrayMetadata.create_default(shape=(0, 4)).to_json()), + CORE_AND_EXTENSIONS_CONTEXT, ), - (dict(ZarrV3ArrayMetadata.create_default(shape=(0, 4)).to_json()), CORE_AND_EXTENSIONS), ], ids=[ "default", @@ -173,7 +176,7 @@ def test_error_an_array_in_consolidated_metadata() -> None: def test_an_array_in_consolidated_metadata_reads_in_the_scope_it_is_read_in() -> None: """An array listed in consolidated metadata is read in the scope the member is read in: a lenient `gzip` accepts the level the core `gzip` refuses, and the model round-trips in that scope.""" member = {"kind": "inline", "must_understand": False, "metadata": {"a": BAD_GZIP_ARRAY}} - lenient = CORE.update(codecs={**CORE.codecs, "gzip": LENIENT_GZIP}) + lenient = CORE_CONTEXT.update(codecs={**CORE_CONTEXT.codecs, "gzip": LENIENT_GZIP}) reading = read_consolidated_metadata_v3(member, context=lenient) assert reading.problems == () assert reading.metadata is not None diff --git a/packages/zarr-metadata/tests/model/test_extra_fields.py b/packages/zarr-metadata/tests/model/test_extra_fields.py index 86684ce928..06982cc252 100644 --- a/packages/zarr-metadata/tests/model/test_extra_fields.py +++ b/packages/zarr-metadata/tests/model/test_extra_fields.py @@ -23,7 +23,7 @@ from zarr_metadata.problems import MetadataValidationError from zarr_metadata.typed_json import UserJSONValue from zarr_metadata.v3.definition import ( - CORE, + CORE_CONTEXT, ArrayExtraFieldDefinition, GroupExtraFieldDefinition, ZarrV3Context, @@ -51,8 +51,8 @@ class AcmeNoteConfiguration(TypedDict, closed=True): ACME_NOTE = ArrayExtraFieldDefinition(name="acme.note", configuration=AcmeNoteConfiguration) """A reader's own definition of the `acme.note` member of an array, whose numbers must be finite.""" -CONTEXT = CORE.update( - array_extra_fields=[*CORE.array_extra_fields.values(), ACME_TILING, ACME_NOTE] +CONTEXT = CORE_CONTEXT.update( + array_extra_fields=[*CORE_CONTEXT.array_extra_fields.values(), ACME_TILING, ACME_NOTE] ) ARRAY: dict[str, Any] = dict(ZarrV3ArrayMetadata.create_default(shape=(4,)).to_json()) @@ -66,10 +66,10 @@ class AcmeNoteConfiguration(TypedDict, closed=True): ("document", "context", "claimed", "must_understand"), [ ({**ARRAY, "acme.tiling": TILING}, CONTEXT, {"acme.tiling"}, {}), - ({**ARRAY, "acme.tiling": TILING}, CORE, set(), {"acme.tiling": TILING}), + ({**ARRAY, "acme.tiling": TILING}, CORE_CONTEXT, set(), {"acme.tiling": TILING}), ({**ARRAY, "acme.other": {"must_understand": False}}, CONTEXT, set(), {}), ({**ARRAY, "acme.other": 5}, CONTEXT, set(), {"acme.other": 5}), - ({**GROUP, "consolidated_metadata": INLINE}, CORE, {"consolidated_metadata"}, {}), + ({**GROUP, "consolidated_metadata": INLINE}, CORE_CONTEXT, {"consolidated_metadata"}, {}), ({**GROUP, "consolidated_metadata": INLINE}, ZarrV3Context(), set(), {}), ( {**GROUP, "consolidated_metadata": None}, @@ -77,8 +77,13 @@ class AcmeNoteConfiguration(TypedDict, closed=True): set(), {"consolidated_metadata": None}, ), - ({**ARRAY, "consolidated_metadata": INLINE}, CORE, set(), {}), - ({**GROUP, "consolidated_metadata": NAN_LISTING}, CORE, {"consolidated_metadata"}, {}), + ({**ARRAY, "consolidated_metadata": INLINE}, CORE_CONTEXT, set(), {}), + ( + {**GROUP, "consolidated_metadata": NAN_LISTING}, + CORE_CONTEXT, + {"consolidated_metadata"}, + {}, + ), ({**GROUP, "consolidated_metadata": NAN_LISTING}, ZarrV3Context(), set(), {}), ({**ARRAY, "acme.other": {"x": math.inf}}, CONTEXT, set(), {"acme.other": {"x": math.inf}}), ], @@ -102,7 +107,7 @@ def test_a_scope_reads_the_extra_fields_it_claims( claimed: set[str], must_understand: dict[str, object], ) -> None: - """Each member the spec does not define is an extra field. A member whose key a definition in scope claims is an `AcceptedField`, and a reader understands it. Any other member is an `UnclaimedField`, kept as written, and `must_understand_fields` lists it unless it says `"must_understand": false`. Nothing reads an unclaimed member, so like attributes it may hold any number, finite or not. A definition of a group's extra field does not read a member of an array, and `CORE` reads a group's `consolidated_metadata`. The document is written back as it was read.""" + """Each member the spec does not define is an extra field. A member whose key a definition in scope claims is an `AcceptedField`, and a reader understands it. Any other member is an `UnclaimedField`, kept as written, and `must_understand_fields` lists it unless it says `"must_understand": false`. Nothing reads an unclaimed member, so like attributes it may hold any number, finite or not. A definition of a group's extra field does not read a member of an array, and `CORE_CONTEXT` reads a group's `consolidated_metadata`. The document is written back as it was read.""" model: ZarrV3ArrayMetadata | ZarrV3GroupMetadata if document["node_type"] == "array": model = ZarrV3ArrayMetadata(document, context=context) @@ -167,16 +172,16 @@ def test_an_extra_field_is_a_field_of_the_reading() -> None: def test_the_simplest_spelling_of_an_extra_field_is_plain_json() -> None: - """`canonical_of` returns plain JSON for an extra field, with no read-only views inside, so `json.dumps` writes it. `CORE` claims a group's `consolidated_metadata`, whose configuration holds objects with no nested field.""" - model = ZarrV3GroupMetadata({**GROUP, "consolidated_metadata": INLINE}, context=CORE) + """`canonical_of` returns plain JSON for an extra field, with no read-only views inside, so `json.dumps` writes it. `CORE_CONTEXT` claims a group's `consolidated_metadata`, whose configuration holds objects with no nested field.""" + model = ZarrV3GroupMetadata({**GROUP, "consolidated_metadata": INLINE}, context=CORE_CONTEXT) simplest = canonical_of(model.extra_fields["consolidated_metadata"], ()) assert json.loads(json.dumps(simplest)) == {"consolidated_metadata": INLINE} def test_error_a_claimed_extra_field_that_is_not_an_object() -> None: - """A definition reads a member whose value is an object. Any other value of a claimed member is a problem at the member's key, as `CORE` refuses a `consolidated_metadata` of null.""" + """A definition reads a member whose value is an object. Any other value of a claimed member is a problem at the member's key, as `CORE_CONTEXT` refuses a `consolidated_metadata` of null.""" with pytest.raises(MetadataValidationError) as raised: - ZarrV3GroupMetadata({**GROUP, "consolidated_metadata": None}, context=CORE) + ZarrV3GroupMetadata({**GROUP, "consolidated_metadata": None}, context=CORE_CONTEXT) assert [(p.loc, p.kind) for p in raised.value.problems] == [ (("consolidated_metadata",), "invalid_type") ] @@ -230,7 +235,7 @@ def test_error_resolve_a_member_the_spec_defines_as_an_extra_field( member: dict[str, Any], ) -> None: """`resolve` reads a member the spec defines, read as an extra field, as a `RefusedField` with one `invalid_value` problem that says the spec defines it, for an array and for a group.""" - read, problems = resolve(member, kind, CORE) + read, problems = resolve(member, kind, CORE_CONTEXT) assert isinstance(read, RefusedField) assert [(p.loc, p.kind) for p in problems] == [((), "invalid_value")] assert "the spec defines" in problems[0].message diff --git a/packages/zarr-metadata/tests/model/test_group.py b/packages/zarr-metadata/tests/model/test_group.py index 9f62600b74..7755b188f8 100644 --- a/packages/zarr-metadata/tests/model/test_group.py +++ b/packages/zarr-metadata/tests/model/test_group.py @@ -49,8 +49,8 @@ from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON from zarr_metadata.v3.codec.gzip import GZIP_CODEC, GzipCodecConfiguration from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, CodecDefinition, ) from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON @@ -148,7 +148,9 @@ def _v2_consolidated_problems(document: object) -> list[tuple[tuple[str | int, . def _v3_field_problems(field: object) -> list[tuple[tuple[str | int, ...], str]]: - return [(p.loc, p.kind) for p in resolve(field, CodecDefinition, CORE_AND_EXTENSIONS)[1]] + return [ + (p.loc, p.kind) for p in resolve(field, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[1] + ] @pytest.mark.parametrize( @@ -639,7 +641,9 @@ def test_error_a_consolidated_envelope_reports_the_members_it_lacks_in_its_order def test_group_update_keeps_the_documents_it_holds() -> None: """`update` does not read the consolidated documents again: a child read in an extended scope keeps each field as it was read, and the group keeps its scope.""" - context = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), ACME_X]) + context = CORE_AND_EXTENSIONS_CONTEXT.update( + codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_X] + ) child = ZarrV3ArrayMetadata.create_default( context=context, shape=(4,), codecs=(LITTLE, {"name": "acme.x"}) ) @@ -654,18 +658,18 @@ def test_group_update_keeps_the_documents_it_holds() -> None: def test_group_update_writes_the_consolidated_metadata_it_is_given() -> None: - """`update` takes consolidated metadata as a document writes it, or as a `ZarrV3ConsolidatedMetadata`, whose document it writes; the group holds the same member either way. `UNSET` removes it. `zstd` is an extension, so `CORE` leaves it unclaimed, and the member read in the group's scope holds it as unclaimed.""" + """`update` takes consolidated metadata as a document writes it, or as a `ZarrV3ConsolidatedMetadata`, whose document it writes; the group holds the same member either way. `UNSET` removes it. `zstd` is an extension, so `CORE_CONTEXT` leaves it unclaimed, and the member read in the group's scope holds it as unclaimed.""" zstd = {"name": "zstd", "configuration": {"level": 3, "checksum": False}} member = cast("Any", _inline(a=_array(codecs=[LITTLE, zstd]))) - held = ZarrV3ConsolidatedMetadata(member, context=CORE) + held = ZarrV3ConsolidatedMetadata(member, context=CORE_CONTEXT) child = held.metadata["a"] assert isinstance(child, ZarrV3ArrayMetadata) assert isinstance(child.codecs[1], UnclaimedField) - group = ZarrV3GroupMetadata.create_default(context=CORE) + group = ZarrV3GroupMetadata.create_default(context=CORE_CONTEXT) updated = group.update(consolidated_metadata=held) assert updated.consolidated_metadata == held.to_json() assert updated == group.update(consolidated_metadata=member) - assert ZarrV3ConsolidatedMetadata(updated.consolidated_metadata, context=CORE) == held + assert ZarrV3ConsolidatedMetadata(updated.consolidated_metadata, context=CORE_CONTEXT) == held removed = updated.update(consolidated_metadata=UNSET) assert removed.consolidated_metadata is UNSET @@ -690,7 +694,7 @@ def _fields_of_an_array(*at: str | int) -> list[tuple[str | int, ...]]: # the outer group also lists. The listed group's own listing is # checked against the outer one, and its documents are not models # of the outer member. The listed group's member is its extra field, - # which `CORE` reads as an envelope. + # which `CORE_CONTEXT` reads as an envelope. ( _inline(g=_group(consolidated_metadata=_inline(b=_array())), **{"g/b": _array()}), ["g", "g/b"], @@ -859,9 +863,9 @@ def counted( reads.append(configuration) yield from () - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( codecs={ - **CORE_AND_EXTENSIONS.codecs, + **CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": dataclasses.replace(GZIP_CODEC, rules=counted), } ) diff --git a/packages/zarr-metadata/tests/model/test_pair.py b/packages/zarr-metadata/tests/model/test_pair.py index 0f1b06341e..f551fa9c8d 100644 --- a/packages/zarr-metadata/tests/model/test_pair.py +++ b/packages/zarr-metadata/tests/model/test_pair.py @@ -34,8 +34,8 @@ from zarr_metadata.v3.codec.bytes import BYTES_CODEC from zarr_metadata.v3.codec.zstd import ZSTD_CODEC from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, CodecDefinition, ZarrV3Context, ) @@ -57,7 +57,7 @@ name="bytes", configuration=EmptyConfiguration, kind="array_bytes", size="static" ) """A private `bytes` definition: a different meaning under the core name, with no configuration.""" -PRIVATE = CORE.update(codecs={**CORE.codecs, "bytes": MY_BYTES}) +PRIVATE = CORE_CONTEXT.update(codecs={**CORE_CONTEXT.codecs, "bytes": MY_BYTES}) SPELLED_OUT: dict[str, Any] = { **ARRAY, "data_type": {"name": "uint8"}, @@ -69,8 +69,8 @@ @pytest.mark.parametrize( ("document", "context", "expected_context"), [ - (ARRAY, None, CORE_AND_EXTENSIONS), - (ARRAY, CORE, CORE), + (ARRAY, None, CORE_AND_EXTENSIONS_CONTEXT), + (ARRAY, CORE_CONTEXT, CORE_CONTEXT), (SPELLED_OUT, ZarrV3Context(), ZarrV3Context()), ], ids=["default-scope", "core", "empty-scope"], @@ -78,7 +78,7 @@ def test_a_model_is_its_document_read_in_its_scope( document: dict[str, Any], context: ZarrV3Context | None, expected_context: ZarrV3Context ) -> None: - """A model built from a document holds that document, as written and refined, and the scope it was read in. The scope is `CORE_AND_EXTENSIONS` when none is given.""" + """A model built from a document holds that document, as written and refined, and the scope it was read in. The scope is `CORE_AND_EXTENSIONS_CONTEXT` when none is given.""" model = ZarrV3ArrayMetadata(document, context=context) assert model.context == expected_context assert model.to_json() == refine_json(document)[0] @@ -141,7 +141,7 @@ def test_error_a_document_with_a_problem_is_refused_at_construction() -> None: ("left", "right", "equal"), [ (ZarrV3ArrayMetadata(ARRAY), ZarrV3ArrayMetadata(SPELLED_OUT), True), - (ZarrV3ArrayMetadata(ARRAY, context=CORE), ZarrV3ArrayMetadata(ARRAY), True), + (ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT), ZarrV3ArrayMetadata(ARRAY), True), (ZarrV3ArrayMetadata(ARRAY), ZarrV3ArrayMetadata(ARRAY, context=PRIVATE), False), (ZarrV3ArrayMetadata(ARRAY), ZarrV3ArrayMetadata(ARRAY, context=ZarrV3Context()), False), (ZarrV3ArrayMetadata(ARRAY), ZarrV3ArrayMetadata({**ARRAY, "attributes": {}}), True), @@ -158,7 +158,9 @@ def test_models_are_equal_by_what_their_documents_mean( @pytest.mark.parametrize( - "context", [None, CORE, PRIVATE, ZarrV3Context()], ids=["default", "core", "private", "empty"] + "context", + [None, CORE_CONTEXT, PRIVATE, ZarrV3Context()], + ids=["default", "core", "private", "empty"], ) def test_a_model_round_trips_through_its_document_in_its_scope( context: ZarrV3Context | None, @@ -183,7 +185,7 @@ def test_error_a_model_whose_scope_does_not_pickle_says_so() -> None: ) model = ZarrV3ArrayMetadata( {**ARRAY, "codecs": ["bytes", "acme.c"]}, - context=CORE.update(codecs=[*CORE.codecs.values(), local]), + context=CORE_CONTEXT.update(codecs=[*CORE_CONTEXT.codecs.values(), local]), ) with pytest.raises((pickle.PicklingError, AttributeError)): pickle.dumps(model) @@ -266,10 +268,10 @@ def test_error_update_refuses_a_document_with_a_problem() -> None: @pytest.mark.parametrize( ("model", "context", "gained"), [ - (ZarrV3ArrayMetadata(ARRAY, context=ZarrV3Context()), CORE, True), - (ZarrV3ArrayMetadata(ARRAY, context=CORE), CORE_AND_EXTENSIONS, False), - (ZarrV3ArrayMetadata(FLOAT, context=ZarrV3Context()), CORE, True), - (ZarrV3ArrayMetadata(SHARDED, context=ZarrV3Context()), CORE, True), + (ZarrV3ArrayMetadata(ARRAY, context=ZarrV3Context()), CORE_CONTEXT, True), + (ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT), CORE_AND_EXTENSIONS_CONTEXT, False), + (ZarrV3ArrayMetadata(FLOAT, context=ZarrV3Context()), CORE_CONTEXT, True), + (ZarrV3ArrayMetadata(SHARDED, context=ZarrV3Context()), CORE_CONTEXT, True), ], ids=["gain", "nothing-to-gain", "gain-data-type-with-fill-value", "gain-of-a-shard"], ) @@ -291,7 +293,7 @@ def test_refined_in_moves_a_model_up_the_order( ("model", "context"), [ (ZarrV3ArrayMetadata(ARRAY), PRIVATE), - (ZarrV3ArrayMetadata(ARRAY, context=CORE), ZarrV3Context()), + (ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT), ZarrV3Context()), ], ids=["conflict", "loss"], ) @@ -311,7 +313,7 @@ def test_with_context_reads_the_document_in_any_scope() -> None: assert private.context == PRIVATE assert private.codecs[0].definition == MY_BYTES assert isinstance(model.with_context(ZarrV3Context()).codecs[0], UnclaimedField) - assert model.with_context(None).context == CORE_AND_EXTENSIONS + assert model.with_context(None).context == CORE_AND_EXTENSIONS_CONTEXT def test_error_with_context_refuses_a_document_the_scope_reads_with_a_problem() -> None: @@ -321,7 +323,7 @@ def test_error_with_context_refuses_a_document_the_scope_reads_with_a_problem() context=ZarrV3Context(), ) with pytest.raises(MetadataValidationError): - loose.with_context(CORE) + loose.with_context(CORE_CONTEXT) @pytest.mark.parametrize( @@ -333,17 +335,17 @@ def test_error_with_context_refuses_a_document_the_scope_reads_with_a_problem() (ZarrV3ArrayMetadata(ARRAY, context=PRIVATE), ZarrV3ArrayMetadata(ARRAY), False), (ZarrV3ArrayMetadata({**ARRAY, "attributes": {"a": 1}}), ZarrV3ArrayMetadata(ARRAY), False), ( - ZarrV3ArrayMetadata(FLOAT, context=CORE), + ZarrV3ArrayMetadata(FLOAT, context=CORE_CONTEXT), ZarrV3ArrayMetadata(FLOAT, context=ZarrV3Context()), True, ), ( - ZarrV3ArrayMetadata(SHARDED, context=CORE), + ZarrV3ArrayMetadata(SHARDED, context=CORE_CONTEXT), ZarrV3ArrayMetadata(SHARDED, context=ZarrV3Context()), True, ), ( - ZarrV3ArrayMetadata(FLOAT, context=CORE), + ZarrV3ArrayMetadata(FLOAT, context=CORE_CONTEXT), ZarrV3ArrayMetadata({**FLOAT, "fill_value": "banana"}, context=ZarrV3Context()), False, ), @@ -393,7 +395,7 @@ def test_refines_orders_models_by_information( def test_a_group_holds_its_consolidated_metadata_as_written() -> None: """A group model holds its `consolidated_metadata` member as written, refined and read-only, and does not read the documents it lists. `ZarrV3ConsolidatedMetadata` reads them: each becomes a model in its scope, and writes its document as written.""" - group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE) + group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE_CONTEXT) member = group.consolidated_metadata assert member == refine_json(MEMBER)[0] with pytest.raises(TypeError): @@ -415,8 +417,8 @@ def counted(configuration: object, nested: Nested) -> Iterator[ValidationProblem calls.append(1) return iter(()) - context = CORE.update( - codecs={**CORE.codecs, "bytes": dataclasses.replace(BYTES_CODEC, rules=counted)} + context = CORE_CONTEXT.update( + codecs={**CORE_CONTEXT.codecs, "bytes": dataclasses.replace(BYTES_CODEC, rules=counted)} ) ZarrV3GroupMetadata(CONSOLIDATED, context=context) assert calls == [] @@ -432,7 +434,11 @@ def _listing(a: dict[str, Any]) -> dict[str, Any]: @pytest.mark.parametrize( ("left", "right", "equal"), [ - (ZarrV3GroupMetadata(CONSOLIDATED), ZarrV3GroupMetadata(CONSOLIDATED, context=CORE), True), + ( + ZarrV3GroupMetadata(CONSOLIDATED), + ZarrV3GroupMetadata(CONSOLIDATED, context=CORE_CONTEXT), + True, + ), ( ZarrV3GroupMetadata(CONSOLIDATED), ZarrV3GroupMetadata(CONSOLIDATED, context=PRIVATE), @@ -446,7 +452,7 @@ def _listing(a: dict[str, Any]) -> dict[str, Any]: ), ( ZarrV3ConsolidatedMetadata(MEMBER), - ZarrV3ConsolidatedMetadata(MEMBER, context=CORE), + ZarrV3ConsolidatedMetadata(MEMBER, context=CORE_CONTEXT), True, ), ( @@ -484,7 +490,7 @@ def test_groups_and_consolidated_metadata_are_equal_by_what_they_read( @pytest.mark.parametrize("document", [GROUP, CONSOLIDATED], ids=["group", "consolidated"]) def test_a_group_round_trips_through_its_document_in_its_scope(document: dict[str, Any]) -> None: """`from_json(g.to_json(), context=g.context) == g`, and pickle preserves both the document and the scope.""" - group = ZarrV3GroupMetadata(document, context=CORE) + group = ZarrV3GroupMetadata(document, context=CORE_CONTEXT) assert ZarrV3GroupMetadata(group.to_json(), context=group.context) == group assert pickle.loads(pickle.dumps(group)) == group @@ -494,18 +500,18 @@ def test_group_update_with_context_and_refined_in_behave_as_the_arrays_do() -> N group = ZarrV3GroupMetadata(CONSOLIDATED, context=ZarrV3Context()) assert group.update(attributes={"g": 2}).consolidated_metadata == group.consolidated_metadata assert isinstance(group.extra_fields["consolidated_metadata"], UnclaimedField) - for moved in (group.refined_in(CORE), group.with_context(PRIVATE)): + for moved in (group.refined_in(CORE_CONTEXT), group.with_context(PRIVATE)): assert isinstance(moved.extra_fields["consolidated_metadata"], AcceptedField) assert moved.consolidated_metadata == group.consolidated_metadata assert moved.refines(group) assert not group.refines(moved) assert moved != group - assert group.refined_in(CORE).context == CORE + assert group.refined_in(CORE_CONTEXT).context == CORE_CONTEXT def test_error_group_refined_in_refuses_a_scope_that_loses_an_extra_field() -> None: - """A group read in `CORE` reads its `consolidated_metadata` as the envelope. A scope that leaves the member unclaimed would lose that reading, so `refined_in` raises `ScopeConflictError`, located at the member.""" - group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE) + """A group read in `CORE_CONTEXT` reads its `consolidated_metadata` as the envelope. A scope that leaves the member unclaimed would lose that reading, so `refined_in` raises `ScopeConflictError`, located at the member.""" + group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE_CONTEXT) with pytest.raises(ScopeConflictError) as raised: group.refined_in(ZarrV3Context()) assert [conflict.loc for conflict in raised.value.conflicts] == [("consolidated_metadata",)] @@ -514,9 +520,9 @@ def test_error_group_refined_in_refuses_a_scope_that_loses_an_extra_field() -> N def test_consolidated_metadata_with_context_and_refined_in_behave_as_the_arrays_do() -> None: """`refined_in` moves every document up the order. `with_context` reads every document in another scope.""" held = ZarrV3ConsolidatedMetadata(MEMBER, context=ZarrV3Context()) - refined = held.refined_in(CORE) + refined = held.refined_in(CORE_CONTEXT) assert refined.refines(held) - assert refined.context == CORE + assert refined.context == CORE_CONTEXT array = refined.metadata["a"] assert isinstance(array, ZarrV3ArrayMetadata) assert isinstance(array.codecs[0], AcceptedField) @@ -527,7 +533,7 @@ def test_consolidated_metadata_with_context_and_refined_in_behave_as_the_arrays_ def test_error_consolidated_metadata_refined_in_refuses_a_conflict() -> None: """`refined_in` raises `ScopeConflictError` for a scope that reads a name of a listed document by another definition, and locates each conflict under that document's path.""" - held = ZarrV3ConsolidatedMetadata(MEMBER, context=CORE) + held = ZarrV3ConsolidatedMetadata(MEMBER, context=CORE_CONTEXT) with pytest.raises(ScopeConflictError) as raised: held.refined_in(PRIVATE) assert {conflict.loc for conflict in raised.value.conflicts} == { @@ -538,12 +544,12 @@ def test_error_consolidated_metadata_refined_in_refuses_a_conflict() -> None: def test_consolidated_metadata_reads_on_its_own() -> None: """`ZarrV3ConsolidatedMetadata(member, context)` reads a member as written, or as a group holds it, with each document as a model in that scope. `read_consolidated_metadata_v3` returns the reading of the same model.""" - held = ZarrV3ConsolidatedMetadata(MEMBER, context=CORE) - group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE) - assert held == ZarrV3ConsolidatedMetadata(group.consolidated_metadata, context=CORE) + held = ZarrV3ConsolidatedMetadata(MEMBER, context=CORE_CONTEXT) + group = ZarrV3GroupMetadata(CONSOLIDATED, context=CORE_CONTEXT) + assert held == ZarrV3ConsolidatedMetadata(group.consolidated_metadata, context=CORE_CONTEXT) assert held.to_json() == refine_json(MEMBER)[0] assert ZarrV3ConsolidatedMetadata.from_json(MEMBER) == ZarrV3ConsolidatedMetadata(MEMBER) - reading = read_consolidated_metadata_v3(MEMBER, context=CORE) + reading = read_consolidated_metadata_v3(MEMBER, context=CORE_CONTEXT) assert reading.problems == () assert reading.metadata == held assert held.reading.metadata is held @@ -575,13 +581,13 @@ def test_error_refined_in_refuses_a_gain_that_surfaces_a_problem() -> None: """A scope that claims a name the current scope left unclaimed may refuse what was written under that name. Then `refined_in` raises `MetadataValidationError`, because the document has a problem in that scope.""" loose = ZarrV3ArrayMetadata({**ARRAY, "fill_value": "banana"}, context=ZarrV3Context()) with pytest.raises(MetadataValidationError) as raised: - loose.refined_in(CORE) + loose.refined_in(CORE_CONTEXT) assert [problem.loc for problem in raised.value.problems] == [("fill_value",)] def test_a_scope_conflict_says_where_each_conflict_sits() -> None: """`refined_in` reports each conflict with the field's location in the document, the same way a problem is located. For example, a loss of `bytes` is reported at `codecs.0`, including inside the shard.""" - model = ZarrV3ArrayMetadata(SHARDED, context=CORE) + model = ZarrV3ArrayMetadata(SHARDED, context=CORE_CONTEXT) with pytest.raises(ScopeConflictError) as raised: model.refined_in(ZarrV3Context()) assert sorted((conflict.key[1], conflict.loc) for conflict in raised.value.conflicts) == [ @@ -598,8 +604,8 @@ def test_a_scope_conflict_says_where_each_conflict_sits() -> None: @pytest.mark.parametrize( "build", [ - lambda: ZarrV3ConsolidatedMetadata(MEMBER, context=CORE_AND_EXTENSIONS), - lambda: read_consolidated_metadata_v3(MEMBER, context=CORE_AND_EXTENSIONS).metadata, + lambda: ZarrV3ConsolidatedMetadata(MEMBER, context=CORE_AND_EXTENSIONS_CONTEXT), + lambda: read_consolidated_metadata_v3(MEMBER, context=CORE_AND_EXTENSIONS_CONTEXT).metadata, ], ids=["constructor", "reader"], ) @@ -612,7 +618,7 @@ def test_with_context_moves_every_listed_model_without_reading_again( assert held.reading.metadata is held for node in held.metadata.values(): assert node.reading.metadata is node - context = CORE.update(codecs=[*CORE.codecs.values(), ZSTD_CODEC]) + context = CORE_CONTEXT.update(codecs=[*CORE_CONTEXT.codecs.values(), ZSTD_CODEC]) moved = held.with_context(context) assert moved.context is context before = held.metadata["a"] diff --git a/packages/zarr-metadata/tests/model/test_pydantic_module.py b/packages/zarr-metadata/tests/model/test_pydantic_module.py index 1458f8be7a..6c14f4b744 100644 --- a/packages/zarr-metadata/tests/model/test_pydantic_module.py +++ b/packages/zarr-metadata/tests/model/test_pydantic_module.py @@ -34,7 +34,7 @@ from zarr_metadata.v2.definition import CORE_V2, ZarrV2Context from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON from zarr_metadata.v3.codec.gzip import GZIP_CODEC -from zarr_metadata.v3.definition import CORE, CORE_AND_EXTENSIONS, ZarrV3Context +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CORE_CONTEXT, ZarrV3Context V3_ARRAY_DOC = dict(ZarrV3ArrayMetadata.create_default(shape=(4,)).to_json()) V2_ARRAY_DOC = dict(ZarrV2ArrayMetadata.create_default(shape=(4,), chunks=(2,)).to_json()) @@ -314,8 +314,8 @@ class Manifest(BaseModel): ("context", "read"), [ (None, True), - (CORE, False), - ({zmp.CONTEXT_KEY: CORE}, False), + (CORE_CONTEXT, False), + ({zmp.CONTEXT_KEY: CORE_CONTEXT}, False), ({"another validator's": 1}, True), ], ids=["none", "a-scope", "a-mapping-holding-one", "a-mapping-holding-none"], @@ -323,24 +323,28 @@ class Manifest(BaseModel): def test_a_v3_field_type_reads_in_the_scope_the_validation_context_holds( context: object, read: bool ) -> None: - """The field type reads in the scope from the validation context, which pydantic passes to every validator. `zstd` is an extension, so `CORE` leaves it unclaimed.""" + """The field type reads in the scope from the validation context, which pydantic passes to every validator. `zstd` is an extension, so `CORE_CONTEXT` leaves it unclaimed.""" model = TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python(_WITH_ZSTD, context=context) assert isinstance(model.codecs[1], AcceptedField) is read class _Bound(BaseModel): - """A model whose field is read in `CORE`, whatever the validation context holds.""" + """A model whose field is read in `CORE_CONTEXT`, whatever the validation context holds.""" - metadata: Annotated[ZarrV3ArrayMetadata, zmp.InScope(CORE)] + metadata: Annotated[ZarrV3ArrayMetadata, zmp.InScope(CORE_CONTEXT)] def test_a_field_bound_to_a_scope_reads_in_that_scope() -> None: - """`Annotated[ZarrV3ArrayMetadata, InScope(CORE)]` reads a document in `CORE`, whatever the validation context holds, and reads an instance again in `CORE`: an instance read in `CORE` with `zstd` unclaimed stays as it is. The field writes the document as written, and a type checker sees the model class.""" - for context in (None, CORE_AND_EXTENSIONS, {zmp.CONTEXT_KEY: CORE_AND_EXTENSIONS}): + """`Annotated[ZarrV3ArrayMetadata, InScope(CORE_CONTEXT)]` reads a document in `CORE_CONTEXT`, whatever the validation context holds, and reads an instance again in `CORE_CONTEXT`: an instance read in `CORE_CONTEXT` with `zstd` unclaimed stays as it is. The field writes the document as written, and a type checker sees the model class.""" + for context in ( + None, + CORE_AND_EXTENSIONS_CONTEXT, + {zmp.CONTEXT_KEY: CORE_AND_EXTENSIONS_CONTEXT}, + ): read = _Bound.model_validate({"metadata": _WITH_ZSTD}, context=context).metadata - assert read.context == CORE + assert read.context == CORE_CONTEXT assert isinstance(read.codecs[1], UnclaimedField) - held = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE) + held = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_CONTEXT) assert _Bound(metadata=held).metadata == held assert _Bound(metadata=held).model_dump()["metadata"] == held.to_json() @@ -349,10 +353,10 @@ def test_a_field_bound_to_a_scope_reads_in_that_scope() -> None: @pytest.mark.parametrize( ("bound", "field_type"), [ - (Annotated[ZarrV3ArrayMetadata, zmp.InScope(CORE)], zmp.ZarrV3ArrayMetadataField), - (Annotated[ZarrV3GroupMetadata, zmp.InScope(CORE)], zmp.ZarrV3GroupMetadataField), + (Annotated[ZarrV3ArrayMetadata, zmp.InScope(CORE_CONTEXT)], zmp.ZarrV3ArrayMetadataField), + (Annotated[ZarrV3GroupMetadata, zmp.InScope(CORE_CONTEXT)], zmp.ZarrV3GroupMetadataField), ( - Annotated[ZarrV3ConsolidatedMetadata, zmp.InScope(CORE)], + Annotated[ZarrV3ConsolidatedMetadata, zmp.InScope(CORE_CONTEXT)], zmp.ZarrV3ConsolidatedMetadataField, ), (Annotated[ZarrV2ArrayMetadata, zmp.InScope(CORE_V2)], zmp.ZarrV2ArrayMetadataField), @@ -383,9 +387,9 @@ def test_a_field_bound_to_a_scope_has_the_schema_of_its_field_type( def test_error_a_field_bound_to_a_scope_refuses_an_instance_it_reads_differently() -> None: - """An instance read where `zstd` is claimed would lose that meaning in `CORE`, so a field bound to `CORE` refuses it with a `ValidationError`, rather than passing it through in its own scope.""" + """An instance read where `zstd` is claimed would lose that meaning in `CORE_CONTEXT`, so a field bound to `CORE_CONTEXT` refuses it with a `ValidationError`, rather than passing it through in its own scope.""" with pytest.raises(ValidationError): - _Bound(metadata=ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS)) + _Bound(metadata=ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS_CONTEXT)) def test_error_a_field_bound_to_a_scope_of_the_other_format() -> None: @@ -397,10 +401,10 @@ def test_error_a_field_bound_to_a_scope_of_the_other_format() -> None: def test_error_a_scope_given_to_validation_reads_an_instance_in_it() -> None: """A scope given in the validation context reads an instance again in that scope, as a bound field does, so an instance that would lose a meaning there is refused. With no scope given, an instance passes through unchanged.""" adapter = TypeAdapter(zmp.ZarrV3ArrayMetadataField) - instance = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS) + instance = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS_CONTEXT) assert adapter.validate_python(instance) is instance with pytest.raises(ValidationError): - adapter.validate_python(instance, context=CORE) + adapter.validate_python(instance, context=CORE_CONTEXT) _READ_IN_SCOPE = [ @@ -430,7 +434,7 @@ def test_error_an_instance_a_scope_finds_problems_in_reports_each_one(validate: ) instance = ZarrV3ArrayMetadata(document, context=ZarrV3Context()) with pytest.raises(ValidationError) as raised: - validate(instance, CORE_AND_EXTENSIONS) + validate(instance, CORE_AND_EXTENSIONS_CONTEXT) assert [(line["type"], line["loc"], line["input"]) for line in raised.value.errors()] == [ ("invalid_value", ("codecs", 1, "configuration", "level"), 99), ("invalid_type", ("codecs", 2, "configuration", "level"), "x"), @@ -439,10 +443,10 @@ def test_error_an_instance_a_scope_finds_problems_in_reports_each_one(validate: @pytest.mark.parametrize("validate", _READ_IN_SCOPE) def test_error_an_instance_a_scope_reads_differently_reports_each_conflict(validate: Any) -> None: # noqa: ANN401 - a validating function - """An instance read where `zstd` is claimed would lose that meaning in `CORE`. pydantic gets one `scope_conflict` line error per conflict, at the field it is in, with that field as written as its input, and a message that names the codec.""" - instance = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS) + """An instance read where `zstd` is claimed would lose that meaning in `CORE_CONTEXT`. pydantic gets one `scope_conflict` line error per conflict, at the field it is in, with that field as written as its input, and a message that names the codec.""" + instance = ZarrV3ArrayMetadata(_WITH_ZSTD, context=CORE_AND_EXTENSIONS_CONTEXT) with pytest.raises(ValidationError) as raised: - validate(instance, CORE) + validate(instance, CORE_CONTEXT) (line,) = raised.value.errors() assert (line["type"], line["loc"]) == ("scope_conflict", ("codecs", 1)) assert line["input"] == instance.to_json()["codecs"][1] @@ -453,7 +457,7 @@ def test_error_an_instance_a_scope_reads_differently_reports_each_conflict(valid _V3_LISTING_WITH_ZSTD = ZarrV3ConsolidatedMetadata( - {**V3_CONSOLIDATED_DOC, "metadata": {"a": _WITH_ZSTD}}, context=CORE_AND_EXTENSIONS + {**V3_CONSOLIDATED_DOC, "metadata": {"a": _WITH_ZSTD}}, context=CORE_AND_EXTENSIONS_CONTEXT ) _V2_LISTING_WITH_ZLIB = ZarrV2ConsolidatedMetadata( { @@ -473,7 +477,7 @@ def test_error_an_instance_a_scope_reads_differently_reports_each_conflict(valid [ pytest.param( lambda value: TypeAdapter(zmp.ZarrV3ConsolidatedMetadataField).validate_python( - value, context=CORE + value, context=CORE_CONTEXT ), _V3_LISTING_WITH_ZSTD, ("metadata", "a", "codecs", 1), @@ -483,7 +487,7 @@ def test_error_an_instance_a_scope_reads_differently_reports_each_conflict(valid ), pytest.param( lambda value: TypeAdapter( - Annotated[ZarrV3ConsolidatedMetadata, zmp.InScope(CORE)] + Annotated[ZarrV3ConsolidatedMetadata, zmp.InScope(CORE_CONTEXT)] ).validate_python(value), _V3_LISTING_WITH_ZSTD, ("metadata", "a", "codecs", 1), @@ -532,8 +536,8 @@ def test_error_a_consolidated_instance_a_scope_reads_differently_reports_each_co @pytest.mark.parametrize( "annotation", [ - Annotated[ZarrV3ArrayMetadata | None, zmp.InScope(CORE)], - Annotated[dict[str, Any], zmp.InScope(CORE)], + Annotated[ZarrV3ArrayMetadata | None, zmp.InScope(CORE_CONTEXT)], + Annotated[dict[str, Any], zmp.InScope(CORE_CONTEXT)], ], ids=["an-optional-model", "a-dict"], ) @@ -553,14 +557,14 @@ def test_error_a_validation_context_names_a_key_of_this_package_it_does_not_have """A mapping validation context with a key that starts like this package's keys, but is not one of them, is a `TypeError`, so a misspelled key is not read as no scope at all.""" with pytest.raises(TypeError, match="zarr_metadata_contxt"): TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python( - _WITH_ZSTD, context={"zarr_metadata_contxt": CORE} + _WITH_ZSTD, context={"zarr_metadata_contxt": CORE_CONTEXT} ) def test_error_a_validation_context_holds_a_scope_that_is_not_one() -> None: with pytest.raises(TypeError, match=zmp.CONTEXT_KEY): TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python( - _WITH_ZSTD, context={zmp.CONTEXT_KEY: "CORE"} + _WITH_ZSTD, context={zmp.CONTEXT_KEY: "CORE_CONTEXT"} ) @@ -707,16 +711,19 @@ def test_each_format_reads_in_its_own_context_key() -> None: from zarr_metadata.definition import UnclaimedField from zarr_metadata.v2.data_type.scalar import UINT_DATA_TYPE_V2 from zarr_metadata.v2.definition import CORE_V2, ZarrV2Context - from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS + from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT adapter = TypeAdapter(zmp.ZarrV2ArrayMetadataField) doc = json.loads(json.dumps(V2_ARRAY_DOC)) small = ZarrV2Context(data_types=[UINT_DATA_TYPE_V2]) - v3_only = adapter.validate_python(doc, context={"zarr_metadata_context": CORE_AND_EXTENSIONS}) + v3_only = adapter.validate_python( + doc, context={"zarr_metadata_context": CORE_AND_EXTENSIONS_CONTEXT} + ) assert v3_only.context is CORE_V2 with pytest.raises(ValidationError): adapter.validate_python( - {**doc, "fill_value": "garbage"}, context={"zarr_metadata_context": CORE_AND_EXTENSIONS} + {**doc, "fill_value": "garbage"}, + context={"zarr_metadata_context": CORE_AND_EXTENSIONS_CONTEXT}, ) own = adapter.validate_python(doc, context={"zarr_metadata_context_v2": small}) assert own.context is small diff --git a/packages/zarr-metadata/tests/model/test_read_array_metadata.py b/packages/zarr-metadata/tests/model/test_read_array_metadata.py index 921aa19aae..7335a9abce 100644 --- a/packages/zarr-metadata/tests/model/test_read_array_metadata.py +++ b/packages/zarr-metadata/tests/model/test_read_array_metadata.py @@ -31,8 +31,8 @@ ) from zarr_metadata.problems import ValidationProblem from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, ChunkGridDefinition, ChunkKeyEncodingDefinition, CodecDefinition, @@ -82,13 +82,18 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o @pytest.mark.parametrize( ("document", "context", "fields", "handed"), [ - (_document(), CORE_AND_EXTENSIONS, [*POINTS, _codec(("codecs", 0))], [(frozenset({4}),)]), + ( + _document(), + CORE_AND_EXTENSIONS_CONTEXT, + [*POINTS, _codec(("codecs", 0))], + [(frozenset({4}),)], + ), # Each codec receives the chunk that the previous codec outputs. ( _document( (4, 6), codecs=[{"name": "transpose", "configuration": {"order": [1, 0]}}, "bytes"] ), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, [*POINTS, _codec(("codecs", 0)), _codec(("codecs", 1))], [(frozenset({4}), frozenset({6})), (frozenset({6}), frozenset({4}))], ), @@ -101,7 +106,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o chunk_grid={"name": "regular", "configuration": {"chunk_shape": [8, 8]}}, codecs=[_shard([4, 4], codecs=[LITTLE, ZSTD], index_codecs=[LITTLE, "crc32c"])], ), - CORE, + CORE_CONTEXT, [ *POINTS, _codec(("codecs", 0)), @@ -119,7 +124,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o chunk_grid={"name": "regular", "configuration": {"chunk_shape": [8, 8]}}, codecs=[_shard([4, 4], codecs=[_shard([2, 2])])], ), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, [ *POINTS, _codec(("codecs", 0)), @@ -139,7 +144,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o chunk_grid={"name": "regular", "configuration": {"chunk_shape": [8, 8]}}, codecs=[_shard([4, 4], index_location="middle", codecs=[LITTLE, ZSTD])], ), - CORE, + CORE_CONTEXT, [ *POINTS, _codec(("codecs", 0), RefusedField), @@ -165,7 +170,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o fill_value={"a": 0, "b": "anything"}, codecs=[LITTLE], ), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, [ POINTS[0], ( @@ -192,7 +197,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o codecs=["bytes", {"name": "gzip", "configuration": {"level": 99}}], storage_transformers=[{"name": "acme.transformer"}], ), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, [ POINTS[0], (("chunk_grid",), ChunkGridDefinition, UnclaimedField), @@ -206,7 +211,7 @@ def _shard(chunk_shape: list[int], codecs: list[object] | None = None, **more: o # A data type that is not JSON is refused. ( _document(data_type=float("nan")), - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, [ (("data_type",), DataTypeDefinition, RefusedField), *POINTS[1:], @@ -273,7 +278,7 @@ def _field_fields( resolved, problems = resolve( cast("dict[str, Any]", document)["codecs"][0], CodecDefinition, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, ("codecs", 0), ) return with_problems(fields_of(resolved, ("codecs", 0)), problems) diff --git a/packages/zarr-metadata/tests/model/test_scope_threading.py b/packages/zarr-metadata/tests/model/test_scope_threading.py index 1a8192d185..ba4a36f170 100644 --- a/packages/zarr-metadata/tests/model/test_scope_threading.py +++ b/packages/zarr-metadata/tests/model/test_scope_threading.py @@ -1,4 +1,4 @@ -"""Tests that every entry point reads in the scope it is given, and in `CORE_AND_EXTENSIONS` when given none.""" +"""Tests that every entry point reads in the scope it is given, and in `CORE_AND_EXTENSIONS_CONTEXT` when given none.""" from __future__ import annotations @@ -13,7 +13,7 @@ ZarrV3ArrayMetadata, ) from zarr_metadata.problems import MetadataValidationError -from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, ZarrV3Context +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, ZarrV3Context if TYPE_CHECKING: from collections.abc import Callable @@ -110,7 +110,7 @@ def test_every_entry_point_reads_in_the_scope_it_is_given(read: Callable[..., ob """Each entry point refuses a gzip level of 12 in the default scope (also selected by `None`), and accepts it in a scope that leaves gzip unclaimed. This shows that each entry point reads in the scope it is given.""" accepted = _accepts(read) assert accepted(EMPTY) is True - assert accepted(CORE_AND_EXTENSIONS) is False + assert accepted(CORE_AND_EXTENSIONS_CONTEXT) is False assert accepted(None) is False @@ -119,7 +119,7 @@ def test_a_group_reads_the_same_in_every_scope(read: Callable[..., object]) -> N """A group document has no fields, and a group does not read the documents its consolidated metadata lists. So each group entry point accepts a group that lists `BAD`, in every scope.""" accepted = _accepts(read) assert accepted(EMPTY) is True - assert accepted(CORE_AND_EXTENSIONS) is True + assert accepted(CORE_AND_EXTENSIONS_CONTEXT) is True assert accepted(None) is True @@ -169,4 +169,4 @@ def test_error_a_v3_store_reader_refuses_a_scope_of_another_format_before_decodi with pytest.raises(TypeError, match="format"): STORE_READERS[name](context=CORE_V2) with pytest.raises(MetadataValidationError): - STORE_READERS[name](context=CORE_AND_EXTENSIONS) + STORE_READERS[name](context=CORE_AND_EXTENSIONS_CONTEXT) diff --git a/packages/zarr-metadata/tests/model/test_scope_threading_v2.py b/packages/zarr-metadata/tests/model/test_scope_threading_v2.py index 677889ec50..ac84ae192e 100644 --- a/packages/zarr-metadata/tests/model/test_scope_threading_v2.py +++ b/packages/zarr-metadata/tests/model/test_scope_threading_v2.py @@ -14,7 +14,7 @@ CORE_V2, ZarrV2Context, ) -from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT if TYPE_CHECKING: from collections.abc import Callable @@ -127,7 +127,7 @@ def test_every_v2_entry_point_reads_in_the_scope_it_is_given(name: str) -> None: def test_error_every_v2_entry_point_refuses_a_scope_of_another_format(name: str) -> None: """A v2 entry point given a v3 scope raises `TypeError`. A scope reads documents of one format, and a v3 scope claims nothing that a v2 document contains.""" with pytest.raises(TypeError, match="format"): - ENTRY_POINTS[name](context=CORE_AND_EXTENSIONS) + ENTRY_POINTS[name](context=CORE_AND_EXTENSIONS_CONTEXT) NOT_JSON = b"{not json" @@ -156,6 +156,6 @@ def test_error_a_v2_store_reader_refuses_a_scope_of_another_format_before_decodi ) -> None: """A v2 entry point that reads a store's bytes raises `TypeError` for a v3 scope before it decodes the bytes, so the wrong scope is reported even when the bytes are not JSON. With a v2 scope, the same bytes raise `MetadataValidationError`.""" with pytest.raises(TypeError, match="format"): - STORE_READERS[name](context=CORE_AND_EXTENSIONS) + STORE_READERS[name](context=CORE_AND_EXTENSIONS_CONTEXT) with pytest.raises(MetadataValidationError): STORE_READERS[name](context=CORE_V2) diff --git a/packages/zarr-metadata/tests/model/test_unclaimed_fields.py b/packages/zarr-metadata/tests/model/test_unclaimed_fields.py index cef80f6c70..332ac4618f 100644 --- a/packages/zarr-metadata/tests/model/test_unclaimed_fields.py +++ b/packages/zarr-metadata/tests/model/test_unclaimed_fields.py @@ -16,7 +16,7 @@ ZarrV3GroupMetadata, ZarrV3NodeMetadata, ) -from zarr_metadata.v3 import CORE +from zarr_metadata.v3 import CORE_CONTEXT from zarr_metadata.v3.definition import ( ArrayExtraFieldDefinition, GroupExtraFieldDefinition, @@ -36,21 +36,21 @@ class _MirrorConfiguration(TypedDict, closed=True): must_understand: NotRequired[bool] -MIRRORING = CORE.update( +MIRRORING = CORE_CONTEXT.update( array_extra_fields=[ - *CORE.array_extra_fields.values(), + *CORE_CONTEXT.array_extra_fields.values(), ArrayExtraFieldDefinition(name="acme.mirror", configuration=_MirrorConfiguration), ] ) -"""`CORE`, with a definition of an `acme.mirror` member that holds a codec.""" +"""`CORE_CONTEXT`, with a definition of an `acme.mirror` member that holds a codec.""" -GROUP_MIRRORING = CORE.update( +GROUP_MIRRORING = CORE_CONTEXT.update( group_extra_fields=[ - *CORE.group_extra_fields.values(), + *CORE_CONTEXT.group_extra_fields.values(), GroupExtraFieldDefinition(name="acme.mirror", configuration=_MirrorConfiguration), ] ) -"""`CORE`, with a definition of an `acme.mirror` member of a group that holds a codec.""" +"""`CORE_CONTEXT`, with a definition of an `acme.mirror` member of a group that holds a codec.""" SHARD = { "name": "sharding_indexed", @@ -65,15 +65,23 @@ class _MirrorConfiguration(TypedDict, closed=True): @pytest.mark.parametrize( ("model", "unclaimed"), [ - (ZarrV3ArrayMetadata(ARRAY, context=CORE), set()), - (ZarrV3ArrayMetadata({**ARRAY, "codecs": ("bytes", ZSTD)}, context=CORE), {("codecs", 1)}), + (ZarrV3ArrayMetadata(ARRAY, context=CORE_CONTEXT), set()), ( - ZarrV3ArrayMetadata({**ARRAY, "codecs": (SHARD,)}, context=CORE), + ZarrV3ArrayMetadata({**ARRAY, "codecs": ("bytes", ZSTD)}, context=CORE_CONTEXT), + {("codecs", 1)}, + ), + ( + ZarrV3ArrayMetadata({**ARRAY, "codecs": (SHARD,)}, context=CORE_CONTEXT), {("codecs", 0, "configuration", "codecs", 1)}, ), - (ZarrV3ArrayMetadata({**ARRAY, "acme": {"x": 1}}, context=CORE), {("acme",)}), - (ZarrV3ArrayMetadata({**ARRAY, "acme": {"must_understand": False}}, context=CORE), set()), - (ZarrV3GroupMetadata({**GROUP, "acme": 1}, context=CORE), {("acme",)}), + (ZarrV3ArrayMetadata({**ARRAY, "acme": {"x": 1}}, context=CORE_CONTEXT), {("acme",)}), + ( + ZarrV3ArrayMetadata( + {**ARRAY, "acme": {"must_understand": False}}, context=CORE_CONTEXT + ), + set(), + ), + (ZarrV3GroupMetadata({**GROUP, "acme": 1}, context=CORE_CONTEXT), {("acme",)}), (ZarrV2ArrayMetadata({**V2_ARRAY, "compressor": {"id": "acme"}}), {("compressor",)}), (ZarrV2GroupMetadata.create_default(), set()), ( @@ -154,8 +162,8 @@ def test_a_model_that_refines_another_has_no_more_to_understand( wider: set[tuple[str | int, ...]], narrower: set[tuple[str | int, ...]], ) -> None: - """A scope that claims more of a document reads a model that refines the model of a scope that claims less, and its `unclaimed_fields` lie at or inside those of the model it refines, for an array and for a group. Here `CORE` leaves an unwaived `acme.mirror` member unclaimed as a whole; the scope that claims the member leaves at most the codec inside it, which lies inside the member, and a waived member is left out by both.""" - claimed_less = kind({**document, "acme.mirror": member}, context=CORE) + """A scope that claims more of a document reads a model that refines the model of a scope that claims less, and its `unclaimed_fields` lie at or inside those of the model it refines, for an array and for a group. Here `CORE_CONTEXT` leaves an unwaived `acme.mirror` member unclaimed as a whole; the scope that claims the member leaves at most the codec inside it, which lies inside the member, and a waived member is left out by both.""" + claimed_less = kind({**document, "acme.mirror": member}, context=CORE_CONTEXT) claimed_more = kind({**document, "acme.mirror": member}, context=context) assert claimed_more.refines(claimed_less) assert set(claimed_less.unclaimed_fields) == wider diff --git a/packages/zarr-metadata/tests/test_contexts.py b/packages/zarr-metadata/tests/test_contexts.py index ef8f408727..f312d2fdbf 100644 --- a/packages/zarr-metadata/tests/test_contexts.py +++ b/packages/zarr-metadata/tests/test_contexts.py @@ -26,7 +26,7 @@ ZarrV2ContextSlots, ZarrV2DataTypeDefinition, ) -from zarr_metadata.v3 import CORE, GZIP_CODEC, ZarrV3Context +from zarr_metadata.v3 import CORE_CONTEXT, GZIP_CODEC, ZarrV3Context from zarr_metadata.v3._context import _KINDS as V3_KINDS from zarr_metadata.v3._context import is_v3_kind from zarr_metadata.v3.definition import CodecDefinition, DataTypeDefinition, ZarrV3ContextSlots @@ -41,7 +41,7 @@ def _other(definition: Any) -> Any: # noqa: ANN401 - a definition of either for V2_ARRAY = ZarrV2ArrayMetadata.create_default().to_json() FORMATS = [ - pytest.param(ZarrV3Context, CORE, CodecDefinition, GZIP_CODEC, 3, id="v3"), + pytest.param(ZarrV3Context, CORE_CONTEXT, CodecDefinition, GZIP_CODEC, 3, id="v3"), pytest.param(ZarrV2Context, CORE_V2, ZarrV2CodecDefinition, GZIP_CODEC_V2, 2, id="v2"), ] @@ -234,7 +234,12 @@ def test_error_a_reader_refuses_what_is_not_a_scope_of_its_format( @pytest.mark.parametrize( ("core", "given", "zarr_format"), - [(CORE, "nope", 3), (CORE, CORE_V2, 3), (CORE_V2, "nope", 2), (CORE_V2, CORE, 2)], + [ + (CORE_CONTEXT, "nope", 3), + (CORE_CONTEXT, CORE_V2, 3), + (CORE_V2, "nope", 2), + (CORE_V2, CORE_CONTEXT, 2), + ], ids=["v3-not-a-scope", "v3-a-v2-scope", "v2-not-a-scope", "v2-a-v3-scope"], ) def test_error_union_refuses_what_is_not_a_scope_of_its_format( @@ -256,7 +261,7 @@ def test_each_declared_slot_has_a_view(cls: Any) -> None: # noqa: ANN401 - eith slots_type = ZarrV3ContextSlots if cls is ZarrV3Context else ZarrV2ContextSlots assert list(cls.kinds) == list(slots_type.__annotations__) assert set(cls.kinds.values()) == (V3_KINDS if cls is ZarrV3Context else V2_KINDS) - scope: Any = CORE if cls is ZarrV3Context else CORE_V2 + scope: Any = CORE_CONTEXT if cls is ZarrV3Context else CORE_V2 for slot, kind in cls.kinds.items(): assert getattr(scope, slot) == scope.definitions_of(kind) diff --git a/packages/zarr-metadata/tests/test_json_schema.py b/packages/zarr-metadata/tests/test_json_schema.py index 7145af7e0c..62ea9358d1 100644 --- a/packages/zarr-metadata/tests/test_json_schema.py +++ b/packages/zarr-metadata/tests/test_json_schema.py @@ -37,8 +37,9 @@ from zarr_metadata.v2.definition import CORE_V2 from zarr_metadata.v3.codec.gzip import GzipCodecConfiguration from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, + ZARR_PYTHON_CONTEXT, ChunkGridDefinition, ChunkKeyEncodingDefinition, CodecDefinition, @@ -472,7 +473,7 @@ def _changed(draw: st.DrawFn, value: object) -> object: return value -_CONTEXTS = (CORE, CORE_AND_EXTENSIONS, ZarrV3Context()) +_CONTEXTS = (CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT, ZARR_PYTHON_CONTEXT, ZarrV3Context()) _FIELD_BASES: tuple[tuple[str, object], ...] = ( *CASES, # Names that nothing claims, so nothing checks their configuration. @@ -498,12 +499,19 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat # it, and whether the scope reads it without a problem. Where the two # differ, a rule found a problem that the schema cannot express. FIELDS: list[tuple[str, object, type[ZarrV3Kind], ZarrV3Context, bool, bool]] = [ - ("object", {"name": "gzip", "configuration": {"level": 5}}, CodecDefinition, CORE, True, True), + ( + "object", + {"name": "gzip", "configuration": {"level": 5}}, + CodecDefinition, + CORE_CONTEXT, + True, + True, + ), ( "out-of-bounds", {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -511,17 +519,17 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "unknown-key", {"name": "gzip", "configuration": {"level": 5, "window": 15}}, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), - ("bare-name-needing-a-configuration", "gzip", CodecDefinition, CORE, False, False), - ("bare-name", "bytes", CodecDefinition, CORE, True, True), + ("bare-name-needing-a-configuration", "gzip", CodecDefinition, CORE_CONTEXT, False, False), + ("bare-name", "bytes", CodecDefinition, CORE_CONTEXT, True, True), ( "understood", {"name": "gzip", "configuration": {"level": 5}, "must_understand": True}, CodecDefinition, - CORE, + CORE_CONTEXT, True, True, ), @@ -529,7 +537,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "not-understood", {"name": "gzip", "configuration": {"level": 5}, "must_understand": False}, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -537,7 +545,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "stray-member", {"name": "gzip", "configuration": {"level": 5}, "version": 2}, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -545,16 +553,16 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "unclaimed", {"name": "zfpy", "configuration": {"mode": 4}}, CodecDefinition, - CORE, + CORE_CONTEXT, True, True, ), - ("unclaimed-bare", "zfpy", CodecDefinition, CORE, True, True), + ("unclaimed-bare", "zfpy", CodecDefinition, CORE_CONTEXT, True, True), ( "unclaimed-not-understood", {"name": "zfpy", "must_understand": False}, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -574,7 +582,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "configuration": {"cname": "lz4", "clevel": 1, "shuffle": "shuffle", "blocksize": 0}, }, CodecDefinition, - CORE, + CORE_CONTEXT, True, False, ), @@ -585,7 +593,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "configuration": {**_INDEXED, "index_codecs": ["bytes", "crc32c"]}, }, CodecDefinition, - CORE, + CORE_CONTEXT, True, True, ), @@ -599,7 +607,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat }, }, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -610,7 +618,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "configuration": {**_INDEXED, "index_codecs": ["bytes", "acme.sum"]}, }, CodecDefinition, - CORE, + CORE_CONTEXT, True, True, ), @@ -625,25 +633,32 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat }, }, CodecDefinition, - CORE, + CORE_CONTEXT, False, False, ), - ("raw-bits", "r16", DataTypeDefinition, CORE, True, True), - ("raw-bits-object", {"name": "r16", "configuration": {}}, DataTypeDefinition, CORE, True, True), + ("raw-bits", "r16", DataTypeDefinition, CORE_CONTEXT, True, True), + ( + "raw-bits-object", + {"name": "r16", "configuration": {}}, + DataTypeDefinition, + CORE_CONTEXT, + True, + True, + ), # The values a raw-bits name carries are not also written in a configuration. ( "raw-bits-configured", {"name": "r16", "configuration": {"bits": 16}}, DataTypeDefinition, - CORE, + CORE_CONTEXT, False, False, ), - ("raw-bits-of-a-size-the-spec-refuses", "r12", DataTypeDefinition, CORE, True, False), + ("raw-bits-of-a-size-the-spec-refuses", "r12", DataTypeDefinition, CORE_CONTEXT, True, False), # `r*` is the notation of the spec's table, not a name: `*` is not # allowed in a name. - ("raw-bits-notation", "r*", DataTypeDefinition, CORE, False, False), + ("raw-bits-notation", "r*", DataTypeDefinition, CORE_CONTEXT, False, False), # The pattern is matched to the end of the name, as the package matches # it, so a validator that matches as Python does must not accept a final # newline. It is also not a name, because a newline is not allowed in one. @@ -651,7 +666,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "raw-bits-and-a-newline", {"name": "r16\n", "configuration": {"x": 1}}, DataTypeDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -672,7 +687,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat }, }, DataTypeDefinition, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, False, False, ), @@ -680,7 +695,7 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "grid-out-of-bounds", {"name": "regular", "configuration": {"chunk_shape": [0]}}, ChunkGridDefinition, - CORE, + CORE_CONTEXT, False, False, ), @@ -688,11 +703,11 @@ def _field_validator(kind: type[ZarrV3Kind], context: int) -> Draft202012Validat "encoding", {"name": "v2", "configuration": {"separator": "/"}}, ChunkKeyEncodingDefinition, - CORE, + CORE_CONTEXT, True, True, ), - ("transformer", {"name": "acme.cache"}, StorageTransformerDefinition, CORE, True, True), + ("transformer", {"name": "acme.cache"}, StorageTransformerDefinition, CORE_CONTEXT, True, True), ] @@ -714,14 +729,14 @@ def test_field_json_schema_says_what_the_scope_reads( ("key", "field"), CASES, ids=[f"{key}:{index}" for index, (key, _) in enumerate(CASES)] ) def test_every_example_field_is_one_its_schema_accepts(key: str, field: object) -> None: - schema = field_json_schema_v3(KINDS[key.split(":")[0]], context=CORE_AND_EXTENSIONS) + schema = field_json_schema_v3(KINDS[key.split(":")[0]], context=ZARR_PYTHON_CONTEXT) assert Draft202012Validator(schema).is_valid(cast("Any", field)) @_PROPERTY @given(st.data()) def test_a_field_read_without_a_problem_is_one_its_schema_accepts(data: st.DataObject) -> None: - # Each example is changed in one place and read in one of three scopes. + # Each example is changed in one place and read in one of four scopes. # If the scope still reads it without a problem, the schema accepts it. key, example = data.draw(st.sampled_from(_FIELD_BASES), label="example") kind = KINDS[key.split(":")[0]] @@ -741,7 +756,7 @@ def test_error_field_json_schema_v3_refuses_a_scope_of_another_format() -> None: def test_field_json_schema_refuses_what_is_not_a_kind() -> None: with pytest.raises(TypeError, match="is not a kind of metadata"): - field_json_schema_v3(Definition, context=CORE) # pyright: ignore[reportArgumentType] + field_json_schema_v3(Definition, context=CORE_CONTEXT) # pyright: ignore[reportArgumentType] # --- a zarr.json ------------------------------------------------------------ @@ -889,7 +904,7 @@ def _node_validator(context: int) -> Draft202012Validator: @_PROPERTY @given(st.data()) def test_a_document_read_without_a_problem_is_one_its_schema_accepts(data: st.DataObject) -> None: - # Each document is changed in one place and read in one of three scopes. + # Each document is changed in one place and read in one of four scopes. # If `read_node_metadata_v3` finds no problem, the schema accepts it. base = data.draw(st.sampled_from(_BASES), label="base") context = data.draw(st.sampled_from(range(len(_CONTEXTS))), label="context") @@ -902,8 +917,8 @@ def test_a_document_read_without_a_problem_is_one_its_schema_accepts(data: st.Da @pytest.mark.parametrize( "context", - [CORE, CORE_AND_EXTENSIONS, ZarrV3Context()], - ids=["core", "core-and-extensions", "empty"], + [CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT, ZARR_PYTHON_CONTEXT, ZarrV3Context()], + ids=["core", "core-and-extensions", "zarr-python", "empty"], ) def test_every_schema_is_a_json_schema_written_the_same_each_time(context: ZarrV3Context) -> None: schema = node_metadata_json_schema_v3(context=context) @@ -959,12 +974,12 @@ def test_a_stray_member_beside_an_unclaimed_name_is_refused_by_the_schema() -> N # The schema rejects it, as the reader does: the envelope of an # unclaimed name holds only the name and its configuration. field = {"name": "zfpy", "version": 2} - assert [(p.loc, p.kind) for p in resolve(field, CodecDefinition, CORE)[1]] == [ + assert [(p.loc, p.kind) for p in resolve(field, CodecDefinition, CORE_CONTEXT)[1]] == [ (("version",), "unknown_key") ] - assert not Draft202012Validator(field_json_schema_v3(CodecDefinition, context=CORE)).is_valid( - field - ) + assert not Draft202012Validator( + field_json_schema_v3(CodecDefinition, context=CORE_CONTEXT) + ).is_valid(field) @pytest.mark.parametrize( @@ -1000,10 +1015,12 @@ def test_the_schema_names_an_extension_as_the_reader_does(name: str) -> None: # A name matches the spec's regex or is a URI. The schema's `pattern` # accepts exactly the names that `well_named` accepts. field = {"name": name} - _, problems = resolve(field, CodecDefinition, CORE) + _, problems = resolve(field, CodecDefinition, CORE_CONTEXT) accepted = len(problems) == 0 assert ( - Draft202012Validator(field_json_schema_v3(CodecDefinition, context=CORE)).is_valid(field) + Draft202012Validator(field_json_schema_v3(CodecDefinition, context=CORE_CONTEXT)).is_valid( + field + ) is accepted ) assert accepted is ( diff --git a/packages/zarr-metadata/tests/test_problem_data.py b/packages/zarr-metadata/tests/test_problem_data.py index c2ecafacb6..9a2f059183 100644 --- a/packages/zarr-metadata/tests/test_problem_data.py +++ b/packages/zarr-metadata/tests/test_problem_data.py @@ -50,7 +50,7 @@ ) from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, DataTypeDefinition, ) @@ -219,18 +219,18 @@ def _raised(read: Callable[[], object]) -> Sequence[ValidationProblem]: (lambda value: check(value, GzipLevelOnly)[1], {"level": 12, "extra": [1]}), (lambda value: GZIP_CODEC.read_configuration(value)[1], {"level": 12, "extra": [1]}), ( - lambda value: resolve(value, CodecDefinition, CORE_AND_EXTENSIONS)[1], + lambda value: resolve(value, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[1], {"name": "gzip", "configuration": {"level": 12}, "must_understand": "yes"}, ), ( lambda value: fill_value_problems( - resolve("uint8", DataTypeDefinition, CORE_AND_EXTENSIONS)[0], value + resolve("uint8", DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT)[0], value ), 300, ), (validate_json, {"a": [math.inf]}), ( - lambda value: resolve(value, CodecDefinition, CORE_AND_EXTENSIONS)[1], + lambda value: resolve(value, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[1], {"name": 1, "configuration": {"a": [math.nan]}, "extra": [1]}, ), (validate_node_name_v3, "__/"), @@ -316,9 +316,9 @@ def test_a_problem_with_what_a_name_carries_is_the_field_s() -> None: # refuses. The problem is located at the field, because the name is what # the document wrote there. The bound on `bits` is not a bound on the # name, so the problem's `ctx` is empty. - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( data_types={ - **CORE_AND_EXTENSIONS.data_types, + **CORE_AND_EXTENSIONS_CONTEXT.data_types, "r*": DataTypeDefinition(name="r*", configuration=WideBits), } ) @@ -391,7 +391,7 @@ def test_a_problem_at_the_first_element_holds_it() -> None: (problem,) = with_input([ValidationProblem(("a", 0), "bad", "invalid_value")], {"a": [7]}) assert problem.input == 7 assert value_at([5, 6], (0,)) == 5 - read = resolve("bytes", DataTypeDefinition, CORE_AND_EXTENSIONS)[0] + read = resolve("bytes", DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT)[0] (found,) = fill_value_problems(read, [-1]) assert (found.loc, found.input) == ((0,), -1) diff --git a/packages/zarr-metadata/tests/test_public_api.py b/packages/zarr-metadata/tests/test_public_api.py index e0095aec31..ab6da5d50f 100644 --- a/packages/zarr-metadata/tests/test_public_api.py +++ b/packages/zarr-metadata/tests/test_public_api.py @@ -88,6 +88,8 @@ def _group_rank(s: str) -> int: "ScaleOffsetCodecMetadata", "ShardingIndexedCodecMetadata", "TransposeCodecMetadata", + "VlenBytesCodecMetadata", + "VlenUtf8CodecMetadata", "ZstdCodecMetadata", # Category C — grid/key canonical unions "RegularChunkGridMetadata", @@ -150,6 +152,10 @@ def _group_rank(s: str) -> int: "StringDataTypeName", "STRING_DATA_TYPE_NAME", "StringFillValue", + # fixed_length_utf32 + "FixedLengthUtf32DataTypeName", + "FIXED_LENGTH_UTF32_DATA_TYPE_NAME", + "FixedLengthUtf32FillValue", # numpy_datetime64 "NumpyDatetime64DataTypeName", "NUMPY_DATETIME64_DATA_TYPE_NAME", @@ -188,6 +194,10 @@ def _group_rank(s: str) -> int: "CastValueCodecName", "CRC32C_CODEC_NAME", "Crc32cCodecName", + "VLEN_BYTES_CODEC_NAME", + "VlenBytesCodecName", + "VLEN_UTF8_CODEC_NAME", + "VlenUtf8CodecName", "GZIP_CODEC_NAME", "GzipCodecName", "SCALE_OFFSET_CODEC_NAME", @@ -327,6 +337,8 @@ def test_all_is_grouped_and_unique() -> None: # A pydantic field's scope, fixed: `Annotated[Model, InScope(CONTEXT)]`. "InScope", "MetadataValidationError", + "FixedLengthUtf32", + "NullTerminatedBytes", "NumpyDatetime64", "NumpyTimeUnit", "NumpyTimedelta64", diff --git a/packages/zarr-metadata/tests/v2/test_definition.py b/packages/zarr-metadata/tests/v2/test_definition.py index 98f62d68a6..2d956e4da8 100644 --- a/packages/zarr-metadata/tests/v2/test_definition.py +++ b/packages/zarr-metadata/tests/v2/test_definition.py @@ -21,7 +21,7 @@ ) from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ) @@ -33,10 +33,10 @@ def test_core_v2_files_every_v2_definition_apart_from_v3() -> None: assert CORE_V2.zarr_format == 2 assert CORE_V2.claimant(ZarrV2CodecDefinition, "gzip") is not GZIP_CODEC assert CORE_V2.claimant(ZarrV2CodecDefinition, "gzip") is not None - assert CORE_AND_EXTENSIONS.claimant(CodecDefinition, "gzip") is GZIP_CODEC + assert CORE_AND_EXTENSIONS_CONTEXT.claimant(CodecDefinition, "gzip") is GZIP_CODEC assert CORE_V2.claimant(ZarrV2DataTypeDefinition, " list[tuple[tuple[str | int def test_core_is_a_subset_of_core_and_extensions() -> None: - """Each slot of `CORE` files a subset of what the same slot of `CORE_AND_EXTENSIONS` files.""" + """Each slot of `CORE_CONTEXT` files a subset of what the same slot of `CORE_AND_EXTENSIONS_CONTEXT` files.""" for kind in ZarrV3Context.kinds.values(): - assert CORE.definitions_of(kind).items() <= CORE_AND_EXTENSIONS.definitions_of(kind).items() + assert ( + CORE_CONTEXT.definitions_of(kind).items() + <= CORE_AND_EXTENSIONS_CONTEXT.definitions_of(kind).items() + ) def test_a_read_field_keeps_the_fields_it_read_inside() -> None: @@ -240,7 +243,7 @@ def test_a_read_field_keeps_the_fields_it_read_inside() -> None: "index_codecs": ["bytes", "crc32c"], }, } - resolved, _ = resolve(shard, CodecDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(shard, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert isinstance(resolved, AcceptedField) assert {loc: inner.json for loc, inner in resolved.nested.items()} == { ("codecs", 0): "bytes", @@ -248,17 +251,19 @@ def test_a_read_field_keeps_the_fields_it_read_inside() -> None: ("index_codecs", 1): "crc32c", } cast_value = {"name": "cast_value", "configuration": {"data_type": "int8"}} - resolved, _ = resolve(cast_value, CodecDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(cast_value, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert isinstance(resolved, AcceptedField) assert resolved.nested[("data_type",)] == INT8 # A field with no nested fields has nothing inside. A field that nothing # in scope claims holds no fields at all. A field refused by its check # or rules keeps the nested fields it read. - assert resolve("int8", DataTypeDefinition, CORE)[0] == INT8 + assert resolve("int8", DataTypeDefinition, CORE_CONTEXT)[0] == INT8 unclaimed = {"name": "acme.cast", "configuration": {"data_type": "int8"}} - assert isinstance(resolve(unclaimed, CodecDefinition, CORE_AND_EXTENSIONS)[0], UnclaimedField) + assert isinstance( + resolve(unclaimed, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT)[0], UnclaimedField + ) refused = {"name": "cast_value", "configuration": {"data_type": "int8", "rounding": 1}} - resolved, _ = resolve(refused, CodecDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(refused, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert isinstance(resolved, RefusedField) assert resolved.nested[("data_type",)] == INT8 @@ -279,7 +284,7 @@ def test_a_read_field_keeps_the_fields_it_read_inside() -> None: def test_a_reading_says_the_name_the_field_was_written_with( field: JSONValue, kind: type[Definition[Any]], name: str | None ) -> None: - assert resolve(field, kind, CORE_AND_EXTENSIONS)[0].name == name + assert resolve(field, kind, CORE_AND_EXTENSIONS_CONTEXT)[0].name == name KINDLESS: Final = Definition(name="acme.kindless", configuration=EmptyConfiguration) @@ -766,7 +771,7 @@ def test_error_must_understand_false_is_refused_wherever_the_model_refuses_it( for problem in read_array_metadata_v3(document).problems if problem.loc[: len(at)] == at ] - assert _locs(resolve(field, kind, CORE_AND_EXTENSIONS)[1]) == by_model + assert _locs(resolve(field, kind, CORE_AND_EXTENSIONS_CONTEXT)[1]) == by_model assert by_model == [(("must_understand",), "invalid_value")] @@ -774,7 +779,7 @@ def _ruled_by( rules: Callable[[GzipCodecConfiguration, Nested], Iterable[ValidationProblem]], ) -> Context: lying = replace(GZIP_CODEC, name="acme.gzip", rules=rules) - return CORE.update(codecs=[*CORE.codecs.values(), lying]) + return CORE_CONTEXT.update(codecs=[*CORE_CONTEXT.codecs.values(), lying]) def test_error_a_rule_that_yields_something_else() -> None: @@ -1044,7 +1049,7 @@ class _HugeFormat(_ClaimsWithNoFormat): "context", [ None, - "CORE", + "CORE_CONTEXT", ZarrV3Context, _ClaimsWithNoFormat(), _BooleanFormatScope(), @@ -1126,7 +1131,9 @@ def test_a_scope_takes_a_name_over() -> None: else [] ), ) - context = CORE_AND_EXTENSIONS.update(codecs={**CORE_AND_EXTENSIONS.codecs, "gzip": strict}) + context = CORE_AND_EXTENSIONS_CONTEXT.update( + codecs={**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": strict} + ) _, found = resolve({"name": "gzip", "configuration": {"level": 0}}, CodecDefinition, context) assert _locs(found) == [(("configuration", "level"), "invalid_value")] assert context.claimant(ChunkGridDefinition, "regular") is REGULAR_CHUNK_GRID @@ -1134,17 +1141,17 @@ def test_a_scope_takes_a_name_over() -> None: def test_a_scope_is_shown_by_how_many_definitions_it_holds() -> None: # Kept short, because `help` shows it as a validator's default argument. - count = sum(len(CORE.definitions_of(kind)) for kind in ZarrV3Context.kinds.values()) - assert repr(CORE) == f"ZarrV3Context(<{count} definitions>)" + count = sum(len(CORE_CONTEXT.definitions_of(kind)) for kind in ZarrV3Context.kinds.values()) + assert repr(CORE_CONTEXT) == f"ZarrV3Context(<{count} definitions>)" def test_a_scope_pickles_as_its_definitions_and_copies_as_itself() -> None: # A model holds the scope it was read in, and takes it along to another process. - again = pickle.loads(pickle.dumps(CORE_AND_EXTENSIONS)) - assert again == CORE_AND_EXTENSIONS - assert again.codecs == CORE_AND_EXTENSIONS.codecs - assert copy.copy(CORE) is CORE - assert copy.deepcopy(CORE) is CORE + again = pickle.loads(pickle.dumps(CORE_AND_EXTENSIONS_CONTEXT)) + assert again == CORE_AND_EXTENSIONS_CONTEXT + assert again.codecs == CORE_AND_EXTENSIONS_CONTEXT.codecs + assert copy.copy(CORE_CONTEXT) is CORE_CONTEXT + assert copy.deepcopy(CORE_CONTEXT) is CORE_CONTEXT LE: Final = {"name": "bytes", "configuration": {"endian": "little"}} @@ -1279,8 +1286,8 @@ def test_two_fields_are_equal_when_they_read_the_same( one: JSONValue, other: JSONValue, kind: type[Definition[Any]], equal: bool ) -> None: """However each field was spelled, equal fields have the same simplest spelling and the same hash, though each is written back as it was read.""" - first, _ = resolve(one, kind, CORE_AND_EXTENSIONS) - second, _ = resolve(other, kind, CORE_AND_EXTENSIONS) + first, _ = resolve(one, kind, CORE_AND_EXTENSIONS_CONTEXT) + second, _ = resolve(other, kind, CORE_AND_EXTENSIONS_CONTEXT) assert isinstance(first, (AcceptedField, UnclaimedField)) assert isinstance(second, (AcceptedField, UnclaimedField)) assert (first == second) is equal @@ -1290,7 +1297,9 @@ def test_two_fields_are_equal_when_they_read_the_same( def test_a_field_copied_or_pickled_is_read_by_a_definition_equal_to_its_own() -> None: - field, _ = resolve({"name": "gzip", "configuration": {"level": 1}}, CodecDefinition, CORE) + field, _ = resolve( + {"name": "gzip", "configuration": {"level": 1}}, CodecDefinition, CORE_CONTEXT + ) for again in (pickle.loads(pickle.dumps(field)), copy.deepcopy(field)): assert again == field assert configuration_of(again, GZIP_CODEC) == {"level": 1} @@ -1508,7 +1517,7 @@ def test_error_a_scope_refuses_a_definition_of_no_kind() -> None: def test_an_extension_is_named_as_the_spec_names_one(name: str, valid: bool) -> None: """An extension name matches the spec's regex `^[a-z][a-z0-9-_.]+$`, or is a URI, which earlier versions of the spec required. Any other name is refused before any definition is consulted.""" for field, at in ((name, ()), ({"name": name}, ("name",))): - resolved, problems = resolve(field, CodecDefinition, CORE) + resolved, problems = resolve(field, CodecDefinition, CORE_CONTEXT) if valid: assert not isinstance(resolved, RefusedField) assert problems == () @@ -1523,7 +1532,7 @@ def test_error_a_bad_name_is_a_problem_when_the_field_is_not_json_too() -> None: # A field whose configuration is not JSON reports its name problem as # the JSON path does, and no definition is asked to claim it. field = {"name": "Acme", "configuration": {"x": float("nan")}} - resolved, problems = resolve(field, CodecDefinition, CORE) + resolved, problems = resolve(field, CodecDefinition, CORE_CONTEXT) assert isinstance(resolved, RefusedField) assert resolved.definition is None assert [(p.loc, p.kind) for p in problems] == [ @@ -1547,7 +1556,7 @@ def test_error_a_field_built_by_hand_is_named_as_the_spec_names_one() -> None: def test_error_a_field_without_a_name_is_missing_one() -> None: - resolved, problems = resolve({"configuration": {}}, CodecDefinition, CORE) + resolved, problems = resolve({"configuration": {}}, CodecDefinition, CORE_CONTEXT) assert isinstance(resolved, RefusedField) assert [(p.loc, p.kind, p.message) for p in problems] == [ (("name",), "missing_key", "missing required key") @@ -1559,7 +1568,9 @@ def test_raw_bits_of_more_than_a_hundred_digits_are_no_size_but_a_name(digits: i # `int` refuses to convert more than 4,300 digits, and no real size has a # hundred digits. Such a name is an extension name, which nothing in # scope claims. - resolved, problems = resolve("r" + "1" * digits, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, problems = resolve( + "r" + "1" * digits, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT + ) assert type(resolved) is UnclaimedField assert problems == () @@ -1573,8 +1584,8 @@ def writes( cast("dict[str, object]", configuration)["level"] = -1 yield from () - context = CORE.update( - codecs={**CORE.codecs, "gzip": dataclasses.replace(GZIP_CODEC, rules=writes)} + context = CORE_CONTEXT.update( + codecs={**CORE_CONTEXT.codecs, "gzip": dataclasses.replace(GZIP_CODEC, rules=writes)} ) with pytest.raises(TypeError, match="does not support item assignment") as raised: resolve({"name": "gzip", "configuration": {"level": 1}}, CodecDefinition, context) @@ -1598,8 +1609,11 @@ def folds_in_place(configuration: GzipCodecConfiguration) -> GzipCodecConfigurat cast("dict[str, object]", configuration)["level"] = 0 return configuration - context = CORE.update( - codecs={**CORE.codecs, "gzip": dataclasses.replace(GZIP_CODEC, canonical=folds_in_place)} + context = CORE_CONTEXT.update( + codecs={ + **CORE_CONTEXT.codecs, + "gzip": dataclasses.replace(GZIP_CODEC, canonical=folds_in_place), + } ) read, _ = resolve({"name": "gzip", "configuration": {"level": 1}}, CodecDefinition, context) with pytest.raises(TypeError, match="does not support item assignment"): @@ -1610,8 +1624,8 @@ def test_error_a_canonical_that_raises_says_which_definition_raised_it() -> None def refuses(configuration: GzipCodecConfiguration) -> GzipCodecConfiguration: raise ValueError("no") - context = CORE.update( - codecs={**CORE.codecs, "gzip": dataclasses.replace(GZIP_CODEC, canonical=refuses)} + context = CORE_CONTEXT.update( + codecs={**CORE_CONTEXT.codecs, "gzip": dataclasses.replace(GZIP_CODEC, canonical=refuses)} ) read, _ = resolve({"name": "gzip", "configuration": {"level": 1}}, CodecDefinition, context) with pytest.raises(ValueError, match="no") as raised: @@ -1622,7 +1636,9 @@ def refuses(configuration: GzipCodecConfiguration) -> GzipCodecConfiguration: def test_error_a_field_s_configuration_and_nested_fields_are_read_only() -> None: """A field's `json`, `configuration` and `nested` attributes are read-only at every level. So a field cannot be changed into a state where its key, `==` and `refines` disagree.""" whole = {**SHARD, "index_codecs": [LE, {"name": "crc32c"}]} - shard, _ = resolve({"name": "sharding_indexed", "configuration": whole}, CodecDefinition, CORE) + shard, _ = resolve( + {"name": "sharding_indexed", "configuration": whole}, CodecDefinition, CORE_CONTEXT + ) assert isinstance(shard, AcceptedField) with pytest.raises(TypeError): shard.configuration["index_location"] = "start" # pyright: ignore[reportIndexIssue] @@ -1637,14 +1653,16 @@ def test_error_a_field_s_configuration_and_nested_fields_are_read_only() -> None assert isinstance(shard.json, Mapping) with pytest.raises(TypeError): shard.json["name"] = "gzip" # pyright: ignore[reportIndexIssue] - unclaimed, _ = resolve({"name": "acme.x", "configuration": {"a": [1]}}, CodecDefinition, CORE) + unclaimed, _ = resolve( + {"name": "acme.x", "configuration": {"a": [1]}}, CodecDefinition, CORE_CONTEXT + ) assert isinstance(unclaimed, UnclaimedField) with pytest.raises(TypeError): unclaimed.configuration["a"] = 2 # pyright: ignore[reportIndexIssue] refused, _ = resolve( {"name": "sharding_indexed", "configuration": {**whole, "index_location": "x"}}, CodecDefinition, - CORE, + CORE_CONTEXT, ) assert isinstance(refused, RefusedField) with pytest.raises(TypeError): @@ -1655,12 +1673,14 @@ def test_a_field_holding_fields_is_copied_and_pickled_whole() -> None: """A field pickles and deep-copies together with the fields it holds, and stays equal to itself, whether it is accepted, refused or unclaimed.""" whole = {**SHARD, "index_codecs": [LE, {"name": "crc32c"}]} fields = [ - resolve({"name": "sharding_indexed", "configuration": whole}, CodecDefinition, CORE)[0], - resolve({"name": "acme.x", "configuration": {"a": [1]}}, CodecDefinition, CORE)[0], + resolve( + {"name": "sharding_indexed", "configuration": whole}, CodecDefinition, CORE_CONTEXT + )[0], + resolve({"name": "acme.x", "configuration": {"a": [1]}}, CodecDefinition, CORE_CONTEXT)[0], resolve( {"name": "sharding_indexed", "configuration": {**whole, "index_location": "x"}}, CodecDefinition, - CORE, + CORE_CONTEXT, )[0], ] for field in fields: @@ -1697,8 +1717,8 @@ def test_a_field_its_caller_refined_reads_as_resolve_reads_it(field: object) -> """`resolve_refined` is `resolve` for a caller that has already refined the field. It gives the same field and the same problems, at the same locations, with the same `input`.""" refined, problems = refine_json(field, ("codecs", 0)) assert problems == () - expected = resolve(field, CodecDefinition, CORE, ("codecs", 0)) - found = resolve_refined(field, refined, problems, CodecDefinition, CORE, ("codecs", 0)) + expected = resolve(field, CodecDefinition, CORE_CONTEXT, ("codecs", 0)) + found = resolve_refined(field, refined, problems, CodecDefinition, CORE_CONTEXT, ("codecs", 0)) assert found == expected assert [(p.loc, p.kind, p.input) for p in found[1]] == [ (p.loc, p.kind, p.input) for p in expected[1] @@ -1711,7 +1731,7 @@ def test_error_a_field_its_caller_found_not_to_be_json_is_refused_unread() -> No refined, problems = refine_json(field, ("codecs", 0)) assert refined is None resolved, found = resolve_refined( - field, refined, problems, CodecDefinition, CORE, ("codecs", 0) + field, refined, problems, CodecDefinition, CORE_CONTEXT, ("codecs", 0) ) assert resolved == RefusedField( json=UNSET, name="gzip", read_as=CodecDefinition, definition=GZIP_CODEC @@ -1719,7 +1739,7 @@ def test_error_a_field_its_caller_found_not_to_be_json_is_refused_unread() -> No assert [(p.loc, p.kind) for p in found] == [ (("codecs", 0, "configuration", "level"), "invalid_type") ] - assert (resolved, found) == resolve(field, CodecDefinition, CORE, ("codecs", 0)) + assert (resolved, found) == resolve(field, CodecDefinition, CORE_CONTEXT, ("codecs", 0)) def test_error_a_refined_field_is_not_read_in_a_scope_of_the_other_format() -> None: @@ -1738,8 +1758,8 @@ def test_error_a_refined_field_is_not_read_in_a_scope_of_the_other_format() -> N @pytest.mark.parametrize( ("field", "kind", "context", "message"), [ - ({"id": "gzip"}, ZarrV2CodecDefinition, CORE, V2_IN_A_V3_SCOPE), - (5, ZarrV2CodecDefinition, CORE, V2_IN_A_V3_SCOPE), + ({"id": "gzip"}, ZarrV2CodecDefinition, CORE_CONTEXT, V2_IN_A_V3_SCOPE), + (5, ZarrV2CodecDefinition, CORE_CONTEXT, V2_IN_A_V3_SCOPE), ({"name": "gzip"}, CodecDefinition, CORE_V2, V3_IN_A_V2_SCOPE), (5, CodecDefinition, CORE_V2, V3_IN_A_V2_SCOPE), ], diff --git a/packages/zarr-metadata/tests/v3/test_every_definition.py b/packages/zarr-metadata/tests/v3/test_every_definition.py index f881aa5b3c..daf731898e 100644 --- a/packages/zarr-metadata/tests/v3/test_every_definition.py +++ b/packages/zarr-metadata/tests/v3/test_every_definition.py @@ -28,8 +28,9 @@ from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE, RawBytesConfiguration from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, + ZARR_PYTHON_CONTEXT, ChunkGridDefinition, ChunkKeyEncodingDefinition, CodecDefinition, @@ -47,7 +48,8 @@ "group_extra_field": GroupExtraFieldDefinition, } -# Fields that read without problems, per extension: at least one per definition in scope. +# Fields that read without problems, per extension: at least one per definition in +# the widest scope the package provides. EXAMPLES: dict[str, tuple[object, ...]] = { "codecs:blosc": ( { @@ -89,6 +91,8 @@ }, }, ), + "codecs:vlen-bytes": ("vlen-bytes", {"name": "vlen-bytes"}), + "codecs:vlen-utf8": ("vlen-utf8", {"name": "vlen-utf8"}), "codecs:transpose": ({"name": "transpose", "configuration": {"order": [2, 1, 0]}},), "codecs:zstd": ({"name": "zstd", "configuration": {"level": 3, "checksum": False}},), "chunk_grid:regular": ({"name": "regular", "configuration": {"chunk_shape": [4, 4]}},), @@ -143,6 +147,13 @@ }, ), "data_type:string": ("string",), + "data_type:fixed_length_utf32": ( + {"name": "fixed_length_utf32", "configuration": {"length_bytes": 48}}, + ), + "data_type:null_terminated_bytes": ( + {"name": "null_terminated_bytes", "configuration": {"length_bytes": 1}}, + ), + "data_type:variable_length_bytes": ("variable_length_bytes",), "data_type:r*": ("r16", "r008"), "group_extra_field:consolidated_metadata": ( {"consolidated_metadata": {"kind": "inline", "must_understand": False, "metadata": {}}}, @@ -161,7 +172,7 @@ def _read(key: str, field: object) -> tuple[type, list[tuple[tuple[str | int, ...], str]]]: """The type of the result of reading `field` in the scope (`AcceptedField`, `UnclaimedField` or `RefusedField`), and the location of each problem.""" - resolved, problems = resolve(field, KINDS[key.split(":")[0]], CORE_AND_EXTENSIONS) + resolved, problems = resolve(field, KINDS[key.split(":")[0]], ZARR_PYTHON_CONTEXT) return type(resolved), [(found.loc, found.kind) for found in problems] @@ -171,7 +182,7 @@ def _problems(key: str, field: object) -> list[tuple[tuple[str | int, ...], str] def test_a_scope_s_definitions_pickle() -> None: # This lets a scope be sent to another process and rebuilt there. - data_types = pickle.loads(pickle.dumps(tuple(CORE_AND_EXTENSIONS.data_types.values()))) + data_types = pickle.loads(pickle.dumps(tuple(CORE_AND_EXTENSIONS_CONTEXT.data_types.values()))) context = ZarrV3Context(data_types=data_types) resolved, _ = resolve("complex64", DataTypeDefinition, context) assert [(p.loc, p.kind) for p in fill_value_problems(resolved, [1, "x"])] == [ @@ -189,9 +200,9 @@ def _every(context: ZarrV3Context) -> set[Definition[Any]]: def test_every_definition_in_scope_has_an_example() -> None: - names = {definition.name for definition in _every(CORE_AND_EXTENSIONS)} + names = {definition.name for definition in _every(ZARR_PYTHON_CONTEXT)} assert names == {key.split(":")[1] for key in EXAMPLES} - assert _every(CORE) < _every(CORE_AND_EXTENSIONS) + assert _every(CORE_CONTEXT) < _every(CORE_AND_EXTENSIONS_CONTEXT) < _every(ZARR_PYTHON_CONTEXT) @pytest.mark.parametrize( @@ -202,12 +213,12 @@ def test_every_example_reads_and_its_simplest_spelling_is_stable(key: str, field # reads the same way, and is its own simplest spelling. assert _read(key, field) == (AcceptedField, []) kind = KINDS[key.split(":")[0]] - simplest, problems = canonicalize(field, kind, CORE_AND_EXTENSIONS) + simplest, problems = canonicalize(field, kind, ZARR_PYTHON_CONTEXT) assert problems == () assert simplest is not None - assert canonicalize(simplest, kind, CORE_AND_EXTENSIONS) == (simplest, ()) + assert canonicalize(simplest, kind, ZARR_PYTHON_CONTEXT) == (simplest, ()) # A field that was already read gives the same spelling as its JSON, without reading it again. - assert canonical_of(*resolve(field, kind, CORE_AND_EXTENSIONS)) == simplest + assert canonical_of(*resolve(field, kind, ZARR_PYTHON_CONTEXT)) == simplest @pytest.mark.parametrize( @@ -339,8 +350,8 @@ def test_the_simplest_spelling(field: dict[str, Any], simplest: object) -> None: "v2": ChunkKeyEncodingDefinition, } kind = kinds.get(field["name"], CodecDefinition) - assert canonicalize(field, kind, CORE_AND_EXTENSIONS) == (simplest, ()) - assert canonical_of(*resolve(field, kind, CORE_AND_EXTENSIONS)) == simplest + assert canonicalize(field, kind, CORE_AND_EXTENSIONS_CONTEXT) == (simplest, ()) + assert canonical_of(*resolve(field, kind, CORE_AND_EXTENSIONS_CONTEXT)) == simplest @pytest.mark.parametrize( @@ -357,13 +368,13 @@ def test_the_simplest_spelling(field: dict[str, Any], simplest: object) -> None: def test_raw_bits_read_as_r_star_with_the_size_their_name_carries( field: object, bits: int, simplest: str ) -> None: - resolved, problems = resolve(field, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, problems = resolve(field, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert problems == () assert isinstance(resolved, AcceptedField) assert resolved.definition is RAW_BYTES_DATA_TYPE assert resolved.json == field assert configuration_of(resolved, RAW_BYTES_DATA_TYPE) == {"bits": bits} - assert canonicalize(field, DataTypeDefinition, CORE_AND_EXTENSIONS) == (simplest, ()) + assert canonicalize(field, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) == (simplest, ()) def test_a_reader_reads_raw_bits_its_own_way_by_defining_r_star() -> None: @@ -371,7 +382,9 @@ def test_a_reader_reads_raw_bits_its_own_way_by_defining_r_star() -> None: # package's own, it reads every raw-bits name. Filed back, the package's # definition reads them again. mine = DataTypeDefinition(name="r*", configuration=RawBytesConfiguration) - context = CORE_AND_EXTENSIONS.update(data_types={**CORE_AND_EXTENSIONS.data_types, "r*": mine}) + context = CORE_AND_EXTENSIONS_CONTEXT.update( + data_types={**CORE_AND_EXTENSIONS_CONTEXT.data_types, "r*": mine} + ) resolved, problems = resolve("r12", DataTypeDefinition, context) assert problems == () assert isinstance(resolved, AcceptedField) @@ -394,7 +407,7 @@ def test_a_reader_reads_raw_bits_its_own_way_by_defining_r_star() -> None: def test_error_r_star_is_notation_and_no_name(field: object, at: tuple[str, ...]) -> None: # `r*` is how the specification's table writes raw bits, not a name a # document can use: `*` is not allowed in an extension name. - resolved, problems = resolve(field, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, problems = resolve(field, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert type(resolved) is RefusedField assert [(p.loc, p.kind) for p in problems] == [(at, "invalid_value")] @@ -423,7 +436,7 @@ def test_error_r_star_is_notation_and_no_name(field: object, at: tuple[str, ...] def test_an_unclaimed_field_keeps_its_configuration_in_its_kind_s_envelope( field: object, kind: type[Definition[Any]], simplest: object ) -> None: - assert canonicalize(field, kind, CORE) == (simplest, ()) + assert canonicalize(field, kind, CORE_CONTEXT) == (simplest, ()) def _shard(codecs: list[object], index_codecs: list[object]) -> dict[str, object]: @@ -492,10 +505,10 @@ def test_error_a_field_with_a_problem_has_no_simplest_spelling( # unknown key, the stray member or the `must_understand`, or would # write a value that is not valid. A field that was already read, # together with its problems, has no simplest spelling either. - simplest, problems = canonicalize(field, kind, CORE_AND_EXTENSIONS) + simplest, problems = canonicalize(field, kind, CORE_AND_EXTENSIONS_CONTEXT) assert simplest is None assert [(problem.loc, problem.kind) for problem in problems] == found - assert canonical_of(*resolve(field, kind, CORE_AND_EXTENSIONS)) is None + assert canonical_of(*resolve(field, kind, CORE_AND_EXTENSIONS_CONTEXT)) is None def test_error_a_canonical_that_does_not_hold_is_the_definitions_fault() -> None: @@ -508,7 +521,9 @@ def test_error_a_canonical_that_does_not_hold_is_the_definitions_fault() -> None canonicalize( {"name": "acme.lying", "configuration": {"level": 1}}, CodecDefinition, - CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), lying]), + CORE_AND_EXTENSIONS_CONTEXT.update( + codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), lying] + ), ) @@ -671,21 +686,24 @@ def test_a_bound_is_its_member_s_type_and_its_problem_holds_it( ) -> None: # The bound is declared on the TypedDict, as pydantic reads a bound, # not in a rule. The problem holds the bound and the value found. - _, problems = resolve(field, kind, CORE_AND_EXTENSIONS) + _, problems = resolve(field, kind, CORE_AND_EXTENSIONS_CONTEXT) assert [(p.loc, p.kind, p.input, dict(p.ctx)) for p in problems] == [ (loc, "invalid_value", value_at(field, loc), ctx) ] STATIC_SIZE = ("bytes", "cast_value", "crc32c", "scale_offset", "transpose") -DYNAMIC_SIZE = ("blosc", "gzip", "sharding_indexed", "zstd") +DYNAMIC_SIZE = ("blosc", "gzip", "sharding_indexed", "vlen-bytes", "vlen-utf8", "zstd") def test_every_codec_says_whether_its_size_is_static() -> None: # This matches zarr-python's `is_fixed_size`. A compressor, and a shard - # (whose empty inner chunks take no bytes), output a size that depends - # on the values. - sizes = {definition.name: definition.size for definition in CORE_AND_EXTENSIONS.codecs.values()} + # (whose empty inner chunks take no bytes), and a variable-length + # codec output a size that depends on the values. + sizes = { + definition.name: definition.size + for definition in CORE_AND_EXTENSIONS_CONTEXT.codecs.values() + } assert sizes == dict.fromkeys(STATIC_SIZE, "static") | dict.fromkeys(DYNAMIC_SIZE, "dynamic") diff --git a/packages/zarr-metadata/tests/v3/test_fill_values.py b/packages/zarr-metadata/tests/v3/test_fill_values.py index 27436365b3..46e2152cfe 100644 --- a/packages/zarr-metadata/tests/v3/test_fill_values.py +++ b/packages/zarr-metadata/tests/v3/test_fill_values.py @@ -40,7 +40,7 @@ from zarr_metadata.v3.data_type._float import FloatWidth, complex_fill_value_rules, float_bits from zarr_metadata.v3.data_type.struct import STRUCT_DATA_TYPE from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, DataTypeDefinition, ) @@ -86,7 +86,7 @@ def acme_point_fill_value_rules( def _problems(data_type: JSONValue, value: object) -> list[tuple[tuple[str | int, ...], str]]: - resolved, found = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, found = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert found == () return [(problem.loc, problem.kind) for problem in fill_value_problems(resolved, value)] @@ -139,7 +139,7 @@ def _problems(data_type: JSONValue, value: object) -> list[tuple[tuple[str | int ], ) def test_every_data_type_takes_its_fill_values(data_type: JSONValue, fill_value: object) -> None: - resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert fill_value_problems(resolved, fill_value) == () @@ -235,7 +235,7 @@ def test_a_fill_value_s_range_is_its_type_s( ) -> None: # `Int8FillValue` is `Annotated[int, Interval(ge=-128, le=127)]`. The # problem holds the range and the value that was found. - resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) problems = fill_value_problems(resolved, fill_value) assert [(p.loc, p.input, dict(p.ctx)) for p in problems] == [ (loc, value_at(fill_value, loc), ctx) @@ -288,8 +288,8 @@ def test_error_an_array_document_s_fill_value_its_data_type_refuses() -> None: def test_error_a_key_the_fill_value_shape_does_not_declare_hides_no_rule() -> None: # The extra key is reported and hidden from the rules, which still check the other keys. - context = CORE_AND_EXTENSIONS.update( - data_types=[*CORE_AND_EXTENSIONS.data_types.values(), ACME_POINT] + context = CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[*CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), ACME_POINT] ) resolved, _ = resolve("acme.point", DataTypeDefinition, context) found = fill_value_problems(resolved, {"x": 99, "extra": 1}) @@ -335,7 +335,7 @@ def _alike(left: object, right: object) -> bool: def _read(data_type: JSONValue) -> AcceptedField[DataTypeDefinition[Any]]: - resolved, found = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, found = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert found == () assert isinstance(resolved, AcceptedField) return resolved @@ -416,7 +416,7 @@ def test_a_fill_value_s_canonical_spelling_is_its_value_s( def test_a_data_type_nothing_in_scope_claims_spells_a_fill_value_as_written() -> None: - unclaimed, _ = resolve("acme.decimal", DataTypeDefinition, CORE_AND_EXTENSIONS) + unclaimed, _ = resolve("acme.decimal", DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) values: list[JSONValue] = [-0.0, True, 1, [1.0], None] for value in values: assert _alike(canonical_fill_value(unclaimed, value), value) @@ -453,7 +453,7 @@ def test_error_a_fill_value_with_a_problem_has_no_canonical_spelling() -> None: assert canonical_fill_value(_read("float32"), "0x7fc0") is UNSET assert canonical_fill_value(_read("int8"), 1.0) is UNSET assert canonical_fill_value(_read("int8"), math.nan) is UNSET - unclaimed, _ = resolve("acme.decimal", DataTypeDefinition, CORE_AND_EXTENSIONS) + unclaimed, _ = resolve("acme.decimal", DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert canonical_fill_value(unclaimed, None) is None @@ -461,9 +461,9 @@ def test_error_a_canonical_spelling_that_raises_says_which_data_type_raised_it() def refuses(configuration: EmptyConfiguration, nested: Nested, value: object) -> JSONValue: raise ValueError("no") - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( data_types=[ - *CORE_AND_EXTENSIONS.data_types.values(), + *CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), dataclasses.replace(ACME_POINT, fill_value_canonical=refuses), ] ) @@ -477,9 +477,9 @@ def test_error_a_canonical_spelling_that_is_not_json_is_refused() -> None: def not_json(configuration: EmptyConfiguration, nested: Nested, value: object) -> JSONValue: return cast("JSONValue", {1, 2}) - context = CORE_AND_EXTENSIONS.update( + context = CORE_AND_EXTENSIONS_CONTEXT.update( data_types=[ - *CORE_AND_EXTENSIONS.data_types.values(), + *CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), dataclasses.replace(ACME_POINT, fill_value_canonical=not_json), ] ) diff --git a/packages/zarr-metadata/tests/v3/test_grid_shapes.py b/packages/zarr-metadata/tests/v3/test_grid_shapes.py index 5d79ea655e..6586624ce5 100644 --- a/packages/zarr-metadata/tests/v3/test_grid_shapes.py +++ b/packages/zarr-metadata/tests/v3/test_grid_shapes.py @@ -22,7 +22,7 @@ chunk_grid_lengths, ) from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, ChunkGridDefinition, ) @@ -39,7 +39,7 @@ def _rectilinear(*specs: JSONValue) -> JSONValue: def _problems(grid: JSONValue, shape: tuple[int, ...]) -> list[tuple[tuple[str | int, ...], str]]: - resolved, found = resolve(grid, ChunkGridDefinition, CORE_AND_EXTENSIONS) + resolved, found = resolve(grid, ChunkGridDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert found == () lengths, problems = chunk_grid_lengths(resolved, shape) # A grid that does not fit the shape gives no lengths along any axis. @@ -75,7 +75,7 @@ def _problems(grid: JSONValue, shape: tuple[int, ...]) -> list[tuple[tuple[str | def test_every_chunk_grid_gives_the_lengths_of_its_chunks_over_a_shape_it_fits( grid: JSONValue, shape: tuple[int, ...], lengths: tuple[set[int] | None, ...] ) -> None: - resolved, _ = resolve(grid, ChunkGridDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(grid, ChunkGridDefinition, CORE_AND_EXTENSIONS_CONTEXT) expected = tuple(None if axis is None else frozenset(axis) for axis in lengths) assert chunk_grid_lengths(resolved, shape) == (expected, ()) @@ -103,8 +103,8 @@ def test_error_rectilinear_chunks_that_do_not_cover_their_dimension(spec: JSONVa def test_a_grid_that_says_nothing_of_the_shape_fits_every_one_its_lengths_unknown() -> None: lenient = ChunkGridDefinition(name="acme.grid", configuration=EmptyConfiguration) - context = CORE_AND_EXTENSIONS.update( - chunk_grids=[*CORE_AND_EXTENSIONS.chunk_grids.values(), lenient] + context = CORE_AND_EXTENSIONS_CONTEXT.update( + chunk_grids=[*CORE_AND_EXTENSIONS_CONTEXT.chunk_grids.values(), lenient] ) resolved, _ = resolve("acme.grid", ChunkGridDefinition, context) assert chunk_grid_lengths(resolved, (3, 4)) == ((None, None), ()) @@ -117,8 +117,8 @@ def test_error_a_grid_whose_chunk_lengths_have_another_number_of_axes() -> None: configuration=EmptyConfiguration, chunk_lengths=lambda c, n, s: (frozenset({1}),), ) - context = CORE_AND_EXTENSIONS.update( - chunk_grids=[*CORE_AND_EXTENSIONS.chunk_grids.values(), flat] + context = CORE_AND_EXTENSIONS_CONTEXT.update( + chunk_grids=[*CORE_AND_EXTENSIONS_CONTEXT.chunk_grids.values(), flat] ) resolved, _ = resolve("acme.grid", ChunkGridDefinition, context) with pytest.raises(ValueError, match="'acme.grid': its chunk_lengths gave 1 axes"): @@ -162,7 +162,9 @@ def test_error_a_grid_whose_chunk_lengths_give_something_else_or_raise( resolved, _ = resolve( "acme.grid", ChunkGridDefinition, - CORE_AND_EXTENSIONS.update(chunk_grids=[*CORE_AND_EXTENSIONS.chunk_grids.values(), odd]), + CORE_AND_EXTENSIONS_CONTEXT.update( + chunk_grids=[*CORE_AND_EXTENSIONS_CONTEXT.chunk_grids.values(), odd] + ), ) with pytest.raises((TypeError, KeyError), match=match) as raised: chunk_grid_lengths(resolved, (3, 4), ("chunk_grid",)) diff --git a/packages/zarr-metadata/tests/v3/test_kinds.py b/packages/zarr-metadata/tests/v3/test_kinds.py index 37fafd05f2..3f92543671 100644 --- a/packages/zarr-metadata/tests/v3/test_kinds.py +++ b/packages/zarr-metadata/tests/v3/test_kinds.py @@ -19,7 +19,7 @@ from zarr_metadata.json_schema import field_json_schema_v3 from zarr_metadata.v2.definition import CORE_V2, ZarrV2Context, ZarrV2DataTypeDefinition from zarr_metadata.v3.definition import ( - CORE, + CORE_CONTEXT, CodecDefinition, DataTypeDefinition, ZarrV3Context, @@ -76,7 +76,7 @@ def test_a_v3_value_that_is_no_field_is_shown_in_the_message() -> None: _, problems = resolve( {"name": "sharding_indexed", "configuration": {"chunk_shape": [1], "codecs": [3]}}, CodecDefinition, - CORE, + CORE_CONTEXT, ) assert any(p.message.endswith("got 3") for p in problems), [p.message for p in problems] @@ -93,7 +93,7 @@ def test_error_a_kind_declares_its_format() -> None: def test_a_scope_has_the_format_of_its_class() -> None: """A scope's `zarr_format` is the format of every kind its class holds: 3 for a `ZarrV3Context` and 2 for a `ZarrV2Context`, including one that files nothing.""" - assert CORE.zarr_format == 3 + assert CORE_CONTEXT.zarr_format == 3 assert CORE_V2.zarr_format == 2 assert ZarrV3Context().zarr_format == 3 assert ZarrV2Context().zarr_format == 2 @@ -106,15 +106,15 @@ def test_error_a_scope_files_one_format() -> None: with pytest.raises(TypeError, match="v2 data type"): ZarrV3Context(data_types=[UINT_DATA_TYPE_V2]) # pyright: ignore[reportArgumentType] with pytest.raises(TypeError, match="v2 data type"): - CORE.update(data_types=[*CORE.data_types.values(), UINT_DATA_TYPE_V2]) # pyright: ignore[reportArgumentType] + CORE_CONTEXT.update(data_types=[*CORE_CONTEXT.data_types.values(), UINT_DATA_TYPE_V2]) # pyright: ignore[reportArgumentType] with pytest.raises(TypeError, match="format"): - CORE.union(CORE_V2) # pyright: ignore[reportArgumentType] + CORE_CONTEXT.union(CORE_V2) # pyright: ignore[reportArgumentType] def test_error_a_field_is_read_in_a_scope_of_its_format() -> None: """`resolve` raises `TypeError` for a scope whose format differs from the kind's. In an empty scope of its format, it reads a field as unclaimed.""" with pytest.raises(TypeError, match="format"): - resolve(" C ) """An array -> array codec that does not declare its output chunk.""" -CONTEXT = CORE_AND_EXTENSIONS.update( - codecs=[*CORE_AND_EXTENSIONS.codecs.values(), ACME_TILE, ACME_SHUFFLE] +CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update( + codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_TILE, ACME_SHUFFLE] ) UINT8 = resolve("uint8", DataTypeDefinition, CONTEXT)[0] @@ -187,7 +187,7 @@ def _lengths(*axes: set[int]) -> tuple[frozenset[int], ...]: CHUNK, [CHUNK, Chunk(), Chunk(None, _of("float32").data_type)], ), - ([_scale(offset=1), "vlen-utf8"], Chunk(), [Chunk(), Chunk()]), + ([_scale(offset=1), "acme.vlen"], Chunk(), [Chunk(), Chunk()]), # Where the specs disagree or say nothing, a data type is not checked. # The numpy time types say they accept any codec of 64-bit integers, # and scale_offset does not say whether it accepts complex numbers. @@ -315,7 +315,7 @@ def test_error_an_arithmetic_codec_handed_values_that_are_no_numbers( ) -> None: # The codec comes before an array -> bytes codec that nothing in scope # claims, which is not checked. - assert _problems([codec, "vlen-utf8"], _of(data_type)) == [ + assert _problems([codec, "acme.vlen"], _of(data_type)) == [ ((0, "configuration"), "invalid_value") ] @@ -330,7 +330,7 @@ def test_error_a_codec_handed_on_what_another_could_not_take_is_judged_too() -> def test_error_a_cast_handed_complex_numbers() -> None: - assert _problems([_cast("float32"), "vlen-utf8"], _of("complex64")) == [ + assert _problems([_cast("float32"), "acme.vlen"], _of("complex64")) == [ ((0, "configuration"), "invalid_value") ] diff --git a/packages/zarr-metadata/tests/v3/test_scope.py b/packages/zarr-metadata/tests/v3/test_scope.py index a99f1ac2df..97b382d3b0 100644 --- a/packages/zarr-metadata/tests/v3/test_scope.py +++ b/packages/zarr-metadata/tests/v3/test_scope.py @@ -29,8 +29,8 @@ from zarr_metadata.v3.data_type.bytes import BYTES_DATA_TYPE from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE from zarr_metadata.v3.definition import ( - CORE, - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, + CORE_CONTEXT, ArrayExtraFieldDefinition, ChunkGridDefinition, ChunkKeyEncodingDefinition, @@ -73,8 +73,8 @@ def _read(data: object, kind: type[Definition[Any]], context: ZarrV3Context) -> ZarrV3Context(codecs=[BYTES_CODEC, GZIP_CODEC]), True, ), - (CORE, pickle.loads(pickle.dumps(CORE)), True), - (CORE, CORE_AND_EXTENSIONS, False), + (CORE_CONTEXT, pickle.loads(pickle.dumps(CORE_CONTEXT)), True), + (CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT, False), (ZarrV3Context(codecs=[GZIP_CODEC]), ZarrV3Context(codecs=[GZIP_CODEC, ZSTD_CODEC]), False), (ZarrV3Context(), ZarrV3Context(), True), ], @@ -91,7 +91,10 @@ def test_a_scope_is_equal_to_another_by_the_definitions_it_files( def test_a_scope_is_a_set_member() -> None: """A scope is hashable, so it can be a dict key or a set member. It was not hashable when its tables were mapping proxies.""" - assert len({CORE, CORE_AND_EXTENSIONS, pickle.loads(pickle.dumps(CORE))}) == 2 + assert ( + len({CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT, pickle.loads(pickle.dumps(CORE_CONTEXT))}) + == 2 + ) def test_error_a_scope_conflict_says_each_disagreement() -> None: @@ -114,14 +117,14 @@ def test_error_a_scope_conflict_says_each_disagreement() -> None: @pytest.mark.parametrize( ("field", "claims"), [ - (_read("gzip", CodecDefinition, CORE), {(CodecDefinition, "gzip"): GZIP_CODEC}), - (_read("zstd", CodecDefinition, CORE), {(CodecDefinition, "zstd"): None}), + (_read("gzip", CodecDefinition, CORE_CONTEXT), {(CodecDefinition, "gzip"): GZIP_CODEC}), + (_read("zstd", CodecDefinition, CORE_CONTEXT), {(CodecDefinition, "zstd"): None}), ( - _read("r16", DataTypeDefinition, CORE), + _read("r16", DataTypeDefinition, CORE_CONTEXT), {(DataTypeDefinition, "r*"): RAW_BYTES_DATA_TYPE}, ), ( - _read(SHARD, CodecDefinition, CORE), + _read(SHARD, CodecDefinition, CORE_CONTEXT), { (CodecDefinition, "sharding_indexed"): SHARDING_INDEXED_CODEC, (CodecDefinition, "bytes"): BYTES_CODEC, @@ -129,7 +132,7 @@ def test_error_a_scope_conflict_says_each_disagreement() -> None: }, ), ( - _read({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE), + _read({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_CONTEXT), {(CodecDefinition, "gzip"): GZIP_CODEC}, ), (RefusedField(json=3, name=None, read_as=CodecDefinition), {}), @@ -146,7 +149,7 @@ def test_claims_of_says_what_a_reading_claimed_of_each_name( def test_error_claims_of_refuses_one_name_read_two_ways() -> None: """Fields read in two scopes that give one name two different definitions have no single set of claims. `claims_of` raises a `ScopeConflictError` that names the key.""" fields = [ - *fields_of(_read("gzip", CodecDefinition, CORE), ("codecs", 0)), + *fields_of(_read("gzip", CodecDefinition, CORE_CONTEXT), ("codecs", 0)), *fields_of(_read("gzip", CodecDefinition, ZarrV3Context(codecs=[MY_GZIP])), ("codecs", 1)), ] with pytest.raises(ScopeConflictError) as raised: @@ -188,61 +191,69 @@ def test_error_claims_of_refuses_one_name_read_two_ways() -> None: @pytest.mark.parametrize( ("field", "other", "expected"), [ - (_read(GZIP_FIELD, CodecDefinition, CORE), _read(GZIP_FIELD, CodecDefinition, CORE), True), ( - _read({"name": "crc32c"}, CodecDefinition, CORE), - _read("crc32c", CodecDefinition, CORE), + _read(GZIP_FIELD, CodecDefinition, CORE_CONTEXT), + _read(GZIP_FIELD, CodecDefinition, CORE_CONTEXT), True, ), ( - _read(ZSTD_FIELD, CodecDefinition, CORE_AND_EXTENSIONS), - _read(ZSTD_FIELD, CodecDefinition, CORE), + _read({"name": "crc32c"}, CodecDefinition, CORE_CONTEXT), + _read("crc32c", CodecDefinition, CORE_CONTEXT), True, ), ( - _read(ZSTD_FIELD, CodecDefinition, CORE), - _read(ZSTD_FIELD, CodecDefinition, CORE_AND_EXTENSIONS), + _read(ZSTD_FIELD, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT), + _read(ZSTD_FIELD, CodecDefinition, CORE_CONTEXT), + True, + ), + ( + _read(ZSTD_FIELD, CodecDefinition, CORE_CONTEXT), + _read(ZSTD_FIELD, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT), False, ), ( - _read(GZIP_FIELD, CodecDefinition, CORE), + _read(GZIP_FIELD, CodecDefinition, CORE_CONTEXT), _read(GZIP_FIELD, CodecDefinition, ZarrV3Context(codecs=[MY_GZIP])), False, ), ( - _read(NESTED_ZSTD, CodecDefinition, CORE_AND_EXTENSIONS), - _read(NESTED_ZSTD, CodecDefinition, CORE), + _read(NESTED_ZSTD, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT), + _read(NESTED_ZSTD, CodecDefinition, CORE_CONTEXT), True, ), ( - _read(NESTED_ZSTD, CodecDefinition, CORE), - _read(NESTED_ZSTD, CodecDefinition, CORE_AND_EXTENSIONS), + _read(NESTED_ZSTD, CodecDefinition, CORE_CONTEXT), + _read(NESTED_ZSTD, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT), False, ), ( - _read({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE), - _read("gzip", CodecDefinition, CORE), + _read({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_CONTEXT), + _read("gzip", CodecDefinition, CORE_CONTEXT), False, ), - (_read("zstd", CodecDefinition, CORE), _read("zstd", CodecDefinition, CORE), True), ( - _read({"name": "zstd", "configuration": {"level": 1}}, CodecDefinition, CORE), - _read({"name": "zstd", "configuration": {"level": 2}}, CodecDefinition, CORE), + _read("zstd", CodecDefinition, CORE_CONTEXT), + _read("zstd", CodecDefinition, CORE_CONTEXT), + True, + ), + ( + _read({"name": "zstd", "configuration": {"level": 1}}, CodecDefinition, CORE_CONTEXT), + _read({"name": "zstd", "configuration": {"level": 2}}, CodecDefinition, CORE_CONTEXT), False, ), ( - _read("crc32c", CodecDefinition, CORE), + _read("crc32c", CodecDefinition, CORE_CONTEXT), _read({"name": "crc32c"}, CodecDefinition, ZarrV3Context()), True, ), ( - _read(BLOSC_LEVEL_ONE, CodecDefinition, CORE), + _read(BLOSC_LEVEL_ONE, CodecDefinition, CORE_CONTEXT), _read(BLOSC_LEVEL_TRUE, CodecDefinition, ZarrV3Context()), False, ), ( - _read(SHARD_HOLDING_REFUSED, CodecDefinition, CORE), - _read(SHARD_HOLDING_REFUSED, CodecDefinition, CORE), + _read(SHARD_HOLDING_REFUSED, CodecDefinition, CORE_CONTEXT), + _read(SHARD_HOLDING_REFUSED, CodecDefinition, CORE_CONTEXT), True, ), ], @@ -279,7 +290,7 @@ def test_refines_orders_readings_by_information( BLOSC_LEVEL_ONE, SHARD_HOLDING_REFUSED, ] -CONTEXTS = [ZarrV3Context(), CORE, CORE_AND_EXTENSIONS] +CONTEXTS = [ZarrV3Context(), CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT] READINGS = [ _read(document, CodecDefinition, context) for document in DOCUMENTS for context in CONTEXTS ] @@ -301,15 +312,15 @@ def test_refines_is_a_partial_order_whose_bottom_is_equality( @pytest.mark.parametrize( ("context", "claims", "gains", "conflicts"), [ - (CORE, {(CodecDefinition, "gzip"): GZIP_CODEC}, (), ()), + (CORE_CONTEXT, {(CodecDefinition, "gzip"): GZIP_CODEC}, (), ()), ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, {(CodecDefinition, "zstd"): None}, ((CodecDefinition, "zstd"),), (), ), ( - CORE, + CORE_CONTEXT, {(CodecDefinition, "zstd"): ZSTD_CODEC}, (), (Conflict((CodecDefinition, "zstd"), ZSTD_CODEC, None),), @@ -320,8 +331,8 @@ def test_refines_is_a_partial_order_whose_bottom_is_equality( (), (Conflict((CodecDefinition, "gzip"), GZIP_CODEC, MY_GZIP),), ), - (CORE, {(CodecDefinition, "acme.x"): None}, (), ()), - (CORE, {(DataTypeDefinition, "r*"): RAW_BYTES_DATA_TYPE}, (), ()), + (CORE_CONTEXT, {(CodecDefinition, "acme.x"): None}, (), ()), + (CORE_CONTEXT, {(DataTypeDefinition, "r*"): RAW_BYTES_DATA_TYPE}, (), ()), ], ids=["agrees", "gain", "loss", "conflict", "unclaimed-both", "raw-bits"], ) @@ -341,13 +352,13 @@ def test_disagreements_says_where_a_scope_reads_claims_otherwise( @pytest.mark.parametrize( ("contexts", "united"), [ - ((CORE, CORE_AND_EXTENSIONS), CORE_AND_EXTENSIONS), + ((CORE_CONTEXT, CORE_AND_EXTENSIONS_CONTEXT), CORE_AND_EXTENSIONS_CONTEXT), ( (ZarrV3Context(codecs=[GZIP_CODEC]), ZarrV3Context(codecs=[ZSTD_CODEC])), ZarrV3Context(codecs=[GZIP_CODEC, ZSTD_CODEC]), ), - ((CORE,), CORE), - ((CORE, CORE), CORE), + ((CORE_CONTEXT,), CORE_CONTEXT), + ((CORE_CONTEXT, CORE_CONTEXT), CORE_CONTEXT), ( (ZarrV3Context(codecs=[BYTES_CODEC]), ZarrV3Context(data_types=[BYTES_DATA_TYPE])), ZarrV3Context(data_types=[BYTES_DATA_TYPE], codecs=[BYTES_CODEC]), @@ -366,7 +377,7 @@ def test_union_is_the_least_scope_above_each( def test_error_union_refuses_one_name_filed_two_ways() -> None: """Scopes that file different definitions under one name have no union. `union` raises a `ScopeConflictError` that names the key and both definitions.""" with pytest.raises(ScopeConflictError) as raised: - CORE.union(ZarrV3Context(codecs=[MY_GZIP])) + CORE_CONTEXT.union(ZarrV3Context(codecs=[MY_GZIP])) (conflict,) = raised.value.conflicts assert (conflict.key, conflict.claimed, conflict.found) == ( (CodecDefinition, "gzip"), @@ -398,21 +409,25 @@ def test_a_gain_is_judged_by_what_the_definition_reads_not_by_spelling() -> None resolve, ) from zarr_metadata.v3.definition import ( - CORE, + CORE_CONTEXT, ChunkKeyEncodingDefinition, ) nothing = ZarrV3Context() spelled_out, _ = resolve( - {"name": "default", "configuration": {"separator": "/"}}, ChunkKeyEncodingDefinition, CORE + {"name": "default", "configuration": {"separator": "/"}}, + ChunkKeyEncodingDefinition, + CORE_CONTEXT, ) - bare, _ = resolve({"name": "default"}, ChunkKeyEncodingDefinition, CORE) + bare, _ = resolve({"name": "default"}, ChunkKeyEncodingDefinition, CORE_CONTEXT) unclaimed, _ = resolve({"name": "default"}, ChunkKeyEncodingDefinition, nothing) assert spelled_out == bare assert refines(bare, unclaimed) assert refines(spelled_out, unclaimed) other, _ = resolve( - {"name": "default", "configuration": {"separator": "."}}, ChunkKeyEncodingDefinition, CORE + {"name": "default", "configuration": {"separator": "."}}, + ChunkKeyEncodingDefinition, + CORE_CONTEXT, ) assert not refines(other, unclaimed) shard = { @@ -424,7 +439,7 @@ def test_a_gain_is_judged_by_what_the_definition_reads_not_by_spelling() -> None "index_location": "end", }, } - read, _ = resolve(shard, CodecDefinition, CORE) + read, _ = resolve(shard, CodecDefinition, CORE_CONTEXT) unread, _ = resolve({**shard, "must_understand": True}, CodecDefinition, nothing) assert refines(read, unread) @@ -440,7 +455,7 @@ def test_a_scope_conflict_error_pickles_and_copies_with_its_conflicts() -> None: import copy with pytest.raises(ScopeConflictError) as raised: - CORE.union(ZarrV3Context(codecs=[OTHER_GZIP])) + CORE_CONTEXT.union(ZarrV3Context(codecs=[OTHER_GZIP])) error = raised.value error.add_note("seen in a union") for again in (pickle.loads(pickle.dumps(error)), copy.copy(error), copy.deepcopy(error)): @@ -457,16 +472,18 @@ def test_a_conflict_names_the_field_as_the_document_writes_it_and_tells_the_defi import dataclasses from zarr_metadata.model import ZarrV3ArrayMetadata - from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS + from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT document = dict( ZarrV3ArrayMetadata.create_default(shape=(2,), data_type="r16", fill_value=[0, 0]).to_json() ) - model = ZarrV3ArrayMetadata(document, CORE_AND_EXTENSIONS) + model = ZarrV3ArrayMetadata(document, CORE_AND_EXTENSIONS_CONTEXT) other = dataclasses.replace(RAW_BYTES_DATA_TYPE, configuration=EmptyConfiguration) with pytest.raises(ScopeConflictError) as raised: model.refined_in( - CORE_AND_EXTENSIONS.update(data_types={**CORE_AND_EXTENSIONS.data_types, "r*": other}) + CORE_AND_EXTENSIONS_CONTEXT.update( + data_types={**CORE_AND_EXTENSIONS_CONTEXT.data_types, "r*": other} + ) ) message = str(raised.value) assert message.startswith("data type 'r16'") @@ -489,11 +506,11 @@ def test_each_slot_is_a_read_only_view_by_name() -> None: "array_extra_fields": ArrayExtraFieldDefinition, "group_extra_fields": GroupExtraFieldDefinition, } - assert CORE.codecs["gzip"] is GZIP_CODEC - assert CORE.data_types["r*"] is RAW_BYTES_DATA_TYPE - assert dict(CORE.storage_transformers) == {} + assert CORE_CONTEXT.codecs["gzip"] is GZIP_CODEC + assert CORE_CONTEXT.data_types["r*"] is RAW_BYTES_DATA_TYPE + assert dict(CORE_CONTEXT.storage_transformers) == {} with pytest.raises(TypeError): - CORE.codecs["gzip"] = MY_GZIP # pyright: ignore[reportIndexIssue] + CORE_CONTEXT.codecs["gzip"] = MY_GZIP # pyright: ignore[reportIndexIssue] def test_error_a_slot_refuses_a_definition_of_another_kind() -> None: diff --git a/packages/zarr-metadata/tests/v3/test_sharding.py b/packages/zarr-metadata/tests/v3/test_sharding.py index 3cb5982a56..0392abc362 100644 --- a/packages/zarr-metadata/tests/v3/test_sharding.py +++ b/packages/zarr-metadata/tests/v3/test_sharding.py @@ -25,7 +25,7 @@ read_pipeline, ) from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, Chunk, CodecDefinition, DataTypeDefinition, @@ -42,7 +42,7 @@ def _dt(name: JSONValue) -> ResolvedField[DataTypeDefinition[Any]]: - return resolve(name, DataTypeDefinition, CORE_AND_EXTENSIONS)[0] + return resolve(name, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT)[0] FLOAT32 = _dt("float32") @@ -81,7 +81,7 @@ def _read( ) -> tuple[tuple[Stage, ...], tuple[ValidationProblem, ...]]: """The stages, and every problem: first each codec's own problems at its location, then the pipeline's.""" read = [ - resolve(codec, CodecDefinition, CORE_AND_EXTENSIONS, (index,)) + resolve(codec, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT, (index,)) for index, codec in enumerate(codecs) ] stages, found = read_pipeline([resolved for resolved, _ in read], chunk) diff --git a/packages/zarr-metadata/tests/v3/test_storage.py b/packages/zarr-metadata/tests/v3/test_storage.py index fadcf36049..58d367119e 100644 --- a/packages/zarr-metadata/tests/v3/test_storage.py +++ b/packages/zarr-metadata/tests/v3/test_storage.py @@ -18,7 +18,7 @@ resolve, ) from zarr_metadata.v3.definition import ( - CORE_AND_EXTENSIONS, + CORE_AND_EXTENSIONS_CONTEXT, DataTypeDefinition, StorageClass, storage_of, @@ -76,7 +76,7 @@ def _struct(*field_types: JSONValue) -> JSONValue: def test_every_data_type_says_how_its_values_are_stored( data_type: JSONValue, storage: StorageClass | None ) -> None: - resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS) + resolved, _ = resolve(data_type, DataTypeDefinition, CORE_AND_EXTENSIONS_CONTEXT) assert storage_of(resolved) == storage @@ -85,7 +85,9 @@ def test_a_data_type_that_says_nothing_of_its_storage_leaves_it_unknown() -> Non resolved, _ = resolve( "acme.quiet", DataTypeDefinition, - CORE_AND_EXTENSIONS.update(data_types=[*CORE_AND_EXTENSIONS.data_types.values(), quiet]), + CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[*CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), quiet] + ), ) assert storage_of(resolved) is None @@ -99,7 +101,9 @@ def test_error_a_storage_that_gives_something_else() -> None: resolved, _ = resolve( "acme.lying", DataTypeDefinition, - CORE_AND_EXTENSIONS.update(data_types=[*CORE_AND_EXTENSIONS.data_types.values(), lying]), + CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[*CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), lying] + ), ) with pytest.raises(TypeError, match="'acme.lying': its storage gives one of"): storage_of(resolved) @@ -115,7 +119,9 @@ def storage(configuration: EmptyConfiguration, nested: Nested) -> StorageClass: resolved, _ = resolve( "acme.raising", DataTypeDefinition, - CORE_AND_EXTENSIONS.update(data_types=[*CORE_AND_EXTENSIONS.data_types.values(), raising]), + CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[*CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), raising] + ), ) with pytest.raises(KeyError) as raised: storage_of(resolved) diff --git a/packages/zarr-metadata/tests/v3/test_zarr_python_scope.py b/packages/zarr-metadata/tests/v3/test_zarr_python_scope.py new file mode 100644 index 0000000000..c80ef4356d --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_zarr_python_scope.py @@ -0,0 +1,168 @@ +"""The `ZARR_PYTHON_CONTEXT` scope, and the definitions it adds for what zarr-python writes. + +`fixed_length_utf32`, `vlen-utf8` and `vlen-bytes` are registered in +zarr-extensions, so they are in `CORE_AND_EXTENSIONS_CONTEXT`. +`null_terminated_bytes` and `variable_length_bytes` are not, so only +`ZARR_PYTHON_CONTEXT` reads them. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from zarr_metadata.definition import ( + AcceptedField, + RefusedField, + UnclaimedField, + canonical_fill_value, + fill_value_problems, + resolve, +) +from zarr_metadata.model import read_array_metadata_v3 +from zarr_metadata.v3.data_type.null_terminated_bytes import NULL_TERMINATED_BYTES_DATA_TYPE +from zarr_metadata.v3.data_type.variable_length_bytes import VARIABLE_LENGTH_BYTES_DATA_TYPE +from zarr_metadata.v3.definition import ( + CORE_AND_EXTENSIONS_CONTEXT, + ZARR_PYTHON_CONTEXT, + DataTypeDefinition, + ZarrV3Context, +) + + +def _document(data_type: object, fill_value: object, codecs: list[object]) -> dict[str, Any]: + return { + "zarr_format": 3, + "node_type": "array", + "shape": [2], + "data_type": data_type, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2]}}, + "chunk_key_encoding": {"name": "default"}, + "fill_value": fill_value, + "codecs": codecs, + "attributes": {}, + } + + +_LITTLE = {"name": "bytes", "configuration": {"endian": "little"}} +_UTF32 = {"name": "fixed_length_utf32", "configuration": {"length_bytes": 12}} +_NULL_TERMINATED = {"name": "null_terminated_bytes", "configuration": {"length_bytes": 4}} + + +# What zarr-python 3.4.2 writes for ` None: + # In `ZARR_PYTHON_CONTEXT` every field is read by a definition and has no problem. + reading = read_array_metadata_v3(document, context=ZARR_PYTHON_CONTEXT) + assert reading.problems == () + assert all(isinstance(field, AcceptedField) for _, field in reading.fields()) + # `CORE_AND_EXTENSIONS_CONTEXT` reads the registered extensions, and leaves + # zarr-python's own data types unclaimed. + data_type = read_array_metadata_v3(document, context=CORE_AND_EXTENSIONS_CONTEXT).data_type + assert isinstance(data_type, AcceptedField) is in_core_and_extensions + assert isinstance(data_type, UnclaimedField) is not in_core_and_extensions + + +def test_zarr_python_is_core_and_extensions_with_zarr_python_s_own_data_types() -> None: + own = CORE_AND_EXTENSIONS_CONTEXT.update( + data_types=[ + *CORE_AND_EXTENSIONS_CONTEXT.data_types.values(), + NULL_TERMINATED_BYTES_DATA_TYPE, + VARIABLE_LENGTH_BYTES_DATA_TYPE, + ] + ) + assert own == ZARR_PYTHON_CONTEXT + + +@pytest.mark.parametrize( + ("data_type", "value", "canonical"), + [ + (_UTF32, "", ""), + (_UTF32, "abc", "abc"), + # A value is padded with null bytes, so trailing ones are the same value. + (_NULL_TERMINATED, "QQAA", "QQ=="), + (_NULL_TERMINATED, "QR==", "QQ=="), + (_NULL_TERMINATED, "", ""), + ("variable_length_bytes", "QR==", "QQ=="), + ("variable_length_bytes", "QQAA", "QQAA"), + ], +) +def test_the_canonical_fill_value(data_type: object, value: str, canonical: str) -> None: + resolved, problems = resolve(data_type, DataTypeDefinition, ZARR_PYTHON_CONTEXT) + assert problems == () + assert fill_value_problems(resolved, value) == () + assert canonical_fill_value(resolved, value) == canonical + + +@pytest.mark.parametrize( + "data_type", + [ + {"name": "fixed_length_utf32", "configuration": {"length_bytes": 0}}, + {"name": "fixed_length_utf32", "configuration": {"length_bytes": 6}}, + {"name": "null_terminated_bytes", "configuration": {"length_bytes": 0}}, + ], + ids=["utf32-zero", "utf32-not-a-multiple-of-4", "null-terminated-zero"], +) +def test_error_a_size_that_holds_no_whole_value(data_type: object) -> None: + resolved, problems = resolve(data_type, DataTypeDefinition, ZARR_PYTHON_CONTEXT) + assert isinstance(resolved, RefusedField) + assert [(p.loc, p.kind) for p in problems] == [ + (("configuration", "length_bytes"), "invalid_value") + ] + + +@pytest.mark.parametrize( + ("data_type", "value"), + [(_UTF32, "abcd"), (_NULL_TERMINATED, "YWJjZGU=")], + ids=["utf32-four-code-points", "null-terminated-five-bytes"], +) +def test_error_a_fill_value_longer_than_the_type(data_type: object, value: str) -> None: + resolved, _ = resolve(data_type, DataTypeDefinition, ZARR_PYTHON_CONTEXT) + assert [(p.loc, p.kind) for p in fill_value_problems(resolved, value)] == [ + ((), "invalid_value") + ] + + +@pytest.mark.parametrize("data_type", [_NULL_TERMINATED, "variable_length_bytes"]) +def test_error_a_fill_value_that_is_not_base64(data_type: object) -> None: + resolved, _ = resolve(data_type, DataTypeDefinition, ZARR_PYTHON_CONTEXT) + assert [(p.loc, p.kind) for p in fill_value_problems(resolved, "YWI")] == [ + ((), "invalid_value") + ] + + +@pytest.mark.parametrize("codec", ["vlen-utf8", "vlen-bytes"]) +def test_error_a_variable_length_codec_for_a_data_type_of_fixed_size(codec: str) -> None: + reading = read_array_metadata_v3(_document(_UTF32, "", [codec]), context=ZARR_PYTHON_CONTEXT) + assert [(p.loc, p.kind) for p in reading.problems] == [ + (("codecs", 0, "configuration"), "invalid_value") + ] + + +def test_error_a_bytes_codec_without_an_endian_for_fixed_length_utf32() -> None: + reading = read_array_metadata_v3(_document(_UTF32, "", ["bytes"]), context=ZARR_PYTHON_CONTEXT) + assert [(p.loc, p.kind) for p in reading.problems] == [ + (("codecs", 0, "configuration", "endian"), "missing_key") + ] + + +def test_an_empty_scope_reads_none_of_them() -> None: + for name in ("fixed_length_utf32", "null_terminated_bytes", "variable_length_bytes"): + assert ZarrV3Context().claimant(DataTypeDefinition, name) is None