fix(zarr-metadata)!: locate every problem in the input and align the v2 readers with v3

packages/zarr-metadata/changes/4574.bugfix.1.md

@@ -0,0 +1,7 @@
+**Breaking:** A problem in a v2 struct dtype is located where the document
+    writes it. A record's problem was located under a `fields` key the
+    document does not have, at `("dtype", "fields", 0, 1)`; it is now at
+    `("dtype", 0, 1)`, and carries the value found there as its `input`.
+    `fields()` places each record's type there too. A definition's
+    functions still find the record types in `nested` at
+    `("fields", 0, 1)`, their place in the configuration.

packages/zarr-metadata/changes/4574.bugfix.2.md

@@ -0,0 +1,4 @@
+A v2 dtype written as the string `"struct"` is refused as a string that is
+    not a NumPy typestr, like any other, instead of being read as a struct
+    with no records. `struct` is the name the definition of a records array
+    is filed under, not part of the v2 dtype grammar.

packages/zarr-metadata/changes/4574.bugfix.3.md

@@ -0,0 +1,5 @@
+A group listed in v3 consolidated metadata whose own document has a problem,
+    such as attributes that are not an object, is reported for that problem
+    alone. It was also reported for disagreeing with the outer listing's copy
+    of the same node, a difference that came only from the problem. A copy
+    whose only problem is an unknown key is still compared.

packages/zarr-metadata/changes/4574.bugfix.4.md

@@ -0,0 +1,7 @@
+A v2 struct record of the wrong length is reported as "expected an array of
+    2 or 3 elements", where the message named only the three-element form. A
+    two-element record whose name is not a string, `[5, "<f4"]`, is reported
+    at the name, where it was reported as a record of the wrong length. The
+    checker behind `zarr_metadata.typed_json.check` tries an array only
+    against the union branches of its length, and names a union of arrays of
+    fixed lengths by those lengths.

packages/zarr-metadata/changes/4574.bugfix.5.md

@@ -0,0 +1,11 @@
+**Breaking:** The v2 readers agree with the v3 readers in three places. A
+    `.zarray` member the spec does not define may hold `NaN` and the
+    infinities, as a v3 member that nothing claims may, in a store and in a
+    `.zmetadata` alike. A `.zmetadata` key that does not name its node's
+    path, such as `/a/.zarray` or `a//b/.zarray`, is refused, as a v3
+    listing refuses such a key, instead of being read as the node at `a` or
+    `a/b`; zarr-python looks an entry up by its node's path and never finds
+    one written otherwise. A `.zgroup` that holds `attributes` is an
+    `invalid_value`, as a `.zarray` that holds them is, not an
+    `unknown_key`, so a caller who tolerates unknown keys does not drop the
+    attributes it holds.

packages/zarr-metadata/changes/4574.bugfix.6.md

@@ -0,0 +1,8 @@
+A v2 typestr with a part its family does not take, such as `<f4[ns]` or
+    `|O4`, is refused with one `invalid_value` problem that names the part:
+    "expected a name 'float' reads, got "<f4[ns]", which encodes a unit that
+    it does not take". It was refused as one `unknown_key` problem per
+    internal configuration key (`unit`, `scale_factor`, `itemsize`), a kind
+    a caller who tolerates unknown keys leaves out, so such a caller read the
+    typestr as valid. A time typestr without a unit, `<M8`, is an
+    `invalid_value` too, not a `missing_key`.

packages/zarr-metadata/changes/4574.bugfix.7.md

@@ -0,0 +1,9 @@
+A definition's rules are located where the document writes the field. A
+    `bytes` codec written as `"bytes"` or `{"name": "bytes"}`, with values
+    of several bytes, reports its missing `endian` at
+    `codecs.N.configuration`, the configuration it does not write, as a
+    codec that requires a configuration does; it was reported at
+    `codecs.N.configuration.endian`, inside a configuration that is not
+    there. A problem with such a field's configuration as a whole, such as a
+    data type of varying size, is at the field. This holds for a
+    definition's `rules` and a codec's `chunk_rules` alike.

packages/zarr-metadata/changes/4574.bugfix.8.md

@@ -0,0 +1,4 @@
+A float given where a document requires one of a set of integers, such as
+    `"zarr_format": 2.0`, is an `invalid_type`, as `4.0` in `shape` is and
+    as a blosc `"shuffle": 1.0` already was. It was an `invalid_value`
+    wherever the set stood alone, so one mistake had two kinds.

packages/zarr-metadata/changes/4574.bugfix.9.md

@@ -0,0 +1,12 @@
+**Breaking:** `read_node_metadata_v2`, `read_repaired_node_metadata_v2`
+    and the v2 models' `from_key_value` locate each problem in the file that
+    holds it, `(".zarray", "order")` or `(".zattrs", "x")`, as
+    `read_consolidated_metadata_v2` locates it in its entry. They located
+    problems in the merged document, so a `.zarray` that wrote `attributes`
+    beside a `.zattrs` that was not an object gave two problems at
+    `("attributes",)`, a location in neither file. An array node's reading
+    locates its fields there too: its new `at` is `(".zarray",)`, so
+    `fields()` gives `(".zarray", "dtype")` and `with_problems` pairs each
+    field with its problems. Its model reads the merged document, so the
+    model's own `reading` locates the field at `("dtype",)`, and no longer
+    equals the node reading.

packages/zarr-metadata/src/zarr_metadata/_hierarchy.py

@@ -16,12 +16,19 @@
 from zarr_metadata._json import ValidationProblem, shown
 
 if TYPE_CHECKING:
-    from collections.abc import Mapping
+    from collections.abc import Mapping, Sequence
 
 NodeType = Literal["array", "group"]
 """The two kinds of node in a hierarchy."""
 
 
+def said(faults: Sequence[str]) -> str:
+    """Join `faults`, each a reason, into one clause: `a`, `a and b`, `a, b and c`."""
+    if len(faults) <= 1:
+        return "".join(faults)
+    return f"{', '.join(faults[:-1])} and {faults[-1]}"
+
+
 def tree_problems(nodes: Mapping[str, NodeType | None]) -> tuple[ValidationProblem, ...]:
     """The reasons `nodes` (the node type of each node, keyed by a valid node path) do not form a tree. Reports one problem per affected node, at that node's path.
 
@@ -129,4 +136,4 @@ def _names(path: str) -> list[str]:
     return [] if path == "/" else path[1:].split("/")
 
 
-__all__ = ["NodeType", "tree_problems"]
+__all__ = ["NodeType", "said", "tree_problems"]

packages/zarr-metadata/src/zarr_metadata/_json.py

@@ -652,11 +652,22 @@ def json_type(value: object) -> str:
 
 
 def refused_kind(value: object, allowed: Sequence[object]) -> ProblemKind:
-    """The kind of problem for `value` outside a closed set: `invalid_type` when no member of the set has its JSON type, else `invalid_value`."""
-    same_type = json_type(value) in {json_type(entry) for entry in allowed}
+    """The kind of problem for `value` outside a closed set: `invalid_type` when no member of the set has its type, else `invalid_value`.
+
+    An integer is a type of its own here, as everywhere a reader expects
+    one: `2.0` for `zarr_format` is the wrong type, as `4.0` in `shape` is.
+    """
+    same_type = _set_type(value) in {_set_type(entry) for entry in allowed}
     return "invalid_value" if same_type else "invalid_type"
 
 
+def _set_type(value: object) -> str:
+    """The type `refused_kind` compares: the JSON type, with an integer told apart from any other number."""
+    if isinstance(value, int) and not isinstance(value, bool):
+        return "an integer"
+    return json_type(value)
+
+
 def is_canonical_json(value: object, *, finite: bool = True) -> TypeGuard[JSONValue]:
     """Whether `value` already uses the concrete containers in `JSONValue`, nested no deeper than `JSON_DEPTH`.
 

packages/zarr-metadata/src/zarr_metadata/_typed_json.py

@@ -454,10 +454,19 @@ def _named(annotation: object, seen: frozenset[object]) -> tuple[str, str]:
         return choices(values), f"values in [{listed}]"
     if is_union(inner):
         # Name each shape once, since two TypedDicts are both "an object".
-        # A broader shape absorbs a narrower one: "a number" absorbs "an
-        # integer", and "a string" absorbs a `Literal` of strings.
+        # Arrays of two or more fixed lengths are named by their lengths:
+        # a union reports an array against itself only when it has none of
+        # them. A broader shape absorbs a narrower one: "a number" absorbs
+        # "an integer", and "a string" absorbs a `Literal` of strings.
         branches = get_args(inner)
-        named = dict.fromkeys(_named(branch, seen) for branch in branches)
+        lengths = sorted({length for length in map(fixed_length, branches) if length is not None})
+        if len(lengths) > 1:
+            listed = f"{', '.join(map(str, lengths[:-1]))} or {lengths[-1]}"
+            arrays = (f"an array of {listed} elements", f"arrays of {listed} elements")
+            rest = [branch for branch in branches if fixed_length(branch) is None]
+            named = dict.fromkeys([arrays, *(_named(branch, seen) for branch in rest)])
+        else:
+            named = dict.fromkeys(_named(branch, seen) for branch in branches)
         if ("a number", "numbers") in named:
             named.pop(("an integer", "integers"), None)
         if ("a string", "strings") in named:
@@ -543,6 +552,22 @@ def shape_of(annotation: object) -> str | None:
     return None
 
 
+def fixed_length(annotation: object) -> int | None:
+    """The length of an array of a fixed length, such as `tuple[str, int]`; None for any other annotation, including `tuple[int, ...]`."""
+    inner = _unaliased(annotation)
+    if get_origin(inner) is not tuple:
+        return None
+    arguments = get_args(inner)
+    if len(arguments) == 2 and arguments[1] is Ellipsis:
+        return None
+    return len(arguments)
+
+
+def _has_length(length: int | None, value: object) -> bool:
+    """Whether `value` has `length` elements, when a branch is an array of a fixed length. Any value passes a branch without one."""
+    return length is None or (is_list_or_tuple(value) and len(value) == length)
+
+
 def has_shape(shape: str | None, value: object) -> bool:
     if shape is None:
         return True
@@ -640,8 +665,8 @@ def parse(value: object, loc: Loc) -> Parsed:
     return parse
 
 
-Branch: TypeAlias = tuple[str | None, Parser, frozenset[str] | None]
-"""A branch of a union: its top-level shape (from `shape_of`), its parser, and its keys if it is a TypedDict."""
+Branch: TypeAlias = tuple[str | None, int | None, Parser, frozenset[str] | None]
+"""A branch of a union: its top-level shape (from `shape_of`), its length if it is an array of a fixed length (from `fixed_length`), its parser, and its keys if it is a TypedDict."""
 
 Tag: TypeAlias = tuple[str, Mapping[tuple[type, object], int]]
 """A key that every branch requires as a `Literal`, and the branch that each of its values selects."""
@@ -657,7 +682,8 @@ def any_of(
     """A member whose type is a union of shapes, parsed by the matching branch.
 
     The parser tries only the branches whose top-level shape matches the
-    value. This way a bad element inside an array is reported at its
+    value, and an array of a fixed length only on an array of that length.
+    This way a bad element inside an array is reported at its
     position inside the array, rather than the whole array being reported
     as wrong. A value that matches no branch's shape is reported once,
     with the expected shapes. When every branch is a TypedDict that
@@ -683,8 +709,8 @@ def parse(value: object, loc: Loc) -> Parsed:
             return _by_tag(branches, tag, value, loc)
         clean: list[tuple[int, int, Parsed]] = []
         failed: list[tuple[int, int, Parsed]] = []
-        for index, (shape, branch, keys) in enumerate(branches):
-            if not has_shape(shape, value):
+        for index, (shape, length, branch, keys) in enumerate(branches):
+            if not has_shape(shape, value) or not _has_length(length, value):
                 continue
             result = branch(value, loc)
             if len(result[1]) != 0:
@@ -716,7 +742,8 @@ def _by_tag(branches: Sequence[Branch], tag: Tag, value: object, loc: Loc) -> Pa
     index = picks.get((type(said), said)) if _hashable(said) else None
     if index is None:
         return value, (outside_of((*loc, key), said, tuple(entry for _, entry in picks)),)
-    return branches[index][1](value, loc)
+    _, _, parse, _ = branches[index]
+    return parse(value, loc)
 
 
 def _keys_of(value: object) -> AbstractSet[str] | None:
@@ -953,7 +980,7 @@ def _union(inner: object, leaf: Leaf, building: _Building) -> Parser | None:
             if isinstance(typeddict, type) and is_typeddict(typeddict)
             else None
         )
-        branches.append((shape_of(branch), member, keys))
+        branches.append((shape_of(branch), fixed_length(branch), member, keys))
     literals = frozenset(
         index for index, branch in enumerate(arguments) if get_origin(_unaliased(branch)) is Literal
     )