Skip to content

fix(dtype): write the generic time unit in brackets in Zarr format 2 metadata - #421

Draft
d-v-b wants to merge 11 commits into
mainfrom
fix/v2-time-unit-brackets
Draft

d-v-b wants to merge 11 commits into
mainfrom
fix/v2-time-unit-brackets

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Fixed Zarr format 2 metadata written for generic time units: the v2 spec requires the unit of a datetime64 or timedelta64 dtype in brackets, but a generic-unit time dtype serialized as NumPy's unitless typestr (<M8). The generic unit is now written explicitly — <M8[generic], or <M8[2generic] with a scale, as scaled generic units already were — which NumPy parses back to the same dtype (np.dtype("M8[generic]") → generic datetime64). The unitless spellings stay readable as the generic unit, per zarr-developers#4580: acceptable for reading, not for writing.

Consequences a reviewer should weigh:

  • The unitless spelling no longer round-trips verbatim: from_json("<M8") reads as the generic unit and writes back as <M8[generic]. A new test pins the read-acceptance and the write-normalization; the dtype fixture generator now brackets every unit.
  • [generic] is bracketed and NumPy-parseable, but "generic" is not in the unit table the spec points to, so strict spec readers (including zarr-metadata's v2 reader, whose NUMPY_TIME_UNITS excludes it) still refuse it. Whether zarr-metadata should accept generic is left to avoid writing invalid zarr v2 datetime metadata zarr-developers/zarr-python#4580.

Fixes zarr-developers#4580

🤖 Generated with Claude Code

d-v-b and others added 7 commits October 29, 2026 10:37
…rs#4554)

* fix(api): drop unreachable scalar branch in save_array

`save_array` rejects anything that is not `NDArrayLike` before it reaches
`np.isscalar(arr)`. numpy scalars lack `__len__` and `__setitem__`, so they
never pass the protocol check, and the `np.array(arr)` fallback was dead.

numpy 2.5.4 changed the `np.isscalar` stub from `TypeGuard` to `TypeIs`,
which lets mypy intersect the narrowed `NDArrayLike` with the scalar union
and report the branch as unreachable. Delete it and pin the error path.

Assisted-by: ClaudeCode:claude-fable-5-1

* docs(changes): add changelog fragment for zarr-developers#4554

Assisted-by: ClaudeCode:claude-opus-5-5
…th 15 updates (zarr-developers#4577)

Bumps the python-dependencies group with 15 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [numpy](https://github.com/numpy/numpy) | `2.5.3` | `2.5.4` |
| [obstore](https://github.com/geospatial-jeff/pyasyncio-benchmark) | `0.11.1` | `0.12.0` |
| [typer](https://github.com/fastapi/typer) | `0.27.2` | `0.27.3` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.168.3` | `6.170.5` |
| [tomlkit](https://github.com/python-poetry/tomlkit) | `0.15.1` | `0.15.2` |
| [uv](https://github.com/astral-sh/uv) | `0.13.4` | `0.13.5` |
| [botocore](https://github.com/boto/botocore) | `1.43.106` | `1.43.117` |
| [moto[s3,server]](https://github.com/getmoto/moto) | `5.2.3` | `5.2.4` |
| [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) | `1.0.6` | `1.0.8` |
| [mkdocstrings-python](https://github.com/mkdocstrings/python) | `2.0.9` | `2.0.10` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.3` | `1.12.5` |
| [griffe-inherited-docstrings](https://github.com/mkdocstrings/griffe-inherited-docstrings) | `1.1.3` | `1.1.4` |
| [ruff](https://github.com/astral-sh/ruff) | `0.16.9` | `0.17.1` |
| [astroid](https://github.com/pylint-dev/astroid) | `4.3.3` | `4.3.4` |
| [mypy](https://github.com/python/mypy) | `2.3.1` | `2.4.0` |



Updates `numpy` from 2.5.3 to 2.5.4
- [Release notes](https://github.com/numpy/numpy/releases)
- [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst)
- [Commits](numpy/numpy@v2.5.3...v2.5.4)

Updates `obstore` from 0.11.1 to 0.12.0
- [Commits](https://github.com/geospatial-jeff/pyasyncio-benchmark/commits)

Updates `typer` from 0.27.2 to 0.27.3
- [Release notes](https://github.com/fastapi/typer/releases)
- [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md)
- [Commits](fastapi/typer@0.27.2...0.27.3)

Updates `hypothesis` from 6.168.3 to 6.170.5
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.168.3...v6.170.5)

Updates `tomlkit` from 0.15.1 to 0.15.2
- [Release notes](https://github.com/python-poetry/tomlkit/releases)
- [Changelog](https://github.com/python-poetry/tomlkit/blob/master/CHANGELOG.md)
- [Commits](python-poetry/tomlkit@0.15.1...0.15.2)

Updates `uv` from 0.13.4 to 0.13.5
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](astral-sh/uv@0.13.4...0.13.5)

Updates `botocore` from 1.43.106 to 1.43.117
- [Commits](boto/botocore@1.43.106...1.43.117)

Updates `moto[s3,server]` from 5.2.3 to 5.2.4
- [Release notes](https://github.com/getmoto/moto/releases)
- [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md)
- [Commits](getmoto/moto@5.2.3...5.2.4)

Updates `mkdocstrings` from 1.0.6 to 1.0.8
- [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases)
- [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/mkdocstrings@1.0.6...1.0.8)

Updates `mkdocstrings-python` from 2.0.9 to 2.0.10
- [Release notes](https://github.com/mkdocstrings/python/releases)
- [Changelog](https://github.com/mkdocstrings/python/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/python@2.0.9...2.0.10)

Updates `markdown-exec[ansi]` from 1.12.3 to 1.12.5
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.3...1.12.5)

Updates `griffe-inherited-docstrings` from 1.1.3 to 1.1.4
- [Release notes](https://github.com/mkdocstrings/griffe-inherited-docstrings/releases)
- [Changelog](https://github.com/mkdocstrings/griffe-inherited-docstrings/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/griffe-inherited-docstrings@1.1.3...1.1.4)

Updates `ruff` from 0.16.9 to 0.17.1
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.16.9...0.17.1)

Updates `astroid` from 4.3.3 to 4.3.4
- [Release notes](https://github.com/pylint-dev/astroid/releases)
- [Changelog](https://github.com/pylint-dev/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v4.3.3...v4.3.4)

Updates `mypy` from 2.3.1 to 2.4.0
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v2.3.1...v2.4.0)

---
updated-dependencies:
- dependency-name: numpy
  dependency-version: 2.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: obstore
  dependency-version: 0.12.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: typer
  dependency-version: 0.27.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.170.5
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: tomlkit
  dependency-version: 0.15.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.13.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: botocore
  dependency-version: 1.43.117
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: moto[s3,server]
  dependency-version: 5.2.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocstrings
  dependency-version: 1.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocstrings-python
  dependency-version: 2.0.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.12.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: griffe-inherited-docstrings
  dependency-version: 1.1.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.17.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: astroid
  dependency-version: 4.3.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mypy
  dependency-version: 2.4.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
)

Bumps the actions group with 4 updates: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv), [CodSpeedHQ/action](https://github.com/codspeedhq/action), [actions/setup-python](https://github.com/actions/setup-python) and [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish).


Updates `astral-sh/setup-uv` from 10.3.0 to 10.3.2
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@1c37ad0...07a047e)

Updates `CodSpeedHQ/action` from 5.5.0 to 5.6.0
- [Release notes](https://github.com/codspeedhq/action/releases)
- [Changelog](https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md)
- [Commits](CodSpeedHQ/action@821a40c...78102a4)

Updates `actions/setup-python` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@5fda3b9...6d4fd22)

Updates `pypa/gh-action-pypi-publish` from 1.14.2 to 1.14.3
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@dc37677...7245906)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 10.3.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: CodSpeedHQ/action
  dependency-version: 5.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: actions/setup-python
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rr-developers#4561)

Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [certifi](https://github.com/certifi/python-certifi).
Bumps the uv group with 1 update in the /packages/zarr-indexing directory: [certifi](https://github.com/certifi/python-certifi).
Bumps the uv group with 1 update in the /packages/zarr-metadata directory: [certifi](https://github.com/certifi/python-certifi).


Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

Updates `certifi` from 2026.7.22 to 2026.10.19
- [Commits](certifi/python-certifi@2026.07.22...2026.10.19)

---
updated-dependencies:
- dependency-name: certifi
  dependency-version: 2026.10.19
  dependency-type: indirect
- dependency-name: certifi
  dependency-version: 2026.10.19
  dependency-type: indirect
- dependency-name: certifi
  dependency-version: 2026.10.19
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [msgpack](https://github.com/msgpack/msgpack-python) from 1.2.1 to 1.2.3.
- [Release notes](https://github.com/msgpack/msgpack-python/releases)
- [Changelog](https://github.com/msgpack/msgpack-python/blob/main/CHANGELOG.md)
- [Commits](msgpack/msgpack-python@v1.2.1...v1.2.3)

---
updated-dependencies:
- dependency-name: msgpack
  dependency-version: 1.2.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…metadata

The v2 spec requires the unit of a datetime64 or timedelta64 dtype in
brackets, but a generic-unit time dtype serialized as NumPy's unitless
typestr (<M8), which the spec forbids. The generic unit is now written
explicitly — <M8[generic], or <M8[2generic] with a scale, as scaled
generic units already were — which NumPy parses back to the same dtype.
The unitless spellings stay readable as the generic unit.

Fixes zarr-developers#4580

Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [tornado](https://github.com/tornadoweb/tornado).


Updates `tornado` from 6.5.9 to 6.6
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.5.9...v6.6.0)

---
updated-dependencies:
- dependency-name: tornado
  dependency-version: '6.6'
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Comment thread src/zarr/core/dtype/npy/time.py Outdated
Comment on lines +501 to +505
if self.unit == "generic":
# NumPy's dtype.str omits the generic unit (and its scale), but the
# v2 spec requires a bracketed unit, so write it explicitly.
scale = "" if self.scale_factor == 1 else self.scale_factor
name += f"[{scale}generic]"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactor the duplicated code into a shared routine

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI text below 🤖

The agent moved the bracketing into TimeDTypeBase._v2_name, which both to_json implementations now call (7541009).

…ime types

The generic-unit bracketing moves to TimeDTypeBase._v2_name, which both
to_json implementations call, instead of being duplicated in each.

Assisted-by: ClaudeCode:claude-fable-5
Comment thread tests/test_dtype/test_npy/test_time.py Outdated
)


def test_unitless_time_v2_reads_as_generic_and_writes_bracketed() -> None:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either parametrize this test, or even better, fold it in to the existing dtype serialization tests (we might not have alias-aware tests, but we should probably add them)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI text below 🤖

The agent folded the one-off test into the shared dtype wrapper as alias-aware cases: json_alias_v2_params on the test base takes (alias, canonical) pairs, asserting both spellings read as the same dtype and write the canonical one. The time types list the unitless generic spellings and NumPy's μs alias (3b6705e).

d-v-b added 2 commits October 29, 2026 16:19
… tests

The one-off unitless-time test becomes json_alias_v2_params on the
shared dtype test base: (alias, canonical) pairs asserting that reading
either spelling gives the same dtype and writing gives the canonical
one. The time types list the unitless generic spellings and NumPy's
microsecond alias; other types can list their aliases the same way.

Assisted-by: ClaudeCode:claude-fable-5
d-v-b added a commit that referenced this pull request Oct 29, 2026
…metadata (zarr-developers#4581)

* fix(dtype): write the generic time unit in brackets in Zarr format 2 metadata

The v2 spec requires the unit of a datetime64 or timedelta64 dtype in
brackets, but a generic-unit time dtype serialized as NumPy's unitless
typestr (<M8), which the spec forbids. The generic unit is now written
explicitly — <M8[generic], or <M8[2generic] with a scale, as scaled
generic units already were — which NumPy parses back to the same dtype.
The unitless spellings stay readable as the generic unit.

Fixes zarr-developers#4580

Assisted-by: ClaudeCode:claude-fable-5

* docs: changelog fragment for #421

Assisted-by: ClaudeCode:claude-fable-5

* refactor(dtype): share the v2 time typestr spelling between the two time types

The generic-unit bracketing moves to TimeDTypeBase._v2_name, which both
to_json implementations call, instead of being duplicated in each.

Assisted-by: ClaudeCode:claude-fable-5

* test(dtype): check v2 JSON alias spellings through the shared wrapper tests

The one-off unitless-time test becomes json_alias_v2_params on the
shared dtype test base: (alias, canonical) pairs asserting that reading
either spelling gives the same dtype and writing gives the canonical
one. The time types list the unitless generic spellings and NumPy's
microsecond alias; other types can list their aliases the same way.

Assisted-by: ClaudeCode:claude-fable-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

avoid writing invalid zarr v2 datetime metadata

1 participant