Skip to content

test: give the sharded coordinate-selection write path real weight in test_oindex - #4331

Draft
d-v-b wants to merge 3 commits into
zarr-developers:mainfrom
d-v-b:test/sharded-oindex-property-reach
Draft

test: give the sharded coordinate-selection write path real weight in test_oindex#4331
d-v-b wants to merge 3 commits into
zarr-developers:mainfrom
d-v-b:test/sharded-oindex-property-reach

Conversation

@d-v-b

@d-v-b d-v-b commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

Summary

#4284 / #4316 fixed orthogonal writes on sharded arrays where the sharding codec re-derives a CoordinateIndexer and must ravel the value, but the property test meant to cover it, test_oindex, reached that path in 2 of 300 examples under the derandomized ci Hypothesis profile — a fixed outcome, so a green run said almost nothing about the fix. The test draws one_of(simple_arrays, rectilinear_arrays), rectilinear arrays cannot shard, and only ~8.6% of simple_arrays draws are sharded.

This adds zarr.testing.strategies.sharded_arrays, a strategy that always yields a sharded Zarr format 3 array (chunk shape, shard shape, subchunk write order and inner codec chain are drawn; the ShardingCodec construction is factored into a private helper shared with arrays() with the same draw order, so existing derandomized sequences are unchanged), and draws it as a third arm in test_oindex and test_vindex. The value-reshape path is now exercised in ~50 of 300 ci examples (13–27% of writes across random seeds) with no measurable change in suite time, and removing the guard in sharding.py makes test_oindex fail (shrunk example: a (1, 3, 1) array with shards (1, 3, 1), selection (array([0]), slice(1, 3), 0), value shape (1, 2)). A Hypothesis event() at the write site makes the reach visible under --hypothesis-show-statistics.

Separately, the docstring of test_set_selection_rejects_value_with_wrong_rank in tests/test_indexing.py claimed storage layout never changes which writes are rejected; it does — a (2, 2, 1) value on oindex[[3, 1], [0, 2]] is accepted by a chunked array (each chunk's piece is (1, 1, 1), which numpy broadcasts) and rejected by a sharded one — so the docstring now states exactly what the test pins.

Found during the pre-release review for #4256.

For reviewers

The design choice worth a look is a dedicated strategy rather than a sharded= knob on arrays(): Zarr format 2 arrays cannot shard, so a knob would have to silently override zarr_formats. Note that the fixed path is reached by a bare-int axis plus one array axis too (OrthogonalIndexer routes through ix_ when n_array_dims > 1 or drop_axes), not only by two or more array axes; the min_dims=2 on the test_oindex arm follows from the guard needing a ≥2-D value. derandomize=True re-rolls the outcome on every edit to the test body, so the ci reach count is only stable while the test is.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions (sharded_arrays is picked up by the existing zarr.testing.strategies API page)
  • New/modified features documented in docs/user-guide/*.md (n/a)
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

… test_oindex

Add zarr.testing.strategies.sharded_arrays, a strategy that always yields a
sharded v3 array (drawing chunk shape, shard shape, subchunk write order and
inner codec chain), and draw it as a third arm in test_oindex and test_vindex.
simple_arrays shards only a few percent of its draws, so under the derandomized
ci profile test_oindex reached the ShardingCodec value-reshape guard
(GH4284/GH4316) in 2 of 300 examples; it now reaches it in about 50 (13-27% of
writes across random seeds), and the test fails without the guard. The
sharded arm in test_oindex draws at least two dimensions, since the guard only
applies to a value with two or more axes. A Hypothesis event at the write site
reports the reach under --hypothesis-show-statistics.

Narrow the docstring of test_set_selection_rejects_value_with_wrong_rank to
the cases it pins: storage layout can change which writes are rejected, e.g.
oindex[[3, 1], [0, 2]] with a (2, 2, 1) value is accepted chunked but raises
sharded.

Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.32%. Comparing base (542ceba) to head (67e69cd).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4331      +/-   ##
==========================================
+ Coverage   94.30%   94.32%   +0.01%     
==========================================
  Files          92       92              
  Lines       12915    12941      +26     
==========================================
+ Hits        12180    12206      +26     
  Misses        735      735              
Files with missing lines Coverage Δ
src/zarr/testing/strategies.py 96.14% <100.00%> (+0.33%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

return ShardingCodec(
subchunk_write_order=subchunk_write_order,
codecs=inner_codecs,
index_codecs=[BytesCodec(), Crc32cCodec()],

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider supporting recursive sharding. A single recursion is likely enough to catch weird issues.

Review asked for recursive sharding in the new strategy; one level is
enough to reach the shard-within-shard paths, which is where the
original GH4280 failure lived.

Building it exposed that the strategy already nested by accident:
passing `shards=` makes `create_array` wrap the drawn `ShardingCodec` in
a second one with the same chunk shape, so every "sharded" draw was a
shard of one-chunk inner shards and the drawn subchunk write order only
ever reached the inner codec, never the outer. `sharded_arrays` now
makes the shard the array's chunk grid and passes the drawn codec as
the serializer, so a single-level draw really is single-level and the
outer codec carries the drawn write order. A drawn (or `nested=`) flag
then wraps that codec in another `ShardingCodec` whose chunk shape is
an integral number of chunks and whose shard is an integral number of
those. `_sharding_codecs` takes an optional inner codec chain to build
the outer layer. (`arrays()` still uses the `shards=` form and so keeps
the accidental one-chunk nesting; left alone here.)

Measured on test_oindex under the ci profile: 24% of writes take the
sharded coordinate-selection path, 16% of examples are nested, and the
outer codec sees all four write orders instead of only `morton`.

Assisted-by: ClaudeCode:claude-fable-5-1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant