test: give the sharded coordinate-selection write path real weight in test_oindex - #4331
Draft
d-v-b wants to merge 3 commits into
Draft
test: give the sharded coordinate-selection write path real weight in test_oindex#4331d-v-b wants to merge 3 commits into
d-v-b wants to merge 3 commits into
Conversation
… test_oindex Add zarr.testing.strategies.sharded_arrays, a strategy that always yields a sharded v3 array (drawing chunk shape, shard shape, subchunk write order and inner codec chain), and draw it as a third arm in test_oindex and test_vindex. simple_arrays shards only a few percent of its draws, so under the derandomized ci profile test_oindex reached the ShardingCodec value-reshape guard (GH4284/GH4316) in 2 of 300 examples; it now reaches it in about 50 (13-27% of writes across random seeds), and the test fails without the guard. The sharded arm in test_oindex draws at least two dimensions, since the guard only applies to a value with two or more axes. A Hypothesis event at the write site reports the reach under --hypothesis-show-statistics. Narrow the docstring of test_set_selection_rejects_value_with_wrong_rank to the cases it pins: storage layout can change which writes are rejected, e.g. oindex[[3, 1], [0, 2]] with a (2, 2, 1) value is accepted chunked but raises sharded. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4331 +/- ##
==========================================
+ Coverage 94.30% 94.32% +0.01%
==========================================
Files 92 92
Lines 12915 12941 +26
==========================================
+ Hits 12180 12206 +26
Misses 735 735
🚀 New features to boost your workflow:
|
d-v-b
commented
Sep 9, 2026
| return ShardingCodec( | ||
| subchunk_write_order=subchunk_write_order, | ||
| codecs=inner_codecs, | ||
| index_codecs=[BytesCodec(), Crc32cCodec()], |
Contributor
Author
There was a problem hiding this comment.
consider supporting recursive sharding. A single recursion is likely enough to catch weird issues.
Review asked for recursive sharding in the new strategy; one level is enough to reach the shard-within-shard paths, which is where the original GH4280 failure lived. Building it exposed that the strategy already nested by accident: passing `shards=` makes `create_array` wrap the drawn `ShardingCodec` in a second one with the same chunk shape, so every "sharded" draw was a shard of one-chunk inner shards and the drawn subchunk write order only ever reached the inner codec, never the outer. `sharded_arrays` now makes the shard the array's chunk grid and passes the drawn codec as the serializer, so a single-level draw really is single-level and the outer codec carries the drawn write order. A drawn (or `nested=`) flag then wraps that codec in another `ShardingCodec` whose chunk shape is an integral number of chunks and whose shard is an integral number of those. `_sharding_codecs` takes an optional inner codec chain to build the outer layer. (`arrays()` still uses the `shards=` form and so keeps the accidental one-chunk nesting; left alone here.) Measured on test_oindex under the ci profile: 24% of writes take the sharded coordinate-selection path, 16% of examples are nested, and the outer codec sees all four write orders instead of only `morton`. Assisted-by: ClaudeCode:claude-fable-5-1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Summary
#4284 / #4316 fixed orthogonal writes on sharded arrays where the sharding codec re-derives a
CoordinateIndexerand must ravel the value, but the property test meant to cover it,test_oindex, reached that path in 2 of 300 examples under the derandomizedciHypothesis profile — a fixed outcome, so a green run said almost nothing about the fix. The test drawsone_of(simple_arrays, rectilinear_arrays), rectilinear arrays cannot shard, and only ~8.6% ofsimple_arraysdraws are sharded.This adds
zarr.testing.strategies.sharded_arrays, a strategy that always yields a sharded Zarr format 3 array (chunk shape, shard shape, subchunk write order and inner codec chain are drawn; theShardingCodecconstruction is factored into a private helper shared witharrays()with the same draw order, so existing derandomized sequences are unchanged), and draws it as a third arm intest_oindexandtest_vindex. The value-reshape path is now exercised in ~50 of 300ciexamples (13–27% of writes across random seeds) with no measurable change in suite time, and removing the guard insharding.pymakestest_oindexfail (shrunk example: a(1, 3, 1)array with shards(1, 3, 1), selection(array([0]), slice(1, 3), 0), value shape(1, 2)). A Hypothesisevent()at the write site makes the reach visible under--hypothesis-show-statistics.Separately, the docstring of
test_set_selection_rejects_value_with_wrong_rankintests/test_indexing.pyclaimed storage layout never changes which writes are rejected; it does — a(2, 2, 1)value onoindex[[3, 1], [0, 2]]is accepted by a chunked array (each chunk's piece is(1, 1, 1), which numpy broadcasts) and rejected by a sharded one — so the docstring now states exactly what the test pins.Found during the pre-release review for #4256.
For reviewers
The design choice worth a look is a dedicated strategy rather than a
sharded=knob onarrays(): Zarr format 2 arrays cannot shard, so a knob would have to silently overridezarr_formats. Note that the fixed path is reached by a bare-int axis plus one array axis too (OrthogonalIndexerroutes throughix_whenn_array_dims > 1 or drop_axes), not only by two or more array axes; themin_dims=2on thetest_oindexarm follows from the guard needing a ≥2-D value.derandomize=Truere-rolls the outcome on every edit to the test body, so thecireach count is only stable while the test is.Author attestation
TODO
sharded_arraysis picked up by the existingzarr.testing.strategiesAPI page)docs/user-guide/*.md(n/a)changes/