Writing dask arrays to a ZipStore causes a corrupt zip file #3516

Activity

1. csubich

   added

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on Oct 10, 2025
2. d-v-b commented on Oct 10, 2025

   

   d-v-b

   on Oct 10, 2025

   Contributor

   More actions

   I am not an expert in the zip archive format, but my guess is that it is not safe to write concurrently. I think if you want to create zipped zarr data via dask, the best approach is probably to use dask to concurrently write data to local storage, and then zip it in a separate step.

   I'm not sure how we can guard against this corruption at runtime without implementing a distributed lock. Open to ideas here.
3. csubich commented on Oct 10, 2025

   

   csubich

   on Oct 10, 2025

   Author

   More actions

   If it were a concurrency bug, then I'd expect it to also happen with non-dask writes to the ZipStore. In the maximal code where I discovered the bug (by way of xarray), a large-ish but in-memory dataset would write successfully to a ZipStore if the chunking was left at default values (delegating it to zarr), but specifying manual chunks via xarray/dask caused this corruption.

   Additionally, this did work correctly in earlier versions of zarr, so I think there's a regression somewhere. In a colab notebook with zarr 2.16.1:

   import dask.array as da
   import numpy as np
   import zarr
   
   print(zarr.__version__)
   # 2.16.1
   
   import numcodecs
   print(numcodecs.__version__)
   # 0.12.1
   
   dask_array = da.zeros((1,),dtype=np.float32)
   
   store = zarr.storage.ZipStore("bug.zarr.zip",mode='w')
   group = zarr.open_group(store=store, mode="w")
   zarr_array = group.create(
       name="data",
       shape=dask_array.shape,
       dtype=dask_array.dtype,
       overwrite=True,
   )
   
   da.to_zarr(dask_array, zarr_array)
   store.close()
   
   store_read = zarr.storage.ZipStore("bug.zarr.zip",mode='r')
   group_read = zarr.open_group(store=store_read, mode="r")
   
   array_read = group_read["data"]
   array_read[0] 
   # np.float32(0.0)
4. d-v-b commented on Oct 10, 2025

   

   d-v-b

   on Oct 10, 2025

   Contributor

   More actions

   Additionally, this did work correctly in earlier versions of zarr, so I think there's a regression somewhere. In a colab notebook with zarr 2.16.1:

   That's good to know — I hope we can restore this functionality. Now I have no intuition for where the data corruption could be coming from, so hopefully someone who does can figure this out!
5. csubich commented on Oct 10, 2025

   

   csubich

   on Oct 10, 2025

   Last edited by csubich

   Author

   More actions

   I was hoping that the parallelism hint would work out, but the current version's ZipStore.set() takes a lock before calling _set() which performs the write, and 2.16.1's ZipStore.__setitem__ does essentially the same thing:

   Unfortunately, I'm getting lost inside dask more deeply than I have time for right now. I can confirm that this still happens with the 'synchronous' scheduler that should eliminate parallelsim, and I can also track it down into how dask.array.blockwise maps load_store_chunk onto the zarr array. However, I cannot get the bug to trigger by calling load_store_chunk myself (even via dask.delayed); easy replication seems to require calling .compute() on a generated (even if trivial) dask graph.

   Edit to add: I'm definitely flummoxed. Even if I modify ZipStore to use a monkeypatched ZipFile that logs-to-console every call to _zf.writestr, I see an identical sequence of writes between da.to_zarr (which corrupts) and a direct slice assignment (which succeeds).
6. fligt commented on Dec 11, 2025

   

   fligt

   on Dec 11, 2025

   Last edited by fligt

   More actions

   Under the hood the dask.array.to_zarr() method in the latest stable dask version '2025.11.0' uses the deprecated zarr.create() function.

   Yesterday @melonora merged a fix to the dask codebase which implements the new zarr.create_array() function that supports zarr sharding: Support zarr sharding through create_array #12153

   My workaround for now is to add dask arrays to a ZipStore by using zarr.create_array() in my code:

   import zarr 
   import dask.array as da
   
   arr = da.arange(20)
   store = zarr.storage.ZipStore('data.zip', mode='w')
   array = zarr.create_array(store=store, data=arr)
   store.close() # otherwise you will get a corrupted file 
   
   print(array)
   
   # output: 
   # <Array zip://data.zip shape=(20,) dtype=int64>
7. melonora commented on Dec 11, 2025

   

   melonora

   on Dec 11, 2025

   Contributor

   More actions

   FYI, the PR got merged in time for the monthly dask release which would happen either end of this week or start of next week.
8. melonora commented on Dec 11, 2025

   

   melonora

   on Dec 11, 2025

   Contributor

   More actions

   also FYI #3612
9. AMBRA7592 commented on Jul 20, 2026

   

   AMBRA7592

   on Jul 20, 2026

   Contributor

   More actions

   I re-ran the original da.to_zarr(dask_array, zarr_array) reproducer against current stable releases and the corruption still occurs.

   Environment:

   • Python 3.14.2
   • zarr 3.2.1
   • dask 2026.7.1
   • numpy 2.5.1
   • macOS arm64

   After store.close(), zipfile.ZipFile(path).testzip() reports zarr.json as the first bad member, and reopening through zarr.open_group(...) raises BadZipFile: Bad magic number for file header. The result is the same with Dask's default scheduler and scheduler="synchronous".

   So the later Dask create_array work and the copy-store workaround do not appear to resolve this original write-to-an-existing-Zarr-array path. This issue remains reproducible on current releases.
10. li-em

    added 2 commits that reference this issue on Oct 5, 2026

    tests: read a zipped OME-Zarr through a ZipStore

    ...

    4c1b759

    tests: read a zipped OME-Zarr through a ZipStore

    ...

    dda7e26
11. d-v-b commented on Oct 27, 2026

    

    d-v-b

    on Oct 27, 2026

    Contributor

    More actions

    this should have been closed by #4450, we can re-open if it's still a problem
12. d-v-b

    closed this as completedon Oct 27, 2026