LocalStore listing lists .partial temp files

[omar-merhebi]

Zarr version

3.2.2.dev112+g50b7e016

Numcodecs version

0.16.5

Python Version

3.12

Operating System

Linux (RHEL 9)

Installation

editable install from source (uv pip install -e .) on a main checkout inside a venv

Description

When LocalStore writes zarr arrays and metadata, it initially writes to a temporary .partial file (full: <stem>.<uuid>.partial) and then it renames it into place when the write is complete. LocalStore.list(), LocalStore.list_prefix(), and LocalStore.list_dir() include these temporary files. So, when a process is writing to a store while another process reads it, those .partial files get listed as if they were real zarr store keys and they also cause a ZarrUserWarning when the group is iterated via Group.members(). When writing arrays concurrently, this clogs up the logs/output with many warnings like this:

ZarrUserWarning: Object at zarr.16442ffbc9dc4bffaae221fef349d64a.partial is not recognized as a component of a Zarr hierarchy.

I've been suppressing these manually. Working on a fix to filter the .partial temps in the listing methods. Happy to PR or take another direction if you prefer!

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr

import asyncio
import tempfile
import warnings
from pathlib import Path

from zarr.storage import LocalStore
from zarr.storage._local import (
    _atomic_write,
)


async def main() -> None:
    root = Path(tempfile.mkdtemp())
    store = LocalStore(root)
    group = zarr.open_group(store=store, mode="w")
    group.create_array("a", shape=(1,), dtype="int32")

    # Begin writing a zarr array and then list the store
    # before the .partial file is renamed.
    with _atomic_write(root / "zarr.json", "wb") as f:
        f.write(b'{"zarr_format": 3, "node_type": "group"}')

        print("list() ->", sorted([k async for k in store.list()]))

        # This should result in the ZarrUserWarning
        with warnings.catch_warnings(record=True) as caught:
            warnings.simplefilter("always")
            list(group.members())
        for w in caught:
            print(f"{w.category.__name__}: {w.message}")


asyncio.run(main())

zarr.print_debug_info()

Additional output

No response

[d-v-b]

thanks for this report. I think the fix is to create the temporary files in a completely different directory, e.g. a system-managed tmp directory. we don't want the store itself to ignore files based on the filename.

[omar-merhebi]

Thanks for the reply. Agreed that the store shouldn't filter out based on file name. The only catch with writing to /tmp (or other equivalent) and then replacing is that it only works if the temp dir and the store are on the same filesystem. I use zarr-python on a linux HPC, so /tmp is on a different filesystem than where the stores end up. On those types of systems, tmp_path.replace() and os.link() in _safe_move() will not work and will end up with OSError ... Invalid cross-device link. shutil.move() would fall back to a copy then delete but that's no longer atomic and ends up writing data twice. This is also the case if you want to write the stores to a different mount on a local machine.

Two other options to consider would be to either 1) put a per-store temp dir alongside the store root (e.g., store at /path/zarr/store and temp files at /path/zarr/store.partial) or 2) write .partial files to a temporary subdir of the root of the store that still gets filtered out when listing. This still would ignore a subdirectory, but it would be a known explicit path that always gets ignored rather than anything with .partial in the name. I'll defer to you but advise 2, since writing to a parent dir of root means it would put files in a directory that the user wasn't expecting it to write to (and, in container setups for example, if a user mounts /data and writes the store directly there, it would fail with the same OSError as above).