Enable dandi validate to operate on Datalad datasets through streaming

[CodyCBakerPhD]

Towards the goal of having a source of internally trusted Truth regarding validation records on a Dandiset, it is necessary to be able to run the validation tool through streamed file contents.

This requires a bit of reworking here on the CLI.

Current strategy is, for simplicity, fsspec on a typical DataLad dataset with special remotes on S3.

Why not datalad-fuse (for now): it needs FUSE and mount privileges, which containers, CI runners, and most HPC nodes don't provide - and these are exactly the places being currently considered as the source of compute for this effort. Reading the git-annex key and URL log with plain git and streaming with fsspec gives the same on-demand, range-request access from any environment that has git and HTTPS; a FUSE-mount workflow can still be layered on later if a "mount everything" use case appears.

AI Summary

Summary

dandi validate can now validate a DataLad / git-annex Dandiset (such as the clones at https://github.com/dandisets) whose content has not been fetched. With the new --missing-file-content=stream policy, the content of each annexed file is streamed on demand (via fsspec) from the URLs registered for it in git-annex, so pynwb and nwbinspector run on the file without downloading the (possibly terabytes of) data. Only git is needed to read the git-annex metadata (neither git-annex nor DataLad has to be installed), plus fsspec[http] (pip install "dandi[extras]") for the streaming.

The motivation is generating validation records for every Dandiset:

$ git clone https://github.com/dandisets/000029
$ dandi validate --missing-file-content=stream --min-severity=INFO -f json_lines -o 000029.jsonl 000029

The docs (docs/source/cmdline/validate.rst) gain a "Validating DataLad Dandisets" section, including how to loop over the subdatasets of the dandisets superdataset, and mention the datalad-fuse alternative (which needs FUSE and so does not work in many containers/CI environments; this PR works anywhere git and HTTPS are available).

What changed

• dandi/support/annex.py (new): parses git-annex keys from the broken symlinks (SHA256E-s<size>--<hash>.nwb), locates a key's URL log in the git-annex branch (local or remote-tracking, e.g. origin/git-annex after a plain git clone) via git cat-file, and provides AnnexReadableFile, a Readable that streams from the first URL that can be opened (direct S3 URLs are preferred over api.dandiarchive.org/.../download/ URLs, which redirect on every request). Uses fsspec's blockcache so h5py's random access only fetches the blocks it touches.
• LocalFileAsset.content_source: an optional Readable an asset reads its content from instead of filepath. pynwb_utils.validate() takes a readable= (results not cached in that case), and NWBAsset runs nwbinspector on the streamed file through inspect_nwbfile_object(), replicating inspect_nwbfile()'s error reporting so the result IDs are identical to local validation.
• validate() core: under stream, each streamable file yields an INFO DANDI.FILE_CONTENT_STREAMED result naming the URL; a broken symlink that cannot be streamed (not annexed / no URL registered) yields DANDI.FILE_CONTENT_MISSING. BIDS content-dependent errors are suppressed for annexed files under stream as they are under only-non-data. A clear error is raised up front if fsspec/aiohttp are missing.
• CLI: stream added to --missing-file-content; dandi validate now also accepts a broken symlink directly as a path argument (click.Path(exists=True) used to reject it with "does not exist").
• Tests (all marked ai_generated): key/URL-log parsing, AnnexRepo on a repo whose git-annex branch is created with git plumbing (both local and origin/ refs), AnnexReadableFile over file:// URLs and over a local range-capable HTTP server (h5py opens the file), and end-to-end validate()/CLI runs asserting that streamed results equal those of validating the same NWB file locally.

Verification

• Real data: dandi validate --missing-file-content=stream on a git clone of dandisets/000029 (6 NWB files, 18 KB–18 MB) streams everything from S3 in ~10 s and produces 77 records (the same pynwb/nwbinspector findings as for local files); the same for 000029 checked out as a git submodule (.git file) and for a single 2.8 MB file of 000035. A 61 GB file of 000003 opens with one HEAD and a single 4 MiB range request.

Limitations / notes

• Zarr assets are not streamed (in the dandisets repos they are separate uninstalled subdatasets, i.e. empty directories that are not validated at all).
  • TODO [follow-up 1]: this will be implemented in follow-ups that facilitate NWB-Zarr integration across the ecosystem.
  • TODO [follow-up 2]: OME-Zarr validation will also be added in a follow-up.
• The BIDS validator cannot stream (though it our use cases, ought not need to), so BIDS errors needing file content are suppressed for annexed files.
  • TODO [follow-up 3]: assess the need for BIDS content validation on existing Dandisets to determine if this is needed. NWB-BIDS integration will NOT require this and always runs with nifti headers disabled.
• Some nwbinspector checks read data arrays (e.g. timestamps), so the amount streamed per file depends on its content.
• The clone must include the git-annex branch (no --single-branch).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HFhMPs6vF5zyeHDvPTvRcA

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 86.43123% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.73%. Comparing base (a0b7aff) to head (22e296b).

Files with missing lines	Patch %	Lines
dandi/support/annex.py	64.74%	55 Missing ⚠️
dandi/tests/fixtures.py	81.81%	6 Missing ⚠️
dandi/files/bases.py	82.14%	5 Missing ⚠️
dandi/pynwb_utils.py	70.00%	3 Missing ⚠️
dandi/validate/_types.py	0.00%	2 Missing ⚠️
dandi/support/tests/test_annex.py	99.47%	1 Missing ⚠️
dandi/validate/_core.py	97.22%	1 Missing ⚠️

Additional details and impacted files

@@                           Coverage Diff                           @@
##           claude/memoize-readable-fingerprint    #1933      +/-   ##
=======================================================================
+ Coverage                                78.45%   78.73%   +0.27%     
=======================================================================
  Files                                       92       94       +2     
  Lines                                    14199    14708     +509     
=======================================================================
+ Hits                                     11140    11580     +440     
- Misses                                    3059     3128      +69     

Flag	Coverage Δ
unittests	78.73% <86.43%> (+0.27%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.