[SimonHeybrock]

Some raw files have defects that break the standard workflows, e.g., an empty transformation NXlog. #785 added failsafe loaders for inspecting such files, but their output cannot be used by workflows. This PR adds a way to fix known defects while loading, without rewriting the file.

What it does

A patch is a list of operations, stored in a TOML file:

[[op]]
kind = "move"
source = "/entry/instrument/detector_carriage/value"
target = "/entry/instrument/detector_carriage/transformations/motion"

[[op]]
kind = "set_attrs"
path = "/entry/instrument/detector_carriage/transformations/motion"
attrs = { depends_on = "...", transformation_type = "translation", vector = [0.0, 0.0, 1.0] }

Operations are set_attrs, delete_attrs, remove, move, set_dataset, applied in order. Patching is opt-in per run:

from ess.reduce.nexus import patch

wf[NeXusFilePatch[SampleRun]] = patch.load_patch(path) + [patch.Remove(path='/entry/broken')]

A patch is a plain list, so a tracked patch file can be amended in memory with extra fixes.

How it works

patch.PatchedH5File wraps an h5py.File and implements the small part of the h5py interface that ScippNexus uses (ScippNexus is duck-typed against h5py). ScippNexus reads through this view, so:

• the file on disk is not modified and nothing is copied; datasets are read from the original file,
• name, parent, and file report paths in the patched tree, so depends_on chains resolve in the patched file,
• new datasets and attributes live in a small in-memory HDF5 file, so they behave like regular h5py objects.

Operations fail if the file does not match the patch (e.g., removing a path that does not exist), so a patch applied to the wrong file fails when the file is opened instead of silently doing nothing.

The patch travels through the workflow as PatchedFilePath in place of the file path, so PreopenNeXusFile=False keeps re-opening the file per access as before.

No ScippNexus changes are needed. Two places in ScippNexus are less duck-typed than its H5Group protocol suggests, and both are handled here:

• snx.File only wraps objects that pass isinstance(..., h5py.File | h5py.Group). PatchedNeXusFile therefore subclasses snx.Group and adds the context-manager and close() behavior of snx.File.
• snx.Attrs calls attrs.get_id(name).get_type().get_cset() to find the encoding of string attributes. Patched attributes are therefore stored in an in-memory HDF5 file, so they have real h5py IDs.

A possible follow-up in ScippNexus: let snx.File accept any h5py-like group, optionally taking ownership, which would make PatchedNeXusFile unnecessary. The protocols in scippnexus/typing.py could also be updated: they omit items, get, and attrs.get_id, which ScippNexus uses, but list create_dataset, create_group, and visititems, which reading does not need.

Discarded alternatives

• Loading the file into memory (BytesIO or the HDF5 core driver) and patching with h5py: reads the whole file into memory, not viable for large event files.
• An overlay HDF5 file with external links into the original: objects reached through an ExternalLink report .file/.parent/.name of the original file, so absolute depends_on paths resolve in the unpatched tree.
• Patching ScippNexus Group objects after construction: requires touching cached private state.

Example patch

ess/loki/nexus_patches/todo_motion_sources.toml fixes the LoKI file used as loki_broken_file in the tests: motion NXlogs of the beamstop axes and the detector carriage have a TODO- source and are empty, while the motor readback was written to <component>/value. With the patch, GenericNeXusWorkflow computes detector and monitor positions for that file.

Not in this PR

• Lookup of patches by instrument and start time (per-patch date ranges), and applying them automatically.
• Recording applied patches in output metadata. Currently only logged.
• Wildcards in paths. Repetitive patches can be generated by a script or built in Python.
• Code that opens files with h5py.File directly (some reflectometry, NMX, and spectroscopy loaders) does not see patches.

🤖 Generated with Claude Code