[SimonHeybrock]

Closes #836.

Adds ess.reduce.nexus.merge, a generic utility to merge the raw NeXus files of consecutive, compatible runs into one file that looks like a single long run. Scientists reduce and check each run first, then merge by hand. The regular single-run workflows then process the merged file.

plan = plan_merge(files, MergeRules.from_toml('loki.toml'))  # reads metadata and NXlogs only
for issue in plan.issues:  # info / warning / error
    print(issue)
plan.write(output)  # refuses if there are errors, or if an input file changed since planning

The rules file sets warn and/or error tolerances on the spread (max - min) of NXlog values over all runs, and lists fixed-size paths that may differ between runs (ignore):

ignore = ["/entry/experiment_description"]

[[rule]]
paths = ["/entry/instrument/*/target_value"]
error = 0

[[rule]]
paths = ["/entry/sample/temperature"]
warn = "0.1 K"
error = "0.5 K"

Choices

• Plain functions, no sciline. The plan holds only metadata and the issues found, never array data, so an essapps WorkflowSpec can output it and a local UI can call write. Writing on the backend waits until essapps supports writing files.
• Copy the first file, then append in place. Every dataset whose first dimension is unlimited (the NXlog and NXevent_data fields) is resized, and the other runs' data is appended into it. Each read and write holds at most 64 MiB of one dataset in memory, so memory use stays bounded for large event data; a small log is copied in one read of its actual size. Because nothing is recreated, the merged file keeps the first file's attributes, hard and soft links, chunking, compression, and fill values.
• NXlog rows repeated at the start of a run are dropped. The file writer starts each NXlog series with values from shortly before the run, which repeat the last rows of the previous run. Plain concatenation would turn a static motor log into a two-row, time-dependent one, and GenericNeXusWorkflow rejects time-dependent transformations. The merge drops leading rows that are identical to the end of the previous run; any other row that is not later than the previous run's last row is an error.
• Fixups after appending. event_index and cue_index get offsets. NXlog minimum_value/maximum_value/average_value are recomputed; average_value is a plain mean, which matches the file writer's values in the coda files. end_time takes the last run's value, and the root attributes file_name and file_time describe the merged file. Each run's per-run NXentry fields (start_time, end_time, entry_identifier, entry_identifier_uuid, scripts) and file name are kept in <entry>/merged_runs/run_<i>. The file is written under a temporary name and renamed when complete.
• Strict about everything else. All other datasets, attributes, dtypes, shapes, and hard-link layout must be identical in all runs, unless the path or a group containing it is in ignore. Runs are sorted by start_time, and runs that overlap in time are an error. So are pulses of a run that are not later than all pulses of the runs before (only the earliest and latest pulse are compared, since messages from Kafka arrive out of order and pulses within a run are not sorted), unsorted NXlog series, NaN in a checked log, NXlog fields or index fields (event_index, cue_index) the merge does not know how to append, and merged files as input. NXlogs that no rule matches are not checked; if their values change, the plan reports them as info.
• NXlog rows with time 0 are an error. 32 of 167 logs in the FREIA coda file end with such rows, and LoKI has them too, which look like padding by the file writer and break the summaries and spreads. Refusing them keeps us aware of whether this happens in real files; it may need reporting to ECDC.
• Tolerances carry units and are converted to the unit of the log, so a producer changing a log's unit (e.g., mm to m) cannot silently change a tolerance. A bare number is dimensionless, as in scipp. Only zero, the same in any unit, needs no unit, so error = 0 works for any log. A log whose unit differs between runs is an error.

Not included

• Rules files for instruments. The LoKI tolerances need input from the scientists.
• The essapps binding.

Test plan

• Unit tests with small HDF5 files written by h5py: event and log concatenation, index offsets, hard and soft links, compression and creation properties kept, all checks and their severities, TOML parsing.
• Manually merged coda_freia_999999_00026865.hdf with a copy made to look like the next run: all times shifted past the end of the original, and the rows from before its start replaced by the last rows of the original. The plan refuses the original files for their time-0 rows. With those rows trimmed, it reports no errors; 136 single-row logs stay single-row, and GenericNeXusWorkflow computes the same detector position from the merged file as from the original. Writing takes about 3 s.

🤖 Generated with Claude Code

[olihammond]

Note from discussion with Judith Houston: In general, samples should be re-run under indentical conditions, so we do not need to worry about merging over non-equivalent configurations.

However, circumstances are possible that we need to be robust to, such as: A user removes their sample from position AT, and then decides the statistics are insufficient from an initial reduction, so before cleaning it they decide to run it again, and insert it into position DT in the changer. In this case, maybe a warning at most, since in this case the sample changer position (and hence different motor positions) should not modify the sample scattering.

[SimonHeybrock]

Note from discussion with Judith Houston: In general, samples should be re-run under indentical conditions, so we do not need to worry about merging over non-equivalent configurations.

Programmers always worry about checking such things, because it is way easier to check early, instead of backtracing once you have have analyzed or published.

However, circumstances are possible that we need to be robust to, such as: A user removes their sample from position AT, and then decides the statistics are insufficient from an initial reduction, so before cleaning it they decide to run it again, and insert it into position DT in the changer. In this case, maybe a warning at most, since in this case the sample changer position (and hence different motor positions) should not modify the sample scattering.

What we have in this PR allows for configuring this using a config file, with distinct warning and error thresholds, as well as which dataset to compare or not.

[nvaytet]

How do you handle histogram monitor data (NXdata)? I guess we need to concatenate along the time dimension? The monitor data is important for normalization.

Is NeXus happy having a non-regular time coordinate?

[SimonHeybrock]

This is not what this is about: _INDEX_FIELDS lists fields that index into datasets that get concatenated, and thus need to be rewritten (see the cumsum code lower down).

[nvaytet]

Ok, maybe I attached the comment in the wrong place, but my questions still stands.

How do you handle monitor data? I could not find any instances of either NXmonitor or NXdata in the merging code.

[nvaytet]

Did you test with real e.g. loki files?

[SimonHeybrock]

You mean in general or this particular test case?

[nvaytet]

I meant in general. I wrote here instead of a general comment because it is then easier to have a conversation without having to copy/paste/quote in general comments

[olihammond]

Adds ess.reduce.nexus.merge, a generic utility to merge the raw NeXus files of consecutive, compatible runs into one file that looks like a single long run. Scientists reduce and check each run first, then merge by hand. The regular single-run workflows then process the merged file.

You may already be aware of this requirement, but because the language here specifically pertains to reduction– I wanted to be 100% clear here that this does not solely apply to sample files. It is also possible that users will want to merge multiple background measurements (so empty cells, direct/blocked beam and flood field data) which are themselves used for reduction of single- or multiple-run sample files.