diff --git a/design/README.md b/design/README.md index 90565b927..e52897a7a 100644 --- a/design/README.md +++ b/design/README.md @@ -57,3 +57,19 @@ and section 11 for integration and manual acceptance. The implementation and user documentation are complete on the unreleased phy 2.2 branch. Final large-dataset smoke testing and save/reopen validation remain. + +## Long-recording curation (proposed) + +The [long-recording curation Phy-side design contract](large-recording-curation.md) +describes bounded local datasets for ~1000-hour sorting datasets, the three spike-index +spaces, optional upstream waveform evidence, full-duration metrics, replay safety, +and the implementation stages. The accompanying synthetic 1000-hour +fixture and reference tests are in `phy/apps/tests/large_recording_fixtures.py` +and `phy/apps/tests/test_large_recording_contract.py`. + +This is a proposal and a reference test contract, **not yet** a `prepare-large` +implementation or an upstream export-format agreement. + +The [integration audit and plan](large-recording-integration-plan.md) refines +the contract with correctness findings, module boundaries, ordered commits in one PR and +acceptance gates. Follow its gates before enabling editable bounded sessions. diff --git a/design/aind-export-handoff.md b/design/aind-export-handoff.md new file mode 100644 index 000000000..924c823f4 --- /dev/null +++ b/design/aind-export-handoff.md @@ -0,0 +1,90 @@ +# AIND export handoff for elapsed-time coordinates + +This note describes the producer work needed before an AIND export can use +Phy's explicit elapsed-tick contract. It is a handoff, not evidence that an +exporter change has been implemented or that a real export has passed +acceptance. The exporter source repository was not available in this worktree; +the steps below are proposed commit-sized changes, with no repository or branch +assumed. + +## Proposed exporter changes + +1. **Establish authoritative acquisition truth.** Obtain acquisition logs or + equivalent source records for clock origin, piece boundaries, dropped or + invalid spans, filler, and the physical recording extent. Resolve the + reported interruptions and the 35 sorting spikes reported inside filler. + Do not infer validity solely from spike times or endpoint interpolation. +2. **Emit a canonical event time axis.** For each sorting spike, emit a + nonnegative integer elapsed tick from the source time origin. Declare + `time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': ...}`. Set + `num_samples` to the exclusive bound that includes the last representable + event tick; retain a physical frame count separately (for example as + `recording.num_frames`). Do not call a tick bound a frame count. +3. **Emit authoritative exposure.** Emit ordered, disjoint integer + `valid_intervals` in the same elapsed-tick coordinates, excluding missing + acquisition time and unusable filler. Ensure every emitted sorting spike + belongs to a valid interval. If any sorting rows fall outside exposure, + resolve them using acquisition truth upstream; never silently drop, move, or + relabel rows. +4. **Emit aligned waveform and window evidence.** Store + `_phy_spikes_subset.sample_index.npy` in canonical elapsed ticks, with each + value equal to `spike_times[subset_spikes[w]]`. Emit producer windows in + elapsed ticks, with boundaries that do not bridge unsupported intervals. + If the windows overlap, retain their original boundaries and ensure their + union accurately represents potential evidence. Add + `recording.sample_index_coordinates: "elapsed_ticks"` and + `recording.window_bounds_coordinates: "elapsed_ticks"` to the manifest when + those sidecars are present. Add `recording.time_base` and authoritative + `valid_intervals`; if `params.py` also declares them, keep the declarations + identical. +5. **Keep raw voltage semantics explicit.** Do not put elapsed ticks into raw + frame-index fields or claim ordinary raw-trace loading works. An export with + a real raw-data path is outside this contract until an explicit validated + event-to-frame mapping is supported. Stored waveform data can be exported + without a raw path. + +## Report evidence separately from verification + +The supplied export report describes 420 frame/clock endpoints for 210 pieces, +with strictly increasing frame endpoints, nondecreasing clock endpoints, and +15 equal adjacent clock pairs. It also reports backward timestamp steps that +are not represented by those endpoints. These are reported properties of the +export metadata, not independent verification of the corresponding acquisition +events. + +The reported `total_dropped_within_pieces_s` was not independently confirmed +against acquisition truth. Endpoint interpolation does not establish the exact +valid-time mask, a sample-accurate raw clock, or reversibility: equal clock +endpoints make the endpoint map non-invertible. The reported 35 spikes inside +filler likewise require independent confirmation and resolution. A 30 kHz +label, monotonic spike times, or a plausible endpoint-derived extent cannot +replace those checks. + +Changing Phy's sampling seed (including setting it to `42`) does not make an +export interoperable. The seed only selects among windows already defined in +Phy's coordinate system; it cannot recover producer acquisition boundaries, +map clock coordinates to frames, or determine which spans contain valid data. +Use compatible producer windows and an explicit coordinate and exposure +contract. + +## Acceptance evidence to return with an export + +Provide the exporter commit or reviewable diff, a short provenance note naming +the authoritative source for boundaries and filler, and machine-checkable +checks showing: + +- every exported spike's elapsed tick and valid-interval membership; +- the exclusive event-axis bound and separately named physical frame count; +- waveform-row alignment of sample indices to `spike_times[subset_spikes]`; +- producer windows' coordinate kind, valid exposure, and source waveform + coverage; +- exact source, retained, and omitted waveform counts under Phy's unchanged + 512 MiB package and 2,000,000-spike budgets. + +If the exporter removes spike rows, it must apply the identical row selection to +every spike-aligned array and rewrite any row-reference arrays such as +`subset_spikes` consistently. Silent removal leaves waveform and metadata rows +misaligned and is not acceptable. Phy can validate the explicit contract and +the exported arrays; only the exporter and acquisition evidence can establish +that the chosen exposure is scientifically authoritative. A local older +recording is not acceptance evidence for a new export. diff --git a/design/aind-native-export-acceptance.md b/design/aind-native-export-acceptance.md new file mode 100644 index 000000000..b7082c825 --- /dev/null +++ b/design/aind-native-export-acceptance.md @@ -0,0 +1,63 @@ +# Native AIND export acceptance + +This checklist accepts an export for Phy software integration only. Passing +checks do not prove scientific validity. The exporter owner must provide +authoritative acquisition evidence for the clock origin, physical extent, +valid exposure, dropped spans, and filler policy. A Phy run cannot supply that +evidence by inferring or converting timestamps or exposure. + +## Before validation + +- [ ] Record the exact Phy revision and native exporter revision in the report. +- [ ] Attach the reviewable exporter change and provenance for acquisition + boundaries and filler classification. +- [ ] Confirm the exporter declares one explicit time base and emits + `num_samples`, `valid_intervals`, waveform sample indices, and producer + windows in that same coordinate system. +- [ ] Confirm every sorting spike is inside authoritative valid exposure and + every waveform sample index equals `spike_times[subset_spikes]`. +- [ ] Resolve spikes inside filler from acquisition evidence upstream. Any + intentional row removal must use the same row mask for every spike-aligned + array and update row references consistently. +- [ ] Keep elapsed ticks separate from physical frame counts. Do not route + elapsed ticks to raw frame-index fields or claim raw-trace loading is + supported by this contract. + +## Phy engineering validation + +1. Run the native export through `tools/validate_aind_export.py` with the two + revisions above, without an output directory. Retain its JSON report, + including input file hashes, time contract, exposure, array counts, and + timings. Resolve every source validation failure at the exporter. +2. Run optional preparation first with Phy's default budgets: 2,000,000 spikes + and 512 MiB. Use a new destination outside the source directory. Record + retained spike and waveform counts, omitted evidence, output size, and + timings. Confirm the original source file hashes are unchanged. +3. If the complete producer evidence exceeds either default cap, run larger + spike and byte caps as distinct, explicitly labeled runs, each to a fresh + destination. Report each cap and its result independently; never treat a + larger-cap success as a pass under the defaults. +4. Open the default bounded output in the Phy GUI. Exercise navigation at the + beginning, middle, and end of retained windows, select units with and + without retained stored waveforms, and verify displayed waveform/channel + mapping and feature behavior. +5. Exercise the three-round SLAY carryforward workflow on the bounded output: + save each round, carry assignments and labels forward, reopen the result, + and compare final assignments and metadata with an independent expected + result. Preserve the round-by-round artifacts and logs. +6. Recheck the native source hashes after GUI and carryforward work. All curation + products must be separate from the source export. + +## Decision record + +- [ ] Phy contract validation passes with no timestamp or exposure inference. +- [ ] Default-budget preparation and output counts are recorded. +- [ ] Larger-cap runs, if needed, are separately reported. +- [ ] GUI and three-round SLAY carryforward evidence is attached. +- [ ] Acquisition authority has approved the clock, exposure, and filler + policy independently of the software checks. + +Record the outcome as **software integration accepted** only when the Phy +checks pass. Record **acquisition truth accepted** separately and only when the +authoritative producer evidence has been reviewed. The first outcome alone is +not scientific acceptance of the export. diff --git a/design/aind-real-data-validation.md b/design/aind-real-data-validation.md new file mode 100644 index 000000000..2edf9d868 --- /dev/null +++ b/design/aind-real-data-validation.md @@ -0,0 +1,137 @@ +# Real AIND software diagnostic + +## Conclusion + +The downloaded export does **not** work unchanged with PR #1428: its time +coordinates and exposure declarations still need upstream correction. A +separately labeled diagnostic derivative exercised the complete Phy workflow +over 469,725,534 rows and exposed one additional Phy defect. That defect is +fixed, with portable regression coverage. Preparation, GUI curation, +save/reopen, full replay, independent verification, and repreparation now pass +on the derivative. + +This is software integration evidence, not scientific acceptance of the +original export. The derivative's exposure is inferred from the JSON endpoint +map, and 33 filler-region spikes were explicitly quarantined. Those choices +require acquisition evidence and producer review before use for curation. + +## Source and unchanged-export checks + +The supplied download script selected computation +`b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`, rather than the earlier reported +`85640506-601d-4826-aa61-d3b822e53c84`. Its recording is +`experiment1_AmplifierData_recording1_group0` from session +`854393_2026-06-09_19-34-26`. + +All 67 downloaded files matched the server's listed sizes, totaling +15,381,599,263 bytes. NumPy headers and payload sizes were checked. The export +contains 469,725,567 spikes, 705 declared units, 140,240 stored waveforms and +177 producer windows. Spike times are monotonic across the complete array. + +The unchanged source fails `prepare-large` with the intended early +`timestamps.json requires an explicit time_base` diagnostic. It declares +10,800,000,000 pipeline frames, while its last spike is elapsed tick +12,188,787,421. Exactly 54,808,765 spikes are beyond the declared extent. +`params.py` and the waveform manifest lack an explicit elapsed-time contract +and valid acquisition intervals. + +All 140,240 waveform centers match the JSON's frame-to-clock conversion within +0.5 tick, while their stored sample indices remain pipeline frames. This +reproduces producer interpolation; it does not establish sample-accurate +acquisition timestamps or exposure. Under that same map, filler regions contain +23, 4 and 6 spikes. These 33 rows differ from the 35 reported for the other +computation. Neither explicit large gap contains spikes. + +## Diagnostic derivative and limits + +The original export was retained unchanged. A separate source was constructed +solely to isolate software behavior: + +- Declare the elapsed-tick contract at 30 kHz, an exclusive extent of + 12,188,787,510 ticks, and canonical sidecar coordinates. +- Copy exact `spike_times[waveform_ids]` into the waveform sample-index sidecar. + Convert each producer `[a, b)` window to + `[round(map(a)), round(map(b - 1)) + 1)` on the elapsed axis. +- Infer diagnostic exposure by excluding only the mapped explicit large gaps + and filler regions. This does not locate unknown dropped frames within + pieces or establish acquisition truth. +- Confirm that coordinate normalization alone still rejects the 33 filler + spikes through the strict exposure-membership check. +- Quarantine those source rows explicitly, apply the identical row selection + to every spike-aligned array, and remap waveform row references. No stored + waveform references a quarantined row. Keep a provenance file listing every + excluded source row and spike tick. + +No conversion or filtering behavior was added to the production importer. +The diagnostic source, outputs, SHA-256 inventory, assumptions and one-off +reproduction scripts are retained beneath the downloaded computation's +`diagnostic-validation/` directory on the validation workstation. + +## Observed Phy defect and fix + +With the actual bounded data, merge, undo/redo, amplitude split and undo/redo +left NumPy integer child IDs as keys in split-event metadata. Projected session +metadata had been normalized, but the retained event had not. Saving failed +while serializing `journal.json` with `TypeError: keys must be str, int, float, +bool or None, not numpy.int64`. + +Both split-event construction paths now normalize metadata before retaining +the event. This preserves IDs and the remainder/selected child order. Two +session API regressions reproduced the failure before the fix; a GUI regression +also exercises merge undo/redo, split, save and reopen. The focused +curation/lifecycle run passed 43 tests. Independent Sol-high review confirmed +the root cause and semantics of the fix. + +## Full workflow results + +Default preparation retained 1,486,361 spikes in 36 windows and 27,865 of +140,240 waveforms. Every unit has spike evidence; 702 of 705 have stored +waveform evidence. The package occupies 532,625,401 bytes, within the +536,870,912-byte cap. The complete producer-window plan is estimated at +2,554,394,546 bytes and 7,298,318 spikes, exceeding both default budgets. + +All default bounded views opened, including waveform-derived features without +exported PCs. Selection included units 574, 578 and 579, which have no retained +stored waveforms. Navigation covered the beginning, middle and final retained +windows. Artificial merge and amplitude-split actions passed undo/redo, an +independent bounded assignment comparison, save and reopen, without logged +application errors. These actions are test operations, not biological curation. + +Full replay wrote a separate output. An independent streamed oracle compared +**all 469,725,534 assignments**, using explicit merge masks and amplitude +comparisons rather than the production replay functions. Edge searches and +weighted/unweighted histograms independently checked the original per-bin +counts, finite amplitude counts and amplitude sums. Exposure arithmetic was +checked against the declared diagnostic intervals, without claiming their +scientific validity. + +Every bounded spike row was checked against its source row. All 27,865 bounded +waveform rows, channel lists and center indices matched their mapped source +rows exactly. Replay preserved immutable array bytes and timestamp provenance. +Repreparation's complete unit counts matched the replay oracle. SHA-256 checks +confirmed that every file in the original download remained unchanged. + +## Measurements and acceptance limits + +Single workstation runs used Linux, Python 3.12.3, phylib 2.7.1, 100,000-row +batches, Xvfb/software OpenGL and Qt's offscreen platform. The workstation has +125 GiB RAM; source files were warm in the filesystem cache. Some later phases +ran concurrently with tests or another verification phase. These are observed +integration costs, not controlled throughput benchmarks or laptop predictions. + +| Phase | Wall time | Peak process RSS | +| --- | ---: | ---: | +| Preparation | 50.1 s | 14.2 GiB | +| GUI actions and reopen | 9.1 s | 0.68 GiB | +| Full replay | 37.0 s | 25.0 GiB | +| Repreparation | 51.0 s | 14.1 GiB | + +Offline RSS includes resident memory-mapped source/output pages. It is not +limited by the bounded package's byte budget. These measurements do not prove a +small process-RSS bound on a much longer source. + +The original export remains blocked until its producer supplies consistent +time coordinates, authoritative exposure and a justified filler-spike policy. +See the [exporter handoff](aind-export-handoff.md). No acquisition logs or +verified exporter source repository were available, and this diagnostic does +not resolve timestamp precision within pieces or chronic unit identity. diff --git a/design/curation-main-gui-result.json b/design/curation-main-gui-result.json new file mode 100644 index 000000000..74117f2e2 --- /dev/null +++ b/design/curation-main-gui-result.json @@ -0,0 +1,76 @@ +{ + "ok": true, + "method": "exact all-row two-way ID mapping", + "revision": "c79666c672ab045e356b31f0764efc2cd0e4bdb4", + "rows": 469725567, + "assignment_dtypes": [ + "int32", + "int64" + ], + "partition_mismatch_rows": 0, + "examples": [], + "input_hashes": { + "left": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "3cbef5e08dc915ddd57aac9c214370a34282d264695f53959c184d6f8f2b7148" + }, + "right": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "b0223191c2cd29d2167b975175d22428bfe444175e19dc68e67aa9a82aea5168" + } + }, + "unmapped_label_ids": { + "left": { + "group": [], + "notes": [] + }, + "right": { + "group": [], + "notes": [] + } + }, + "manual_label_mismatches": [], + "compared_fields": [ + "group", + "notes" + ], + "scope": "complete curation assignments and requested manual labels; baseline GUI rendering errors retained", + "main_revision": "ad64e9138a6556f4785a359c3ee1258864d60f1d", + "bounded_runtime_revision": "06637b24aeca9b0984d7928d459187df46406444", + "reviewed_pr_head": "c79666c672ab045e356b31f0764efc2cd0e4bdb4", + "mapped_units": 703, + "gui_rendering_error_free": false, + "baseline_feature_errors": { + "acceptance": 7, + "reopen": 5 + }, + "actual_restored_view_count": { + "AmplitudeView": 1, + "ClusterScatterView": 1, + "ClusterView": 1, + "CorrelogramView": 1, + "FeatureView": 1, + "FiringRateView": 1, + "ISIView": 1, + "MergePropositionsView": 1, + "ProbeView": 1, + "SimilarityView": 1, + "WaveformView": 1 + }, + "raw_source_parent_scan": { + "units": [ + 40, + 44, + 48 + ], + "rows": 1156852, + "nonfinite": 0, + "equal_to_threshold": 0, + "independently_verified": true + }, + "source_integrity": { + "ok": true, + "raw_files_verified": 67, + "immutable_main_working_files": 19 + } +} diff --git a/design/curation-main-gui-validation.md b/design/curation-main-gui-validation.md new file mode 100644 index 000000000..869a7c3ab --- /dev/null +++ b/design/curation-main-gui-validation.md @@ -0,0 +1,139 @@ +# Main GUI versus bounded PR curation: real-data comparison + +On 26 October 2026, the actual ordinary GUI on current `master` +(`ad64e9138a6556f4785a359c3ee1258864d60f1d`, verified against GitHub) completed +the same action ledger as the bounded PR GUI. Its complete saved output matches +bounded full-source replay on **all 469,725,567 spikes** and requested manual +labels. This extends the earlier ordinary-engine-only reference with an actual +main-branch TemplateModel, controller, graphical split and GUI save/reopen. + +The result is **curation completed with baseline Feature View errors**. It is +not an error-free all-view GUI acceptance claim. The renderer failure and other +limits are retained below. + +## Inputs and isolation + +- Source: Carl's earlier converted engineering export, computation + `b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`, 705 original units, 112.859 elapsed + clock hours. This is not Carl's newer 500-hour dataset, and its inferred + exposure remains unverified. +- Main code: a clean detached worktree at `ad64e913`; the script asserts the + imported `phy` module comes from that worktree. No main code was changed. +- PR runtime: `06637b24`, unchanged by the subsequent tool/documentation commits + through reviewed head `c79666c6`. The previously completed native bounded GUI + run and full replay were reused, with their pinned inputs and ledger. +- Main working data: a separate 15,381,601,942-byte deep copy with symlinks + dereferenced, never hard-linked. Its `params.py` has one final `dat_path = None` + override so ordinary phylib treats the exporter's string sentinel as absent raw + data. No event times, source rows, amplitudes or exposure declarations were + converted for this run. Both parameter-file hashes are recorded locally. +- Environment: Python 3.12.3, phylib 2.7.1, native X11 Qt under Xvfb and software + OpenGL. Each full ordinary GUI process ran under a 96 GiB virtual-memory limit. + +The [shared ledger](curation-mode-acceptance-ledger.json) specifies merge +40/44/48, undo/redo, source-amplitude split, undo/redo, merge the low child with +24, explicit GOOD/MUA and notes, a label-history branch, save/reopen and a no-edit +save. The ordinary GUI used **template** amplitudes, matching `amplitudes.npy`. + +## Native graphical split and save/reopen + +The script ran the real ordinary controller and Supervisor. A real Qt +Alt-right-drag placed the preview; a real `K` key event triggered the standard +split action and its all-spike provider. The resulting two children were checked +against the full parent membership and source predicate, rather than inferred +from child counts alone. + +The viewport was panned so the shared threshold landed on an integer pixel. +Requested and actual thresholds were exactly **-424.0037077454673**. An +independent chunked scan of raw source arrays confirmed all **1,156,852 parent +spikes** have finite amplitudes, with none equal to the threshold. Thus ordinary +display sanitization of nonfinite values does not affect this comparison. + +After the ledger's final save, the process closed the GUI/controller and exited +zero. A fresh process loaded the saved full sorting through the main GUI, +verified the final groups/notes for all three reviewed units, selected them, +saved without edits and exited zero. Using a separate process establishes real +reload and avoids retaining an earlier model through global callbacks. + +Observed single warm-cache runs: + +| Run | Wall time in harness | Peak RSS | +| --- | --- | --- | +| Main GUI ledger and saves | 116.1 s | 33.5 GiB | +| Fresh-process GUI reopen and no-edit save | 44.5 s | 26.5 GiB | + +These are integration observations, not controlled performance benchmarks. + +The caller requested Amplitude and Waveform views, but ordinary saved/default +state restored **11 view types**, including Feature View, Firing Rate, ISI, +Correlogram, Probe, Cluster Scatter and the tables. The original run JSON's +`views` field lists requested views only. The committed summary records the +actual restored inventory and the rendering errors instead of treating that +field as the observed layout. + +## Exact complete-output comparison and integrity + +`tools/compare_curation_modes.py compare` compared the saved main full sorting +after reopen against the existing complete bounded replay. It passed: + +- **469,725,567** rows, with exact row-aligned integer spike times. +- **Zero partition mismatches**, using a two-way raw unit-ID mapping over every + row, spanning 703 final units. +- **Zero group/notes differences** and no unmapped manual-label IDs. + +Ordinary main saved int32 assignments; bounded replay retained int64. Assignment +dtype and numeric IDs need not agree for partition equality. The comparison +checks exact integer memberships, with input hashes recorded in the +[report summary](curation-main-gui-result.json). + +An independent verifier checked all **67 original download files** against the +archived SHA-256 inventory. All **19 immutable working-copy files** (every NPY +except assignments, plus timestamps and suggestions) match the converted source. +The source parameters still match the original hash recorded during preparation. + +An optional working-copy hash helper was started concurrently with the first +GUI run and lost a race with the intentional GUI save of `spike_clusters.npy`. +Its failed log is retained; it is not claimed as a successful input preflight. +The source copy operation, unchanged original hashes, final immutable-file check +and complete-output comparison provide the stated evidence. When reproducing, +run the copy hash helper before launching the GUI. + +## Baseline Feature View failure + +Main emitted **7 Feature View errors during curation and 5 during reopen**: +background feature sampling sometimes requests no IDs in the stored waveform +reservoir. Unmodified phylib 2.7.1 `TemplateModel.get_features()` then invokes PCA +on zero waveforms, causing `ZeroDivisionError` in `_compute_pcs()`. + +A minimal independent reproduction calls the installed `get_features()` with +requested row 2 and stored row 1; their empty intersection raises the same +exception. This is a baseline feature-rendering failure, not an assignment +mutation. No exception was discarded when classifying this acceptance result. + +Bounded `_get_feature_view_spike_ids()` intersects selected and background IDs +with stored waveform rows before projection when dense features are unavailable. +Its inherited feature getter returns an empty bunch for an empty request, so +this particular phylib empty-PCA path differs from ordinary main. No PR #1428 +change is needed to establish the curation equivalence above. An upstream phylib +guard and focused regression would be a separate follow-up; changing the +ordinary baseline during this comparison would hide the observed behavior. + +## Artifacts and remaining acceptance + +Actual main captures are in `docs/images/amplitude-main-gui/`: the native +threshold preview and the reopened full sorting. Empty feature panels in these +captures reflect the recorded baseline issue. They are not a controlled layout +comparison with the separately captured bounded views. + +Reports, exact scripts, screenshots and logs are archived under +`/home/cyrille/GIT/IBL/phy-data/carl/aind-b1b0782a-b3ea-4f34-81d7-b3c6e36452b3/main-versus-pr-gui-acceptance/`. +The separate high-effort audit independently confirmed raw parent finiteness, +the exact comparator result, both process exits, restored inventory and the +rendering-error scope. No additional product-code regression was identified. + +This closes the automated main-GUI versus bounded-GUI comparison for this pinned +dataset and ledger. Carl's manual usability and scientific assessment remain +valuable, particularly on his 500-hour recording. The practical variants in +`curation-user-acceptance.md` remain proposed user tests, not claims that every +variant was executed. Acceptance of Brandon's native exporter still needs that +export and authoritative acquisition exposure. diff --git a/design/curation-mode-acceptance-ledger.json b/design/curation-mode-acceptance-ledger.json new file mode 100644 index 000000000..a64e7b957 --- /dev/null +++ b/design/curation-mode-acceptance-ledger.json @@ -0,0 +1,149 @@ +{ + "aliases": { + "a": 40, + "b": 44, + "c": 48, + "d": 49, + "extra": 24 + }, + "fields": [ + "group", + "notes" + ], + "actions": [ + { + "op": "merge", + "parents": [ + "a", + "b", + "c" + ], + "as": "merged" + }, + { + "op": "undo" + }, + { + "op": "redo" + }, + { + "op": "split", + "parent": "merged", + "threshold": -424.0037077454673, + "selected": "low", + "remainder": "rest" + }, + { + "op": "undo" + }, + { + "op": "redo" + }, + { + "op": "merge", + "parents": [ + "low", + "extra" + ], + "as": "final" + }, + { + "op": "label", + "ids": [ + "final" + ], + "field": "group", + "value": "good" + }, + { + "op": "label", + "ids": [ + "final" + ], + "field": "notes", + "value": "reviewed merged low side" + }, + { + "op": "label", + "ids": [ + "rest" + ], + "field": "group", + "value": "mua" + }, + { + "op": "label", + "ids": [ + "rest" + ], + "field": "notes", + "value": "reviewed threshold remainder" + }, + { + "op": "label", + "ids": [ + "d" + ], + "field": "group", + "value": "mua" + }, + { + "op": "label", + "ids": [ + "d" + ], + "field": "notes", + "value": "reviewed unchanged source unit" + }, + { + "op": "label", + "ids": [ + "rest" + ], + "field": "group", + "value": "noise" + }, + { + "op": "undo" + }, + { + "op": "label", + "ids": [ + "rest" + ], + "field": "group", + "value": "good" + }, + { + "op": "label", + "ids": [ + "rest" + ], + "field": "group", + "value": "mua" + }, + { + "op": "save" + }, + { + "op": "reopen" + }, + { + "op": "save" + } + ], + "expected_labels": { + "final": { + "group": "good", + "notes": "reviewed merged low side" + }, + "rest": { + "group": "mua", + "notes": "reviewed threshold remainder" + }, + "d": { + "group": "mua", + "notes": "reviewed unchanged source unit" + } + } +} diff --git a/design/curation-mode-acceptance-result.json b/design/curation-mode-acceptance-result.json new file mode 100644 index 000000000..a01ef6ef1 --- /dev/null +++ b/design/curation-mode-acceptance-result.json @@ -0,0 +1,51 @@ +{ + "ok": true, + "method": "exact all-row two-way ID mapping", + "revision": "06637b24aeca9b0984d7928d459187df46406444", + "rows": 469725567, + "assignment_dtypes": [ + "int64", + "int64" + ], + "partition_mismatch_rows": 0, + "examples": [], + "input_hashes": { + "left": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "2b9c3d4f8163bd29a0d5eb2b15205b38380d87149535f3e5820442eb05bcd936" + }, + "right": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "b0223191c2cd29d2167b975175d22428bfe444175e19dc68e67aa9a82aea5168" + } + }, + "unmapped_label_ids": { + "left": { + "group": [], + "notes": [] + }, + "right": { + "group": [], + "notes": [] + } + }, + "manual_label_mismatches": [], + "compared_fields": [ + "group", + "notes" + ], + "mapped_units": 703, + "numeric_ids_also_identical": true, + "evidence": "native bounded Qt graphical drag/K and full ordinary Supervisor without GUI views, same PR code revision", + "noop-comparison": { + "ok": true, + "method": "exact matching SHA-256 of complete NPY files and explicit manual-label tables", + "arrays": 18 + }, + "immutable-check": { + "ok": true, + "raw_files_verified": 67, + "immutable_replay_files_per_round": 19, + "rounds": 2 + } +} diff --git a/design/curation-output-contract-result.json b/design/curation-output-contract-result.json new file mode 100644 index 000000000..000098fd6 --- /dev/null +++ b/design/curation-output-contract-result.json @@ -0,0 +1,146 @@ +{ + "ok": true, + "method": "exact all-row two-way ID mapping", + "revision": "eb7983f0cd435c8acd5ef3722e19df00356b3ef8", + "rows": 469725567, + "assignment_dtypes": [ + "int32", + "int64" + ], + "partition_mismatch_rows": 0, + "manual_label_mismatches": [], + "input_hashes": { + "left": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "3cbef5e08dc915ddd57aac9c214370a34282d264695f53959c184d6f8f2b7148" + }, + "right": { + "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c", + "spike_clusters.npy": "b0223191c2cd29d2167b975175d22428bfe444175e19dc68e67aa9a82aea5168" + } + }, + "file_contract": { + "ok": true, + "assignment_oracle": { + "verified": true, + "source_rows": 469725567, + "membership_events": 3, + "threshold_splits": 1 + }, + "immutable_files": [ + "_phy_spikes_subset.channels.npy", + "_phy_spikes_subset.manifest.json", + "_phy_spikes_subset.sample_index.npy", + "_phy_spikes_subset.spikes.npy", + "_phy_spikes_subset.waveforms.npy", + "_phy_spikes_subset.window_bounds.npy", + "_phy_spikes_subset.window_strata.npy", + "_phy_spikes_subset.window_types.npy", + "amplitudes.npy", + "channel_groups.npy", + "channel_map.npy", + "channel_map_si.npy", + "channel_positions.npy", + "curation.json", + "similar_templates.npy", + "spike_templates.npy", + "spike_times.npy", + "template_ind.npy", + "templates.npy", + "timestamps.json" + ], + "native_derived_static_files": [ + "whitening_mat_inv.npy" + ], + "all_metadata_fields": [ + "amplitude_cutoff", + "amplitude_cv_median", + "amplitude_cv_range", + "amplitude_median", + "bombcell_label", + "channel_group", + "default_qc", + "exp_decay", + "firing_range", + "group", + "isi_violations_count", + "isi_violations_ratio", + "main_peak_to_trough_ratio", + "main_to_next_extremum_duration", + "noise_cutoff", + "noise_ratio", + "notes", + "num_negative_peaks", + "num_positive_peaks", + "peak_after_to_trough_ratio", + "peak_after_width", + "peak_before_to_peak_after_ratio", + "peak_before_to_trough_ratio", + "peak_before_width", + "peak_half_width", + "peak_to_trough_duration", + "presence_ratio", + "recovery_slope", + "repolarization_slope", + "rp_contamination", + "rp_violations", + "sd_ratio", + "si_unit_id", + "sliding_rp_violation", + "snr", + "spread", + "sync_spike_2", + "sync_spike_4", + "sync_spike_8", + "trough_half_width", + "trough_width", + "unitrefine_label", + "unitrefine_probability", + "velocity_above", + "velocity_below", + "waveform_baseline_flatness" + ], + "permitted_metadata_differences": [], + "ordinary_aliases": [ + { + "filename": "cluster_si_unit_ids.tsv", + "field": "si_unit_id", + "policy": "unchanged source alias" + }, + { + "filename": "cluster_sync_spike_2.tsv", + "field": "sync_spike_2", + "policy": "byte-pinned blank source column" + }, + { + "filename": "cluster_sync_spike_4.tsv", + "field": "sync_spike_4", + "policy": "byte-pinned blank source column" + }, + { + "filename": "cluster_sync_spike_8.tsv", + "field": "sync_spike_8", + "policy": "byte-pinned blank source column" + } + ], + "preparation_verified": true, + "saved_generation_verified": "96fce3cd8e02474f8a175ddbb89c571c", + "historical_parents": "recorded cumulative ancestry; current parent bound to supplied manifest" + }, + "file_inventory_counts": { + "ordinary": { + "required": 70, + "optional_artifacts": [ + ".phy", + "cluster_info.tsv", + "curation_review.json", + "phy.log" + ] + }, + "replay": { + "required": 73, + "optional_artifacts": [] + } + }, + "scope": "Full source logical output contract with independent cumulative predicate execution over every source row; explicit optional artifacts have their own checks, only GUI caches/logs excluded from byte identity." +} diff --git a/design/curation-output-equivalence.md b/design/curation-output-equivalence.md new file mode 100644 index 000000000..0035c4f5d --- /dev/null +++ b/design/curation-output-equivalence.md @@ -0,0 +1,204 @@ +# Executable curation equivalence contract + +Carl's acceptance requirement is that applying supported decisions to retained +evidence and replaying them over the full source produces the same complete +curation as applying those decisions to the full recording directly. A useful +regression must exercise actual controllers and saved files, including spikes +that were neither retained nor drawn. + +This follow-up stays on PR #1428's `feature/long-recording-curation` branch. +The separate empty-waveform feature projection fix belongs to +[phylib PR #70](https://github.com/cortex-lab/phylib/pull/70); bounded mode does +not require a dependency bump for that guard. + +## Permanent controller test + +`phy/apps/large_recording/tests/test_controller_equivalence.py` creates a finite +660-spike, six-unit sorting with an explicit acquisition gap, imported producer +metadata and sparse stored waveforms. Three window/budget/seed configurations +retain genuine subsets; every active unit has unseen source spikes. + +One shared action runner drives ordinary TemplateController/Supervisor and the +manifest-selected bounded controller/Supervisor. It never substitutes manual +`BoundedSession.record_*()` calls for controller actions. The amplitude panel +receives real Qt Alt-right-drag events and the normal split QAction. Only five +points may be drawn while the parent contains more spikes. The source includes +amplitudes exactly equal to the requested threshold; they stay in the remainder. +Viewport calibration accommodates floating pixel transforms without changing +the spike values or bypassing the split action. + +The same ledger exercises: + +1. Merge, undo/redo, amplitude split, undo/redo, and merge one child with another + source unit; explicit GOOD/MUA, noise, cleared classification and custom notes; + undo/redo of reviewed labels, a new edit discarding the redo branch, save, + close, reopen and save again. +2. Prepare the full replay for another round, merge and split again, review both + new memberships, edit/undo/redo an explicitly seeded note, save and reopen. +3. Prepare/save/reopen/apply without edits. Complete NPY files and manual group + and notes tables must be byte-identical to the preceding replay. +4. Prepare again and leave a separate merge unreviewed. The checker verifies + bounded clearing and explicitly admits only supported native ancestor + inheritance, rather than pretending these imported values must agree. + +Expected reviewed labels are written independently in `EXPECTED_ONE/TWO`. +Each round compares full saved ordinary data with full bounded replay using the +original source and the actual final bounded package. Original inputs are hashed +before and after; GUI callbacks and mapped resources are explicitly cleaned up. +The GUI layout is scoped to amplitude and tables, not an all-view rendering test. + +An early extra snapshot assertion exposed ordinary inherited-note undo behavior: +editing an inherited parent note on a derived unit and undoing it restores an +empty value, rather than the displayed inherited note. The ledger seeds an +explicit reviewed note before testing its edit/undo, matching the explicit-label +contract. That initial diagnostic is retained locally; it is not claimed as a +successful inherited-note undo check or fixed by this PR. + +## File-by-file contract + +The opt-in checker is `tools/curation_output_contract.py`, called by +`tools/compare_curation_modes.py compare --source ...`. The existing basic +comparison remains available for complete partition/selected-label comparisons. + +| Output class | Required check | +| --- | --- | +| `spike_clusters.npy` | Every source row belongs to the same final partition through a two-way ID bijection. An independent chunked oracle also executes cumulative merges/split predicates on original source rows and checks exact replay IDs. Assignment dtype/vector-column representation and native allocated IDs may differ. | +| Immutable NPY arrays and producer sidecars | Every full array/file matches the original bytes, including times, amplitudes, features, templates, channels, waveform rows/channels/samples, producer window plans, timestamps and suggestion bytes when present. | +| Imported and curator metadata | All fields are checked. Untouched units retain source values. New identities clear bounded producer fields unless explicitly reviewed. Every surviving label event requires an independent final expectation, including clear/null. Native differences must be supported ancestor values or empty values and are reported individually. | +| Metadata files/schema | Replay preserves producer basenames, delimiters and columns. Native canonical field tables may supersede byte-identical source aliases. Entirely blank native source columns may retain retired rows only when their original file is byte-identical. Missing blank tables still fail. Typed sidecars and tables must agree. | +| Parameters | Original Python structure outside managed extent/exposure/raw-path overrides is preserved; evaluated parameter values and effective recording clock/exposure/raw reference are checked. This is not a general proof of executable Python equivalence. | +| Journal/lineage | Exact operation/version/clearing-payload/predicate schemas, journal hash, current parent/preparation binding, final identity/revision, cumulative allocated ancestry, retired identities, metadata snapshots and explicit label intent are checked. | +| Saved generation | Pointer, descriptor and all checkpoint file hashes bind the export's journal, metadata, schema, allocator, provenance and reviews to the saved session. Every retained row's full output assignment must equal its saved assignment. | +| Suggestions/reviews | Suggestion bytes and source-hash bindings survive; inherited decisions match semantically. This action ledger makes no new proposition decisions. Their separate acceptance procedure remains in force. | +| Native derived whitening inverse | When absent from source but materialized by ordinary Phy, it must equal the inverse of the source whitening matrix, or the channel-count identity when no matrix exists. It is not an arbitrary extra-file exemption. | +| GUI state/derived summaries | `.phy` caches and logs may differ. Optional generated cluster summaries must have the complete active catalog and agree with their manual labels. Other optional source snapshots are byte-checked. Unknown or missing assets fail. | + +The checker does not call the bounded assignment projector or event metadata +interpreter. It uses the source format reader for admission and independently +executes cumulative membership intent and tracks clearing and label intent. +The assignment oracle reads original scalar attributes for every source spike, +including spikes omitted from every bounded package. It applies earlier-round +actions before later actions, skips history already incorporated in the supplied +original source, and compares the resulting IDs exactly with the replay. Split +child order, comparator equality and parent-local nonfinite policy are checked. +Memory use is bounded by the requested chunk size and action history. +Native producer inheritance +is constrained to supported ancestor values/empty values; it does not reconstruct +every intermediate largest-parent inheritance choice. Historical parent revisions +are recorded ancestry, not fetched historical files. The current parent and +saved generation are verified against the supplied final package. + +`--source` without a package supports first-round checks but explicitly reports +unverified preparation/saved-generation bindings. The full acceptance command is: + +```bash +uv run --no-sync python tools/compare_curation_modes.py compare \ + /data/ordinary-final /data/bounded-full-output \ + --source /data/original-source --bounded-dir /data/final-bounded \ + --expectations /data/expected-labels.json --chunk-size 1000000 \ + --output /data/full-output-comparison.json +``` + +The report path must be fresh and outside every input folder. Expectations map +replay unit IDs to actual planned final decisions, not values copied from the +export under test. Use the original source across repeated rounds. Immutable +input preservation is checked by the test's pinned before/after inventory; a +post-hoc CLI source path alone cannot establish that its caller never edited it. + +## Corruption regressions and scope + +`test_output_contract.py` includes positive canonical/native-alias/blank-column +and implicit-group cases, and deliberate corruptions of arrays, memberships, +manual labels, new/untouched producer values, source aliases, metadata layout, +parameters, missing/extra files, malformed provenance, journal payloads, +predicates, parent bindings, allocator state, retired identities, review binding +and generated catalog coverage. Corrupt metadata tests repair table/JSON/revision +consistency so policy checks must catch them. Journal tests repair the correct +canonical JSON hash including its trailing newline, including source-only checks. +A standalone CLI regression catches import/path problems outside pytest. + +A two-merge case swaps derived unit IDs while preserving the same partitions +and identical labels. The basic comparison accepts that rename; the strict +assignment oracle rejects its now-incorrect recorded source-parent relationships. + +A subsequent independent audit exposed a checker false acceptance: a split +threshold could be changed in both journal and provenance, with repaired +checkpoint hashes, while leaving the old assignments in place. Schema, ancestry +and saved-row agreement did not establish predicate semantics. The independent +all-source oracle closes that gap; the corruption is now a regression, including +source-only and cumulative-round checks. This was a validation gap, not evidence +that the normal curation producer generated an incorrect split. + +These tests guard the supported workflows and declared exceptions. They do not +prove equivalence for every possible action sequence or arbitrary lasso/feature +cuts, which cannot assign unseen spikes in bounded mode. Controller tests use +finite source amplitudes; ordinary display sanitization of nonfinite values can +differ from raw bounded predicates. Existing raw predicate regressions test +nonfinite/equality policy separately. + +## Real-data validation + +The existing main-GUI versus PR comparison uses Carl's converted engineering +dataset: **469,725,567 spikes**, a 112.859-hour elapsed clock, and unverified +inferred exposure. It is not the newer 500-hour recording or native-export +acceptance. The stronger file stage passed against the hash-pinned all-row +mapping, checking **46 metadata fields**, **20 immutable files**, the actual +saved generation and full file inventory. It recognized the unchanged plural +SI-ID alias, three byte-identical blank source columns, and a valid native +derived whitening inverse. These are explicit file-format artifacts. + +The initial strict run correctly surfaced a checker admission gap: ordinary +loading skips entirely blank imported columns, leaving their original files +unchanged after source IDs retire. A narrow byte-pinned rule and a forged-value +regression were added. The first standalone run also found a tool import-path +error, now covered by an actual subprocess CLI regression. Both diagnostic logs +are retained; neither is described as a product curation mismatch. + +The final standalone CLI run passed over all **469,725,567 rows**, including +all 46 metadata fields, 20 immutable files, the exact saved generation and full +file inventories. It reported zero unexpected partition or manual-label +mismatches and zero producer-value differences for this particular saved pair. +The subsequent predicate-oracle run independently checked all 469,725,567 +original rows through two merges and one threshold split, with no assignment +contradictions. The runtime checker is commit `eb7983f0`; the complete local +report is `contract-comparison-predicate-oracle-committed.json` in the acceptance +directory below. +The [committed report summary](curation-output-contract-result.json) records the +runtime revision, input hashes, checked fields/files and admitted native artifacts. + +After the predicate-oracle fix, `make test-full` passed **1,041 tests** (499 core +and 542 application) with one existing benchmark skip, including lint and +formatting. The complete **52-case file-contract target** includes coherent +current/inherited threshold forgeries, an unseen-row-only contradiction, and +an already-curated source whose incorporated history must be skipped. +The **26 literal oracle cases** cover all four operators, equality, child +ordering, scalar vector/column attributes, wide allocated IDs, nonfinite policy +and mixed cumulative actions across chunk boundaries. Independent targeted +verification also passed all three controller plans and 18 comparison cases. +The Sol 6.1 high auditor independently reran the original forged-checkpoint +reproduction: it is now rejected at source spike 4, and the implementation review +found no further issue. +Documentation and cross-platform CI links are recorded at handoff. Local +reports and reproducibility scripts remain in the dataset's +`main-versus-pr-gui-acceptance/` directory; no source or prior baseline output was +modified by the checker. + +The independent high-effort audit found and regression-tested missing blank +table coverage, journal/history agreement, new-unit ancestry, malformed history +and invalid event payloads. Its final read found no remaining must-fix issue. + +## Parallel phylib follow-up + +[phylib PR #70](https://github.com/cortex-lab/phylib/pull/70) returns the existing +zero-filled feature tensor when a request has no stored waveform overlap or +no channels, instead of attempting PCA on an empty array. Five real on-disk +regressions cover empty requests, no coverage, empty channels, partial coverage +and complete coverage. Three fail before the guard; **308 full-suite tests** and +cross-platform/minimum-SpikeInterface CI pass afterward. + +A separate copied real sorting was reopened in unmodified main Phy with only +the isolated patched phylib on `PYTHONPATH`. It exited zero with **zero +tracebacks**, versus five Feature View errors in the original equivalent reopen. +Complete assignments and group/notes files remained byte-identical. Original +validation data and installed environments were preserved. This addresses the +recorded baseline renderer failure and does not change PR #1428's dependency pin. diff --git a/design/curation-user-acceptance.md b/design/curation-user-acceptance.md new file mode 100644 index 000000000..3101d3409 --- /dev/null +++ b/design/curation-user-acceptance.md @@ -0,0 +1,277 @@ +# Curation-mode user acceptance protocol + +Status: protocol for the graphical bounded curation workflow; run both GUI +branches against the same full-source data before accepting the feature. + +Permanent controller-to-controller tests and the stronger whole-output checker +are now described in [the executable equivalence contract](curation-output-equivalence.md). +They supplement the manual protocol and the pinned main-branch GUI trial below. + +## Purpose and evidence boundary + +This is an end-to-end acceptance check for ordinary and bounded curation. It +compares their complete final spike assignments and explicitly reviewed manual +labels after applying the same user action ledger. It does not compare bounded +evidence arrays with full-source arrays, nor does it treat a streamed reference +as ordinary GUI evidence. +The harness comparison tests live in +`phy/apps/large_recording/tests/test_curation_comparison.py`. + +The ordinary GUI run exercises the usual complete-data workflow. The bounded +GUI run exercises retained evidence, semantic action journaling, and full-source +replay. Separately, `tools/compare_curation_modes.py reference --engine +ordinary` executes full Supervisor commands without GUI views attached; it is +an ordinary engine reference, not evidence from the ordinary GUI. The +`--engine streamed` reference is a further independent streamed baseline, also +not GUI evidence. + +## Freeze inputs and outputs + +1. Pin the exact ordinary and bounded code revisions under review. Record both + revisions and the comparison tool revision. +2. Preserve an immutable, complete snapshot of the source sorting. Create a + separate writable working copy for the ordinary GUI, which saves in place. + Record hashes for `spike_times.npy`, `spike_clusters.npy`, and + `amplitudes.npy`. Do not use the bounded evidence package as the ordinary + input. +3. Prepare the bounded package from the immutable source and retain its manifest, + window plan, journal, and preparation report. Use fresh output directories + outside the immutable source copies. +4. Use the same selected source units, threshold, action order, and explicit + final manual labels in both GUI runs. Save each result and preserve its + final full output. Keep screenshots or a short screen recording of the + relevant preview and actions alongside the ledger. + +For real data, record the dataset's actual extent and time axis from metadata. +Do not use a generated 500-hour example as evidence about a different real +recording. In particular, Carl's currently available converted dataset is +469 million spikes and 112.9 hours on its declared clock, with a frame value +near 100 hours; it is not a 500-hour dataset. Treat any newer source revision +as a separate input and recapture its metadata. + +## Graphical action ledger + +Record one shared `ledger.json`. The following schema illustrates the order and +the required alias mapping; replace example aliases, IDs, threshold, and labels +with those actually used. The actions describe user intent, including history +and save boundaries. + +```json +{ + "aliases": {"unit_a": 40, "unit_b": 44, "unit_c": 48}, + "actions": [ + {"op": "merge", "parents": ["unit_a", "unit_b"], "as": "merged"}, + { + "op": "split", + "parent": "merged", + "threshold": 1.5, + "selected": "low", + "remainder": "rest" + }, + {"op": "undo"}, + {"op": "redo"}, + {"op": "label", "ids": ["low"], "field": "group", "value": "good"}, + {"op": "label", "ids": ["low"], "field": "notes", "value": "reviewed low side"}, + {"op": "label", "ids": ["rest"], "field": "group", "value": "mua"}, + {"op": "label", "ids": ["rest"], "field": "notes", "value": "reviewed remainder"}, + {"op": "label", "ids": ["unit_c"], "field": "group", "value": "mua"}, + {"op": "label", "ids": ["unit_c"], "field": "notes", "value": "reviewed unchanged unit"}, + {"op": "save"}, + {"op": "reopen"}, + {"op": "save"} + ], + "expected_labels": { + "low": {"group": "good"}, + "rest": {"group": "mua"}, + "unit_c": {"group": "mua"} + }, + "fields": ["group", "notes"] +} +``` + +Use the actual final unit identities returned by each run to fill the aliases. +In ordinary Amplitude View, select the **template** amplitude type, which reads +`amplitudes.npy`; feature and template-feature amplitudes define different +quantities. Run ordinary GUI acceptance with the graphical threshold interaction +(Alt plus right-drag, then `K`) and bounded GUI acceptance with the same threshold. Use +source units whose amplitudes are all finite for this graphical equivalence +check. Bounded splitting assigns finite values strictly below the threshold to +the selected side; equal and nonfinite values stay in the remainder. Require +evidence on both sides before committing. Test nonfinite values separately +through the raw source predicate engine: ordinary TemplateModel displays +nonfinite amplitudes as zero, so its graphical selection need not match that +raw predicate. The ordinary comparison engine deliberately uses raw source +`amplitudes.npy` values rather than displayed values. + +Record intentional no-ops and state boundaries in the acceptance log alongside +the shared action ledger when relevant: for example an invalid threshold +rejected before mutation, an undo with nothing available, a no-change label, a +save/reopen/save cycle without edits, or a second save without edits. The +reference harness ledger currently supports merge, split, label, undo, redo, +save, and reopen; do not put unsupported no-op operations into its `actions` +array. Record the expected result (no mutation, rejection, or unchanged output) +so both GUI branches are checked consistently. + +After the final redo and before concluding, assign explicit final `group` and +`notes` values to every transformed or deliberately reviewed final unit, plus +an unchanged control. Other untouched units retain and compare their source +labels. This keeps the comparison about reviewed manual labels, not +legacy producer-label inheritance. The protocol compares only the requested +manual fields. Producer metrics such as SNR may intentionally be cleared on +transformed memberships in bounded mode. Membership-scoped quality automation +may also give transformed units different automatic status; record and review +that behavior separately instead of treating it as a manual-label mismatch. + +## Complete-output comparison + +Export the ordinary GUI result and run `phy apply-curation` on the bounded +session to produce its complete full-source result. The bounded GUI assignment +at `K` covers retained evidence rows; replay applies its recorded predicate to +the unseen source spikes. Save both results to separate complete output +directories. +Put the JSON report in a fresh path outside both compared directories and both +immutable source copies. Compare the complete output directories (not the +bounded package or its retained rows): + +```bash +uv run python tools/compare_curation_modes.py compare \ + /data/ordinary-final /data/bounded-replay-final \ + --fields group notes --output /data/curation-comparison.json +``` + +The report checks equal source row counts and exact row-aligned `spike_times`, +then proves partition equivalence using a two-way raw cluster-ID mapping over +all assignment rows in chunks. It checks the requested manual labels through +that mapping. A label attached to an ID with no assigned spikes is reported as +unmapped and fails acceptance; empty or unmapped label tables are not silently +ignored. `--fields` is an explicit allowlist: include the manual fields reviewed +in the ledger and do not add producer metrics or automatic quality fields whose +bounded membership policy intentionally differs. Review the full report, +including input hashes, ID mapping, examples, and label diagnostics. A zero exit +status and `ok: true` are required. + +This check establishes final partition and chosen manual-label equivalence for +the pinned inputs and ledger. It does not prove identical intermediate UI +behavior, producer metadata, automatic quality calculations, or scientific +validity of the reviewed labels. Preserve GUI evidence and compare reports with +the pinned revisions and immutable source hashes. + +## Practical user-test matrix + +Start with one shared ledger, then add small variants rather than attempting +every combination in one session. These are proposed user checks; the observed +automated run is recorded separately below. + +| Check | What to verify | +| --- | --- | +| Merge and review | Merge three units, split the result, then merge one child with another original unit. Explicitly assign GOOD, MUA, noise, a cleared/unsorted group and custom notes on final units. Try reversing merge-candidate order in a separate run. | +| History | Undo and redo a merge, split and label. Undo, make a different edit and confirm redo is discarded. Track the actual new IDs rather than assuming allocation reuse. | +| Thresholds | Compare graphical `K` with the exact recorded numeric threshold. Test equality, a rejected one-sided threshold, and selection/context changes that clear a preview. Test raw nonfinite values separately from ordinary graphical equivalence. | +| Persistence | Save, close, reopen and save without edits. Apply, prepare again, save without edits and apply again. Compare complete assignments and explicit manual labels after both rounds. | +| Sampling and gaps | Repeat a small ledger with a different seed, window plan or budget while both children retain evidence. Keep the same authoritative gap/exposure declaration and source predicate. The complete result should be independent of the displayed sample. | +| SLAY decisions | Separately accept and reject suggestions, transform an overlapping unit and check stale reasons. Preserve decisions through further rounds using the existing `design/slay-roundtrip-validation.md` procedure. The comparison reference ledger does not implement proposition operations. | + +## Observed engineering acceptance, 26 October 2026 + +Runtime code and comparison tool were pinned at +`06637b24aeca9b0984d7928d459187df46406444` on PR #1428's existing +`feature/long-recording-curation` branch. The source was Carl's converted test +overlay for computation `b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`: **469,725,567 +spikes**, 705 original units and a 112.859-hour elapsed clock. Its inferred +exposure remains unverified acquisition truth; this is engineering integration +evidence, not native-export acceptance or a run on Carl's newer 500-hour data. + +The bounded package contained 1,486,361 evidence spikes. A native X11 Qt GUI +performed the real `Alt`-right drag and `K` action, then saved and reopened. +The shared [recorded ledger](curation-mode-acceptance-ledger.json) includes: + +1. Merge 40, 44 and 48; undo and redo. +2. Graphically split the merged unit using the exact preview threshold + **-424.0037077454673** in raw source amplitude units; undo and redo. +3. Merge the low-amplitude child with original unit 24. +4. Explicitly label that new unit GOOD, the remainder MUA and unchanged + control 49 MUA, with custom notes. Change a label to noise, undo it, then + branch with a different label and restore the intended final MUA value. +5. Save, reopen and save again without edits; apply to the full source. + +The independent reference ran the same ledger through the complete ordinary +Supervisor/Clustering and ClusterMeta implementations without attaching GUI +views. It used raw source amplitudes for the predicate. Both paths used the +same PR revision: this was **not a main-branch versus PR ordinary GUI trial**. +The ordinary run finished in 3:31 with 29.8 GiB peak RSS and no swap, under a +96 GiB virtual-memory ceiling. The bounded GUI plus first replay took 53.3 s. +These are single warm-cache observations, not controlled benchmarks. + +The independent complete-output comparison passed over **all 469,725,567 rows**: +exact row-aligned sample times, **zero partition mismatches**, a bijection over +703 final units, and **zero group/notes mismatches or unmapped label IDs**. +The numerical unit IDs also happened to agree; that is not required by the +partition test. Complete NPY file hashes for assignments differ because the +ordinary reference writes a vector and replay retains the source column shape; +the comparator normalizes both shapes and checks every assignment exactly. +The [committed report summary](curation-mode-acceptance-result.json) retains +the comparison method, revision, counts, diagnostics and input hashes. + +A further preparation from that full output, direct session save, reopen, +no-edit save and replay completed in 121.8 s. All **18 complete NPY files**, +including assignments and spike times, and both explicit group/notes tables +matched the preceding replay byte-for-byte by SHA-256. This second cycle tests +persistence without edits; it is separate from the native GUI save/reopen test. +An independent hash verifier also confirmed that all **67 original download +files** remained unchanged, and that **19 immutable source files in each full +output** matched the converted input. + +### Screenshots and reproduction + +The three images in `docs/images/amplitude-threshold-workflow/` show a separate +disposable demonstration on the newly prepared real output, selecting unit 708. +They show its sampled overview before the drag, the orange threshold preview +with matching waveform highlights, and the two children after actual `K`. +The exact threshold is **-180.09097605065006**, with retained children of 3721 +and 3643 spikes and 34 highlighted waveform traces. This is a separate split +from the shared comparison ledger. All 88 files in the input bounded package +retained their hashes; the capture operates on a temporary copy. + +```bash +QT_QPA_PLATFORM=xcb LIBGL_ALWAYS_SOFTWARE=1 \ + xvfb-run -a -s '-screen 0 2600x1600x24 +extension GLX' \ + uv run --no-sync python tools/capture_long_recording_views.py \ + --amplitude-split --bounded-dir /data/fresh-bounded \ + --cluster-ids 708 --output-dir /data/amplitude-screenshots + +QT_QPA_PLATFORM=offscreen uv run --no-sync python \ + tools/compare_curation_modes.py reference /data/full-source \ + design/curation-mode-acceptance-ledger.json /data/ordinary-reference \ + --engine ordinary --chunk-size 1000000 + +uv run --no-sync python tools/compare_curation_modes.py compare \ + /data/ordinary-reference /data/bounded-full-output --fields group notes \ + --chunk-size 1000000 --output /data/comparison.json +``` + +The reference command uses full ordinary in-memory curation and needs memory +appropriate for the source. Its comparison artifact is assignments, times, +manual-label tables and provenance, not a complete sorting export. Choose the +actual active unit in a fresh bounded package when reproducing the screenshots. +Pixel quantization can move the requested threshold slightly; `capture.json` +records the exact committed value. + +Local reports, scripts and logs are archived in +`/home/cyrille/GIT/IBL/phy-data/carl/aind-b1b0782a-b3ea-4f34-81d7-b3c6e36452b3/curation-mode-acceptance/`. +The initial one-off native GUI script needed a harness correction from QWidget +focus handling to QWindow activation; the corrected GUI and reference processes +both exited zero. The final capture tool also exits zero. + +`make test-full` passed **960 tests** (499 core and 461 application), with one +existing benchmark skip, including lint and formatting. The 24 added tests cover +six graphical workflows and eighteen independent comparisons, including raw +nonfinite semantics, ID renumbering, metadata differences, callback cleanup, +history and output protection. The independent Sol 6.1-high audit found no +remaining concrete blocker. Documentation checks and CI are recorded at handoff. + +The later [main-GUI comparison](curation-main-gui-validation.md) completed this +same ledger on unmodified main and matched every assignment and manual label. +It retains baseline Feature View rendering errors as an explicit limitation. +Carl's manual usability and scientific review remain useful, together with the +small practical variants in the matrix. Neither automated run establishes the +scientific validity of splits or of the acquisition exposure declaration. diff --git a/design/large-recording-curation.md b/design/large-recording-curation.md new file mode 100644 index 000000000..26f6d482e --- /dev/null +++ b/design/large-recording-curation.md @@ -0,0 +1,348 @@ +# Long-recording curation in Phy 2.x: Phy-side design contract + +**Status:** implementation present; acceptance validation is ongoing. This +specification is for Phy-side behavior; it does not authorize changes to the +AIND/Code Ocean export pipeline. The current branch implements the bounded +workflow behind an explicit manifest while retaining standard behavior for +ordinary datasets. See the [integration plan](large-recording-integration-plan.md) +and [validation record](large-recording-validation.md) for implementation scope, +checks, and remaining acceptance work. + +The [integration audit and plan](large-recording-integration-plan.md) refines +this proposal with assignment revisions, transactional persistence, capability +gates and ordered commit acceptance criteria for one implementation PR. Its enablement gates take precedence +over the original implementation sequence in Section 9. + +## 1. Objective and boundaries + +Curate chronic Neuropixels sorting datasets of roughly 1000 hours without opening hundreds +of millions or billions of spikes in the existing in-memory Phy clustering model. +The workstation downloads the **complete sorter output once**; downloading the +complete raw voltage recording is **not required**. All preparation, interactive +curation, full-data replay, and regeneration run locally: + +```text +complete, immutable sorting (local disk) + | prepare-large: streamed/batch + v +bounded, temporally distributed Phy dataset + full-duration summaries + | existing Template GUI + Unit Stability + v +semantic curation actions (not only sampled row assignments) + | apply-curation: streamed/batch against complete sorting + v +new complete curated sorting (original untouched) + | prepare-large again, potentially different evidence windows + +-----------------------> new bounded GUI dataset +``` + +The **complete spike-level sorting stays outside the interactive GUI**. +The bounded dataset must reproduce useful current Phy workflows rather than +pretend to be a statistically representative replacement for every spike. +Full-duration plots and rates come from separate complete-sorting summaries. + +Out of scope: cloud services, rewriting phylib for out-of-core clustering, +longitudinal unit-identity graphs (phy3), automatic change-point detection, and +implementing upstream waveform extraction in this repository. + +## 2. The three index spaces (essential contract) + +All index values below are **zero-based row positions**, not timestamps, sample +numbers, SpikeInterface unit IDs, or stable IDs assigned by a sorter. + +| Symbol | Meaning | Where it lives | +| --- | --- | --- | +| `g` | Full-sorting spike row index, in immutable source order | Original `spike_times.npy` / `spike_clusters.npy` | +| `b` | Bounded spike row index used by the GUI | Bounded `spike_times.npy` / `spike_clusters.npy` | +| `w` | Row in the stored waveform/channel tensors | `_phy_spikes_subset.waveforms.npy` / `.channels.npy` | + +**Canonical mapping:** the bounded package writes +`_phy_spike_global_ids.npy`, a one-dimensional strictly increasing `int64` +array of length `N_bounded`. It is `G[b] = g`. The underscore prefix is +intentional: phylib treats ordinary `spike_*.npy` files as per-spike +attributes, so `spike_global_ids.npy` would be needlessly interpreted as one. + +**Dataset-local waveform IDs:** `_phy_spikes_subset.spikes.npy[w] = b` in the +bounded package. In the *upstream full export*, the same filename instead +contains `g` because that file accompanies the full sorting. The file **never** +means an intrinsic immutable/global ID; it indexes whichever Phy dataset is in +its directory. + +Example: + +```text +full indices selected by prepare-large, G: + [14, 19, 27, 117, 203, 839] +bounded rows b: + [ 0, 1, 2, 3, 4, 5] +upstream waveform full indices: + [14, 117, 839, 1902] +intersection with G: + [14, 117, 839] +bounded _phy_spikes_subset.spikes.npy: + [ 0, 3, 5] +``` + +Recover full source row numbers for waveforms by composition: + +```python +G = np.load('_phy_spike_global_ids.npy', mmap_mode='r') +B = np.load('_phy_spikes_subset.spikes.npy', mmap_mode='r') +full_waveform_rows = G[B] # [14, 117, 839] +``` + +`prepare-large` must preserve source row order, subset all per-spike arrays +consistently, retain absolute spike sample times, and remap matching waveform +rows **along with the corresponding channel/waveform data**. A sorted +intersection / `np.searchsorted` implementation needs no billion-element +inverse lookup array. Sorting by sample time is not a substitute for preserving +row identity; use the source export's exact row order (including ties). + +## 3. Source and optional upstream waveform inputs + +The source is a locally accessible standard Phy/SpikeInterface export (or an +adapter yielding equivalent arrays). The original spike index `g` refers to +**the exact exported row ordering**, not an assumed row in an unrelated +SpikeInterface spike vector. Source arrays and their provenance must be +validated before processing; never infer a row mapping from spike timestamps. + +The existing, unmodified Phy waveform files are: + +| Filename | Meaning | +| --- | --- | +| `_phy_spikes_subset.spikes.npy` | `int64[N_waveforms]`: source full spike row indices, sorted/unique | +| `_phy_spikes_subset.channels.npy` | `int32[N_waveforms, C_sparse]`: model channel indices, with `-1` padding; `channel_map.npy` maps these to raw-file columns | +| `_phy_spikes_subset.waveforms.npy` | numeric `[N_waveforms, N_samples, C_sparse]`: aligned waveform data, zero on padded channels | + +The upstream AIND pipeline **may later** supply the following proposed files; +these names/schema are *not yet finalized with its developers*: + +| Optional file | Initial proposal | +| --- | --- | +| `_phy_spikes_subset.window_bounds.npy` | `int64[W, 2]` recording sample intervals, `[start, end)` | +| `_phy_spikes_subset.window_types.npy` | `uint8[W]`: `0` survey, `1` rescue | +| `_phy_spikes_subset.window_strata.npy` | `int32[W]`: survey stratum index, `-1` for rescue | +| `_phy_spikes_subset.manifest.json` | Small versioned scalar metadata: source identity, time base, extractor geometry, parameters, seed, array semantics | + +There are deliberately **no structured NumPy dtypes**, mixed-purpose `N x 5` +matrices, or massive JSON window arrays. The upstream manifest is useful for +planning downstream windows but is not required for fixture-based development. +The presence of a window denotes *potential* waveform coverage, not one stored +waveform for every spike within that interval: per-unit waveform caps still +apply. The three established waveform files remain authoritative for which +individual spikes actually have stored waveforms. + +In the absence of the proposed window sidecars, `prepare-large` may use the +existing three files and their spike-index/time distribution as available +evidence, with reduced guarantees about raw-I/O locality and survey windows. +A dataset with neither raw traces nor pre-extracted waveforms must not claim +that Waveform/Feature View evidence is available. + +The full export's identifier/fingerprint should combine an immutable sorting +asset ID when available, full spike count, and a digest of a canonical +row-ordered immutable spike array (`spike_times.npy` is one candidate). The +upstream reservoir must be verified against that *exact* source. A mutable +`spike_clusters.npy` hash must not be the sole identity used across curation +iterations. Byte-level hashes in examples are illustrative; the definitive +AIND provenance method is a pending joint decision. + +Coordinate identity and assignment revision are separate: the latter binds the +initial full cluster assignments and metadata for replay. Each curated output +records its parent revision and journal digest. Validate relevant immutable +feature, acquisition and time-base provenance as well as spike rows; equal +timestamp arrays alone do not establish source identity. Production hashing +must stream file contents rather than read a complete array into memory. + +## 4. Bounded-package Phy-side output + +`prepare-large` creates a **new directory**, not an in-place rewrite of the +original. At minimum it contains ordinary, internally consistent Phy files +(`params.py`, `spike_times.npy`, `spike_templates.npy`, `spike_clusters.npy`, +channel/template arrays, and present per-spike attributes), plus: + +| File | Role | +| --- | --- | +| `_phy_spike_global_ids.npy` | Required `int64[N_bounded]` sorted mapping `b -> g`, no duplicates | +| `_phy_curation_manifest.json` | Versioned local contract; immutable source ID/fingerprint, complete/bounded counts, durations, preparation algorithm/parameters/seed, evidence limitations and file names | +| `_phy_spikes_subset.{spikes,channels,waveforms}.npy` | Existing Phy-format stored waveforms, *remapped to bounded row indices* (if available) | +| Full-duration metrics | Separate time-binned evidence, described in Section 6; do not pretend proxy metrics are full metrics | + +Unselected spike rows are absent from the GUI arrays, **not deleted from the +complete sorting**. Absolute `spike_times.npy` values remain in acquisition +sample coordinates and require a dtype capable of representing 1000 h at the +source sample rate (typically `int64`, not `int32`). For 30 kHz, 1000 h is +108 billion samples. Keep recording start/end, session/block boundaries, valid +acquisition exposure, and gaps distinct from periods of zero firing. + +Do not rely on phylib's fallback of estimating `model.duration` from the last +spike when raw traces are absent. The future GUI metrics/time-axis integration +must take the authoritative duration/coverage metadata instead. Standard +`params.py` still identifies the bounded Phy dataset; the manifest augments it. + +## 5. Preparation algorithm: scientific and memory constraints + +A first version should: + +1. Scan the **full** local sorting in a bounded-memory batch pass; calculate + full-duration per-unit time bins and coverage/exposure. +2. Partition *valid acquisition time* into coarse strata. Select short shared + contiguous survey windows at deterministic seeded offsets across sessions/ + blocks (illustrative defaults: one 30 s window per 1 h stratum). +3. Include **all units and all spikes within each chosen window** as initial + bounded candidates; preserve source indices and absolute times. +4. Enforce configurable **total spike / RAM budgets**. Add shared rescue windows + for underrepresented units, prioritizing benefit to several units and + temporally separated evidence; retain bounded deterministic selection. +5. Where a waveform reservoir exists, favor windows with usable waveform + coverage and report per-unit counts and uncovered regions. Selecting a + window does not guarantee all its spikes have stored waveforms. +6. Write an atomic, versioned bounded package. Verify array-length alignment, + sorted global IDs, waveform/feature alignment, provenance, coverage and + source immutability before marking it usable. + +These choices are **heuristics**, not a claim of unbiased sampling, complete +physiological representativeness, or equivalent CCG/ISI statistics. Never label +metrics from the bounded subset as full-data measurements. Windows identified +manually in a future Unit Stability View may motivate a regenerated package; +automated anomaly detection is out of scope. + +The upstream `20_000` waveform cap per unit is **not** a mandatory local budget: +for 1000 units x 20k waveforms x 90 samples x 32 channels x 4 bytes it would +approach **230 GB** uncompressed. Extraction, transfer, storage and bounded +GUI consumption require independently configurable *total* byte/spike budgets. + +## 6. Unit Stability evidence from the full recording + +A Unit Stability View should work on ordinary 1-10 h recordings too. For a +selected cluster it plots time-binned **full-sorting** spike count / firing rate, +amplitude median/quantiles, depth/position and presence/coverage when those +quantities exist in the source. Use an explicit time axis, valid acquisition +exposure per bin, data-availability flags, and recording gaps. Counts divided +by wall-clock bin width are wrong for acquisition gaps. No missing metric should +be silently synthesized. + +Initial storage may use regular homogeneous `.npy` arrays plus a small manifest, +with explicit row-to-cluster-ID mapping and bin edges; exact metric schema and +plot integration belong to the Unit Stability PR. At each full-data curation +iteration, merged/split clusters must receive **correctly recomputed or +incrementally updated** summaries, not merely the stale parent-unit curves. + +Current `BaseController.get_mean_firing_rate()` divides the loaded cluster's +`n_spikes` by `model.duration`; this would severely undercount a bounded unit. +When a bounded package is detected, its full-count/exposure-aware value must +replace that calculation wherever presented. The same provenance distinction +applies to the Cluster View, correlograms, ISI and other proxy-based statistics. + +Window statistics need explicit observation intervals. Do not connect ISIs or +CCG pairs across independent segments or unsupported gaps. CCG normalization +must use a declared lag-exposure estimator for those intervals; initially hide +unsupported normalized diagnostics. Whole-window sampling, rescue selection +and incomplete waveform coverage do not establish unbiased full-data quality +metrics or chronic biological unit identity. + +## 7. Semantic curation and full-data replay + +The current `Clustering` operates on arrays of loaded spike rows, creates new +cluster IDs during `assign/split`, and saves those local assignments. They +cannot by themselves define the fate of billions of unobserved spikes. + +Before mutations in a bounded GUI session, record **versioned semantic events** +with preconditions, affected source clusters, deterministic operation parameters, +resolved cluster IDs and provenance. Track edits through existing undo/redo and +save. Apply them in order against the *complete* local sorting to write a new +curated output (never overwrite the original). + +| Operation | Initial replay policy | +| --- | --- | +| Merge complete units | Reapply to every source spike in the units; replayable | +| Cluster quality/metadata labels | Replay per resulting cluster identity; replayable | +| Threshold split | Replay only if the predicate/feature definition is serialized and an equivalent full-data per-spike attribute exists; otherwise disallow | +| Lasso or arbitrary selected spike-ID split | Not replayable for unseen spikes by default; **block in bounded mode** unless a valid full-data rule is supplied | +| Undo/redo | Journal must represent the committed final event sequence and allow exact save/reopen, not duplicate reverted edits | + +Unbounded/ordinary Phy behavior must remain unchanged. It is not acceptable to +silently treat a selection of bounded spike IDs as a valid full-unit split. +Existing `curation.json` merge proposals and `curation_review.json` are separate +producer/review artifacts; the new semantic event journal must not overwrite +either file. Re-running `prepare-large` after `apply-curation` may reuse the same +source spike/waveform bytes because curation changes assignments, not spike +identity. + +For the first editable release, support merges and labels only. Keep all splits +blocked until their specific replay rules pass end-to-end tests. Allocate IDs +from the complete unit catalog, including units absent from the subset. Record +resolved inherited metadata, rather than recomputing it from full parent counts. +Commit journal, assignments and metadata as one validated save generation; +reopen must recover a coherent generation and allocator state. Validate operation +intent before mutation, including plugin and console entry points. + +## 8. Test/reference fixture and future acceptance + +`phy/apps/tests/large_recording_fixtures.py` deterministically writes a minimal +32-spike, **1000 h** full Phy-format sorting and a bounded eight-spike package, +including upstream waveforms for eight different source rows and remapped +bounded waveforms for five. The fixture intentionally omits the raw binary. +`phy/apps/tests/test_large_recording_contract.py` asserts: + +- exact `int64` sample indexing well past the 32-bit limit, source-order and + time preservation; +- full -> bounded -> waveform mapping, including copied waveform/channel rows; +- existing Phy waveform filenames, row/dtype/shape and `-1` padding invariants; +- simple window array dtypes, sample-coordinate bounds and shared row alignment; +- source fingerprint agreement, separate source-vs-bounded identity; +- empty/disjoint intersections and existing phylib load compatibility. + +**These are reference contract tests only:** they do not claim that +`prepare-large`, Unit Stability or semantic replay already exists. Subsequent +PRs should reuse the fixture but replace the reference remapping with tests of +the production command. Add mismatched-source, reordered-row, excessive-memory, +partially covered units, gap, sparse feature and replay/undo corruption tests +before enabling bounded curation on real datasets. + +## 9. Ownership and implementation order + +1. **This PR:** Phy-side design, source-coordinate/format reference fixture and + tests only. No GUI changes, no new CLI, no upstream exporter changes. +2. **Unit Stability:** full-data summaries and selected-unit GUI view, initially + usable on normal recordings. Keep user-visible full/partial provenance clear. +3. **Semantic curation journal:** integration with `Supervisor`, event semantics, + undo/save/reopen and blocking unsafe splits in bounded mode. +4. **`prepare-large`:** separate streaming batch CLI, input validation, window + selection, output/remapped waveform files and bounded-manifest generation. +5. **`apply-curation`:** validate/replay semantic actions against full local + sorting, write a new curated full sorting, verify, then regenerate a bounded + package. Test deterministic repeatability and failures atomically. + +The integration plan now organizes these stages as commits within a single PR, +rather than separate PRs. **Do not ship an editable bounded GUI without replay safety**. + +Agent starting points: `AGENTS.md`, `design/README.md`, +`phy/apps/__init__.py` (Click CLI), `phy/apps/template/gui.py` (controller), +`phy/apps/base.py` (views/rates/save), `phy/cluster/{clustering,supervisor}.py` +(actions/history), `phy/apps/template/tests/test_gui.py`, and +`phy/apps/tests/test_large_recording_contract.py`. The existing waveform +reader is in the separate `phylib/io/model.py` dependency. + +## 10. Decisions not frozen by this Phy-side PR + +The upstream team should review the exact optional window-sidecar names, AIND +sorting asset identity/fingerprinting, segment/time/recording-block semantics, +waveform extraction alignment, total storage budget, and whether windows or a +more general sampling-plan format fits its pipeline best. Phy should accept the +existing three-file waveform format regardless. Values such as `1 h`, `30 s`, +`2,000` and `20,000` are preliminary, not schema requirements. + +Code Ocean handles **creation of a reusable evidence reservoir**; Phy handles +**bounded selection, GUI coordinate remapping, full-duration summaries, +semantic curation and replay**. Neither side should infer an absent waveform +or silently redefine source spike IDs. + +## Implementation references + +- [Phy `Clustering.assign`](../phy/cluster/clustering.py) and + [Supervisor](../phy/cluster/supervisor.py) for current row-based edits. +- [Template GUI](../phy/apps/template/gui.py) and + [Waveform/Feature mixins](../phy/apps/base.py) for standard stored-waveform + support and full/proxy metric safeguards. +- [Existing stored-waveform GUI test](../phy/apps/template/tests/test_gui.py). +- [SpikeInterface `export_to_phy` source (AIND inspected commit)](https://github.com/SpikeInterface/spikeinterface/blob/896ea840da72b4a5f06dcf8870ad820d20fe7031/src/spikeinterface/exporters/to_phy.py). diff --git a/design/large-recording-integration-plan.md b/design/large-recording-integration-plan.md new file mode 100644 index 000000000..de9e7043d --- /dev/null +++ b/design/large-recording-integration-plan.md @@ -0,0 +1,283 @@ +# Long recording audit and integration plan + +## Assessment + +The downloaded contract bundle is a useful foundation. Keep the bounded GUI, +immutable full sorting, explicit source-row mapping, and full-data replay model. +Do not enable editing until replay and persistence agree with the GUI. A 1000-hour +timestamp fixture validates coordinate range; it does not validate capacity or +scientific representativeness. + +This review targets phy `7e439491` and the downloaded +`phy_large_recording_contract_bundle.zip`. The accompanying contract remains a +proposal. This plan refines its implementation order and supplies additional +requirements; the phases below govern enablement. + +## Findings that affect correctness + +1. **Identity needs two layers.** A spike-coordinate fingerprint identifies + immutable rows and their time/channel/feature provenance. An assignment + revision identifies the starting full cluster assignments and metadata. + Replay must validate both. A times-only hash cannot distinguish recordings + with identical timestamps or changes to feature values. Include the acquisition + identity, time base, relevant immutable array digests and schema in coordinate + identity; hash assignments and metadata separately for each revision. Record + parent revision, journal digest, output revision and preparation identity. + Compute hashes incrementally, without `read_bytes()` on production arrays. + +2. **Cluster IDs must come from the full namespace.** + [Clustering](../phy/cluster/clustering.py) initializes its allocator from the + maximum loaded cluster ID. A subset can omit the highest full ID. Seed the + allocator from a full unit catalog and persist its high-water mark. Record + explicit output IDs in events and reject collisions during replay. Units with + no bounded evidence remain in the catalog and output, with missing evidence + visible to the user. + +3. **Save is a transaction.** + [BaseController](../phy/apps/base.py) currently saves assignments and metadata + separately; [Supervisor.save](../phy/cluster/supervisor.py) emits further save + events and clears dirty state. Adding another journal file to that chain does + not make it crash consistent. Bounded mode needs immutable save generations + containing assignments, metadata, journal and identity, published by a single + commit pointer. Reopen only complete validated generations. Preserve the last + valid generation after interruption; keep dirty state after a failed commit. + Define whether undo survives reopen. Recommended v1: reopen establishes a new + undo baseline while retaining the committed semantic history for replay. + +4. **Record resolved metadata.** `Clustering._do_merge` selects the largest + parent using loaded counts, and Supervisor inherits its metadata. Full and + sampled counts can choose different parents. Serialize the resolved group and + custom labels in merge events, including an explicit policy for conflicting + labels. Replay must not infer this choice from full counts. + +5. **Capture intent before mutation.** + [AmplitudeView.on_request_split](../phy/cluster/views/amplitude.py) returns + selected row IDs and clears the threshold. A listener after clustering cannot + recover the predicate. Route bounded edits through a command service before + any mutation. Gate direct assign/split, plugins, console operations, shortcuts, + merge proposals and threshold providers at the mutation boundary. V1 supports + whole-unit merges and labels; block every split until its full-data semantics + are implemented and tested. + +6. **Statistics require observation intervals.** Preserve shared contiguous + windows and segment boundaries. ISIs must not connect windows or acquisition + gaps. CCGs must exclude pairs crossing unsupported boundaries, and their + normalization must match observed lag exposure; the existing full-duration + baseline in `BaseController._get_correlograms_rate` is not valid for a window + subset. Initially suppress unsupported normalization and diagnostic claims. + Label window statistics as sampled evidence. Rescue selection and waveform + availability introduce bias; they do not justify extrapolating quality, + contamination, refractory violations or chronic unit identity to the full run. + +7. **Define time and channel coordinates.** V1 accepts one validated monotonic + sample axis with a fixed sampling rate and explicit valid acquisition + intervals. Reject unsupported segment layouts rather than concatenate silently. + Preserve tied timestamps and source row order. Sparse waveform channel IDs + index model channels; `channel_map.npy` maps these to raw-file columns. They + are not interchangeable with hardware or SpikeInterface channel IDs. Test a + nonidentity map, sparse padding, whitening/scaling and waveform alignment. + Keep sample coordinates as integers and CPU times as float64; subtract the + viewport origin before converting fine-scale plotting positions to float32. + +8. **All preparation products need budgets.** A dense units-by-time-bin summary, + exact quantile calculation, waveform gather, or full index inversion can + exceed RAM despite streamed input. Specify batch, output, summary, cache and + GUI memory limits, plus temporary disk estimates. Distinguish peak working + memory from file-backed pages. Whole shared windows exceeding the spike budget + must be shortened, rejected or replaced with documented deterministic rules; + do not silently thin spikes within a window and retain its statistical claims. + +9. **Persist the final window plan.** Store retained intervals, segment/block + identity, survey/rescue classification and valid exposure in the bounded + package. A seed and parameters do not identify trimmed or replaced windows. + Count acquisition exposure once when composing merged-unit summaries. + +## Module boundaries + +Use a small `phy/apps/large_recording/` package, with pure services callable +without Qt and thin Click/controller adapters. Refine filenames while coding; +keep these responsibilities separate: + +| Responsibility | Contract | +| --- | --- | +| Source adapter and validation | Stream canonical arrays; classify per-spike, template, channel and cluster data explicitly; reject ambiguous unsupported arrays | +| Manifest and identity | Validate versions, paths, coordinate identity, assignment revisions, exposure and capabilities | +| Preparation | Deterministic window plan, bounded output, streamed waveform remapping, budget report | +| Summaries | Full counts/exposure, availability, declared exact or approximate estimators and multiresolution queries | +| Semantic commands and journal | Operation validation, explicit IDs/metadata, history cursor and transactional commit | +| Replay | Pure event interpretation, chunked output assignments, invariants and lineage | +| GUI adapter | Capability gating, evidence labels, bounded model/controller integration | + +Do not load the complete source through TemplateModel or Clustering in batch +commands. Do not hide full-data scans in a selected-unit GUI callback. Limit +changes to phylib to demonstrated loader/coordinate fixes, with separate tests +and dependency requirements. Preserve existing ordinary-dataset behavior. + +## One implementation PR with ordered commits and acceptance gates + +Implement the complete workflow on `feature/long-recording-curation` in the +main phy checkout. Use the stages below as reviewable commit groups within one +PR. Intermediate capability gates support development; merge the PR only after +the complete prepare, curate, save, replay and regenerate workflow passes its +acceptance checks. Predicate splits require their own full-data equivalence +checks within the same PR if included in its supported functionality. + +### 1. Contract and reference fixtures + +Start with the revised contract, this plan and format fixtures. +Fix the bundle's missing `dtype` in `params.py`, allow equal sample times, and +reject malformed reference row IDs. Run the real released-phylib loader test +without accepting a skip. Its success proves this small file layout loads, +not that all GUI workflows or long-recording statistics are supported. + +### 2. Source validation and summary engine + +Implement identities, assignment revisions, array inventory, exposure and +capability validation independently of Qt. Start with full counts and rates; +add amplitude/depth only with defined units and available source attributes. +Do not add per-bin quantiles until their exact/approximate algorithm, merge +behavior and memory costs are specified. Medians cannot be merged by averaging +parent medians; invalidate or recompute unavailable derived summaries. + +Gate: exact tiny-oracle counts; missing exposure; gaps; source/feature mismatch; +reordered rows; column-vector input arrays; unknown versions; unsupported +segments; bounded peak memory as source size grows; actionable disk estimates. + +### 3. Read-only preparation and inspection + +Implement `prepare-large` with deterministic shared windows, rescue policies, +per-unit evidence reports and explicit budgets. Write into a temporary sibling +directory and publish only after complete validation. Rebuild paths in params, +subset every registered per-spike array, retain template-indexed structures, and +avoid copying stale `.phy` caches and derived cluster metrics. Unknown attributes +must be explicitly included, excluded with a report, or rejected. + +Add a bounded-manifest GUI adapter with authoritative duration, full/proxy +counts and unsupported-view safeguards. Until phase 6, require read-only bounded +sessions, including save and plugin mutation entry points. + +Gate: production command vs independently specified expected arrays; shuffled +channel mapping; actual `get_waveforms()` values; sparse PC feature alignment; +missing/partial waveform coverage; tied samples; zero evidence units; empty +selection policy; deterministic budgets; interrupted publication; no source +writes; GUI open/close and time-axis precision tests. + +### 4. Semantic commands and durable saves + +Implement merge/label command validation, full-catalog ID allocation, resolved +metadata, undo/redo active history and transactional generations. Keep the +feature disabled for editable bounded sessions while replay is incomplete. +Existing `curation.json` and `curation_review.json` retain their separate roles. + +Gate: merge chains; conflicting labels; undo then branch; redo; reopen baseline; +omitted high-ID units; proposal-driven merges; rejected direct split/assign; +failed-save dirty state; fault injection at every commit boundary; assignments +equal the journal's projection onto bounded rows. + +### 5. Full-data replay + +Implement `apply-curation` against an exact source assignment revision. For v1 +merges, resolve the final source-unit mapping once, then stream assignments into +a new full dataset. Preserve row count/order, times and immutable arrays; write +metadata and lineage coherently. Reject repeated application to the wrong parent +revision. Publish only a verified complete output. Reprepare from that revision +with the same coordinate identity and fresh revision-dependent summaries. + +Gate: compare every assignment with an independent tiny oracle; preserve all +unseen spikes; replay chains and labels exactly; reject mismatched parent or +corrupted journal; interruption never exposes a complete-looking partial output; +same source/events produce the same semantic result; source bytes unchanged. + +### 6. Enable bounded merges and labels with Unit Stability + +Expose editable bounded mode only when phases 3–5 pass together. Integrate a +Unit Stability view that distinguishes acquisition exposure, evidence windows, +available measurements and unresolved identity. Add bounded status and operation +capabilities to the existing workflow. Count/exposure summaries can update for +merges; quantiles and other nonadditive metrics require recomputation or explicit +unavailability. Keep all splits blocked. + +Gate: prepare → inspect → merge/label → undo/redo → save/reopen → replay → +reprepare → verify. Include ordinary 1–10 h datasets to prevent regressions. +Run `make test-full`, `make doc-check`, and `uv build` when CLI/package changes +are included. Complete manual smoke testing with an actual chronic dataset. + +### 7. Optional predicate splits + +Add one operation at a time after the initial workflow is accepted. Store feature +identity/units, predicate operator, threshold, missing-value policy, parent +revision and explicit child IDs. The same predicate must evaluate identically +on GUI evidence and all source spikes. Stored waveform-derived or local PCA +amplitudes do not automatically define a full-data predicate. Lasso splits +remain unavailable without a separately validated full-data rule. + +## Standard recording compatibility gate + +Recording length must never select a new behavior implicitly. Activate bounded +curation only through an explicit validated bounded manifest. Without that +manifest, retain the standard loader, selector, metrics, mutation capabilities, +save format and history behavior. A malformed bounded manifest must produce an +actionable error rather than silently enter standard editable mode. Optional +Unit Stability on ordinary recordings must not replace their existing views, +change their default layout or introduce full-source scans at GUI startup. + +Keep the semantic command, replay and transactional save implementation behind +the bounded adapter wherever possible. Any necessary shared-code change requires +focused standard-mode regression coverage in addition to bounded-mode tests. +Do not impose bounded split restrictions or journal files on ordinary datasets. + +Before merging the single PR, run `make test-full` and verify ordinary datasets +with raw traces, stored waveforms only, and sparse features. Cover standard +merge and label inheritance, lasso/threshold/template splits, undo/redo and +branching, save/reopen assignments and metadata, merge proposals, plugin-driven +edits, channel mapping, sampled views and existing rate/ISI/CCG behavior. Assert +that ordinary saves do not create bounded journals or manifests. Compare startup +memory and selection latency on the same ordinary datasets before and after the +change; any meaningful regression must be explained and resolved before merge. +Existing tests and these checks establish compatibility evidence; the plan +itself does not guarantee compatibility before implementation. + +## Capacity and scientific acceptance + +Use three kinds of evidence: tiny exact-oracle datasets for correctness; +generated high-density on-disk datasets for resource behavior; real chronic +recordings for usability and scientific acceptance. Duration, spike count, +number of units, array widths and available waveform coverage are separate axes. +Measure preparation/replay throughput, peak process memory, temporary disk, +GUI startup and selected-unit update latency on documented hardware. Establish +acceptance budgets from those measurements before advertising capacity. + +Have a domain curator review known drift, rare/bursting units, transient +artifacts, gaps and intentionally incorrect merges. Compare selected evidence +with withheld contiguous windows and full-data diagnostics where available. +Record what the curator can and cannot infer; finite selected evidence cannot +guarantee biological identity throughout 1000 hours. Software integrity and +scientific evidence sufficiency are separate release criteria. + +## Producer decisions + +Agree with the producer team on source asset identity, full unit IDs, authoritative +duration/exposure, segment mapping, waveform time/channel/amplitude semantics, +reservoir coverage and total storage costs. Existing three-file waveforms remain +supported; proposed window sidecars must remain optional until jointly agreed. +The inspected [SpikeInterface exporter revision](https://github.com/SpikeInterface/spikeinterface/blob/896ea840da72b4a5f06dcf8870ad820d20fe7031/src/spikeinterface/exporters/to_phy.py) +supports one segment and distinguishes Phy channel indices from exported original +channel IDs. Do not assume that arbitrary multi-session inputs already satisfy +this contract or that a deployed exporter matches this inspected revision. + +## Implementation and validation status + +The branch now includes batched source validation and preparation, full-source +summaries, bounded GUI curation and checkpoints, semantic replay, Unit Stability, +and dedicated long-recording tests. The feature is implemented, but its +acceptance remains incomplete: this test and synthetic benchmark evidence does +not replace real chronic-recording review by domain curators. + +On 2026-10-13, `make test-full` completed with 745 passed and 1 skipped. Three +additional bounded GUI lifecycle regressions pass in +`phy/apps/large_recording/tests/test_gui_lifecycle.py`. +[`design/large-recording-validation.md`](large-recording-validation.md) records +the synthetic capacity runs and their limits. The separate 32-spike, 1000-hour +fixture verifies sample-coordinate handling and file mapping; it is not a +billion-spike capacity test or evidence of chronic biological identity. Curator +acceptance on real chronic recordings remains outstanding. diff --git a/design/large-recording-time-coordinates.md b/design/large-recording-time-coordinates.md new file mode 100644 index 000000000..3c04bc9bf --- /dev/null +++ b/design/large-recording-time-coordinates.md @@ -0,0 +1,244 @@ +# Event coordinates and acquisition exposure + +This document defines the narrow time-coordinate extension for the AIND export +reported against PR #1428. It supplements the +[large-recording contract](large-recording-curation.md) and its +[integration plan](large-recording-integration-plan.md). It does not authorize +changes to an upstream export or establish that this particular chronic +recording is ready for curation. + +## Supported coordinates + +Phy-facing event times are nonnegative integer coordinates on one monotonic +axis. Two explicitly distinguished meanings are supported: + +- `acquisition_samples`: sample indices on the acquisition axis supported by the + original contract. +- `elapsed_ticks`: elapsed seconds from the source recording's time origin, + multiplied by a declared tick frequency and quantized to integers. Gaps remain + visible on this axis. A tick is a time coordinate, not a recorded voltage + sample. + +The optional producer declaration is a Python dictionary in `params.py`: + +```python +sample_rate = 30000.0 +waveform_sample_rate = 30000.0 +time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': 30000.0} +num_samples = 12188787510 +# Illustrative values only; the producer must supply authoritative exposure. +valid_intervals = [[0, 100000], [200000, 12188787510]] +``` + +The same declaration can occur as `recording.time_base` in +`_phy_spikes_subset.manifest.json`. If both locations declare it, they must +agree. `kind` is one of the two strings above; `ticks_per_second` is positive +and finite. Unknown kinds and conflicting declarations are errors. + +For this implementation, `ticks_per_second`, `params.py`'s `sample_rate`, +`waveform_sample_rate` when declared, and the waveform manifest's +`recording.sampling_frequency_hz` when declared must be equal. This keeps the +existing loader and waveform time axes correct for the reported 30 kHz export. +Supporting unequal event tick and physical waveform sample frequencies requires +separate waveform-axis plumbing and is outside this extension. + +The physical and event frequencies nevertheless have different meanings. In +elapsed mode, `sample_rate` converts event ticks to seconds; the waveform +frequency describes the spacing between samples inside each stored waveform. +Clock drift or a piecewise clock map must not stretch the waveform's local +sample axis. + +Without an explicit declaration, retain the original acquisition-sample +behavior and identity representation for existing sources. A source containing +`timestamps.json` must provide an explicit time-base declaration before this +bounded workflow accepts it. This is an intentional conservative admission +rule: the file's presence signals acquisition metadata whose exposure semantics +have not been established by the legacy defaults. It is not proof that every +such source actually has mixed coordinates. The error must explain how to +declare authoritative coordinates and exposure; it must not parse the JSON and +guess a conversion. + +## Bounds and exposure + +For both kinds, the legacy names `num_samples` and `valid_intervals` describe +the canonical event axis: + +- `num_samples` is its exclusive positive integer upper bound. In elapsed mode + it is a tick extent, not the number of acquired frames. +- `valid_intervals` contains ordered, disjoint, nonempty integer `[start, end)` + intervals inside that extent. They declare valid acquisition exposure on the + same event axis. +- Every spike belongs to a valid interval. Time ties and original source row + order remain allowed and preserved. +- Exposure in seconds is the sum of interval lengths divided by the tick + frequency, counted once across units. Gaps and filler are not zero firing. + +An explicit elapsed-time source needs authoritative exposure rather than a +silent `[0, num_samples)` default. The producer must exclude missing acquisition +time and filler that does not contain usable recording data. If sorting spikes +remain in excluded intervals, reject the source with an actionable error. Do +not silently drop, move or relabel them: these actions change source row +identity or scientific interpretation. A future flagged-spike policy needs its +own full-data counting, selection and replay contract. + +The upper bound must include the producer's final representable event tick. +An endpoint mapping may describe the last sample rather than an exclusive end; +do not blindly substitute its rounded clock span for `num_samples`. A physical +pipeline frame count can be retained as separately named producer provenance, +for example `recording.num_frames`, but cannot also define the elapsed extent. +`--duration` remains a consistency check and does not override conflicting +producer metadata. + +Reject a first or last source spike outside the declared extent before a full +validation/hash scan. Still perform the complete batched validation for internal +reversals, out-of-range rows and invalid exposure membership. The early check +is a diagnostic improvement, not a replacement for validation. + +## Waveform and window sidecars + +In explicit elapsed mode, the waveform manifest must declare the coordinates of +each present time-bearing sidecar: + +```json +{ + "format": "phy_spikes_subset", + "format_version": 1, + "recording": { + "sampling_frequency_hz": 30000.0, + "num_samples": 12188787510, + "time_base": {"kind": "elapsed_ticks", "ticks_per_second": 30000.0}, + "window_bounds_coordinates": "elapsed_ticks", + "sample_index_coordinates": "elapsed_ticks" + } +} +``` + +The example omits `valid_intervals`; a real elapsed export must declare the +authoritative intervals in `params.py` or `recording`, with agreement when both +declare them. `window_bounds_coordinates` is required when window bounds are +present; `sample_index_coordinates` is required when sample indices are +present. Their values equal `time_base.kind`. A contradictory +`recording.frame_coordinates` declaration is an error, not an invitation to +reinterpret an array. Existing producer prose describing frames before +clock conversion must be corrected explicitly for an elapsed export. + +The sidecars have these meanings and validation obligations: + +- `_phy_spikes_subset.window_bounds.npy`: `int64[W, 2]` event-coordinate + `[start, end)` intervals, ordered by start, within the declared extent and + valid exposure. Overlapping windows are unioned without double counting; + acquisition block boundaries remain distinct. A window cannot bridge an + unsupported interruption. +- Optional `_phy_spikes_subset.window_types.npy`: integer `[W]`, conventionally + `uint8`, with `0` survey and `1` rescue; absent types default to survey. +- Optional `_phy_spikes_subset.window_strata.npy`: integer `[W]`, conventionally + `int32`, nonnegative survey stratum IDs and `-1` for rescue. Supplied arrays + align with the window bounds; absent strata receive deterministic defaults. +- Optional `_phy_spikes_subset.sample_index.npy`: integer `[N_waveforms]`, aligned + with waveform rows, containing canonical event-center coordinates. For every + waveform row `w`, it equals + `spike_times[subset_spikes[w]]`. It is not a source spike row ID or a raw-file + sample offset. + +The established waveform triplet remains authoritative for actual waveform +availability. A window denotes potential evidence, not a waveform for every +spike in it. When sidecars are supplied as producer evidence, waveform centers +must lie in the declared window union. Reject malformed sidecars rather than +falling back to an independently generated plan that hides the error. + +The existing row spaces remain unchanged: full spike row `g`, bounded spike row +`b`, waveform row `w`. Remap waveform row IDs through the selected full-to-bounded +mapping, and gather sample-index, waveform and channel rows together. Replay +preserves the original full arrays byte for byte. Supporting a sample-index +sidecar that intentionally remains in pipeline-frame coordinates is a separate +future provenance extension; this implementation does not infer its mapping. + +Preparation should prefer validated producer evidence windows while keeping +whole windows, the configured spike/byte budgets, and deterministic omission +reporting. Do not enlarge memory or disk budgets merely because many waveforms +were exported. Report the source, retained and omitted waveform counts so a +small bounded package cannot appear to preserve the complete reservoir. + +## Raw voltage access + +Elapsed event ticks must never flow into raw voltage indexing. This first +extension rejects an elapsed source with a real raw-data path until a separate +explicit event-to-raw-frame mapping and its validation exist. Absent raw paths, +including an export's literal `None` sentinel, must be recognized consistently. +The bounded package continues to use stored waveforms without raw data. + +Do not replace raw indices with elapsed ticks in an exporter and then claim +ordinary raw-trace loading works. Converting waveform centers for this contract +does not convert the physical voltage stream or establish a reversible map. + +## Identity, replay and metadata preservation + +Explicit time-base metadata participates in coordinate identity alongside the +canonical extent, exposure, immutable arrays, channel geometry and existing +acquisition provenance. Changing the coordinate kind, tick frequency or +exposure must invalidate a bounded package prepared against different metadata. +Keep the legacy identity shape unchanged for sources without a declaration. + +Carry explicit metadata into bounded `params.py` and the bounded manifest. +Validate it when reopening and replaying, rather than trusting a free-form field +that is never consumed. Preserve it in complete replay outputs and their +lineage. Repreparing a replay output must produce the same coordinate identity. +Assignment revisions remain distinct from that identity. Coordinate changes +cannot be repaired by retaining a journal against an obsolete source identity. + +Optional upstream sampling-policy windows do not by themselves change immutable +spike identity, but their validation and package hashes must bind the exact +evidence that the GUI uses. A waveform-aligned sample-index array does carry +immutable row provenance and must be included in the corresponding hashes and +copy/gather paths. + +## Why Phy does not convert the supplied timestamps JSON + +The supplied file contains 420 frame/clock endpoints for 210 pieces. Frame +endpoints increase strictly; clock endpoints are nondecreasing with 15 equal +adjacent pairs. Its metadata additionally reports underlying backward timestamp +steps that are not represented by those endpoint pairs. Consequently, the +endpoint map does not establish an invertible or sample-accurate raw clock. + +Interpolation can reproduce the reported exported spike ticks without proving +the exposure between endpoints. Piecewise dropped samples, filler and true +gaps need producer knowledge. Mapping just the endpoints of a window that +crosses a gap would incorrectly declare the gap observed. The reported +`total_dropped_within_pieces_s` also does not locate every unsupported interval. +Neither the apparent 30 kHz unit nor monotonic exported spikes makes elapsed +ticks equivalent to physical acquisition samples. + +The producer owns clock conversion and the final validity mask. Phy validates +the resulting explicit contract. Acquisition logs remain necessary to resolve +the reported interruptions and the 35 sorting spikes inside filler before +declaring this export suitable for exposure-aware statistics. + +## Regression and acceptance gates + +Before considering the extension complete, require concrete checks for: + +1. Unchanged legacy preparation, identity, summaries, GUI behavior and replay; + explicit acquisition-sample sources also round-trip. +2. Explicit elapsed data with tied spikes, long `int64` coordinates, gaps and + endpoint boundaries; exact independently calculated full counts and exposure. +3. Missing/conflicting time declarations, unsupported rates, raw paths, + contradictory frame metadata, and first/last out-of-bounds spikes; errors + identify the conflicting coordinate meanings before the expensive scan. +4. Spikes in filler/excluded exposure, internal reversals and invalid interval + boundaries; none are silently dropped or accepted as valid exposure. +5. Malformed producer sidecars, policy labels without bounds, invalid + dtypes/lengths, overlapping-window union without double counting, windows + crossing gaps, waveform centers outside windows, and sample-index values + inconsistent with their source spike rows. +6. Budget-constrained producer-window selection with exact retained/omitted + waveform reporting and deterministic complete-window selection. +7. Prepare, merge/label, save/reopen, replay and reprepare preserving explicit + metadata, immutable rows and waveform alignment. Mutating time metadata or + sample-index provenance must invalidate the appropriate identity/checksum. +8. Window-local ISI/CCG calculations excluding unsupported cross-boundary pairs, + physical waveform sample spacing, and no elapsed-coordinate raw indexing. + +These establish software behavior. They do not establish that endpoint-derived +exposure is scientifically precise or that the finite waveform evidence is +sufficient for chronic unit identity. Domain-curator acceptance remains required +by the existing validation plan. diff --git a/design/large-recording-validation.md b/design/large-recording-validation.md new file mode 100644 index 000000000..afcee5825 --- /dev/null +++ b/design/large-recording-validation.md @@ -0,0 +1,114 @@ +# Long-recording implementation validation + +This record summarizes the current implementation checks and their limits. The +results validate the measured code paths and fixtures described here. They do +not establish capacity for every source layout or scientific suitability for +chronic curation. + +## Automated checks + +After the real-data split-journal fix described in the +[AIND diagnostic record](aind-real-data-validation.md), `make test-full` passed +**821 tests with 1 existing benchmark skip** (496 core and 325 application +tests). Lint, formatting, documentation checks, and package build passed. + +On 2026-10-24, the AIND time-coordinate follow-up passed `make test-full`: +**818 passed, 1 existing benchmark skip** (496 core tests and 322 application +tests). The environment used Python 3.12.3, released phylib 2.7.1, Linux, +Xvfb, software OpenGL, and Qt's offscreen platform. The target included lint +and formatting checks. The strict documentation check and `uv build` also +passed; the wheel contains all 12 long-recording runtime modules. + +The follow-up has a 96-test focused source/preparation/replay run and an +additional GUI regression for explicit elapsed clocks. Its independent +discontinuous-clock oracle verifies per-bin exposure and unit counts, exact +waveform sample-index alignment, merge assignments, save/reopen, replay, +repreparation, timestamp provenance, and unchanged source bytes. The GUI test +opens all default bounded views, checks exposure-based firing rate and window +navigation, and confirms that elapsed ticks do not enable raw traces. All of +these tests are included in the full-suite result above. + +Separate architecture, correctness, and final audit reviews used Sol at high +reasoning effort. The correctness review found a legacy coordinate-name +mismatch and a generated-policy bypass of producer waveform-window membership +validation; both received regression coverage and were corrected. The final +independent audit found no remaining blocking defect in the reviewed changes. + +These results validate the [explicit coordinate contract](large-recording-time-coordinates.md), +not the supplied export's acquisition truth. During that initial follow-up the +complete export and its acquisition logs were unavailable; its spike-array and +filler-spike claims were attributed to Carl's report. A different computation +was subsequently downloaded and tested as an explicitly modified engineering +derivative, as recorded in the AIND diagnostic above. No exporter repository +was identified or modified. The +[exporter handoff](aind-export-handoff.md) records the upstream work and real-data +acceptance evidence still needed. + +On 2026-10-13, `make test-full` completed with **756 passed and 1 +skipped** (496 non-application tests and 260 application tests). Final channel +geometry and nonidentity waveform-mapping additions passed a further focused +source/preparation run of **33 tests**. Lint, format checking, package build, +and the strict documentation build also passed. The dedicated bounded GUI lifecycle file passed all three +regressions: reentrant observer rollback with the GUI attached, merge-proposal +review through undo/redo/save/reopen, and pending full metrics for threshold +children in the default bounded views. + +The reference long-recording fixture is deliberately sparse: 32 spikes spread +over a 1000-hour sample clock. It checks long `int64` coordinates, tied times, +source-row mapping, waveform remapping, and loader compatibility. It is not a +billion-spike capacity test, a realistic chronic recording, or evidence of +biological identity across a recording. + +## Synthetic preparation and replay runs + +The reproducible benchmark is `uv run python +tools/benchmark_long_recording.py`. It writes synthetic source arrays in +batches, with no raw voltage data, and runs preparation and replay in separate +processes. Both runs below used Python 3.13.3, NumPy 2.4.3, Linux +`x86_64`/glibc 2.39, 30 kHz sampling, 1000 units, a 1000-hour sample axis, +100,000-row batches, 30-second windows, 3600-second strata, a 100,000-spike +budget, a 512 MiB output budget, seed 0, and 3600-second summary bins. The +oracle independently streamed every source and output assignment row and +matched the expected merge in both cases. + +| Synthetic source spikes | Source files | Bounded package | Bounded spikes | Curated output | Prepare time / peak RSS | Replay time / peak RSS | Oracle rows | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 250,000 | 4,032,892 B | 8,625,460 B | 2,389 | 5,038,086 B | 1.902 s / 215,773,184 B | 0.114 s / 214,847,488 B | 250,000 | +| 2,000,000 | 32,032,892 B | 9,157,120 B | 16,676 | 40,038,086 B | 1.227 s / 214,433,792 B | 0.292 s / 215,285,760 B | 2,000,000 | + +Each measurement is a single run, not a throughput curve. Preparation and +replay durations exclude child-process startup; peak RSS is the process high +water mark. The source is a regular synthetic distribution with tiny templates +and no stored waveforms, raw data, irregular acquisition gaps, or real feature +loads. These figures therefore do not predict every preparation cost or GUI +interaction on a chronic dataset. The full JSON reports are retained at +`/tmp/phy-long-recording-benchmark-250k.json` and +`/tmp/phy-long-recording-benchmark-2m.json` on the validation workstation. + +## Ordinary-dataset smoke comparison + +A separate three-run comparison used fresh subprocesses, configs, and caches in +the same Python environment. It compared baseline `a30d5c65` with current +`f212f069e5b5b19469cfa73781dac08de093d9a1` on a 12,000-spike, two-cluster +dataset with 1,024 stored waveform rows of shape `[40, 4]` and dense template +features. Counts and output dimensions matched exactly. + +| Measure (median of 3) | Baseline | Current | +| --- | ---: | ---: | +| Controller startup | 0.03879 s | 0.02822 s | +| Waveform provider | 0.00353 s | 0.00293 s | +| Feature provider | 0.00372 s | 0.00302 s | + +This small case showed no regression; the apparent gains are within the range +where run-to-run noise matters. The comparison did not measure Qt rendering, +raw-voltage I/O, or large-dataset selection churn. Its report is retained at +`/tmp/phy-standard-recording-comparison.json` on the validation workstation. + +## Remaining acceptance work + +Real chronic-recording review by domain curators remains outstanding. It must +cover drift, rare and bursting units, artifacts, gaps, waveform coverage, +intentionally incorrect merges, withheld contiguous windows, and full-data +diagnostics where available. Software integrity checks and synthetic throughput +do not establish that the evidence is scientifically sufficient or that a +cluster represents one biological unit throughout 1000 hours. diff --git a/design/slay-roundtrip-validation.md b/design/slay-roundtrip-validation.md new file mode 100644 index 000000000..7af25b863 --- /dev/null +++ b/design/slay-roundtrip-validation.md @@ -0,0 +1,84 @@ +# SLAY persistence validation + +The follow-up to PR #1428 preserves producer `curation.json` bytes and the +complete source-bound review ledger through bounded preparation, committed +save, full replay, and re-preparation. Suggestions retain original IDs; pending +suggestions affected by merges or threshold splits become stale. Historical +accept/reject decisions take precedence over that derived status. + +## Integrity and regression coverage + +Preparation hashes the producer document and an immutable inherited review +snapshot. Session loading uses committed checkpoint reviews before loose review +copies. Direct saves and saves with the proposition feature disabled preserve +the ledger; suggestions without decisions receive an empty bound ledger. +Generation extra documents must belong to the checkpoint hash inventory. +Replay stages both documents, checks copied bytes, and publishes hashes and +cumulative merge/split retirement records in lineage. + +Focused regressions cover three no-op rounds in direct, enabled and disabled +modes; accepted, rejected and orphan history; cumulative merge/split retirement; +malformed, conflicting or absent loose copies; mutated bound documents; missing +or null checkpoint ledgers; and mutations during copying. Old bounded packages +with unbound proposition artifacts fail explicitly instead of dropping history. +Ordinary unbounded loading keeps its existing compatibility behavior. + +A separate Sol-high read-only audit identified copy-time mutation, legacy +silent-drop and null-ledger bypasses. Each received a fix and regression; the +final audit reported no remaining blocking findings. + +## Real-data fixture and limits + +The fixture is Carl's converted overlay of computation +`b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`, with 469,725,567 spikes, +705 original units, 140,240 waveform rows and 93 suggestions. It preserves the +33 filler-edge spikes using Carl's test-only exposure intervals. These intervals +have not been established as acquisition truth; this is an engineering test, +not acceptance of Brandon's native exporter. See +`aind-real-data-validation.md` for the original coordinate mismatch and exposure +limitations. + +The roundtrip uses Carl's explicit 8,000,000-spike / 3,000,000,000-byte caps to +retain all waveform rows. Through the ordinary proposition-review and merge +path, it accepts `[40,44,48] -> 705` and `[49,50,51,53] -> 706` and marks both +results good. Subsequent rounds perform no additional curation. + +The acceptance oracle requires two accepted and 91 pending suggestions in each +round, identical suggestion bytes and review records, save/reopen consistency, +and an independent streamed comparison of every full-source assignment against +the two original merge groups. Small regression fixtures provide overlapping +stale-suggestion coverage, because Carl's remaining 91 suggestions are disjoint +from these groups. + +The separate validator run on this same test overlay passed with default +2,000,000-spike / 536,870,912-byte caps: 1,486,361 evidence spikes, 27,865 of +140,240 waveform rows retained, and 532,625,749 output bytes. It records that +the exporter revision is Carl's test conversion and that acquisition truth is +unestablished. A real native export and authoritative gap/filler policy are +still required; see `aind-native-export-acceptance.md`. + +Local reports and reproduction scripts are retained under +`/home/cyrille/GIT/IBL/phy-data/carl/aind-b1b0782a-b3ea-4f34-81d7-b3c6e36452b3/slay-validation`. +Large arrays and access credentials are not repository artifacts. + +## Results + +- All three rounds retained exactly two accepted and 91 pending suggestions, + identical producer bytes and review records, and matching assignments for + all 469,725,567 source rows. Each bounded package retained all 140,240 + waveform rows and 7,298,318 evidence spikes at Carl's larger caps. +- The independent hash oracle verified all 67 original files were unchanged + and 12 immutable files per full replay matched the converted source. Both + good labels and the seven retired parent records survived every round. +- A separate final-code controller with proposition loading disabled saved and + reopened the real third-round checkpoint with both accepted reviews intact. +- `make test-full` passed: 854 tests (499 core, 355 applications), with one + existing benchmark skip. Lint and formatting passed. The corrected disabled + controller tests subsequently passed all 22 persistence cases. `make doc-check` + passed; packaging was unchanged. +- The combined three-round script completed every functional assertion and + wrote its success report, then exited with signal 11 during process shutdown. + A separate repeated six-controller/three-GUI lifecycle run with event + processing and reference cleanup exited successfully. The combined harness's + shutdown fault is retained as a limitation; its cause is not established as + a product defect or a data-integrity failure. diff --git a/design/unit-metadata-roundtrip-validation.md b/design/unit-metadata-roundtrip-validation.md new file mode 100644 index 000000000..140d25256 --- /dev/null +++ b/design/unit-metadata-roundtrip-validation.md @@ -0,0 +1,111 @@ +# Unit metadata and display feedback: seven-item acceptance + +This follow-up implements Carl's three-page **PR1428 - unit metadata across +curation rounds.pdf**, received 25 October 2026. Changes remain in PR #1428's +`feature/long-recording-curation` branch. No phylib or exporter change is needed. + +| PDF item | Behavior and evidence | +| --- | --- | +| 1. Source IDs and filenames | Schema descriptors preserve source basenames, CSV/TSV delimiters and column ordering. `cluster_si_unit_ids.tsv` remains plural with header `si_unit_id`; new merge/split IDs have blank scalar source IDs. Complete parent metadata is retained in history. | +| 2. Stale pipeline metrics/classifications | Membership-policy events clear imported fields and quality on new membership. Explicit new-unit labels survive unchanged rounds; prior decisions remain historical. GUI, journal, checkpoint and export use the same projection. | +| 3. Parameter-file growth | One final managed override block replaces itself; only recognizable old generated suffixes are removed. Original Python code, comments, CRLF, encoding cookies and BOMs survive. Raw-data sentinel strings normalize to Python `None`. | +| 4. Eight-unit cap | Default 32, saved nonnegative setting; zero shows all and any positive integer is accepted. Display IDs and cache capacity are independent; truncation never changes merge membership. | +| 5. Legend | Removed outright from every metric axis. Authoritative cluster/Merge colors are retained. | +| 6. Camera action | Unit Stability captures its actual Qt FigureCanvas, retaining the usual screenshot path/naming/return contract. Regression tests trigger the screenshot action and inspect the PNG. Real capture also calls that same view method. | +| 7. Rate readability | Default saved width 0.6 points. Dataset-local display bins default to native resolution and permit aligned coarsening. Counts/exposure and finite measurement sums/counts add before division; zero exposure remains unavailable and final short edges are retained. | + +## Metadata compatibility and policy + +Imported values remain unchanged for untouched units, including legacy empty +strings and typed JSON values. New producer values are blank even when parent +values agree. New memberships are unreviewed; an explicit `705 = good` applies +to unit 705 until it is transformed again. No producer SNR, classifier, or +violation metric is recomputed or copied from a preferred parent. + +Layout descriptors validate filenames, columns, duplicate fields and collisions. +Typed JSON/table disagreement fails with file, field and unit context. Read-only +source aliases may be symlinks; output writes refuse symlink targets. Legacy +omitted-null rows and Python scalar representations remain readable, and old +blank values retain their canonical revision hashes. + +Version-1 event meaning is preserved. `BoundedSession.migrate_metadata()` records +an explicit policy transition, recovering source names from the exact original +source identity and revision. Saving creates a new immutable generation without +changing the base manifest, assignments, allocation IDs or review ledger. It +clears ambiguous resolved values on active derived units while retaining explicit +label journal intent. Already exported values without their earlier journal +cannot establish curator intent; they remain imported values rather than being +retroactively called explicit decisions. The guide includes migration usage. + +## Real-data acceptance + +The fixture is Carl's earlier converted engineering test export, computation +`b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`: 469,725,567 spikes, 705 original +units and a 112.859-hour elapsed clock. Its exposure declaration remains a test +conversion, not independently verified acquisition truth. Carl's newly mentioned +500-hour export was not available locally. + +Fresh outputs exercise: + +1. Accept `[40,44,48] -> 705` and `[49,50,51,53] -> 706` through the GUI + proposition workflow; assert producer fields and quality clear before explicit + good labels and a custom curator note are recorded. +2. Re-prepare, split 705 into 707/708 by a source-amplitude predicate, assert new + membership values clear, then explicitly label both children. +3. Re-prepare, merge 707/708 into 709, assert clearing again, then label 709. + +The independent Qt-free oracle checked every one of the 469,725,567 assignments +in **each round**, all unchanged source metadata cells, blank producer cells, +explicit group labels, and the two accepted review records. All 45 original +table filenames survive. The 91 remaining propositions remain pending. Params +are exactly **892 bytes in all three outputs**, with one managed block. +History grows from 5 to 8 to 11 records rather than losing parent decisions. + +The same oracle verified all **67 original download files** were unchanged and +**12 immutable files per output** matched the converted source. A separate copy +of the earlier reviewed bounded package was migrated: its manifest, assignments +and accepted reviews were unchanged; ambiguous producer values became blank and +explicit good labels on 705/706 survived. The data-only process exited zero. + +The combined GUI/data harness completed all assertions and printed PASS, then +exited 143 (SIGTERM). The signal's source is unknown; no traceback or verified +product termination path was found. This is retained as a combined-harness +limitation, not described as a clean process exit. Independent data verification, +the test suites, and both standalone capture processes exited normally. The +original independent checker also needed a harness-only fix to flatten column +cluster arrays before combining them with scalar amplitude masks; saved outputs +were unchanged and all rows were subsequently rechecked. + +## Display and regression checks + +Refreshed real captures use units 24, 26, 27 and 32, with no legend and 0.6-point +rate lines. The actual camera image is included in +`docs/images/unit-metadata-refinements/`. A generated **513-hour, twelve-unit** +fixture uses 6156 native five-minute bins and exact ten-minute display coarsening +to 3078 bins. It is synthetic stress coverage, not Carl's 500-hour data. + +Focused coverage includes metadata aliases/multi-column CSVs, typed values, +safe names and conflicts, blank legacy revisions, immutable old generation +reopen, migration identity, snapshot saves, policy event versions, provenance, +undo/redo, encoding/BOM/CRLF preservation, all display settings, actual camera +capture, cache limits, nonuniform-grid rejection, zero-exposure groups, merged +weighted means and final partial bins. + +Local verification ran `make test-full` (499 core + 431 application tests, one +existing benchmark skip), then the application target after two added symlink +regressions (433 passed). The final parameter/replay run passed 23 cases, +including two added encoding regressions. The final metadata run passed 24 cases, including two added structured-label +serialization regressions. This covers 936 distinct current tests; full CI runs the final tree together. Lint and format checks passed. +Documentation and wheel checks are recorded with the final handoff. + +`make doc-check` and `uv build` passed. The built wheel contains both new +runtime modules (`metadata.py` and `parameters.py`) with bytes matching the +working source. All refreshed image files were visually inspected. Source +symlink regressions skip only the specific Windows missing-privilege condition; +they run normally on systems supporting symlink creation. + +Local reports/scripts are under +`/home/cyrille/GIT/IBL/phy-data/carl/aind-b1b0782a-b3ea-4f34-81d7-b3c6e36452b3/pdf-metadata-validation/`. +Captures are in sibling `pdf-refinements-screenshots` and +`pdf-refinements-synthetic-513h` directories. Arrays and credentials are not +repository artifacts. diff --git a/design/unit-stability-comparison-validation.md b/design/unit-stability-comparison-validation.md new file mode 100644 index 000000000..a08b07fb6 --- /dev/null +++ b/design/unit-stability-comparison-validation.md @@ -0,0 +1,74 @@ +# Unit Stability comparison validation + +Carl's PR #1428 feedback requested independent metric visibility and comparison +of multiple clusters on long recordings. The view now overlays up to eight +selected units using authoritative selection colors, with saved firing-rate, +amplitude and depth toggles. The display limit does not change merge membership. +Existing preparation, export, journal and replay formats are unchanged. + +## Data and screenshots + +The real-data gallery uses the existing converted AIND test export from +computation `b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`. Its clock spans +112.859 hours including gaps, with 113 summary bins and 705 source units. +This is the earlier local export, not Carl's newly mentioned 500-hour recording. +The conversion's acquisition mask remains unverified; these are software +integration checks, not scientific validation of exposure. + +Before and after captures select the same units, 40, 44 and 48, in a +2100 × 1200 Qt window. The baseline at `6ac9010d` displays only unit 40. +The new view displays all three, permits a full-height rate-only panel, and +preserves the same IDs and colors in Merge View. The authoritative table +presentation order is 40, 48, 44; colors follow that shared selection mapping. +Rate and amplitude are present; depth remains explicitly unavailable. + +The screenshot tool opens a disposable copy of the bounded package and checked +all 72 input file hashes remained unchanged. Full counts read from the source +summaries are 612,536 (40), 207,903 (44), and 336,413 (48). + +A separate generated 500-hour fixture contains 120,000 spikes, four units, +rate/amplitude/depth summaries and 1,000 bins. Each selected unit has exactly +30,000 source spikes. It supplies duration and missing-real-depth test coverage; +it must not be presented as Carl's 500-hour recording. + +Both capture processes exited normally after explicitly draining deferred Qt +deletions. Captures use Xvfb, Qt's X11 platform and software Mesa. Single runs, +concurrent with the test suite, measured selection/block at 0.372 s on the AIND +package and 0.623 s on the synthetic fixture. The two rate-only redraws together +took 0.094 s and 0.529 s respectively. These are observations, not benchmark +guarantees or CI timing thresholds. + +The gallery is in `docs/images/unit-stability-comparison/`. Reproduction options +are documented in `docs/long-recordings.md`; local capture manifests, source +hashes, scripts and logs are retained under +`/home/cyrille/GIT/IBL/phy-data/carl/stability-comparison/`. + +## Regression coverage and review + +Focused tests exercise exact per-unit rates with partial/zero exposure and a +short final bin, finite-value weighted merged means, pending split children +beside valid units, units without sampled evidence, row-at-a-time aggregation, +and absence of complete-spike-array reads in bounded mode. Rate-only loading +does not open trend arrays; toggle redraws reuse cached summaries. + +Selection tests cover normal/Similarity and Merge ordering, authoritative +colors, initial attachment during Merge mode, deferred hidden-view updates, +empty selection, frozen updates, display truncation, and merge/split undo/redo. +All eight metric visibility combinations retain stable axes/callback counts; +saved flags and QAction checks survive closing and reopening the view. +Ordinary optional Unit Stability retains its continuous-acquisition assumption +and refreshes cached counts on cluster edits. + +The implementation and existing adapter tests passed 25 focused cases. +An independent Sol 6.1-high audit found and reproduced a hidden-view palette +reset; the scoped fix and regression were included. Final review found no +remaining product blocker. A minimal standalone ordinary-GUI probe completed +all count assertions but exited 139 during interpreter teardown; repeating it +with explicit deferred Qt deletion exited normally. This remains a harness +cleanup limitation consistent with the earlier roundtrip record, not a verified +new product defect. + +Final local validation: `make test-full` passed 864 tests (499 core, 365 +application), with one existing benchmark skip; its lint and formatting checks +passed. `make doc-check` passed, including strict site generation and current +generated documentation. No packaging or dependency changes were made. diff --git a/docs/api.md b/docs/api.md index a5f0f2506..52733d154 100644 --- a/docs/api.md +++ b/docs/api.md @@ -10117,6 +10117,13 @@ Component that brings manual clustering facilities to a GUI: Whether automatic navigation and similar-cluster selection skip noise and MUA clusters. The default is True. +* `clustering_factory : callable, optional` + Construct the clustering model. Defaults to Clustering; bounded curation + supplies a guarded implementation with the same constructor interface. + +* `cluster_meta_factory : callable, optional` + Construct metadata from groups and labels. Defaults to create_cluster_meta. + **Events** When this component is attached to a GUI, the following events are emitted: @@ -13197,6 +13204,15 @@ Create a trace view. --- +#### TemplateController.create_unit_stability_view + + +**`TemplateController.create_unit_stability_view(self)`** + +Optionally inspect selected-unit activity over the loaded recording. + +--- + #### TemplateController.create_waveform_view diff --git a/docs/changelog.md b/docs/changelog.md index 2842a69f3..cdc53ac77 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -11,10 +11,65 @@ been included in a stable release. The current entries cover all user-visible changes committed since 23 July 2026; test-only commits are represented by the behavior they verify rather than listed separately. +### Long recordings + +- SLAY merge suggestions and saved accept/reject decisions now survive bounded + curation, full-sorting replay, and subsequent preparation. Suggestions retain + their original IDs; pending suggestions affected by merges or splits become + stale rather than being silently reassigned. +- Fixed saving bounded amplitude splits after merge undo/redo when NumPy cluster + IDs appear in journal metadata. +- Added `prepare-large` and `apply-curation` for bounded shared-window evidence, + full-duration exposure-aware summaries, semantic merges/labels/source-amplitude + splits, and replay into a new complete sorting. +- Added Unit Stability, precise window-relative amplitude/feature inspection, + sampled interval/correlogram views, and explicit full-versus-evidence counts. + Standard datasets retain their existing curation workflow and default layout. +- Bounded Evidence Amplitude now defaults to an all-retained-windows overview on + the recording clock, with a saved overview/window scope toggle. Inspecting a + window switches the view to that window. `Alt`-right-drag previews a source + amplitude threshold. `K` splits retained evidence and records the predicate; + `apply-curation` evaluates every source spike. Finite values strictly below + the threshold are selected; equality and nonfinite values stay in the remainder. +- Added source-grounded whole-output curation acceptance checks, with explicit + metadata-policy differences, saved-generation binding, and independent + all-source validation of cumulative merge and split predicates. Permanent controller + tests run the same actions through full and bounded modes across four rounds + and three sampling configurations, including points omitted from display. +- Unit Stability overlays up to 32 selected units by default, with consistent + colors and no legend. Its saved display limit accepts zero to show all selected + units, without a hard positive upper limit. Saved view settings include the + positive rate-line width (0.6 points by default) and the **Show firing rate**, + **Show amplitude**, and **Show depth** toggles, which default on. Its + dataset-local display-bin setting defaults to native resolution and permits + exact coarsening by integer multiples of the native bin width. The screenshot + action captures the Matplotlib Qt canvas. Full-source rates use acquisition + exposure, and amplitude splits mark only affected children’s metrics as + pending. On ordinary recordings, firing rates identify their + continuous-acquisition assumption. +- Bounded replay preserves producer metadata table filenames, delimiters, and + column layouts. New merge and split memberships clear producer values and + source IDs; quality labels are membership-scoped and start unreviewed until + assigned. Legacy journals retain inherited-metadata behavior until explicitly + migrated with the bounded session `migrate_metadata()` API. Replay parameter + overrides are kept in one managed final block while original source code + remains intact. +- Bounded saves use validated atomic generations and reject conflicting saves + from stale sessions. Unsupported selected-row splits are blocked. +- Sparse cluster IDs no longer require dense allocations during clustering. +- Bounded amplitude plots label seconds from window start and display the + window's recording-time range above the plot. +- Long-recording preparation accepts explicitly declared elapsed-tick + coordinates and can select whole validated producer evidence windows, while + keeping the existing spike and output-byte budgets. + ### Documentation - Documented uv-first installation of exact previous phy releases in separate environments, including how to intentionally replace a tool installation. +- Added a long-recording workflow guide covering bounded evidence preparation, + supported curation, full-sorting replay, and limits on interpreting sampled + measurements. ### Added diff --git a/docs/cli.md b/docs/cli.md index 0bb61a764..16b554a1f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -25,6 +25,35 @@ GUI commands accept: --clear-cache remove and rebuild the dataset .phy cache ``` +## Long recordings + +Prepare bounded evidence, curate it with `template-gui`, then replay supported +actions into a new full sorting. See the [long-recording guide](long-recordings.md) +for input requirements, interpretation limits, and the complete workflow. + +```bash +phy prepare-large /data/recording/params.py /data/recording-bounded \ + --duration 3600000 \ + --valid-intervals /data/recording/valid-intervals.npy \ + --max-spikes 2000000 \ + --max-bytes 536870912 \ + --window-seconds 30 \ + --stratum-seconds 3600 \ + --seed 0 \ + --bin-seconds 3600 \ + --batch-size 100000 + +phy template-gui /data/recording-bounded/params.py + +phy apply-curation /data/recording/params.py /data/recording-bounded \ + /data/recording-curated --batch-size 100000 +``` + +`--duration` is acquisition duration in seconds. `--valid-intervals` reads an +integer `.npy` array of `[start, end)` sample pairs. `--max-bytes` caps the +prepared package size, not process memory. The output directories must be new; +the original sorting remains the replay parent. + ## Raw trace viewer ```bash diff --git a/docs/images/amplitude-main-gui/main-reopened.png b/docs/images/amplitude-main-gui/main-reopened.png new file mode 100644 index 000000000..6d09833a2 Binary files /dev/null and b/docs/images/amplitude-main-gui/main-reopened.png differ diff --git a/docs/images/amplitude-main-gui/main-threshold-preview.png b/docs/images/amplitude-main-gui/main-threshold-preview.png new file mode 100644 index 000000000..af6f834fd Binary files /dev/null and b/docs/images/amplitude-main-gui/main-threshold-preview.png differ diff --git a/docs/images/amplitude-threshold-workflow/amplitude-after.png b/docs/images/amplitude-threshold-workflow/amplitude-after.png new file mode 100644 index 000000000..5f13a31ed Binary files /dev/null and b/docs/images/amplitude-threshold-workflow/amplitude-after.png differ diff --git a/docs/images/amplitude-threshold-workflow/amplitude-before.png b/docs/images/amplitude-threshold-workflow/amplitude-before.png new file mode 100644 index 000000000..397411ccd Binary files /dev/null and b/docs/images/amplitude-threshold-workflow/amplitude-before.png differ diff --git a/docs/images/amplitude-threshold-workflow/amplitude-preview.png b/docs/images/amplitude-threshold-workflow/amplitude-preview.png new file mode 100644 index 000000000..4d1fddd6d Binary files /dev/null and b/docs/images/amplitude-threshold-workflow/amplitude-preview.png differ diff --git a/docs/images/long-recording-evidence.png b/docs/images/long-recording-evidence.png new file mode 100644 index 000000000..294d03924 Binary files /dev/null and b/docs/images/long-recording-evidence.png differ diff --git a/docs/images/long-recording-overview.png b/docs/images/long-recording-overview.png new file mode 100644 index 000000000..cfd729bf7 Binary files /dev/null and b/docs/images/long-recording-overview.png differ diff --git a/docs/images/unit-metadata-refinements/carl-camera.png b/docs/images/unit-metadata-refinements/carl-camera.png new file mode 100644 index 000000000..0ec124cbb Binary files /dev/null and b/docs/images/unit-metadata-refinements/carl-camera.png differ diff --git a/docs/images/unit-metadata-refinements/carl-coarsened-rate.png b/docs/images/unit-metadata-refinements/carl-coarsened-rate.png new file mode 100644 index 000000000..a880270ad Binary files /dev/null and b/docs/images/unit-metadata-refinements/carl-coarsened-rate.png differ diff --git a/docs/images/unit-metadata-refinements/carl-native-rate.png b/docs/images/unit-metadata-refinements/carl-native-rate.png new file mode 100644 index 000000000..46175875b Binary files /dev/null and b/docs/images/unit-metadata-refinements/carl-native-rate.png differ diff --git a/docs/images/unit-metadata-refinements/synthetic-513h-coarsened.png b/docs/images/unit-metadata-refinements/synthetic-513h-coarsened.png new file mode 100644 index 000000000..c09fc1d2a Binary files /dev/null and b/docs/images/unit-metadata-refinements/synthetic-513h-coarsened.png differ diff --git a/docs/images/unit-metadata-refinements/synthetic-513h-native.png b/docs/images/unit-metadata-refinements/synthetic-513h-native.png new file mode 100644 index 000000000..afb66fbbf Binary files /dev/null and b/docs/images/unit-metadata-refinements/synthetic-513h-native.png differ diff --git a/docs/images/unit-stability-comparison/after-merge.png b/docs/images/unit-stability-comparison/after-merge.png new file mode 100644 index 000000000..79d632f23 Binary files /dev/null and b/docs/images/unit-stability-comparison/after-merge.png differ diff --git a/docs/images/unit-stability-comparison/after-overview.png b/docs/images/unit-stability-comparison/after-overview.png new file mode 100644 index 000000000..be2aa1aae Binary files /dev/null and b/docs/images/unit-stability-comparison/after-overview.png differ diff --git a/docs/images/unit-stability-comparison/after-rate-only.png b/docs/images/unit-stability-comparison/after-rate-only.png new file mode 100644 index 000000000..457b1b4e5 Binary files /dev/null and b/docs/images/unit-stability-comparison/after-rate-only.png differ diff --git a/docs/images/unit-stability-comparison/before-overview.png b/docs/images/unit-stability-comparison/before-overview.png new file mode 100644 index 000000000..74ebcc426 Binary files /dev/null and b/docs/images/unit-stability-comparison/before-overview.png differ diff --git a/docs/images/unit-stability-comparison/synthetic-500h-overview.png b/docs/images/unit-stability-comparison/synthetic-500h-overview.png new file mode 100644 index 000000000..a93b42325 Binary files /dev/null and b/docs/images/unit-stability-comparison/synthetic-500h-overview.png differ diff --git a/docs/long-recordings.md b/docs/long-recordings.md new file mode 100644 index 000000000..27aa113a2 --- /dev/null +++ b/docs/long-recordings.md @@ -0,0 +1,465 @@ +# Curating long recordings + +Phy's long-recording workflow prepares a bounded evidence dataset for interactive +review, then replays supported curation actions against the complete local +sorting. Preparation reads source arrays in batches; it does not load the full +sorting into the GUI. The source dataset is retained unchanged, and replay +writes a separate output dataset. + +This workflow supports a single monotonic event axis. Existing sources without +an explicit time-base declaration use `acquisition_samples`, the legacy +fixed-rate sample-coordinate contract. Sources with `time_base` may instead +declare `elapsed_ticks`: nonnegative integer elapsed-time coordinates with a +declared `ticks_per_second`. In either case, `num_samples` is the exclusive +upper bound of the canonical coordinate axis, and `valid_intervals` are +half-open `[start, end)` intervals on that same axis. For elapsed ticks, +`num_samples` is a tick bound, not a physical frame count. Explicit elapsed +sources must provide authoritative valid intervals; Phy cannot infer +acquisition gaps or filler from periods with no spikes. Multi-segment recordings +are not supported. + +The `time_base` declaration is a dictionary such as +`{'kind': 'elapsed_ticks', 'ticks_per_second': 30000.0}` in `params.py` (or +`recording.time_base` in the waveform manifest). For this workflow, +`ticks_per_second`, `sample_rate`, and the physical waveform sample rate must +be equal. Unequal event and waveform rates need separate waveform-axis support. +A source containing `timestamps.json` must declare its coordinate contract +explicitly; Phy does not parse that file to guess a conversion. Elapsed ticks +cannot index raw voltage, so an elapsed source with a real `dat_path` is not +supported. Stored waveforms can be used without raw data. + +For explicit elapsed exports, the waveform manifest must declare +`recording.sample_index_coordinates` and +`recording.window_bounds_coordinates` as `elapsed_ticks` when those sidecars are +present. The waveform-aligned `sample_index[w]` must equal +`spike_times[subset_spikes[w]]`; it is neither a source row ID nor a raw frame +offset. The manifest's `sample_index_coordinates` and +`window_bounds_coordinates` must each equal the declared `time_base.kind`. +Producer window bounds describe canonical elapsed coordinates and must not +cross unsupported gaps. The 512 MiB output cap applies to package bytes, not +process RSS; waveform source, retained, and omitted counts are reported in the +CLI and manifest. + +The screenshots below show the bounded GUI views using a deterministic +synthetic eight-hour sorting. These captures document the interface before +the latest Unit Stability display refinements; their unit cap and legend are +historical. The overview plots full-duration rates and +available source means; gray marks the synthetic acquisition gap and cyan marks +the retained evidence windows. The evidence view places windowed amplitudes at +top left, sampled ISI at top right, sampled correlograms at middle right, and +features below. They illustrate the interface and its sampling context; they +are not biological data or scientific examples. + +![Full-duration Unit Stability with a synthetic acquisition gap](images/long-recording-overview.png) + +![Windowed source amplitudes, features, sampled ISI and correlogram](images/long-recording-evidence.png) + +Recreate these images from source with Qt's X11 platform and software OpenGL +(the script writes a temporary synthetic source and bounded package): + +```bash +QT_QPA_PLATFORM=xcb LIBGL_ALWAYS_SOFTWARE=1 \ + xvfb-run -a -s '-screen 0 2400x1600x24 +extension GLX' \ + uv run python tools/capture_long_recording_views.py +``` + +### Comparing clusters on a real recording + +These actual Qt captures use the same converted AIND test export, selected +clusters **40, 44, and 48**, 113 summary bins, and 2100 × 1200 window size. +The elapsed clock spans 112.9 hours including declared gaps. Rate and source +amplitude are available; depth trends are unavailable. The exposure mask is +the test conversion's declaration, not independent acquisition verification. + +| Before: only unit 40 displayed | After: all three selected units | +| --- | --- | +| ![Before: three selected clusters but one stability curve](images/unit-stability-comparison/before-overview.png) | ![After: selected cluster rate and amplitude overlays](images/unit-stability-comparison/after-overview.png) | + +Turning off **Show amplitude** and **Show depth** expands the rate comparison: + +![Rate-only comparison of units 40, 44 and 48](images/unit-stability-comparison/after-rate-only.png) + +These captures predate the current Unit Stability display refinements; the +overlaid rates remain representative, while any legend or unit-display limit +visible in them is historical. The same colors and cluster IDs follow the staged +Merge candidates: + +![Merge View candidates match the firing-rate curves](images/unit-stability-comparison/after-merge.png) + +To capture an existing package, use a fresh output directory outside it. The +script opens a temporary copy and verifies the input file hashes are unchanged: + +```bash +QT_QPA_PLATFORM=xcb LIBGL_ALWAYS_SOFTWARE=1 \ + xvfb-run -a -s '-screen 0 2400x1600x24 +extension GLX' \ + uv run python tools/capture_long_recording_views.py --comparison \ + --bounded-dir /data/recording-bounded --cluster-ids 40 44 48 \ + --dataset-label 'Recording test export' --output-dir /tmp/stability-captures +``` + +A separate [synthetic 500-hour example](images/unit-stability-comparison/synthetic-500h-overview.png) +exercises rate, amplitude, and depth together over 1000 bins. It is generated +test data, not Carl's 500-hour recording. Reproduce it with `--comparison +--synthetic-hours 500 --output-dir /tmp/stability-500h` instead of `--bounded-dir` +and `--cluster-ids`. + +### Latest display refinements + +The following captures show the current controls on Carl's earlier converted +112.9-hour test export, with units **24, 26, 27, and 32**. The camera image was +saved through Unit Stability's own screenshot action. Legends are removed and +native rate lines use the new 0.6-point default. Declared exposure remains the +conversion's test mask, not independent acquisition verification. + +![Camera capture with no legend and thin rate lines](images/unit-metadata-refinements/carl-camera.png) + +A separate generated 513-hour fixture displays twelve selected units with +6156 native five-minute bins. Selecting ten-minute display bins groups these +into 3078 bins by adding counts/exposure and measurement sums/counts; prepared +files remain unchanged. This is generated test data, not Carl's 500-hour export. + +| Native five-minute bins | Ten-minute display bins | +| --- | --- | +| ![Twelve synthetic units at native resolution](images/unit-metadata-refinements/synthetic-513h-native.png) | ![The same units with exact display coarsening](images/unit-metadata-refinements/synthetic-513h-coarsened.png) | + +Reproduce the generated stress captures with `--comparison --synthetic-hours +513 --synthetic-units 12 --summary-bin-seconds 300 --output-dir +/tmp/stability-513h`. The capture script also exercises the view's camera action +and writes `camera-rate-only.png`. Display coarsening starts at the native +prepared resolution; it never changes preparation's default bin size. + +## Prepare bounded evidence + +Run `prepare-large` with the source `params.py` and a new output directory: + +```bash +phy prepare-large /data/recording/params.py /data/recording-bounded \ + --duration 3600000 \ + --valid-intervals /data/recording/valid-intervals.npy \ + --max-spikes 2000000 \ + --max-bytes 536870912 \ + --window-seconds 30 \ + --stratum-seconds 3600 \ + --seed 0 \ + --bin-seconds 3600 \ + --batch-size 100000 +``` + +`--duration` is in seconds and must agree with declared acquisition metadata. +The `.npy` valid-interval file contains an integer array of shape `[W, 2]` with +ordered, nonoverlapping coordinate pairs. Without it, legacy acquisition-sample +sources treat the full declared extent as valid exposure. Explicit elapsed +sources require authoritative `valid_intervals` in metadata or supplied +explicitly through the CLI/API; there is no full-extent +default for them. + +`--max-bytes` limits the bounded output package size, including its summaries and +window plan. It is not a process-memory or peak-RSS limit. The source unit IDs +must fit nonnegative `int32`; sample coordinates and full-source spike row IDs +use `int64`. + +Preparation uses `--window-policy auto` by default. When validated producer +windows are available, it selects among those whole windows and preserves their +acquisition boundaries. Producer-window mode requires valid producer windows; +it does not generate rescue windows to compensate for missing or malformed +producer evidence. Without producer windows, the original shared-window planner +uses the configured window length, strata, and seed. Inspect the saved window +plan in the output package when documenting or repeating an analysis. +Windows that exceed the budget are omitted, so a high-density interval can +reduce retained evidence. Preparation does not shorten an over-budget window or +silently thin spikes inside it. The generated planner supplements its survey of +shared windows with deterministic rescue evidence for units missed by the +survey. This improves the chance of seeing rare units, but makes the evidence +selection unequal: rescue windows must not be treated as a probability sample. +The explicit `producer` policy uses only producer windows. The `auto` policy +can supplement producer windows with generated rescue evidence for units still +missing from the selected evidence. + +Overlapping producer windows are allowed; selection uses their union while +retaining their acquisition boundaries. Preparation records exact +full-recording spike counts and rates using valid exposure. If source scalar +amplitudes or depths are available, it +also records per-bin finite-value means and availability counts. It does not +invent missing measurements or convert source values to physical units. Stored +waveforms are included only where available in the source; raw traces are not +required when stored waveform data are sufficient for the views being used. +The default spike and byte budgets remain 2,000,000 spikes and 512 MiB. Output +byte budgets describe package bytes, not resident memory. Waveform availability +can exceed the bounded output budget: preparation reports waveform counts +retained, omitted, and available in the source on the CLI, and stores the counts +in the manifest so truncation is visible. +Producer metadata tables retain their source filenames, delimiters, and column +order, including multi-column CSV files. For a new merge or split membership, +producer values and source-unit IDs are blank; imported custom fields are also +cleared. Quality labels belong to the current unit +membership: a new membership starts unreviewed until explicitly labeled. +Untouched memberships keep their labels, and label history remains in the +curation journal. Existing legacy journals keep their inherited-metadata +behavior until explicitly migrated with the bounded session's +`migrate_metadata()` API. This migration preserves label intent while adopting +membership-scoped metadata. Use the full-count, rate, amplitude-mean, and +depth-mean summaries for the measurements they explicitly cover; producer +metrics are not recomputed during merge or split. + +For an already saved legacy bounded session, close the GUI and explicitly +migrate before applying its curation. Supply the original producer params if +the source has moved; this also recovers its original metadata filenames: + +```python +from phy.apps.large_recording.curation import BoundedSession + +session = BoundedSession('/data/recording-bounded') +try: + session.migrate_metadata('/data/original-recording/params.py') + session.save() +finally: + session.close() +``` + +Migration publishes a new save generation and retains the older generations, +assignments, review decisions, and explicit label events. A legacy session +remains readable under its old rules until migrated. Already exported values +alone cannot reveal whether a label was inherited or explicitly assigned; keep +the original source and saved bounded journal to recover that distinction. +The same applies to metadata filenames already changed by an older export. +Conflicting JSON/table metadata is reported with the filename, field, and unit; +reconcile the input representations rather than relying on silent precedence. + +Replay keeps source parameter code intact and maintains one final marked block +for its `num_samples`, `valid_intervals`, and `dat_path` overrides. Original +assignments can remain in the file because they may be used by other user +expressions; the managed block, rather than every declaration in the file, is +the stable replay override. + +## Review and curate + +Open the bounded output with the normal GUI command: + +```bash +phy template-gui /data/recording-bounded/params.py +``` + +The bounded manifest explicitly selects this mode. Bounded mode has its own +configuration namespace and default view layout; it does not replace the normal +Phy layout or restore ordinary saved dock geometry. The GUI keeps full-recording +counts and rates distinct from counts in selected evidence rows. Unit Stability +shows time-binned counts/rates, available source scalar-amplitude means, +evidence windows, and acquisition gaps. Missing measurements and units without +evidence remain visible as unavailable. + +Unit Stability overlays up to 32 selected clusters by default for responsiveness, +using the same +colors as the cluster and Merge views and no legend. The view menu's display +limit can be changed; zero displays all selected units, with no hard positive +upper limit. A nonzero limit affects only the displayed curves, not cluster +selection or merge membership. Use the **Show firing rate**, **Show amplitude**, +and **Show depth** toggles to choose plotted measurements. These choices are +saved across sessions; all three are on by default, and turning all three off +leaves the view empty. The default rate-line width is 0.6 points; its positive +width setting is saved with the view. The display-bin setting is local to the +dataset: zero uses the prepared native bins (or the view's native bins on an +ordinary recording), while a coarser bin groups adjacent bins by adding counts +and exposure, and additive trend summaries, exactly. It accepts only integer +multiples of the native bin width, so it cannot display finer bins. The default +is native resolution, preserving the prepared bin size (for example, 300 or +3600 seconds). The screenshot action captures the Matplotlib Qt canvas. The +full-source counts and rates use acquisition exposure, including declared gaps; +source amplitude and depth summaries appear only when available. + +Feature View uses window-relative seconds and labels the current window with its +recording-time range. Unit Stability keeps the full-recording time context +visible while individual windows are inspected. + +The Sampled ISI view excludes intervals crossing retained-window boundaries. +The Sampled Correlogram view shows raw pair counts from retained windows; it +does not apply a full-recording rate baseline. Both views are explicitly labeled +as sampled evidence. Its central bin counts unordered within-unit pairs and +combines both temporal orientations for cross-unit pairs whose absolute delay +is less than one bin, so tied sample times do not depend on source row order. +For exact pair counts, each unit may have at most 46,340 spikes in an evidence +window; the view reports an error for a larger per-unit window. Prepare shorter +windows if this limit is reached. + +The Evidence Amplitude view opens in a sampled **overview**: its horizontal +axis uses the original recording clock and displays retained evidence windows +together. Use the view menu's **Toggle amplitude scope** action to switch between +this overview and the current evidence window. **View > Inspect evidence +window** selects a retained window and switches Evidence Amplitude to that +window context; the overview remains available by toggling the scope back. +The window scope uses seconds from that window's start and shows its recording- +time range. Changing the inspected window resets its time origin. +Displayed points may be further limited by the view's point budget. This only +limits what is drawn: split predicates are evaluated over every source spike +when applied. + +Bounded curation supports whole-unit merges, cluster labels, and source-amplitude +splits. Outside Merge mode, select exactly one unit in Evidence Amplitude and +use `Alt` plus right-drag to place or adjust the horizontal threshold preview. +Finite source amplitudes strictly below the threshold are selected; equality +and nonfinite values stay in the remainder. The preview is visual evidence from +the displayed sample. Pressing `K` requires evidence on both sides of the +threshold and commits the predicate. Preview state clears when selection or +view context changes. The command action **Split by source amplitude** remains +available for directly entering a threshold. +In bounded mode, `K` splits the retained evidence rows and records the semantic +source-amplitude predicate; `phy apply-curation` later evaluates that predicate +against every source spike. Thus the bounded GUI assignment is provisional for +unseen spikes until full-source replay. Source `amplitudes.npy` values are used +during replay. A split marks affected child units' full-data metrics as pending +individually; unrelated selected units retain their available metrics. +Resolving pending full-data counts and +trends requires full replay followed by re-preparation. Merge and label actions +are journaled with the bounded assignments and metadata in an atomic checkpoint +generation. +Reopening loads the last complete generation. A reopened session starts a new +undo baseline. Each save checks that the checkpoint has not changed since this +session opened. If another session has saved first, the stale save fails; reopen +the bounded dataset before continuing to avoid overwriting that generation. + +Lasso, feature, and waveform splits remain unavailable in bounded mode because +the selected rows do not define how unseen source spikes should be assigned. +Ordinary datasets retain their standard split tools, including the graphical +amplitude-threshold preview. + +### Comparing ordinary and bounded curation + +Perform the same documented merges, amplitude threshold splits and explicit +manual-label decisions on separate copies of one source. Compare the ordinary +saved sorting against the complete `apply-curation` output, rather than against +the retained evidence package: + +```bash +uv run python tools/compare_curation_modes.py compare \ + /data/ordinary-final /data/bounded-full-output \ + --fields group notes --output /data/comparison.json +``` + +The report checks every spike's time and cluster partition through a two-way +ID mapping, then checks the requested manual labels. Numeric cluster IDs may +differ between runs. Use a fresh report path outside both input folders. +This basic mode compares only the requested manual fields; it excludes producer +metrics whose inheritance policy may differ. + +For acceptance, enable the stronger whole-output contract with the original +immutable source, the bounded package used for the final replay, and independent +expected final curator decisions: + +```bash +uv run python tools/compare_curation_modes.py compare \ + /data/ordinary-final /data/bounded-full-output \ + --source /data/original-source --bounded-dir /data/final-bounded \ + --expectations /data/expected-labels.json --output /data/full-comparison.json +``` + +The expectations file maps replay unit IDs to reviewed fields, for example +`{"708": {"group": "good", "notes": "reviewed"}, "49": {"group": null}}`. +Write it from the planned decisions and the allocated ID ledger, rather than +copying values from an output being tested. Include every surviving explicit +label decision, including cleared labels. Across multiple rounds, keep +`--source` pointing to the original source and supply the final round's package. +Put the report outside every input folder. + +This mode checks every imported metadata field, byte-identical immutable +arrays/sidecars, preserved replay table layouts, parameters, and the exact saved +generation's journal, metadata, review state and retained assignments. Unknown +or missing files fail. An independent chunked oracle applies the cumulative +merge and threshold-split history to every original source spike, including +spikes omitted from the bounded evidence, and checks exact replay assignments. +It evaluates original scalar attributes, comparator equality and the declared +nonfinite policy across repeated curation rounds. +New memberships must have cleared producer values unless +explicitly reviewed; native ancestor inheritance is checked and reported as a +permitted difference. Native source aliases and untouched blank source columns +are admitted only when their original bytes match. A generated native whitening +inverse is checked against its source matrix. GUI caches/logs may differ; +generated cluster summaries must still have the correct catalog and manual +labels. Existing suggestion decisions must survive unchanged; new SLAY decisions +use the separate proposition acceptance procedure. + +Without `--bounded-dir`, first-round source checks do not verify preparation or +saved-generation binding. Later-round verification requires that package. +The report states which bindings were verified; use the complete command above +for acceptance. The repository's `design/curation-output-equivalence.md` records +the contract, permanent controller tests and real-data evidence. + +For graphical split comparisons, choose finite source amplitudes: ordinary +TemplateModel displays nonfinite amplitudes as zero, while bounded replay +keeps them in the remainder. Select **template** amplitudes in ordinary +Amplitude View to compare its stored `amplitudes.npy` values; feature amplitudes +represent a different quantity. The tool also provides independent ordinary-engine +and streamed references; neither replaces a full ordinary GUI acceptance run. +The repository's `design/curation-user-acceptance.md` provides the action ledger, +additional user checks and evidence limits. + +## Unit Stability on ordinary recordings + +Unit Stability can also be opened as an optional view for an ordinary dataset. +It evaluates selected units and bins their loaded spikes over the recording +duration. Its firing rate assumes continuous acquisition because ordinary +datasets do not supply valid acquisition intervals to the view; the plot labels +that assumption. The Unit Stability view menu's **Show firing rate**, +**Show amplitude**, and **Show depth** toggles and rate-line width are saved +across sessions, with all three toggles on by default. The display-bin setting +is dataset-local. The ordinary default view layout remains unchanged. +Use the bounded manifest workflow above when known acquisition gaps must be +represented explicitly. + +## Apply to the complete sorting + +After saving the bounded session, replay its journal against the exact source +assignment revision used during preparation: + +```bash +phy apply-curation /data/recording/params.py /data/recording-bounded \ + /data/recording-curated --batch-size 100000 +``` + +Replay checks the source identity and parent assignment revision, streams the +complete assignment array in batches, and writes a new full sorting. It rejects +a journal prepared from a different source revision. Keep the original source +and bounded package; they establish the lineage of the curated output. To review +the curated full sorting with bounded evidence again, run `prepare-large` on +the curated output to produce a fresh package and revision-specific summaries; +this resolves pending full-data counts and trends for split children. Replay +applies the recorded threshold split to every source spike using its source +amplitude value. + +When the source includes SLAY suggestions in `curation.json`, preparation and +replay preserve that file exactly, together with saved accept/reject decisions +in `curation_review.json`. Subsequent bounded rounds retain those decisions, +including rounds with no new edits or with the proposition view disabled. +There is no need to copy either file by hand. + +Suggestions keep their original cluster IDs and source order. A saved acceptance +remains part of the review history even if its resulting cluster is later +merged or split. Pending suggestions whose original clusters disappear become +stale; known merge or split causes are shown without assigning the suggestion +to replacement clusters automatically. Legacy outputs without that history can +only report that an original cluster is missing. + +Bounded packages bind suggestions and review history to file hashes. A committed +checkpoint supplies the saved decisions even if a loose review copy is missing +or outdated. Changing the bound suggestion file requires preparing a new +package; incompatible or damaged documents stop saving or replay. +Older bounded packages with unbound suggestion or review documents require +migration before they can be replayed with this integrity check. Retain their +source and saved checkpoints so existing curation can be recovered; preparing +the original source again alone does not transfer those saved edits. + +## Interpreting evidence + +Full counts and exposure-based rates describe the complete sorting. Window plots, +waveforms, feature scatterplots, and other bounded measurements describe only +the retained evidence. Survey and rescue selection can overrepresent some +periods or units, and waveform availability can vary across units and time. +These views do not establish full-recording quality, contamination, refractory +violations, or continuous biological identity. Use them to guide review, not as +scientific claims about unseen spikes. Long-duration coordinates preserve the +declared source time axis; silence alone is never evidence that the acquisition was +inactive. + +For reproducible synthetic capacity measurements, run +`uv run python tools/benchmark_long_recording.py`. The benchmark reports +preparation and replay time and peak memory in separate processes and checks +replay against a streamed oracle. It is capped at two million synthetic spikes; +its results are not a substitute for measurements on real chronic datasets. diff --git a/mkdocs.yml b/mkdocs.yml index bf8a4ad12..f59e59417 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -17,6 +17,7 @@ nav: - How-to and customization: - Configuration and state: configuration.md - Performance and spike sampling: performance.md + - Curating long recordings: long-recordings.md - Change keyboard shortcuts: keyboard_customization.md - Plugin introduction: customization.md - Plugin examples: plugins.md diff --git a/phy/apps/__init__.py b/phy/apps/__init__.py index db51602ea..a74019d0d 100644 --- a/phy/apps/__init__.py +++ b/phy/apps/__init__.py @@ -174,6 +174,72 @@ def cli_template_describe(ctx, params_path): template_describe(params_path) +# ------------------------------------------------------------------------------ +# Long-recording batch commands +# ------------------------------------------------------------------------------ + + +@phycli.command('prepare-large') +@click.argument('params-path', type=click.Path(exists=True, dir_okay=False, path_type=Path)) +@click.argument('output-dir', type=click.Path(path_type=Path)) +@click.option('--duration', 'duration_seconds', type=click.FloatRange(min=0, min_open=True)) +@click.option('--valid-intervals', type=click.Path(exists=True, dir_okay=False, path_type=Path)) +@click.option('--max-spikes', type=click.IntRange(min=1), default=2000000, show_default=True) +@click.option('--max-bytes', type=click.IntRange(min=1), default=536870912, show_default=True) +@click.option('--window-seconds', type=click.FloatRange(min=0, min_open=True), default=30.0) +@click.option('--stratum-seconds', type=click.FloatRange(min=0, min_open=True), default=3600.0) +@click.option('--seed', type=click.IntRange(min=0), default=0) +@click.option('--bin-seconds', type=click.FloatRange(min=0, min_open=True), default=3600.0) +@click.option('--batch-size', type=click.IntRange(min=1), default=100000) +@click.option( + '--window-policy', + type=click.Choice(['auto', 'producer', 'generated']), + default='auto', + show_default=True, + help='Prefer producer waveform windows, require them, or generate new windows.', +) +def cli_prepare_large(params_path, output_dir, valid_intervals=None, **kwargs): + """Prepare bounded shared-window evidence and complete-sorting summaries. + + VALID_INTERVALS is an optional int64 .npy array of acquisition [start, end) tick pairs. + Duration/exposure must be authoritative; it is never inferred from spike silence. + """ + import json + + import numpy as np + + from .large_recording.prepare import prepare_large + + try: + intervals = np.load(valid_intervals, allow_pickle=False) if valid_intervals else None + output = prepare_large(params_path, output_dir, valid_intervals=intervals, **kwargs) + except (ValueError, OSError) as error: + raise click.ClickException(str(error)) from error + click.echo(f'Prepared bounded evidence: {output}') + manifest = json.loads((output / '_phy_curation_manifest.json').read_text(encoding='utf-8')) + coverage = manifest['waveform_coverage'] + click.echo( + f'Stored waveforms: {coverage["retained_rows"]:,} retained / ' + f'{coverage["source_rows"]:,} source; {coverage["dropped_rows"]:,} omitted.' + ) + + +@phycli.command('apply-curation') +@click.argument('source-params', type=click.Path(exists=True, dir_okay=False, path_type=Path)) +@click.argument('bounded-dir', type=click.Path(exists=True, file_okay=False, path_type=Path)) +@click.argument('output-dir', type=click.Path(path_type=Path)) +@click.option('--batch-size', type=click.IntRange(min=1), default=100000) +def cli_apply_curation(source_params, bounded_dir, output_dir, batch_size): + """Replay committed bounded curation into a new complete sorting.""" + from .large_recording.replay import apply_curation + + try: + output = apply_curation(source_params, bounded_dir, output_dir, batch_size=batch_size) + except (ValueError, OSError) as error: + raise click.ClickException(str(error)) from error + click.echo(f'Published complete curated sorting: {output}') + + # ------------------------------------------------------------------------------ # Kwik GUI # ------------------------------------------------------------------------------ @@ -224,6 +290,11 @@ def cli_alf_convert(ctx, subdirs, out_dir): from phylib.io.merge import Merger from phylib.io.model import load_model + if any((Path(path) / '_phy_curation_manifest.json').exists() for path in subdirs): + raise click.ClickException( + 'Apply bounded curation to the complete sorting before ALF conversion.' + ) + out_dir = Path(out_dir) if len(subdirs) >= 2: @@ -253,6 +324,11 @@ def template_extract_waveforms( """Extract spike waveforms.""" from phylib.io.model import load_model + if (Path(params_path).parent / '_phy_curation_manifest.json').exists(): + raise click.ClickException( + 'Extract waveforms on the original source and regenerate bounded evidence; bounded package assets are immutable.' + ) + model = load_model(params_path) model.save_spikes_subset_waveforms( max_n_spikes_per_template=n_spikes_per_cluster, max_n_channels=nc diff --git a/phy/apps/base.py b/phy/apps/base.py index 593ddfcff..63c8a7fc2 100644 --- a/phy/apps/base.py +++ b/phy/apps/base.py @@ -2449,7 +2449,11 @@ def create_gui(self, default_views=None, **kwargs): subtitle=str(self.dir_path), config_dir=self.config_dir, local_path=self.cache_dir / 'state.json', - default_state_path=Path(inspect.getfile(self.__class__)).parent / 'static/state.json', + default_state_path=getattr( + self, + 'default_state_path', + Path(inspect.getfile(self.__class__)).parent / 'static/state.json', + ), view_creator=self.view_creator, default_views=default_views, enable_threading=self._enable_threading, diff --git a/phy/apps/large_recording/__init__.py b/phy/apps/large_recording/__init__.py new file mode 100644 index 000000000..373145000 --- /dev/null +++ b/phy/apps/large_recording/__init__.py @@ -0,0 +1 @@ +"""Bounded evidence preparation and full-sorting curation replay.""" diff --git a/phy/apps/large_recording/adapter.py b/phy/apps/large_recording/adapter.py new file mode 100644 index 000000000..4923c9096 --- /dev/null +++ b/phy/apps/large_recording/adapter.py @@ -0,0 +1,517 @@ +"""Manifest-gated template controller for bounded recording evidence.""" + +from copy import deepcopy +from pathlib import Path + +import numpy as np +from phylib.io.model import get_template_params +from phylib.utils import Bunch, connect, unconnect + +from phy.cluster._propositions import ( + MergePropositionController, + decode_curation_mapping, + decode_review_mapping, +) + +from ..base import _allocate_spike_counts, _sample_spikes_evenly +from ..template.gui import TemplateController +from .curation import BoundedSession +from .model import BoundedTemplateModel +from .statistics import window_correlograms, window_isi +from .supervisor import BoundedSupervisor +from .views import EvidenceAmplitudeView, SampledCorrelogramView, SampledISIView + + +class BoundedTemplateController(TemplateController): + gui_name = 'BoundedTemplateGUI' + # Bounded mode has its own state namespace and initial view list. Importing + # ordinary saved dock geometry would suppress the new evidence views. + default_state_path = None + default_views = ( + 'UnitStabilityView', + 'WaveformView', + 'EvidenceAmplitudeView', + 'FeatureView', + 'ProbeView', + 'SampledISIView', + 'SampledCorrelogramView', + ) + + def __init__(self, dir_path=None, model=None, **kwargs): + if model is not None: + raise ValueError( + 'Open bounded packages from their directory, without a supplied model.' + ) + self.bounded_session = BoundedSession(dir_path) + self.evidence_window_index = 0 + kwargs['enable_threading'] = False + kwargs.update(get_template_params(Path(dir_path) / 'params.py')) + kwargs.pop('dir_path', None) + super().__init__(dir_path=dir_path, **kwargs) + + def _create_model(self, dir_path=None, **kwargs): + model = BoundedTemplateModel(dir_path=dir_path, session=self.bounded_session, **kwargs) + if model.sample_rate != self.bounded_session.manifest['sample_rate']: + model.close() + raise ValueError('Bounded params sampling rate disagrees with the manifest.') + model.spike_clusters = self.bounded_session.assignments.copy() + model.metadata = deepcopy(self.bounded_session.metadata) + model.duration = self.bounded_session.manifest['num_samples'] / model.sample_rate + + def reject_save(*args, **kwargs): + raise ValueError('Save bounded curation through supervisor.save() as one transaction.') + + model.save_spike_clusters = reject_save + model.save_metadata = reject_save + model.save_spikes_subset_waveforms = reject_save + model.freeze_source() + return model + + def _cache_methods(self): + # Ordinary persistent feature caches omit the current evidence window. + # Bounded queries are already capped; evaluate them against live state. + return + + def close(self, close_model=True): + super().close(close_model=close_model) + if close_model: + self.bounded_session.close() + + def _set_supervisor(self): + session = self.bounded_session + propositions = self._load_merge_propositions() + self.supervisor = BoundedSupervisor( + spike_clusters=self.model.spike_clusters.copy(), + cluster_groups=self.model.metadata.get('group', {}), + cluster_metrics=self.cluster_metrics, + cluster_labels=self.model.metadata, + similarity=self.similarity_functions[self.similarity], + session=session, + merge_propositions=propositions, + ) + self.supervisor._validate_native() + + def _load_merge_propositions(self): + session = self.bounded_session + session.validate_proposition_documents() + self._merge_proposition_sha256 = None + if not self.enable_merge_propositions: + return None + item = session.proposition_documents.get('curation.json') + if item is None: + return None + from .curation import _read_json + + catalog = decode_curation_mapping( + _read_json(session.path / item['filename']), retired_units=session.retired_units + ) + self._merge_proposition_sha256 = item['sha256'] + reviews = session.extra_documents.get('curation_review.json') + if reviews is not None: + _, mapping = decode_review_mapping(reviews) + catalog = catalog._replace(reviews=mapping) + return MergePropositionController(catalog) + + def _set_cluster_metrics(self): + super()._set_cluster_metrics() + self.cluster_metrics['full_count'] = self.get_full_count + self.cluster_metrics['evidence_count'] = lambda c: self.supervisor.n_spikes(c) + self.cluster_metrics['waveform_count'] = self.get_waveform_count + self.cluster_metrics['evidence'] = lambda c: ( + 'pending replay' if self.get_full_count(c) is None else 'sampled windows' + ) + + def _set_view_creator(self): + super()._set_view_creator() + for name in ( + 'ISIView', + 'CorrelogramView', + 'FiringRateView', + 'TraceView', + 'TraceImageView', + ): + self.view_creator.pop(name, None) + self.view_creator.pop('AmplitudeView', None) + self.view_creator['EvidenceAmplitudeView'] = self.create_evidence_amplitude_view + self.view_creator['UnitStabilityView'] = self.create_unit_stability_view + self.view_creator['SampledISIView'] = lambda: SampledISIView(cluster_stat=self._get_isi) + self.view_creator['SampledCorrelogramView'] = lambda: SampledCorrelogramView( + correlograms=self._window_correlograms, sample_rate=self.model.sample_rate + ) + + def _get_waveforms_dict(self): + functions = super()._get_waveforms_dict() + if not self.bounded_session.manifest['capabilities']['stored_waveforms']: + functions.pop('waveforms', None) + functions.pop('mean_waveforms', None) + return functions + + def create_waveform_view(self): + view = super().create_waveform_view() + if view is None: + return + plot = view.plot + + def plot_and_clear_preview(**kwargs): + # Sampling, waveform mode, and channel layout may change the drawn + # traces. Require a fresh preview rather than leaving stale colors. + result = plot(**kwargs) + for amplitude_view in getattr(self, '_evidence_amplitude_views', ()): + amplitude_view.clear_amplitude_split_threshold() + return result + + view.plot = plot_and_clear_preview + return view + + def create_feature_view(self): + if ( + self.model.features is None + and not self.bounded_session.manifest['capabilities']['stored_waveforms'] + ): + return + view = super().create_feature_view() + if view is not None: + view.ex_status = self._window_status() + return view + + def _source_units(self, cluster_id): + version = self.bounded_session.events + cached = getattr(self, '_lineage_cache', None) + if cached is not None and cached[0] is version: + return cached[1].get(int(cluster_id)) + units = {int(c): {int(c)} for c in self.bounded_session.unit_ids} + for event in self.bounded_session.events: + if event['operation'] == 'merge': + parents = [units.pop(c) for c in event['parents']] + units[event['target']] = ( + None if any(p is None for p in parents) else set.union(*parents) + ) + elif event['operation'] == 'threshold_split': + units.pop(event['parent']) + units.update(dict.fromkeys(event['children'])) + self._lineage_cache = version, units + return units.get(int(cluster_id)) + + def get_full_count(self, cluster_id): + units = self._source_units(cluster_id) + if units is None: + return None + if not hasattr(self, '_full_counts'): + self._full_counts = dict( + zip(self.bounded_session.unit_ids, self.bounded_session.manifest['counts']) + ) + counts = self._full_counts + return sum(int(counts[c]) for c in units) + + def get_waveform_count(self, cluster_id): + stored = self.model.spike_waveforms + if stored is None: + return 0 + if ( + not hasattr(self, '_waveform_membership_cache') + or self._waveform_membership_cache[0] is not self.bounded_session.events + ): + labels = self.supervisor.clustering.spike_clusters[stored.spike_ids] + ids, counts = np.unique(labels, return_counts=True) + self._waveform_membership_cache = self.bounded_session.events, dict(zip(ids, counts)) + return int(self._waveform_membership_cache[1].get(cluster_id, 0)) + + def get_mean_firing_rate(self, cluster_id): + count = self.get_full_count(cluster_id) + if count is None: + return None + intervals = np.asarray(self.bounded_session.manifest['valid_intervals']) + exposure = np.sum(intervals[:, 1] - intervals[:, 0]) / self.model.sample_rate + return count / exposure + + @property + def evidence_windows(self): + if not hasattr(self, '_evidence_windows'): + self._evidence_windows = np.load( + self.dir_path / self.bounded_session.manifest['windows_file'] + ) + return self._evidence_windows + + def _get_isi(self, cluster_id): + ids = self.supervisor.clustering.spikes_per_cluster.get(cluster_id, []) + samples = self.model.spike_samples[ids] + return Bunch(data=window_isi(samples, self.evidence_windows, self.model.sample_rate)) + + def _window_correlograms(self, cluster_ids, bin_size, window_size): + return window_correlograms( + self.model.spike_samples, + self.supervisor.clustering.spike_clusters, + self.evidence_windows, + cluster_ids, + self.model.sample_rate, + bin_size, + window_size, + ) + + def create_unit_stability_view(self): + from .stability import UnitStabilityView + + return UnitStabilityView(self) + + @property + def current_evidence_window(self): + return self.evidence_windows[self.evidence_window_index] + + def _window_ids(self, cluster_id=None): + start, end = self.current_evidence_window + if cluster_id is None: + first, last = np.searchsorted(self.model.spike_samples, [start, end]) + return np.arange(first, last, dtype=np.int64) + ids = self.supervisor.clustering.spikes_per_cluster.get( + cluster_id, np.array([], dtype=np.int64) + ) + times = self.model.spike_samples[ids] + first, last = np.searchsorted(times, [start, end]) + return ids[first:last] + + def _window_time(self, ids): + # Subtract int64 samples before floating conversion, even at hour 1000. + return ( + self.model.spike_samples[ids] - self.current_evidence_window[0] + ) / self.model.sample_rate + + def _get_feature_view_spike_ids(self, cluster_id=None, load_all=False): + ids = self._window_ids(cluster_id) + if self.model.features is None: + if self.model.spike_waveforms is None: + return np.array([], dtype=np.int64) + ids = np.intersect1d(ids, self.model.spike_waveforms.spike_ids) + if getattr(self.model, 'features_rows', None) is not None: + ids = np.intersect1d(ids, self.model.features_rows) + limit = self.n_spikes_features_background if cluster_id is None else self.n_spikes_features + return ids if load_all else _sample_spikes_evenly(ids, limit) + + def _get_feature_view_spike_times(self, cluster_id=None, load_all=False): + ids = self._get_feature_view_spike_ids(cluster_id, load_all=load_all) + if not len(ids): + return + start, end = self.current_evidence_window + return Bunch( + data=self._window_time(ids), + spike_ids=ids, + lim=(0.0, (end - start) / self.model.sample_rate), + ) + + def _resolve_spike_amplitudes(self, spike_ids, name, first_cluster): + if name == 'source': + return self.bounded_session.attributes['amplitudes.npy'].reshape(-1)[spike_ids] + return super()._resolve_spike_amplitudes(spike_ids, name, first_cluster) + + def _evidence_amplitudes(self, cluster_ids, load_all=False, scope='window'): + def eligible_ids(cluster_id=None): + if cluster_id is None: + # Sample conceptual row coordinates before gathering assignments. + # Interactive updates must not scan all retained spikes. + first, last = 0, self.model.n_spikes + if scope == 'window': + first, last = np.searchsorted( + self.model.spike_samples, self.current_evidence_window + ) + count = min(last - first, self.n_spikes_amplitudes_background) + if not count: + return np.array([], dtype=np.int64) + ids = np.arange(count, dtype=np.int64) + if count > 1: + ids *= last - first - 1 + ids //= count - 1 + return ids + first + if scope == 'window': + return self._window_ids(cluster_id) + return self.supervisor.clustering.spikes_per_cluster.get( + cluster_id, np.array([], dtype=np.int64) + ) + + def spike_times(ids): + if scope == 'window': + return self._window_time(ids) + return self.model.spike_samples[ids] / self.model.sample_rate + + active = [c for c in cluster_ids if c is not None] + eligible = [eligible_ids(c) for c in active] + counts = _allocate_spike_counts( + [len(ids) for ids in eligible], + per_cluster=None if load_all else self.n_spikes_amplitudes, + total=None if load_all else self.n_spikes_amplitudes_total, + ) + values = self.bounded_session.attributes['amplitudes.npy'].reshape(-1) + output = [] + if None in cluster_ids: + ids = eligible_ids() + ids = ids[~np.isin(self.supervisor.clustering.spike_clusters[ids], active)] + ids = _sample_spikes_evenly(ids, self.n_spikes_amplitudes_background) + ids = ids[np.isfinite(values[ids])] + output.append( + Bunch(amplitudes=values[ids], spike_ids=ids, spike_times=spike_times(ids)) + ) + for ids, count in zip(eligible, counts): + ids = _sample_spikes_evenly(ids, int(count)) + ids = ids[np.isfinite(values[ids])] + output.append( + Bunch(amplitudes=values[ids], spike_ids=ids, spike_times=spike_times(ids)) + ) + return output + + def create_evidence_amplitude_view(self): + array = self.bounded_session.attributes.get('amplitudes.npy') + if array is None or (array.ndim != 1 and array.shape[1:] != (1,)): + return + start, end = self.current_evidence_window + + def amplitudes(cluster_ids, load_all=False): + return self._evidence_amplitudes( + cluster_ids, load_all=load_all, scope=view.amplitude_scope + ) + + def split_is_eligible(): + return ( + len(self.supervisor.selected) == 1 + and not self.supervisor.selection.state.is_merge_mode + and 'amplitudes.npy' + in self.bounded_session.capabilities['predicate_split_attributes'] + ) + + view = EvidenceAmplitudeView( + amplitudes={'source': amplitudes}, + amplitudes_type='source', + duration=(end - start) / self.model.sample_rate, + split_is_eligible=split_is_eligible, + ) + view._controller = self + self._evidence_amplitude_views = getattr(self, '_evidence_amplitude_views', []) + self._evidence_amplitude_views.append(view) + self._update_amplitude_scope(view) + + @connect(sender=self.supervisor) + def on_select(sender, cluster_ids, **kwargs): + view.clear_amplitude_split_threshold() + self.supervisor._update_graphical_split_action() + + @connect(sender=self.supervisor) + def on_bounded_state_changed(sender): + view.clear_amplitude_split_threshold() + + @connect(event='amplitude_split_preview_changed', sender=view) + def on_preview(sender, **kwargs): + self.supervisor._update_graphical_split_action() + + @connect(event='view_attached', sender=view) + def on_attached(sender, gui): + def visibility_changed(visible): + if not visible: + view.clear_amplitude_split_threshold() + self.supervisor._update_graphical_split_action() + + view.dock.visibilityChanged.connect(visibility_changed) + + @connect(event='selected_channel_changed') + @connect(event='selected_feature_changed') + def on_context_changed(sender): + if getattr(sender, 'gui', None) is self.supervisor.gui: + view.clear_amplitude_split_threshold() + + @connect(event='close_view', sender=view) + def on_close(view_, gui): + view.clear_amplitude_split_threshold() + self._evidence_amplitude_views.remove(view) + for callback in ( + on_select, + on_bounded_state_changed, + on_preview, + on_context_changed, + on_attached, + on_close, + ): + unconnect(callback) + self.supervisor._update_graphical_split_action() + + return view + + def _update_amplitude_scope(self, view): + start, end = self.current_evidence_window + overview = view.amplitude_scope == 'overview' + view.duration = self.model.duration if overview else (end - start) / self.model.sample_rate + if overview: + view.window_caption = 'Sampled overview | All retained evidence windows' + if hasattr(view, 'window_label'): + view.window_label.setText(view.window_caption) + else: + view.set_window_context( + self.evidence_window_index, + start / self.model.sample_rate, + end / self.model.sample_rate, + ) + view.ex_status = ( + 'All retained windows; original recording clock; sampled source amplitudes.' + if overview + else self._window_status() + ) + if hasattr(view, 'time_axis_label'): + view.time_axis_label.setText( + 'Recording time (seconds)' if overview else 'Seconds from window start' + ) + view.canvas.panzoom.reset() + if view.cluster_ids: + view.plot() + view.update_status() + + def _window_status(self): + start, end = self.current_evidence_window + return ( + f'Evidence window {self.evidence_window_index}: {start / self.model.sample_rate:.3f}–' + f'{end / self.model.sample_rate:.3f} s; time axis is seconds from window start; nonfinite amplitudes omitted.' + ) + + def inspect_evidence_window(self, index): + index = int(index) + if not 0 <= index < len(self.evidence_windows): + raise ValueError('Evidence window index is outside the retained plan.') + self.evidence_window_index = index + for view in getattr(self, '_evidence_amplitude_views', ()): + view.amplitude_scope = 'window' + gui = getattr(self.supervisor, 'gui', None) + if gui is not None: + for view in gui.list_views('FeatureView'): + view.ex_status = self._window_status() + view.canvas.panzoom.reset() + if view.cluster_ids: + view.plot() + view.update_status() + + def create_gui(self, *args, **kwargs): + gui = super().create_gui(*args, **kwargs) + gui.setWindowTitle( + f'phy — Bounded evidence ({self.model.n_spikes:,} / ' + f'{self.bounded_session.manifest["num_spikes"]:,} spikes)' + ) + + @self.supervisor.actions.add(prompt=True, name='Split by source amplitude') + def split_by_source_amplitude(threshold): + """Split one selected unit below a source amplitude threshold (source units).""" + ids = self.supervisor.selected + if len(ids) != 1: + raise ValueError('Select one unit for a source-amplitude threshold split.') + return self.supervisor.split_source_amplitude(ids[0], float(threshold)) + + @gui.view_actions.add(prompt=True, prompt_default=lambda: self.evidence_window_index) + def inspect_evidence_window(index): + """Inspect a retained evidence window by its zero-based index.""" + self.inspect_evidence_window(index) + + split_action = self.supervisor.actions.get('split') + if split_action is not None: + self.supervisor._update_graphical_split_action() + split_action.setToolTip( + 'Selected-row splits cannot assign unseen source spikes; use a source-amplitude predicate.' + ) + predicate_action = self.supervisor.actions.get('Split by source amplitude') + if predicate_action is not None: + predicate_action.setEnabled( + 'amplitudes.npy' in self.bounded_session.capabilities['predicate_split_attributes'] + ) + + return gui diff --git a/phy/apps/large_recording/curation.py b/phy/apps/large_recording/curation.py new file mode 100644 index 000000000..3738cc778 --- /dev/null +++ b/phy/apps/large_recording/curation.py @@ -0,0 +1,1121 @@ +"""Semantic bounded curation and crash-consistent, immutable save generations.""" + +import copy +import hashlib +import json +import os +import shutil +import uuid +import warnings +from contextlib import contextmanager +from pathlib import Path + +import numpy as np + +from .metadata import POLICY, complete_schema, validate_schema +from .source import _time_base, hash_json + +MANIFEST = '_phy_curation_manifest.json' +BASE_ASSIGNMENTS = '_phy_spike_clusters_base.npy' +BOOLEAN_CAPABILITIES = ( + 'merge', + 'label', + 'split', + 'stored_waveforms', + 'raw_traces', + 'normalized_correlograms', +) + + +def _json_bytes(value): + return ( + json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False) + '\n' + ).encode() + + +def _digest(value): + return hashlib.sha256(_json_bytes(value)).hexdigest() + + +def _hash_file(path): + digest = hashlib.sha256() + with Path(path).open('rb') as stream: + while chunk := stream.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def _read_json(path): + with Path(path).open(encoding='utf8') as stream: + return json.load(stream) + + +def _write_json(path, value): + with Path(path).open('wb') as stream: + stream.write(_json_bytes(value)) + stream.flush() + os.fsync(stream.fileno()) + + +def _fsync_directory(path): + if os.name != 'nt': + descriptor = os.open(path, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _safe_file(root, name): + if ( + not isinstance(name, str) + or Path(name).name != name + or name in ('', '.', '..') + or any(c in name for c in ('\\', '\0', '\n', '\r')) + ): + raise ValueError('Manifest/checkpoint filename must be a plain local filename.') + path = Path(root) / name + if path.is_symlink(): + raise ValueError('Manifest/checkpoint files must not be symbolic links.') + return path + + +def _validate_digest(value): + if ( + not isinstance(value, str) + or len(value) != 64 + or any(c not in '0123456789abcdef' for c in value) + ): + raise ValueError('Invalid SHA-256 digest.') + return value + + +@contextmanager +def _commit_lock(root): + """Hold a process-safe OS lock; process exit releases it automatically.""" + with _safe_file(root, 'commit.lock').open('a+b') as stream: + if os.name == 'nt': + import msvcrt + + # Windows byte-range locking requires at least one byte in the file. + stream.seek(0, os.SEEK_END) + if stream.tell() == 0: + stream.write(b'\0') + stream.flush() + stream.seek(0) + msvcrt.locking(stream.fileno(), msvcrt.LK_LOCK, 1) + try: + yield + finally: + stream.seek(0) + msvcrt.locking(stream.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + fcntl.flock(stream.fileno(), fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(stream.fileno(), fcntl.LOCK_UN) + + +def _current_pointer(root): + try: + return _read_json(root / 'current.json') + except FileNotFoundError: + return None + + +def _integer(value, name): + if isinstance(value, bool) or not isinstance(value, (int, np.integer)) or value < 0: + raise ValueError(f'{name} must be a nonnegative integer.') + return int(value) + + +def _validate_capabilities(manifest): + capabilities = manifest.get('capabilities', {}) + if not isinstance(capabilities, dict): + raise ValueError('Bounded capabilities must be a mapping.') + if ( + set(capabilities) + - set(BOOLEAN_CAPABILITIES) + - { + 'full_attributes', + 'predicate_split_attributes', + } + ): + raise ValueError('Unsupported bounded capability declaration.') + result = {} + for name in BOOLEAN_CAPABILITIES: + value = capabilities.get(name, False) + if type(value) is not bool: + raise ValueError(f'Bounded capability {name!r} must be a boolean.') + result[name] = value + if result['split']: + raise ValueError('General selected-row splits are unsupported in bounded mode.') + for name in ('full_attributes', 'predicate_split_attributes'): + values = capabilities.get(name, []) + if ( + not isinstance(values, list) + or any(not isinstance(value, str) for value in values) + or len(set(values)) != len(values) + ): + raise ValueError(f'Bounded capability {name!r} must be a unique string list.') + result[name] = list(values) + if set(result['predicate_split_attributes']) - {'amplitudes.npy'}: + raise ValueError( + 'Unsupported bounded predicate attribute; only raw amplitudes are supported.' + ) + return result + + +def _field_name(field): + if ( + not isinstance(field, str) + or not field + or field in ('.', '..') + or any(char in field for char in ('/', '\\', '\0', '\n', '\r', '\t')) + ): + raise ValueError('Metadata fields must be nonempty safe local field names.') + return field + + +def normalize_metadata(metadata): + """Return JSON-compatible metadata with integer cluster keys.""" + if not isinstance(metadata, dict): + raise ValueError('Cluster metadata must be a field-to-cluster mapping.') + result = {} + for field, values in metadata.items(): + _field_name(field) + if not isinstance(values, dict): + raise ValueError('Invalid cluster metadata field.') + result[field] = {} + for key, value in values.items(): + try: + cluster = int(key) + except (ValueError, TypeError) as error: + raise ValueError('Invalid metadata cluster ID.') from error + if str(cluster) != str(key) or cluster < 0: + raise ValueError('Invalid metadata cluster ID.') + _json_bytes(value) + result[field][cluster] = copy.deepcopy(value) + return result + + +def validate_predicate(predicate): + """Validate a threshold on a canonical full-source scalar attribute.""" + if not isinstance(predicate, dict) or set(predicate) != { + 'attribute', + 'operator', + 'threshold', + 'nonfinite', + }: + raise ValueError('Threshold requires attribute/operator/threshold/nonfinite.') + if predicate['attribute'] not in ('amplitudes.npy', 'spike_templates.npy'): + raise ValueError('Unsupported full-source threshold attribute.') + if predicate['operator'] not in ('<', '<=', '>', '>='): + raise ValueError('Unsupported threshold comparator.') + threshold = predicate['threshold'] + if isinstance(threshold, bool) or not isinstance(threshold, (int, float)): + raise ValueError('Threshold must be a finite number.') + if not np.isfinite(threshold) or predicate['nonfinite'] not in ('keep', 'reject'): + raise ValueError('Threshold needs a finite threshold and explicit nonfinite policy.') + return copy.deepcopy(predicate) + + +def evaluate_predicate(values, predicate): + values = np.asarray(values).reshape(-1) + finite = np.isfinite(values) + if predicate['nonfinite'] == 'reject' and not finite.all(): + raise ValueError('Nonfinite source values violate threshold policy.') + operators = {'<': np.less, '<=': np.less_equal, '>': np.greater, '>=': np.greater_equal} + return finite & operators[predicate['operator']](values, predicate['threshold']) + + +def interpret_events(unit_ids, metadata, events): + """Validate event intent and resolve the final catalog and metadata.""" + active = set(map(int, unit_ids)) + if not active or any(i < 0 for i in active): + raise ValueError('Full source unit catalog must contain nonnegative IDs.') + allocated = set(active) + metadata = normalize_metadata(metadata) + if not isinstance(events, list): + raise ValueError('Semantic journal must be an ordered event list.') + next_id = max(active) + 1 + for event_index, event in enumerate(events): + if not isinstance(event, dict) or event.get('version') not in (1, 2): + raise ValueError('Unsupported semantic journal event version.') + operation = event.get('operation') + if operation == 'metadata_policy': + if ( + set(event) != {'version', 'operation', 'policy', 'schema'} + or event['version'] != 2 + or event['policy'] != POLICY + ): + raise ValueError('Malformed metadata policy migration event.') + validate_schema(event['schema']) + # Old resolved values have no reliable curator intent. Preserve only + # explicit label events on the current membership scope. + derived = set() + labels = {} + for previous in events[:event_index]: + op = previous['operation'] + if op == 'merge': + derived.add(previous['target']) + elif op == 'threshold_split': + derived.update(previous['children']) + elif op == 'label': + for unit in previous['ids']: + labels[(previous['field'], unit)] = previous['value'] + for field, values in metadata.items(): + for unit in derived & active: + values[unit] = copy.deepcopy(labels.get((field, unit))) + elif operation == 'merge': + if set(event) != {'version', 'operation', 'parents', 'target', 'metadata'}: + raise ValueError('Malformed merge event.') + parents = [_integer(i, 'Parent ID') for i in event['parents']] + target = _integer(event['target'], 'Target ID') + if target > np.iinfo(np.int32).max: + raise ValueError('Target cluster ID exceeds the current Phy loader namespace.') + if len(parents) < 2 or len(set(parents)) != len(parents): + raise ValueError('Merge requires at least two distinct complete units.') + if not set(parents) <= active or target in allocated or target < next_id: + raise ValueError('Merge references inactive units or colliding target ID.') + resolved = event['metadata'] + if not isinstance(resolved, dict) or any(not isinstance(f, str) for f in resolved): + raise ValueError('Merge metadata must contain resolved field values.') + if event['version'] == 2: + expected = dict.fromkeys(set(metadata) | {'group'}) + if resolved != expected: + raise ValueError('New membership must clear producer metadata and quality.') + _json_bytes(resolved) + for field in resolved: + _field_name(field) + active.difference_update(parents) + active.add(target) + allocated.add(target) + next_id = target + 1 + for values in metadata.values(): + for parent in parents: + values.pop(parent, None) + for field, value in resolved.items(): + metadata.setdefault(field, {})[target] = copy.deepcopy(value) + elif operation == 'label': + if set(event) != {'version', 'operation', 'field', 'ids', 'value'}: + raise ValueError('Malformed label event.') + ids = [_integer(i, 'Label ID') for i in event['ids']] + field = event['field'] + _field_name(field) + if not ids or len(set(ids)) != len(ids) or not set(ids) <= active: + raise ValueError('Label references inactive units or invalid metadata field.') + _json_bytes(event['value']) + values = metadata.setdefault(field, {}) + for cluster in ids: + values[cluster] = copy.deepcopy(event['value']) + elif operation == 'threshold_split': + if set(event) != { + 'version', + 'operation', + 'parent', + 'children', + 'predicate', + 'metadata', + }: + raise ValueError('Malformed threshold split event.') + parent = _integer(event['parent'], 'Parent ID') + children = [_integer(i, 'Child ID') for i in event['children']] + if any(child > np.iinfo(np.int32).max for child in children): + raise ValueError('Child cluster ID exceeds the current Phy loader namespace.') + validate_predicate(event['predicate']) + if parent not in active or len(children) != 2 or len(set(children)) != 2: + raise ValueError('Threshold split requires an active parent and two children.') + if any(i in allocated or i < next_id for i in children): + raise ValueError('Threshold split child ID collision.') + resolved = normalize_metadata(event['metadata']) + if event['version'] == 2: + expected = {field: dict.fromkeys(children) for field in set(metadata) | {'group'}} + if resolved != expected: + raise ValueError('New membership must clear producer metadata and quality.') + if any(not set(v) <= set(children) for v in resolved.values()): + raise ValueError('Split metadata must refer only to explicit children.') + active.remove(parent) + active.update(children) + allocated.update(children) + next_id = max(children) + 1 + for values in metadata.values(): + values.pop(parent, None) + for field, values in resolved.items(): + metadata.setdefault(field, {}).update(values) + else: + raise ValueError(f'Unsupported semantic operation: {operation!r}.') + return active, metadata, next_id + + +def project_assignments(assignments, events, attributes=None): + """Interpret a validated journal against a bounded array or full-source batch.""" + output = np.asarray(assignments, dtype=np.int64).reshape(-1).copy() + attributes = attributes or {} + for event in events: + if event['operation'] == 'merge': + output[np.isin(output, event['parents'])] = event['target'] + elif event['operation'] == 'threshold_split': + predicate = event['predicate'] + attribute = predicate['attribute'] + if attribute not in attributes: + raise ValueError(f'Full-source threshold attribute is unavailable: {attribute}.') + mask = output == event['parent'] + values = np.asarray(attributes[attribute]).reshape(-1) + if len(values) != len(output): + raise ValueError('Threshold attribute is not a scalar per-spike array.') + selected = np.zeros(len(output), dtype=bool) + selected[mask] = evaluate_predicate(values[mask], predicate) + output[mask & ~selected] = event['children'][0] + output[mask & selected] = event['children'][1] + return output + + +class BoundedSession: + """Bounded semantic state; reopen establishes a new undo baseline. + + Snapshot/restore integrate with the controller's undo stack. Saving publishes + one pointer after every generation file has been flushed and verified. + """ + + def __init__(self, dataset_path): + self.global_ids = None + self.base_assignments = None + self.attributes = {} + try: + self._initialize(dataset_path) + except BaseException: + self.close() + raise + + def _initialize(self, dataset_path): + self.path = Path(dataset_path) + if self.path.is_file(): + self.path = self.path.parent + self.manifest = _read_json(self.path / MANIFEST) + manifest = self.manifest + required = { + 'source', + 'sample_rate', + 'num_samples', + 'valid_intervals', + 'num_spikes', + 'bounded_num_spikes', + 'unit_ids', + 'counts', + 'next_cluster_id', + 'global_spike_ids_file', + 'windows_file', + } + if manifest.get('format') != 'phy_bounded_curation' or manifest.get('format_version') != 1: + raise ValueError('Unsupported bounded curation manifest version.') + if not required <= set(manifest): + raise ValueError( + 'Incomplete bounded curation manifest; regenerate with prepare-large.' + ) + source = manifest['source'] + if not isinstance(source, dict) or not {'identity', 'revision', 'path'} <= set(source): + raise ValueError('Bounded manifest needs source identity and assignment revision.') + if not isinstance(source['identity'], dict) or not isinstance(source['revision'], str): + raise ValueError('Invalid source identity or revision.') + identity = source['identity'] + identity_fields = {'format_version', 'acquisition', 'time_base', 'arrays', 'sha256'} + if ( + not identity_fields <= set(identity) + or identity.get('format_version') != 1 + or identity['sha256'] + != hash_json({k: v for k, v in identity.items() if k != 'sha256'}) + or len(source['revision']) != 64 + or any(c not in '0123456789abcdef' for c in source['revision']) + ): + raise ValueError('Invalid source coordinate identity or assignment revision digest.') + n = _integer(manifest['bounded_num_spikes'], 'Bounded spike count') + full_n = _integer(manifest['num_spikes'], 'Full spike count') + samples = _integer(manifest['num_samples'], 'Recording sample count') + if not n or n > full_n or not samples or not np.isfinite(manifest['sample_rate']): + raise ValueError('Invalid bounded counts or recording duration.') + if manifest['sample_rate'] <= 0: + raise ValueError('Sampling rate must be positive.') + expected_time_base = { + 'sample_rate': manifest['sample_rate'], + 'num_samples': samples, + 'valid_intervals': manifest['valid_intervals'], + } + if manifest.get('time_base') is not None: + contract = _time_base(manifest['time_base']) + if contract['ticks_per_second'] != manifest['sample_rate']: + raise ValueError('Bounded tick frequency disagrees with sampling rate.') + expected_time_base['contract'] = contract + timestamps_hash = identity['time_base'].get('timestamps_sha256') + if timestamps_hash is not None: + if ( + not isinstance(timestamps_hash, str) + or len(timestamps_hash) != 64 + or any(c not in '0123456789abcdef' for c in timestamps_hash) + ): + raise ValueError('Invalid source timestamps digest.') + expected_time_base['timestamps_sha256'] = timestamps_hash + if identity['time_base'] != expected_time_base: + raise ValueError('Bounded time base disagrees with immutable source identity.') + self.capabilities = _validate_capabilities(manifest) + self.unit_ids = [_integer(i, 'Unit ID') for i in manifest['unit_ids']] + if any(unit > np.iinfo(np.int32).max for unit in self.unit_ids): + raise ValueError('Unit IDs exceed the current Phy loader namespace.') + if not self.unit_ids or len(set(self.unit_ids)) != len(self.unit_ids): + raise ValueError('Full unit catalog must contain distinct unit IDs.') + counts = np.asarray(manifest['counts']) + if ( + counts.shape != (len(self.unit_ids),) + or counts.dtype.kind not in 'iu' + or np.any(counts < 0) + or sum(map(int, counts)) != full_n + ): + raise ValueError('Full unit counts do not match the source spike count/catalog.') + from phy.cluster._propositions import MergePropositionCatalog + + MergePropositionCatalog((), (), retired_units=manifest.get('retired_units', {})) + if set(manifest.get('retired_units', {})) & set(map(str, self.unit_ids)): + raise ValueError('Retired units cannot remain in the active source catalog.') + self.base_next_cluster_id = _integer(manifest['next_cluster_id'], 'Next cluster ID') + if self.base_next_cluster_id <= max(self.unit_ids): + raise ValueError('Cluster allocator collides with the complete source catalog.') + intervals = np.asarray(manifest['valid_intervals']) + if ( + intervals.ndim != 2 + or intervals.shape[1] != 2 + or not len(intervals) + or intervals.dtype.kind not in 'iu' + ): + raise ValueError('Valid acquisition intervals must be integer [start, end) pairs.') + if ( + np.any(intervals[:, 0] < 0) + or np.any(intervals[:, 1] > samples) + or np.any(intervals[:, 1] <= intervals[:, 0]) + or np.any(intervals[1:, 0] < intervals[:-1, 1]) + ): + raise ValueError('Invalid or overlapping acquisition intervals.') + self.global_ids = np.load( + _safe_file(self.path, manifest['global_spike_ids_file']), mmap_mode='r' + ) + if self.global_ids.dtype != np.int64 or self.global_ids.shape != (n,): + raise ValueError('Global spike IDs must be a one-dimensional int64 mapping.') + if ( + np.any(self.global_ids < 0) + or np.any(self.global_ids >= full_n) + or np.any(self.global_ids[1:] <= self.global_ids[:-1]) + ): + raise ValueError('Global spike IDs must be sorted, unique source row indices.') + windows = np.load(_safe_file(self.path, manifest['windows_file']), mmap_mode='r') + if ( + windows.ndim != 2 + or windows.shape[1] != 2 + or not len(windows) + or windows.dtype != np.int64 + ): + raise ValueError('Evidence windows must be int64 sample intervals.') + if ( + np.any(windows[:, 0] < 0) + or np.any(windows[:, 1] > samples) + or np.any(windows[:, 1] <= windows[:, 0]) + or np.any(windows[1:, 0] < windows[:-1, 1]) + ): + raise ValueError('Invalid or overlapping evidence windows.') + for start, stop in windows: + if not any(left <= start and stop <= right for left, right in intervals): + raise ValueError('Evidence windows cross unsupported acquisition gaps.') + for name, digest in manifest.get('bounded_files', {}).items(): + _validate_digest(digest) + if _hash_file(_safe_file(self.path, name)) != digest: + raise ValueError(f'Bounded package integrity check failed: {name}.') + hashed_files = manifest.get('bounded_files', {}) + if not { + BASE_ASSIGNMENTS, + manifest['global_spike_ids_file'], + manifest['windows_file'], + 'spike_times.npy', + } <= set(hashed_files): + raise ValueError('Bounded manifest lacks required immutable array digests.') + self.base_assignments = np.load(self.path / BASE_ASSIGNMENTS, mmap_mode='r') + if ( + self.base_assignments.shape != (n,) + or self.base_assignments.dtype.kind not in 'iu' + or not set(np.unique(self.base_assignments)) <= set(self.unit_ids) + ): + raise ValueError('Immutable bounded assignments do not match the full catalog.') + self.metadata_schema = validate_schema( + manifest.get('metadata_schema', {'version': 1, 'tables': []}) + ) + if manifest.get('metadata_policy') not in (None, POLICY): + raise ValueError('Unsupported bounded metadata policy.') + self.base_metadata = normalize_metadata( + manifest.get('base_metadata', manifest.get('metadata', {})) + ) + if any(not set(values) <= set(self.unit_ids) for values in self.base_metadata.values()): + raise ValueError('Base metadata refers to units outside the full source catalog.') + bounded_times = np.load(self.path / 'spike_times.npy', mmap_mode='r').reshape(-1) + if ( + bounded_times.shape != (n,) + or bounded_times.dtype.kind not in 'iu' + or np.any(bounded_times[1:] < bounded_times[:-1]) + ): + raise ValueError('Bounded spike sample coordinates have invalid alignment/order.') + window_ids = np.searchsorted(windows[:, 0], bounded_times, side='right') - 1 + if np.any(window_ids < 0) or np.any(bounded_times >= windows[window_ids, 1]): + raise ValueError('Bounded spikes lie outside the recorded evidence windows.') + self.attributes = {} + for name in ('amplitudes.npy', 'spike_templates.npy'): + if (self.path / name).exists(): + self.attributes[name] = np.load(self.path / name, mmap_mode='r') + for name in self.capabilities['predicate_split_attributes']: + array = self.attributes.get(name) + if ( + array is None + or name not in identity['arrays'] + or not (array.ndim == 1 or array.shape[1:] == (1,)) + or len(array) != n + or array.dtype.kind not in 'iuf' + ): + raise ValueError( + f'Bounded predicate attribute is not available as a source scalar: {name}.' + ) + self.assignments = np.asarray(self.base_assignments, dtype=np.int64).copy() + self.metadata = copy.deepcopy(self.base_metadata) + self.events = [] + self.extra_documents = {} + if 'proposition_documents' not in manifest and any( + (self.path / name).exists() for name in ('curation.json', 'curation_review.json') + ): + raise ValueError( + 'Legacy bounded propositions have no preparation binding; reprepare the source.' + ) + self.proposition_documents = manifest.get('proposition_documents', {}) + if not isinstance(self.proposition_documents, dict) or set(self.proposition_documents) - { + 'curation.json', + 'curation_review.json', + }: + raise ValueError('Invalid bounded proposition document inventory.') + for name, item in self.proposition_documents.items(): + expected_name = ( + '_phy_curation_review_base.json' if name == 'curation_review.json' else name + ) + if ( + not isinstance(item, dict) + or set(item) != {'filename', 'sha256'} + or item['filename'] != expected_name + or hashed_files.get(expected_name) != item['sha256'] + ): + raise ValueError('Proposition documents must have immutable preparation digests.') + _validate_digest(item['sha256']) + value = _read_json(_safe_file(self.path, expected_name)) + if not isinstance(value, dict): + raise ValueError('Proposition documents must contain JSON objects.') + if name == 'curation_review.json': + self.extra_documents[name] = value + if ( + 'curation.json' in self.proposition_documents + and 'curation_review.json' not in self.extra_documents + ): + from phy.cluster._propositions import encode_review_mapping + + self.extra_documents['curation_review.json'] = encode_review_mapping( + {}, source_sha256=self.proposition_documents['curation.json']['sha256'] + ) + self.validate_proposition_documents() + self.next_cluster_id = self.base_next_cluster_id + self.checkpoint_path = None + self._expected_pointer = None + pointer_path = self.path / '.phy-curation' / 'current.json' + if pointer_path.exists(): + self._load_checkpoint(pointer_path) + if self.metadata_policy is None: + warnings.warn( + 'Legacy bounded metadata preserves inherited values; use ' + 'migrate_metadata() to adopt membership-scoped metadata.', + UserWarning, + stacklevel=2, + ) + + def validate_proposition_documents(self): + """Keep producer input and its inherited ledger bound to preparation.""" + for name, item in self.proposition_documents.items(): + path = _safe_file(self.path, item['filename']) + if not path.is_file() or _hash_file(path) != item['sha256']: + raise ValueError(f'Bounded proposition document changed: {name}.') + if 'proposition_documents' in self.manifest: + for name in ('curation.json',): + if name not in self.proposition_documents and (self.path / name).exists(): + raise ValueError('Unbound bounded proposition document.') + review = self.extra_documents.get('curation_review.json') + has_review = 'curation_review.json' in self.extra_documents + if has_review and 'proposition_documents' not in self.manifest: + raise ValueError( + 'Legacy bounded reviews have no preparation binding; reprepare the source.' + ) + if 'curation.json' in self.proposition_documents and not isinstance(review, dict): + raise ValueError('Bounded proposition checkpoints require a review ledger mapping.') + if has_review: + from phy.cluster._propositions import decode_review_mapping + + source, _ = decode_review_mapping(review) + expected = self.proposition_documents.get('curation.json', {}).get('sha256') + if ( + source.get('filename') != 'curation.json' + or expected is None + or source.get('sha256') != expected + ): + raise ValueError('Bounded review source filename/hash does not match preparation.') + + @property + def retired_units(self): + """Cumulative retirement history, without retargeting source suggestions.""" + result = copy.deepcopy(self.manifest.get('retired_units', {})) + # Events have already passed the complete-unit journal validator. + interpret_events(self.unit_ids, self.base_metadata, self.events) + for event in self.events: + if event['operation'] == 'merge': + for parent in event['parents']: + result[str(parent)] = {'operation': 'merge', 'replacements': [event['target']]} + elif event['operation'] == 'threshold_split': + result[str(event['parent'])] = { + 'operation': 'threshold_split', + 'replacements': list(event['children']), + } + return result + + def _validated_state(self, assignments, metadata, events, next_cluster_id): + self._validate_event_capabilities(events) + _, projected_metadata, minimum = interpret_events( + self.unit_ids, self.base_metadata, events + ) + projected = project_assignments(self.base_assignments, events, self.attributes) + values = np.asarray(assignments) + if values.dtype.kind not in 'iu' or values.shape != projected.shape: + raise ValueError('Checkpoint assignments have invalid shape or dtype.') + if not np.array_equal(values, projected): + raise ValueError('Checkpoint assignments disagree with semantic journal projection.') + if normalize_metadata(metadata) != projected_metadata: + raise ValueError('Checkpoint metadata disagrees with semantic journal projection.') + minimum = max(minimum, self.base_next_cluster_id) + next_cluster_id = _integer(next_cluster_id, 'Next cluster ID') + if next_cluster_id < minimum: + raise ValueError('Checkpoint allocator collides with journal IDs.') + return projected, projected_metadata, next_cluster_id + + def _load_checkpoint(self, pointer_path): + pointer = _read_json(pointer_path) + generation = _safe_file(pointer_path.parent / 'generations', pointer['generation']) + descriptor = _read_json(generation / 'checkpoint.json') + if _digest(descriptor) != pointer['sha256']: + raise ValueError('Checkpoint commit pointer digest is invalid.') + if descriptor.get('version') != 1 or descriptor.get('manifest_sha256') != _digest( + self.manifest + ): + raise ValueError('Checkpoint belongs to a different bounded manifest.') + for name, digest in descriptor['files'].items(): + _validate_digest(digest) + if _hash_file(_safe_file(generation, name)) != digest: + raise ValueError(f'Checkpoint corruption: {name}.') + if not {'spike_clusters.npy', 'journal.json', 'metadata.json'} <= set(descriptor['files']): + raise ValueError('Incomplete checkpoint file inventory.') + events = _read_json(generation / 'journal.json') + for event in events: + if event.get('operation') == 'threshold_split': + event['metadata'] = normalize_metadata(event['metadata']) + metadata = _read_json(generation / 'metadata.json') + assignments = np.load(generation / 'spike_clusters.npy', mmap_mode='r') + values, metadata, next_id = self._validated_state( + assignments, metadata, events, descriptor['next_cluster_id'] + ) + self.assignments, self.metadata = values, metadata + self.events = events + if 'metadata_policy' in descriptor and ( + descriptor['metadata_policy'] != self.metadata_policy + or descriptor.get('metadata_schema') != self.effective_metadata_schema + or descriptor.get('metadata_provenance') != self.metadata_provenance + ): + raise ValueError( + 'Checkpoint metadata policy/schema/provenance disagrees with journal.' + ) + self.next_cluster_id = next_id + self.checkpoint_path = generation + self._expected_pointer = copy.deepcopy(pointer) + documents = descriptor.get('extra_documents', []) + if ( + not isinstance(documents, list) + or any(not isinstance(name, str) for name in documents) + or len(set(documents)) != len(documents) + or not set(documents) <= set(descriptor['files']) + or set(documents) + & {'checkpoint.json', 'spike_clusters.npy', 'journal.json', 'metadata.json'} + ): + raise ValueError('Extra checkpoint documents must be in the hash inventory.') + self.extra_documents = { + name: _read_json(_safe_file(generation, name)) for name in documents + } + self.validate_proposition_documents() + + def snapshot(self): + return { + 'assignments': self.assignments.copy(), + 'metadata': copy.deepcopy(self.metadata), + 'events': copy.deepcopy(self.events), + 'next_cluster_id': self.next_cluster_id, + } + + def journal_state(self): + """Return a compact history prefix without copying bounded spike arrays. + + Commands append event dictionaries and never modify earlier events. The + tuple shares those immutable event records between history entries. + """ + return {'events': tuple(self.events), 'next_cluster_id': self.next_cluster_id} + + def restore_journal(self, state): + """Restore an active history prefix while preserving allocator high-water.""" + events = list(state['events']) + self._validate_event_capabilities(events) + _, metadata, minimum = interpret_events(self.unit_ids, self.base_metadata, events) + next_id = _integer(state['next_cluster_id'], 'Next cluster ID') + if next_id < max(minimum, self.base_next_cluster_id): + raise ValueError('History allocator collides with semantic journal IDs.') + assignments = project_assignments(self.base_assignments, events, self.attributes) + self.assignments, self.metadata, self.events = assignments, metadata, events + self.next_cluster_id = max(self.next_cluster_id, next_id) + + def close(self): + """Release file-backed bounded readers independently of the GUI model.""" + for array in (self.global_ids, self.base_assignments, *self.attributes.values()): + mapping = getattr(array, '_mmap', None) + if mapping is not None and not mapping.closed: + mapping.close() + + def restore(self, snapshot): + assignments, metadata, next_id = self._validated_state(**snapshot) + self.assignments, self.metadata = assignments, metadata + self.events = copy.deepcopy(snapshot['events']) + self.next_cluster_id = next_id + + def _append(self, event): + events = self.events + [event] + self._validate_event_capabilities(events) + _, metadata, next_id = interpret_events(self.unit_ids, self.base_metadata, events) + assignments = project_assignments(self.base_assignments, events, self.attributes) + self.assignments, self.metadata, self.events = assignments, metadata, events + self.next_cluster_id = max(self.next_cluster_id, next_id) + return copy.deepcopy(event) + + def _require_capability(self, operation, predicate=None): + if operation == 'threshold_split': + predicate = validate_predicate(predicate) + if predicate['attribute'] not in self.capabilities['predicate_split_attributes']: + raise ValueError( + 'Predicate split capability is unavailable in this bounded package.' + ) + elif not self.capabilities.get(operation, False): + raise ValueError( + f'{operation.capitalize()} capability is disabled in this bounded package.' + ) + + def _validate_event_capabilities(self, events): + policy = self.manifest.get('metadata_policy') == POLICY + for event in events: + if isinstance(event, dict): + operation = event.get('operation') + if operation == 'metadata_policy': + if policy: + raise ValueError('Metadata policy migration has already been applied.') + policy = True + elif operation in ('merge', 'label', 'threshold_split'): + if event.get('version') != (2 if policy else 1): + raise ValueError('Semantic event version disagrees with metadata policy.') + self._require_capability(operation, event.get('predicate')) + + @property + def metadata_policy(self): + return self._metadata_policy(self.events) + + def _metadata_policy(self, events): + return ( + POLICY + if ( + self.manifest.get('metadata_policy') == POLICY + or any(e['operation'] == 'metadata_policy' for e in events) + ) + else None + ) + + @property + def effective_metadata_schema(self): + return self._metadata_schema(self.events, self.metadata) + + def _metadata_schema(self, events, metadata): + schema = self.metadata_schema + for event in events: + if event['operation'] == 'metadata_policy': + schema = event['schema'] + return complete_schema(schema, metadata) + + @property + def metadata_provenance(self): + return self._metadata_provenance(self.events) + + def _metadata_provenance(self, events): + result = copy.deepcopy(self.manifest.get('metadata_provenance', {'history': []})) + metadata = copy.deepcopy(self.base_metadata) + for index, event in enumerate(events): + operation = event['operation'] + if operation in ('merge', 'threshold_split'): + parents = event['parents'] if operation == 'merge' else [event['parent']] + children = [event['target']] if operation == 'merge' else event['children'] + result.setdefault('history', []).append( + { + 'operation': operation, + 'source_revision': self.manifest['source']['revision'], + 'event_index': index, + **( + {'predicate': copy.deepcopy(event['predicate'])} + if operation == 'threshold_split' + else {} + ), + 'parents': parents, + 'children': children, + 'parent_metadata': { + field: {str(unit): values.get(unit) for unit in parents} + for field, values in metadata.items() + }, + 'policy': POLICY if event['version'] == 2 else 'legacy_inherited', + } + ) + elif operation == 'label' or operation == 'metadata_policy': + result.setdefault('history', []).append(copy.deepcopy(event)) + # Event metadata is already resolved and validated. Scan it once + # instead of replaying every prefix of a potentially long journal. + if operation in ('merge', 'threshold_split'): + for values in metadata.values(): + for parent in parents: + values.pop(parent, None) + for field, resolved in event['metadata'].items(): + values = {event['target']: resolved} if operation == 'merge' else resolved + metadata.setdefault(field, {}).update(copy.deepcopy(values)) + elif operation == 'label': + values = metadata.setdefault(event['field'], {}) + for unit in event['ids']: + values[unit] = copy.deepcopy(event['value']) + elif operation == 'metadata_policy': + _, metadata, _ = interpret_events( + self.unit_ids, self.base_metadata, events[: index + 1] + ) + return result + + def migrate_metadata(self, source_params=None): + """Explicitly clear ambiguous inherited values; retain label journal intent. + + The original manifest and generations remain immutable. Saving commits a + policy transition and schema in the same atomic generation as assignments. + Reopen a supervisor after migrating an already-open GUI session. + """ + if self.metadata_policy == POLICY: + return None + from .source import Source + + source_params = source_params or self.manifest.get('source', {}).get('path') + if not source_params: + raise ValueError('Migration requires source_params to recover original table names.') + with Source( + source_params, + duration_seconds=self.manifest['num_samples'] / self.manifest['sample_rate'], + valid_intervals=self.manifest['valid_intervals'], + ) as source: + if source.identity != self.manifest['source']['identity']: + raise ValueError('Migration source identity does not match preparation.') + if source.revision != self.manifest['source']['revision']: + raise ValueError('Migration source revision does not match preparation.') + if source.metadata != self.base_metadata: + raise ValueError('Migration base metadata does not match source metadata.') + schema = source.metadata_schema + return self._append( + {'version': 2, 'operation': 'metadata_policy', 'policy': POLICY, 'schema': schema} + ) + + def preflight_merge(self, parents, target, metadata=None): + self._require_capability('merge') + event = { + 'version': 2 if self.metadata_policy else 1, + 'operation': 'merge', + 'parents': [int(i) for i in parents], + 'target': _integer(target, 'Target ID'), + 'metadata': ( + dict.fromkeys(set(self.metadata) | {'group'}) + if self.metadata_policy + else copy.deepcopy(metadata or {}) + ), + } + if event['target'] < self.next_cluster_id: + raise ValueError('Merge target is below the complete catalog allocator.') + interpret_events(self.unit_ids, self.base_metadata, self.events + [event]) + return event + + def record_merge(self, parents, target, metadata=None): + return self._append(self.preflight_merge(parents, target, metadata)) + + def preflight_label(self, field, ids, value): + self._require_capability('label') + complete_schema(self.effective_metadata_schema, {**self.metadata, field: {}}) + event = { + 'version': 2 if self.metadata_policy else 1, + 'operation': 'label', + 'field': field, + 'ids': [int(i) for i in ids], + 'value': copy.deepcopy(value), + } + interpret_events(self.unit_ids, self.base_metadata, self.events + [event]) + return event + + def record_label(self, field, ids, value): + return self._append(self.preflight_label(field, ids, value)) + + def record_threshold_split(self, parent, children, predicate, metadata=None): + """Apply an explicitly enabled full-source scalar predicate.""" + self._require_capability('threshold_split', predicate) + children = [int(i) for i in children] + if any(i < self.next_cluster_id for i in children): + raise ValueError('Threshold children collide with the complete catalog allocator.') + return self._append( + { + 'version': 2 if self.metadata_policy else 1, + 'operation': 'threshold_split', + 'parent': int(parent), + 'children': children, + 'predicate': validate_predicate(predicate), + 'metadata': ( + {field: dict.fromkeys(children) for field in set(self.metadata) | {'group'}} + if self.metadata_policy + else normalize_metadata(metadata or {}) + ), + } + ) + + def preflight_split(self, parent, selected_id, remainder_id, predicate, metadata=None): + self._require_capability('threshold_split', predicate) + children = [int(remainder_id), int(selected_id)] + if any(i < self.next_cluster_id for i in children): + raise ValueError('Threshold children collide with the complete catalog allocator.') + event = { + 'version': 2 if self.metadata_policy else 1, + 'operation': 'threshold_split', + 'parent': int(parent), + 'children': children, + 'predicate': validate_predicate(predicate), + 'metadata': ( + {field: dict.fromkeys(children) for field in set(self.metadata) | {'group'}} + if self.metadata_policy + else normalize_metadata(metadata or {}) + ), + } + interpret_events(self.unit_ids, self.base_metadata, self.events + [event]) + project_assignments(self.base_assignments, self.events + [event], self.attributes) + return event + + def record_split(self, parent, selected_id, remainder_id, predicate, metadata=None): + return self._append( + self.preflight_split(parent, selected_id, remainder_id, predicate, metadata) + ) + + def save( + self, + assignments=None, + metadata=None, + events=None, + next_cluster_id=None, + extra_documents=None, + ): + """Validate, flush a generation, then atomically replace its commit pointer.""" + self.validate_proposition_documents() + assignments = self.assignments if assignments is None else assignments + metadata = self.metadata if metadata is None else metadata + events = self.events if events is None else events + next_cluster_id = self.next_cluster_id if next_cluster_id is None else next_cluster_id + values, metadata, next_id = self._validated_state( + assignments, metadata, events, next_cluster_id + ) + if extra_documents is not None and not isinstance(extra_documents, dict): + raise ValueError('Extra checkpoint documents must be a filename-to-JSON mapping.') + documents = copy.deepcopy(self.extra_documents) + if extra_documents is not None: + documents.update(copy.deepcopy(extra_documents)) + if not isinstance(documents, dict): + raise ValueError('Extra checkpoint documents must be a filename-to-JSON mapping.') + for name, value in documents.items(): + _safe_file(self.path, name) + if name in ('checkpoint.json', 'spike_clusters.npy', 'journal.json', 'metadata.json'): + raise ValueError('Extra document collides with a checkpoint state file.') + _json_bytes(value) + previous_documents = self.extra_documents + try: + self.extra_documents = documents + self.validate_proposition_documents() + finally: + self.extra_documents = previous_documents + root = self.path / '.phy-curation' + generations = root / 'generations' + generations.mkdir(parents=True, exist_ok=True) + _fsync_directory(root) + _fsync_directory(self.path) + with _commit_lock(root): + if _current_pointer(root) != self._expected_pointer: + raise ValueError( + 'Bounded curation checkpoint changed in another session; ' + 'reopen the dataset before saving to avoid losing its changes.' + ) + return self._save_generation(generations, values, metadata, events, next_id, documents) + + def _save_generation(self, generations, values, metadata, events, next_id, documents): + root = generations.parent + identifier = uuid.uuid4().hex + stage = generations / f'.{identifier}.tmp' + final = generations / identifier + stage.mkdir() + pointer_tmp = root / f'.current-{identifier}.tmp' + try: + np.save(stage / 'spike_clusters.npy', values) + # Windows _commit (os.fsync) requires a writable descriptor. + with (stage / 'spike_clusters.npy').open('r+b') as stream: + os.fsync(stream.fileno()) + _write_json(stage / 'journal.json', events) + _write_json(stage / 'metadata.json', metadata) + for name, value in documents.items(): + _write_json(stage / name, value) + descriptor = { + 'version': 1, + 'manifest_sha256': _digest(self.manifest), + 'next_cluster_id': next_id, + 'metadata_policy': self._metadata_policy(events), + 'metadata_schema': self._metadata_schema(events, metadata), + 'metadata_provenance': self._metadata_provenance(events), + 'files': { + name: _hash_file(stage / name) + for name in ('spike_clusters.npy', 'journal.json', 'metadata.json', *documents) + }, + 'extra_documents': sorted(documents), + } + _write_json(stage / 'checkpoint.json', descriptor) + _fsync_directory(stage) + os.replace(stage, final) + _fsync_directory(generations) + pointer = {'generation': identifier, 'sha256': _digest(descriptor)} + _write_json(pointer_tmp, pointer) + self.validate_proposition_documents() + os.replace(pointer_tmp, root / 'current.json') + _fsync_directory(root) + except BaseException: + shutil.rmtree(stage, ignore_errors=True) + pointer_tmp.unlink(missing_ok=True) + raise + self.assignments, self.metadata, self.events = values, metadata, copy.deepcopy(events) + self.next_cluster_id, self.checkpoint_path = next_id, final + self.extra_documents = documents + self._expected_pointer = pointer + return final diff --git a/phy/apps/large_recording/metadata.py b/phy/apps/large_recording/metadata.py new file mode 100644 index 000000000..446a3e018 --- /dev/null +++ b/phy/apps/large_recording/metadata.py @@ -0,0 +1,211 @@ +"""Lossless producer table layout and typed canonical unit metadata.""" + +import ast +import csv +import json +from pathlib import Path + +SCHEMA_FILE = '_phy_cluster_metadata_schema.json' +POLICY = 'membership_v1' + + +def normalize_value(value): + """Empty table cells represent missing values, including in typed JSON.""" + return None if value == '' else value + + +def cell(value): + if value is None: + return '' + if isinstance(value, (dict, list, bool)): + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False) + return str(value) + + +def _legacy_cell_matches(text, value): + if text == str(value): + return True + if not isinstance(value, (dict, list)): + return False + try: + # Old csv.writer used Python repr, while the JSON sidecar sorted keys + # and converted tuples to lists. Compare JSON meaning without exec or + # Python's bool/int equality conflating true with 1. + candidate = json.loads(json.dumps(ast.literal_eval(text), allow_nan=False)) + options = {'sort_keys': True, 'separators': (',', ':'), 'allow_nan': False} + return json.dumps(candidate, **options) == json.dumps(value, **options) + except (SyntaxError, TypeError, ValueError): + return False + + +def validate_schema(schema): + if ( + not isinstance(schema, dict) + or set(schema) != {'version', 'tables'} + or schema['version'] != 1 + ): + raise ValueError('Unsupported metadata table schema.') + if not isinstance(schema['tables'], list): + raise ValueError('Metadata schema tables must be an ordered list.') + names, fields = set(), set() + for table in schema['tables']: + if not isinstance(table, dict) or set(table) != {'filename', 'delimiter', 'columns'}: + raise ValueError('Invalid metadata table descriptor.') + name, columns = table['filename'], table['columns'] + if ( + not isinstance(name, str) + or Path(name).name != name + or '\\' in name + or any(c in name for c in '\0\n\r') + or not name.startswith('cluster_') + or Path(name).suffix not in ('.tsv', '.csv') + or Path(name).stem.casefold() == 'cluster_info' + or name.casefold() in names + ): + raise ValueError(f'Unsafe, reserved or colliding metadata filename: {name!r}.') + if table['delimiter'] != ('\t' if name.endswith('.tsv') else ','): + raise ValueError(f'Invalid metadata delimiter: {name}.') + if ( + not isinstance(columns, list) + or len(columns) < 2 + or columns.count('cluster_id') != 1 + or any( + not isinstance(f, str) or not f or any(c in f for c in '/\\\0\n\r\t') + for f in columns + ) + or len(set(columns)) != len(columns) + ): + raise ValueError(f'Invalid or duplicate metadata columns: {name}.') + for field in columns: + if field != 'cluster_id': + if field in fields: + raise ValueError(f'Duplicate metadata field {field!r} in {name}.') + fields.add(field) + names.add(name.casefold()) + return {'version': 1, 'tables': sorted(schema['tables'], key=lambda t: t['filename'])} + + +def complete_schema(schema, metadata): + if 'cluster_id' in metadata: + raise ValueError('cluster_id is reserved for the metadata unit index.') + schema = json.loads(json.dumps(validate_schema(schema))) + fields = {f for t in schema['tables'] for f in t['columns']} + for field in sorted(set(metadata) - fields): + schema['tables'].append( + { + 'filename': f'cluster_{field}.tsv', + 'delimiter': '\t', + 'columns': ['cluster_id', field], + } + ) + return validate_schema(schema) + + +def read_metadata(path): + path = Path(path) + schema_path = path / SCHEMA_FILE + declared = json.loads(schema_path.read_text()) if schema_path.exists() else None + tables, parsed = [], {} + filenames = sorted([*path.glob('cluster_*.tsv'), *path.glob('cluster_*.csv')]) + for filename in filenames: + if filename.stem == 'cluster_info': + continue + with filename.open(encoding='utf-8', newline='') as stream: + reader = csv.reader(stream, delimiter='\t' if filename.suffix == '.tsv' else ',') + columns = next(reader, []) + table = { + 'filename': filename.name, + 'delimiter': '\t' if filename.suffix == '.tsv' else ',', + 'columns': columns, + } + tables.append(table) + validate_schema({'version': 1, 'tables': tables}) + for field in columns: + if field != 'cluster_id': + parsed.setdefault(field, {}) + seen = set() + for row in reader: + if len(row) != len(columns): + raise ValueError(f'Malformed metadata row in {filename.name}.') + raw_id = row[columns.index('cluster_id')] + try: + unit = int(raw_id) + except ValueError as error: + raise ValueError(f'Invalid unit ID in {filename.name}: {raw_id!r}.') from error + if unit < 0 or unit in seen: + raise ValueError(f'Duplicate or negative unit {unit} in {filename.name}.') + seen.add(unit) + for field, value in zip(columns, row): + if field != 'cluster_id': + parsed.setdefault(field, {})[unit] = ( + value if declared is None else normalize_value(value) + ) + schema = validate_schema({'version': 1, 'tables': tables}) + if declared is not None and validate_schema(declared) != schema: + raise ValueError('Metadata table files or columns disagree with metadata schema.') + sidecar = path / '_phy_cluster_metadata.json' + if not sidecar.exists(): + if declared is not None: + raise ValueError('Metadata schema requires the canonical typed JSON sidecar.') + return parsed, schema + raw = json.loads(sidecar.read_text(encoding='utf-8')) + if not isinstance(raw, dict): + raise ValueError('Cluster metadata JSON must be a field mapping.') + canonical = {} + for field, values in raw.items(): + if not isinstance(values, dict): + raise ValueError('Cluster metadata fields must map integer cluster IDs to values.') + canonical[field] = {} + for key, value in values.items(): + unit = int(key) + if str(unit) != key or unit < 0: + raise ValueError('Cluster metadata IDs must be nonnegative canonical integers.') + canonical[field][unit] = value + complete_schema(schema, canonical) # Validate field-derived filenames as well. + for table in tables: + for field in table['columns']: + if field == 'cluster_id': + continue + left, right = parsed.get(field, {}), canonical.get(field, {}) + for unit in set(left) | set(right): + if declared is None and unit not in left and right.get(unit) is None: + continue # Legacy writer omitted rows for JSON null values. + if ( + declared is None + and unit in left + and unit in right + and _legacy_cell_matches(left[unit], right[unit]) + ): + continue # Legacy csv.writer used Python representations for typed values. + if unit not in left or unit not in right or cell(left[unit]) != cell(right[unit]): + raise ValueError( + f'Metadata conflict in {table["filename"]}, field {field}, unit {unit}.' + ) + if declared is not None and set(canonical) != set(parsed): + raise ValueError('Canonical metadata fields disagree with table schema.') + return canonical, complete_schema(schema, canonical) + + +def write_metadata(path, metadata, schema): + """Keep source basenames, delimiters and column order, including blank rows.""" + schema = complete_schema(schema, metadata) + filenames = [table['filename'] for table in schema['tables']] + for filename in [*filenames, '_phy_cluster_metadata.json', SCHEMA_FILE]: + if (path / filename).is_symlink(): + raise ValueError(f'Metadata output must not overwrite a symbolic link: {filename}.') + for table in schema['tables']: + fields = [f for f in table['columns'] if f != 'cluster_id'] + units = sorted({u for f in fields for u in metadata.get(f, {})}) + with (path / table['filename']).open('w', encoding='utf-8', newline='') as stream: + writer = csv.writer(stream, delimiter=table['delimiter']) + writer.writerow(table['columns']) + for unit in units: + writer.writerow( + [ + unit if f == 'cluster_id' else cell(metadata.get(f, {}).get(unit)) + for f in table['columns'] + ] + ) + (path / '_phy_cluster_metadata.json').write_text(json.dumps(metadata, sort_keys=True) + '\n') + (path / SCHEMA_FILE).write_text(json.dumps(schema, sort_keys=True) + '\n') + return schema diff --git a/phy/apps/large_recording/model.py b/phy/apps/large_recording/model.py new file mode 100644 index 000000000..9f0de0389 --- /dev/null +++ b/phy/apps/large_recording/model.py @@ -0,0 +1,65 @@ +"""Bounded template loading without dense caches indexed by cluster namespace.""" + +from types import MappingProxyType + +import numpy as np +from phylib.io.model import TemplateModel + + +class BoundedTemplateModel(TemplateModel): + """Load committed assignments and original templates as separate concepts. + + The controller derives live unit evidence from Supervisor assignments. + phylib's eager merged-cluster waveform cache allocates by the maximum + absolute ID and does not track subsequent GUI edits; bounded mode does not + expose that cache as current scientific evidence. + """ + + def __init__(self, *args, session, **kwargs): + self._bounded_session = session + super().__init__(*args, **kwargs) + self.sparse_clusters = None + + def freeze_source(self): + """Prevent public model edits from bypassing semantic curation.""" + names = set() + for name, value in vars(self).items(): + if not name.startswith('_') and isinstance(value, np.ndarray): + value.flags.writeable = False + names.add(name) + self.metadata = MappingProxyType( + {name: MappingProxyType(values) for name, values in self.metadata.items()} + ) + names.update( + ( + 'metadata', + 'save_spike_clusters', + 'save_metadata', + 'save_spikes_subset_waveforms', + 'sample_rate', + 'duration', + 'n_channels', + 'n_channels_dat', + 'n_samples_waveforms', + 'dtype', + 'offset', + ) + ) + self._protected_source_names = frozenset(names) + + def __setattr__(self, name, value): + if name in getattr(self, '_protected_source_names', ()): + raise ValueError('Bounded model source is read-only; use semantic curation commands.') + super().__setattr__(name, value) + + def _load_spike_clusters(self): + return self._bounded_session.assignments.copy() + + def get_merge_map(self): + return {}, np.array([], dtype=np.int64) + + def cluster_waveforms(self): + return None + + def get_cluster_mean_waveforms(self, *args, **kwargs): + raise ValueError('Use the controller waveform/template providers for live bounded units.') diff --git a/phy/apps/large_recording/parameters.py b/phy/apps/large_recording/parameters.py new file mode 100644 index 000000000..b76a300ca --- /dev/null +++ b/phy/apps/large_recording/parameters.py @@ -0,0 +1,119 @@ +"""Maintain replay-owned parameter overrides without rewriting user Python.""" + +import ast +import io +import tokenize + +BEGIN = '# BEGIN PHY CURATION PARAMETERS' +END = '# END PHY CURATION PARAMETERS' +KEYS = ('num_samples', 'valid_intervals', 'dat_path') + + +def _assignments(nodes): + values = {} + for node, key in zip(nodes, KEYS, strict=True): + if ( + not isinstance(node, ast.Assign) + or len(node.targets) != 1 + or not isinstance(node.targets[0], ast.Name) + or node.targets[0].id != key + or node.col_offset != 0 + ): + raise ValueError('Not a generated parameter assignment.') + values[key] = ast.literal_eval(node.value) + return values + + +def _without_managed_block(text): + markers = [] + for token in tokenize.generate_tokens(io.StringIO(text).readline): + if token.type == tokenize.COMMENT and token.start[1] == 0 and token.string in (BEGIN, END): + markers.append((token.string, token.start[0])) + if not markers: + return text + if len(markers) != 2 or [item[0] for item in markers] != [BEGIN, END]: + raise ValueError('Malformed Phy curation parameter block.') + lines = text.splitlines(keepends=True) + first, last = markers[0][1] - 1, markers[1][1] - 1 + if ''.join(lines[last + 1 :]).strip(): + raise ValueError('Phy curation parameter block must be the final content in params.py.') + nodes = ast.parse(''.join(lines[first + 1 : last])).body + if len(nodes) != 3: + raise ValueError('Unexpected code inside Phy curation parameter block.') + try: + _assignments(nodes) + except (ValueError, TypeError) as error: + raise ValueError('Unexpected code inside Phy curation parameter block.') from error + return ''.join(lines[:first]) + + +def _strip_legacy_suffix(text, overrides): + """Remove only recognizable duplicated trailing triples from the old writer.""" + while True: + nodes = ast.parse(text).body + if len(nodes) < 6: + return text + tail = nodes[-3:] + try: + values = _assignments(tail) + except (ValueError, TypeError): + return text + if values != overrides: + return text + previous = { + target.id + for node in nodes[:-3] + if isinstance(node, ast.Assign) + for target in node.targets + if isinstance(target, ast.Name) + } + if not set(KEYS) <= previous: + return text + lines = text.splitlines(keepends=True) + if any(node.lineno != node.end_lineno for node in tail): + return text + if [node.lineno for node in tail] != list(range(tail[0].lineno, tail[0].lineno + 3)): + return text + if any( + lines[node.lineno - 1].rstrip('\r\n') != f'{key} = {values[key]!r}' + for node, key in zip(tail, KEYS, strict=True) + ): + return text + if ''.join(lines[tail[-1].end_lineno :]).strip(): + return text + # The old writer inserted one blank line immediately before every triple. + stop = tail[0].lineno - 1 + if stop and not lines[stop - 1].strip(): + stop -= 1 + text = ''.join(lines[:stop]) + + +def rewrite_replay_params(text, *, num_samples, valid_intervals, dat_path): + """Return source text with one stable, final block of replay-owned values. + + Original assignments may feed later user expressions and therefore remain + intact. Only this marked block and confidently identified legacy generated + suffixes are replaced; imports, comments and executable source are retained. + """ + if not isinstance(text, str): + raise TypeError('Parameter source must be text.') + ast.parse(text) # Validate syntax without executing user source again. + overrides = dict(zip(KEYS, (num_samples, valid_intervals, dat_path), strict=True)) + # Recognize old suffix values before normalizing legacy no-raw-data strings. + prefix = _strip_legacy_suffix(_without_managed_block(text), overrides) + paths = dat_path if isinstance(dat_path, (list, tuple)) else [dat_path] + if all( + path is None or (isinstance(path, str) and path.strip() in ('', 'None')) for path in paths + ): + overrides['dat_path'] = None + newline = '\r\n' if '\r\n' in text else '\n' + if prefix and not prefix.endswith(newline * 2): + prefix += newline if prefix.endswith(newline) else newline * 2 + block = newline.join([BEGIN, *(f'{key} = {ascii(overrides[key])}' for key in KEYS), END, '']) + return prefix + block + + +def rewrite_replay_params_bytes(contents, **overrides): + """Preserve a Python encoding cookie/BOM as well as original line endings.""" + encoding, _ = tokenize.detect_encoding(io.BytesIO(contents).readline) + return rewrite_replay_params(contents.decode(encoding), **overrides).encode(encoding) diff --git a/phy/apps/large_recording/prepare.py b/phy/apps/large_recording/prepare.py new file mode 100644 index 000000000..eddbea5eb --- /dev/null +++ b/phy/apps/large_recording/prepare.py @@ -0,0 +1,696 @@ +"""Publish a self-contained bounded evidence package from a full sorting.""" + +import json +import os +import shutil +import tempfile +from pathlib import Path + +import numpy as np + +from .metadata import POLICY, SCHEMA_FILE, write_metadata +from .source import ( + PER_SPIKE, + STATIC, + WAVEFORMS, + Source, + close_array, + hash_file, + hash_json, + publish_directory, +) +from .summaries import build_summaries, scalar_attributes, summary_size + + +def _row_bounds(times, windows): + return [ + ( + int(np.searchsorted(times, start, side='left')), + int(np.searchsorted(times, end, side='left')), + ) + for start, end in windows + ] + + +def _union(windows, block_starts=()): + result = [] + for start, end in sorted(windows, key=lambda row: (int(row[0]), int(row[1]))): + if result and ( + start < result[-1][1] or (start == result[-1][1] and start not in block_starts) + ): + result[-1][1] = max(result[-1][1], int(end)) + else: + result.append([int(start), int(end)]) + return result + + +def _selected_waveform_count(source, windows): + ids = source.arrays.get('_phy_spikes_subset.spikes.npy') + if ids is None: + return 0 + return sum( + int(np.searchsorted(ids, stop) - np.searchsorted(ids, start)) + for start, stop in _row_bounds(source.arrays['spike_times.npy'].reshape(-1), windows) + ) + + +def _producer_windows(source): + """Validate optional producer plans without loading waveform tensors.""" + stem = '_phy_spikes_subset.' + bounds = source.arrays.get(stem + 'window_bounds.npy') + types = source.arrays.get(stem + 'window_types.npy') + strata = source.arrays.get(stem + 'window_strata.npy') + if bounds is None: + if types is not None or strata is not None: + raise ValueError('Producer window types/strata require window_bounds.') + return None, None + if bounds.ndim != 2 or bounds.shape[1] != 2 or bounds.dtype.kind not in 'iu': + raise ValueError('Producer window_bounds must be an integer (W, 2) array.') + for label, values in (('types', types), ('strata', strata)): + if values is not None and ( + values.ndim != 1 or len(values) != len(bounds) or values.dtype.kind not in 'iu' + ): + raise ValueError(f'Producer window_{label} must be an aligned integer vector.') + manifest_path = source.path / (stem + 'manifest.json') + info = json.loads(manifest_path.read_text()) if manifest_path.exists() else {} + coordinates = info.get('recording', {}).get('window_bounds_coordinates') + clock = getattr(source, 'time_base', None) + kind = clock['kind'] if clock else 'acquisition_samples' + if coordinates is not None and coordinates != kind: + raise ValueError('Producer window_bounds_coordinates disagrees with the source time_base.') + if kind == 'elapsed_ticks' and coordinates is None: + raise ValueError('Elapsed-tick producer windows require window_bounds_coordinates.') + previous = -1 + for index, (first, last) in enumerate(bounds): + first, last = int(first), int(last) + block = int(np.searchsorted(source.valid_intervals[:, 0], first, side='right') - 1) + if first < previous or first < 0 or last <= first or last > source.num_samples: + raise ValueError( + 'Producer window bounds must be sorted, positive, and inside recording.' + ) + if block < 0 or last > source.valid_intervals[block, 1]: + raise ValueError('Producer window crosses an acquisition block or gap.') + previous = first + code = int(types[index]) if types is not None else 0 + stratum = int(strata[index]) if strata is not None else (-1 if code == 1 else 0) + if code not in (0, 1): + raise ValueError('Producer window_types codes must be 0 (survey) or 1 (rescue).') + if stratum < -1 or (code == 1 and stratum != -1) or (code == 0 and stratum < 0): + raise ValueError( + 'Producer window_strata must be nonnegative for survey and -1 for rescue.' + ) + if source.arrays.keys() >= WAVEFORMS: + union = _union(bounds, set(source.valid_intervals[:, 0])) + if _selected_waveform_count(source, union) != len(source.arrays[stem + 'spikes.npy']): + raise ValueError('Stored waveform rows fall outside the producer window plan.') + provenance = { + 'sha256': hash_json( + { + 'files': { + name: hash_file(source.path / name) + for name in ( + stem + 'window_bounds.npy', + stem + 'window_types.npy', + stem + 'window_strata.npy', + ) + if name in source.arrays + }, + 'coordinates': coordinates, + } + ), + 'window_bounds_coordinates': coordinates, + 'num_windows': len(bounds), + } + return (bounds, types, strata), provenance + + +def _distributed_indices(count): + """Visit sorted windows in a reproducible dyadic spread, using constant memory.""" + if not count: + return + bits = (count - 1).bit_length() + for value in range(1 << bits): + index, remainder = 0, value + for _ in range(bits): + index = (index << 1) | (remainder & 1) + remainder >>= 1 + if index < count: + yield index + + +def prepare_large( + params_path, + output_dir, + duration_seconds=None, + valid_intervals=None, + max_spikes=2000000, + max_bytes=536870912, + window_seconds=30, + stratum_seconds=3600, + seed=0, + batch_size=100000, + bin_seconds=3600, + window_policy='auto', +): + """Choose complete shared windows, write bounded arrays, and publish atomically. + + Survey windows use a seeded offset within each acquisition stratum. A second + pass tries one first-spike rescue window for every unit without evidence. + Candidate windows that exceed either budget are explicitly omitted; spikes + within accepted windows are never independently thinned. + """ + with Source(params_path, duration_seconds, valid_intervals, batch_size) as source: + return _prepare_source( + source, + output_dir, + max_spikes, + max_bytes, + window_seconds, + stratum_seconds, + seed, + batch_size, + bin_seconds, + window_policy, + ) + + +def _prepare_source( + source, + output_dir, + max_spikes, + max_bytes, + window_seconds, + stratum_seconds, + seed, + batch_size, + bin_seconds, + window_policy, +): + if window_policy not in ('auto', 'producer', 'generated'): + raise ValueError('window_policy must be auto, producer, or generated.') + producer, producer_provenance = _producer_windows(source) + available = ( + producer is not None and bool(len(producer[0])) and source.arrays.keys() >= WAVEFORMS + ) + if window_policy == 'producer' and not available: + raise ValueError( + 'Producer policy requires nonempty window bounds and stored waveform data.' + ) + resolved_policy = 'producer' if available and window_policy != 'generated' else 'generated' + output_dir = Path(output_dir).resolve() + missing_static = { + 'templates.npy', + 'channel_map.npy', + 'channel_positions.npy', + } - source.arrays.keys() + if missing_static: + raise ValueError(f'Bounded GUI preparation requires: {", ".join(sorted(missing_static))}.') + if output_dir.exists(): + raise ValueError(f'Output already exists: {output_dir}.') + if output_dir == source.path or source.path in output_dir.parents: + raise ValueError('Bounded output must be outside the source dataset directory.') + if int(max_spikes) != max_spikes or max_spikes <= 0 or max_bytes <= 0: + raise ValueError('max_spikes and max_bytes must be positive budgets.') + widths = [] + for label, seconds in ( + ('window_seconds', window_seconds), + ('stratum_seconds', stratum_seconds), + ): + width = float(seconds) * source.sample_rate + if ( + not np.isfinite(width) + or width < 1 + or not np.isclose(width, round(width), rtol=0, atol=1e-6) + ): + raise ValueError(f'{label} must specify a positive integer sample width.') + widths.append(int(round(width))) + width, stratum = widths + times = source.arrays['spike_times.npy'].reshape(-1) + assignments = source.arrays['spike_clusters.npy'].reshape(-1) + summary_bytes = summary_size(source, bin_seconds)[0] + producer_documents = [ + source.path / name + for name in ('curation.json', 'curation_review.json') + if (source.path / name).is_file() + ] + producer_hashes = {path.name: hash_file(path) for path in producer_documents} + fixed_bytes = ( + sum( + path.stat().st_size * (2 if path.name == 'curation_review.json' else 1) + for path in producer_documents + ) + + summary_bytes + + sum((source.path / name).stat().st_size for name in source.arrays.keys() & STATIC) + ) + # Reserve headers, manifest, params, metadata, and window records explicitly. + metadata_bytes = sum( + len(str(value).encode('utf-8')) + 64 + for values in source.metadata.values() + for value in values.values() + ) + overhead = 32768 + metadata_bytes + 1024 * len(source.arrays) + fixed_bytes += overhead + # Candidate records remain present even when rejected. Reserve them before + # selection so an omission cannot silently exceed the byte budget. + if resolved_policy == 'producer': + fixed_bytes += len(producer[0]) * 512 + row_bytes = 16 # int64 global row mapping and baseline assignment + for name in source.arrays.keys() & PER_SPIKE: + array = source.arrays[name] + row_bytes += array.dtype.itemsize * int(np.prod(array.shape[1:], dtype=np.int64)) + waveform_sidecars = source.arrays.keys() & {'_phy_spikes_subset.sample_index.npy'} + waveform_bytes = 0 + if source.arrays.keys() >= WAVEFORMS: + for name in WAVEFORMS | waveform_sidecars: + array = source.arrays[name] + waveform_bytes += array.dtype.itemsize * int(np.prod(array.shape[1:], dtype=np.int64)) + if fixed_bytes > max_bytes: + raise ValueError( + f'Output budget exceeded before selecting evidence: estimated ' + f'{fixed_bytes} bytes; increase max_bytes or bin_seconds.' + ) + windows, plan, omitted = [], [], [] + rng = np.random.default_rng(seed) + + def accept(start, end, kind, segment, stratum_id): + candidate = _union([*windows, [start, end]], set(source.valid_intervals[:, 0])) + n = sum(stop - first for first, stop in _row_bounds(times, candidate)) + n_waveforms = _selected_waveform_count(source, candidate) + size = ( + fixed_bytes + + n * row_bytes + + n_waveforms * waveform_bytes + + (len(candidate) + len(plan) + len(omitted) + 1) * 256 + ) + record = { + 'start': int(start), + 'end': int(end), + 'kind': kind, + 'segment': 0, + 'block': int(segment), + 'stratum': int(stratum_id), + } + if (len(plan) + len(omitted) + 1) * 256 + fixed_bytes > max_bytes: + raise ValueError('Window plan metadata exceeds max_bytes; increase max_bytes.') + if n > max_spikes or size > max_bytes: + omitted.append( + {**record, 'reason': 'spike_budget' if n > max_spikes else 'byte_budget'} + ) + return False + windows[:] = candidate + plan.append(record) + return True + + producer_estimate = None + if resolved_policy == 'producer': + bounds, types, strata = producer + # Counting intersections uses sparse ID binary searches, never waveform data. + union = _union(bounds, set(source.valid_intervals[:, 0])) + all_spikes = sum(stop - first for first, stop in _row_bounds(times, union)) + all_waveforms = _selected_waveform_count(source, union) + producer_estimate = { + 'spikes': all_spikes, + 'waveform_rows': all_waveforms, + 'bytes': fixed_bytes + + all_spikes * row_bytes + + all_waveforms * waveform_bytes + + (len(bounds) + len(union)) * 256, + } + del union + for index in _distributed_indices(len(bounds)): + first, last = map(int, bounds[index]) + block = int(np.searchsorted(source.valid_intervals[:, 0], first, side='right') - 1) + code = int(types[index]) if types is not None else 0 + kind = 'survey' if code == 0 else 'rescue' + stratum_id = int(strata[index]) if strata is not None else (-1 if code else index) + accept(first, last, kind, block, stratum_id) + else: + # Candidate enumeration is scalar and bounded by output-window metadata, + # rather than constructing recording-length arrays. + for segment, (first, last) in enumerate(source.valid_intervals): + for start in range(int(first), int(last), stratum): + end = min(start + stratum, int(last)) + length = min(width, end - start) + offset = int(rng.integers(0, end - start - length + 1)) + accept( + start + offset, start + offset + length, 'survey', segment, start // stratum + ) + if (len(plan) + len(omitted)) * 256 + fixed_bytes > max_bytes: + raise ValueError( + 'Window plan metadata exceeds max_bytes; increase stratum_seconds.' + ) + evidence = set() + for first, stop in _row_bounds(times, windows): + for start in range(first, stop, batch_size): + evidence.update(np.unique(assignments[start : min(start + batch_size, stop)]).tolist()) + missing = set(source.unit_ids.tolist()) - evidence + for batch in source.iter_batches(): + if window_policy == 'producer': + break + if not missing: + break + units, positions = np.unique(batch['spike_clusters.npy'], return_index=True) + for unit, position in zip(units.tolist(), positions.tolist(), strict=True): + if unit not in missing: + continue + sample = int(batch['spike_times.npy'][position]) + segment = int(np.searchsorted(source.valid_intervals[:, 0], sample, side='right') - 1) + first, last = source.valid_intervals[segment] + start = max(int(first), min(sample - width // 2, int(last) - width)) + end = min(start + width, int(last)) + accept(start, end, 'rescue', segment, -1) + # One deterministic rescue attempt per unit, including rejected attempts. + missing.remove(unit) + row_bounds = _row_bounds(times, windows) + n_bounded = sum(stop - first for first, stop in row_bounds) + if not n_bounded: + raise ValueError( + 'No bounded evidence fits the budgets; increase max_spikes/max_bytes ' + 'or shorten window_seconds (tied spikes remain together).' + ) + output_dir.parent.mkdir(parents=True, exist_ok=True) + estimated_bytes = ( + fixed_bytes + + n_bounded * row_bytes + + _selected_waveform_count(source, windows) * waveform_bytes + + (len(windows) + len(plan) + len(omitted)) * 256 + ) + free = shutil.disk_usage(output_dir.parent).free + if free < estimated_bytes: + raise OSError( + f'Insufficient disk space for bounded preparation: estimated {estimated_bytes:,} bytes, {free:,} available.' + ) + temporary = Path( + tempfile.mkdtemp(prefix=f'.{output_dir.name}.prepare-', dir=output_dir.parent) + ) + global_ids, base, destinations = None, None, {} + try: + global_ids = np.lib.format.open_memmap( + temporary / '_phy_spike_global_ids.npy', mode='w+', dtype=np.int64, shape=(n_bounded,) + ) + for name in source.arrays.keys() & PER_SPIKE: + array = source.arrays[name] + destinations[name] = np.lib.format.open_memmap( + temporary / name, mode='w+', dtype=array.dtype, shape=(n_bounded, *array.shape[1:]) + ) + base = np.lib.format.open_memmap( + temporary / '_phy_spike_clusters_base.npy', + mode='w+', + dtype=np.int64, + shape=(n_bounded,), + ) + cursor = 0 + bounded_counts = {int(unit): 0 for unit in source.unit_ids} + for first, stop in row_bounds: + for start in range(first, stop, batch_size): + end = min(start + batch_size, stop) + length = end - start + global_ids[cursor : cursor + length] = np.arange(start, end, dtype=np.int64) + base[cursor : cursor + length] = assignments[start:end] + for name, target in destinations.items(): + target[cursor : cursor + length] = source.arrays[name][start:end] + ids, counts = np.unique(assignments[start:end], return_counts=True) + for unit, count in zip(ids.tolist(), counts.tolist(), strict=True): + bounded_counts[unit] += count + cursor += length + for array in (global_ids, base, *destinations.values()): + array.flush() + for name in source.arrays.keys() & STATIC: + shutil.copyfile(source.path / name, temporary / name) + if source.arrays.keys() >= WAVEFORMS: + _copy_waveforms(source, temporary, global_ids, row_bounds) + waveform_counts = {int(unit): 0 for unit in source.unit_ids} + waveform_ids_path = temporary / '_phy_spikes_subset.spikes.npy' + if waveform_ids_path.exists(): + waveform_ids = np.load(waveform_ids_path, mmap_mode='r') + for start in range(0, len(waveform_ids), batch_size): + ids, counts = np.unique( + base[waveform_ids[start : start + batch_size]], return_counts=True + ) + for unit, count in zip(ids, counts): + waveform_counts[int(unit)] += int(count) + waveform_ids._mmap.close() + np.save(temporary / '_phy_curation_windows.npy', np.asarray(windows, dtype=np.int64)) + metadata = source.metadata + write_metadata(temporary, metadata, source.metadata_schema) + # Standard loader stays self-contained; no original raw path is referenced. + params = { + key: source.params[key] + for key in ( + 'sample_rate', + 'n_channels_dat', + 'dtype', + 'offset', + 'hp_filtered', + 'acquisition_id', + 'depth_axis', + 'amplitude_units', + 'depth_units', + 'position_units', + 'waveform_units', + ) + if key in source.params + } + params.update( + dat_path=None, + sample_rate=source.sample_rate, + num_samples=source.num_samples, + valid_intervals=source.valid_intervals.tolist(), + ) + if getattr(source, 'time_base', None) is not None: + params['time_base'] = source.time_base + params.setdefault('dtype', 'int16') + params.setdefault('n_channels_dat', len(source.arrays.get('channel_map.npy', []))) + (temporary / 'params.py').write_text( + ''.join(f'{key} = {value!r}\n' for key, value in params.items()), encoding='utf-8' + ) + summaries = build_summaries(source, temporary, bin_seconds, max_bytes=max_bytes) + final_windows = [] + for start, end in windows: + contributors = [ + record for record in plan if record['start'] < end and record['end'] > start + ] + kinds = sorted({record['kind'] for record in contributors}) + final_windows.append( + { + 'start': start, + 'end': end, + 'segment': 0, + 'block': contributors[0]['block'], + 'kind': kinds[0] if len(kinds) == 1 else 'mixed', + 'kinds': kinds, + 'strata': sorted({record['stratum'] for record in contributors}), + 'exposure_seconds': (end - start) / source.sample_rate, + } + ) + manifest = { + 'format': 'phy_bounded_curation', + 'format_version': 1, + 'source': { + 'identity': source.identity, + 'revision': source.revision, + 'path': str(source.path), + }, + 'sample_rate': source.sample_rate, + 'num_samples': source.num_samples, + 'duration_seconds': source.num_samples / source.sample_rate, + 'valid_intervals': source.valid_intervals.tolist(), + 'num_spikes': source.n_spikes, + 'bounded_num_spikes': n_bounded, + 'unit_ids': source.unit_ids.tolist(), + 'counts': source.counts.tolist(), + 'bounded_counts': [bounded_counts[int(unit)] for unit in source.unit_ids], + 'waveform_counts': [waveform_counts[int(unit)] for unit in source.unit_ids], + 'waveform_coverage': { + 'source_rows': len(source.arrays['_phy_spikes_subset.spikes.npy']) + if source.arrays.keys() >= WAVEFORMS + else 0, + 'retained_rows': sum(waveform_counts.values()), + 'dropped_rows': ( + len(source.arrays['_phy_spikes_subset.spikes.npy']) + if source.arrays.keys() >= WAVEFORMS + else 0 + ) + - sum(waveform_counts.values()), + }, + 'producer_window_plan': producer_provenance, + 'producer_window_estimate': producer_estimate, + 'next_cluster_id': source.next_cluster_id, + 'global_spike_ids_file': '_phy_spike_global_ids.npy', + 'base_assignments_file': '_phy_spike_clusters_base.npy', + 'windows_file': '_phy_curation_windows.npy', + 'windows': final_windows, + 'accepted_window_candidates': plan, + 'omitted_windows': omitted, + 'metadata': metadata, + 'base_metadata': metadata, + 'metadata_schema': source.metadata_schema, + 'metadata_policy': POLICY, + 'metadata_provenance': source.metadata_provenance, + 'summaries': summaries, + **{key: value for key, value in summaries.items() if key.endswith('_file')}, + 'selection': { + 'algorithm': 'shared_windows_v1', + 'window_policy': window_policy, + 'resolved_window_policy': resolved_policy, + 'seed': int(seed), + 'window_seconds': window_seconds, + 'stratum_seconds': stratum_seconds, + }, + 'budgets': { + 'max_spikes': int(max_spikes), + 'max_bytes': int(max_bytes), + 'batch_size': int(batch_size), + 'summary_bytes': int(summary_bytes), + 'estimated_bytes': int( + fixed_bytes + + n_bounded * row_bytes + + _selected_waveform_count(source, windows) * waveform_bytes + + (len(windows) + len(plan) + len(omitted)) * 256 + ), + }, + 'capabilities': { + 'merge': True, + 'label': True, + 'split': False, + 'full_attributes': sorted(scalar_attributes(source)), + 'predicate_split_attributes': ( + ['amplitudes.npy'] if 'amplitude' in scalar_attributes(source) else [] + ), + 'stored_waveforms': bool(sum(waveform_counts.values())), + 'raw_traces': False, + 'normalized_correlograms': False, + }, + 'source_array_inventory': { + name: { + 'role': 'per_spike' + if name in PER_SPIKE + else 'stored_waveform' + if name in WAVEFORMS + else 'waveform_sidecar' + if name in waveform_sidecars + else 'static' + if name in STATIC + else 'excluded_window_sidecar', + 'shape': list(array.shape), + 'dtype': str(array.dtype), + 'bytes': int(array.nbytes), + } + for name, array in source.arrays.items() + }, + 'excluded_files': sorted( + p.name + for p in source.path.iterdir() + if p.name not in (PER_SPIKE | STATIC | WAVEFORMS | waveform_sidecars) + and p.name + not in ( + 'params.py', + '_phy_cluster_metadata.json', + SCHEMA_FILE, + 'curation.json', + 'curation_review.json', + ) + and (not p.name.startswith('cluster_') or p.stem == 'cluster_info') + ), + } + if getattr(source, 'time_base', None) is not None: + manifest['time_base'] = source.time_base + manifest['retired_units'] = source.retired_units + manifest['proposition_documents'] = {} + for document in producer_documents: + # Root review is a convenience copy; its inherited baseline is immutable. + name = ( + '_phy_curation_review_base.json' + if document.name == 'curation_review.json' + else document.name + ) + shutil.copyfile(document, temporary / name) + manifest['proposition_documents'][document.name] = { + 'filename': name, + 'sha256': producer_hashes[document.name], + } + if hash_file(temporary / name) != producer_hashes[document.name]: + raise ValueError('Producer proposition document changed during preparation.') + if name != document.name: + shutil.copyfile(document, temporary / document.name) + bound_paths = [ + *temporary.glob('*.npy'), + temporary / 'params.py', + temporary / '_phy_cluster_metadata.json', + temporary / SCHEMA_FILE, + *(temporary / table['filename'] for table in source.metadata_schema['tables']), + *(temporary / item['filename'] for item in manifest['proposition_documents'].values()), + ] + manifest['bounded_files'] = {p.name: hash_file(p) for p in sorted(bound_paths)} + manifest['preparation_identity'] = hash_json(manifest) + (temporary / '_phy_curation_manifest.json').write_text( + json.dumps(manifest, indent=2, sort_keys=True) + '\n', encoding='utf-8' + ) + actual_bytes = sum(p.stat().st_size for p in temporary.iterdir()) + if actual_bytes > max_bytes: + raise ValueError(f'Final output exceeds max_bytes: {actual_bytes} bytes.') + for array in (global_ids, base, *destinations.values()): + close_array(array) + for path in temporary.iterdir(): + if path.is_file(): + # Windows _commit (os.fsync) requires a writable descriptor. + with path.open('r+b') as stream: + os.fsync(stream.fileno()) + _fsync_directory(temporary) + if output_dir.exists(): + raise ValueError(f'Output already exists: {output_dir}.') + from .curation import BoundedSession + + checked = BoundedSession(temporary) + checked.close() + publish_directory(temporary, output_dir) + _fsync_directory(output_dir.parent) + except BaseException: + for array in (global_ids, base, *destinations.values()): + close_array(array) + shutil.rmtree(temporary, ignore_errors=True) + raise + return output_dir + + +def _fsync_directory(path): + if os.name != 'nt': + descriptor = os.open(path, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _copy_waveforms(source, output, global_ids, row_bounds): + """Intersect sparse row IDs with selected spans in bounded working memory.""" + upstream_ids = source.arrays['_phy_spikes_subset.spikes.npy'] + spans = [ + (int(np.searchsorted(upstream_ids, first)), int(np.searchsorted(upstream_ids, stop))) + for first, stop in row_bounds + ] + n = sum(stop - first for first, stop in spans) + targets = {} + waveform_names = WAVEFORMS | (source.arrays.keys() & {'_phy_spikes_subset.sample_index.npy'}) + try: + for name in waveform_names: + original = source.arrays[name] + targets[name] = np.lib.format.open_memmap( + output / name, mode='w+', dtype=original.dtype, shape=(n, *original.shape[1:]) + ) + cursor = 0 + for first, stop in spans: + for start in range(first, stop, source.batch_size): + end = min(start + source.batch_size, stop) + length = end - start + targets['_phy_spikes_subset.spikes.npy'][cursor : cursor + length] = ( + np.searchsorted(global_ids, upstream_ids[start:end]) + ) + for name in waveform_names - {'_phy_spikes_subset.spikes.npy'}: + targets[name][cursor : cursor + length] = source.arrays[name][start:end] + cursor += length + for array in targets.values(): + array.flush() + finally: + for array in targets.values(): + close_array(array) diff --git a/phy/apps/large_recording/replay.py b/phy/apps/large_recording/replay.py new file mode 100644 index 000000000..ec6fdfda8 --- /dev/null +++ b/phy/apps/large_recording/replay.py @@ -0,0 +1,261 @@ +"""Stream semantic curation into a new complete sorting, never the source.""" + +import os +import shutil +import tempfile +from pathlib import Path + +import numpy as np + +from .curation import ( + BoundedSession, + _digest, + _fsync_directory, + _write_json, + interpret_events, + project_assignments, +) +from .metadata import complete_schema, write_metadata +from .parameters import rewrite_replay_params_bytes +from .source import PER_SPIKE, Source, _has_raw_paths, close_array, hash_file, publish_directory + + +def _merge_mapping(unit_ids, events): + """Resolve a whole-unit merge journal once, independently of spike count.""" + lineage = {int(unit): [int(unit)] for unit in unit_ids} + for event in events: + if event['operation'] == 'merge': + members = [] + for parent in event['parents']: + members.extend(lineage.pop(parent)) + lineage[event['target']] = members + mapping = {old: new for new, members in lineage.items() for old in members} + keys = np.array(sorted(mapping), dtype=np.int64) + targets = np.array([mapping[int(key)] for key in keys], dtype=np.int64) + return keys, targets + + +def _write_metadata(path, metadata, schema=None): + return write_metadata(path, metadata, schema or {'version': 1, 'tables': []}) + + +def apply_curation(source_params, bounded_dir, output_dir, batch_size=100000): + """Validate identity/revision and replay only the last committed generation. + + The output is published as one new directory after validation. Full source + arrays remain file-backed; only one assignment batch and the unit map reside + in memory. Arbitrary selected-row splits have no journal interpretation. + """ + session = BoundedSession(bounded_dir) + source = None + try: + if session.checkpoint_path is None: + raise ValueError('Save a bounded semantic checkpoint before applying curation.') + manifest = session.manifest + source = Source( + source_params, + duration_seconds=manifest['num_samples'] / manifest['sample_rate'], + valid_intervals=manifest['valid_intervals'], + batch_size=batch_size, + ) + return _apply_curation(source, session, output_dir, batch_size) + finally: + if source is not None: + source.close() + session.close() + + +def _apply_curation(source, session, output_dir, batch_size): + session.validate_proposition_documents() + manifest = session.manifest + if 'metadata_schema' in manifest and source.metadata_schema != manifest['metadata_schema']: + raise ValueError('Source metadata schema changed since bounded preparation.') + if ( + 'metadata_provenance' in manifest + and source.metadata_provenance != manifest['metadata_provenance'] + ): + raise ValueError('Source metadata provenance changed since bounded preparation.') + expected = manifest['source'] + if source.identity != expected['identity']: + raise ValueError('Source coordinate/feature identity does not match bounded preparation.') + if source.revision != expected['revision']: + raise ValueError('Source assignment revision does not match journal parent revision.') + if source.metadata != session.base_metadata: + raise ValueError('Bounded base metadata does not match the source metadata revision.') + if ( + source.unit_ids.tolist() != manifest['unit_ids'] + or source.n_spikes != manifest['num_spikes'] + ): + raise ValueError('Source catalog or spike count does not match bounded preparation.') + for name, item in session.proposition_documents.items(): + if hash_file(source.path / name) != item['sha256']: + raise ValueError('Source proposition documents changed since preparation.') + # Identity/revision bind the complete source. Validate the actual projection + # too, so a tampered bounded base cannot silently replay unrelated evidence. + source_assignments = source.arrays['spike_clusters.npy'].reshape(-1) + for start in range(0, len(session.global_ids), batch_size): + rows = session.global_ids[start : start + batch_size] + if not np.array_equal( + source_assignments[rows], + session.base_assignments[start : start + batch_size], + ): + raise ValueError('Immutable bounded assignments do not match source rows.') + for name in sorted(source.arrays.keys() & (PER_SPIKE - {'spike_clusters.npy'})): + bounded = np.load(session.path / name, mmap_mode='r', allow_pickle=False) + try: + if bounded.shape != (len(session.global_ids), *source.arrays[name].shape[1:]): + raise ValueError(f'Bounded attribute has invalid source alignment: {name}.') + for start in range(0, len(session.global_ids), batch_size): + rows = session.global_ids[start : start + batch_size] + if not np.array_equal( + source.arrays[name][rows], + bounded[start : start + batch_size], + equal_nan=True, + ): + raise ValueError(f'Bounded attribute does not match source rows: {name}.') + finally: + close_array(bounded) + active_units, metadata, _ = interpret_events( + source.unit_ids, session.base_metadata, session.events + ) + output = Path(output_dir).resolve() + if output.exists(): + raise FileExistsError(f'Curated output already exists: {output}.') + if output == source.path or source.path in output.parents: + raise ValueError('Curated output must be outside the immutable source directory.') + output.parent.mkdir(parents=True, exist_ok=True) + estimated_bytes = sum( + (source.path / name).stat().st_size + for name in source.arrays + if name != 'spike_clusters.npy' + ) + estimated_bytes += source.n_spikes * 8 + 1024 * 1024 + free = shutil.disk_usage(output.parent).free + if free < estimated_bytes: + raise OSError( + f'Insufficient disk space for full replay: estimated at least {estimated_bytes:,} bytes, {free:,} available.' + ) + stage = Path(tempfile.mkdtemp(prefix=f'.{output.name}-', dir=output.parent)) + curated = None + assignments = None + try: + # Copy only validated canonical assets, not GUI caches or producer reviews. + for name in source.arrays: + if name != 'spike_clusters.npy': + shutil.copyfile(source.path / name, stage / name) + shutil.copyfile(source.params_path, stage / 'params.py') + upstream_manifest = source.path / '_phy_spikes_subset.manifest.json' + if upstream_manifest.exists(): + shutil.copyfile(upstream_manifest, stage / upstream_manifest.name) + if source.time_base is not None and source.timestamps_path.exists(): + shutil.copyfile(source.timestamps_path, stage / 'timestamps.json') + # Preserve source params semantics and make external raw paths absolute. + # The full export can contain no raw traces; immutable array rows suffice. + dat_path = source.params.get('dat_path') + if _has_raw_paths(dat_path): + raw_paths = dat_path if isinstance(dat_path, (list, tuple)) else [dat_path] + resolved = [ + str((source.path / raw).resolve()) if _has_raw_paths(raw) else raw + for raw in raw_paths + ] + dat_path = resolved if isinstance(dat_path, (list, tuple)) else resolved[0] + params_path = stage / 'params.py' + params_path.write_bytes( + rewrite_replay_params_bytes( + params_path.read_bytes(), + num_samples=source.num_samples, + valid_intervals=source.valid_intervals.tolist(), + dat_path=dat_path, + ) + ) + assignments = np.lib.format.open_memmap( + stage / 'spike_clusters.npy', + mode='w+', + dtype=np.int64, + shape=source.arrays['spike_clusters.npy'].shape, + ) + flat = assignments.reshape(-1) + has_split = any(event['operation'] == 'threshold_split' for event in session.events) + keys, targets = ( + (None, None) if has_split else _merge_mapping(source.unit_ids, session.events) + ) + for batch in source.iter_batches(): + values = batch['spike_clusters.npy'] + if has_split: + projected = project_assignments(values, session.events, batch) + else: + positions = np.searchsorted(keys, values) + projected = targets[positions] + flat[batch['start'] : batch['stop']] = projected + assignments.flush() + assignments._mmap.close() + del flat, assignments + assignments = None + _write_metadata(stage, metadata, complete_schema(source.metadata_schema, metadata)) + _write_json(stage / '_phy_curation_journal.json', session.events) + proposition_hashes = {} + item = session.proposition_documents.get('curation.json') + if item is not None: + shutil.copyfile(session.path / item['filename'], stage / 'curation.json') + proposition_hashes['curation.json'] = hash_file(stage / 'curation.json') + if proposition_hashes['curation.json'] != item['sha256']: + raise ValueError('Producer propositions changed during replay.') + review = session.extra_documents.get('curation_review.json') + if review is not None and item is not None: + _write_json(stage / 'curation_review.json', review) + proposition_hashes['curation_review.json'] = hash_file(stage / 'curation_review.json') + # Re-reading verifies alignment, full catalog and immutable byte digests. + curated = Source(stage / 'params.py', batch_size=batch_size) + if curated.identity != source.identity or curated.n_spikes != source.n_spikes: + raise ValueError('Curated output changed immutable source coordinates.') + if curated.metadata != metadata: + raise ValueError('Curated output metadata failed exact round-trip validation.') + actual = curated.arrays['spike_clusters.npy'].reshape(-1) + for batch in source.iter_batches(): + if has_split: + expected_batch = project_assignments( + batch['spike_clusters.npy'], session.events, batch + ) + else: + expected_batch = targets[np.searchsorted(keys, batch['spike_clusters.npy'])] + if not np.array_equal(actual[batch['start'] : batch['stop']], expected_batch): + raise ValueError('Curated output assignments failed semantic replay verification.') + _write_json( + stage / '_phy_curation_lineage.json', + { + 'format': 'phy_curation_lineage', + 'format_version': 1, + 'coordinate_identity': source.identity, + 'parent_revision': source.revision, + 'journal_sha256': _digest(session.events), + 'revision': curated.revision, + 'preparation_sha256': _digest(manifest), + 'next_cluster_id': session.next_cluster_id, + 'unit_ids': sorted(active_units), + 'retired_units': session.retired_units, + 'proposition_documents': proposition_hashes, + 'metadata_provenance': session.metadata_provenance, + 'metadata_policy': session.metadata_policy, + }, + ) + curated.close() + for path in stage.iterdir(): + if path.is_file(): + # Windows _commit (os.fsync) requires a writable descriptor. + with path.open('r+b') as stream: + os.fsync(stream.fileno()) + _fsync_directory(stage) + # Refuse replacement even when a racing caller creates the target. + if output.exists(): + raise FileExistsError(f'Curated output already exists: {output}.') + session.validate_proposition_documents() + publish_directory(stage, output) + _fsync_directory(output.parent) + except BaseException: + if curated is not None: + curated.close() + if assignments is not None: + assignments._mmap.close() + shutil.rmtree(stage, ignore_errors=True) + raise + return output diff --git a/phy/apps/large_recording/source.py b/phy/apps/large_recording/source.py new file mode 100644 index 000000000..ee7534735 --- /dev/null +++ b/phy/apps/large_recording/source.py @@ -0,0 +1,696 @@ +"""Validated, file-backed source sorting on one monotonic integer time axis.""" + +import ctypes +import errno +import hashlib +import json +import os +import runpy +import sys +from pathlib import Path + +import numpy as np + +from .metadata import read_metadata + +PER_SPIKE = frozenset( + ( + 'spike_times.npy', + 'spike_clusters.npy', + 'spike_templates.npy', + 'amplitudes.npy', + 'pc_features.npy', + 'template_features.npy', + 'spike_positions.npy', + 'spike_depths.npy', + ) +) +STATIC = frozenset( + ( + 'templates.npy', + 'templates_ind.npy', + 'template_ind.npy', + 'channel_map.npy', + 'channel_positions.npy', + 'channel_shanks.npy', + 'channel_map_si.npy', + 'channel_groups.npy', + 'whitening_mat.npy', + 'whitening_mat_inv.npy', + 'similar_templates.npy', + 'pc_feature_ind.npy', + 'template_feature_ind.npy', + ) +) +WAVEFORMS = frozenset( + ( + '_phy_spikes_subset.spikes.npy', + '_phy_spikes_subset.channels.npy', + '_phy_spikes_subset.waveforms.npy', + ) +) +WAVEFORM_SIDECARS = frozenset(('_phy_spikes_subset.sample_index.npy',)) +WINDOW_SIDECARS = frozenset( + ( + '_phy_spikes_subset.window_bounds.npy', + '_phy_spikes_subset.window_types.npy', + '_phy_spikes_subset.window_strata.npy', + ) +) + + +def hash_file(path, batch_bytes=1024 * 1024): + """Hash bytes incrementally, including the array header.""" + digest = hashlib.sha256() + with Path(path).open('rb') as stream: + for block in iter(lambda: stream.read(batch_bytes), b''): + digest.update(block) + return digest.hexdigest() + + +def hash_json(value): + """Hash canonical JSON, normalizing integer metadata dictionary keys.""" + + def normalize(item): + if isinstance(item, dict): + return {str(key): normalize(val) for key, val in item.items()} + if isinstance(item, (list, tuple)): + return [normalize(val) for val in item] + if isinstance(item, np.generic): + return item.item() + return item + + payload = json.dumps(normalize(value), sort_keys=True, separators=(',', ':'), allow_nan=False) + return hashlib.sha256(payload.encode('utf-8')).hexdigest() + + +def assignment_revision(path, metadata): + """Identify mutable assignments and resolved cluster metadata.""" + path = Path(path) + if path.is_dir(): + path = path / 'spike_clusters.npy' + return hash_json({'assignments': hash_file(path), 'metadata': metadata}) + + +def close_array(array): + """Release a file-backed array handle, including on Windows.""" + mapping = getattr(array, '_mmap', None) + if mapping is not None and not mapping.closed: + mapping.close() + + +def publish_directory(staged, destination): + """Atomically publish a complete directory without replacing any target. + + Ordinary POSIX rename can replace an empty destination directory. Native + exclusive rename prevents a racing publication from silently replacing it. + """ + if os.name == 'nt': + os.rename(staged, destination) + return + library = ctypes.CDLL(None, use_errno=True) + if sys.platform.startswith('linux'): + rename = getattr(library, 'renameat2', None) + if rename is None: + raise OSError(errno.ENOSYS, 'Atomic exclusive directory publication is unavailable.') + rename.argtypes = [ + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ] + rename.restype = ctypes.c_int + result = rename(-100, os.fsencode(staged), -100, os.fsencode(destination), 1) + elif sys.platform == 'darwin': + rename = getattr(library, 'renamex_np', None) + if rename is None: + raise OSError(errno.ENOSYS, 'Atomic exclusive directory publication is unavailable.') + rename.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint] + rename.restype = ctypes.c_int + result = rename(os.fsencode(staged), os.fsencode(destination), 4) + else: + raise OSError(errno.ENOSYS, 'Atomic exclusive directory publication is unavailable.') + if result: + code = ctypes.get_errno() + raise OSError(code, os.strerror(code), str(destination)) + + +def _vector(array, name): + if array.ndim == 1 or (array.ndim == 2 and array.shape[1] == 1): + return array.reshape(-1) + raise ValueError(f'{name} must be a vector or column vector.') + + +def _metadata(path): + return read_metadata(path)[0] + + +def _time_base(value): + """Normalize an explicit producer clock without inferring conversions.""" + if not isinstance(value, dict) or set(value) != {'kind', 'ticks_per_second'}: + raise ValueError('time_base must declare kind and ticks_per_second.') + if value['kind'] not in ('acquisition_samples', 'elapsed_ticks'): + raise ValueError('time_base.kind must be acquisition_samples or elapsed_ticks.') + rate = value['ticks_per_second'] + if isinstance(rate, bool) or not isinstance(rate, (int, float)): + raise ValueError('time_base.ticks_per_second must be positive and finite.') + rate = float(rate) + if not np.isfinite(rate) or rate <= 0: + raise ValueError('time_base.ticks_per_second must be positive and finite.') + return {'kind': value['kind'], 'ticks_per_second': rate} + + +def _has_raw_paths(dat_path): + paths = dat_path if isinstance(dat_path, (list, tuple)) else [dat_path] + return any(path is not None and str(path).strip() not in ('', 'None') for path in paths) + + +class Source: + """Read source arrays in batches without constructing a GUI model. + + Duration must come from producer metadata or an explicit argument: the last + spike is never used as acquisition exposure. Intervals are half-open samples. + """ + + def __init__( + self, params_path, duration_seconds=None, valid_intervals=None, batch_size=100000 + ): + self.arrays = {} + try: + self._initialize(params_path, duration_seconds, valid_intervals, batch_size) + except BaseException: + self.close() + raise + + def _initialize( + self, params_path, duration_seconds=None, valid_intervals=None, batch_size=100000 + ): + self.params_path = Path(params_path).resolve() + self.path = self.params_path.parent + self.params = { + key: value + for key, value in runpy.run_path(str(self.params_path)).items() + if not key.startswith('__') + } + self.batch_size = int(batch_size) + if self.batch_size <= 0: + raise ValueError('batch_size must be positive.') + if 'dtype' in self.params: + try: + raw_dtype = np.dtype(self.params['dtype']) + except (TypeError, ValueError) as error: + raise ValueError('params.py dtype must be a valid raw numeric dtype.') from error + if not np.issubdtype(raw_dtype, np.number): + raise ValueError('params.py dtype must be a valid raw numeric dtype.') + self.sample_rate = float(self.params.get('sample_rate', 0)) + if not np.isfinite(self.sample_rate) or self.sample_rate <= 0: + raise ValueError('A positive finite sample_rate is required.') + upstream = self.path / '_phy_spikes_subset.manifest.json' + recording = {} + info = {} + if upstream.exists(): + info = json.loads(upstream.read_text(encoding='utf-8')) + if info.get('format') != 'phy_spikes_subset' or info.get('format_version') != 1: + raise ValueError('Unsupported waveform manifest format or version.') + recording = info.get('recording', {}) + upstream_rate = recording.get('sampling_frequency_hz', self.sample_rate) + if upstream_rate != self.sample_rate: + raise ValueError('Waveform manifest sampling rate disagrees with params.py.') + self.upstream_recording = recording + declared_clocks = [ + _time_base(container['time_base']) + for container in (self.params, recording) + if 'time_base' in container + ] + if len(declared_clocks) == 2 and declared_clocks[0] != declared_clocks[1]: + raise ValueError('time_base disagrees between params and waveform manifest.') + self.time_base = declared_clocks[0] if declared_clocks else None + self.timestamps_path = self.path / 'timestamps.json' + if self.time_base is None and self.timestamps_path.exists(): + raise ValueError( + 'timestamps.json requires an explicit time_base in params.py or manifest ' + 'recording. Convert num_samples, valid_intervals and waveform sidecars ' + 'to the spike_times.npy coordinates; automatic timestamp mapping is unsupported.' + ) + if self.time_base is not None: + if self.time_base['ticks_per_second'] != self.sample_rate: + raise ValueError('time_base ticks_per_second must equal sample_rate.') + for container in (self.params, recording): + if container.get('waveform_sample_rate', self.sample_rate) != self.sample_rate: + raise ValueError('waveform_sample_rate must equal sample_rate.') + kind = self.time_base['kind'] + if recording.get('frame_coordinates', kind) != kind: + raise ValueError('recording.frame_coordinates must equal time_base.kind.') + for suffix, field in ( + ('window_bounds', 'window_bounds_coordinates'), + ('sample_index', 'sample_index_coordinates'), + ): + if (self.path / f'_phy_spikes_subset.{suffix}.npy').exists(): + coordinates = recording.get(field) + if coordinates is None and kind == 'acquisition_samples': + coordinates = kind + if coordinates != kind: + raise ValueError(f'recording.{field} must equal time_base.kind.') + if kind == 'elapsed_ticks' and _has_raw_paths(self.params.get('dat_path')): + raise ValueError('elapsed_ticks cannot reference raw dat_path traces.') + if self.params.get('num_segments', recording.get('num_segments', 1)) != 1: + raise ValueError('Only a single monotonic acquisition segment is supported.') + duration_samples = None + if duration_seconds is not None: + duration_samples = float(duration_seconds) * self.sample_rate + if ( + not np.isfinite(duration_samples) + or duration_samples <= 0 + or duration_samples > np.iinfo(np.int64).max + or not np.isclose(duration_samples, round(duration_samples), rtol=0, atol=1e-6) + ): + raise ValueError('duration_seconds must specify a positive integer sample count.') + duration_samples = int(round(duration_samples)) + if ( + 'num_samples' in self.params + and 'num_samples' in recording + and self.params['num_samples'] != recording['num_samples'] + ): + raise ValueError( + 'Producer sample counts disagree between params and waveform manifest.' + ) + declared_samples = self.params.get('num_samples', recording.get('num_samples')) + if declared_samples is None: + declared_samples = duration_samples + elif duration_samples is not None and int(declared_samples) != duration_samples: + raise ValueError('Explicit duration disagrees with acquisition metadata.') + if declared_samples is None: + raise ValueError( + 'Acquisition exposure is missing; provide duration_seconds or num_samples.' + ) + self.num_samples = int(declared_samples) + if ( + self.num_samples <= 0 + or self.num_samples > np.iinfo(np.int64).max + or self.num_samples != declared_samples + ): + raise ValueError('num_samples must be a positive int64 sample count.') + if ( + 'valid_intervals' in self.params + and 'valid_intervals' in recording + and not np.array_equal( + np.asarray(self.params['valid_intervals']), + np.asarray(recording['valid_intervals']), + ) + ): + raise ValueError( + 'Producer valid_intervals disagree between params and waveform manifest.' + ) + declared_intervals = self.params.get('valid_intervals', recording.get('valid_intervals')) + if valid_intervals is not None and declared_intervals is not None: + if not np.array_equal(np.asarray(valid_intervals), np.asarray(declared_intervals)): + raise ValueError('Explicit valid_intervals disagree with acquisition metadata.') + if valid_intervals is None: + valid_intervals = declared_intervals + if ( + self.time_base is not None + and self.time_base['kind'] == 'elapsed_ticks' + and valid_intervals is None + ): + raise ValueError('elapsed_ticks requires explicit authoritative valid_intervals.') + intervals = [[0, self.num_samples]] if valid_intervals is None else valid_intervals + raw_intervals = np.asarray(intervals) + if raw_intervals.ndim != 2 or raw_intervals.shape[1] != 2 or not len(raw_intervals): + raise ValueError('valid_intervals must be nonempty sample [start, end) pairs.') + self.valid_intervals = raw_intervals.astype(np.int64) + if ( + not np.array_equal(raw_intervals, self.valid_intervals) + or np.any(self.valid_intervals[:, 0] < 0) + or np.any(self.valid_intervals[:, 1] > self.num_samples) + or np.any(self.valid_intervals[:, 1] <= self.valid_intervals[:, 0]) + or np.any(self.valid_intervals[1:, 0] < self.valid_intervals[:-1, 1]) + ): + raise ValueError( + 'valid_intervals must be ordered, disjoint integer acquisition samples.' + ) + sample_index_path = self.path / '_phy_spikes_subset.sample_index.npy' + if ( + self.time_base is None + and sample_index_path.exists() + and recording.get('sample_index_coordinates', 'acquisition_samples') + != 'acquisition_samples' + ): + raise ValueError( + 'Legacy waveform sample_index coordinates must be acquisition_samples; ' + 'declare an explicit time_base for elapsed_ticks.' + ) + names = {p.name for p in self.path.glob('*.npy')} + unknown = names - PER_SPIKE - STATIC - WAVEFORMS - WAVEFORM_SIDECARS - WINDOW_SIDECARS + # Replay lineage and original optional sidecars are retained; bounded + # outputs deliberately cannot be used as full-source sorting inputs. + if unknown: + raise ValueError(f'Unregistered source arrays: {", ".join(sorted(unknown))}.') + self.arrays = { + name: np.load(self.path / name, mmap_mode='r', allow_pickle=False) + for name in sorted(names) + } + for required in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy'): + if required not in self.arrays: + raise ValueError(f'Missing required source array {required}.') + self.n_spikes = len(_vector(self.arrays['spike_times.npy'], 'spike_times.npy')) + self.metadata, self.metadata_schema = read_metadata(self.path) + self.validate() + counts = {} + for batch in self.iter_batches(): + ids, values = np.unique(batch['spike_clusters.npy'], return_counts=True) + for cluster_id, count in zip(ids.tolist(), values.tolist(), strict=True): + counts[cluster_id] = counts.get(cluster_id, 0) + count + for values in self.metadata.values(): + for cluster_id in values: + counts.setdefault(cluster_id, 0) + lineage_path = self.path / '_phy_curation_lineage.json' + lineage = {} + if lineage_path.exists(): + lineage = json.loads(lineage_path.read_text(encoding='utf-8')) + if ( + lineage.get('format') != 'phy_curation_lineage' + or lineage.get('format_version') != 1 + ): + raise ValueError('Unsupported curation lineage format or version.') + for unit in lineage.get('unit_ids', []): + if not isinstance(unit, int) or unit < 0: + raise ValueError('Lineage unit IDs must be nonnegative integers.') + counts.setdefault(unit, 0) + self.metadata_provenance = lineage.get('metadata_provenance', {'history': []}) + self.retired_units = lineage.get('retired_units', {}) + if not isinstance(self.retired_units, dict): + raise ValueError('Lineage retired_units must be a mapping.') + for unit, record in self.retired_units.items(): + if ( + not isinstance(unit, str) + or not unit.isdecimal() + or str(int(unit)) != unit + or int(unit) > np.iinfo(np.int32).max + or not isinstance(record, dict) + or set(record) != {'operation', 'replacements'} + or record['operation'] not in ('merge', 'threshold_split') + ): + raise ValueError('Invalid lineage retired unit record.') + replacements = record['replacements'] + if ( + not isinstance(replacements, list) + or not replacements + or any( + type(value) is not int or value < 0 or value > np.iinfo(np.int32).max + for value in replacements + ) + or len(set(replacements)) != len(replacements) + or len(replacements) != (1 if record['operation'] == 'merge' else 2) + or int(unit) in replacements + or int(unit) in counts + ): + raise ValueError('Invalid lineage retired unit replacements.') + documents = lineage.get('proposition_documents', {}) + if not isinstance(documents, dict): + raise ValueError('Lineage proposition documents must be a mapping.') + for name, digest in documents.items(): + if ( + name not in ('curation.json', 'curation_review.json') + or not isinstance(digest, str) + or len(digest) != 64 + or any(c not in '0123456789abcdef' for c in digest) + or (self.path / name).is_symlink() + or hash_file(self.path / name) != digest + ): + raise ValueError('Lineage proposition document integrity check failed.') + if any(unit < 0 or unit > np.iinfo(np.int32).max for unit in counts): + raise ValueError( + 'Unit IDs must fit nonnegative int32 for the current Phy loader; spike sample/row coordinates remain int64.' + ) + self.unit_ids = np.array(sorted(counts), dtype=np.int64) + minimum_next = int(self.unit_ids[-1]) + 1 if len(self.unit_ids) else 0 + self.next_cluster_id = lineage.get('next_cluster_id', minimum_next) + if not isinstance(self.next_cluster_id, int) or self.next_cluster_id < minimum_next: + raise ValueError('Lineage next_cluster_id collides with the source unit catalog.') + self.counts = np.array( + [counts[cluster_id] for cluster_id in self.unit_ids], dtype=np.int64 + ) + immutable = sorted(names - {'spike_clusters.npy'} - WINDOW_SIDECARS) + # Optional subset planning metadata describes a sampling policy, not a + # new immutable spike-coordinate identity. + acquisition = { + key: self.params[key] + for key in ( + 'acquisition_id', + 'n_channels_dat', + 'dtype', + 'offset', + 'depth_axis', + 'amplitude_units', + 'depth_units', + 'position_units', + 'waveform_units', + ) + if key in self.params + } + self.identity = { + 'format_version': 1, + 'acquisition': acquisition, + 'time_base': { + 'sample_rate': self.sample_rate, + 'num_samples': self.num_samples, + 'valid_intervals': self.valid_intervals.tolist(), + }, + 'arrays': {name: hash_file(self.path / name) for name in immutable}, + } + if self.time_base is not None: + self.identity['time_base']['contract'] = self.time_base.copy() + if self.timestamps_path.exists(): + self.identity['time_base']['timestamps_sha256'] = hash_file(self.timestamps_path) + self.identity['sha256'] = hash_json(self.identity) + declared_source = info.get('source_sorting', {}) + if not isinstance(declared_source, dict): + raise ValueError('Waveform manifest source_sorting must be a mapping.') + if declared_source.get('num_spikes', self.n_spikes) != self.n_spikes: + raise ValueError('Waveform reservoir source spike count does not match this sorting.') + declared_times = declared_source.get('spike_times_sha256') + if ( + declared_times is not None + and declared_times != self.identity['arrays']['spike_times.npy'] + ): + raise ValueError( + 'Waveform reservoir source coordinate fingerprint does not match this sorting.' + ) + self.revision = assignment_revision(self.path, self.metadata) + if lineage: + if ( + lineage.get('coordinate_identity') != self.identity + or lineage.get('revision') != self.revision + ): + raise ValueError( + 'Curation lineage does not match immutable identity or assignments.' + ) + + def close(self): + """Release canonical array mappings after a batch command.""" + for array in self.arrays.values(): + close_array(array) + self.arrays.clear() + + def __enter__(self): + return self + + def __exit__(self, *args): + self.close() + + def iter_batches(self): + """Yield bounded row slices; vector fields use one-dimensional views.""" + for start in range(0, self.n_spikes, self.batch_size): + stop = min(start + self.batch_size, self.n_spikes) + batch = {'start': start, 'stop': stop} + for name in self.arrays.keys() & PER_SPIKE: + array = self.arrays[name] + if name in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy'): + array = _vector(array, name) + batch[name] = array[start:stop] + yield batch + + def validate(self): + """Check source row, clock, sparse channel, and feature alignment.""" + for name in self.arrays.keys() & PER_SPIKE: + if self.arrays[name].shape[0] != self.n_spikes: + raise ValueError(f'{name} is not aligned with source spikes.') + for name in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy'): + array = _vector(self.arrays[name], name) + if not np.issubdtype(array.dtype, np.integer): + raise ValueError(f'{name} must contain integer coordinates.') + times = _vector(self.arrays['spike_times.npy'], 'spike_times.npy') + # Monotonic sources expose a clock mismatch at their endpoints. Check + # these before scanning hundreds of millions of rows or hashing assets; + # the complete monotonicity and bounds validation below remains required. + if len(times) and (times[0] < 0 or times[-1] >= self.num_samples): + raise ValueError( + 'Spike time endpoints fall outside declared num_samples; check that ' + 'spike_times.npy, num_samples, valid_intervals and waveform sidecars ' + 'use the same time base. Convert producer metadata to canonical ' + 'coordinates before preparing this sorting.' + ) + previous = -1 + for batch in self.iter_batches(): + times = batch['spike_times.npy'] + if ( + np.any(times < previous) + or np.any(times[1:] < times[:-1]) + or np.any(times < 0) + or np.any(times >= self.num_samples) + ): + raise ValueError( + 'Spike sample times must be monotonic and within acquisition bounds.' + ) + if len(times): + previous = int(times[-1]) + interval_ids = np.searchsorted(self.valid_intervals[:, 0], times, side='right') - 1 + if np.any(interval_ids < 0) or np.any(times >= self.valid_intervals[interval_ids, 1]): + raise ValueError('Spike times fall outside valid acquisition intervals.') + for name in ('spike_clusters.npy', 'spike_templates.npy'): + if np.any(batch[name] < 0): + raise ValueError(f'{name} must contain nonnegative IDs.') + templates = self.arrays.get('templates.npy') + if templates is not None and np.any(batch['spike_templates.npy'] >= len(templates)): + raise ValueError('spike_templates.npy references an unavailable template.') + waveforms = self.arrays.keys() & WAVEFORMS + if waveforms and waveforms != WAVEFORMS: + raise ValueError('Stored waveforms require spikes, channels, and waveforms together.') + sample_index = self.arrays.get('_phy_spikes_subset.sample_index.npy') + if sample_index is not None and not waveforms: + raise ValueError('Stored waveform sample_index requires the waveform triplet.') + if waveforms: + ids = self.arrays['_phy_spikes_subset.spikes.npy'] + if ids.ndim != 1 or ids.dtype != np.dtype('int64'): + raise ValueError('Stored waveform row IDs must be a one-dimensional int64 array.') + channels = self.arrays['_phy_spikes_subset.channels.npy'] + values = self.arrays['_phy_spikes_subset.waveforms.npy'] + n_channels = len(self.arrays.get('channel_map.npy', [])) + if ( + values.ndim != 3 + or channels.shape != (len(ids), values.shape[2]) + or len(values) != len(ids) + or not np.issubdtype(channels.dtype, np.integer) + ): + raise ValueError('Stored waveform tensors and channels are not aligned.') + if sample_index is not None and ( + sample_index.ndim != 1 + or not np.issubdtype(sample_index.dtype, np.integer) + or len(sample_index) != len(ids) + ): + raise ValueError('Stored waveform sample_index must be an aligned integer vector.') + last = -1 + # Sparse-padding checks create temporary boolean arrays and gathers; + # cap their tensor bytes independently of the source row batch size. + row_bytes = max(1, values.dtype.itemsize * int(np.prod(values.shape[1:]))) + waveform_batch = min(self.batch_size, max(1, 8 * 1024 * 1024 // row_bytes)) + for start in range(0, len(ids), waveform_batch): + rows = ids[start : start + waveform_batch] + slots = channels[start : start + waveform_batch] + if ( + np.any(rows <= last) + or np.any(rows[1:] <= rows[:-1]) + or np.any(rows < 0) + or np.any(rows >= self.n_spikes) + ): + raise ValueError('Stored waveform row IDs must be sorted, unique source rows.') + if sample_index is not None: + coordinates = sample_index[start : start + waveform_batch] + if ( + np.any(coordinates < 0) + or np.any(coordinates >= self.num_samples) + or not np.array_equal( + coordinates, self.arrays['spike_times.npy'].reshape(-1)[rows] + ) + ): + raise ValueError( + 'Stored waveform sample_index must equal spike_times.npy at waveform ' + 'row IDs in canonical time-base coordinates.' + ) + if len(rows): + last = int(rows[-1]) + if np.any((slots < -1) | (slots >= n_channels)): + raise ValueError( + 'Sparse waveform channels must index model channels or use -1.' + ) + block = values[start : start + waveform_batch] + if np.any(np.moveaxis(block, 1, 2)[slots == -1] != 0): + raise ValueError('Sparse waveform padding channels must contain zeros.') + channel_map = self.arrays.get('channel_map.npy') + if channel_map is not None: + channels = _vector(channel_map, 'channel_map.npy') + if ( + not np.issubdtype(channels.dtype, np.integer) + or np.any(channels < 0) + or len(np.unique(channels)) != len(channels) + ): + raise ValueError('channel_map.npy must contain unique nonnegative raw column IDs.') + raw_channels = self.params.get('n_channels_dat') + if raw_channels is not None and ( + int(raw_channels) != raw_channels + or raw_channels <= 0 + or np.any(channels >= raw_channels) + ): + raise ValueError('channel_map.npy contains columns outside n_channels_dat.') + for name in ('channel_groups.npy', 'channel_shanks.npy', 'channel_map_si.npy'): + if name in self.arrays and len(_vector(self.arrays[name], name)) != len(channels): + raise ValueError(f'{name} must align with model channels.') + for name in ('whitening_mat.npy', 'whitening_mat_inv.npy'): + if name in self.arrays and self.arrays[name].shape != ( + len(channels), + len(channels), + ): + raise ValueError(f'{name} must contain a model-channel square matrix.') + templates = self.arrays.get('templates.npy') + if templates is not None: + # phylib uses singular template_ind; KS2's plural sidecar does + # not turn a dense tensor into a sparse tensor in the loader. + sparse = self.arrays.get('template_ind.npy') + if templates.ndim != 3: + raise ValueError('templates.npy must be a template/sample/channel tensor.') + if sparse is None and templates.shape[2] != len(channels): + raise ValueError('Dense templates must align with model channels.') + for index_name in ('template_ind.npy', 'templates_ind.npy'): + sparse = self.arrays.get(index_name) + if sparse is None: + continue + if sparse.shape != (len(templates), templates.shape[2]): + raise ValueError(f'{index_name} must align with sparse templates.') + if not np.issubdtype(sparse.dtype, np.integer): + raise ValueError(f'{index_name} must contain integer channel indices.') + if np.any((sparse < -1) | (sparse >= len(channels))): + raise ValueError(f'{index_name} contains invalid model channel IDs.') + positions = self.arrays.get('channel_positions.npy') + if positions is not None and ( + positions.shape != (len(channels), 2) or not np.all(np.isfinite(positions)) + ): + raise ValueError( + 'channel_positions.npy must contain finite x/y model channel coordinates.' + ) + for feature, indices in ( + ('pc_features.npy', 'pc_feature_ind.npy'), + ('template_features.npy', 'template_feature_ind.npy'), + ): + if feature in self.arrays and indices not in self.arrays: + raise ValueError(f'{feature} requires {indices}.') + if feature in self.arrays: + values = self.arrays[feature] + lookup = self.arrays[indices] + if lookup.ndim != 2 or values.ndim < 2 or values.shape[-1] != lookup.shape[1]: + raise ValueError(f'{feature} and {indices} sparse axes do not agree.') + templates = self.arrays.get('templates.npy') + if templates is not None and len(lookup) != len(templates): + raise ValueError(f'{indices} must align with templates.') + if not np.issubdtype(lookup.dtype, np.integer): + raise ValueError(f'{indices} must contain integer indices.') + limit = ( + len(self.arrays.get('channel_map.npy', [])) + if feature == 'pc_features.npy' + else len(lookup) + ) + for start in range(0, len(lookup), self.batch_size): + rows = lookup[start : start + self.batch_size] + if np.any((rows < -1) | (rows >= limit)): + raise ValueError(f'{indices} contains invalid sparse indices.') + return self diff --git a/phy/apps/large_recording/stability.py b/phy/apps/large_recording/stability.py new file mode 100644 index 000000000..b7c847ee0 --- /dev/null +++ b/phy/apps/large_recording/stability.py @@ -0,0 +1,547 @@ +"""Selected-unit full-duration evidence with explicit acquisition exposure.""" + +from collections import OrderedDict +from sys import getsizeof + +import numpy as np +from phylib.utils import Bunch, connect, unconnect + +from phy.cluster.views import ManualClusteringView +from phy.gui.qt import screenshot, screenshot_default_path +from phy.plot.plot import PlotCanvasMpl +from phy.utils.color import selected_cluster_color + + +class UnitStabilityView(ManualClusteringView): + """Compare exact rates and available source trends for selected units.""" + + plot_canvas_class = PlotCanvasMpl + # Apply the display limit locally, retaining the full presentation selection. + max_n_clusters = 0 + display_limit = 32 + rate_line_width = 0.6 + display_bin_seconds = 0.0 + summary_cache_bytes = 64 * 1024 * 1024 + _default_position = 'right' + ex_status = 'Full-duration evidence; sampling cannot establish chronic unit identity.' + _metrics = ('firing_rate', 'amplitude', 'depth') + + def __init__(self, controller): + super().__init__() + self.controller = controller + self.canvas.subplots(3, 1, sharex=True) + self.show_firing_rate = self.show_amplitude = self.show_depth = True + self.state_attrs += tuple(f'show_{metric}' for metric in self._metrics) + self.state_attrs += ('display_limit', 'rate_line_width') + self.local_state_attrs += ('display_bin_seconds',) + self._summary_arrays = {} + self._selected_summaries = OrderedDict() + self._common_data = None + self._attaching = False + self._footer = self.canvas.figure.text(0.12, 0.025, '', fontsize=8) + + def _array(self, name): + if name not in self._summary_arrays: + self._summary_arrays[name] = np.load(self.controller.dir_path / name, mmap_mode='r') + return self._summary_arrays[name] + + def _common(self): + if self._common_data is not None: + return self._common_data + controller = self.controller + session = getattr(controller, 'bounded_session', None) + if session is None: + edges = np.linspace(0, controller.model.duration, 101) + self._common_data = Bunch( + edges=edges, + exposure=np.diff(edges), + windows=[], + intervals=[[0, controller.model.duration]], + footer='', + ) + else: + manifest = session.manifest + summary = manifest.get('summaries', manifest) + edges = self._array(summary['edges_file']) / controller.model.sample_rate + missing = [c for c, n in zip(session.unit_ids, manifest['bounded_counts']) if not n] + footer = '' + if missing: + footer = ( + f'{len(missing)} source units without evidence: ' + f'{", ".join(map(str, missing[:12]))}' + (' …' if len(missing) > 12 else '') + ) + self._common_data = Bunch( + edges=edges, + exposure=self._array(summary['exposure_file']), + windows=controller.evidence_windows / controller.model.sample_rate, + intervals=np.asarray(manifest['valid_intervals']) / controller.model.sample_rate, + summary=summary, + lookup={int(c): i for i, c in enumerate(session.unit_ids)}, + footer=footer, + ) + return self._common_data + + def get_data(self, cluster_id, metrics=None): + """Load only requested metrics, caching bounded selected-unit summary rows. + + Counts share the manifest's exposure. Trend sums are combined with their + finite-value counts, never averaged as equally weighted unit means. + """ + metrics = self._metrics if metrics is None else metrics + common = self._common() + data = self._selected_summaries.get(cluster_id) + controller = self.controller + session = getattr(controller, 'bounded_session', None) + if data is None: + data = Bunch(common.copy()) + data.update( + counts=None, + trends={}, + trend_sums={}, + trend_counts={}, + loaded=set(), + pending=False, + display=None, + ) + if session is None: + data.provenance = 'Loaded recording; continuous acquisition assumed.' + else: + units = controller._source_units(cluster_id) + data.pending = units is None + data.rows = [] if units is None else [common.lookup[c] for c in units] + data.provenance = ( + 'Split evidence pending full-data replay.' + if data.pending + else 'Complete source sorting; exact counts and finite-value means.' + ) + self._selected_summaries[cluster_id] = data + else: + self._selected_summaries.move_to_end(cluster_id) + if data.pending: + self._trim_cache() + return data + for metric in metrics: + if metric in data.loaded: + continue + if metric == 'firing_rate': + if session is None: + data.counts = np.histogram(controller.get_spike_times(cluster_id), data.edges)[ + 0 + ] + else: + counts = np.zeros(len(data.exposure), dtype=np.int64) + grid = self._array(common.summary['counts_file']) + for row in data.rows: + counts += grid[row] + data.counts = counts + data.rates = np.full(len(data.exposure), np.nan) + np.divide(data.counts, data.exposure, out=data.rates, where=data.exposure > 0) + elif session is not None: + sum_file, count_file = f'{metric}_sum_file', f'{metric}_count_file' + if sum_file in common.summary and count_file in common.summary: + total = np.zeros(len(data.exposure)) + valid = np.zeros(len(data.exposure), dtype=np.int64) + sums = self._array(common.summary[sum_file]) + counts = self._array(common.summary[count_file]) + for row in data.rows: + total += sums[row] + valid += counts[row] + mean = np.full(len(data.exposure), np.nan) + np.divide(total, valid, out=mean, where=(valid > 0) & (data.exposure > 0)) + data.trend_sums[metric] = total + data.trend_counts[metric] = valid + data.trends[metric] = mean + data.loaded.add(metric) + data.display = None + self._trim_cache() + return data + + @property + def displayed_ids(self): + """Presentation subset; the supervisor selection remains authoritative.""" + return self.cluster_ids[: self.display_limit] if self.display_limit else self.cluster_ids + + @staticmethod + def _data_bytes(data): + # Common arrays are shared/mapped, and are deliberately outside the row cache. + overhead = getsizeof(data) + getsizeof(data.loaded) + rows = data.get('rows', []) + overhead += getsizeof(rows) + sum(map(getsizeof, rows)) + arrays = [data.get('counts'), data.get('rates')] + for key in ('trends', 'trend_sums', 'trend_counts'): + overhead += getsizeof(data.get(key, {})) + arrays.extend(data.get(key, {}).values()) + display = data.get('display') + if display is not None: + overhead += getsizeof(display) + getsizeof(display.trends) + arrays.extend([display.edges, display.exposure, display.counts, display.rates]) + arrays.extend(display.trends.values()) + return overhead + sum( + a.nbytes + getsizeof(a) - (a.nbytes if a.flags.owndata else 0) + for a in arrays + if isinstance(a, np.ndarray) + ) + + def _trim_cache(self): + total = sum(self._data_bytes(data) for data in self._selected_summaries.values()) + while self._selected_summaries and total > self.summary_cache_bytes: + _, data = self._selected_summaries.popitem(last=False) + total -= self._data_bytes(data) + + def _display_factor(self, seconds=None): + seconds = self.display_bin_seconds if seconds is None else float(seconds) + if not np.isfinite(seconds) or seconds < 0: + raise ValueError('Display bin size must be zero (native) or finite and positive.') + if seconds == 0: + return 1 + widths = np.diff(self._common().edges) + native = widths[0] if len(widths) else np.nan + if ( + not np.all(np.isfinite(widths)) + or not np.isfinite(native) + or native <= 0 + or not np.allclose(widths[:-1], native, rtol=1e-10, atol=1e-10) + or widths[-1] <= 0 + or (widths[-1] > native and not np.isclose(widths[-1], native, rtol=1e-10, atol=1e-10)) + ): + raise ValueError( + 'Display bin size in seconds requires uniform native bins, ' + 'with an optional shorter final bin. Use 0 for native bins.' + ) + factor = seconds / native + if ( + not np.isfinite(factor) + or factor < 1 + or not np.isclose(factor, round(factor), rtol=1e-10, atol=1e-10) + ): + raise ValueError( + 'Display bin size must be an integer multiple of the native bin size.' + ) + return int(round(factor)) + + def get_display_data(self, cluster_id, metrics=None): + """Group adjacent native bins using additive evidence, without accessing spikes.""" + data = self.get_data(cluster_id, metrics) + factor = self._display_factor() + if data.pending or factor == 1: + return data + if data.display is None or data.display.factor != factor: + starts = np.arange(0, len(data.exposure), min(factor, len(data.exposure))) + exposure = np.add.reduceat(data.exposure, starts) + counts = None if data.counts is None else np.add.reduceat(data.counts, starts) + rates = np.full(len(starts), np.nan) + if counts is not None: + np.divide(counts, exposure, out=rates, where=exposure > 0) + trends = {} + for metric, total in data.trend_sums.items(): + sums = np.add.reduceat(total, starts) + valid = np.add.reduceat(data.trend_counts[metric], starts) + mean = np.full(len(starts), np.nan) + np.divide(sums, valid, out=mean, where=(valid > 0) & (exposure > 0)) + trends[metric] = mean + data.display = Bunch( + factor=factor, + edges=np.r_[data.edges[starts], data.edges[-1]], + exposure=exposure, + counts=counts, + rates=rates, + trends=trends, + pending=False, + provenance=data.provenance, + ) + display = data.display + self._trim_cache() + return display + + def set_display_limit(self, value): + """Set maximum displayed units (0 for all; otherwise a positive integer).""" + number = float(value) + if not np.isfinite(number) or number != int(number) or number < 0: + raise ValueError('Display limit must be an nonnegative integer (0 shows all).') + self.display_limit = int(number) + self.plot() + + def set_rate_line_width(self, value): + """Set rate line width in points (finite and positive).""" + number = float(value) + if not np.isfinite(number) or number <= 0: + raise ValueError('Rate line width must be finite and positive.') + self.rate_line_width = number + self.plot() + + def set_display_bin_seconds(self, value): + """Set display bin size in seconds (0 uses native prepared bins).""" + self._display_factor(value) + self.display_bin_seconds = float(value) + for data in self._selected_summaries.values(): + data.display = None + self.plot() + + def set_state(self, state): + state = dict(state) + for key, default in ( + ('display_limit', 32), + ('rate_line_width', 0.6), + ('display_bin_seconds', 0.0), + ): + if key not in state: + continue + try: + number = float(state[key]) + valid = np.isfinite(number) + if key == 'display_limit': + valid = valid and number.is_integer() and number >= 0 + elif key == 'rate_line_width': + valid = valid and number > 0 + else: + self._display_factor(number) + state[key] = int(number) if key == 'display_limit' and valid else number + if not valid: + state[key] = default + except (ValueError, TypeError, OverflowError): + state[key] = default + super().set_state(state) + + def screenshot(self, dir=None): + """Save the actual Matplotlib Qt canvas using normal screenshot naming.""" + path = screenshot_default_path(self, dir=dir) + return screenshot(self.canvas.canvas, path=path) + + @property + def status(self): + n = len(self.cluster_ids) + return ( + f'Showing first {self.display_limit} of {n} selected units; ' + 'display truncation does not change the selection or merge operation.' + if self.display_limit and n > self.display_limit + else '' + ) + + def _layout_axes(self, visible): + axes = list(self.canvas.iter_ax()) + for metric, ax in zip(self._metrics, axes): + ax.clear() + ax.set_visible(metric in visible) + for index, metric in enumerate(visible): + ax = axes[self._metrics.index(metric)] + height = 0.76 / len(visible) + ax.set_position( + [ + 0.12, + 0.15 + (len(visible) - 1 - index) * height, + 0.84, + height * (0.84 if len(visible) > 1 else 1), + ] + ) + ax.tick_params(axis='x', labelbottom=index == len(visible) - 1) + if index == len(visible) - 1: + ax.set_xlabel('Elapsed acquisition clock (hours)') + return axes + + def plot(self, **kwargs): + visible = [metric for metric in self._metrics if getattr(self, f'show_{metric}')] + axes = self._layout_axes(visible) + self._footer.set_text('') + # Reuse an existing axis for the explicitly empty panel. + if not visible: + axes[0].set_visible(True) + axes[0].set_position([0.12, 0.15, 0.84, 0.76]) + axes[0].set_axis_off() + axes[0].text( + 0.02, + 0.6, + 'No metrics enabled. Use Show firing rate, amplitude or depth.', + transform=axes[0].transAxes, + ) + else: + common = self._common() + self._footer.set_text(common.footer) + first_ax = axes[self._metrics.index(visible[0])] + notes = [] + unavailable = {metric: [] for metric in visible} + if not self.cluster_ids: + notes.append('Select a unit to inspect full-duration evidence.') + for index, cluster_id in enumerate(self.displayed_ids): + data = self.get_display_data(cluster_id, visible) + if data.pending: + notes.append(f'Unit {cluster_id}: {data.provenance}') + continue + color = selected_cluster_color(self.cluster_color_index(cluster_id, index)) + x = (data.edges[:-1] + data.edges[1:]) / 7200 + for metric in visible: + ax = axes[self._metrics.index(metric)] + values = data.rates if metric == 'firing_rate' else data.trends.get(metric) + if values is not None and np.any(np.isfinite(values)): + ax.plot( + x, + values, + color=color, + label=str(cluster_id), + linewidth=self.rate_line_width if metric == 'firing_rate' else 1.5, + ) + elif metric != 'firing_rate': + unavailable[metric].append(cluster_id) + labels = ('Rate (Hz)', 'Mean amplitude\n(source units)', 'Mean depth\n(source units)') + for metric in visible: + ax = axes[self._metrics.index(metric)] + ax.set_axis_on() + ax.set_ylabel(labels[self._metrics.index(metric)]) + if unavailable[metric]: + missing = ', '.join(map(str, unavailable[metric])) + message = ( + f'{metric.title()} measurement unavailable.' + if not ax.lines + else f'{metric.title()} measurement unavailable for units: {missing}.' + ) + ax.text(0.02, 0.96, message, transform=ax.transAxes, va='top', fontsize=8) + for start, end in common.windows: + ax.axvspan(start / 3600, end / 3600, color='cyan', alpha=0.15) + previous = 0 + for start, end in common.intervals: + if start > previous: + ax.axvspan(previous / 3600, start / 3600, color='gray', alpha=0.4) + previous = end + if previous < common.edges[-1]: + ax.axvspan(previous / 3600, common.edges[-1] / 3600, color='gray', alpha=0.4) + ax.set_xlim(common.edges[0] / 3600, common.edges[-1] / 3600) + provenance = ( + 'Complete source sorting; exact counts and finite-value means.' + if getattr(self.controller, 'bounded_session', None) is not None + else 'Loaded recording; continuous acquisition assumed.' + ) + first_ax.set_title(provenance, fontsize=9) + if notes: + first_ax.text( + 0.02, + 0.96, + '\n'.join(notes), + transform=first_ax.transAxes, + va='top', + fontsize=8, + ) + self.update_status() + self.canvas.update() + + def on_select(self, cluster_ids=None, **kwargs): + if self._attaching: + supervisor = self.controller.supervisor + cluster_ids = supervisor.selected + self._update_cluster_color_indices(supervisor, cluster_ids) + self.cluster_ids = tuple(cluster_ids or ()) + selected = set(self.displayed_ids) + self._selected_summaries = OrderedDict( + (c, d) for c, d in self._selected_summaries.items() if c in selected + ) + self.plot(**kwargs) + + def _update_cluster_color_indices(self, sender, cluster_ids): + # Hidden-view replay already adopted the authoritative palette at receipt. + if sender is not self: + super()._update_cluster_color_indices(sender, cluster_ids) + + def on_select_threaded(self, sender, cluster_ids, gui=None, **kwargs): + # The base intentionally ignores empty selections; this evidence panel must clear. + if not cluster_ids: + if ( + not self.auto_update + or self._closed + or sender.__class__.__name__ in ('ClusterView', 'SimilarityView') + ): + return + self._update_cluster_color_indices(sender, cluster_ids) + if self.defer_hidden_updates and not self._dock_visible: + self.cluster_ids = () + self._pending_selection = ([], dict(kwargs)) + return + self._pending_selection = None + self.on_select([], **kwargs) + return + return super().on_select_threaded(sender, cluster_ids, gui=gui, **kwargs) + + def _toggle_metric(self, metric, checked): + setattr(self, f'show_{metric}', bool(checked)) + if hasattr(self, 'actions'): + self.actions.get(f'toggle_{metric}').setChecked(bool(checked)) + if not self._closed: + self.plot() + + def toggle_firing_rate(self, checked): + """Show firing rate.""" + self._toggle_metric('firing_rate', checked) + + def toggle_amplitude(self, checked): + """Show amplitude.""" + self._toggle_metric('amplitude', checked) + + def toggle_depth(self, checked): + """Show depth.""" + self._toggle_metric('depth', checked) + + def _invalidate(self): + self._selected_summaries.clear() + + def _release_cache(self): + self._summary_arrays.clear() + self._selected_summaries.clear() + self._common_data = None + + def close(self): + """Release mapped summary references and cached rows, including unattached views.""" + result = super().close() + self._release_cache() + return result + + def attach(self, gui): + self._attaching = True + try: + super().attach(gui) + finally: + self._attaching = False + for metric in self._metrics: + self.actions.add( + getattr(self, f'toggle_{metric}'), + checkable=True, + checked=getattr(self, f'show_{metric}'), + docstring=f'Show {metric.replace("_", " ")}.', + ) + self.actions.get(f'toggle_{metric}').setText(f'Show {metric.replace("_", " ")}') + + for action in ( + self.set_display_limit, + self.set_rate_line_width, + self.set_display_bin_seconds, + ): + self.actions.add(action, prompt=True) + + @self.actions.add(prompt=True) + def show_source_unit(cluster_id): + """Inspect a full source unit, including one without sampled evidence.""" + session = getattr(self.controller, 'bounded_session', None) + if session is not None and int(cluster_id) not in session.unit_ids: + raise ValueError('Unknown source unit ID.') + self._cluster_color_index_by_id = {} + self.on_select([int(cluster_id)]) + + @connect(event='bounded_state_changed', sender=self.controller.supervisor) + def on_state(sender): + self._invalidate() + if not self._closed: + self.plot() + + @connect(event='cluster', sender=self.controller.supervisor) + def on_cluster(sender, up): + self._invalidate() + # Supervisor selection callbacks may precede this listener. Refresh + # ordinary data after invalidation even if the selected IDs stayed put. + if ( + getattr(self.controller, 'bounded_session', None) is None + and self.auto_update + and not self._closed + and not (self.defer_hidden_updates and not self._dock_visible) + ): + self.plot() + + @connect(event='close_view', sender=self) + def on_close(sender, *args): + unconnect(on_state, on_cluster, on_close) + self._release_cache() diff --git a/phy/apps/large_recording/statistics.py b/phy/apps/large_recording/statistics.py new file mode 100644 index 000000000..04742de84 --- /dev/null +++ b/phy/apps/large_recording/statistics.py @@ -0,0 +1,67 @@ +"""Descriptive statistics restricted to retained contiguous observation windows.""" + +import numpy as np +from phylib.stats import correlograms + + +def window_isi(samples, windows, sample_rate): + intervals = [] + for start, end in windows: + first, last = np.searchsorted(samples, [start, end]) + intervals.append(np.diff(samples[first:last]) / sample_rate) + return np.concatenate(intervals) if intervals else np.array([], dtype=np.float64) + + +def window_correlograms( + samples, clusters, windows, cluster_ids, sample_rate, bin_size, window_size +): + """Exact window-local pair counts, without a full-duration rate baseline. + + Positive/negative lag bins follow phylib's floor-to-bin convention. The + central bin counts unordered within-unit pairs and cross-unit pairs with + absolute delays below one bin. Cross-unit central counts sum both temporal + orientations, making them independent of equal-time source row order. + """ + selected = np.isin(clusters, cluster_ids) + order = np.argsort(cluster_ids) + sorted_ids = np.asarray(cluster_ids)[order] + result = None + for start, end in windows: + first, last = np.searchsorted(samples, [start, end]) + mask = selected[first:last] + ticks = samples[first:last][mask] - start + if len(ticks) and ticks.max() >= 2**50: + raise ValueError( + 'Observation window is too long for exact correlogram sample conversion.' + ) + # phylib truncates float times * rate to integer samples. A common + # quarter-sample offset preserves every integer tick after conversion + # without changing any pair delay, including late-clock recordings. + times = (ticks.astype(np.float64) + 0.25) / sample_rate + # Avoid phylib's lookup allocation indexed by absolute cluster ID. + labels = order[np.searchsorted(sorted_ids, clusters[first:last][mask])] + if np.any(np.bincount(labels, minlength=len(cluster_ids)) > 46340): + raise ValueError( + 'Evidence window has too many spikes for exact correlogram counts; prepare shorter windows.' + ) + positive = correlograms( + times, + labels, + sample_rate=sample_rate, + cluster_ids=np.arange(len(cluster_ids)), + bin_size=bin_size, + window_size=window_size, + symmetrize=False, + ).astype(np.int64) + # Accumulate before symmetrization to preserve exact central counts and + # int64 totals across many windows. Self-spike pairs are never counted. + if result is None: + result = positive + else: + result += positive + if result is None: + raise ValueError('Correlograms require at least one observation window.') + center = result[..., 0] + result[..., 0].T + np.fill_diagonal(center, np.diag(result[..., 0])) + result[..., 0] = center + return np.dstack((result[..., 1:][..., ::-1].transpose(1, 0, 2), result)) diff --git a/phy/apps/large_recording/summaries.py b/phy/apps/large_recording/summaries.py new file mode 100644 index 000000000..2f498fab5 --- /dev/null +++ b/phy/apps/large_recording/summaries.py @@ -0,0 +1,122 @@ +"""Exact file-backed counts and additive source-measurement summaries.""" + +from contextlib import ExitStack +from pathlib import Path + +import numpy as np + +from .source import close_array + + +def scalar_attributes(source): + """Return available scalar measurements without inventing physical units.""" + attributes = {} + amplitude = source.arrays.get('amplitudes.npy') + if amplitude is not None and (amplitude.ndim == 1 or amplitude.shape[1:] == (1,)): + attributes['amplitude'] = amplitude.reshape(-1) + depth = source.arrays.get('spike_depths.npy') + if depth is not None and (depth.ndim == 1 or depth.shape[1:] == (1,)): + attributes['depth'] = depth.reshape(-1) + elif (positions := source.arrays.get('spike_positions.npy')) is not None: + axis = source.params.get('depth_axis') + if axis is not None: + if ( + not isinstance(axis, int) + or positions.ndim != 2 + or not 0 <= axis < positions.shape[1] + ): + raise ValueError('depth_axis must declare a valid spike_positions.npy coordinate.') + attributes['depth'] = positions[:, axis] + return attributes + + +def summary_size(source, bin_seconds=3600): + """Estimate exact array bytes before allocating the dense output grid.""" + step = float(bin_seconds) * source.sample_rate + if not np.isfinite(step) or step < 1 or not np.isclose(step, round(step), rtol=0, atol=1e-6): + raise ValueError('bin_seconds must specify a positive integer sample width.') + step = int(round(step)) + bins = (source.num_samples + step - 1) // step + size = len(source.unit_ids) * bins * 8 * (1 + 2 * len(scalar_attributes(source))) + size += (bins * 2 + 1) * 8 + return size, step, bins + + +def build_summaries(source, output_dir, bin_seconds=3600, max_bytes=536870912): + """Stream exact full counts, exposure, and finite-value sums/counts. + + Exposure is one shared vector. Summing parent-unit counts on merge must + divide by this vector once. Scalar means are sums/counts, never medians. + """ + size, step, bins = summary_size(source, bin_seconds) + if size + 1024 * (1 + 2 * len(scalar_attributes(source))) > max_bytes: + raise ValueError( + f'Summary budget exceeded: need at least {size} array bytes; ' + 'increase bin_seconds or max_bytes.' + ) + with ExitStack() as stack: + output_dir = Path(output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + result = { + 'counts_file': '_phy_summary_counts.npy', + 'edges_file': '_phy_summary_edges.npy', + 'exposure_file': '_phy_summary_exposure.npy', + } + shape = (len(source.unit_ids), bins) + counts = np.lib.format.open_memmap( + output_dir / result['counts_file'], mode='w+', dtype=np.int64, shape=shape + ) + stack.callback(close_array, counts) + counts[:] = 0 + attributes = scalar_attributes(source) + grids = {} + for field in attributes: + for kind, dtype in (('sum', np.float64), ('count', np.int64)): + key = f'{field}_{kind}_file' + result[key] = f'_phy_summary_{field}_{kind}.npy' + array = np.lib.format.open_memmap( + output_dir / result[key], mode='w+', dtype=dtype, shape=shape + ) + stack.callback(close_array, array) + array[:] = 0 + grids[field, kind] = array + edges = np.lib.format.open_memmap( + output_dir / result['edges_file'], mode='w+', dtype=np.int64, shape=(bins + 1,) + ) + stack.callback(close_array, edges) + # Bounded chunks also avoid a recording-duration-sized temporary arange. + for start in range(0, bins + 1, source.batch_size): + stop = min(start + source.batch_size, bins + 1) + edges[start:stop] = np.arange(start, stop, dtype=np.int64) * step + edges[-1] = source.num_samples + exposure = np.lib.format.open_memmap( + output_dir / result['exposure_file'], mode='w+', dtype=np.float64, shape=(bins,) + ) + stack.callback(close_array, exposure) + exposure[:] = 0 + for start, end in source.valid_intervals: + first, last = int(start // step), int((end - 1) // step) + for offset in range(first, last + 1, source.batch_size): + stop = min(offset + source.batch_size, last + 1) + exposure[offset:stop] += ( + np.minimum(edges[offset + 1 : stop + 1], end) + - np.maximum(edges[offset:stop], start) + ) / source.sample_rate + for batch in source.iter_batches(): + rows = np.searchsorted(source.unit_ids, batch['spike_clusters.npy']) + columns = batch['spike_times.npy'] // step + np.add.at(counts, (rows, columns), 1) + for field, values in attributes.items(): + values = values[batch['start'] : batch['stop']] + finite = np.isfinite(values) + np.add.at(grids[field, 'sum'], (rows[finite], columns[finite]), values[finite]) + np.add.at(grids[field, 'count'], (rows[finite], columns[finite]), 1) + for array in (counts, edges, exposure, *grids.values()): + array.flush() + result['estimators'] = { + 'counts': 'exact', + 'exposure': 'acquisition_seconds', + **dict.fromkeys(attributes, 'finite_sum_and_count'), + } + result['measurement_units'] = {field: f'source {field} units' for field in attributes} + return result diff --git a/phy/apps/large_recording/supervisor.py b/phy/apps/large_recording/supervisor.py new file mode 100644 index 000000000..b6ee1cd77 --- /dev/null +++ b/phy/apps/large_recording/supervisor.py @@ -0,0 +1,526 @@ +"""Guarded native commands paired with semantic state, history and checkpoints.""" + +from contextlib import contextmanager +from copy import deepcopy +from functools import wraps +from types import MappingProxyType + +import numpy as np +from phylib.utils import emit + +from phy.cluster._history import History +from phy.cluster._utils import ClusterMeta +from phy.cluster.clustering import Clustering +from phy.cluster.supervisor import Supervisor + +from .curation import evaluate_predicate, interpret_events + + +class MutationPolicy: + """Only semantic commands may mutate native bounded state.""" + + def __init__(self): + self.depth = 0 + + def require(self): + if not self.depth: + raise ValueError( + 'Bounded curation requires a semantic command. Use supervisor.merge(), ' + 'supervisor.label(), or split_source_amplitude(); arbitrary splits are unavailable.' + ) + + @contextmanager + def command(self): + self.depth += 1 + try: + yield + finally: + self.depth -= 1 + + +class GuardedClustering(Clustering): + def __init__(self, *args, policy, **kwargs): + self.policy = policy + super().__init__(*args, **kwargs) + + @property + def spike_clusters(self): + result = self._spike_clusters.view() + result.flags.writeable = bool(self.policy.depth) + return result + + @property + def cluster_ids(self): + result = self._cluster_ids.view() + result.flags.writeable = False + return result + + @property + def spikes_per_cluster(self): + return MappingProxyType(self._spikes_per_cluster) + + def _update_cluster_ids(self, *args, **kwargs): + super()._update_cluster_ids(*args, **kwargs) + for ids in self._spikes_per_cluster.values(): + ids.flags.writeable = False + + def merge(self, *args, **kwargs): + self.policy.require() + return super().merge(*args, **kwargs) + + def assign(self, *args, **kwargs): + self.policy.require() + return super().assign(*args, **kwargs) + + def reset(self): + self.policy.require() + return super().reset() + + def undo(self): + self.policy.require() + return super().undo() + + def redo(self): + self.policy.require() + return super().redo() + + +class GuardedMeta(ClusterMeta): + def __init__(self, policy): + self.policy = policy + super().__init__() + + def set(self, *args, **kwargs): + self.policy.require() + return super().set(*args, **kwargs) + + def add_field(self, *args, **kwargs): + self.policy.require() + return super().add_field(*args, **kwargs) + + def from_dict(self, *args, **kwargs): + self.policy.require() + return super().from_dict(*args, **kwargs) + + def undo(self): + self.policy.require() + return super().undo() + + def redo(self): + self.policy.require() + return super().redo() + + +class SessionHistory(History): + def __init__(self, session): + self.session = session + super().__init__(session.journal_state()) + + def capture(self): + self.add(self.session.journal_state()) + + def undo(self): + self.back() + self.session.restore_journal(self.current_item) + + def redo(self): + self.forward() + self.session.restore_journal(self.current_item) + + +def semantic_command(method): + @wraps(method) + def execute(self, *args, **kwargs): + with self._transaction(): + return method(self, *args, **kwargs) + + return execute + + +class BoundedSupervisor(Supervisor): + """Native UI/history operations paired with a validated semantic journal.""" + + def __init__(self, *args, session, **kwargs): + self.session = session + self.policy = MutationPolicy() + self.select_actions = None + self.cluster_view = None + self.similarity_view = None + + def clustering_factory(*args, **kwargs): + return GuardedClustering(*args, policy=self.policy, **kwargs) + + def meta_factory(groups, labels): + meta = GuardedMeta(self.policy) + data = {} + for field, values in labels.items(): + for cluster, value in values.items(): + data.setdefault(cluster, {})[field] = value + with self.policy.command(): + meta.add_field('group') + meta.from_dict(data) + return meta + + # Generation state supersedes disposable clustering caches. + kwargs['context'] = None + kwargs['new_cluster_id'] = session.next_cluster_id + kwargs['clustering_factory'] = clustering_factory + kwargs['cluster_meta_factory'] = meta_factory + super().__init__(*args, **kwargs) + self.session_history = SessionHistory(session) + + @contextmanager + def _transaction(self): + """Roll native and semantic state back if an observer or command fails.""" + clustering = self.clustering + arrays = ( + clustering._spike_clusters.copy(), + dict(clustering._spikes_per_cluster), + clustering._cluster_ids.copy(), + ) + data, fields = deepcopy(self.cluster_meta._data), dict(self.cluster_meta._fields) + histories = [ + clustering._undo_stack, + self.cluster_meta._undo_stack, + self._global_history, + self.session_history, + ] + if self.merge_propositions is not None: + histories.append(self.merge_propositions._history) + history_states = [(list(h._history), h._index) for h in histories] + journal, selection, dirty = ( + self.session.journal_state(), + self.selection.snapshot(), + self._is_dirty, + ) + propositions = ( + self.merge_propositions.snapshot() if self.merge_propositions is not None else None + ) + workflow, columns = self._workflow_context(), list(self.columns) + try: + yield + except Exception: + clustering._spike_clusters, clustering._spikes_per_cluster, clustering._cluster_ids = ( + arrays + ) + self.cluster_meta._data, self.cluster_meta._fields = data, fields + for history, (items, index) in zip(histories, history_states): + history._history, history._index = items, index + self.session.restore_journal(journal) + self.session.next_cluster_id = max( + self.session.next_cluster_id, clustering.new_cluster_id() + ) + self.selection.restore(selection) + if propositions is not None: + self.merge_propositions.restore(propositions) + self._is_dirty = dirty + self.columns = columns + self._lineage_cache = None + if self.gui is not None: + self._reset_cluster_view() + self._restore_history_context( + selection, + { + 'mode': 'merge' if selection.is_merge_mode else 'post_merge', + 'tables': workflow, + }, + 'undo', + ) + self._refresh_bounded() + self._update_save_feedback() + raise + + def _require_capability(self, name): + if not self.session.manifest.get('capabilities', {}).get(name, False): + raise ValueError(f'This bounded package does not allow {name} operations.') + + def get_cluster_info(self, cluster_id, exclude=()): + info = super().get_cluster_info(cluster_id, exclude=exclude) + # Producer TSV labels may use metric names. They must not overwrite + # authoritative complete-sorting counts/rates with inherited values. + for name in ( + 'n_spikes', + 'fr', + 'full_count', + 'evidence_count', + 'waveform_count', + 'evidence', + ): + if name not in exclude and name in self.cluster_metrics: + info[name] = self.cluster_metrics[name](cluster_id) + return info + + def _select_after_merge(self, *args, **kwargs): + if self.gui is not None: + return super()._select_after_merge(*args, **kwargs) + + def _select_after_split(self, *args, **kwargs): + if self.gui is not None: + return super()._select_after_split(*args, **kwargs) + + def _restore_history_context(self, *args, **kwargs): + if self.gui is not None: + return super()._restore_history_context(*args, **kwargs) + + def _reset_cluster_view(self, *args, **kwargs): + if self.gui is not None: + return super()._reset_cluster_view(*args, **kwargs) + + def _workflow_context(self): + return super()._workflow_context() if self.gui is not None else {} + + def _validate_native(self, metadata=True): + if not np.array_equal(self.clustering.spike_clusters, self.session.assignments): + raise ValueError('Native assignments disagree with the bounded semantic command.') + if metadata: + active, expected, _ = interpret_events( + self.session.unit_ids, self.session.base_metadata, self.session.events + ) + for field in set(self.cluster_meta.fields) | set(expected): + for cluster in active: + actual = ( + self.cluster_meta.get(field, cluster) + if field in self.cluster_meta.fields + else None + ) + if actual != expected.get(field, {}).get(cluster): + raise ValueError( + 'Native metadata disagrees with the bounded semantic command.' + ) + + def _synchronize_metadata(self): + # Native ClusterMeta.undo does not retain nonstacked inherited merge + # labels. The semantic projection is authoritative in bounded mode. + data = {} + for field, values in self.session.metadata.items(): + if field not in self.cluster_meta.fields: + with self.policy.command(): + self.cluster_meta.add_field(field) + for cluster, value in values.items(): + data.setdefault(cluster, {})[field] = value + self.cluster_meta._data = data + + def _refresh_bounded(self): + if self.merge_propositions is not None: + self.merge_propositions.project_live_ids( + self.clustering.cluster_ids, retired_units=self.session.retired_units + ) + if self.gui is not None: + data = self.cluster_info + self.cluster_view.change(data) + self.similarity_view.change(data) + if self.merge_view is not None: + self.merge_view.change(data) + emit('bounded_state_changed', self) + + def _capture(self): + if self.session.metadata_policy: + self._synchronize_metadata() + self._validate_native() + self.session_history.capture() + self._global_history.add_to_current_action(self.session_history) + self._is_dirty = True + self._update_save_feedback() + self._synchronize_metadata() + self._refresh_bounded() + + @semantic_command + def merge(self, cluster_ids=None, to=None): + self._require_capability('merge') + parents = list(self.selected if cluster_ids is None else cluster_ids) + if len(parents) < 2: + return + target = max(self.clustering.new_cluster_id(), self.session.next_cluster_id) + target = target if to is None else int(to) + self.session.preflight_merge(parents, target) + with self.policy.command(): + out = super().merge(cluster_ids=cluster_ids, to=target) + if out is None: + return + resolved = { + field: self.cluster_meta.get(field, target) for field in self.cluster_meta.fields + } + self.session.record_merge(parents, target, metadata=resolved) + self._capture() + return out + + @semantic_command + def label(self, name, value, cluster_ids=None): + self._require_capability('label') + if name in ( + 'n_spikes', + 'fr', + 'full_count', + 'evidence_count', + 'waveform_count', + 'evidence', + ): + raise ValueError('Bounded count/rate/evidence columns are read-only metrics.') + ids = list(self.selected if cluster_ids is None else np.atleast_1d(cluster_ids)) + if not ids or self.selection.state.is_merge_mode: + return + # Preflight against a copy before touching native metadata. + before = self.session.snapshot() + self.session.record_label(name, ids, value) + self.session.restore(before) + with self.policy.command(): + super().label(name, value, cluster_ids=ids) + self.session.record_label(name, ids, value) + self._capture() + + def _graphical_split_views(self): + if self.gui is None: + return [] + return [ + view + for view in self.gui.list_views('EvidenceAmplitudeView') + if not view._closed + and view.gui is self.gui + and view._dock_visible + and view.split_threshold is not None + and np.isfinite(view.split_threshold) + and list(view.cluster_ids) == list(self.selected) + and len(self.selected) == 1 + and not self.selection.state.is_merge_mode + and view.amplitudes_type == 'source' + ] + + def _update_graphical_split_action(self): + actions = getattr(self, 'actions', None) + action = actions.get('split') if actions is not None else None + if action is not None: + capable = 'amplitudes.npy' in self.session.capabilities['predicate_split_attributes'] + action.setEnabled(capable and len(self._graphical_split_views()) == 1) + + def _set_merge_mode_ui(self, active): + if self.gui is None: + return + super()._set_merge_mode_ui(active) + if active: + for view in self.gui.list_views('EvidenceAmplitudeView'): + view.clear_amplitude_split_threshold() + self._update_graphical_split_action() + + def split(self, spike_ids=None, spike_clusters_rel=0): + if spike_ids is not None or spike_clusters_rel != 0: + raise ValueError( + 'Lasso, feature and waveform splits cannot be replayed on unseen spikes. ' + 'Use Split by source amplitude for a full-source scalar predicate.' + ) + views = self._graphical_split_views() + if len(views) != 1: + raise ValueError('Exactly one visible source-amplitude threshold preview is required.') + view = views[0] + threshold = view.split_threshold + cluster_id = int(self.selected[0]) + try: + out = self.split_source_amplitude(cluster_id, threshold) + except Exception as error: + # A transaction may have published transient native selection updates + # before rollback. Keep the user's predicate available for correction. + view.split_threshold = threshold + view._replot_displayed_amplitudes() + emit( + 'amplitude_split_preview_changed', + view, + cluster_id=cluster_id, + amplitudes_type='source', + threshold=threshold, + ) + view._show_split_status(str(error)) + raise + view.clear_split_selection() + self._update_graphical_split_action() + return out + + @semantic_command + def split_source_amplitude(self, cluster_id, threshold): + if 'amplitudes.npy' not in self.session.manifest.get('capabilities', {}).get( + 'predicate_split_attributes', [] + ): + raise ValueError('This package does not support source-amplitude predicate splits.') + if self.selection.state.is_merge_mode: + raise ValueError('Leave Merge mode before splitting by source amplitude.') + if 'amplitudes.npy' not in self.session.attributes: + raise ValueError('The source has no scalar per-spike amplitudes.') + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': float(threshold), + 'nonfinite': 'keep', + } + ids = self.clustering.spikes_per_cluster.get(int(cluster_id), np.array([], dtype=np.int64)) + selected = ids[ + evaluate_predicate(self.session.attributes['amplitudes.npy'][ids], predicate) + ] + if not len(selected) or len(selected) == len(ids): + raise ValueError('Both sides of the threshold must have evidence spikes.') + selected_id = max(self.clustering.new_cluster_id(), self.session.next_cluster_id) + remainder_id = selected_id + 1 + self.clustering._new_cluster_id = selected_id + metadata = { + field: { + child: self.cluster_meta.get(field, cluster_id) + for child in (selected_id, remainder_id) + } + for field in self.cluster_meta.fields + } + before = self.session.snapshot() + self.session.record_threshold_split( + int(cluster_id), [remainder_id, selected_id], predicate, metadata=metadata + ) + self.session.restore(before) + with self.policy.command(): + out = Supervisor.split(self, selected) + self.session.record_threshold_split( + int(cluster_id), [remainder_id, selected_id], predicate, metadata=metadata + ) + self._capture() + return out + + @semantic_command + def undo(self): + with self.policy.command(): + super().undo() + self._after_history() + + @semantic_command + def redo(self): + with self.policy.command(): + super().redo() + self._after_history() + + def _after_history(self): + self._validate_native(metadata=False) + # Never reuse allocated IDs after branching or restoring a snapshot. + self.session.next_cluster_id = max( + self.session.next_cluster_id, self.clustering.new_cluster_id() + ) + self._is_dirty = True + self._update_save_feedback() + self._synchronize_metadata() + self._refresh_bounded() + + def save(self): + self._validate_native() + self.session.validate_proposition_documents() + documents = None + if self.merge_propositions is not None: + documents = dict(self.session.extra_documents) + documents.update( + { + 'curation_review.json': self.merge_propositions.catalog.review_mapping( + source_sha256=self.session.proposition_documents['curation.json']['sha256'] + ) + } + ) + self.session.save( + assignments=self.clustering.spike_clusters, + metadata=self.session.metadata, + next_cluster_id=max(self.session.next_cluster_id, self.clustering.new_cluster_id()), + extra_documents=documents, + ) + if self.merge_propositions is not None: + self.merge_propositions.mark_saved() + self._is_dirty = False + self._update_save_feedback(saved=True) diff --git a/phy/apps/large_recording/tests/__init__.py b/phy/apps/large_recording/tests/__init__.py new file mode 100644 index 000000000..ba239ecd8 --- /dev/null +++ b/phy/apps/large_recording/tests/__init__.py @@ -0,0 +1 @@ +"""Long-recording workflow regression tests.""" diff --git a/phy/apps/large_recording/tests/test_adapter.py b/phy/apps/large_recording/tests/test_adapter.py new file mode 100644 index 000000000..0c44051ac --- /dev/null +++ b/phy/apps/large_recording/tests/test_adapter.py @@ -0,0 +1,450 @@ +"""Bounded GUI correctness and ordinary-controller isolation.""" + +import json + +import numpy as np +import pytest +from click.testing import CliRunner +from phylib.utils import connect, unconnect + +from phy.apps import phycli +from phy.apps.large_recording.adapter import BoundedTemplateController +from phy.apps.large_recording.curation import BoundedSession +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.tests.test_source import make_sorting +from phy.apps.template.gui import TemplateController + + +def make_controller(tmp_path, sparse=False, propositions=None): + params = make_gui_sorting(tmp_path / 'source') + if propositions is not None: + (params.parent / 'curation.json').write_text(json.dumps(propositions)) + output = prepare_large( + params, + tmp_path / 'bounded', + window_seconds=3 if sparse else 20, + stratum_seconds=20, + max_spikes=5 if sparse else 100, + bin_seconds=5, + batch_size=2, + ) + controller = TemplateController( + dir_path=output, + config_dir=tmp_path / 'config', + plugins=[], + enable_threading=False, + clear_cache=True, + ) + return params, controller + + +def make_gui_sorting(path): + params = make_sorting(path) + times = np.load(path / 'spike_times.npy') + np.save(path / 'spike_times.npy', times * 30000) + params.write_text( + params.read_text() + .replace('sample_rate = 1.0', 'sample_rate = 30000.0') + .replace('num_samples = 20', 'num_samples = 600000') + ) + return params + + +def test_manifest_factory_reserves_complete_ids_and_guards_mutations(tmp_path): + _, controller = make_controller(tmp_path, sparse=True) + try: + assert isinstance(controller, BoundedTemplateController) + with pytest.raises(ValueError): + controller.model.spike_clusters[0] = 123 + with pytest.raises(ValueError): + controller.model.spike_clusters = np.array([123]) + with pytest.raises(ValueError): + controller.model.sample_rate = 1.0 + with pytest.raises(TypeError): + controller.model.metadata['group'][0] = 'noise' + supervisor = controller.supervisor + assert 99 not in supervisor.clustering.cluster_ids + assert supervisor.clustering.new_cluster_id() == 100 + assert controller.get_full_count(0) == 4 + assert controller.get_mean_firing_rate(0) == 0.2 + for operation in ( + lambda: supervisor.clustering.merge([0, 1]), + lambda: supervisor.clustering.assign([0]), + lambda: supervisor.clustering.split([0]), + supervisor.clustering.undo, + supervisor.clustering.redo, + supervisor.clustering.reset, + lambda: supervisor.cluster_meta.set('group', [0], 'noise', add_to_stack=False), + lambda: supervisor.cluster_meta.from_dict({0: {'group': 'noise'}}), + lambda: controller.model.save_spike_clusters(supervisor.clustering.spike_clusters), + lambda: controller.model.save_metadata('group', {0: 'noise'}), + lambda: supervisor.split([0]), + ): + with pytest.raises(ValueError): + operation() + with pytest.raises(ValueError): + supervisor.clustering.spike_clusters[0] = 8 + assert not controller.bounded_session.events + with pytest.raises(ValueError): + supervisor.cluster_meta.add_field('group', 'noise') + with pytest.raises(TypeError): + supervisor.clustering.spikes_per_cluster[0] = np.array([0]) + with pytest.raises(ValueError): + supervisor.clustering.spikes_per_cluster[0][0] = 1 + assert 'CorrelogramView' not in controller.view_creator + assert 'FiringRateView' not in controller.view_creator + finally: + controller.close() + + +def test_loader_and_native_history_do_not_allocate_by_sparse_unit_id(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'source') + labels = np.load(params.parent / 'spike_clusters.npy').astype(np.int64) + labels[labels == 99] = 2_000_000_000 + np.save(params.parent / 'spike_clusters.npy', labels) + (params.parent / 'cluster_group.tsv').write_text( + 'cluster_id\tgroup\n0\tgood\n1\tmua\n2\tgood\n2000000000\tnoise\n' + ) + output = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + controller = TemplateController(dir_path=output, config_dir=tmp_path / 'config', plugins=[]) + try: + assert controller.model.sparse_clusters is None + first = controller.supervisor.merge([0, 1]) + assert first.added == [2_000_000_001] + controller.supervisor.merge([2, 2_000_000_001]) + controller.supervisor.undo() + assert 2_000_000_001 in controller.supervisor.clustering.cluster_ids + values = controller._window_correlograms([2, 2_000_000_001], 0.001, 0.004) + assert values.shape == (2, 2, 5) + controller.supervisor.save() + finally: + controller.close() + reopened = TemplateController(dir_path=output, config_dir=tmp_path / 'config', plugins=[]) + try: + assert 2_000_000_001 in reopened.model.cluster_ids + assert reopened.model.sparse_clusters is None + finally: + reopened.close() + + +def test_merge_label_history_save_reopen_replay_regenerate(tmp_path): + params, controller = make_controller(tmp_path) + path = controller.dir_path + try: + supervisor = controller.supervisor + original_file = (path / 'spike_clusters.npy').read_bytes() + up = supervisor.merge([0, 1]) + assert up.added == [100] + assert controller.get_full_count(100) == 7 + supervisor.label('group', 'noise', [100]) + supervisor.undo() + assert controller.bounded_session.metadata['group'][100] is None + assert supervisor.cluster_meta.get('group', 100) is None + supervisor.redo() + assert controller.bounded_session.metadata['group'][100] == 'noise' + supervisor.undo() + supervisor.undo() + np.testing.assert_array_equal( + supervisor.clustering.spike_clusters, controller.bounded_session.base_assignments + ) + supervisor.merge([1, 2]) + assert supervisor.clustering.new_cluster_id() == 102 + supervisor.label('quality', 'reviewed', [101]) + supervisor.save() + assert not supervisor.is_dirty() + assert (path / 'spike_clusters.npy').read_bytes() == original_file + expected = supervisor.clustering.spike_clusters.copy() + finally: + controller.close() + reopened = TemplateController( + dir_path=path, config_dir=tmp_path / 'config', plugins=[], enable_threading=False + ) + try: + np.testing.assert_array_equal(reopened.supervisor.clustering.spike_clusters, expected) + assert reopened.supervisor.clustering.new_cluster_id() == 102 + assert reopened.supervisor.cluster_meta.get('quality', 101) == 'reviewed' + assert reopened.supervisor._global_history.current_position == 0 + finally: + reopened.close() + output = apply_curation(params, path, tmp_path / 'curated', batch_size=2) + source = np.load(params.parent / 'spike_clusters.npy') + oracle = source.copy() + oracle[np.isin(source, [1, 2])] = 101 + np.testing.assert_array_equal(np.load(output / 'spike_clusters.npy'), oracle) + regenerated = prepare_large( + output / 'params.py', + tmp_path / 'regenerated', + window_seconds=20, + stratum_seconds=20, + bin_seconds=5, + ) + assert BoundedSession(regenerated).unit_ids == [0, 99, 101] + + +def test_source_threshold_replays_every_spike_and_full_metrics_pending(tmp_path): + params, controller = make_controller(tmp_path) + try: + supervisor = controller.supervisor + up = supervisor.split_source_amplitude(0, 4) + assert set(up.added) == {100, 101} + assert controller.get_full_count(100) is None + assert controller.get_mean_firing_rate(101) is None + supervisor.undo() + assert controller.get_full_count(0) == 4 + supervisor.redo() + supervisor.save() + output = apply_curation(params, controller.dir_path, tmp_path / 'curated', batch_size=2) + source = np.load(params.parent / 'spike_clusters.npy') + amplitudes = np.load(params.parent / 'amplitudes.npy') + oracle = source.copy() + oracle[(source == 0) & (amplitudes < 4)] = 100 + oracle[(source == 0) & ~(amplitudes < 4)] = 101 + np.testing.assert_array_equal(np.load(output / 'spike_clusters.npy'), oracle) + finally: + controller.close() + + +def test_failed_save_leaves_dirty_previous_checkpoint(tmp_path, monkeypatch): + _, controller = make_controller(tmp_path) + try: + supervisor = controller.supervisor + supervisor.merge([0, 1]) + supervisor.save() + supervisor.label('group', 'noise', [100]) + from phy.apps.large_recording import curation + + original = curation._write_json + + def fail(path, value): + if path.name == 'journal.json': + raise OSError('interrupted') + return original(path, value) + + monkeypatch.setattr(curation, '_write_json', fail) + with pytest.raises(OSError, match='interrupted'): + supervisor.save() + assert supervisor.is_dirty() + assert BoundedSession(controller.dir_path).metadata['group'][100] is None + finally: + controller.close() + + +def test_standard_controller_retains_existing_path_and_save_format(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'ordinary') + controller = TemplateController( + dir_path=params.parent, + config_dir=tmp_path / 'config', + plugins=[], + enable_threading=False, + sample_rate=30000, + n_channels_dat=3, + dtype='int16', + ) + gui = controller.create_gui(default_views=[], do_prompt_save=False) + try: + assert type(controller) is TemplateController + assert controller.gui_name == 'TemplateGUI' + assert controller.default_views[0] == 'WaveformView' + assert 'FiringRateView' in controller.view_creator + up = controller.supervisor.merge([0, 1]) + controller.supervisor.save() + assert up.added == [100] + assert not (params.parent / '.phy-curation').exists() + assert not (params.parent / '_phy_curation_manifest.json').exists() + assert 100 in np.load(params.parent / 'spike_clusters.npy') + finally: + gui.close() + controller.close() + + +def test_malformed_manifest_fails_before_model_loading(tmp_path): + params = make_sorting(tmp_path / 'ordinary') + (params.parent / '_phy_curation_manifest.json').write_text(json.dumps({'format_version': 99})) + with pytest.raises(ValueError, match='Unsupported'): + TemplateController(dir_path=params.parent, plugins=[]) + + +def test_cli_preparation_and_replay(tmp_path): + params = make_sorting(tmp_path / 'source') + runner = CliRunner() + output = tmp_path / 'bounded' + result = runner.invoke( + phycli, + [ + 'prepare-large', + str(params), + str(output), + '--window-seconds', + '20', + '--stratum-seconds', + '20', + '--bin-seconds', + '5', + '--batch-size', + '2', + ], + ) + assert result.exit_code == 0, result.output + session = BoundedSession(output) + session.record_merge([0, 1], 100, {'group': 'good'}) + session.save() + result = runner.invoke( + phycli, + [ + 'apply-curation', + str(params), + str(output), + str(tmp_path / 'curated'), + '--batch-size', + '2', + ], + ) + assert result.exit_code == 0, result.output + + +def test_stability_gui_and_exact_trends(qtbot, tmp_path): + _, controller = make_controller(tmp_path, sparse=True) + gui = controller.create_gui(default_views=['UnitStabilityView'], do_prompt_save=False) + try: + assert 'Bounded evidence' in gui.windowTitle() + view = gui.list_views('UnitStabilityView')[0] + data = view.get_data(0) + assert data.counts.sum() == 4 + np.testing.assert_allclose(data.trends['amplitude'], [1.5, np.nan, 6, 9], equal_nan=True) + view.on_select([99]) # Unit absent from evidence still has full-duration counts. + assert view.get_data(99).counts.sum() == 2 + controller.supervisor.merge([0, 1]) + controller.supervisor.block() + assert view.get_data(100).counts.sum() == 7 + finally: + gui.close() + controller.close() + + +def test_observer_mutation_rolls_back_native_and_semantic_state(tmp_path): + _, controller = make_controller(tmp_path) + supervisor = controller.supervisor + once = [] + + @connect(event='cluster', sender=supervisor.clustering) + def mutate(sender, up): + if not once: + once.append(True) + sender.assign([0]) + + try: + with pytest.raises(ValueError, match='Native assignments'): + supervisor.merge([0, 1]) + np.testing.assert_array_equal( + supervisor.clustering.spike_clusters, controller.bounded_session.base_assignments + ) + assert not controller.bounded_session.events + assert supervisor._global_history.current_position == 0 + assert not supervisor.is_dirty() + finally: + unconnect(mutate) + controller.close() + + +def test_evidence_navigation_never_reuses_old_feature_cache(tmp_path): + _, controller = make_controller(tmp_path, sparse=True) + try: + first = controller._get_features(0, channel_ids=[0, 1], load_all=True) + assert first.spike_ids.tolist() == [0, 3] + controller.inspect_evidence_window(1) + assert not controller._get_feature_view_spike_ids(0, load_all=True).size + assert not controller._get_features(0, channel_ids=[0, 1], load_all=True) + finally: + controller.close() + + +def test_late_clock_evidence_preserves_submillisecond_positions(tmp_path): + params = make_gui_sorting(tmp_path / 'source') + origin = 1000 * 3600 * 30000 + times = np.load(params.parent / 'spike_times.npy') + origin + times[1:4] = origin + np.array([30, 31, 60]) + np.save(params.parent / 'spike_times.npy', times) + params.write_text( + params.read_text().replace('num_samples = 600000', f'num_samples = {origin + 600000}') + + f'valid_intervals = [[{origin}, {origin + 600000}]]\n' + ) + output = prepare_large( + params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20, bin_seconds=3600 + ) + controller = TemplateController(dir_path=output, config_dir=tmp_path / 'config', plugins=[]) + try: + positions = controller._window_time(np.array([1, 2])) + assert positions[1] - positions[0] == pytest.approx(1 / 30000) + assert np.float32(positions[0]) != np.float32(positions[1]) + finally: + controller.close() + + +def test_empty_window_and_nonfinite_amplitude_measurements(qtbot, tmp_path): + _, controller = make_controller(tmp_path, sparse=True) + view = controller.create_evidence_amplitude_view() + try: + controller.inspect_evidence_window(1) + view.on_select([0]) + assert view.data_bounds == (0, 0, view.duration, 1) + assert all(not len(b.amplitudes) for b in view._displayed_bunchs) + finally: + view.canvas.close() + controller.close() + + +def test_proposition_reviews_checkpoint_and_failed_review_history(tmp_path): + from phy.cluster._propositions import MergePropositionController, decode_curation_mapping + + mapping = {'format_version': '2', 'unit_ids': [0, 1, 2, 99], 'merges': [{'unit_ids': [0, 1]}]} + _, controller = make_controller(tmp_path, propositions=mapping) + supervisor = controller.supervisor + producer = controller.dir_path / 'curation.json' + supervisor.merge_propositions = MergePropositionController(decode_curation_mapping(mapping)) + key = supervisor.merge_propositions.catalog.propositions[0].key + try: + with pytest.raises(RuntimeError, match='observer'), supervisor._transaction(): + supervisor.merge_propositions.accept(key, [0, 1], 100) + raise RuntimeError('observer') + assert supervisor.merge_propositions._history.current_position == 0 + assert not supervisor.merge_propositions.catalog.reviews + supervisor.merge_propositions.reject(key) + supervisor.save() + controller.close() + reopened = TemplateController( + dir_path=producer.parent, config_dir=tmp_path / 'config', plugins=[] + ) + try: + assert ( + reopened.supervisor.merge_propositions.catalog.status_for(key).value == 'rejected' + ) + assert not reopened.supervisor.is_dirty() + finally: + reopened.close() + finally: + controller.close() + + +def test_nonfinite_source_amplitudes_are_omitted_from_plots_and_kept_in_replay(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'source') + amplitudes = np.load(params.parent / 'amplitudes.npy') + amplitudes[3] = np.inf + amplitudes[6] = np.nan + np.save(params.parent / 'amplitudes.npy', amplitudes) + output = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + controller = TemplateController(dir_path=output, config_dir=tmp_path / 'config', plugins=[]) + view = controller.create_evidence_amplitude_view() + try: + view.on_select([0]) + assert np.all(np.isfinite(view.data_bounds)) + assert all(np.all(np.isfinite(b.amplitudes)) for b in view._displayed_bunchs) + controller.supervisor.split_source_amplitude(0, 4) + controller.supervisor.save() + curated = apply_curation(params, output, tmp_path / 'curated') + labels = np.load(curated / 'spike_clusters.npy') + assert labels[0] == 100 + assert labels[3] == labels[6] == 101 + finally: + view.canvas.close() + controller.close() diff --git a/phy/apps/large_recording/tests/test_assignment_oracle.py b/phy/apps/large_recording/tests/test_assignment_oracle.py new file mode 100644 index 000000000..c1b098b64 --- /dev/null +++ b/phy/apps/large_recording/tests/test_assignment_oracle.py @@ -0,0 +1,107 @@ +"""Literal all-source expectations for the independent acceptance oracle.""" + +from types import SimpleNamespace + +import numpy as np +import pytest + +from tools.curation_output_contract import _assignment_oracle + + +def _split( + parent, children, operator='<', threshold=4, nonfinite='keep', attribute='amplitudes.npy' +): + return { + 'operation': 'threshold_split', + 'parents': [parent], + 'children': children, + 'predicate': { + 'attribute': attribute, + 'operator': operator, + 'threshold': threshold, + 'nonfinite': nonfinite, + }, + } + + +def _check(original, values, expected, history, chunk_size=2, attribute='amplitudes.npy'): + source = SimpleNamespace( + arrays={ + 'spike_clusters.npy': np.asarray(original, dtype=np.int16), + attribute: np.asarray(values), + } + ) + curated = SimpleNamespace(arrays={'spike_clusters.npy': np.asarray(expected)}) + return _assignment_oracle(source, curated, history, chunk_size) + + +@pytest.mark.parametrize('attribute', ['amplitudes.npy', 'spike_templates.npy']) +@pytest.mark.parametrize('column', [False, True]) +@pytest.mark.parametrize( + ('operator', 'expected'), + [ + ('<', [40001, 40000, 40000, 9]), + ('<=', [40001, 40001, 40000, 9]), + ('>', [40000, 40000, 40001, 9]), + ('>=', [40000, 40001, 40001, 9]), + ], +) +def test_predicate_boundaries_and_child_order(attribute, column, operator, expected): + values = np.array([1, 4, 7, 2]) + if column: + values = values[:, None] + report = _check( + [0, 0, 0, 9], + values, + expected, + [_split(0, [40000, 40001], operator=operator, attribute=attribute)], + attribute=attribute, + ) + assert report == { + 'verified': True, + 'source_rows': 4, + 'membership_events': 1, + 'threshold_splits': 1, + } + + +@pytest.mark.parametrize('operator', ['<', '<=', '>', '>=']) +def test_nonfinite_keep_stays_in_remainder(operator): + _check( + [0, 0, 0, 9], + [np.nan, np.inf, -np.inf, np.nan], + [10, 10, 10, 9], + [_split(0, [10, 11], operator=operator)], + ) + + +def test_nonfinite_reject_only_applies_to_current_parent(): + history = [_split(0, [10, 11], nonfinite='reject')] + _check([0, 0, 9], [1, 7, np.nan], [11, 10, 9], history) + with pytest.raises(ValueError, match='rejects nonfinite'): + _check([0, 0, 9], [np.nan, 7, 1], [10, 10, 9], history) + + +@pytest.mark.parametrize('chunk_size', [1, 2, 7, 100]) +def test_cumulative_merge_split_remerge_and_split(chunk_size): + history = [ + {'operation': 'merge', 'parents': [0, 1], 'children': [10]}, + _split(10, [11, 12]), + {'operation': 'merge', 'parents': [12, 2], 'children': [13]}, + {'operation': 'label', 'field': 'group', 'ids': [13], 'value': 'good'}, + _split(13, [14, 15], operator='>=', threshold=2), + ] + original, values = [0, 1, 0, 1, 2, 2, 9], [1, 3, 4, 7, 2, 5, 0] + expected = [14, 15, 11, 11, 15, 15, 9] + _check(original, values, expected, history, chunk_size) + # Wrong rows remain detectable across chunk boundaries and after subsequent merges. + for row in (0, 3, 6): + wrong = expected.copy() + wrong[row] = 15 if expected[row] != 15 else 14 + with pytest.raises(ValueError, match=f'at spike {row}$'): + _check(original, values, wrong, history, chunk_size) + + +def test_reject_nonscalar_attribute(): + with pytest.raises(ValueError, match='scalar per source spike'): + _check([0, 0], [[1, 2], [3, 4]], [11, 11], [_split(0, [10, 11])]) diff --git a/phy/apps/large_recording/tests/test_cli.py b/phy/apps/large_recording/tests/test_cli.py new file mode 100644 index 000000000..573e95fc3 --- /dev/null +++ b/phy/apps/large_recording/tests/test_cli.py @@ -0,0 +1,43 @@ +"""Batch CLI coverage and window-policy regressions.""" + +import json + +from click.testing import CliRunner + +from phy.apps import phycli +from phy.apps.large_recording.tests.test_source import make_sorting + + +def test_prepare_cli_reports_waveform_coverage_and_window_policy(tmp_path): + params = make_sorting(tmp_path / 'source') + output = tmp_path / 'bounded' + result = CliRunner().invoke( + phycli, + [ + 'prepare-large', + str(params), + str(output), + '--window-policy', + 'generated', + '--window-seconds', + '20', + '--stratum-seconds', + '20', + ], + ) + assert result.exit_code == 0, result.output + assert '5 retained / 5 source; 0 omitted' in result.output + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + assert manifest['selection']['window_policy'] == 'generated' + + +def test_prepare_cli_explains_missing_producer_windows(tmp_path): + params = make_sorting(tmp_path / 'source') + output = tmp_path / 'bounded' + result = CliRunner().invoke( + phycli, + ['prepare-large', str(params), str(output), '--window-policy', 'producer'], + ) + assert result.exit_code == 1 + assert 'producer' in result.output.lower() + assert not output.exists() diff --git a/phy/apps/large_recording/tests/test_controller_equivalence.py b/phy/apps/large_recording/tests/test_controller_equivalence.py new file mode 100644 index 000000000..514a62e7e --- /dev/null +++ b/phy/apps/large_recording/tests/test_controller_equivalence.py @@ -0,0 +1,410 @@ +"""The same user commands on a full sorting and genuine bounded evidence.""" + +import gc +import shutil +from contextlib import contextmanager + +import numpy as np +import pytest +from phylib.io.model import load_model +from phylib.utils.event import _EVENT + +import tools.compare_curation_modes as comparison +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import hash_file +from phy.apps.large_recording.tests.test_adapter import make_gui_sorting +from phy.apps.template.gui import TemplateController +from phy.gui.qt import QApplication, QCoreApplication, QEvent +from phy.plot.tests import mouse_drag +from phy.plot.transform import NDC, Range + + +def _make_source(path): + params = make_gui_sorting(path) + seconds = np.r_[np.arange(50), np.arange(60, 120)] + units = np.array([0, 1, 2, 50, 51, 99], dtype=np.int32) + clusters = np.tile(units, len(seconds)) + times = np.repeat(seconds * 30000, len(units)) + np.tile( + np.arange(len(units)) * 300, len(seconds) + ) + amplitudes = np.repeat( + np.where(seconds % 2 == 0, 1, np.where(seconds % 4 == 1, 4, 7)), len(units) + ).astype(np.float32) + arrays = { + 'spike_times.npy': times.astype(np.int64), + 'spike_clusters.npy': clusters, + 'spike_templates.npy': (np.arange(len(times)) % 3).astype(np.int32), + 'amplitudes.npy': amplitudes, + 'spike_positions.npy': np.column_stack((np.zeros(len(times)), times / 30000)), + 'spike_depths.npy': (times / 30000).astype(np.float64), + 'pc_features.npy': np.arange(len(times) * 4, dtype=np.float32).reshape(-1, 2, 2), + 'whitening_mat.npy': np.eye(3, dtype=np.float32), + 'whitening_mat_inv.npy': np.eye(3, dtype=np.float32), + } + waveform_ids = np.arange(0, len(times), 7, dtype=np.int64) + arrays.update( + { + '_phy_spikes_subset.spikes.npy': waveform_ids, + '_phy_spikes_subset.channels.npy': np.tile([0, 1], (len(waveform_ids), 1)), + '_phy_spikes_subset.waveforms.npy': np.arange( + len(waveform_ids) * 8 * 2, dtype=np.float32 + ).reshape(-1, 8, 2), + } + ) + for name, values in arrays.items(): + np.save(path / name, values) + params.write_text( + params.read_text().replace('num_samples = 600000', 'num_samples = 3600000') + + 'valid_intervals = [[0, 1500000], [1800000, 3600000]]\n' + ) + (path / 'cluster_group.tsv').write_text( + 'cluster_id\tgroup\n0\tgood\n1\tmua\n2\tgood\n50\tgood\n51\tmua\n99\tnoise\n' + ) + (path / 'cluster_snr.tsv').write_text( + 'cluster_id\tsnr\n0\t8\n1\t10\n2\t12\n50\t16\n51\t18\n99\t14\n' + ) + (path / 'cluster_si_unit_ids.tsv').write_text( + 'cluster_id\tsi_unit_id\n0\t0\n1\t1\n2\t2\n50\t50\n51\t51\n99\t99\n' + ) + (path / 'cluster_notes.tsv').write_text( + 'cluster_id\tnotes\n50\tproducer parent A\n51\tproducer parent B\n99\toriginal control\n' + ) + return params + + +def _hashes(path): + return {str(p.relative_to(path)): hash_file(p) for p in path.rglob('*') if p.is_file()} + + +@contextmanager +def _open(path, config, bounded): + baseline = {id(item) for item in _EVENT._callbacks} + model = None if bounded else load_model(path / 'params.py') + controller = TemplateController( + dir_path=path, + model=model, + config_dir=config, + plugins=[], + enable_threading=False, + ) + controller.n_spikes_amplitudes = 5 + controller.n_spikes_amplitudes_total = 5 + controller.n_spikes_amplitudes_background = 8 + view_name = 'EvidenceAmplitudeView' if bounded else 'AmplitudeView' + gui = None + try: + # Explicit view_count prevents saved/default layout from adding other + # renderers: the controller, Supervisor and real amplitude provider stay native. + gui = controller.create_gui( + view_count={view_name: 1}, default_views=[view_name], do_prompt_save=False + ) + gui.resize(900, 700) + gui.show() + QApplication.processEvents() + view = gui.get_view(view_name) + if not bounded: + view.amplitudes_type = 'template' + yield controller, gui, view + finally: + if gui is not None: + gui.close() + gui.deleteLater() + QCoreApplication.sendPostedEvents(None, QEvent.DeferredDelete) + QApplication.processEvents() + controller.close() + # Global callbacks may own controller/history cycles after closing a GUI. + # Retain exactly the callbacks that preceded this isolated controller. + _EVENT._callbacks = [item for item in _EVENT._callbacks if id(item) in baseline] + del controller, model + gc.collect() + + +def _snapshot(supervisor): + return ( + supervisor.clustering.spike_clusters.copy(), + { + field: { + int(unit): supervisor.cluster_meta.get(field, int(unit)) + for unit in supervisor.clustering.cluster_ids + } + for field in ('group', 'notes') + }, + ) + + +def _assert_snapshot(supervisor, expected): + actual = _snapshot(supervisor) + np.testing.assert_array_equal(actual[0], expected[0]) + assert actual[1] == expected[1] + + +def _split(qtbot, controller, gui, view, parent, threshold): + supervisor = controller.supervisor + supervisor.select([parent]) + supervisor.block() + qtbot.wait(20) + view.plot() + ids = supervisor.clustering.spikes_per_cluster[parent].copy() + amplitudes = np.asarray(controller.model.amplitudes[ids]).reshape(-1) + assert np.all(np.isfinite(amplitudes)) + drawn = next(b for b in view.get_clusters_data() if b.cluster_id == parent) + assert len(drawn.spike_ids) <= 5 < len(ids), 'The display must not be the split input.' + selected = amplitudes < threshold + assert selected.any() and (~selected).any(), 'Both children must have retained evidence.' + width, height = view.canvas.get_size() + point = (width // 3, height // 2) + target = Range(view.data_bounds, NDC).apply([[0, threshold]])[0][1] + normalized_y = 1 - 2 * point[1] / height + pan = view.canvas.panzoom.pan + view.canvas.panzoom.pan = ( + pan[0], + normalized_y / view.canvas.panzoom._zoom_aspect()[1] - target, + ) + # Move the viewport by representable steps if inverse-transform rounding + # lands just above the requested boundary; no stored value is rewritten. + draw_threshold = threshold + for _ in range(16): + if view._threshold_from_window_pos(point) <= threshold: + break + draw_threshold = np.nextafter(draw_threshold, -np.inf) + below = Range(view.data_bounds, NDC).apply([[0, draw_threshold]])[0][1] + view.canvas.panzoom.pan = ( + pan[0], + normalized_y / view.canvas.panzoom._zoom_aspect()[1] - below, + ) + mouse_drag( + qtbot, + view.canvas, + (point[0] - 10, point[1]), + point, + button='right', + modifiers=('Alt',), + ) + assert view.split_threshold == pytest.approx(threshold, abs=1e-12, rel=0) + np.testing.assert_array_equal(selected, amplitudes < view.split_threshold) + action = supervisor.actions.get('split') + assert action.isEnabled() + action.trigger() + supervisor.block() + after = supervisor.clustering.spike_clusters[ids] + low, rest = np.unique(after[selected]), np.unique(after[~selected]) + assert len(low) == len(rest) == 1 and low[0] != rest[0] + return int(low[0]), int(rest[0]) + + +def _run_ledger(qtbot, path, config, bounded, actions, aliases): + aliases = dict(aliases) + history, future = [], [] + opened = _open(path, config, bounded) + controller, gui, view = opened.__enter__() + try: + for action in actions: + op, *args = action + supervisor = controller.supervisor + if op == 'reopen': + opened.__exit__(None, None, None) + opened = _open(path, config, bounded) + controller, gui, view = opened.__enter__() + history.clear() + future.clear() + continue + if op == 'save': + supervisor.save() + continue + if op in ('undo', 'redo'): + origin, destination = (history, future) if op == 'undo' else (future, history) + current = _snapshot(supervisor) + expected = origin.pop() if origin else current + if expected is not current: + destination.append(current) + getattr(supervisor, op)() + supervisor.block() + _assert_snapshot(supervisor, expected) + continue + history.append(_snapshot(supervisor)) + future.clear() + if op == 'merge': + result, *parents = args + up = supervisor.merge([aliases[parent] for parent in parents]) + supervisor.block() + assert len(up.added) == 1 + aliases[result] = int(up.added[0]) + elif op == 'split': + parent, low, rest, threshold = args + aliases[low], aliases[rest] = _split( + qtbot, controller, gui, view, aliases[parent], threshold + ) + elif op == 'label': + parent, field, value = args + supervisor.label(field, value, cluster_ids=[aliases[parent]]) + supervisor.block() + assert supervisor.cluster_meta.get(field, aliases[parent]) == value + else: + raise AssertionError(op) + supervisor = controller.supervisor + state = _snapshot(supervisor) + supervisor.save() + finally: + opened.__exit__(None, None, None) + return aliases, state + + +ROUND_ONE = [ + ('merge', 'merged', 'a', 'b'), + ('undo',), + ('redo',), + ('split', 'merged', 'low', 'rest', 4.0), + ('undo',), + ('redo',), + ('merge', 'final', 'low', 'c'), + ('label', 'final', 'group', 'good'), + ('label', 'rest', 'group', 'mua'), + ('label', 'final', 'notes', 'reviewed merged child'), + ('label', 'rest', 'notes', 'reviewed remainder'), + ('label', 'rest', 'group', 'noise'), + ('undo',), + ('redo',), + ('undo',), + ('label', 'rest', 'notes', 'branch kept'), + ('redo',), # A new edit discarded the noise redo branch. + ('label', 'control', 'group', None), + ('label', 'control', 'notes', 'cleared group on unchanged membership'), + ('save',), + ('reopen',), + ('save',), +] + +ROUND_TWO = [ + ('merge', 'reunited', 'final', 'rest'), + ('split', 'reunited', 'second_low', 'second_rest', 4.0), + ('label', 'second_low', 'group', 'good'), + ('label', 'second_rest', 'group', 'mua'), + ('label', 'second_low', 'notes', 'reviewed round two low'), + # Seed an explicit note before editing it: ordinary inherited parent notes + # and bounded new-membership clearing intentionally have different baselines. + ('label', 'second_rest', 'notes', 'round two draft'), + ('label', 'second_rest', 'notes', 'reviewed round two remainder'), + ('undo',), + ('redo',), + ('save',), + ('reopen',), + ('save',), +] + + +EXPECTED_ONE = { + 'final': {'group': 'good', 'notes': 'reviewed merged child'}, + 'rest': {'group': 'mua', 'notes': 'branch kept'}, + 'control': {'group': None, 'notes': 'cleared group on unchanged membership'}, +} +EXPECTED_TWO = { + 'second_low': {'group': 'good', 'notes': 'reviewed round two low'}, + 'second_rest': {'group': 'mua', 'notes': 'reviewed round two remainder'}, + 'control': {'group': None, 'notes': 'cleared group on unchanged membership'}, +} +NO_EDITS = [('save',), ('reopen',), ('save',)] +UNREVIEWED = [('merge', 'unreviewed', 'review_a', 'review_b'), *NO_EDITS] + + +@pytest.mark.parametrize('window,budget,seed', [(2, 32, 0), (3, 48, 11), (4, 48, 23)]) +def test_same_controller_actions_match_full_replay(qtbot, tmp_path, capsys, window, budget, seed): + params = _make_source(tmp_path / 'source') + source_hashes = _hashes(params.parent) + ordinary = tmp_path / 'ordinary' + shutil.copytree(params.parent, ordinary) + ordinary_aliases = bounded_aliases = { + 'a': 0, + 'b': 1, + 'c': 2, + 'review_a': 50, + 'review_b': 51, + 'control': 99, + } + source_params = params + for round_index, (actions, expected_manual) in enumerate( + zip( + (ROUND_ONE, ROUND_TWO, NO_EDITS, UNREVIEWED), + (EXPECTED_ONE, EXPECTED_TWO, EXPECTED_TWO, EXPECTED_TWO), + strict=True, + ) + ): + round_source_hashes = _hashes(source_params.parent) + package = prepare_large( + source_params, + tmp_path / f'bounded-{round_index}', + seed=seed + round_index, + max_spikes=budget, + window_seconds=window, + stratum_seconds=30, + bin_seconds=10, + batch_size=17, + ) + source_labels = np.load(source_params.parent / 'spike_clusters.npy').reshape(-1) + kept = np.load(package / '_phy_spike_global_ids.npy') + assert 0 < len(kept) < len(source_labels) + for unit in np.unique(source_labels): + assert np.any(source_labels[kept] == unit) + assert np.count_nonzero(source_labels[kept] == unit) < np.count_nonzero( + source_labels == unit + ), 'Every active unit must contain unseen source spikes.' + ordinary_aliases, ordinary_state = _run_ledger( + qtbot, + ordinary, + tmp_path / 'config-ordinary', + False, + actions, + ordinary_aliases, + ) + bounded_aliases, bounded_state = _run_ledger( + qtbot, + package, + tmp_path / 'config-bounded', + True, + actions, + bounded_aliases, + ) + for aliases, state in ( + (ordinary_aliases, ordinary_state), + (bounded_aliases, bounded_state), + ): + for alias, fields in expected_manual.items(): + for field, value in fields.items(): + assert state[1][field][aliases[alias]] == value + expected_labels = { + bounded_aliases[alias]: fields for alias, fields in expected_manual.items() + } + output = apply_curation( + source_params, package, tmp_path / f'curated-{round_index}', batch_size=17 + ) + report = comparison.compare_outputs( + ordinary, + output, + fields=('group', 'notes'), + chunk_size=13, + source=params.parent, + expected_labels=expected_labels, + bounded_dir=package, + ) + assert report['ok'], report.get('error', report) + assert report['file_contract']['ok'], report + for difference in report['permitted_membership_policy_differences']: + if difference['field'] in ('group', 'notes'): + assert round_index == 3, difference + assert difference['right_id'] == bounded_aliases['unreviewed'], difference + else: + assert difference['field'] in ('snr', 'si_unit_id'), difference + assert difference['right'] == '', difference + assert report['rows'] == len(source_labels) + if round_index == 2: + no_op_files = [p.name for p in source_params.parent.glob('*.npy')] + [ + 'cluster_group.tsv', + 'cluster_notes.tsv', + ] + for name in no_op_files: + assert hash_file(source_params.parent / name) == hash_file(output / name), name + assert _hashes(source_params.parent) == round_source_hashes + assert _hashes(params.parent) == source_hashes + source_params = output / 'params.py' + captured = capsys.readouterr() + assert 'Traceback (most recent call last)' not in captured.err, captured.err diff --git a/phy/apps/large_recording/tests/test_curation.py b/phy/apps/large_recording/tests/test_curation.py new file mode 100644 index 000000000..ed667ef8d --- /dev/null +++ b/phy/apps/large_recording/tests/test_curation.py @@ -0,0 +1,458 @@ +"""Exact bounded journal projection and transactional persistence regressions.""" + +import json +import shutil + +import numpy as np +import pytest + +from phy.apps.large_recording.curation import BASE_ASSIGNMENTS, BoundedSession +from phy.apps.large_recording.source import Source, hash_file +from phy.apps.tests.large_recording_fixtures import make_long_recording_fixture + + +def make_session(tmp_path): + fixture = make_long_recording_fixture(tmp_path) + # The highest complete-source unit is intentionally absent from evidence. + clusters = np.load(fixture.full / 'spike_clusters.npy') + clusters[-1] = 99 + np.save(fixture.full / 'spike_clusters.npy', clusters) + (fixture.full / 'cluster_group.tsv').write_text('cluster_id\tgroup\n0\tgood\n1\tnoise\n') + source = Source(fixture.full / 'params.py', batch_size=5) + rows = np.array([0, 2, 5, 8, 13, 21, 25], dtype=np.int64) + for name in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy', 'amplitudes.npy'): + np.save(fixture.bounded / name, source.arrays[name][rows]) + np.save(fixture.bounded / BASE_ASSIGNMENTS, clusters[rows]) + np.save(fixture.bounded / '_phy_spike_global_ids.npy', rows) + np.save(fixture.bounded / 'windows.npy', np.array([[0, source.num_samples]], dtype=np.int64)) + manifest = { + 'format': 'phy_bounded_curation', + 'format_version': 1, + 'source': { + 'identity': source.identity, + 'revision': source.revision, + 'path': str(source.params_path), + }, + 'sample_rate': source.sample_rate, + 'num_samples': source.num_samples, + 'valid_intervals': source.valid_intervals.tolist(), + 'num_spikes': source.n_spikes, + 'bounded_num_spikes': len(rows), + 'unit_ids': source.unit_ids.tolist(), + 'counts': source.counts.tolist(), + 'next_cluster_id': 100, + 'global_spike_ids_file': '_phy_spike_global_ids.npy', + 'windows_file': 'windows.npy', + 'base_metadata': source.metadata, + 'capabilities': { + 'merge': True, + 'label': True, + 'split': False, + 'predicate_split_attributes': ['amplitudes.npy'], + }, + 'bounded_files': { + name: hash_file(fixture.bounded / name) + for name in ( + BASE_ASSIGNMENTS, + '_phy_spike_global_ids.npy', + 'windows.npy', + 'spike_times.npy', + ) + }, + } + (fixture.bounded / '_phy_curation_manifest.json').write_text(json.dumps(manifest)) + return source, BoundedSession(fixture.bounded) + + +@pytest.mark.parametrize('command', ['record_threshold_split', 'record_split']) +def test_split_numpy_metadata_keys_save_and_reopen(tmp_path, command): + source, session = make_session(tmp_path) + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 0.5, + 'nonfinite': 'keep', + } + children = [np.int64(100), np.int64(101)] + metadata = {'group': {children[0]: 'good', children[1]: 'noise'}} + try: + if command == 'record_threshold_split': + session.record_threshold_split(0, children, predicate, metadata) + else: + session.record_split(0, *children, predicate, metadata) + session.save() + with_metadata = session.events[-1]['metadata']['group'] + assert all(type(key) is int for key in with_metadata) + reopened = BoundedSession(session.path) + try: + np.testing.assert_array_equal(reopened.assignments, session.assignments) + assert reopened.metadata['group'][100] == 'good' + assert reopened.metadata['group'][101] == 'noise' + finally: + reopened.close() + finally: + session.close() + source.close() + + +def test_merge_label_snapshot_branch_reopen(tmp_path): + source, session = make_session(tmp_path) + baseline = session.snapshot() + session.record_merge([0, 1], 100, {'group': 'good'}) + first = session.snapshot() + session.record_label('group', [100], 'mua') + second = session.snapshot() + session.restore(first) + assert session.metadata['group'][100] == 'good' + session.restore(second) + assert session.metadata['group'][100] == 'mua' + session.restore(baseline) + session.record_merge([1, 2], 100, {'group': 'noise'}) + session.record_label('custom', [99, 100], 'reviewed') + session.save(extra_documents={'supplemental.json': {'accepted': [100]}}) + reopened = BoundedSession(session.path) + assert reopened.next_cluster_id == 101 + assert reopened.events == session.events + assert reopened.metadata == session.metadata + assert np.array_equal(reopened.assignments, session.assignments) + assert reopened.extra_documents == {'supplemental.json': {'accepted': [100]}} + assert source.unit_ids.tolist() == [0, 1, 2, 99] + assert 99 not in session.base_assignments + + +@pytest.mark.parametrize('parents,target', [([0, 1], 99), ([0, 0], 100), ([0, 3], 100)]) +def test_merge_collision_and_invalid_parents_do_not_mutate(tmp_path, parents, target): + _, session = make_session(tmp_path) + before = session.snapshot() + with pytest.raises(ValueError): + session.record_merge(parents, target) + assert session.events == before['events'] + assert np.array_equal(session.assignments, before['assignments']) + + +def test_save_requires_exact_semantic_projection(tmp_path): + _, session = make_session(tmp_path) + session.record_merge([0, 1], 100, {'group': 'good'}) + with pytest.raises(ValueError, match='projection'): + session.save(assignments=session.base_assignments) + with pytest.raises(ValueError, match='metadata'): + session.save(metadata={}) + assert not (session.path / '.phy-curation' / 'current.json').exists() + + +@pytest.mark.parametrize('fail_at', ['journal.json', 'checkpoint.json', 'pointer']) +def test_interrupted_save_recovers_previous_generation(tmp_path, monkeypatch, fail_at): + from phy.apps.large_recording import curation + + _, session = make_session(tmp_path) + session.record_merge([0, 1], 100, {'group': 'good'}) + session.save() + previous = session.snapshot() + session.record_label('group', [100], 'mua') + original = curation._write_json + + def fail(path, value): + if path.name == fail_at or (fail_at == 'pointer' and path.name.startswith('.current-')): + raise OSError('injected interrupted write') + return original(path, value) + + monkeypatch.setattr(curation, '_write_json', fail) + with pytest.raises(OSError, match='interrupted'): + session.save() + reopened = BoundedSession(session.path) + assert reopened.events == previous['events'] + assert reopened.metadata == previous['metadata'] + assert session.metadata['group'][100] == 'mua' + + +def test_checkpoint_corruption_is_rejected(tmp_path): + _, session = make_session(tmp_path) + session.save() + (session.checkpoint_path / 'journal.json').write_text('[]\nchanged') + with pytest.raises(ValueError, match='corruption'): + BoundedSession(session.path) + + +def test_bound_base_corruption_is_rejected(tmp_path): + _, session = make_session(tmp_path) + session.close() + shutil.copyfile(session.path / 'spike_times.npy', session.path / BASE_ASSIGNMENTS) + with pytest.raises(ValueError, match='integrity'): + BoundedSession(session.path) + + +def test_threshold_projection_is_serialized_and_nonfinite_policy_explicit(tmp_path): + _, session = make_session(tmp_path) + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 15.0, + 'nonfinite': 'keep', + } + session.record_split(2, 100, 101, predicate, {'group': {100: 'noise', 101: 'good'}}) + expected = session.base_assignments.astype(np.int64).copy() + mask = expected == 2 + selected = np.load(session.path / 'amplitudes.npy') < 15 + expected[mask & selected] = 100 + expected[mask & ~selected] = 101 + assert np.array_equal(session.assignments, expected) + session.save() + assert BoundedSession(session.path).events == session.events + with pytest.raises(ValueError, match='Unsupported'): + session.record_split(0, 102, 103, dict(predicate, attribute='waveform_peak')) + + +@pytest.mark.parametrize('boundary', ['generation', 'pointer']) +def test_atomic_publication_failure_preserves_previous_commit(tmp_path, monkeypatch, boundary): + from phy.apps.large_recording import curation + + _, session = make_session(tmp_path) + session.save() + session.record_merge([0, 1], 100, {'group': 'good'}) + original = curation.os.replace + + def fail(source, destination): + if (boundary == 'generation' and destination.parent.name == 'generations') or ( + boundary == 'pointer' and destination.name == 'current.json' + ): + raise OSError('injected atomic publication failure') + return original(source, destination) + + monkeypatch.setattr(curation.os, 'replace', fail) + with pytest.raises(OSError, match='publication'): + session.save() + assert BoundedSession(session.path).events == [] + assert len(session.events) == 1 + + +def test_nonfinite_predicate_keep_and_reject_are_explicit(): + from phy.apps.large_recording.curation import evaluate_predicate + + values = np.array([1.0, np.nan, np.inf, -np.inf, 3.0]) + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 2.0, + 'nonfinite': 'keep', + } + assert evaluate_predicate(values, predicate).tolist() == [True, False, False, False, False] + with pytest.raises(ValueError, match='Nonfinite'): + evaluate_predicate(values, dict(predicate, nonfinite='reject')) + + +def test_semantically_corrupt_checkpoint_rejected_even_with_valid_file_hashes(tmp_path): + from phy.apps.large_recording.curation import _digest + + _, session = make_session(tmp_path) + session.record_merge([0, 1], 100, {'group': 'good'}) + session.save() + generation = session.checkpoint_path + journal = json.loads((generation / 'journal.json').read_text()) + journal[0]['target'] = 99 + (generation / 'journal.json').write_text(json.dumps(journal)) + descriptor = json.loads((generation / 'checkpoint.json').read_text()) + descriptor['files']['journal.json'] = hash_file(generation / 'journal.json') + (generation / 'checkpoint.json').write_text(json.dumps(descriptor)) + pointer = {'generation': generation.name, 'sha256': _digest(descriptor)} + (session.path / '.phy-curation' / 'current.json').write_text(json.dumps(pointer)) + with pytest.raises(ValueError, match='colliding'): + BoundedSession(session.path) + + +def test_compact_history_preserves_allocator_and_reconstructs_assignments(tmp_path): + _, session = make_session(tmp_path) + before = session.journal_state() + session.record_merge([0, 1], 100, {'group': 'good'}) + after = session.journal_state() + merged = session.assignments.copy() + assert set(after) == {'events', 'next_cluster_id'} + assert after['events'][0] is session.events[0] + session.restore_journal(before) + assert session.events == [] + assert session.next_cluster_id == 101 + assert np.array_equal(session.assignments, session.base_assignments) + session.restore_journal(after) + assert np.array_equal(session.assignments, merged) + session.restore_journal(before) + session.record_merge([1, 2], 101, {'group': 'noise'}) + assert session.next_cluster_id == 102 + + +def test_session_close_releases_own_file_mappings(tmp_path): + _, session = make_session(tmp_path) + mappings = [ + array._mmap + for array in ( + session.global_ids, + session.base_assignments, + *session.attributes.values(), + ) + ] + session.close() + assert all(mapping.closed for mapping in mappings) + session.close() + + +def test_stale_session_save_rejected_without_losing_first_commit(tmp_path): + _, first = make_session(tmp_path) + second = BoundedSession(first.path) + first.record_merge([0, 1], 100, {'group': 'good'}) + second.record_merge([1, 2], 100, {'group': 'noise'}) + first.save(extra_documents={'supplemental.json': {'accepted': [100]}}) + with pytest.raises(ValueError, match='reopen'): + second.save(extra_documents={'supplemental.json': {'rejected': [100]}}) + reopened = BoundedSession(first.path) + assert reopened.events == first.events + assert reopened.extra_documents == first.extra_documents + assert second.events != reopened.events + # The winning session can commit again without reopening. + first.record_label('group', [100], 'mua') + first.save() + assert BoundedSession(first.path).metadata['group'][100] == 'mua' + + +def test_failed_save_releases_commit_lock(tmp_path, monkeypatch): + from phy.apps.large_recording import curation + + _, session = make_session(tmp_path) + session.save() + other = BoundedSession(session.path) + original = curation._write_json + + def fail(path, value): + if path.name == 'journal.json': + raise OSError('injected locked save failure') + return original(path, value) + + with monkeypatch.context() as patch: + patch.setattr(curation, '_write_json', fail) + with pytest.raises(OSError, match='locked'): + session.save() + other.record_label('group', [0], 'mua') + other.save() + assert BoundedSession(session.path).metadata['group'][0] == 'mua' + + +def _concurrent_save_worker(path, barrier, results, label): + session = BoundedSession(path) + try: + session.record_label('group', [0], label) + barrier.wait(timeout=20) + try: + session.save() + except ValueError as error: + results.put(('conflict', str(error))) + else: + results.put(('saved', label)) + finally: + session.close() + + +def test_concurrent_process_saves_compare_generation_under_lock(tmp_path): + import multiprocessing + + _, session = make_session(tmp_path) + session.save() + context = multiprocessing.get_context('spawn') + barrier = context.Barrier(2) + results = context.Queue() + processes = [ + context.Process( + target=_concurrent_save_worker, + args=(str(session.path), barrier, results, label), + ) + for label in ('good', 'noise') + ] + for process in processes: + process.start() + try: + outcomes = [results.get(timeout=30) for _ in processes] + for process in processes: + process.join(timeout=10) + assert process.exitcode == 0 + assert sorted(status for status, _ in outcomes) == ['conflict', 'saved'] + winner = next(value for status, value in outcomes if status == 'saved') + assert 'reopen' in next(value for status, value in outcomes if status == 'conflict') + assert BoundedSession(session.path).metadata['group'][0] == winner + finally: + for process in processes: + if process.is_alive(): + process.terminate() + process.join(timeout=5) + results.close() + + +def _replace_capabilities(session, capabilities): + session.close() + path = session.path / '_phy_curation_manifest.json' + manifest = json.loads(path.read_text()) + if capabilities is None: + manifest.pop('capabilities', None) + else: + manifest['capabilities'] = capabilities + path.write_text(json.dumps(manifest)) + return BoundedSession(session.path) + + +@pytest.mark.parametrize( + 'capabilities', [None, {}, {'merge': False, 'label': False, 'split': False}] +) +def test_readonly_capabilities_reject_public_mutations_without_state_changes( + tmp_path, capabilities +): + _, session = make_session(tmp_path) + session = _replace_capabilities(session, capabilities) + before = session.snapshot() + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 15.0, + 'nonfinite': 'keep', + } + commands = [ + lambda: session.preflight_merge([0, 1], 100), + lambda: session.record_merge([0, 1], 100), + lambda: session.preflight_label('group', [0], 'good'), + lambda: session.record_label('group', [0], 'good'), + lambda: session.preflight_split(2, 100, 101, predicate), + lambda: session.record_split(2, 100, 101, predicate), + lambda: session.record_threshold_split(2, [101, 100], predicate), + ] + for command in commands: + with pytest.raises(ValueError, match='capability'): + command() + assert session.events == before['events'] + assert session.metadata == before['metadata'] + assert session.next_cluster_id == before['next_cluster_id'] + assert np.array_equal(session.assignments, before['assignments']) + + +@pytest.mark.parametrize( + 'capabilities', + [ + {'merge': 'true'}, + {'label': 'false'}, + {'stored_waveforms': 1}, + {'raw_traces': None}, + {'split': True}, + {'predicate_split_attributes': 'amplitudes.npy'}, + {'predicate_split_attributes': ['amplitudes.npy', 'amplitudes.npy']}, + {'predicate_split_attributes': ['spike_templates.npy']}, + {'predicate_split_attributes': ['waveform_peak']}, + {'predicate_split_attributes': [True]}, + {'full_attributes': 'amplitude'}, + ], +) +def test_malformed_or_unsupported_capabilities_fail_validation(tmp_path, capabilities): + _, session = make_session(tmp_path) + with pytest.raises(ValueError, match='capability|capabilities|unsupported|Unsupported'): + _replace_capabilities(session, capabilities) + + +def test_predicate_capability_requires_an_available_scalar_source_attribute(tmp_path): + _, session = make_session(tmp_path) + session.close() + values = np.load(session.path / 'amplitudes.npy') + np.save(session.path / 'amplitudes.npy', np.column_stack([values, values])) + with pytest.raises(ValueError, match='source scalar'): + BoundedSession(session.path) diff --git a/phy/apps/large_recording/tests/test_curation_comparison.py b/phy/apps/large_recording/tests/test_curation_comparison.py new file mode 100644 index 000000000..ae6757634 --- /dev/null +++ b/phy/apps/large_recording/tests/test_curation_comparison.py @@ -0,0 +1,255 @@ +"""Independent full ordinary Supervisor versus bounded journal/replay acceptance.""" + +import numpy as np +import pytest + +import tools.compare_curation_modes as comparison +from phy.apps.large_recording.curation import BoundedSession +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.tests.test_source import make_sorting + + +def ledger(): + return { + 'aliases': {'a': 0, 'b': 1, 'c': 99}, + 'fields': ['group', 'notes'], + 'actions': [ + {'op': 'merge', 'parents': ['a', 'b'], 'as': 'ab'}, + {'op': 'undo'}, + {'op': 'redo'}, + { + 'op': 'split', + 'parent': 'ab', + 'threshold': 4.0, + 'selected': 'low', + 'remainder': 'rest', + }, + {'op': 'label', 'ids': ['low'], 'field': 'group', 'value': 'good'}, + {'op': 'label', 'ids': ['rest'], 'field': 'group', 'value': 'noise'}, + {'op': 'label', 'ids': ['low'], 'field': 'notes', 'value': 'finite below 4'}, + {'op': 'label', 'ids': ['rest'], 'field': 'notes', 'value': 'equal/nonfinite'}, + {'op': 'undo'}, + {'op': 'redo'}, + {'op': 'save'}, + {'op': 'reopen'}, + {'op': 'save'}, + ], + 'expected_labels': { + 'low': {'group': 'good', 'notes': 'finite below 4'}, + 'rest': {'group': 'noise', 'notes': 'equal/nonfinite'}, + }, + } + + +@pytest.mark.parametrize('budget,seed', [(5, 0), (9, 11)]) +def test_ordinary_supervisor_against_bounded_all_rows(tmp_path, budget, seed): + params = make_sorting(tmp_path / 'source') + values = np.load(params.parent / 'amplitudes.npy') + values[:5] = [np.nan, np.inf, -np.inf, 4, 3] + np.save(params.parent / 'amplitudes.npy', values) + bounded = prepare_large( + params, + tmp_path / 'bounded', + max_spikes=budget, + seed=seed, + window_seconds=3, + stratum_seconds=20, + batch_size=2, + ) + ordinary = tmp_path / 'ordinary' + comparison.run_reference(params.parent, ledger(), ordinary) + session = BoundedSession(bounded) + before = session.journal_state() + merged = session.next_cluster_id + session.record_merge([0, 1], merged) + after = session.journal_state() + session.restore_journal(before) + session.restore_journal(after) + low, rest = session.next_cluster_id, session.next_cluster_id + 1 + session.record_threshold_split( + merged, + [rest, low], + {'attribute': 'amplitudes.npy', 'operator': '<', 'threshold': 4.0, 'nonfinite': 'keep'}, + ) + for alias, unit in [('low', low), ('rest', rest)]: + for field, value in ledger()['expected_labels'][alias].items(): + session.record_label(field, [unit], value) + session.save() + session.close() + reopened = BoundedSession(bounded) + reopened.save() + reopened.close() + replay = tmp_path / 'replay' + apply_curation(params, bounded, replay, batch_size=2) + result = comparison.compare_outputs(ordinary, replay, chunk_size=2) + assert result['ok'], result + assert result['rows'] == len(np.load(params.parent / 'spike_clusters.npy')) + streamed = tmp_path / 'streamed' + report = comparison.run_reference( + params.parent, ledger(), streamed, engine='streamed', chunk_size=2 + ) + assert report['evidence'] == 'streamed reference, NOT full ordinary GUI' + assert comparison.compare_outputs(ordinary, streamed, chunk_size=3)['ok'] + + +def outputs(tmp_path, left, right): + paths = [tmp_path / 'left', tmp_path / 'right'] + for path, ids in zip(paths, (left, right), strict=True): + path.mkdir() + np.save(path / 'spike_clusters.npy', np.asarray(ids)) + np.save(path / 'spike_times.npy', np.arange(len(ids), dtype=np.int64)) + return paths + + +def test_exact_mapping_detects_conflict_across_chunks(tmp_path): + left, right = outputs(tmp_path, [0, 1, 0, 1], [8, 9, 8, 8]) + report = comparison.compare_outputs(left, right, chunk_size=1) + assert not report['ok'] + assert report['partition_mismatch_rows'] == 1 + assert report['examples'][0]['row'] == 3 + + +def test_semantic_labels_follow_permuted_ids(tmp_path): + left, right = outputs(tmp_path, [0, 1, 0], [100, 200, 100]) + comparison.write_labels(left, {'group': {0: 'good', 1: 'noise'}}) + comparison.write_labels(right, {'group': {100: 'good', 200: 'noise'}}) + assert comparison.compare_outputs(left, right)['ok'] + comparison.write_labels(right, {'group': {100: 'noise', 200: 'good'}}) + assert len(comparison.compare_outputs(left, right)['manual_label_mismatches']) == 2 + + +@pytest.mark.parametrize('bad', ['rows', 'nan', 'times', 'shape']) +def test_invalid_outputs_fail_as_json_data(tmp_path, bad): + left, right = outputs(tmp_path, [0, 1], [8, 9]) + if bad == 'rows': + np.save(right / 'spike_clusters.npy', np.array([8])) + elif bad == 'nan': + np.save(right / 'spike_clusters.npy', np.array([8.0, np.nan])) + elif bad == 'times': + np.save(right / 'spike_times.npy', np.array([1, 0])) + else: + np.save(right / 'spike_clusters.npy', np.zeros((2, 2), dtype=int)) + result = comparison.compare_outputs(left, right) + assert not result['ok'] and result['error'] + + +def test_unmapped_zero_spike_label_units_require_explicit_review(tmp_path): + left, right = outputs(tmp_path, [0, 1], [8, 9]) + comparison.write_labels(left, {'group': {0: 'good', 1: 'noise', 99: 'mua'}}) + comparison.write_labels(right, {'group': {8: 'good', 9: 'noise', 100: 'mua'}}) + result = comparison.compare_outputs(left, right) + assert not result['ok'] + assert result['unmapped_label_ids']['left']['group'] == [99] + + +def test_csv_multiple_manual_fields_and_generated_info(tmp_path): + left, right = outputs(tmp_path, [0, 1], [8, 9]) + (left / 'cluster_manual.csv').write_text('cluster_id,group,notes\n0,good,yes\n1,noise,no\n') + (left / 'cluster_info.tsv').write_text('cluster_id\tgroup\n0\twrong\n') + comparison.write_labels( + right, {'group': {8: 'good', 9: 'noise'}, 'notes': {8: 'yes', 9: 'no'}} + ) + assert comparison.compare_outputs(left, right)['ok'] + + +def test_reference_handles_column_clusters(tmp_path): + params = make_sorting(tmp_path / 'source', columns=True) + comparison.run_reference(params.parent, ledger(), tmp_path / 'ordinary') + comparison.run_reference( + params.parent, ledger(), tmp_path / 'streamed', engine='streamed', chunk_size=2 + ) + assert comparison.compare_outputs(tmp_path / 'ordinary', tmp_path / 'streamed')['ok'] + + +@pytest.mark.parametrize('fail', [False, True]) +def test_reference_releases_supervisor_callbacks_and_states(tmp_path, monkeypatch, fail): + import gc + import weakref + + from phylib.utils.event import _EVENT + + from phy.cluster.supervisor import Supervisor + + params = make_sorting(tmp_path / 'source') + before = {id(item) for item in _EVENT._callbacks} + refs = [] + original = Supervisor.__init__ + + def track(self, *args, **kwargs): + original(self, *args, **kwargs) + refs.append(weakref.ref(self)) + + monkeypatch.setattr(Supervisor, '__init__', track) + actions = [{'op': 'reopen'}, {'op': 'reopen'}] + if fail: + actions.append({'op': 'label', 'ids': ['missing'], 'field': 'group', 'value': 'good'}) + definition = {'aliases': {'a': 0}, 'fields': ['group'], 'actions': actions} + if fail: + with pytest.raises(KeyError): + comparison.run_reference(params.parent, definition, tmp_path / 'reference') + else: + comparison.run_reference(params.parent, definition, tmp_path / 'reference') + gc.collect() + assert len(refs) == 3 + assert all(ref() is None for ref in refs) + assert {id(item) for item in _EVENT._callbacks} == before + + +@pytest.mark.parametrize('inside', [True, False]) +def test_compare_cli_cannot_overwrite_input_or_existing_report(tmp_path, monkeypatch, inside): + import sys + + left, right = outputs(tmp_path, [0, 1], [8, 9]) + target = left / 'spike_clusters.npy' if inside else tmp_path / 'existing.json' + if not inside: + target.write_text('preserve this report') + before = target.read_bytes() + monkeypatch.setattr( + sys, + 'argv', + ['compare_curation_modes', 'compare', str(left), str(right), '--output', str(target)], + ) + assert comparison.main() == 1 + assert target.read_bytes() == before + + +@pytest.mark.parametrize('threshold', [-100.0, 100.0]) +def test_streamed_empty_split_rejected_and_snapshots_closed(tmp_path, threshold): + params = make_sorting(tmp_path / 'source') + definition = { + 'aliases': {'a': 0}, + 'actions': [ + { + 'op': 'split', + 'parent': 'a', + 'threshold': threshold, + 'selected': 'low', + 'remainder': 'rest', + } + ], + } + with pytest.raises(ValueError, match='both selected and remainder'): + comparison.run_reference( + params.parent, definition, tmp_path / 'reference', engine='streamed', chunk_size=2 + ) + assert not list((tmp_path / 'reference').glob('.history-*.npy')) + + +def test_streamed_allocator_and_inactive_aliases_after_undo(tmp_path): + params = make_sorting(tmp_path / 'source') + actions = [ + {'op': 'merge', 'parents': ['a', 'b'], 'as': 'first'}, + {'op': 'undo'}, + {'op': 'merge', 'parents': ['a', 'b'], 'as': 'second'}, + ] + definition = {'aliases': {'a': 0, 'b': 1}, 'actions': actions} + report = comparison.run_reference( + params.parent, definition, tmp_path / 'reference', engine='streamed', chunk_size=2 + ) + assert report['aliases']['second'] > report['aliases']['first'] + actions.append({'op': 'label', 'ids': ['first'], 'field': 'group', 'value': 'good'}) + with pytest.raises(ValueError, match='active units'): + comparison.run_reference( + params.parent, definition, tmp_path / 'invalid', engine='streamed', chunk_size=2 + ) diff --git a/phy/apps/large_recording/tests/test_elapsed_workflow.py b/phy/apps/large_recording/tests/test_elapsed_workflow.py new file mode 100644 index 000000000..aeba29650 --- /dev/null +++ b/phy/apps/large_recording/tests/test_elapsed_workflow.py @@ -0,0 +1,82 @@ +"""Independent discontinuous-clock preparation and replay oracle.""" + +import json +from contextlib import closing + +import numpy as np +import pytest + +from phy.apps.large_recording.curation import BoundedSession +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import Source, hash_file +from phy.apps.large_recording.tests.test_source import make_sorting + + +def test_elapsed_producer_evidence_replay_and_exposure(tmp_path): + params = make_sorting(tmp_path / 'source') + contract = {'kind': 'elapsed_ticks', 'ticks_per_second': 1.0} + intervals = [[0, 7], [10, 20]] + params.write_text( + params.read_text() + f'time_base = {contract!r}\nvalid_intervals = {intervals!r}\n' + ) + source_path = params.parent + samples = np.load(source_path / 'spike_times.npy') + labels = np.load(source_path / 'spike_clusters.npy') + waveform_ids = np.load(source_path / '_phy_spikes_subset.spikes.npy') + np.save(source_path / '_phy_spikes_subset.sample_index.npy', samples[waveform_ids]) + np.save(source_path / '_phy_spikes_subset.window_bounds.npy', np.array(intervals, 'int64')) + (source_path / 'timestamps.json').write_text('{"provenance": "producer-owned clock"}\n') + (source_path / '_phy_spikes_subset.manifest.json').write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': { + 'time_base': contract, + 'window_bounds_coordinates': 'elapsed_ticks', + 'sample_index_coordinates': 'elapsed_ticks', + }, + } + ) + ) + original = {p.name: hash_file(p) for p in source_path.iterdir() if p.is_file()} + bounded = prepare_large(params, tmp_path / 'bounded', bin_seconds=5) + manifest = json.loads((bounded / '_phy_curation_manifest.json').read_text()) + assert manifest['time_base'] == contract + assert manifest['selection']['resolved_window_policy'] == 'producer' + np.testing.assert_array_equal(np.load(bounded / '_phy_spike_global_ids.npy'), np.arange(12)) + np.testing.assert_array_equal( + np.load(bounded / '_phy_spikes_subset.sample_index.npy'), samples[waveform_ids] + ) + # Independent exposure oracle, including the partial bin before the gap. + np.testing.assert_array_equal(np.load(bounded / manifest['exposure_file']), [5, 2, 5, 5]) + counts = np.load(bounded / manifest['counts_file']) + for row, unit in enumerate(manifest['unit_ids']): + expected, _ = np.histogram(samples[labels == unit], bins=[0, 5, 10, 15, 20]) + np.testing.assert_array_equal(counts[row], expected) + with closing(BoundedSession(bounded)) as session: + session.record_merge([0, 1], 100) + session.save() + with closing(BoundedSession(bounded)) as reopened: + np.testing.assert_array_equal(reopened.assignments, np.where(labels < 2, 100, labels)) + output = apply_curation(params, bounded, tmp_path / 'curated') + np.testing.assert_array_equal( + np.load(output / 'spike_clusters.npy'), np.where(labels < 2, 100, labels) + ) + assert (output / 'timestamps.json').read_bytes() == ( + source_path / 'timestamps.json' + ).read_bytes() + with Source(params) as source, Source(output / 'params.py') as curated: + assert source.identity == curated.identity + again = prepare_large(output / 'params.py', tmp_path / 'again', bin_seconds=5) + with closing(BoundedSession(again)) as session: + assert session.manifest['time_base'] == contract + assert original == {p.name: hash_file(p) for p in source_path.iterdir() if p.is_file()} + # A metadata edit cannot change the coordinate interpretation of saved evidence. + path = again / '_phy_curation_manifest.json' + altered = json.loads(path.read_text()) + altered['time_base']['kind'] = 'acquisition_samples' + path.write_text(json.dumps(altered)) + with pytest.raises(ValueError, match='time base'): + BoundedSession(again) diff --git a/phy/apps/large_recording/tests/test_export_validator.py b/phy/apps/large_recording/tests/test_export_validator.py new file mode 100644 index 000000000..dc4808ee3 --- /dev/null +++ b/phy/apps/large_recording/tests/test_export_validator.py @@ -0,0 +1,207 @@ +"""Focused contract and report tests for the AIND export validator.""" + +import json + +import tools.validate_aind_export as validator +from phy.apps.large_recording.source import hash_file +from phy.apps.large_recording.tests.test_source import make_sorting +from tools.validate_aind_export import main, validate_export + + +def test_validator_reports_engineering_evidence_and_input_hashes(tmp_path): + params = make_sorting(tmp_path / 'source') + params.write_text( + params.read_text() + + "time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': 1.0}\n" + + 'valid_intervals = [[0, 20]]\n' + ) + report = validate_export( + params, + phy_revision='phy-test-revision', + exporter_revision='exporter-test-revision', + batch_size=4, + ) + + assert report['status'] == 'passed' + assert report['failures'] == [] + assert report['engineering_validation_only'] is True + assert report['acquisition_truth_established'] is False + assert report['source']['num_spikes'] == 12 + assert report['source']['num_samples'] == 20 + assert report['source']['exposure']['valid_samples'] == 20 + assert report['source']['input_files']['spike_times.npy']['sha256'] + assert report['phy_revision'] == 'phy-test-revision' + assert report['exporter_revision'] == 'exporter-test-revision' + assert report['timing_seconds']['total'] >= 0 + + +def test_validator_preparation_report_matches_manifest_and_preserves_source(tmp_path): + params = make_sorting(tmp_path / 'source') + params.write_text( + params.read_text().replace('num_samples = 20', 'num_samples = 120') + + "time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': 1.0}\n" + + 'valid_intervals = [[0, 120]]\n' + ) + source_path = params.parent + before_hashes = { + path.name: hash_file(path) for path in source_path.iterdir() if path.is_file() + } + output_dir = tmp_path / 'bounded' + + report = validate_export( + params, + phy_revision='phy-test-revision', + exporter_revision='exporter-test-revision', + output_dir=output_dir, + batch_size=4, + ) + + assert report['status'] == 'passed', report['failures'] + output = report['prepared_output'] + manifest = json.loads((output_dir / '_phy_curation_manifest.json').read_text()) + disk_bytes = sum(path.stat().st_size for path in output_dir.rglob('*') if path.is_file()) + assert output['num_spikes'] == manifest['bounded_num_spikes'] + assert output['num_units'] == len(manifest['unit_ids']) + assert output['waveform_coverage'] == manifest['waveform_coverage'] + assert output['budgets'] == manifest['budgets'] + assert output['output_bytes'] == disk_bytes + assert report['source']['input_files'] == { + name: {'bytes': (source_path / name).stat().st_size, 'sha256': digest} + for name, digest in before_hashes.items() + } + assert { + path.name: hash_file(path) for path in source_path.iterdir() if path.is_file() + } == before_hashes + + +def test_validator_reports_source_rejection_as_json_failure(tmp_path): + params = make_sorting(tmp_path / 'source') + (params.parent / 'timestamps.json').write_text(json.dumps({'frames': []})) + + report = validate_export( + params, + phy_revision='phy-test-revision', + exporter_revision='exporter-test-revision', + ) + + assert report['status'] == 'failed' + assert len(report['failures']) == 1 + assert 'explicit time_base' in report['failures'][0]['message'] + assert report['acquisition_truth_established'] is False + + +def test_validator_requires_explicit_aind_time_and_exposure_contract(tmp_path): + params = make_sorting(tmp_path / 'source') + + report = validate_export( + params, + phy_revision='phy-test-revision', + exporter_revision='exporter-test-revision', + ) + + assert report['status'] == 'failed' + assert ( + report['failures'][0]['message'] == 'AIND export requires an explicit time_base contract.' + ) + + +def test_validator_rejects_nested_preparation_destination(tmp_path): + params = make_sorting(tmp_path / 'source') + params.write_text( + params.read_text() + + "time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': 1.0}\n" + + 'valid_intervals = [[0, 20]]\n' + ) + + report = validate_export( + params, + phy_revision='phy-test-revision', + exporter_revision='exporter-test-revision', + output_dir=params.parent / 'derived', + ) + + assert report['status'] == 'failed' + assert 'outside the immutable source' in report['failures'][0]['message'] + assert not (params.parent / 'derived').exists() + + +def test_cli_report_cannot_overwrite_source_params(tmp_path, capsys): + params = make_sorting(tmp_path / 'source') + original = params.read_bytes() + + result = main( + [ + str(params), + '--phy-revision', + 'phy-test-revision', + '--exporter-revision', + 'exporter-test-revision', + '--report', + str(params), + ] + ) + + assert result == 1 + assert params.read_bytes() == original + report = json.loads(capsys.readouterr().out) + assert report['status'] == 'failed' + assert 'outside the immutable source' in report['failures'][0]['message'] + + +def test_cli_report_cannot_target_future_prepared_output(tmp_path, capsys): + params = make_sorting(tmp_path / 'source') + params.write_text( + params.read_text() + + "time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': 1.0}\n" + + 'valid_intervals = [[0, 20]]\n' + ) + output_dir = tmp_path / 'prepared' + report_path = output_dir / 'params.py' + + result = main( + [ + str(params), + '--phy-revision', + 'phy-test-revision', + '--exporter-revision', + 'exporter-test-revision', + '--output-dir', + str(output_dir), + '--report', + str(report_path), + ] + ) + + assert result == 1 + assert not output_dir.exists() + report = json.loads(capsys.readouterr().out) + assert report['status'] == 'failed' + assert 'prepared output directories' in report['failures'][0]['message'] + + +def test_cli_report_exclusive_create_preserves_racing_file(tmp_path, monkeypatch, capsys): + params = make_sorting(tmp_path / 'source') + report_path = tmp_path / 'report.json' + + def create_report_during_validation(*args, **kwargs): + report_path.write_text('keep me', encoding='utf-8') + return {'status': 'passed'} + + monkeypatch.setattr(validator, 'validate_export', create_report_during_validation) + result = main( + [ + str(params), + '--phy-revision', + 'phy-test-revision', + '--exporter-revision', + 'exporter-test-revision', + '--report', + str(report_path), + ] + ) + + assert result == 1 + assert report_path.read_text(encoding='utf-8') == 'keep me' + report = json.loads(capsys.readouterr().out) + assert report['status'] == 'failed' + assert 'already exists' in report['failures'][0]['message'] diff --git a/phy/apps/large_recording/tests/test_graphical_split.py b/phy/apps/large_recording/tests/test_graphical_split.py new file mode 100644 index 000000000..b2a45c16e --- /dev/null +++ b/phy/apps/large_recording/tests/test_graphical_split.py @@ -0,0 +1,311 @@ +"""Graphical threshold previews commit full-source semantic commands.""" + +import numpy as np +import pytest +from phylib.utils import connect, emit, unconnect + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.tests.test_adapter import make_controller, make_gui_sorting +from phy.apps.template.gui import TemplateController +from phy.gui.qt import Qt +from phy.plot.tests import mouse_click, mouse_drag +from phy.plot.transform import NDC, Range + + +def drag_threshold(qtbot, view, threshold): + qtbot.wait(50) + width, height = view.canvas.get_size() + ndc = Range(view.data_bounds, NDC).apply([[0, threshold]])[0] + screen = (ndc + np.asarray(view.canvas.panzoom.pan)) * view.canvas.panzoom._zoom_aspect() + pixel = ((screen[0] + 1) * width / 2, (1 - screen[1]) * height / 2) + mouse_drag( + qtbot, view.canvas, (pixel[0] - 10, pixel[1]), pixel, button='right', modifiers=('Alt',) + ) + assert view.split_threshold == pytest.approx(threshold, abs=0.03) + + +def test_drag_k_semantic_commit_waveform_preview_history_save_replay(qtbot, tmp_path): + params, controller = make_controller(tmp_path) + gui = controller.create_gui( + default_views=['EvidenceAmplitudeView', 'WaveformView'], do_prompt_save=False + ) + supervisor = controller.supervisor + requests = [] + + @connect(event='request_split', sender=supervisor) + def on_request(sender): + requests.append(True) + return np.array([0], dtype=np.int64) + + try: + gui.show() + supervisor.select([0]) + supervisor.block() + view = gui.get_view('EvidenceAmplitudeView') + waveform = gui.get_view('WaveformView') + waveform.waveforms_type = 'waveforms' + waveform.plot() + drag_threshold(qtbot, view, 4.5) + threshold = view.split_threshold + drawn = waveform._displayed_bunchs[0].spike_ids + amplitudes = controller.bounded_session.attributes['amplitudes.npy'] + np.testing.assert_array_equal( + waveform._highlighted_spike_ids, drawn[amplitudes[drawn] < threshold] + ) + assert waveform._highlighted_spike_color == view.split_preview_color + assert supervisor.actions.get('split').isEnabled() + assert view.on_request_split(supervisor).size == 0 + before = supervisor.clustering.spike_clusters.copy() + view.canvas.setFocus() + qtbot.keyClick(view.canvas, Qt.Key_K) + supervisor.block() + assert not requests + assert controller.bounded_session.events[-1]['operation'] == 'threshold_split' + predicate = controller.bounded_session.events[-1]['predicate'] + assert predicate['threshold'] == threshold + after = supervisor.clustering.spike_clusters.copy() + assert view.split_threshold is None + assert not supervisor.actions.get('split').isEnabled() + supervisor.undo() + supervisor.block() + np.testing.assert_array_equal(supervisor.clustering.spike_clusters, before) + supervisor.redo() + supervisor.block() + np.testing.assert_array_equal(supervisor.clustering.spike_clusters, after) + supervisor.save() + bounded = controller.dir_path + finally: + unconnect(on_request) + gui.close() + controller.close() + + reopened = TemplateController(dir_path=bounded, config_dir=tmp_path / 'config', plugins=[]) + try: + np.testing.assert_array_equal(reopened.supervisor.clustering.spike_clusters, after) + result = apply_curation(params, bounded, tmp_path / 'replay', batch_size=2) + source = np.load(params.parent / 'spike_clusters.npy') + values = np.load(params.parent / 'amplitudes.npy') + children = reopened.bounded_session.events[-1]['children'] + expected = source.copy() + expected[(source == 0) & (values < threshold)] = children[1] + expected[(source == 0) & ~(values < threshold)] = children[0] + np.testing.assert_array_equal(np.load(result / 'spike_clusters.npy'), expected) + finally: + reopened.close() + + +def test_overview_original_clock_all_windows_budget_and_window_toggle(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'source') + bounded = prepare_large( + params, tmp_path / 'bounded', window_seconds=3, stratum_seconds=5, bin_seconds=5 + ) + controller = TemplateController(dir_path=bounded, config_dir=tmp_path / 'config', plugins=[]) + gui = controller.create_gui(default_views=['EvidenceAmplitudeView'], do_prompt_save=False) + try: + gui.show() + controller.supervisor.select([0]) + controller.supervisor.block() + view = gui.get_view('EvidenceAmplitudeView') + assert view.amplitude_scope == 'overview' + data = view.get_clusters_data(load_all=True)[0] + assert len(data.spike_ids) > len(controller._window_ids(0)) + np.testing.assert_allclose( + data.spike_times, controller.model.spike_samples[data.spike_ids] / 30000 + ) + controller.n_spikes_amplitudes = 2 + controller.n_spikes_amplitudes_total = 2 + assert len(view.get_clusters_data()[-1].spike_ids) <= 2 + drag_threshold(qtbot, view, 4.5) + controller.inspect_evidence_window(0) + assert view.split_threshold is None + assert view.amplitude_scope == 'window' + local = view.get_clusters_data(load_all=True)[0] + np.testing.assert_array_equal(local.spike_ids, controller._window_ids(0)) + view.toggle_amplitude_scope() + assert view.amplitude_scope == 'overview' + assert view.state['amplitude_scope'] == 'overview' + finally: + gui.close() + controller.close() + + +def test_invalid_preview_and_row_splits_preserve_state(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(default_views=['EvidenceAmplitudeView'], do_prompt_save=False) + supervisor = controller.supervisor + try: + gui.show() + supervisor.select([0]) + supervisor.block() + view = gui.get_view('EvidenceAmplitudeView') + drag_threshold(qtbot, view, 4.5) + before = supervisor.clustering.spike_clusters.copy() + with pytest.raises(ValueError, match='cannot be replayed'): + supervisor.split([0]) + assert view.split_threshold is not None + view.split_threshold = -1 + with pytest.raises(ValueError, match='Both sides'): + supervisor.split() + assert view.split_threshold == -1 + supervisor.actions.get('split').trigger() + supervisor.block() + assert view.dock.status == 'Both sides of the threshold must have evidence spikes.' + assert view.split_threshold == -1 + assert supervisor.actions.get('split').isEnabled() + np.testing.assert_array_equal(supervisor.clustering.spike_clusters, before) + assert controller.bounded_session.events == [] + view.clear_split_selection() + view.canvas.lasso.add((-0.5, -0.5)) + view.canvas.lasso.add((0.5, -0.5)) + view.canvas.lasso.add((0, 0.5)) + assert view.on_request_split(supervisor).size == 0 + with pytest.raises(ValueError, match='Exactly one'): + supervisor.split() + assert not supervisor.actions.get('split').isEnabled() + finally: + gui.close() + controller.close() + + +def test_preview_invalidated_by_context_and_hidden_dock(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui( + default_views=['EvidenceAmplitudeView', 'WaveformView'], do_prompt_save=False + ) + supervisor = controller.supervisor + try: + gui.show() + supervisor.select([0]) + supervisor.block() + view = gui.get_view('EvidenceAmplitudeView') + waveform = gui.get_view('WaveformView') + drag_threshold(qtbot, view, 4.5) + waveform.plot() + assert view.split_threshold is None + for event in ('selected_channel_changed', 'selected_feature_changed'): + drag_threshold(qtbot, view, 4.5) + emit(event, waveform) + assert view.split_threshold is None + assert not len(waveform._highlighted_spike_ids) + drag_threshold(qtbot, view, 4.5) + waveform.next_waveforms_type() + assert view.split_threshold is None + waveform.waveforms_type = 'waveforms' + waveform.plot() + assert not len(waveform._highlighted_spike_ids) + drag_threshold(qtbot, view, 4.5) + mouse_click(qtbot, view.canvas, (20, 20), button='right', modifiers=('Control',)) + assert view.split_threshold is None + drag_threshold(qtbot, view, 4.5) + view.actions.get('clear_amplitude_split_threshold').trigger() + assert view.split_threshold is None + drag_threshold(qtbot, view, 4.5) + view.dock.hide() + assert view.split_threshold is None + assert not supervisor.actions.get('split').isEnabled() + view.dock.show() + drag_threshold(qtbot, view, 4.5) + supervisor.select([0, 1]) + supervisor.block() + assert view.split_threshold is None + assert not view._can_set_split_threshold() + supervisor.select([0]) + supervisor.block() + drag_threshold(qtbot, view, 4.5) + supervisor.toggle_merge_mode() + supervisor.block() + assert view.split_threshold is None + assert not view._can_set_split_threshold() + finally: + gui.close() + controller.close() + + +def test_commit_scope_other_gui_ambiguous_and_closed_previews(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(default_views=['EvidenceAmplitudeView'], do_prompt_save=False) + supervisor = controller.supervisor + other = None + try: + gui.show() + supervisor.select([0]) + supervisor.block() + first = gui.get_view('EvidenceAmplitudeView') + second = gui.create_and_add_view('EvidenceAmplitudeView') + supervisor.select([0]) + supervisor.block() + # Simulate contradictory preview state without the exclusive activation + # event, as a plugin or restored stale callback could create it. + first.split_threshold = 3 + second.split_threshold = 4 + with pytest.raises(ValueError, match='Exactly one'): + supervisor.split() + second.clear_split_selection() + # A foreign view answering the global legacy event cannot intercept K. + other = controller.create_evidence_amplitude_view() + other.on_select([0]) + other.split_threshold = 6 + other.gui = object() + foreign_requests = [] + + @connect(event='request_split') + def foreign_request(sender): + foreign_requests.append(True) + return np.array([1], dtype=np.int64) + + try: + supervisor.split() + supervisor.block() + assert not foreign_requests + assert controller.bounded_session.events[-1]['predicate']['threshold'] == 3 + finally: + unconnect(foreign_request) + supervisor.undo() + supervisor.block() + supervisor.select([0]) + supervisor.block() + drag_threshold(qtbot, first, 4.5) + # Closing the originating dock invalidates its pending predicate. + first.close() + assert first.split_threshold is None + with pytest.raises(ValueError, match='Exactly one'): + supervisor.split() + finally: + if other is not None: + other.canvas.close() + gui.close() + controller.close() + + +def test_failed_semantic_observer_preserves_preview_and_assignments(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(default_views=['EvidenceAmplitudeView'], do_prompt_save=False) + supervisor = controller.supervisor + try: + gui.show() + supervisor.select([0]) + supervisor.block() + view = gui.get_view('EvidenceAmplitudeView') + drag_threshold(qtbot, view, 4.5) + threshold = view.split_threshold + before = supervisor.clustering.spike_clusters.copy() + + @connect(event='cluster', sender=supervisor.clustering) + def fail(sender, up): + raise RuntimeError('observer failed') + + try: + with pytest.raises(RuntimeError, match='observer failed'): + supervisor.split() + supervisor.block() + np.testing.assert_array_equal(supervisor.clustering.spike_clusters, before) + assert controller.bounded_session.events == [] + assert supervisor.selected == [0] + assert view.split_threshold == threshold + assert supervisor.actions.get('split').isEnabled() + finally: + unconnect(fail) + finally: + gui.close() + controller.close() diff --git a/phy/apps/large_recording/tests/test_gui_lifecycle.py b/phy/apps/large_recording/tests/test_gui_lifecycle.py new file mode 100644 index 000000000..c612ad425 --- /dev/null +++ b/phy/apps/large_recording/tests/test_gui_lifecycle.py @@ -0,0 +1,240 @@ +"""GUI transaction, proposition review, and pending-metric regressions.""" + +import json + +import numpy as np +import pytest +from phylib.utils import connect, unconnect + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.tests.test_adapter import make_controller, make_gui_sorting +from phy.apps.template.gui import TemplateController + + +def test_elapsed_producer_windows_open_and_navigate_in_gui(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'source') + intervals = [[0, 210000], [300000, 600000]] + contract = {'kind': 'elapsed_ticks', 'ticks_per_second': 30000.0} + params.write_text( + params.read_text() + f'time_base = {contract!r}\nvalid_intervals = {intervals!r}\n' + ) + np.save(params.parent / '_phy_spikes_subset.window_bounds.npy', np.array(intervals, 'int64')) + (params.parent / '_phy_spikes_subset.manifest.json').write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': {'window_bounds_coordinates': 'elapsed_ticks'}, + } + ) + ) + output = prepare_large(params, tmp_path / 'bounded', window_policy='producer', bin_seconds=5) + controller = TemplateController( + dir_path=output, config_dir=tmp_path / 'config', plugins=[], enable_threading=False + ) + gui = None + try: + gui = controller.create_gui(do_prompt_save=False) + controller.supervisor.select([0, 1]) + controller.supervisor.block() + controller.inspect_evidence_window(1) + assert controller.model.sample_rate == 30000.0 + assert controller.model.traces is None + assert controller.get_mean_firing_rate(0) == pytest.approx(4 / 17) + view = gui.get_view('EvidenceAmplitudeView') + assert view.duration == 10 + assert view.window_label.text() == ('Evidence window 1 | Recording time: 10.000–20.000 s') + finally: + if gui is not None: + gui.close() + controller.close() + + +def test_window_time_labels_follow_navigation(qtbot, tmp_path): + _, controller = make_controller(tmp_path, sparse=True) + gui = controller.create_gui(default_views=['EvidenceAmplitudeView'], do_prompt_save=False) + try: + view = gui.get_view('EvidenceAmplitudeView') + assert view.time_axis_label.text() == 'Recording time (seconds)' + assert view.amplitude_scope == 'overview' + initial = view.window_label.text() + controller.inspect_evidence_window(1) + start, end = controller.current_evidence_window / controller.model.sample_rate + assert view.window_label.text() == ( + f'Evidence window 1 | Recording time: {start:.3f}–{end:.3f} s' + ) + assert view.window_label.text() != initial + assert view.duration == end - start + finally: + gui.close() + controller.close() + + +def test_gui_observer_mutation_rolls_back_views_selection_and_journal(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(default_views=[], do_prompt_save=False) + supervisor = controller.supervisor + once = [] + + try: + supervisor.select([0]) + supervisor.block() + assignments = supervisor.clustering.spike_clusters.copy() + events = list(controller.bounded_session.events) + selection = supervisor.selection.snapshot() + cluster_ids = supervisor.cluster_view.get_ids() + similarity_ids = supervisor.similarity_view.get_ids() + histories = tuple( + (history._index, len(history._history)) + for history in ( + supervisor.clustering._undo_stack, + supervisor.cluster_meta._undo_stack, + supervisor._global_history, + supervisor.session_history, + ) + ) + dirty = supervisor.is_dirty() + + @connect(event='cluster', sender=supervisor.clustering) + def mutate(sender, up): + if not once: + once.append(True) + sender.assign([0]) + + with pytest.raises((ValueError, KeyError)): + supervisor.merge([0, 1]) + supervisor.block() + + np.testing.assert_array_equal(supervisor.clustering.spike_clusters, assignments) + assert controller.bounded_session.events == events + assert supervisor.selection.snapshot() == selection + assert supervisor.cluster_view.get_ids() == cluster_ids + assert supervisor.similarity_view.get_ids() == similarity_ids + assert ( + tuple( + (history._index, len(history._history)) + for history in ( + supervisor.clustering._undo_stack, + supervisor.cluster_meta._undo_stack, + supervisor._global_history, + supervisor.session_history, + ) + ) + == histories + ) + assert supervisor.is_dirty() == dirty + finally: + if 'mutate' in locals(): + unconnect(mutate) + gui.close() + controller.close() + + +def test_gui_proposition_review_undo_redo_and_checkpoint_reopen(qtbot, tmp_path): + params = make_gui_sorting(tmp_path / 'source') + proposal = { + 'format_version': '2', + 'unit_ids': [0, 1, 2, 99], + 'merges': [{'unit_ids': [0, 1]}], + } + (params.parent / 'curation.json').write_text(json.dumps(proposal), encoding='utf-8') + bounded = prepare_large( + params, + tmp_path / 'bounded', + window_seconds=20, + stratum_seconds=20, + bin_seconds=5, + batch_size=2, + ) + config = tmp_path / 'config' + controller = TemplateController( + dir_path=bounded, config_dir=config, plugins=[], enable_threading=False + ) + gui = controller.create_gui(default_views=[], do_prompt_save=False) + supervisor = controller.supervisor + proposition = supervisor.merge_propositions.catalog.propositions[0] + key = proposition.key + + try: + supervisor._review_merge_proposition(supervisor.merge_propositions_view, key) + assert supervisor.selection.state.is_merge_mode + assert supervisor.selected_merge == [0, 1] + + merged = supervisor.merge() + supervisor.block() + assert merged.added == [100] + assert supervisor.merge_propositions.catalog.status_for(key).value == 'accepted' + assert controller.bounded_session.events[-1]['target'] == 100 + + supervisor.undo() + supervisor.block() + assert supervisor.merge_propositions.catalog.status_for(key).value == 'pending' + assert controller.bounded_session.events == [] + + supervisor.redo() + supervisor.block() + assert supervisor.merge_propositions.catalog.status_for(key).value == 'accepted' + assert controller.bounded_session.events[-1]['target'] == 100 + supervisor.save() + finally: + gui.close() + controller.close() + + reopened = TemplateController( + dir_path=bounded, config_dir=config, plugins=[], enable_threading=False + ) + try: + key = reopened.supervisor.merge_propositions.catalog.propositions[0].key + assert reopened.supervisor.merge_propositions.catalog.status_for(key).value == 'accepted' + assert reopened.bounded_session.events[-1]['target'] == 100 + finally: + reopened.close() + + +def test_default_bounded_views_handle_threshold_children_with_pending_metrics(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(do_prompt_save=False) + try: + supervisor = controller.supervisor + update = supervisor.split_source_amplitude(0, 4) + supervisor.block() + assert set(update.added) == {100, 101} + for cluster_id in update.added: + row = supervisor.cluster_view._model.row_by_id(cluster_id) + assert row['fr'] is None + assert row['full_count'] is None + assert row['evidence'] == 'pending replay' + finally: + gui.close() + controller.close() + + +def test_merge_redo_then_amplitude_split_saves_and_reopens(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + bounded = controller.dir_path + gui = controller.create_gui(default_views=[], do_prompt_save=False) + supervisor = controller.supervisor + try: + supervisor.merge([0, 1]) + supervisor.block() + supervisor.undo() + supervisor.block() + supervisor.redo() + supervisor.block() + split = supervisor.split_source_amplitude(2, 6.0) + supervisor.block() + assignments = supervisor.clustering.spike_clusters.copy() + supervisor.save() + children = list(split.added) + finally: + gui.close() + controller.close() + reopened = TemplateController( + dir_path=bounded, config_dir=tmp_path / 'config', plugins=[], enable_threading=False + ) + try: + np.testing.assert_array_equal(reopened.bounded_session.assignments, assignments) + assert set(reopened.bounded_session.events[-1]['children']) == set(children) + assert len(reopened.bounded_session.events) == 2 + finally: + reopened.close() diff --git a/phy/apps/large_recording/tests/test_metadata.py b/phy/apps/large_recording/tests/test_metadata.py new file mode 100644 index 000000000..6ded32f26 --- /dev/null +++ b/phy/apps/large_recording/tests/test_metadata.py @@ -0,0 +1,496 @@ +"""Membership-scoped metadata, source layouts and explicit legacy migration.""" + +import csv +import json + +import numpy as np +import pytest + +from phy.apps.large_recording.curation import BoundedSession, interpret_events +from phy.apps.large_recording.metadata import ( + POLICY, + SCHEMA_FILE, + complete_schema, + read_metadata, + validate_schema, + write_metadata, +) +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import Source, hash_file +from phy.apps.large_recording.tests.test_curation import make_session +from phy.apps.large_recording.tests.test_source import make_sorting + + +def prepared(params, destination): + prepare_large( + params, + destination, + window_seconds=20, + stratum_seconds=20, + max_spikes=100, + bin_seconds=5, + batch_size=2, + ) + return BoundedSession(destination) + + +def metadata_sorting(tmp_path): + params = make_sorting(tmp_path / 'source') + (params.parent / 'cluster_si_unit_ids.tsv').write_text( + 'cluster_id\tsi_unit_id\n0\t40\n1\t44\n2\t48\n88\tempty\n99\t99\n' + ) + (params.parent / 'cluster_producer.csv').write_text( + 'unknown,cluster_id,bombcell,snr\nsua,0,good,12.26\nsua,1,good,12.92\n' + 'sua,2,mua,12.16\n,88,,\nunchanged,99,noise,3\n' + ) + return params + + +def test_three_round_membership_and_noop_export(tmp_path): + params = metadata_sorting(tmp_path) + original = Source(params) + expected_unchanged = {field: values.get(99) for field, values in original.metadata.items()} + expected_empty = {field: values.get(88) for field, values in original.metadata.items()} + original.close() + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 4.0, + 'nonfinite': 'keep', + } + previous_history = 0 + for round_id in range(3): + session = prepared(params, tmp_path / f'bounded-{round_id}') + assert session.metadata_policy == POLICY + if round_id == 0: + session.record_label('group', [0], 'good') + session.record_merge([0, 1], 100, {'group': 'good', 'snr': 12.26}) + children = [100] + elif round_id == 1: + assert session.metadata['group'][100] == 'good' + session.record_threshold_split( + 100, [101, 102], predicate, {'group': {101: 'good', 102: 'good'}} + ) + children = [101, 102] + else: + session.record_merge([101, 102, 2], 103, {'group': 'good'}) + children = [103] + for values in session.metadata.values(): + assert all(values[unit] is None for unit in children) + for field, value in expected_unchanged.items(): + assert session.metadata[field].get(99) == value + for field, value in expected_empty.items(): + assert session.metadata[field].get(88) == value + session.record_label('group', children, 'good') + session.save() + output = apply_curation(params, session.path, tmp_path / f'round-{round_id}') + assert (output / 'cluster_si_unit_ids.tsv').exists() + assert not (output / 'cluster_si_unit_id.tsv').exists() + with (output / 'cluster_producer.csv').open(newline='') as stream: + rows = list(csv.reader(stream)) + assert rows[0] == ['unknown', 'cluster_id', 'bombcell', 'snr'] + for unit in children: + assert next(row for row in rows[1:] if row[1] == str(unit)) == ['', str(unit), '', ''] + current = Source(output / 'params.py') + assert current.metadata == session.metadata + history = current.metadata_provenance['history'] + membership = [ + record for record in history if record['operation'] in ('merge', 'threshold_split') + ] + latest = membership[-1] + assert latest['parent_metadata']['group'][str(latest['parents'][0])] == 'good' + if round_id == 0: + assert latest['parent_metadata']['group']['1'] == 'mua' + assert len(history) > previous_history + previous_history = len(history) + current.close() + session.close() + # A full no-op round retains the explicit label on this membership. + noop = prepared(output / 'params.py', tmp_path / f'noop-bounded-{round_id}') + assert all(noop.metadata['group'][unit] == 'good' for unit in children) + noop.save() + noop_output = apply_curation( + output / 'params.py', noop.path, tmp_path / f'noop-output-{round_id}' + ) + noop.close() + params = noop_output / 'params.py' + + +def test_typed_tables_roundtrip_and_conflicts(tmp_path): + schema = { + 'version': 1, + 'tables': [ + { + 'filename': 'cluster_si_unit_ids.tsv', + 'delimiter': '\t', + 'columns': ['cluster_id', 'si_unit_id'], + }, + { + 'filename': 'cluster_metrics.csv', + 'delimiter': ',', + 'columns': ['snr', 'cluster_id', 'flag', 'details'], + }, + ], + } + metadata = { + 'si_unit_id': {0: 40, 1: None}, + 'snr': {0: 12.26, 1: None}, + 'flag': {0: True, 1: False}, + 'details': {0: ['a', 1], 1: {'x': 2}}, + 'group': {0: 'good', 1: None}, + } + write_metadata(tmp_path, metadata, schema) + assert read_metadata(tmp_path)[0] == metadata + assert read_metadata(tmp_path)[1] == complete_schema(schema, metadata) + metric = tmp_path / 'cluster_metrics.csv' + metric.write_text(metric.read_text().replace('12.26', '99.0')) + with pytest.raises(ValueError, match='cluster_metrics.csv, field snr, unit 0'): + read_metadata(tmp_path) + + +@pytest.mark.parametrize( + 'filename', ['../cluster_metric.tsv', 'cluster_info.tsv', 'cluster_bad\\evil.tsv', 'params.py'] +) +def test_unsafe_or_reserved_filenames(filename): + with pytest.raises(ValueError, match='filename'): + validate_schema( + { + 'version': 1, + 'tables': [ + {'filename': filename, 'delimiter': '\t', 'columns': ['cluster_id', 'metric']} + ], + } + ) + + +def test_duplicate_fields_and_casefold_collisions(tmp_path): + (tmp_path / 'cluster_a.tsv').write_text('cluster_id\tsnr\n0\t1\n') + (tmp_path / 'cluster_b.csv').write_text('cluster_id,snr\n0,1\n') + with pytest.raises(ValueError, match='Duplicate metadata field'): + read_metadata(tmp_path) + with pytest.raises(ValueError, match='colliding'): + validate_schema( + { + 'version': 1, + 'tables': [ + { + 'filename': 'cluster_A.tsv', + 'delimiter': '\t', + 'columns': ['cluster_id', 'a'], + }, + { + 'filename': 'cluster_a.tsv', + 'delimiter': '\t', + 'columns': ['cluster_id', 'b'], + }, + ], + } + ) + + +def test_legacy_blank_omitted_rows_and_conflicting_edit(tmp_path): + (tmp_path / 'cluster_metric.tsv').write_text('cluster_id\tmetric\n0\t4\n2\t\n') + (tmp_path / '_phy_cluster_metadata.json').write_text( + json.dumps({'metric': {'0': 4, '1': None, '2': ''}}) + ) + metadata, schema = read_metadata(tmp_path) + assert metadata['metric'] == {0: 4, 1: None, 2: ''} + write_metadata(tmp_path, metadata, schema) + assert read_metadata(tmp_path)[0] == metadata + assert '1\t\n' in (tmp_path / 'cluster_metric.tsv').read_text() + (tmp_path / 'cluster_metric.tsv').write_text('cluster_id\tmetric\n0\t9\n1\t\n2\t\n') + with pytest.raises(ValueError, match='field metric, unit 0'): + read_metadata(tmp_path) + + +def test_legacy_migration_atomic_reopen_history_and_reviews(tmp_path): + source, session = make_session(tmp_path) + session.record_merge([0, 1], 100, {'group': 'good', 'producer': 9}) + session.record_label('group', [100], 'good') + session.record_merge([100, 2], 101, {'group': 'good', 'producer': 9}) + session.record_label('comment', [101], 'explicit decision') + review = {'opaque_review': ['unchanged']} + legacy_generation = session.save(extra_documents={'review.json': review}) + manifest_hash = hash_file(session.path / '_phy_curation_manifest.json') + old_hash = hash_file(legacy_generation / 'metadata.json') + before = session.snapshot() + event = session.migrate_metadata() + assert event['operation'] == 'metadata_policy' + assert session.metadata['group'][101] is None # Parent's explicit label expires. + assert session.metadata['producer'][101] is None + assert session.metadata['comment'][101] == 'explicit decision' + migrated = session.snapshot() + session.restore(before) + assert session.metadata['group'][101] == 'good' + session.restore(migrated) + assert session.metadata['group'][101] is None + np.testing.assert_array_equal(session.assignments, before['assignments']) + assert session.next_cluster_id == before['next_cluster_id'] + session.save() + reopened = BoundedSession(session.path) + assert reopened.metadata_policy == POLICY + assert reopened.metadata == session.metadata + assert reopened.extra_documents['review.json'] == review + assert hash_file(session.path / '_phy_curation_manifest.json') == manifest_hash + assert hash_file(legacy_generation / 'metadata.json') == old_hash + reopened.record_label('group', [101], 'good') + assert reopened.metadata['group'][101] == 'good' + reopened.record_merge([101, 99], 102, {'group': 'good'}) + assert reopened.metadata['group'][102] is None + source.close() + session.close() + reopened.close() + + +def test_legacy_interpretation_is_unchanged(): + events = [ + { + 'version': 1, + 'operation': 'merge', + 'parents': [0, 1], + 'target': 2, + 'metadata': {'group': 'good', 'snr': 12.26}, + } + ] + _, metadata, _ = interpret_events([0, 1], {'snr': {0: 12.26, 1: 8}}, events) + assert metadata == {'group': {2: 'good'}, 'snr': {2: 12.26}} + + +def test_blank_legacy_metadata_preserves_revision_and_v1_values(tmp_path): + from phy.apps.large_recording.source import assignment_revision + + params = make_sorting(tmp_path / 'full') + (params.parent / 'cluster_metric.tsv').write_text('cluster_id\tmetric\n0\t\n') + with Source(params) as source: + legacy = {'group': {0: 'good', 1: 'mua', 2: 'good', 99: 'noise'}, 'metric': {0: ''}} + assert source.metadata == legacy + assert source.revision == assignment_revision(params.parent, legacy) + _, projected, _ = interpret_events([0, 1], {'metric': {0: ''}}, []) + assert projected == {'metric': {0: ''}} + + +def test_save_snapshot_policy_schema_and_provenance_use_supplied_state(tmp_path): + source, session = make_session(tmp_path) + session.record_merge([0, 1], 100, {'group': 'good'}) + legacy = session.snapshot() + session.migrate_metadata() + session.record_label('group', [100], 'good') + # Save an earlier valid history prefix, as undo/history clients can do. + session.save(**legacy) + reopened = BoundedSession(session.path) + assert reopened.events == legacy['events'] + assert reopened.metadata == legacy['metadata'] + assert reopened.metadata_policy is None + assert len(reopened.metadata_provenance['history']) == 1 + source.close() + session.close() + reopened.close() + + +def test_migration_rejects_same_metadata_with_changed_source_coordinates(tmp_path): + source, session = make_session(tmp_path) + source.close() + amplitudes = np.load(source.path / 'amplitudes.npy') + amplitudes[0] += 1 + np.save(source.path / 'amplitudes.npy', amplitudes) + with pytest.raises(ValueError, match='identity'): + session.migrate_metadata() + assert not session.events + session.close() + + +def test_gui_membership_quality_metadata_and_history(tmp_path): + from phy.apps.large_recording.tests.test_adapter import make_controller + + _, controller = make_controller(tmp_path) + try: + supervisor = controller.supervisor + supervisor.label('producer_metric', 42, [0, 1]) + supervisor.label('classification', 'sua', [0, 1]) + supervisor.merge([0, 1]) + session = controller.bounded_session + for field in ('producer_metric', 'classification', 'group'): + assert session.metadata[field][100] is None + assert supervisor.cluster_meta.get(field, 100) is None + assert supervisor.cluster_meta.get('group', 2) == 'good' + supervisor.label('group', 'good', [100]) + supervisor.undo() + assert supervisor.cluster_meta.get('group', 100) is None + supervisor.redo() + assert supervisor.cluster_meta.get('group', 100) == 'good' + supervisor.split_source_amplitude(100, 4) + children = session.events[-1]['children'] + for field in ('producer_metric', 'classification', 'group'): + assert all(supervisor.cluster_meta.get(field, child) is None for child in children) + supervisor.undo() + assert supervisor.cluster_meta.get('group', 100) == 'good' + supervisor.redo() + assert all(supervisor.cluster_meta.get('group', child) is None for child in children) + supervisor.save() + finally: + controller.close() + + +def test_migration_keeps_explicit_target_quality_and_old_generation_reopens(tmp_path): + source, session = make_session(tmp_path) + # The original spelling cannot be inferred from the group header alone. + (source.path / 'cluster_group.tsv').rename(source.path / 'cluster_quality_archive.tsv') + session.record_merge([0, 1], 100, {'group': 'noise'}) + session.record_label('group', [100], 'good') + session.save() + pointer_path = session.path / '.phy-curation' / 'current.json' + legacy_pointer = pointer_path.read_bytes() + legacy_metadata = session.snapshot()['metadata'] + session.migrate_metadata() + assert session.metadata['group'][100] == 'good' + assert ( + session.effective_metadata_schema['tables'][0]['filename'] == 'cluster_quality_archive.tsv' + ) + session.save() + migrated_pointer = pointer_path.read_bytes() + pointer_path.write_bytes(legacy_pointer) + with pytest.warns(UserWarning, match='Legacy bounded metadata'): + old = BoundedSession(session.path) + assert old.metadata == legacy_metadata + assert old.metadata_policy is None + old.close() + pointer_path.write_bytes(migrated_pointer) + migrated = BoundedSession(session.path) + assert migrated.metadata['group'][100] == 'good' + assert migrated.metadata_policy == POLICY + migrated.close() + source.close() + session.close() + + +def test_v2_resolved_metadata_cannot_smuggle_parent_measurements(): + event = { + 'version': 2, + 'operation': 'merge', + 'parents': [0, 1], + 'target': 2, + 'metadata': {'group': 'good', 'snr': 12.26}, + } + with pytest.raises(ValueError, match='clear producer metadata'): + interpret_events([0, 1], {'snr': {0: 12.26, 1: 8}}, [event]) + + +def test_legacy_typed_python_table_representations(tmp_path): + metadata = {'flag': {'0': True}, 'details': {'0': ['a', 1]}} + (tmp_path / '_phy_cluster_metadata.json').write_text(json.dumps(metadata)) + (tmp_path / 'cluster_flag.tsv').write_text('cluster_id\tflag\n0\tTrue\n') + (tmp_path / 'cluster_details.tsv').write_text("cluster_id\tdetails\n0\t['a', 1]\n") + assert read_metadata(tmp_path)[0] == {'flag': {0: True}, 'details': {0: ['a', 1]}} + + +def test_merge_empty_catalog_unit_does_not_inherit_identity(tmp_path): + params = metadata_sorting(tmp_path) + session = prepared(params, tmp_path / 'bounded') + session.record_merge([0, 88], 100) + assert session.metadata['si_unit_id'][100] is None + session.save() + output = apply_curation(params, session.path, tmp_path / 'out') + with Source(output / 'params.py') as curated: + assert 88 not in curated.unit_ids + assert curated.metadata['si_unit_id'][100] is None + assert curated.counts[curated.unit_ids.tolist().index(100)] == 4 + session.close() + + +def test_replay_rejects_unsourced_manifest_metadata(tmp_path): + source, session = make_session(tmp_path) + session.close() + manifest_path = session.path / '_phy_curation_manifest.json' + manifest = json.loads(manifest_path.read_text()) + manifest['base_metadata']['group']['0'] = 'fabricated' + manifest_path.write_text(json.dumps(manifest)) + session = BoundedSession(session.path) + session.save() + with pytest.raises(ValueError, match='base metadata does not match'): + apply_curation(source.params_path, session.path, tmp_path / 'out') + assert not (tmp_path / 'out').exists() + source.close() + session.close() + + +def test_replay_rejects_changed_original_table_basename(tmp_path): + params = metadata_sorting(tmp_path) + session = prepared(params, tmp_path / 'bounded') + session.save() + (params.parent / 'cluster_si_unit_ids.tsv').rename(params.parent / 'cluster_si_unit_id.tsv') + with pytest.raises(ValueError, match='metadata schema changed'): + apply_curation(params, session.path, tmp_path / 'out') + session.close() + + +def test_readonly_source_symlink_preserves_table_alias_and_target(tmp_path): + params = make_sorting(tmp_path / 'source') + producer = tmp_path / 'immutable-producer-si.tsv' + producer.write_text('cluster_id\tsi_unit_id\n0\t40\n1\t44\n2\t48\n99\t99\n') + alias = params.parent / 'cluster_si_unit_ids.tsv' + _symlink_or_skip(alias, producer) + original_hash = hash_file(producer) + with Source(params) as source: + assert source.metadata['si_unit_id'][0] == '40' + descriptor = next( + table for table in source.metadata_schema['tables'] if 'si_unit_id' in table['columns'] + ) + assert descriptor['filename'] == 'cluster_si_unit_ids.tsv' + session = prepared(params, tmp_path / 'bounded') + session.save() + output = apply_curation(params, session.path, tmp_path / 'out') + assert (output / 'cluster_si_unit_ids.tsv').is_file() + assert not (output / 'cluster_si_unit_ids.tsv').is_symlink() + assert alias.is_symlink() + assert hash_file(producer) == original_hash + session.close() + + +def test_metadata_output_refuses_existing_symlink_without_mutating_target(tmp_path): + producer = tmp_path / 'producer.tsv' + producer.write_text('immutable producer data') + destination = tmp_path / 'out' + destination.mkdir() + _symlink_or_skip(destination / 'cluster_si_unit_ids.tsv', producer) + schema = { + 'version': 1, + 'tables': [ + { + 'filename': 'cluster_si_unit_ids.tsv', + 'delimiter': '\t', + 'columns': ['cluster_id', 'si_unit_id'], + } + ], + } + original_hash = hash_file(producer) + with pytest.raises(ValueError, match='output must not overwrite a symbolic link'): + write_metadata(destination, {'si_unit_id': {0: '40'}}, schema) + assert hash_file(producer) == original_hash + + +def _symlink_or_skip(path, target): + try: + path.symlink_to(target) + except OSError as error: + if getattr(error, 'winerror', None) == 1314: + pytest.skip('This Windows account has no symbolic-link privilege.') + raise + + +@pytest.mark.parametrize('value', [{'z': (1, 2), 'a': True}, [{'z': 2, 'a': True}]]) +def test_legacy_structured_label_key_order_and_tuple_json_equivalence(tmp_path, value): + metadata = {'details': {0: value}} + table = tmp_path / 'cluster_details.tsv' + with table.open('w', newline='') as stream: + writer = csv.writer(stream, delimiter='\t') + writer.writerow(['cluster_id', 'details']) + writer.writerow([0, value]) + sidecar = tmp_path / '_phy_cluster_metadata.json' + sidecar.write_text(json.dumps(metadata, sort_keys=True)) + actual, _ = read_metadata(tmp_path) + assert actual['details'][0] == json.loads(json.dumps(value)) + # A changed boolean is a real conflict, not legacy serialization variance. + table.write_text(table.read_text().replace('True', '1')) + with pytest.raises(ValueError, match='field details, unit 0'): + read_metadata(tmp_path) diff --git a/phy/apps/large_recording/tests/test_output_contract.py b/phy/apps/large_recording/tests/test_output_contract.py new file mode 100644 index 000000000..7cdf49e05 --- /dev/null +++ b/phy/apps/large_recording/tests/test_output_contract.py @@ -0,0 +1,774 @@ +"""Reject undeclared full-output changes, including internally consistent metadata edits.""" + +import csv +import hashlib +import json +import shutil +from types import SimpleNamespace + +import numpy as np +import pytest + +import tools.compare_curation_modes as comparison +from phy.apps.large_recording.curation import BoundedSession +from phy.apps.large_recording.metadata import SCHEMA_FILE, read_metadata, write_metadata +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import assignment_revision, hash_file +from phy.apps.large_recording.tests.test_source import make_sorting + + +def _inventory(path): + return { + str(item.relative_to(path)): hash_file(item) for item in path.rglob('*') if item.is_file() + } + + +def _pair(tmp_path, aliases=False, twin=False, split=False, unclassified=False, sparse=False): + params = make_sorting(tmp_path / 'source') + source = params.parent + assert sum((twin, split, unclassified)) <= 1 + if unclassified: + (source / 'cluster_group.tsv').unlink() + params.write_text(params.read_text() + "exporter_tag = 'fixture-v1'\n") + (source / 'cluster_snr.tsv').write_text( + 'cluster_id\tsnr\n0\t12\n1\t12\n2\t12\n99\t12\n' + if twin + else 'cluster_id\tsnr\n0\t12\n1\t13\n2\t22\n99\t3\n' + ) + (source / 'cluster_notes.tsv').write_text( + 'cluster_id\tnotes\n0\tfirst\n1\tsecond\n2\tcontrol\n99\tlast\n' + ) + (source / 'cluster_empty.tsv').write_text('cluster_id\tempty\n0\t\n1\t\n2\t\n99\t\n') + np.save(source / '_phy_spikes_subset.window_bounds.npy', np.array([[0, 20]], dtype=np.int64)) + (source / 'curation.json').write_text( + '{"format_version":"2","unit_ids":[0,1,2,99],"merges":[{"unit_ids":[0,1]}]}\n' + ) + if aliases: + (source / 'cluster_si_unit_ids.tsv').write_text( + 'cluster_id\tsi_unit_id\n0\t10\n1\t11\n2\t12\n99\t99\n' + ) + (source / 'cluster_import.csv').write_text( + 'producer,cluster_id,score\nfirst,0,1.5\nsecond,1,2.5\ncontrol,2,3.5\nlast,99,4.5\n' + ) + original = _inventory(source) + fields = list(read_metadata(source)[0]) + if 'group' not in fields: + fields.append('group') + definition = { + 'aliases': {'a': 0, 'b': 1, 'c': 2, 'd': 99}, + 'fields': fields, + 'actions': [ + {'op': 'merge', 'parents': ['a', 'b'], 'as': 'merged'}, + {'op': 'label', 'ids': ['merged'], 'field': 'group', 'value': 'good'}, + {'op': 'label', 'ids': ['merged'], 'field': 'notes', 'value': 'reviewed'}, + {'op': 'save'}, + ], + 'expected_labels': {'merged': {'group': 'good', 'notes': 'reviewed'}}, + } + if split: + definition['actions'] = [ + {'op': 'merge', 'parents': ['a', 'b'], 'as': 'merged'}, + { + 'op': 'split', + 'parent': 'merged', + 'threshold': 4.0, + 'selected': 'low', + 'remainder': 'rest', + }, + *[ + {'op': 'label', 'ids': [alias], 'field': field, 'value': value} + for alias in ('low', 'rest') + for field, value in (('group', 'good'), ('notes', 'reviewed')) + ], + {'op': 'save'}, + ] + definition['expected_labels'] = { + alias: {'group': 'good', 'notes': 'reviewed'} for alias in ('low', 'rest') + } + if unclassified: + definition['actions'] = [ + a for a in definition['actions'] if not (a['op'] == 'label' and a['field'] == 'group') + ] + definition['expected_labels'] = {'merged': {'notes': 'reviewed'}} + if twin: + definition['actions'][-1:-1] = [ + {'op': 'merge', 'parents': ['c', 'd'], 'as': 'twin'}, + {'op': 'label', 'ids': ['twin'], 'field': 'group', 'value': 'good'}, + {'op': 'label', 'ids': ['twin'], 'field': 'notes', 'value': 'reviewed'}, + ] + ordinary = tmp_path / 'ordinary-full' + shutil.copytree(source, ordinary) + reference = tmp_path / 'ordinary-reference' + comparison.run_reference(source, definition, reference) + for item in reference.iterdir(): + if item.name == 'spike_clusters.npy' or item.name.startswith('cluster_'): + shutil.copyfile(item, ordinary / item.name) + bounded = prepare_large( + params, + tmp_path / 'bounded', + window_seconds=2 if sparse else 20, + stratum_seconds=5 if sparse else 20, + max_spikes=6 if sparse else 100, + window_policy='generated' if sparse else 'auto', + bin_seconds=5, + batch_size=2, + ) + session = BoundedSession(bounded) + try: + merged = session.next_cluster_id + assert merged == 100 + session.record_merge([0, 1], merged) + targets = [merged] + if split: + session.record_threshold_split( + merged, + [102, 101], + { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 4.0, + 'nonfinite': 'keep', + }, + ) + targets = [101, 102] + for unit in targets: + if not unclassified: + session.record_label('group', [unit], 'good') + session.record_label('notes', [unit], 'reviewed') + if twin: + session.record_merge([2, 99], 101) + session.record_label('group', [101], 'good') + session.record_label('notes', [101], 'reviewed') + session.save() + finally: + session.close() + right = apply_curation(params, bounded, tmp_path / 'replay', batch_size=2) + return SimpleNamespace( + source=source, + ordinary=ordinary, + right=right, + bounded=bounded, + expected={ + unit: ( + {'notes': 'reviewed'} if unclassified else {'group': 'good', 'notes': 'reviewed'} + ) + for unit in ([101, 102] if split else [100, 101] if twin else [100]) + }, + original=original, + ) + + +def _compare(pair, bounded=True): + return comparison.compare_outputs( + pair.ordinary, + pair.right, + fields=('group', 'notes'), + chunk_size=2, + source=pair.source, + expected_labels=pair.expected, + bounded_dir=pair.bounded if bounded else None, + ) + + +@pytest.fixture +def contract_pair(tmp_path): + pair = _pair(tmp_path) + try: + yield pair + finally: + assert _inventory(pair.source) == pair.original + + +def _change_json(path, mutate): + content = json.loads(path.read_text()) + mutate(content) + path.write_text(json.dumps(content) + '\n') + + +def _refresh_revision(right): + metadata, _ = read_metadata(right) + _change_json( + right / '_phy_curation_lineage.json', + lambda document: document.update(revision=assignment_revision(right, metadata)), + ) + + +def _edit_metadata(right, field, unit, value): + metadata, schema = read_metadata(right) + metadata[field][unit] = value + write_metadata(right, metadata, schema) + # Keep the typed sidecar, every TSV and the lineage revision consistent. + # The checker must reject the metadata policy violation, not just a stale hash. + _refresh_revision(right) + + +def _journal_hash(journal): + payload = json.dumps(journal, sort_keys=True, separators=(',', ':'), allow_nan=False) + '\n' + return hashlib.sha256(payload.encode('utf-8')).hexdigest() + + +def _forge_split_threshold(pair, threshold, inherited=False): + """Change intent coherently, leaving assignments and curator expectations untouched.""" + journal_path = pair.right / '_phy_curation_journal.json' + lineage_path = pair.right / '_phy_curation_lineage.json' + journal = json.loads(journal_path.read_text()) + lineage = json.loads(lineage_path.read_text()) + lineage['metadata_provenance']['history'][1]['predicate']['threshold'] = threshold + if not inherited: + journal[1]['predicate']['threshold'] = threshold + lineage['journal_sha256'] = _journal_hash(journal) + journal_path.write_text(json.dumps(journal) + '\n') + manifest_path = pair.bounded / '_phy_curation_manifest.json' + manifest = json.loads(manifest_path.read_text()) + if inherited: + manifest['metadata_provenance']['history'][1]['predicate']['threshold'] = threshold + manifest_path.write_text(json.dumps(manifest) + '\n') + lineage['preparation_sha256'] = _journal_hash(manifest) + lineage_path.write_text(json.dumps(lineage) + '\n') + pointer_path = pair.bounded / '.phy-curation/current.json' + pointer = json.loads(pointer_path.read_text()) + generation = pair.bounded / '.phy-curation/generations' / pointer['generation'] + saved_journal = generation / 'journal.json' + saved_journal.write_text(json.dumps(journal) + '\n') + descriptor_path = generation / 'checkpoint.json' + descriptor = json.loads(descriptor_path.read_text()) + descriptor['files']['journal.json'] = hash_file(saved_journal) + descriptor['metadata_provenance'] = lineage['metadata_provenance'] + descriptor['manifest_sha256'] = _journal_hash(manifest) + descriptor_path.write_text(json.dumps(descriptor) + '\n') + pointer['sha256'] = _journal_hash(descriptor) + pointer_path.write_text(json.dumps(pointer) + '\n') + + +@pytest.mark.parametrize('bounded', [False, True]) +def test_coherent_threshold_forgery_rejected_by_source_assignments(tmp_path, bounded): + pair = _pair(tmp_path, split=True) + try: + assert _compare(pair, bounded=bounded)['ok'] + # All amplitudes are <1000; the unchanged output still has two children. + # Journal, cumulative provenance, saved generation and all hashes agree. + _forge_split_threshold(pair, 1000.0) + assert comparison.compare_outputs(pair.ordinary, pair.right)['ok'] + result = _compare(pair, bounded=bounded) + assert not result['ok'], result + assert 'cumulative source predicates' in result['error'], result + finally: + assert _inventory(pair.source) == pair.original + + +def test_threshold_oracle_checks_spikes_absent_from_bounded_evidence(tmp_path): + pair = _pair(tmp_path, split=True, sparse=True) + try: + assert _compare(pair)['ok'] + manifest = json.loads((pair.bounded / '_phy_curation_manifest.json').read_text()) + retained = np.load(pair.bounded / manifest['global_spike_ids_file']) + amplitudes = np.load(pair.source / 'amplitudes.npy') + parents = np.isin(np.load(pair.source / 'spike_clusters.npy'), [0, 1]) + changed = parents & ((amplitudes < 4.0) != (amplitudes < 5.0)) + assert np.flatnonzero(changed).tolist() == [4] + assert not changed[retained].any() + # Saved retained assignments still obey either threshold; only a check + # over all original-source rows can detect this contradiction. + _forge_split_threshold(pair, 5.0) + result = _compare(pair) + assert not result['ok'], result + assert 'cumulative source predicates' in result['error'], result + finally: + assert _inventory(pair.source) == pair.original + + +def test_inherited_split_semantics_checked_after_no_edit_round(tmp_path): + pair = _pair(tmp_path, split=True) + try: + pair.bounded = prepare_large( + pair.right / 'params.py', + tmp_path / 'bounded-next', + window_seconds=20, + stratum_seconds=20, + max_spikes=100, + bin_seconds=5, + batch_size=2, + ) + session = BoundedSession(pair.bounded) + try: + assert session.events == [] + session.save() + finally: + session.close() + pair.right = apply_curation( + pair.right / 'params.py', pair.bounded, tmp_path / 'replay-next', batch_size=2 + ) + assert _compare(pair)['ok'] + # There are no current membership events: checking just the latest + # journal would miss this forged threshold in an earlier round. + assert json.loads((pair.right / '_phy_curation_journal.json').read_text()) == [] + _forge_split_threshold(pair, 1000.0, inherited=True) + assert comparison.compare_outputs(pair.ordinary, pair.right)['ok'] + result = _compare(pair) + assert not result['ok'], result + assert 'cumulative source predicates' in result['error'], result + finally: + assert _inventory(pair.source) == pair.original + + +def test_curated_original_history_is_not_reapplied_to_source_assignments(tmp_path): + pair = _pair(tmp_path, split=True) + original = pair.source + try: + pair.source = pair.right + curated_source_inventory = _inventory(pair.source) + pair.ordinary = tmp_path / 'ordinary-curated-source' + shutil.copytree(pair.source, pair.ordinary) + pair.bounded = prepare_large( + pair.source / 'params.py', + tmp_path / 'bounded-from-curated-source', + window_seconds=20, + stratum_seconds=20, + max_spikes=100, + bin_seconds=5, + batch_size=2, + ) + session = BoundedSession(pair.bounded) + try: + assert session.events == [] + session.save() + finally: + session.close() + pair.right = apply_curation( + pair.source / 'params.py', + pair.bounded, + tmp_path / 'replay-from-curated-source', + batch_size=2, + ) + pair.expected = {} + result = _compare(pair) + assert result['ok'], result + oracle = result['file_contract']['assignment_oracle'] + assert oracle['membership_events'] == oracle['threshold_splits'] == 0 + assert oracle['source_rows'] == 12 + assert _inventory(pair.source) == curated_source_inventory + finally: + assert _inventory(original) == pair.original + + +@pytest.mark.parametrize('aliases', [False, True]) +def test_complete_file_contract_accepts_declared_metadata_differences(tmp_path, aliases): + pair = _pair(tmp_path, aliases=aliases) + try: + result = _compare(pair) + assert result['ok'], result + assert result['rows'] == 12 + metadata, _ = read_metadata(pair.right) + assert metadata['snr'][100] is None + assert metadata['snr'][2] == '22' + assert metadata['group'][100] == 'good' + if aliases: + for name in ('cluster_si_unit_ids.tsv', 'cluster_import.csv'): + assert (pair.ordinary / name).read_bytes() == (pair.source / name).read_bytes() + assert (pair.ordinary / 'cluster_si_unit_id.tsv').exists() + assert (pair.ordinary / 'cluster_producer.tsv').exists() + finally: + assert _inventory(pair.source) == pair.original + + +@pytest.mark.parametrize('name', ['cluster_si_unit_ids.tsv', 'cluster_import.csv']) +def test_ordinary_source_alias_allowance_requires_unchanged_bytes(tmp_path, name): + pair = _pair(tmp_path, aliases=True) + try: + assert _compare(pair)['ok'] + path = pair.ordinary / name + # Change only a retired parent's value in the loose producer alias. + # Active canonical fields and the curation outputs still agree. + if name.endswith('.tsv'): + path.write_text(path.read_text().replace('0\t10', '0\t777')) + else: + path.write_text(path.read_text().replace('first,0,1.5', 'forged,0,1.5')) + assert not _compare(pair)['ok'] + finally: + assert _inventory(pair.source) == pair.original + + +def test_first_round_source_binding_checks_journal_without_bounded_package(contract_pair): + pair = contract_pair + assert _compare(pair, bounded=False)['ok'] + path = pair.right / '_phy_curation_journal.json' + journal = json.loads(path.read_text()) + journal[-1]['value'] = 'forged label intent' + path.write_text(json.dumps(journal) + '\n') + _change_json( + pair.right / '_phy_curation_lineage.json', + lambda document: document.update(journal_sha256=_journal_hash(journal)), + ) + result = _compare(pair, bounded=False) + assert not result['ok'] + assert 'Journal details disagree with cumulative history' in result['error'] + + +def test_ordinary_derived_whitening_inverse_must_be_correct(contract_pair): + pair = contract_pair + assert not (pair.source / 'whitening_mat.npy').exists() + assert not (pair.source / 'whitening_mat_inv.npy').exists() + inverse = np.eye(3, dtype=np.float64) + path = pair.ordinary / 'whitening_mat_inv.npy' + np.save(path, inverse) + assert _compare(pair)['ok'] + inverse[0, 0] = 2 + np.save(path, inverse) + result = _compare(pair) + assert not result['ok'] + assert 'Invalid native derived whitening inverse' in result['error'] + + +@pytest.mark.parametrize( + 'corruption', + [ + 'amplitude', + 'waveform', + 'channels', + 'window_bounds', + 'assignment', + 'manual_label', + 'changed_membership_producer', + 'untouched_producer', + 'ordinary_untouched_producer', + 'clock', + 'exposure', + 'auxiliary_parameter', + 'new_parameter', + 'missing_asset', + 'unknown_file', + 'ordinary_unknown_file', + 'metadata_filename', + 'metadata_header', + 'metadata_schema', + 'journal', + 'parent_revision', + 'review_binding', + 'ordinary_missing_empty_table', + 'unrecorded_empty_field', + 'malformed_history', + 'parent_metadata_history', + ], +) +def test_complete_file_contract_rejects_output_corruption(contract_pair, corruption): + pair = contract_pair + right = pair.right + assert _compare(pair)['ok'] + assets = { + 'amplitude': ('amplitudes.npy', (0,)), + 'waveform': ('_phy_spikes_subset.waveforms.npy', (0, 0, 0)), + 'channels': ('channel_positions.npy', (0, 0)), + 'window_bounds': ('_phy_spikes_subset.window_bounds.npy', (0, 1)), + } + if corruption in assets: + name, index = assets[corruption] + values = np.load(right / name) + values[index] += 1 + np.save(right / name, values) + elif corruption == 'assignment': + values = np.load(right / 'spike_clusters.npy') + values[0] = 2 + np.save(right / 'spike_clusters.npy', values) + _refresh_revision(right) + elif corruption == 'manual_label': + _edit_metadata(right, 'group', 100, 'noise') + elif corruption == 'changed_membership_producer': + _edit_metadata(right, 'snr', 100, '999') + elif corruption == 'untouched_producer': + _edit_metadata(right, 'snr', 2, '999') + elif corruption == 'ordinary_untouched_producer': + path = pair.ordinary / 'cluster_snr.tsv' + path.write_text(path.read_text().replace('2\t22', '2\t999')) + elif corruption in ('clock', 'exposure', 'auxiliary_parameter', 'new_parameter'): + changes = { + 'clock': 'sample_rate = 2.0\n', + 'exposure': 'valid_intervals = [[0, 3], [5, 20]]\n', + 'auxiliary_parameter': "exporter_tag = 'changed'\n", + 'new_parameter': "unexplained_option = 'changed'\n", + } + path = right / 'params.py' + path.write_text(path.read_text() + changes[corruption]) + elif corruption == 'missing_asset': + (right / 'spike_templates.npy').unlink() + elif corruption == 'unknown_file': + (right / 'unexplained.txt').write_text('This is not an allowed generated output.\n') + elif corruption == 'ordinary_unknown_file': + (pair.ordinary / 'unexplained.txt').write_text('Not an ordinary GUI cache or table.\n') + elif corruption == 'metadata_filename': + (right / 'cluster_snr.tsv').rename(right / 'cluster_snrs.tsv') + + def rename(document): + for table in document['tables']: + if table['filename'] == 'cluster_snr.tsv': + table['filename'] = 'cluster_snrs.tsv' + + _change_json(right / SCHEMA_FILE, rename) + elif corruption == 'metadata_header': + path = right / 'cluster_snr.tsv' + with path.open(newline='') as stream: + rows = list(csv.reader(stream, delimiter='\t')) + with path.open('w', newline='') as stream: + csv.writer(stream, delimiter='\t').writerows([list(reversed(row)) for row in rows]) + + def reorder(document): + for table in document['tables']: + if table['filename'] == 'cluster_snr.tsv': + table['columns'].reverse() + + _change_json(right / SCHEMA_FILE, reorder) + elif corruption == 'metadata_schema': + _change_json(right / SCHEMA_FILE, lambda document: document.update(version=2)) + elif corruption == 'journal': + path = right / '_phy_curation_journal.json' + journal = json.loads(path.read_text()) + journal[0]['target'] = 101 + path.write_text(json.dumps(journal) + '\n') + _change_json( + right / '_phy_curation_lineage.json', + lambda document: document.update(journal_sha256=_journal_hash(journal)), + ) + elif corruption == 'parent_revision': + _change_json( + right / '_phy_curation_lineage.json', + lambda document: document.update(parent_revision='0' * 64), + ) + elif corruption == 'review_binding': + path = right / 'curation_review.json' + _change_json(path, lambda document: document['source'].update(sha256='0' * 64)) + _change_json( + right / '_phy_curation_lineage.json', + lambda document: document['proposition_documents'].update( + {'curation_review.json': hash_file(path)} + ), + ) + elif corruption == 'ordinary_missing_empty_table': + (pair.ordinary / 'cluster_empty.tsv').unlink() + elif corruption == 'unrecorded_empty_field': + metadata, schema = read_metadata(right) + metadata['invented'] = dict.fromkeys(metadata['group']) + write_metadata(right, metadata, schema) + _refresh_revision(right) + elif corruption == 'malformed_history': + _change_json( + right / '_phy_curation_lineage.json', + lambda document: document.update(metadata_provenance=[]), + ) + elif corruption == 'parent_metadata_history': + + def change_snapshot(document): + document['metadata_provenance']['history'][0]['parent_metadata']['snr']['0'] = '999' + + _change_json(right / '_phy_curation_lineage.json', change_snapshot) + else: + raise AssertionError(corruption) + result = _compare(pair) + assert not result['ok'], (corruption, result) + if corruption == 'journal': + assert 'Journal details disagree with cumulative history' in result['error'] + elif corruption == 'ordinary_missing_empty_table': + assert 'Missing native metadata field/table coverage' in result['error'] + elif corruption == 'unrecorded_empty_field': + assert 'Unexpected replay metadata fields without label intent' in result['error'] + elif corruption == 'parent_metadata_history': + assert 'Parent metadata provenance disagrees with independent history' in result['error'] + elif corruption == 'malformed_history': + assert result['error'] + + +def test_strict_contract_standalone_cli(contract_pair, tmp_path): + import subprocess + import sys + from pathlib import Path + + pair = contract_pair + repository = Path(__file__).resolve().parents[4] + expectations = tmp_path / 'expectations.json' + expectations.write_text(json.dumps(pair.expected)) + output = tmp_path / 'cli-report.json' + result = subprocess.run( + [ + sys.executable, + str(repository / 'tools/compare_curation_modes.py'), + 'compare', + str(pair.ordinary), + str(pair.right), + '--source', + str(pair.source), + '--bounded-dir', + str(pair.bounded), + '--expectations', + str(expectations), + '--output', + str(output), + '--chunk-size', + '2', + ], + cwd=repository, + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stdout + result.stderr + report = json.loads(output.read_text()) + assert report['ok'] and report['file_contract']['saved_generation_verified'] + + +def test_source_only_contract_checks_new_unit_row_ancestry(tmp_path): + pair = _pair(tmp_path, twin=True) + assert _compare(pair)['ok'] + values = np.load(pair.right / 'spike_clusters.npy') + swapped = np.where(values == 100, 101, 100).astype(values.dtype) + np.save(pair.right / 'spike_clusters.npy', swapped) + _refresh_revision(pair.right) + # Same partitions and identical labels allow this rename in the basic check; + # the stored provenance still names the WRONG source parents for those IDs. + assert comparison.compare_outputs(pair.ordinary, pair.right)['ok'] + result = comparison.compare_outputs( + pair.ordinary, + pair.right, + source=pair.source, + expected_labels=pair.expected, + chunk_size=2, + ) + assert not result['ok'] and 'Assignment disagrees' in result['error'] + assert _inventory(pair.source) == pair.original + + +@pytest.mark.parametrize('corruption', ['allocator', 'retirement']) +def test_saved_generation_and_retired_identity_binding(contract_pair, corruption): + pair = contract_pair + path = pair.right / '_phy_curation_lineage.json' + if corruption == 'allocator': + _change_json(path, lambda d: d.update(next_cluster_id=d['next_cluster_id'] + 1)) + else: + _change_json( + path, + lambda d: d['retired_units'].update( + {'999': {'operation': 'merge', 'replacements': [100]}} + ), + ) + result = _compare(pair) + assert not result['ok'] + assert ('saved checkpoint' if corruption == 'allocator' else 'independent history') in result[ + 'error' + ] + + +def test_generated_summary_requires_complete_active_catalog(contract_pair): + pair = contract_pair + path = pair.ordinary / 'cluster_info.tsv' + path.write_text( + 'cluster_id\tgroup\tnotes\n2\tgood\tcontrol\n99\tnoise\tlast\n100\tgood\treviewed\n' + ) + assert _compare(pair)['ok'] + path.write_text('cluster_id\tgroup\tnotes\n99\tnoise\tlast\n100\tgood\treviewed\n') + result = _compare(pair) + assert not result['ok'] and 'cluster_info catalog' in result['error'] + + +def test_native_blank_source_column_can_retain_retired_rows(contract_pair): + pair = contract_pair + path = pair.ordinary / 'cluster_empty.tsv' + shutil.copyfile(pair.source / path.name, path) + result = _compare(pair) + assert result['ok'], result + assert any( + a['policy'] == 'byte-pinned blank source column' + for a in result['file_contract']['ordinary_aliases'] + ) + path.write_text(path.read_text().replace('0\t\n', '0\tforged\n')) + assert not _compare(pair)['ok'] + + +def test_membership_can_introduce_implicit_unreviewed_group(tmp_path): + pair = _pair(tmp_path, unclassified=True) + result = _compare(pair) + assert result['ok'], result.get('error', result) + assert _inventory(pair.source) == pair.original + + +@pytest.mark.parametrize('corruption', ['metadata', 'extra_payload', 'single_parent']) +def test_source_only_rejects_invalid_membership_journal(contract_pair, corruption): + pair = contract_pair + journal_path = pair.right / '_phy_curation_journal.json' + lineage_path = pair.right / '_phy_curation_lineage.json' + journal = json.loads(journal_path.read_text()) + lineage = json.loads(lineage_path.read_text()) + if corruption == 'metadata': + journal[0]['metadata']['snr'] = 'forged producer metadata' + elif corruption == 'extra_payload': + journal[0]['unexplained'] = True + else: + journal[0]['parents'] = [0] + lineage['metadata_provenance']['history'][0]['parents'] = [0] + journal_path.write_text(json.dumps(journal) + '\n') + lineage['journal_sha256'] = _journal_hash(journal) + lineage_path.write_text(json.dumps(lineage) + '\n') + report = comparison.compare_outputs( + pair.ordinary, + pair.right, + source=pair.source, + expected_labels=pair.expected, + chunk_size=2, + ) + assert not report['ok'] + expected = { + 'metadata': 'clearing policy', + 'extra_payload': 'event schema', + 'single_parent': 'parent cardinality', + } + assert expected[corruption] in report['error'], report + + +@pytest.mark.parametrize( + 'key,value', + [ + ('operator', '=='), + ('attribute', 'spike_depths.npy'), + ('threshold', True), + ('nonfinite', 'drop'), + ], +) +def test_source_only_rejects_invalid_threshold_predicate(tmp_path, key, value): + pair = _pair(tmp_path, split=True) + assert _compare(pair)['ok'] + path = pair.right / '_phy_curation_journal.json' + journal = json.loads(path.read_text()) + journal[1]['predicate'][key] = value + path.write_text(json.dumps(journal) + '\n') + lineage_path = pair.right / '_phy_curation_lineage.json' + lineage = json.loads(lineage_path.read_text()) + lineage['journal_sha256'] = _journal_hash(journal) + lineage['metadata_provenance']['history'][1]['predicate'][key] = value + lineage_path.write_text(json.dumps(lineage) + '\n') + result = comparison.compare_outputs( + pair.ordinary, + pair.right, + source=pair.source, + expected_labels=pair.expected, + chunk_size=2, + ) + assert not result['ok'] and 'predicate schema' in result['error'], result + assert _inventory(pair.source) == pair.original + + +def test_native_transformed_producer_difference_is_ancestry_bounded(contract_pair): + pair = contract_pair + path = pair.ordinary / 'cluster_snr.tsv' + with path.open(newline='') as stream: + rows = list(csv.reader(stream, delimiter='\t')) + rows = [row for row in rows if row[0] != '100'] + rows.append(['100', '12']) # A real source parent's metric, not a new measurement. + with path.open('w', newline='') as stream: + csv.writer(stream, delimiter='\t').writerows(rows) + result = _compare(pair) + assert result['ok'], result + assert any( + d['field'] == 'snr' and d['right_id'] == 100 and d['left'] == '12' and d['right'] == '' + for d in result['permitted_membership_policy_differences'] + ) + rows[-1] = ['100', '999'] + with path.open('w', newline='') as stream: + csv.writer(stream, delimiter='\t').writerows(rows) + result = _compare(pair) + assert not result['ok'] and 'unsupported ancestor value' in result['error'] diff --git a/phy/apps/large_recording/tests/test_parameters.py b/phy/apps/large_recording/tests/test_parameters.py new file mode 100644 index 000000000..68db02648 --- /dev/null +++ b/phy/apps/large_recording/tests/test_parameters.py @@ -0,0 +1,145 @@ +"""Replay parameters retain Python semantics and stop accumulating overrides.""" + +import ast + +import pytest + +from phy.apps.large_recording.parameters import ( + BEGIN, + END, + rewrite_replay_params, + rewrite_replay_params_bytes, +) + +VALUES = {'num_samples': 20, 'valid_intervals': [[0, 8], [10, 20]], 'dat_path': None} + + +@pytest.mark.parametrize('ending', ['', '\n', '\r\n']) +def test_managed_parameters_are_idempotent_and_keep_user_source(ending): + original = '# User comment\nimport pathlib\nnum_samples = 7\n' + original += 'derived = num_samples * 2\nvalid_intervals = [[0, 7]]\n' + original += 'alias = dat_path = "relative.bin"' + ending + rewritten = rewrite_replay_params(original, **VALUES) + assert rewritten.startswith(original) + for _ in range(3): + assert rewrite_replay_params(rewritten, **VALUES) == rewritten + namespace = {} + exec(rewritten, namespace) + assert namespace['derived'] == 14 and namespace['alias'] == 'relative.bin' + assert {key: namespace[key] for key in VALUES} == VALUES + assert rewritten.count(BEGIN) == rewritten.count(END) == 1 + + +def test_legacy_literal_suffixes_collapse_without_deleting_originals(): + original = "num_samples = 20\nvalid_intervals = [[0, 8], [10, 20]]\ndat_path = r'None'\n" + legacy = original + ( + "\nnum_samples = 20\nvalid_intervals = [[0, 8], [10, 20]]\ndat_path = 'None'\n" * 3 + ) + values = dict(VALUES, dat_path='None') + rewritten = rewrite_replay_params(legacy, **values) + assert rewritten.startswith(original) + assert len(rewritten) < len(legacy) + assert rewrite_replay_params(rewritten, **VALUES) == rewritten + assert 'dat_path = None' in rewritten + assert ( + sum( + isinstance(node, ast.Assign) + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == 'num_samples' + for node in ast.parse(rewritten).body + ) + == 2 + ) + + +def test_nonliteral_chained_conditional_and_commented_assignments_remain(): + original = 'num_samples = 10 + 10\nvalid_intervals = [[0, 20]]\ndat_path = None\n' + original += 'if True:\n num_samples = 19\n' + original += "num_samples = valid_intervals = dat_path = 'keep'\n" + original += 'num_samples = 20 # user annotation\nvalid_intervals = [[0, 8], [10, 20]]\ndat_path = None\n' + assert rewrite_replay_params(original, **VALUES).startswith(original) + + +@pytest.mark.parametrize( + 'source', + [ + BEGIN + '\nnum_samples = 1\n', + END + '\n' + BEGIN + '\n', + BEGIN + '\nprint("keep")\n' + END + '\n', + BEGIN + + '\nnum_samples = 20\nvalid_intervals = []\ndat_path = None\n' + + END + + '\nprint("keep")\n', + ], +) +def test_malformed_or_modified_managed_blocks_fail_without_deleting_code(source): + with pytest.raises(ValueError): + rewrite_replay_params(source, **VALUES) + + +def test_marker_strings_and_quoted_paths_are_safe(): + source = f'documentation = """\n{BEGIN}\n{END}\n"""\n' + path = "a'quote\\folder/\u00e9lectrode.bin" + result = rewrite_replay_params(source, **dict(VALUES, dat_path=path)) + assert result.startswith(source) + namespace = {} + exec(result, namespace) + assert namespace['dat_path'] == path + + +def test_crlf_prefix_is_preserved_byte_for_byte(): + original = '# \u00e9lectrode\r\nnum_samples = 7\r\nderived = num_samples * 2\r\n' + result = rewrite_replay_params(original, **VALUES) + assert result.startswith(original) + assert '\n' not in result.replace('\r\n', '') + assert rewrite_replay_params(result, **VALUES) == result + + +@pytest.mark.parametrize('encoding', ['utf-8-sig', 'cp1252']) +def test_python_encoding_and_bom_survive_with_unicode_generated_paths(encoding): + text = f'# coding: {encoding}\r\n# \u00e9lectrode\r\nnum_samples = 7\r\n' + original = text.encode(encoding) + values = dict(VALUES, dat_path='\u00e9lectrode/\u4e2d\u6587.bin') + result = rewrite_replay_params_bytes(original, **values) + assert result.startswith(original) + assert rewrite_replay_params_bytes(result, **values) == result + namespace = {} + exec(result, namespace) + assert namespace['dat_path'] == values['dat_path'] + + +def test_three_replay_rounds_preserve_crlf_and_stable_file_size(tmp_path): + from phy.apps.large_recording.curation import BoundedSession + from phy.apps.large_recording.prepare import prepare_large + from phy.apps.large_recording.replay import apply_curation + from phy.apps.large_recording.tests.test_curation import make_session + + source, session = make_session(tmp_path) + params = source.params_path + original = params.read_text().replace('\n', '\r\n').encode('utf8') + params.write_bytes(original) + source.close() + session.save() + sizes = [] + for index in range(3): + output = apply_curation(params, session.path, tmp_path / f'output-{index}') + session.close() + params = output / 'params.py' + content = params.read_bytes() + assert content.startswith(original) + assert content.count(BEGIN.encode()) == 1 + assert b'\n' not in content.replace(b'\r\n', b'') + sizes.append(len(content)) + if index < 2: + bounded = prepare_large( + params, + tmp_path / f'bounded-{index}', + window_seconds=20, + stratum_seconds=20, + max_spikes=100, + bin_seconds=5, + batch_size=2, + ) + session = BoundedSession(bounded) + session.save() + assert len(set(sizes)) == 1 diff --git a/phy/apps/large_recording/tests/test_prepare.py b/phy/apps/large_recording/tests/test_prepare.py new file mode 100644 index 000000000..51c15d46a --- /dev/null +++ b/phy/apps/large_recording/tests/test_prepare.py @@ -0,0 +1,228 @@ +"""Production bounded preparation: budgets, rows, waveforms, and publication.""" + +import json + +import numpy as np +import pytest +from phylib.io.model import load_model + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.source import PER_SPIKE, hash_file +from phy.apps.large_recording.tests.test_source import make_sorting + + +@pytest.mark.parametrize('channel_map', [[0, 1, 2], [2, 0, 1]]) +def test_prepare_exact_arrays_waveforms_and_source_unchanged(tmp_path, channel_map): + params = make_sorting(tmp_path / 'full', columns=True) + np.save(params.parent / 'channel_map.npy', np.asarray(channel_map, dtype=np.int32)) + before = {p.name: hash_file(p) for p in params.parent.iterdir()} + output = prepare_large( + params, + tmp_path / 'bounded', + max_spikes=8, + window_seconds=3, + stratum_seconds=20, + seed=0, + bin_seconds=5, + batch_size=2, + ) + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + windows = np.load(output / manifest['windows_file']) + samples = np.load(params.parent / 'spike_times.npy').reshape(-1) + expected_ids = np.array( + [ + index + for index, sample in enumerate(samples) + if any(start <= sample < end for start, end in windows) + ], + dtype=np.int64, + ) + global_ids = np.load(output / manifest['global_spike_ids_file']) + np.testing.assert_array_equal(global_ids, expected_ids) + assert len(global_ids) <= 8 + for name in PER_SPIKE: + if (params.parent / name).exists(): + np.testing.assert_array_equal( + np.load(output / name), np.load(params.parent / name)[expected_ids] + ) + source_waveform_ids = np.load(params.parent / '_phy_spikes_subset.spikes.npy') + expected_waveform_rows = [ + row for row, spike in enumerate(source_waveform_ids) if spike in expected_ids + ] + np.testing.assert_array_equal( + global_ids[np.load(output / '_phy_spikes_subset.spikes.npy')], + source_waveform_ids[expected_waveform_rows], + ) + for suffix in ('channels', 'waveforms'): + np.testing.assert_array_equal( + np.load(output / f'_phy_spikes_subset.{suffix}.npy'), + np.load(params.parent / f'_phy_spikes_subset.{suffix}.npy')[expected_waveform_rows], + ) + assert before == {p.name: hash_file(p) for p in params.parent.iterdir()} + assert 'dat_path = None' in (output / 'params.py').read_text() + assert manifest['next_cluster_id'] == 100 + assert sum(manifest['counts']) == len(samples) + assert sum(manifest['bounded_counts']) == len(global_ids) + model = load_model(output / 'params.py') + try: + ids = np.load(output / '_phy_spikes_subset.spikes.npy') + channels = np.load(output / '_phy_spikes_subset.channels.npy') + sparse = np.load(output / '_phy_spikes_subset.waveforms.npy') + expected = np.zeros((len(ids), 8, 3), dtype=np.float32) + for row, slots in enumerate(channels): + for slot, channel in enumerate(slots): + if channel >= 0: + expected[row, :, channel] = sparse[row, :, slot] + np.testing.assert_array_equal(model.get_waveforms(ids, channel_ids=[0, 1, 2]), expected) + finally: + model.close() + + +def test_deterministic_budgets_and_zero_evidence_catalog(tmp_path): + params = make_sorting(tmp_path / 'full') + outputs = [ + prepare_large( + params, + tmp_path / name, + max_spikes=5, + window_seconds=3, + stratum_seconds=20, + seed=0, + bin_seconds=5, + batch_size=2, + ) + for name in ('one', 'two') + ] + manifests = [ + json.loads((path / '_phy_curation_manifest.json').read_text()) for path in outputs + ] + assert manifests[0] == manifests[1] + assert manifests[0]['unit_ids'][-1] == 99 + assert manifests[0]['bounded_counts'][-1] == 0 + assert manifests[0]['omitted_windows'] + for path in outputs: + ids = np.load(path / '_phy_spike_global_ids.npy') + times = np.load(path / 'spike_times.npy') + assert len(ids) <= 5 + # All three tied samples at 1 retain their separate source identities. + assert np.count_nonzero(times == 1) == 2 + + +def test_rejected_budget_or_interrupted_publication_leaves_no_output(tmp_path, monkeypatch): + params = make_sorting(tmp_path / 'full') + with pytest.raises(ValueError, match='No bounded evidence'): + prepare_large( + params, tmp_path / 'empty', max_spikes=1, window_seconds=3, stratum_seconds=20, seed=0 + ) + with pytest.raises(ValueError, match='Output budget'): + prepare_large(params, tmp_path / 'too-big', max_bytes=10) + import phy.apps.large_recording.prepare as module + + def fail(*args, **kwargs): + raise OSError('simulated interruption') + + monkeypatch.setattr(module, 'build_summaries', fail) + with pytest.raises(OSError, match='interruption'): + prepare_large(params, tmp_path / 'interrupted', window_seconds=20, stratum_seconds=20) + assert not (tmp_path / 'interrupted').exists() + assert not list(tmp_path.glob('.interrupted.prepare-*')) + + +def test_prepare_missing_waveforms_is_explicit(tmp_path): + params = make_sorting(tmp_path / 'full') + for path in params.parent.glob('_phy_spikes_subset.*.npy'): + path.unlink() + output = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + assert not manifest['capabilities']['stored_waveforms'] + assert not (output / '_phy_spikes_subset.waveforms.npy').exists() + + +def test_spikeinterface_singular_sparse_template_indices(tmp_path): + params = make_sorting(tmp_path / 'full') + templates = np.load(params.parent / 'templates.npy')[:, :, :2] + templates[1, :, 1] = 0 + np.save(params.parent / 'templates.npy', templates) + indices = np.array([[0, 2], [1, -1], [2, 0]], dtype=np.int32) + np.save(params.parent / 'template_ind.npy', indices) + output = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + model = load_model(output / 'params.py') + try: + np.testing.assert_array_equal(model.sparse_templates.cols, indices) + finally: + model.close() + + +def test_adjacent_acquisition_blocks_retain_observation_boundaries(tmp_path): + params = make_sorting(tmp_path / 'full') + output = prepare_large( + params, + tmp_path / 'bounded', + valid_intervals=[[0, 10], [10, 20]], + window_seconds=10, + stratum_seconds=10, + ) + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + np.testing.assert_array_equal(np.load(output / manifest['windows_file']), [[0, 10], [10, 20]]) + assert manifest['windows'][0]['block'] != manifest['windows'][1]['block'] + + +def test_disk_preflight_rejects_before_writing_package(tmp_path, monkeypatch): + from types import SimpleNamespace + + from phy.apps.large_recording import prepare + + params = make_sorting(tmp_path / 'full') + monkeypatch.setattr(prepare.shutil, 'disk_usage', lambda path: SimpleNamespace(free=0)) + with pytest.raises(OSError, match='Insufficient disk'): + prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + assert not (tmp_path / 'bounded').exists() + assert not list(tmp_path.glob('.bounded.prepare-*')) + + +def test_final_windows_records_match_retained_samples_and_gaps(tmp_path): + params = make_sorting(tmp_path / 'full') + output = prepare_large( + params, + tmp_path / 'bounded', + valid_intervals=[[0, 7], [10, 20]], + window_seconds=5, + stratum_seconds=20, + bin_seconds=5, + ) + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + intervals = np.load(output / manifest['windows_file']) + assert [[row['start'], row['end']] for row in manifest['windows']] == intervals.tolist() + for row in manifest['windows']: + start, end = row['start'], row['end'] + assert any(first <= start and end <= last for first, last in [[0, 7], [10, 20]]) + assert row['exposure_seconds'] == end - start + assert manifest['capabilities']['predicate_split_attributes'] == ['amplitudes.npy'] + + +def test_prepare_protects_loader_and_preserves_producer_documents(tmp_path): + params = make_sorting(tmp_path / 'full') + (params.parent / 'curation.json').write_text( + json.dumps({'format_version': '2', 'unit_ids': [0, 1], 'merges': [{'unit_ids': [0, 1]}]}) + ) + (params.parent / 'curation_review.json').write_text( + json.dumps( + { + 'format_version': '1', + 'source': { + 'filename': 'curation.json', + 'sha256': hash_file(params.parent / 'curation.json'), + }, + 'reviews': {}, + } + ) + ) + output = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + manifest = json.loads((output / '_phy_curation_manifest.json').read_text()) + assert manifest['bounded_files']['params.py'] == hash_file(output / 'params.py') + assert manifest['bounded_files']['_phy_cluster_metadata.json'] == hash_file( + output / '_phy_cluster_metadata.json' + ) + for name in ('curation.json', 'curation_review.json'): + assert (output / name).read_bytes() == (params.parent / name).read_bytes() + assert name not in manifest['excluded_files'] diff --git a/phy/apps/large_recording/tests/test_producer_windows.py b/phy/apps/large_recording/tests/test_producer_windows.py new file mode 100644 index 000000000..9c071d5e1 --- /dev/null +++ b/phy/apps/large_recording/tests/test_producer_windows.py @@ -0,0 +1,177 @@ +"""Producer plans retain complete windows and preserve sparse waveform row alignment.""" + +import json + +import numpy as np +import pytest + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.tests.test_source import make_sorting + +STEM = '_phy_spikes_subset.' + + +def producer_sorting(path): + params = make_sorting(path) + bounds = np.array([[0, 3], [5, 7], [10, 12], [14, 15], [18, 20]]) + np.save(path / (STEM + 'window_bounds.npy'), bounds) + np.save(path / (STEM + 'window_types.npy'), np.zeros(5, dtype=np.uint8)) + np.save(path / (STEM + 'window_strata.npy'), np.arange(5, dtype=np.int32)) + return params + + +def read_manifest(path): + return json.loads((path / '_phy_curation_manifest.json').read_text()) + + +def test_auto_prefers_producer_and_exact_waveform_remapping(tmp_path): + params = producer_sorting(tmp_path / 'source') + output = prepare_large(params, tmp_path / 'bounded', window_policy='producer', batch_size=2) + manifest = read_manifest(output) + assert manifest['selection']['resolved_window_policy'] == 'producer' + assert manifest['waveform_coverage'] == { + 'source_rows': 5, + 'retained_rows': 5, + 'dropped_rows': 0, + } + np.testing.assert_array_equal( + np.load(output / manifest['windows_file']), + np.load(params.parent / (STEM + 'window_bounds.npy')), + ) + ids = np.load(output / manifest['global_spike_ids_file']) + upstream_ids = np.load(params.parent / (STEM + 'spikes.npy')) + np.testing.assert_array_equal(ids[np.load(output / (STEM + 'spikes.npy'))], upstream_ids) + for suffix in ('waveforms.npy', 'channels.npy'): + np.testing.assert_array_equal( + np.load(output / (STEM + suffix)), np.load(params.parent / (STEM + suffix)) + ) + assert manifest['producer_window_plan']['sha256'] + assert manifest['producer_window_estimate']['spikes'] == 12 + auto = prepare_large(params, tmp_path / 'auto') + assert read_manifest(auto)['selection']['resolved_window_policy'] == 'producer' + + +def test_budget_distributes_complete_windows_across_recording(tmp_path): + params = producer_sorting(tmp_path / 'source') + output = prepare_large(params, tmp_path / 'bounded', window_policy='producer', max_spikes=7) + manifest = read_manifest(output) + # Dyadic traversal takes the first and last windows before exhausting budget. + assert manifest['bounded_num_spikes'] == 7 + assert [(r['start'], r['end']) for r in manifest['accepted_window_candidates']] == [ + (0, 3), + (18, 20), + ] + assert manifest['omitted_windows'] + assert all(r['reason'] == 'spike_budget' for r in manifest['omitted_windows']) + assert manifest['waveform_coverage']['dropped_rows'] > 0 + + +def test_generated_override_and_missing_producer(tmp_path): + params = producer_sorting(tmp_path / 'source') + output = prepare_large( + params, + tmp_path / 'bounded', + window_policy='generated', + window_seconds=20, + stratum_seconds=20, + ) + assert read_manifest(output)['selection']['resolved_window_policy'] == 'generated' + params = make_sorting(tmp_path / 'legacy') + with pytest.raises(ValueError, match='requires nonempty'): + prepare_large(params, tmp_path / 'missing', window_policy='producer') + with pytest.raises(ValueError, match='window_policy'): + prepare_large(params, tmp_path / 'bad', window_policy='invalid') + + +@pytest.mark.parametrize( + 'name,value,match', + [ + ('window_bounds.npy', np.array([0, 3]), 'integer'), + ('window_bounds.npy', np.array([[0.0, 3.0]]), 'integer'), + ('window_bounds.npy', np.array([[3, 2]]), 'bounds'), + ('window_bounds.npy', np.array([[0, 21]]), 'bounds'), + ('window_bounds.npy', np.array([[5, 6], [0, 3]]), 'sorted'), + ('window_types.npy', np.zeros(4, dtype=np.uint8), 'aligned'), + ('window_types.npy', np.full(5, 2, dtype=np.uint8), 'codes'), + ('window_strata.npy', np.full(5, -2), 'strata'), + ], +) +def test_invalid_sidecars_rejected_even_for_generated(tmp_path, name, value, match): + params = producer_sorting(tmp_path / 'source') + np.save(params.parent / (STEM + name), value) + if name == 'window_bounds.npy': + for suffix in ('window_types.npy', 'window_strata.npy'): + (params.parent / (STEM + suffix)).unlink() + with pytest.raises(ValueError, match=match): + prepare_large(params, tmp_path / 'bounded', window_policy='generated') + + +@pytest.mark.parametrize('intervals', [[[0, 10], [10, 20]], [[0, 9], [10, 20]]]) +def test_producer_window_cannot_cross_block_or_gap(tmp_path, intervals): + params = producer_sorting(tmp_path / 'source') + with params.open('a') as stream: + stream.write(f'valid_intervals = {intervals!r}\n') + np.save(params.parent / (STEM + 'window_bounds.npy'), np.array([[0, 11]])) + for suffix in ('window_types.npy', 'window_strata.npy'): + (params.parent / (STEM + suffix)).unlink() + with pytest.raises(ValueError, match='block|gap'): + prepare_large(params, tmp_path / 'bounded') + + +def test_overlaps_union_without_duplicate_spikes(tmp_path): + params = producer_sorting(tmp_path / 'source') + np.save(params.parent / (STEM + 'window_bounds.npy'), np.array([[0, 12], [10, 20]])) + for suffix in ('window_types.npy', 'window_strata.npy'): + (params.parent / (STEM + suffix)).unlink() + output = prepare_large(params, tmp_path / 'bounded', window_policy='producer') + manifest = read_manifest(output) + assert manifest['bounded_num_spikes'] == 12 + np.testing.assert_array_equal(np.load(output / manifest['windows_file']), [[0, 20]]) + + +@pytest.mark.parametrize('policy', ['auto', 'producer', 'generated']) +def test_waveforms_outside_producer_windows_rejected(tmp_path, policy): + params = producer_sorting(tmp_path / 'source') + np.save(params.parent / (STEM + 'window_bounds.npy'), np.array([[0, 3]])) + for suffix in ('window_types.npy', 'window_strata.npy'): + (params.parent / (STEM + suffix)).unlink() + with pytest.raises(ValueError, match='outside'): + prepare_large(params, tmp_path / 'bounded', window_policy=policy) + + +def test_legacy_acquisition_coordinate_declaration(tmp_path): + params = producer_sorting(tmp_path / 'source') + (params.parent / (STEM + 'manifest.json')).write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': {'window_bounds_coordinates': 'acquisition_samples'}, + } + ) + ) + output = prepare_large(params, tmp_path / 'bounded', window_policy='producer') + assert read_manifest(output)['waveform_coverage']['retained_rows'] == 5 + + +def test_wrong_window_coordinates_rejected_for_generated(tmp_path): + params = producer_sorting(tmp_path / 'source') + (params.parent / (STEM + 'manifest.json')).write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': {'window_bounds_coordinates': 'elapsed_ticks'}, + } + ) + ) + with pytest.raises(ValueError, match='coordinates|time_base'): + prepare_large(params, tmp_path / 'bounded', window_policy='generated') + + +def test_producer_policy_requires_waveform_triplet(tmp_path): + params = producer_sorting(tmp_path / 'source') + for suffix in ('spikes.npy', 'channels.npy', 'waveforms.npy'): + (params.parent / (STEM + suffix)).unlink() + with pytest.raises(ValueError, match='requires nonempty'): + prepare_large(params, tmp_path / 'bounded', window_policy='producer') diff --git a/phy/apps/large_recording/tests/test_proposition_persistence.py b/phy/apps/large_recording/tests/test_proposition_persistence.py new file mode 100644 index 000000000..2df49f444 --- /dev/null +++ b/phy/apps/large_recording/tests/test_proposition_persistence.py @@ -0,0 +1,330 @@ +"""Producer bytes, durable decisions and retirement survive bounded replay rounds.""" + +import json +from pathlib import Path + +import pytest + +from phy.apps.large_recording.adapter import BoundedTemplateController +from phy.apps.large_recording.curation import BoundedSession, _digest +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import Source, hash_file +from phy.apps.large_recording.tests.test_adapter import make_gui_sorting +from phy.apps.template.gui import TemplateController +from phy.cluster._propositions import PropositionReview, encode_review_mapping, proposition_key + + +class PropositionsDisabledController(BoundedTemplateController): + enable_merge_propositions = False + + +def prepare(tmp_path): + params = make_gui_sorting(tmp_path / 'source') + producer = b'{"format_version":"2", "unit_ids":[0,1,2,99], "merges":[{"unit_ids":[0,1]},{"unit_ids":[1,2]}], "producer": {"unchanged":true}}\n' + (params.parent / 'curation.json').write_bytes(producer) + review = encode_review_mapping( + { + proposition_key([1, 2]): PropositionReview('rejected'), + proposition_key([77, 78]): PropositionReview('accepted', [77, 78], 90), + }, + source_sha256=hash_file(params.parent / 'curation.json'), + ) + (params.parent / 'curation_review.json').write_text(json.dumps(review)) + bounded = prepare_large(params, tmp_path / 'bounded', window_seconds=20, stratum_seconds=20) + return params, bounded, producer, review + + +@pytest.mark.parametrize('mode', ['direct', 'enabled', 'disabled']) +def test_three_noop_rounds_preserve_exact_source_and_complete_ledger(tmp_path, mode): + params, bounded, producer, review = prepare(tmp_path) + original_reviews = review['reviews'] + for round_index in range(3): + if mode == 'direct': + session = BoundedSession(bounded) + session.save() + session.close() + else: + controller_class = ( + PropositionsDisabledController if mode == 'disabled' else TemplateController + ) + controller = controller_class( + dir_path=bounded, config_dir=tmp_path / 'config', plugins=[] + ) + try: + if mode == 'disabled': + assert controller.supervisor.merge_propositions is None + else: + assert controller.supervisor.merge_propositions is not None + controller.supervisor.save() + finally: + controller.close() + # A committed generation outranks every loose-root review state. + root_review = bounded / 'curation_review.json' + if round_index == 0: + root_review.write_text('{malformed') + elif round_index == 1: + root_review.write_text(json.dumps(dict(review, reviews={}))) + else: + root_review.unlink(missing_ok=True) + reopened = TemplateController(dir_path=bounded, config_dir=tmp_path / 'config', plugins=[]) + try: + assert ( + reopened.supervisor.merge_propositions.catalog.review_mapping()['reviews'] + == original_reviews + ) + finally: + reopened.close() + output = apply_curation(params, bounded, tmp_path / f'curated-{round_index}') + assert (output / 'curation.json').read_bytes() == producer + assert ( + json.loads((output / 'curation_review.json').read_text())['reviews'] + == original_reviews + ) + lineage = json.loads((output / '_phy_curation_lineage.json').read_text()) + assert lineage['proposition_documents'] == { + name: hash_file(output / name) for name in ('curation.json', 'curation_review.json') + } + params = output / 'params.py' + bounded = prepare_large( + params, tmp_path / f'bounded-{round_index}', window_seconds=20, stratum_seconds=20 + ) + + +def test_accepted_and_rejected_decisions_and_retirements_survive_replay(tmp_path): + params, bounded, producer, review = prepare(tmp_path) + controller = TemplateController(dir_path=bounded, config_dir=tmp_path / 'config', plugins=[]) + try: + supervisor = controller.supervisor + key = proposition_key([0, 1]) + supervisor.merge_propositions.accept(key, [0, 1], 100) + supervisor.merge([0, 1], to=100) + supervisor.save() + expected_reviews = supervisor.merge_propositions.catalog.review_mapping()['reviews'] + finally: + controller.close() + output = apply_curation(params, bounded, tmp_path / 'curated') + assert (output / 'curation.json').read_bytes() == producer + assert json.loads((output / 'curation_review.json').read_text())['reviews'] == expected_reviews + source = Source(output / 'params.py') + try: + assert source.retired_units == { + '0': {'operation': 'merge', 'replacements': [100]}, + '1': {'operation': 'merge', 'replacements': [100]}, + } + finally: + source.close() + second = prepare_large( + output / 'params.py', tmp_path / 'second', window_seconds=20, stratum_seconds=20 + ) + session = BoundedSession(second) + session.record_split( + 100, + 101, + 102, + {'attribute': 'amplitudes.npy', 'operator': '<', 'threshold': 4.0, 'nonfinite': 'keep'}, + ) + session.save() + session.close() + final = apply_curation(output / 'params.py', second, tmp_path / 'final') + lineage = json.loads((final / '_phy_curation_lineage.json').read_text()) + assert lineage['retired_units']['100'] == { + 'operation': 'threshold_split', + 'replacements': [102, 101], + } + assert lineage['retired_units']['0'] == {'operation': 'merge', 'replacements': [100]} + + +@pytest.mark.parametrize('target', ['curation.json', '_phy_curation_review_base.json']) +def test_mutation_after_load_rejects_save_and_replay(tmp_path, target): + params, bounded, _, _ = prepare(tmp_path) + session = BoundedSession(bounded) + session.save() + (bounded / target).write_text('{}') + try: + with pytest.raises(ValueError, match='changed'): + session.save() + with pytest.raises(ValueError, match='integrity'): + apply_curation(params, bounded, tmp_path / 'bad') + assert not (tmp_path / 'bad').exists() + finally: + session.close() + + +def test_generation_extra_document_requires_hash_inventory(tmp_path): + _, bounded, _, _ = prepare(tmp_path) + session = BoundedSession(bounded) + generation = session.save() + session.close() + descriptor = json.loads((generation / 'checkpoint.json').read_text()) + descriptor['files'].pop('curation_review.json') + (generation / 'checkpoint.json').write_text(json.dumps(descriptor)) + pointer = bounded / '.phy-curation/current.json' + data = json.loads(pointer.read_text()) + data['sha256'] = _digest(descriptor) + pointer.write_text(json.dumps(data)) + with pytest.raises(ValueError, match='hash inventory'): + BoundedSession(bounded) + + +@pytest.mark.parametrize('operation', ['prepare', 'replay']) +def test_copy_race_rejects_producer_mutation_without_publication(tmp_path, monkeypatch, operation): + from phy.apps.large_recording import prepare as preparation + from phy.apps.large_recording import replay + + params, bounded, _, _ = prepare(tmp_path) + session = BoundedSession(bounded) + session.save() + session.close() + module = preparation if operation == 'prepare' else replay + original_copy = module.shutil.copyfile + + def mutate_copy(source, destination, *args, **kwargs): + result = original_copy(source, destination, *args, **kwargs) + if Path(destination).name == 'curation.json': + Path(destination).write_text('{}') + return result + + monkeypatch.setattr(module.shutil, 'copyfile', mutate_copy) + output = tmp_path / 'bad' + with pytest.raises(ValueError, match='changed'): + if operation == 'prepare': + prepare_large(params, output, window_seconds=20, stratum_seconds=20) + else: + apply_curation(params, bounded, output) + assert not output.exists() + + +@pytest.mark.parametrize( + 'filename, digest', + [ + ('../curation.json', 'valid'), + ('other.json', 'valid'), + ('curation.json', None), + ('curation.json', '0' * 64), + ('curation.json', 'xyz'), + ], +) +def test_bounded_inherited_review_requires_exact_filename_and_hash(tmp_path, filename, digest): + params, _, _, review = prepare(tmp_path) + review['source']['filename'] = filename + if digest is None: + review['source'].pop('sha256') + elif digest != 'valid': + review['source']['sha256'] = digest + (params.parent / 'curation_review.json').write_text(json.dumps(review)) + with pytest.raises(ValueError, match='filename/hash'): + prepare_large(params, tmp_path / 'bad', window_seconds=20, stratum_seconds=20) + assert not (tmp_path / 'bad').exists() + + +def test_gui_save_rejects_producer_mutation_since_load(tmp_path): + _, bounded, _, _ = prepare(tmp_path) + controller = TemplateController(dir_path=bounded, config_dir=tmp_path / 'config', plugins=[]) + try: + (bounded / 'curation.json').write_text('{}') + with pytest.raises(ValueError, match='changed'): + controller.supervisor.save() + assert not (bounded / '.phy-curation/current.json').exists() + finally: + controller.close() + + +def test_replay_rejects_source_producer_mutation(tmp_path): + params, bounded, _, _ = prepare(tmp_path) + session = BoundedSession(bounded) + session.save() + session.close() + (params.parent / 'curation.json').write_text('{}') + with pytest.raises(ValueError, match='changed since preparation'): + apply_curation(params, bounded, tmp_path / 'bad') + assert not (tmp_path / 'bad').exists() + + +def test_legacy_bounded_propositions_require_repreparation(tmp_path): + _, bounded, _, _ = prepare(tmp_path) + path = bounded / '_phy_curation_manifest.json' + manifest = json.loads(path.read_text()) + manifest.pop('proposition_documents') + path.write_text(json.dumps(manifest)) + with pytest.raises(ValueError, match='reprepare'): + BoundedSession(bounded) + + +def test_legacy_checkpoint_review_without_loose_root_requires_repreparation(tmp_path): + from phy.apps.large_recording.curation import _hash_file + from phy.apps.large_recording.tests.test_curation import make_session + + _, session = make_session(tmp_path) + generation = session.save(extra_documents={'supplemental.json': {}}) + session.close() + (generation / 'supplemental.json').rename(generation / 'curation_review.json') + descriptor_path = generation / 'checkpoint.json' + descriptor = json.loads(descriptor_path.read_text()) + descriptor['extra_documents'] = ['curation_review.json'] + descriptor['files'].pop('supplemental.json') + descriptor['files']['curation_review.json'] = _hash_file(generation / 'curation_review.json') + descriptor_path.write_text(json.dumps(descriptor)) + pointer_path = session.path / '.phy-curation/current.json' + pointer = json.loads(pointer_path.read_text()) + pointer['sha256'] = _digest(descriptor) + pointer_path.write_text(json.dumps(pointer)) + with pytest.raises(ValueError, match='reprepare'): + BoundedSession(session.path) + + +@pytest.mark.parametrize('mode', ['direct', 'disabled']) +def test_suggestions_without_reviews_publish_empty_bound_ledger(tmp_path, mode): + params, _, producer, _ = prepare(tmp_path) + (params.parent / 'curation_review.json').unlink() + bounded = prepare_large(params, tmp_path / 'empty', window_seconds=20, stratum_seconds=20) + if mode == 'direct': + session = BoundedSession(bounded) + session.save() + session.close() + else: + controller = PropositionsDisabledController( + dir_path=bounded, config_dir=tmp_path / 'config', plugins=[] + ) + try: + assert controller.supervisor.merge_propositions is None + controller.supervisor.save() + finally: + controller.close() + output = apply_curation(params, bounded, tmp_path / 'curated') + assert (output / 'curation.json').read_bytes() == producer + assert json.loads((output / 'curation_review.json').read_text()) == { + 'format_version': '1', + 'source': {'filename': 'curation.json', 'sha256': hash_file(output / 'curation.json')}, + 'reviews': {}, + } + + +def test_null_review_cannot_remove_inherited_decisions(tmp_path): + _, bounded, _, review = prepare(tmp_path) + session = BoundedSession(bounded) + try: + with pytest.raises(ValueError, match='ledger mapping'): + session.save(extra_documents={'curation_review.json': None}) + assert session.extra_documents['curation_review.json'] == review + assert not (bounded / '.phy-curation/current.json').exists() + finally: + session.close() + + +def test_committed_generation_cannot_omit_required_review_ledger(tmp_path): + _, bounded, _, _ = prepare(tmp_path) + session = BoundedSession(bounded) + generation = session.save() + session.close() + descriptor_path = generation / 'checkpoint.json' + descriptor = json.loads(descriptor_path.read_text()) + descriptor['extra_documents'] = [] + descriptor['files'].pop('curation_review.json') + descriptor_path.write_text(json.dumps(descriptor)) + pointer_path = bounded / '.phy-curation/current.json' + pointer = json.loads(pointer_path.read_text()) + pointer['sha256'] = _digest(descriptor) + pointer_path.write_text(json.dumps(pointer)) + with pytest.raises(ValueError, match='ledger mapping'): + BoundedSession(bounded) diff --git a/phy/apps/large_recording/tests/test_replay.py b/phy/apps/large_recording/tests/test_replay.py new file mode 100644 index 000000000..a0d2aa50e --- /dev/null +++ b/phy/apps/large_recording/tests/test_replay.py @@ -0,0 +1,191 @@ +"""Full assignment oracle, revision binding and atomic publication regressions.""" + +import json + +import numpy as np +import pytest + +from phy.apps.large_recording.replay import apply_curation +from phy.apps.large_recording.source import Source, hash_file +from phy.apps.large_recording.tests.test_curation import make_session + + +def test_full_replay_chains_preserve_unseen_units_and_immutable_bytes(tmp_path): + source, session = make_session(tmp_path) + original_hashes = {name: hash_file(source.path / name) for name in source.arrays} + original = source.arrays['spike_clusters.npy'].copy() + session.record_merge([0, 1], 100, {'group': 'good', 'confidence': 0.8}) + session.record_label('group', [100], 'mua') + session.record_merge([100, 2], 101, {'group': 'good', 'confidence': None}) + session.record_label('custom', [99, 101], 'verified') + session.save() + output = apply_curation(source.params_path, session.path, tmp_path / 'curated', batch_size=3) + expected = np.where(np.isin(original, [0, 1, 2]), 101, original) + assert np.array_equal(np.load(output / 'spike_clusters.npy'), expected) + assert expected[-1] == 99 + curated = Source(output / 'params.py') + assert curated.metadata == session.metadata + assert curated.identity == source.identity + assert curated.revision != source.revision + lineage = json.loads((output / '_phy_curation_lineage.json').read_text()) + assert lineage['parent_revision'] == source.revision + assert lineage['revision'] == curated.revision + for name, digest in original_hashes.items(): + assert hash_file(source.path / name) == digest + if name != 'spike_clusters.npy': + assert hash_file(output / name) == digest + with pytest.raises(ValueError, match='revision'): + apply_curation(output / 'params.py', session.path, tmp_path / 'twice') + assert not (tmp_path / 'twice').exists() + second = apply_curation(source.params_path, session.path, tmp_path / 'second', batch_size=7) + assert hash_file(second / 'spike_clusters.npy') == hash_file(output / 'spike_clusters.npy') + + +def test_source_revision_and_feature_identity_are_independently_bound(tmp_path): + source, session = make_session(tmp_path) + session.save() + clusters = source.arrays['spike_clusters.npy'].copy() + amplitudes = source.arrays['amplitudes.npy'].copy() + source.close() + clusters[0] = 2 + np.save(source.path / 'spike_clusters.npy', clusters) + with pytest.raises(ValueError, match='revision'): + apply_curation(source.params_path, session.path, tmp_path / 'wrong_revision') + amplitudes[0] += 1 + np.save(source.path / 'amplitudes.npy', amplitudes) + with pytest.raises(ValueError, match='identity'): + apply_curation(source.params_path, session.path, tmp_path / 'wrong_feature') + assert not (tmp_path / 'wrong_revision').exists() + assert not (tmp_path / 'wrong_feature').exists() + + +def test_full_threshold_split_matches_scalar_oracle(tmp_path): + source, session = make_session(tmp_path) + original = source.arrays['spike_clusters.npy'].copy() + amplitudes = source.arrays['amplitudes.npy'].copy() + predicate = { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 15.0, + 'nonfinite': 'keep', + } + session.record_split(2, 100, 101, predicate, {'group': {100: 'noise', 101: 'good'}}) + session.record_merge([0, 100], 102, {'group': 'mua'}) + session.save() + expected = original.astype(np.int64) + expected[(original == 2) & (amplitudes < 15)] = 100 + expected[(original == 2) & ~(amplitudes < 15)] = 101 + expected[np.isin(expected, [0, 100])] = 102 + output = apply_curation(source.params_path, session.path, tmp_path / 'split', batch_size=4) + assert np.array_equal(np.load(output / 'spike_clusters.npy'), expected) + assert Source(output / 'params.py').metadata == session.metadata + + +def test_failed_replay_does_not_publish_output(tmp_path, monkeypatch): + from phy.apps.large_recording import replay + + source, session = make_session(tmp_path) + session.save() + original = replay._write_json + + def fail(path, value): + if path.name == '_phy_curation_lineage.json': + raise OSError('injected replay interruption') + return original(path, value) + + monkeypatch.setattr(replay, '_write_json', fail) + with pytest.raises(OSError, match='interruption'): + apply_curation(source.params_path, session.path, tmp_path / 'interrupted') + assert not (tmp_path / 'interrupted').exists() + assert not list(tmp_path.glob('.interrupted-*')) + + +def test_replay_requires_committed_checkpoint_and_new_output(tmp_path): + source, session = make_session(tmp_path) + with pytest.raises(ValueError, match='Save'): + apply_curation(source.params_path, session.path, tmp_path / 'unsaved') + session.save() + with pytest.raises(FileExistsError): + apply_curation(source.params_path, session.path, source.path) + + +def test_replay_preserves_empty_split_children_and_allocator_lineage(tmp_path): + source, session = make_session(tmp_path) + session.record_split( + 2, + 100, + 101, + { + 'attribute': 'amplitudes.npy', + 'operator': '<', + 'threshold': 100.0, + 'nonfinite': 'keep', + }, + ) + session.save() + output = apply_curation(source.params_path, session.path, tmp_path / 'empty_child') + curated = Source(output / 'params.py') + assert 101 in curated.unit_ids + assert curated.counts[curated.unit_ids.tolist().index(101)] == 0 + assert curated.next_cluster_id == 102 + + +def test_bounded_threshold_attribute_must_match_exact_source_rows(tmp_path): + source, session = make_session(tmp_path) + session.save() + session.close() + # Even a manifest that does not hash this optional attribute must not allow + # evidence values different from those used by the full-data predicate. + values = np.load(session.path / 'amplitudes.npy') + values[0] += 1 + np.save(session.path / 'amplitudes.npy', values) + with pytest.raises(ValueError, match='does not match source rows'): + apply_curation(source.params_path, session.path, tmp_path / 'bad_evidence') + assert not (tmp_path / 'bad_evidence').exists() + + +@pytest.mark.parametrize('kind', ['acquisition_samples', 'elapsed_ticks']) +def test_replay_preserves_explicit_clock_metadata_and_timestamp_bytes(tmp_path, kind): + source, session = make_session(tmp_path) + params = source.params_path + intervals = source.valid_intervals.tolist() + rate = source.sample_rate + source.close() + session.close() + contract = {'kind': kind, 'ticks_per_second': rate} + params.write_text( + params.read_text() + + f'\ntime_base = {contract!r}\nvalid_intervals = {intervals!r}\ndat_path = "None"\n' + ) + timestamps = params.parent / 'timestamps.json' + timestamps.write_bytes(b'{"producer": "clock mapping", "gaps": [4, 8]}\n') + upstream_path = params.parent / '_phy_spikes_subset.manifest.json' + upstream = json.loads(upstream_path.read_text()) + upstream['recording'].update( + { + 'time_base': contract, + 'frame_coordinates': kind, + 'window_bounds_coordinates': kind, + 'sample_index_coordinates': kind, + } + ) + upstream_path.write_text(json.dumps(upstream)) + source = Source(params) + manifest_path = session.path / '_phy_curation_manifest.json' + manifest = json.loads(manifest_path.read_text()) + manifest['source']['identity'] = source.identity + manifest['time_base'] = contract + manifest_path.write_text(json.dumps(manifest)) + from phy.apps.large_recording.curation import BoundedSession + + session = BoundedSession(session.path) + session.record_merge([0, 1], 100) + session.save() + output = apply_curation(params, session.path, tmp_path / 'curated') + assert (output / 'timestamps.json').read_bytes() == timestamps.read_bytes() + with Source(output / 'params.py') as curated: + assert curated.identity == source.identity + assert curated.time_base == contract + assert curated.params['dat_path'] is None + session.close() + source.close() diff --git a/phy/apps/large_recording/tests/test_source.py b/phy/apps/large_recording/tests/test_source.py new file mode 100644 index 000000000..a0642ffe7 --- /dev/null +++ b/phy/apps/large_recording/tests/test_source.py @@ -0,0 +1,469 @@ +"""Exact source clock, alignment, and identity regressions.""" + +import json + +import numpy as np +import pytest + +from phy.apps.large_recording.source import Source, hash_file + + +def make_sorting(path, columns=False): + path.mkdir() + samples = np.array([0, 1, 1, 2, 5, 6, 10, 11, 14, 18, 19, 19], dtype=np.int64) + clusters = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 99, 99], dtype=np.int32) + templates = np.arange(len(samples), dtype=np.int32) % 3 + arrays = { + 'spike_times.npy': samples, + 'spike_clusters.npy': clusters, + 'spike_templates.npy': templates, + 'amplitudes.npy': np.arange(len(samples), dtype=np.float32), + 'spike_positions.npy': np.column_stack((np.zeros(len(samples)), samples * 2)), + 'spike_depths.npy': samples.astype(np.float64) * 2, + 'templates.npy': np.arange(3 * 8 * 3, dtype=np.float32).reshape(3, 8, 3), + 'channel_map.npy': np.array([2, 0, 1], dtype=np.int32), + 'channel_positions.npy': np.array([[0, 0], [0, 10], [0, 20]]), + 'pc_features.npy': np.arange(len(samples) * 4, dtype=np.float32).reshape(-1, 2, 2), + 'pc_feature_ind.npy': np.array([[0, 2], [2, 1], [1, 0]], dtype=np.int32), + } + for name, array in arrays.items(): + if columns and name in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy'): + array = array[:, None] + np.save(path / name, array) + ids = np.array([0, 2, 3, 8, 11], dtype=np.int64) + channels = np.array([[0, -1], [2, 1], [1, -1], [2, -1], [0, 1]], dtype=np.int32) + waveforms = np.zeros((len(ids), 8, 2), dtype=np.float32) + for row, spike in enumerate(ids): + for slot, channel in enumerate(channels[row]): + if channel != -1: + waveforms[row, :, slot] = spike + channel + np.arange(8) / 10 + for suffix, array in (('spikes', ids), ('channels', channels), ('waveforms', waveforms)): + np.save(path / f'_phy_spikes_subset.{suffix}.npy', array) + (path / 'params.py').write_text( + "sample_rate = 1.0\nnum_samples = 20\ndat_path = None\nn_channels_dat = 3\ndtype = 'int16'\n" + ) + (path / 'cluster_group.tsv').write_text( + 'cluster_id\tgroup\n0\tgood\n1\tmua\n2\tgood\n99\tnoise\n' + ) + return path / 'params.py' + + +@pytest.mark.parametrize('columns', [False, True]) +def test_source_batches_ties_counts_columns(tmp_path, columns): + source = Source(make_sorting(tmp_path / 'full', columns), batch_size=2) + assert source.n_spikes == 12 + assert source.unit_ids.tolist() == [0, 1, 2, 99] + assert source.counts.tolist() == [4, 3, 3, 2] + assert all(batch['spike_times.npy'].ndim == 1 for batch in source.iter_batches()) + assert all(isinstance(array, np.memmap) for array in source.arrays.values()) + assert source.metadata['group'][99] == 'noise' + + +def test_declared_waveform_source_identity_is_checked(tmp_path): + params = make_sorting(tmp_path / 'full') + manifest = { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': {'num_samples': 20, 'sampling_frequency_hz': 1}, + 'source_sorting': {'num_spikes': 12, 'spike_times_sha256': '0' * 64}, + } + path = params.parent / '_phy_spikes_subset.manifest.json' + path.write_text(json.dumps(manifest)) + with pytest.raises(ValueError, match='fingerprint'): + Source(params) + manifest['source_sorting']['spike_times_sha256'] = hash_file(params.parent / 'spike_times.npy') + path.write_text(json.dumps(manifest)) + Source(params).close() + manifest['source_sorting']['num_spikes'] = 13 + path.write_text(json.dumps(manifest)) + with pytest.raises(ValueError, match='spike count'): + Source(params) + + +def test_failed_source_validation_closes_readers_and_rejects_unsupported_ids( + tmp_path, monkeypatch +): + params = make_sorting(tmp_path / 'full') + labels = np.load(params.parent / 'spike_clusters.npy').astype(np.int64) + labels[-1] = 2**31 + np.save(params.parent / 'spike_clusters.npy', labels) + original = np.load + readers = [] + + def read(*args, **kwargs): + array = original(*args, **kwargs) + readers.append(array) + return array + + monkeypatch.setattr(np, 'load', read) + with pytest.raises(ValueError, match='int32'): + Source(params) + assert all(array._mmap.closed for array in readers) + + +def test_identity_assignment_and_feature_layers(tmp_path): + params = make_sorting(tmp_path / 'full') + source = Source(params) + source.close() + clusters = np.load(params.parent / 'spike_clusters.npy') + clusters[clusters == 1] = 0 + np.save(params.parent / 'spike_clusters.npy', clusters) + merged = Source(params) + merged.close() + assert merged.identity == source.identity + assert merged.revision != source.revision + values = np.load(params.parent / 'amplitudes.npy') + values[0] = 99 + np.save(params.parent / 'amplitudes.npy', values) + altered = Source(params) + altered.close() + assert altered.identity != merged.identity + assert altered.revision == merged.revision + assert ( + hash_file(params.parent / 'amplitudes.npy') == altered.identity['arrays']['amplitudes.npy'] + ) + + +def test_exposure_gaps_and_missing_duration(tmp_path): + params = make_sorting(tmp_path / 'full') + source = Source(params, valid_intervals=[[0, 7], [10, 20]]) + assert source.valid_intervals.tolist() == [[0, 7], [10, 20]] + with pytest.raises(ValueError, match='outside valid'): + Source(params, valid_intervals=[[0, 5], [10, 20]]) + params.write_text(params.read_text().replace('num_samples = 20\n', '')) + with pytest.raises(ValueError, match='exposure is missing'): + Source(params) + assert Source(params, duration_seconds=20).num_samples == 20 + with pytest.raises(ValueError, match='integer sample count'): + Source(params, duration_seconds=20.5) + + +@pytest.mark.parametrize( + 'failure', ['order', 'alignment', 'unknown', 'segment', 'waveforms', 'ids'] +) +def test_reject_ambiguous_or_misaligned_sources(tmp_path, failure): + params = make_sorting(tmp_path / 'full') + path = params.parent + if failure == 'order': + np.save(path / 'spike_times.npy', np.arange(12)[::-1]) + elif failure == 'alignment': + np.save(path / 'pc_features.npy', np.zeros((11, 2, 2))) + elif failure == 'unknown': + np.save(path / 'mysterious.npy', np.ones(12)) + elif failure == 'segment': + params.write_text(params.read_text() + 'num_segments = 2\n') + elif failure == 'waveforms': + (path / '_phy_spikes_subset.channels.npy').unlink() + else: + np.save(path / '_phy_spikes_subset.spikes.npy', np.array([0, 2, 2, 8, 11], dtype=np.int64)) + with pytest.raises(ValueError): + Source(params, batch_size=2) + + +def test_metadata_json_preserves_typed_values(tmp_path): + params = make_sorting(tmp_path / 'full') + (params.parent / 'cluster_group.tsv').write_text('cluster_id\tgroup\n0\tgood\n') + metadata = {'group': {'0': 'good'}, 'custom': {'99': None, '2': 4}} + (params.parent / '_phy_cluster_metadata.json').write_text(json.dumps(metadata)) + assert Source(params).metadata == {'group': {0: 'good'}, 'custom': {99: None, 2: 4}} + + +def test_replay_lineage_retains_zero_count_catalog_and_allocator(tmp_path): + params = make_sorting(tmp_path / 'full') + source = Source(params) + lineage = { + 'format': 'phy_curation_lineage', + 'format_version': 1, + 'coordinate_identity': source.identity, + 'revision': source.revision, + 'unit_ids': [0, 1, 2, 99, 103], + 'next_cluster_id': 105, + } + (params.parent / '_phy_curation_lineage.json').write_text(json.dumps(lineage)) + reopened = Source(params) + assert reopened.next_cluster_id == 105 + assert reopened.unit_ids[-1] == 103 + assert reopened.counts[-1] == 0 + assert reopened.identity == source.identity + lineage['next_cluster_id'] = 103 + (params.parent / '_phy_curation_lineage.json').write_text(json.dumps(lineage)) + with pytest.raises(ValueError, match='collides'): + Source(params) + + +def test_unknown_upstream_manifest_version_is_rejected(tmp_path): + params = make_sorting(tmp_path / 'full') + (params.parent / '_phy_spikes_subset.manifest.json').write_text( + json.dumps({'format': 'phy_spikes_subset', 'format_version': 2}) + ) + with pytest.raises(ValueError, match='Unsupported waveform manifest'): + Source(params) + + +def test_integer_clock_beyond_32_bits(tmp_path): + from phy.apps.tests.large_recording_fixtures import make_long_recording_fixture + + fixture = make_long_recording_fixture(tmp_path) + source = Source(fixture.full / 'params.py', batch_size=3) + assert source.num_samples == 1000 * 3600 * 30000 + assert source.arrays['spike_times.npy'][-1] > np.iinfo(np.int32).max + assert source.n_spikes == 32 + + +def test_source_heap_does_not_scale_with_spike_arrays(tmp_path): + import tracemalloc + + peaks = [] + for spikes in (1000, 250000): + path = tmp_path / str(spikes) + path.mkdir() + for name, dtype in ( + ('spike_times.npy', np.int64), + ('spike_clusters.npy', np.int32), + ('spike_templates.npy', np.int32), + ): + values = np.lib.format.open_memmap( + path / name, mode='w+', dtype=dtype, shape=(spikes,) + ) + for start in range(0, spikes, 1024): + stop = min(start + 1024, spikes) + values[start:stop] = np.arange(start, stop) if name == 'spike_times.npy' else 0 + values.flush() + del values + (path / 'params.py').write_text(f'sample_rate = 1\nnum_samples = {spikes}\n') + tracemalloc.start() + source = Source(path / 'params.py', batch_size=1024) + _, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + assert source.n_spikes == spikes + assert sum(source.counts) == spikes + peaks.append(peak) + # This checks allocated heap; file-backed OS pages are intentionally separate. + assert peaks[-1] < 4 * 1024 * 1024 + assert peaks[-1] - peaks[0] < 3 * 1024 * 1024 + + +def test_explicit_intervals_cannot_override_changed_producer_exposure(tmp_path): + params = make_sorting(tmp_path / 'full') + params.write_text(params.read_text() + 'valid_intervals = [[0, 7], [10, 20]]\n') + with pytest.raises(ValueError, match='disagree with acquisition metadata'): + Source(params, valid_intervals=[[0, 20]]) + with Source(params) as source: + arrays = list(source.arrays.values()) + assert source.valid_intervals.tolist() == [[0, 7], [10, 20]] + assert not source.arrays + assert all(array._mmap.closed for array in arrays) + + +def test_exporter_channel_sidecars_and_raw_map_validation(tmp_path): + params = make_sorting(tmp_path / 'full') + np.save( + params.parent / 'channel_map_si.npy', np.array(['hardware2', 'hardware0', 'hardware1']) + ) + np.save(params.parent / 'channel_groups.npy', np.array([1, 1, 0])) + with Source(params) as source: + assert 'channel_map_si.npy' in source.arrays + params.write_text(params.read_text().replace('n_channels_dat = 3', 'n_channels_dat = 2')) + with pytest.raises(ValueError, match='outside n_channels_dat'): + Source(params) + + +@pytest.mark.parametrize( + 'positions', [np.zeros((3, 1)), np.zeros((3, 3)), np.full((3, 2), np.nan)] +) +def test_channel_geometry_must_match_gui_contract(tmp_path, positions): + params = make_sorting(tmp_path / 'source') + np.save(params.parent / 'channel_positions.npy', positions) + with pytest.raises(ValueError, match='finite x/y'): + Source(params) + + +def test_exclusive_publish_preserves_racing_empty_destination(tmp_path): + from phy.apps.large_recording.source import publish_directory + + staged = tmp_path / 'stage' + staged.mkdir() + (staged / 'payload').write_text('complete') + destination = tmp_path / 'destination' + destination.mkdir() + with pytest.raises(FileExistsError): + publish_directory(staged, destination) + assert destination.is_dir() and not list(destination.iterdir()) + assert (staged / 'payload').read_text() == 'complete' + destination.rmdir() + publish_directory(staged, destination) + assert not staged.exists() + assert (destination / 'payload').read_text() == 'complete' + + +def test_clock_endpoint_mismatch_fails_before_full_scan_or_hash(tmp_path, monkeypatch): + from phy.apps.large_recording import source as module + + params = make_sorting(tmp_path / 'source') + times = np.load(params.parent / 'spike_times.npy') + times[-1] = 100 + np.save(params.parent / 'spike_times.npy', times) + + def unexpected(*args, **kwargs): + pytest.fail('Clock endpoint mismatch must fail before scans or hashing.') + + monkeypatch.setattr(Source, 'iter_batches', unexpected) + monkeypatch.setattr(module, 'hash_file', unexpected) + with pytest.raises(ValueError, match='same time base'): + Source(params) + + +def clock_manifest(params, recording): + path = params.parent / '_phy_spikes_subset.manifest.json' + path.write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'recording': recording, + } + ) + ) + return path + + +def test_explicit_clocks_and_legacy_identity(tmp_path): + params = make_sorting(tmp_path / 'source') + with Source(params) as source: + legacy = source.identity + assert source.time_base is None + # Existing exports preserve their exact identity schema. + assert set(legacy['time_base']) == {'sample_rate', 'num_samples', 'valid_intervals'} + contract = {'kind': 'acquisition_samples', 'ticks_per_second': 1} + params.write_text(params.read_text() + f'time_base = {contract!r}\n') + clock_manifest(params, {'time_base': contract}) + with Source(params) as source: + assert source.time_base == contract + assert source.identity != legacy + explicit = source.identity + timestamps = params.parent / 'timestamps.json' + timestamps.write_text('{"mapping": "producer-owned"}') + with Source(params) as source: + assert source.identity != explicit + assert source.identity['time_base']['timestamps_sha256'] == hash_file(timestamps) + timestamps.write_text('{"mapping": "changed"}') + with Source(params) as source: + assert source.identity['time_base']['timestamps_sha256'] == hash_file(timestamps) + + +@pytest.mark.parametrize( + 'contract', + [ + None, + {}, + {'kind': 'clock', 'ticks_per_second': 1}, + {'kind': 'elapsed_ticks', 'ticks_per_second': 0}, + {'kind': 'elapsed_ticks', 'ticks_per_second': float('inf')}, + {'kind': 'elapsed_ticks', 'ticks_per_second': True}, + {'kind': 'elapsed_ticks', 'ticks_per_second': '1'}, + ], +) +def test_invalid_explicit_clock_contract(tmp_path, contract): + params = make_sorting(tmp_path / 'source') + clock_manifest(params, {'time_base': contract}) + with pytest.raises(ValueError, match='time_base'): + Source(params) + + +def test_timestamps_without_contract_fail_before_arrays(tmp_path, monkeypatch): + params = make_sorting(tmp_path / 'source') + (params.parent / 'timestamps.json').write_text('{}') + monkeypatch.setattr(np, 'load', lambda *a, **kw: pytest.fail('Unexpected array read')) + with pytest.raises(ValueError, match='explicit time_base'): + Source(params) + + +@pytest.mark.parametrize( + 'failure', + [ + 'clock', + 'ticks', + 'waveform_rate', + 'manifest_rate', + 'frame_coordinates', + 'missing_intervals', + 'raw', + 'window_coordinates', + 'sample_coordinates', + ], +) +def test_explicit_elapsed_clock_rejects_ambiguous_producer_metadata(tmp_path, failure): + params = make_sorting(tmp_path / 'source') + contract = {'kind': 'elapsed_ticks', 'ticks_per_second': 1} + recording = {'time_base': contract, 'valid_intervals': [[0, 20]]} + if failure == 'clock': + params.write_text( + params.read_text() + + "time_base = {'kind': 'acquisition_samples', 'ticks_per_second': 1}\n" + ) + elif failure == 'ticks': + recording['time_base'] = {**contract, 'ticks_per_second': 2} + elif failure == 'waveform_rate': + params.write_text(params.read_text() + 'waveform_sample_rate = 2\n') + elif failure == 'manifest_rate': + recording['sampling_frequency_hz'] = 2 + elif failure == 'frame_coordinates': + recording['frame_coordinates'] = 'before any clock-time conversion' + elif failure == 'missing_intervals': + recording.pop('valid_intervals') + elif failure == 'raw': + params.write_text(params.read_text() + "dat_path = ['None', 'raw.bin']\n") + elif failure == 'window_coordinates': + np.save(params.parent / '_phy_spikes_subset.window_bounds.npy', [[0, 20]]) + else: + np.save(params.parent / '_phy_spikes_subset.sample_index.npy', [0, 1, 2, 14, 19]) + recording['sample_index_coordinates'] = 'acquisition_samples' + clock_manifest(params, recording) + with pytest.raises(ValueError): + Source(params) + + +@pytest.mark.parametrize('raw', [None, 'None', '', [], [None, 'None', '']]) +def test_elapsed_clock_canonical_sidecars_and_no_raw_sentinels(tmp_path, raw): + params = make_sorting(tmp_path / 'source') + contract = {'kind': 'elapsed_ticks', 'ticks_per_second': 1} + params.write_text(params.read_text() + f'dat_path = {raw!r}\n') + recording = { + 'time_base': contract, + 'valid_intervals': [[0, 7], [10, 20]], + 'frame_coordinates': 'elapsed_ticks', + 'window_bounds_coordinates': 'elapsed_ticks', + 'sample_index_coordinates': 'elapsed_ticks', + } + clock_manifest(params, recording) + np.save(params.parent / '_phy_spikes_subset.window_bounds.npy', [[0, 7], [10, 20]]) + np.save(params.parent / '_phy_spikes_subset.sample_index.npy', [0, 1, 2, 14, 19]) + with Source(params) as source: + assert source.time_base == contract + assert source.upstream_recording == recording + assert source.valid_intervals.tolist() == [[0, 7], [10, 20]] + + +@pytest.mark.parametrize( + 'values', + [ + [0, 1, 2, 14], + [[0], [1], [2], [14], [19]], + [0.0, 1.0, 2.0, 14.0, 19.0], + [-1, 1, 2, 14, 19], + [0, 1, 2, 14, 20], + [0, 1, 2, 13, 19], + ], +) +def test_waveform_sample_index_requires_exact_canonical_alignment(tmp_path, values): + params = make_sorting(tmp_path / 'source') + np.save(params.parent / '_phy_spikes_subset.sample_index.npy', values) + with pytest.raises(ValueError, match='sample_index'): + Source(params) + + +def test_waveform_sample_index_legacy_default_is_acquisition_samples(tmp_path): + params = make_sorting(tmp_path / 'source', columns=True) + np.save(params.parent / '_phy_spikes_subset.sample_index.npy', [0, 1, 2, 14, 19]) + with Source(params) as source: + assert '_phy_spikes_subset.sample_index.npy' in source.identity['arrays'] diff --git a/phy/apps/large_recording/tests/test_stability.py b/phy/apps/large_recording/tests/test_stability.py new file mode 100644 index 000000000..50412cae8 --- /dev/null +++ b/phy/apps/large_recording/tests/test_stability.py @@ -0,0 +1,605 @@ +"""Exact selected-unit comparisons and scoped GUI selection behavior.""" + +from itertools import product +from types import MappingProxyType + +import numpy as np +import pytest +from phylib.utils import Bunch + +from phy.apps.large_recording.stability import UnitStabilityView +from phy.apps.large_recording.tests.test_adapter import make_controller +from phy.utils.color import selected_cluster_color + + +@pytest.fixture +def summary_view(qtbot, tmp_path): + arrays = { + 'edges': [0, 10, 20, 25], + 'exposure': [8.0, 0.0, 3.0], + 'counts': [[4, 0, 3], [12, 0, 6], [8, 0, 0]], + 'amplitude_sum': [[4.0, 0.0, 15.0], [30.0, 0.0, 8.0], [0.0, 0.0, 0.0]], + 'amplitude_count': [[2, 0, 3], [3, 0, 1], [0, 0, 0]], + 'depth_sum': [[40.0, 0.0, 150.0], [300.0, 0.0, 80.0], [0.0, 0.0, 0.0]], + 'depth_count': [[2, 0, 3], [3, 0, 1], [0, 0, 0]], + } + summary = {} + for key, values in arrays.items(): + name = f'{key}.npy' + np.save(tmp_path / name, np.asarray(values)) + summary[f'{key}_file'] = name + membership = {1: [1], 2: [2], 3: [3], 4: [1, 2], 5: None} + controller = Bunch( + dir_path=tmp_path, + model=Bunch(sample_rate=1.0, duration=25.0), + evidence_windows=np.array([[0, 2], [22, 24]]), + bounded_session=Bunch( + unit_ids=[1, 2, 3], + manifest={ + 'summaries': summary, + 'bounded_counts': [1, 1, 0], + 'valid_intervals': [[0, 8], [20, 23]], + }, + ), + _source_units=lambda cluster: membership[cluster], + get_spike_times=lambda cluster: pytest.fail('Bounded views must never read spike arrays'), + ) + view = UnitStabilityView(controller) + yield view + view.close() + + +@pytest.mark.parametrize('selected', [[1, 2], [1, 2, 3]]) +def test_exact_overlays_and_shared_exposure(summary_view, selected): + view = summary_view + view.on_select(selected) + axes = list(view.canvas.iter_ax()) + assert len(axes[0].lines) == len(selected) + assert [line.get_label() for line in axes[0].lines] == list(map(str, selected)) + expected = {1: [0.5, np.nan, 1.0], 2: [1.5, np.nan, 2.0], 3: [1.0, np.nan, 0.0]} + for index, (cluster, line) in enumerate(zip(selected, axes[0].lines)): + np.testing.assert_allclose(line.get_ydata(), expected[cluster], equal_nan=True) + np.testing.assert_allclose(line.get_xdata(), np.array([5, 15, 22.5]) / 3600) + np.testing.assert_allclose(line.get_color(), selected_cluster_color(index)) + assert '3' in view._footer.get_text() # Unit without bounded evidence remains inspectable. + assert len(axes[1].lines) == 2 # No fabricated all-NaN trend for unit 3. + assert any( + 'Amplitude measurement unavailable for units: 3' in t.get_text() for t in axes[1].texts + ) == (3 in selected) + + +def test_weighted_merged_trends_and_pending_do_not_hide_valid_units(summary_view): + view = summary_view + merged = view.get_data(4) + np.testing.assert_array_equal(merged.counts, [16, 0, 9]) + np.testing.assert_allclose(merged.rates, [2.0, np.nan, 3.0], equal_nan=True) + np.testing.assert_allclose(merged.trends['amplitude'], [6.8, np.nan, 5.75], equal_nan=True) + np.testing.assert_allclose(merged.trends['depth'], [68.0, np.nan, 57.5], equal_nan=True) + view.on_select([5, 4]) + ax = next(view.canvas.iter_ax()) + assert len(ax.lines) == 1 and ax.lines[0].get_label() == '4' + assert any('Unit 5: Split evidence pending' in t.get_text() for t in ax.texts) + + +def test_lazy_metrics_selected_cache_and_all_toggle_layouts(summary_view, monkeypatch): + view = summary_view + loads = [] + original = view._array + + def load(name): + loads.append(name) + return original(name) + + monkeypatch.setattr(view, '_array', load) + view.show_amplitude = view.show_depth = False + view.on_select([1, 2]) + assert not any('amplitude' in name or 'depth' in name for name in loads) + common = view._common() + lookups = common.lookup + view.toggle_amplitude(True) + assert any('amplitude' in name for name in loads) + assert not any('depth' in name for name in loads) + loads.clear() + view.toggle_amplitude(False) + view.toggle_amplitude(True) + assert not loads + assert view._common() is common and view._common().lookup is lookups + axes = list(view.canvas.iter_ax()) + callback_counts = {k: len(v) for k, v in view.canvas.canvas.callbacks.callbacks.items()} + for _ in range(2): + for combination in product((False, True), repeat=3): + for metric, enabled in zip(view._metrics, combination): + getattr(view, f'toggle_{metric}')(enabled) + assert tuple(view.state[f'show_{metric}'] for metric in view._metrics) == combination + assert list(view.canvas.iter_ax()) == axes + assert { + k: len(v) for k, v in view.canvas.canvas.callbacks.callbacks.items() + } == callback_counts + visible = [ax for ax in axes if ax.get_visible()] + if any(combination): + assert len(visible) == sum(combination) + assert visible[-1].get_xlabel() == 'Elapsed acquisition clock (hours)' + assert all(not ax.get_xlabel() for ax in visible[:-1]) + assert visible[0].get_position().y1 == pytest.approx( + 0.91 if len(visible) == 1 else 0.15 + 0.76 - 0.16 * 0.76 / len(visible) + ) + else: + assert any('No metrics enabled' in t.get_text() for t in visible[0].texts) + assert view.state.show_depth + view.on_select([3]) + assert set(view._selected_summaries) == {3} + + +def test_selection_limit_palette_and_clear_lifecycle(summary_view): + view = summary_view + view.display_limit = 8 + # Repeat source rows under distinct cluster IDs to test the display bound. + view.controller._source_units = lambda cluster: [1] + selected = list(range(20, 30)) + sender = Bunch( + selection_color_indices=MappingProxyType({c: 9 - i for i, c in enumerate(selected)}) + ) + view.on_select_threaded(sender, selected) + assert view.cluster_ids == tuple(selected) + assert selected == list(range(20, 30)) + assert len(view._selected_summaries) == 8 + ax = next(view.canvas.iter_ax()) + assert len(ax.lines) == 8 + assert 'Showing first 8 of 10 selected units' in view.status + assert 'does not change the selection or merge operation' in view.status + for i, line in enumerate(ax.lines): + np.testing.assert_allclose(line.get_color(), selected_cluster_color(9 - i)) + view.auto_update = False + view.on_select_threaded(sender, []) + assert view.cluster_ids == tuple(selected) + view.auto_update = True + view.defer_hidden_updates = True + view._dock_visible = False + view.on_select_threaded(sender, selected) + view._dock_visible = True + view._flush_pending_selection() + np.testing.assert_allclose(ax.lines[0].get_color(), selected_cluster_color(9)) + view._dock_visible = False + view.on_select_threaded(sender, []) + assert view.cluster_ids == () and view._pending_selection == ([], {}) + assert len(ax.lines) == 8 + view._dock_visible = True + view._flush_pending_selection() + assert not ax.lines + view.on_select([1]) + view._closed = True + view.on_select_threaded(sender, []) + assert view.cluster_ids == (1,) + view._closed = False + + +def test_ordinary_continuous_assumption_and_invalidation(qtbot): + times = [np.array([0.0, 1.0, 9.0])] + calls = [] + + def get_times(cluster): + calls.append(cluster) + return times[0] + + controller = Bunch(model=Bunch(duration=10.0), get_spike_times=get_times) + view = UnitStabilityView(controller) + try: + view.on_select([7, 8]) + data = view.get_data(7) + assert data.counts.sum() == 3 and not data.trends + assert 'continuous acquisition assumed' in data.provenance + np.testing.assert_allclose(data.exposure, 0.1) + assert calls == [7, 8] + times[0] = np.array([2.0]) + view._invalidate() + assert view.get_data(7).counts.sum() == 1 + finally: + view.close() + + +def test_gui_initial_authoritative_selection_actions_and_curation(qtbot, tmp_path, monkeypatch): + _, controller = make_controller(tmp_path, sparse=True) + gui = controller.create_gui(default_views=[], do_prompt_save=False) + supervisor = controller.supervisor + try: + supervisor.select([0]) + supervisor.block() + supervisor.similarity_view.select([1]) + supervisor.block() + view = UnitStabilityView(controller) + view.attach(gui) + assert view.cluster_ids == tuple(supervisor.selected) == (0, 1) + for metric in view._metrics: + assert view.actions.get(f'toggle_{metric}').isChecked() + view.actions.get('toggle_amplitude').trigger() + assert not view.show_amplitude + assert not view.actions.get('toggle_amplitude').isChecked() + view.toggle_depth(False) + assert not view.actions.get('toggle_depth').isChecked() + # Exercise the real dock camera callback with the Matplotlib Qt widget. + from phy.apps.large_recording import stability + + default_path = stability.screenshot_default_path + paths = [] + + def screenshot_path(widget, dir=None): + path = default_path(widget, dir=tmp_path) + paths.append(path) + return path + + monkeypatch.setattr(stability, 'screenshot_default_path', screenshot_path) + view.dock._dock_widgets['screenshot'].click() + assert paths[-1].read_bytes().startswith(b'\x89PNG\r\n\x1a\n') + assert paths[-1].name.endswith('_UnitStabilityView.png') + assert view.screenshot(tmp_path) == paths[-1] + view.actions.run('set_rate_line_width', 0.75) + view.actions.run('set_display_limit', 64) + view.actions.run('set_display_bin_seconds', 20) + assert 'display_bin_seconds' in view.local_state_attrs + assert 'display_bin_seconds' not in view.state_attrs + state = view.state.copy() + supervisor.toggle_merge_mode() + supervisor.block() + supervisor.similarity_view.select([2, 1]) + supervisor.block() + other = UnitStabilityView(controller) + other.attach(gui) + assert other.cluster_ids == tuple(supervisor.selected) + assert other.cluster_ids + supervisor.add_to_merge((2,), insertion=1) + supervisor.block() + assert other.cluster_ids == tuple(supervisor.selected) + assert other.cluster_ids[1] == 2 + supervisor.similarity_view.sort_by('id', 'desc') + supervisor.block() + assert other.cluster_ids == tuple(supervisor.selected) + for index, line in enumerate(next(other.canvas.iter_ax()).lines): + cluster = int(line.get_label()) + np.testing.assert_allclose( + line.get_color(), + selected_cluster_color(supervisor.selection_color_indices[cluster]), + ) + # Save and restore through GUI view state, including QAction checks. + other.close() + view.close() + restored = UnitStabilityView(controller) + restored.attach(gui) + assert restored.rate_line_width == 0.75 + assert restored.display_limit == 64 + assert restored.display_bin_seconds == 20 + assert restored.show_firing_rate + assert not restored.show_amplitude and not restored.show_depth + assert not restored.actions.get('toggle_amplitude').isChecked() + assert not restored.actions.get('toggle_depth').isChecked() + view = restored + view.set_state(state) + assert not view.show_amplitude and not view.show_depth + supervisor.toggle_merge_mode() + supervisor.block() + supervisor.merge([0, 1]) + supervisor.block() + assert view.get_data(100).counts.sum() == 7 + assert view.get_display_data(100).counts.sum() == 7 + supervisor.undo() + supervisor.block() + assert view.get_data(0).counts.sum() == 4 + assert view.get_display_data(0).counts.sum() == 4 + supervisor.redo() + supervisor.block() + assert view.get_data(100).counts.sum() == 7 + assert view.get_display_data(100).counts.sum() == 7 + supervisor.undo() + supervisor.block() + up = supervisor.split_source_amplitude(0, 2) + supervisor.block() + children = up.added + for cluster in children: + assert view.get_data(cluster).pending + view.on_select([children[0], 1]) + assert [line.get_label() for line in next(view.canvas.iter_ax()).lines] == ['1'] + supervisor.undo() + supervisor.block() + assert view.get_data(0).counts.sum() == 4 + assert view.get_display_data(0).counts.sum() == 4 + supervisor.redo() + supervisor.block() + assert view.get_data(children[0]).pending + view.close() + assert not view._summary_arrays and not view._selected_summaries + finally: + gui.close() + controller.close() + + +def test_500_hour_500_bin_comparison_stays_bounded(summary_view): + view = summary_view + bins = 500 + view._common_data = Bunch( + edges=np.linspace(0, 500 * 3600, bins + 1), + exposure=np.full(bins, 3600.0), + windows=np.array([[0, 30], [499 * 3600, 499 * 3600 + 30]]), + intervals=[[0, 500 * 3600]], + footer='', + lookup={1: 0, 2: 1, 3: 2}, + summary={'counts_file': 'virtual_counts'}, + ) + view._summary_arrays['virtual_counts'] = np.ones((3, bins), dtype=np.int64) + view.controller._source_units = lambda cluster: [1] + view.on_select(list(range(8))) + assert len(view._selected_summaries) == 8 + assert all(len(line.get_xdata()) == bins for line in next(view.canvas.iter_ax()).lines) + + +def test_summary_aggregation_reads_one_row_at_a_time(summary_view): + view = summary_view + reads = [] + + class RowGrid: + def __init__(self, values): + self.values = values + + def __getitem__(self, row): + assert isinstance(row, (int, np.integer)), 'Never materialize multiple unit rows' + reads.append(row) + return self.values[row] + + for key in ('counts', 'amplitude_sum', 'amplitude_count', 'depth_sum', 'depth_count'): + name = f'{key}.npy' + view._summary_arrays[name] = RowGrid(np.load(view.controller.dir_path / name)) + data = view.get_data(4) + np.testing.assert_array_equal(data.counts, [16, 0, 9]) + assert reads == [0, 1] + [0, 0, 1, 1] * 2 + + +def test_ordinary_gui_cluster_event_refreshes_cached_counts(qtbot, tmp_path): + _, controller = make_controller(tmp_path) + gui = controller.create_gui(default_views=[], do_prompt_save=False) + times = [np.array([0.0, 1.0])] + ordinary = Bunch( + supervisor=controller.supervisor, + model=Bunch(duration=20.0), + get_spike_times=lambda cluster: times[0], + ) + view = UnitStabilityView(ordinary) + try: + controller.supervisor.select([0]) + controller.supervisor.block() + view.attach(gui) + assert view.get_data(0).counts.sum() == 2 + times[0] = np.array([0.0, 1.0, 2.0, 3.0]) + controller.supervisor.label('group', 'noise', cluster_ids=[0]) + controller.supervisor.block() + ax = next(view.canvas.iter_ax()) + assert np.nansum(ax.lines[0].get_ydata() * 0.2) == pytest.approx(4) + assert view.get_data(0).counts.sum() == 4 + assert view.get_display_data(0).counts.sum() == 4 + finally: + view.close() + gui.close() + controller.close() + + +def test_display_limit_all_and_large_values_preserve_selection(summary_view): + view = summary_view + assert view.display_limit == 32 + view.controller._source_units = lambda cluster: [1] + selected = list(range(40)) + view.on_select(selected) + assert view.cluster_ids == tuple(selected) + assert len(view.displayed_ids) == len(next(view.canvas.iter_ax()).lines) == 32 + assert 'merge operation' in view.status + for limit in (0, 64, 128): + view.set_display_limit(limit) + assert view.displayed_ids == tuple(selected) + assert len(next(view.canvas.iter_ax()).lines) == 40 + assert not view.status + assert view.cluster_ids == tuple(selected) + for ax in view.canvas.iter_ax(): + assert ax.get_legend() is None + assert all(line.get_linewidth() == 0.6 for line in next(view.canvas.iter_ax()).lines) + view.set_rate_line_width(0.7) + assert all(line.get_linewidth() == 0.7 for line in next(view.canvas.iter_ax()).lines) + + +@pytest.mark.parametrize('value', [None, 'bad', np.nan, np.inf, -1, 0]) +def test_invalid_line_width_state_and_action(summary_view, value): + view = summary_view + with pytest.raises((ValueError, TypeError)): + view.set_rate_line_width(value) + view.set_state({'rate_line_width': value}) + assert view.rate_line_width == 0.6 + + +@pytest.mark.parametrize('value', [None, 'bad', np.nan, np.inf, -1, 1.5]) +def test_invalid_display_limit_state_and_action(summary_view, value): + view = summary_view + with pytest.raises((ValueError, TypeError)): + view.set_display_limit(value) + view.set_state({'display_limit': value}) + assert view.display_limit == 32 + + +@pytest.mark.parametrize('value', [None, 'bad', np.nan, np.inf, -1, 5, 15]) +def test_invalid_display_bins_state_and_action(summary_view, value): + view = summary_view + with pytest.raises((ValueError, TypeError)): + view.set_display_bin_seconds(value) + view.set_state({'display_bin_seconds': value}) + assert view.display_bin_seconds == 0 + + +@pytest.mark.parametrize( + 'seconds, expected_edges, counts, exposure, amplitude', + [ + (0, [0, 10, 20, 25], [16, 0, 9], [8, 0, 3], [6.8, np.nan, 5.75]), + (20, [0, 20, 25], [16, 9], [8, 3], [6.8, 5.75]), + (30, [0, 25], [25], [11], [57 / 9]), + (100, [0, 25], [25], [11], [57 / 9]), + ], +) +def test_display_rebin_exact_merged_primitives( + summary_view, seconds, expected_edges, counts, exposure, amplitude +): + view = summary_view + view.set_display_bin_seconds(seconds) + data = view.get_display_data(4) + np.testing.assert_array_equal(data.edges, expected_edges) + np.testing.assert_array_equal(data.counts, counts) + np.testing.assert_array_equal(data.exposure, exposure) + np.testing.assert_allclose(data.trends['amplitude'], amplitude, equal_nan=True) + expected_rates = np.full(len(exposure), np.nan) + np.divide(counts, exposure, out=expected_rates, where=np.asarray(exposure) > 0) + np.testing.assert_allclose(data.rates, expected_rates, equal_nan=True) + view.on_select([4]) + axes = list(view.canvas.iter_ax()) + for ax in axes: + np.testing.assert_allclose( + ax.lines[0].get_xdata(), (data.edges[:-1] + data.edges[1:]) / 7200 + ) + assert all(ax.get_legend() is None for ax in axes) + + +def test_300_second_native_bins_and_weighted_trends(summary_view, monkeypatch): + view = summary_view + common = view._common() + common.edges = np.array([0, 300, 600, 900, 1000]) + common.exposure = np.array([300.0, 0.0, 100.0, 40.0]) + common.intervals = [[0, 300], [600, 700], [900, 940]] + # Means 1 and 9 have different finite counts: their aggregate is 7, not 5. + view._summary_arrays.update( + { + 'counts.npy': np.array([[30, 0, 20, 4]]), + 'amplitude_sum.npy': np.array([[1.0, 0.0, 27.0, 20.0]]), + 'amplitude_count.npy': np.array([[1, 0, 3, 2]]), + 'depth_sum.npy': np.array([[10.0, 0.0, 270.0, 200.0]]), + 'depth_count.npy': np.array([[1, 0, 3, 2]]), + } + ) + source = view.get_data(1) + assert view._display_factor() == 1 + np.testing.assert_allclose(source.rates, [0.1, np.nan, 0.2, 0.1], equal_nan=True) + monkeypatch.setattr(view, '_array', lambda name: pytest.fail('Cached rows need no I/O')) + for seconds, edges, counts, exposures, means in ( + (600, [0, 600, 1000], [30, 24], [300, 140], [1, 47 / 5]), + (900, [0, 900, 1000], [50, 4], [400, 40], [7, 10]), + (1800, [0, 1000], [54], [440], [8]), + ): + view.set_display_bin_seconds(seconds) + data = view.get_display_data(1) + np.testing.assert_array_equal(data.edges, edges) + np.testing.assert_array_equal(data.counts, counts) + np.testing.assert_array_equal(data.exposure, exposures) + np.testing.assert_allclose(data.trends['amplitude'], means) + assert view.get_display_data(1) is data + view.set_display_bin_seconds(0) + assert view.get_display_data(1) is source + + +def test_cache_budget_independent_of_limit_and_released(summary_view): + view = summary_view + view.controller._source_units = lambda cluster: [1] + view.display_limit = 0 + budget = 2 * view._data_bytes(view.get_data(-1)) + view._invalidate() + view.summary_cache_bytes = budget + for cluster in range(20): + data = view.get_display_data(cluster) + assert data.counts.sum() == 7 + assert sum(map(view._data_bytes, view._selected_summaries.values())) <= budget + assert 19 in view._selected_summaries and 0 not in view._selected_summaries + view.summary_cache_bytes = 0 + assert view.get_data(30).counts.sum() == 7 + assert not view._selected_summaries + view._invalidate() + assert not view._selected_summaries + + +def test_6156_bins_many_units_cached_operation_bounds(summary_view, monkeypatch): + view = summary_view + bins = 6156 + common = view._common() + common.edges = np.arange(bins + 1) * 300.0 + common.exposure = np.full(bins, 300.0) + common.summary = {'counts_file': 'virtual_counts'} + common.intervals = [[0, bins * 300]] + rows = [] + + class Grid: + def __getitem__(self, row): + rows.append(row) + return np.ones(bins, dtype=np.int64) + + view._summary_arrays['virtual_counts'] = Grid() + view.controller._source_units = lambda cluster: [1] + view.show_amplitude = view.show_depth = False + view.on_select(list(range(12))) + assert len(rows) == 12 + assert len(next(view.canvas.iter_ax()).lines) == 12 + assert all(len(line.get_ydata()) == bins for line in next(view.canvas.iter_ax()).lines) + monkeypatch.setattr(view, '_array', lambda name: pytest.fail('Must reuse cached source rows')) + view.set_display_bin_seconds(900) + view.plot() + assert len(rows) == 12 + assert all(len(line.get_ydata()) == bins // 3 for line in next(view.canvas.iter_ax()).lines) + view.set_display_bin_seconds(0) + assert len(rows) == 12 + + +def test_display_gap_and_unattached_cache_release(summary_view): + view = summary_view + common = view._common() + common.exposure = np.array([0.0, 0.0, 3.0]) + view.set_display_bin_seconds(20) + data = view.get_display_data(4) + assert data.exposure[0] == 0 + assert np.isnan(data.rates[0]) + assert np.isnan(data.trends['amplitude'][0]) + assert np.isnan(data.trends['depth'][0]) + assert view._selected_summaries and view._summary_arrays + view.close() + assert not view._selected_summaries and not view._summary_arrays + assert view._common_data is None + + +def test_pending_summaries_also_obey_cache_budget(summary_view): + view = summary_view + view.controller._source_units = lambda cluster: None + pending = view.get_data(5) + view.summary_cache_bytes = 2 * view._data_bytes(pending) + for cluster in range(20): + assert view.get_data(cluster).pending + assert len(view._selected_summaries) <= 2 + assert ( + sum(map(view._data_bytes, view._selected_summaries.values())) <= view.summary_cache_bytes + ) + + +@pytest.mark.parametrize( + 'edges', + [ + [0, 10, 25, 30], + [0, 10, 20, 35], + [0, 10, 20, 20], + [0, 10, 20, np.inf], + ], +) +def test_seconds_coarsening_rejects_nonuniform_or_invalid_native_bins(summary_view, edges): + view = summary_view + view._common().edges = np.asarray(edges, dtype=float) + assert view._display_factor(0) == 1 + with pytest.raises(ValueError, match='requires uniform native bins'): + view.set_display_bin_seconds(20) + assert view.display_bin_seconds == 0 + view.set_state({'display_bin_seconds': 20}) + assert view.display_bin_seconds == 0 + + +def test_seconds_coarsening_supports_short_final_native_bin(summary_view): + view = summary_view + # The fixture's native widths are 10, 10, 5 seconds. + assert view._display_factor(20) == 2 + view.set_display_bin_seconds(20) + data = view.get_display_data(1) + np.testing.assert_array_equal(data.edges, [0, 20, 25]) + np.testing.assert_array_equal(data.counts, [4, 3]) + np.testing.assert_array_equal(data.exposure, [8, 3]) diff --git a/phy/apps/large_recording/tests/test_statistics.py b/phy/apps/large_recording/tests/test_statistics.py new file mode 100644 index 000000000..1bee50516 --- /dev/null +++ b/phy/apps/large_recording/tests/test_statistics.py @@ -0,0 +1,43 @@ +"""Independent pair-count oracle and observation-boundary regressions.""" + +import numpy as np + +from phy.apps.large_recording.statistics import window_correlograms, window_isi + + +def test_window_isi_excludes_missing_observation_periods_and_preserves_ties(): + samples = np.array([0, 1, 1, 2, 1000, 1002], dtype=np.int64) + windows = np.array([[0, 3], [1000, 1003]], dtype=np.int64) + np.testing.assert_array_equal(window_isi(samples, windows, 1), [1, 0, 1, 2]) + + +def test_late_sample_ccg_matches_all_pairs_oracle_with_sparse_ids(): + origin = 1000 * 3600 * 30000 + samples = origin + np.array([0, 1, 30, 30, 31, 60, 300, 301, 330], dtype=np.int64) + clusters = np.array([2_000_000_000, 2, 2, 2_000_000_000, 2, 2, 2, 2_000_000_000, 2]) + windows = np.array([[origin, origin + 100], [origin + 300, origin + 400]], dtype=np.int64) + ids = [2_000_000_000, 2] + actual = window_correlograms(samples, clusters, windows, ids, 30000, 0.001, 0.004) + expected = np.zeros((2, 2, 5), dtype=np.int64) + lookup = {cluster: index for index, cluster in enumerate(ids)} + for start, end in windows: + rows = [i for i, tick in enumerate(samples) if start <= tick < end] + for index, left in enumerate(rows): + for right in rows[index + 1 :]: + delay = int(samples[right] - samples[left]) // 30 + if delay > 2: + continue + a, b = lookup[clusters[left]], lookup[clusters[right]] + expected[a, b, 2 + delay] += 1 + if a != b or delay != 0: + expected[b, a, 2 - delay] += 1 + np.testing.assert_array_equal(actual, expected) + + +def test_cross_unit_central_pairs_do_not_depend_on_tied_row_order(): + samples = np.array([0, 0, 10, 10], dtype=np.int64) + windows = np.array([[0, 20]], dtype=np.int64) + a = window_correlograms(samples, np.array([0, 1, 1, 0]), windows, [0, 1], 30000, 0.001, 0.004) + b = window_correlograms(samples, np.array([1, 0, 1, 0]), windows, [0, 1], 30000, 0.001, 0.004) + np.testing.assert_array_equal(a, b) + assert a[0, 1, 2] == 4 diff --git a/phy/apps/large_recording/tests/test_summaries.py b/phy/apps/large_recording/tests/test_summaries.py new file mode 100644 index 000000000..5eb4d03e2 --- /dev/null +++ b/phy/apps/large_recording/tests/test_summaries.py @@ -0,0 +1,67 @@ +"""Full counts and acquisition exposure compared with an independent tiny oracle.""" + +import numpy as np +import pytest + +from phy.apps.large_recording.source import Source +from phy.apps.large_recording.summaries import build_summaries +from phy.apps.large_recording.tests.test_source import make_sorting + + +def test_exact_counts_exposure_and_additive_measurements(tmp_path): + params = make_sorting(tmp_path / 'full', columns=True) + amplitude = np.load(params.parent / 'amplitudes.npy') + amplitude[1] = np.nan + np.save(params.parent / 'amplitudes.npy', amplitude[:, None]) + source = Source(params, valid_intervals=[[0, 7], [10, 20]], batch_size=2) + files = build_summaries(source, tmp_path / 'summary', bin_seconds=5) + counts = np.load(tmp_path / 'summary' / files['counts_file']) + expected = np.zeros((4, 4), dtype=np.int64) + expected_sum = np.zeros((4, 4)) + expected_n = np.zeros((4, 4), dtype=np.int64) + units = np.load(params.parent / 'spike_clusters.npy').reshape(-1) + times = np.load(params.parent / 'spike_times.npy').reshape(-1) + for unit, time, value in zip(units, times, amplitude, strict=True): + row = list(source.unit_ids).index(unit) + column = int(time // 5) + expected[row, column] += 1 + if np.isfinite(value): + expected_sum[row, column] += value + expected_n[row, column] += 1 + np.testing.assert_array_equal(counts, expected) + np.testing.assert_array_equal(counts.sum(axis=1), source.counts) + np.testing.assert_array_equal( + np.load(tmp_path / 'summary' / files['edges_file']), [0, 5, 10, 15, 20] + ) + np.testing.assert_array_equal( + np.load(tmp_path / 'summary' / files['exposure_file']), [5, 2, 5, 5] + ) + np.testing.assert_array_equal( + np.load(tmp_path / 'summary' / files['amplitude_sum_file']), expected_sum + ) + np.testing.assert_array_equal( + np.load(tmp_path / 'summary' / files['amplitude_count_file']), expected_n + ) + assert files['measurement_units']['amplitude'] == 'source amplitude units' + assert 'depth_sum_file' in files + + +def test_summary_budget_checked_before_output_allocation(tmp_path): + source = Source(make_sorting(tmp_path / 'full')) + with pytest.raises(ValueError, match='Summary budget'): + build_summaries(source, tmp_path / 'summary', bin_seconds=1, max_bytes=50) + assert not (tmp_path / 'summary').exists() + + +def test_position_coordinates_require_declared_depth_axis(tmp_path): + from phy.apps.large_recording.summaries import scalar_attributes + + params = make_sorting(tmp_path / 'full') + (params.parent / 'spike_depths.npy').unlink() + with Source(params) as source: + assert 'depth' not in scalar_attributes(source) + params.write_text(params.read_text() + 'depth_axis = 1\n') + with Source(params) as source: + np.testing.assert_array_equal( + scalar_attributes(source)['depth'], source.arrays['spike_positions.npy'][:, 1] + ) diff --git a/phy/apps/large_recording/views.py b/phy/apps/large_recording/views.py new file mode 100644 index 000000000..d4877a15f --- /dev/null +++ b/phy/apps/large_recording/views.py @@ -0,0 +1,78 @@ +"""Explicitly sampled statistics and source-amplitude evidence views.""" + +import numpy as np + +from phy.cluster.views import AmplitudeView, CorrelogramView, ISIView +from phy.gui.qt import QLabel, Qt + + +class SampledISIView(ISIView): + ex_status = 'Selected windows only; intervals crossing window boundaries excluded.' + + +class SampledCorrelogramView(CorrelogramView): + ex_status = 'Selected windows only; raw pair counts, no full-recording rate baseline.' + + +class EvidenceAmplitudeView(AmplitudeView): + """Sampled source amplitudes with a full-recording clock or local window.""" + + def __init__(self, *args, **kwargs): + self._amplitude_scope = 'overview' + super().__init__(*args, **kwargs) + self.state_attrs += ('amplitude_scope',) + + @property + def amplitude_scope(self): + return self._amplitude_scope + + @amplitude_scope.setter + def amplitude_scope(self, scope): + if scope not in ('overview', 'window'): + raise ValueError('Amplitude scope must be overview or window.') + self.clear_amplitude_split_threshold() + self._amplitude_scope = scope + if hasattr(self, '_controller'): + self._controller._update_amplitude_scope(self) + + def toggle_amplitude_scope(self): + """Toggle the full recording overview and current evidence window.""" + self.amplitude_scope = 'window' if self.amplitude_scope == 'overview' else 'overview' + + def on_request_split(self, sender=None): + # Bounded commits consume a semantic predicate, never sampled row IDs. + return np.array([], dtype=np.int64) + + window_caption = 'Recording time unavailable' + + def set_window_context(self, index, start_seconds, end_seconds): + self.window_caption = ( + f'Evidence window {index} | Recording time: {start_seconds:.3f}–{end_seconds:.3f} s' + ) + if hasattr(self, 'window_label'): + self.window_label.setText(self.window_caption) + + def attach(self, gui): + super().attach(gui) + self.actions.add(self.toggle_amplitude_scope, show_shortcut=False) + layout = self.dock.widget().layout() + self.window_label = QLabel(self.window_caption) + self.window_label.setWordWrap(True) + self.time_axis_label = QLabel( + 'Recording time (seconds)' + if self.amplitude_scope == 'overview' + else 'Seconds from window start' + ) + for label in (self.window_label, self.time_axis_label): + label.setAlignment(Qt.AlignCenter) + label.setStyleSheet('color: white; background-color: black; padding: 4px;') + layout.insertWidget(0, self.window_label) + layout.insertWidget(layout.count() - 1, self.time_axis_label) + + def _get_data_bounds(self, bunchs): + if not any(len(bunch.amplitudes) for bunch in bunchs): + return (0, 0, self.duration, 1) + xmin, ymin, xmax, ymax = super()._get_data_bounds(bunchs) + if ymin == ymax: + ymin, ymax = ymin - 0.5, ymax + 0.5 + return xmin, ymin, xmax, ymax diff --git a/phy/apps/template/gui.py b/phy/apps/template/gui.py index bda606c42..d8c1db3ba 100644 --- a/phy/apps/template/gui.py +++ b/phy/apps/template/gui.py @@ -81,6 +81,14 @@ class TemplateController(WaveformMixin, FeatureMixin, TemplateMixin, TraceMixin, # Internal methods # ------------------------------------------------------------------------- + def __new__(cls, dir_path=None, *args, **kwargs): + if cls is TemplateController and dir_path is not None: + if (Path(dir_path) / '_phy_curation_manifest.json').exists(): + from ..large_recording.adapter import BoundedTemplateController + + return object.__new__(BoundedTemplateController) + return object.__new__(cls) + def _get_waveforms_dict(self): waveforms_dict = super()._get_waveforms_dict() # Remove waveforms and mean_waveforms if there is no raw data file. @@ -142,6 +150,7 @@ def _get_template_features(self, cluster_ids, load_all=False): def _set_view_creator(self): super()._set_view_creator() self.view_creator['TemplateFeatureView'] = self.create_template_feature_view + self.view_creator['UnitStabilityView'] = self.create_unit_stability_view # Public methods # ------------------------------------------------------------------------- @@ -255,6 +264,12 @@ def create_template_feature_view(self): return return TemplateFeatureView(coords=self._get_template_features) + def create_unit_stability_view(self): + """Optionally inspect selected-unit activity over the loaded recording.""" + from ..large_recording.stability import UnitStabilityView + + return UnitStabilityView(self) + # ------------------------------------------------------------------------------ # Template commands @@ -268,9 +283,14 @@ def template_gui(params_path, **kwargs): # pragma: no cover dir_path = p.parent _add_log_file(dir_path / 'phy.log') - model = load_model(params_path) + bounded = (dir_path / '_phy_curation_manifest.json').exists() + model = None if bounded else load_model(params_path) # Automatically export spike waveforms when using compressed raw ephys. - if model.spike_waveforms is None and isinstance(model.traces, MtscompEphysReader): + if ( + model is not None + and model.spike_waveforms is None + and isinstance(model.traces, MtscompEphysReader) + ): # TODO: customizable values below. model.save_spikes_subset_waveforms(max_n_spikes_per_template=500, max_n_channels=16) @@ -285,6 +305,22 @@ def template_gui(params_path, **kwargs): # pragma: no cover def template_describe(params_path): """Describe a template dataset.""" + if (Path(params_path).parent / '_phy_curation_manifest.json').exists(): + from ..large_recording.curation import BoundedSession + + session = BoundedSession(Path(params_path).parent) + try: + print(f'Bounded evidence: {len(session.assignments):,} spikes') + print( + f'Complete source: {session.manifest["num_spikes"]:,} spikes, {len(session.unit_ids):,} units' + ) + print( + f'Duration: {session.manifest["num_samples"] / session.manifest["sample_rate"]:.3f} s' + ) + print(f'Committed curation: {len(session.events)} semantic operations') + finally: + session.close() + return model = load_model(params_path) model.describe() model.close() diff --git a/phy/apps/tests/large_recording_fixtures.py b/phy/apps/tests/large_recording_fixtures.py new file mode 100644 index 000000000..7041213bd --- /dev/null +++ b/phy/apps/tests/large_recording_fixtures.py @@ -0,0 +1,153 @@ +"""Small, deterministic on-disk example of a 1000-hour two-stage Phy dataset. + +This is reference test data, not a prepare-large implementation. The fake +recording has 32 spikes spread across 1000 hours, not 1000 hours of raw traces. +The full and bounded folders use ordinary Phy filenames. +""" + +import hashlib +import json +import shutil +from dataclasses import dataclass +from pathlib import Path + +import numpy as np + + +@dataclass(frozen=True) +class LongRecordingFixture: + full: Path + bounded: Path + global_ids: np.ndarray + upstream_waveform_ids: np.ndarray + bounded_waveform_ids: np.ndarray + + +def _sha256(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _reference_remap(global_ids, upstream_ids): + """Reference intersection for sorted, unique, nonnegative int64 row IDs.""" + global_ids = np.asarray(global_ids) + upstream_ids = np.asarray(upstream_ids) + for ids in (global_ids, upstream_ids): + if ids.ndim != 1 or ids.dtype != np.dtype('int64'): + raise ValueError('Row IDs must be one-dimensional int64 arrays.') + if np.any(ids < 0) or np.any(ids[1:] <= ids[:-1]): + raise ValueError('Row IDs must be nonnegative, sorted, and unique.') + if not len(global_ids) or not len(upstream_ids): + empty = np.array([], dtype=np.int64) + return empty, empty.copy() + positions = np.searchsorted(global_ids, upstream_ids) + keep = positions < len(global_ids) + keep[keep] &= global_ids[positions[keep]] == upstream_ids[keep] + return positions[keep].astype(np.int64), np.flatnonzero(keep).astype(np.int64) + + +def make_long_recording_fixture(root): + """Write one full Phy dataset and its expected bounded counterpart.""" + root = Path(root) + full, bounded = root / 'full', root / 'bounded' + full.mkdir(parents=True) + bounded.mkdir(parents=True) + + fs = 30_000 + num_samples = 1000 * 3600 * fs # Far beyond signed int32 sample indices. + samples = np.linspace(100, num_samples - 100, 32, dtype=np.int64) + samples[8] = samples[5] # Simultaneous spikes retain distinct source row identities. + samples[6:8] = samples[5] + units = (np.arange(32) % 3).astype(np.int32) + amplitudes = np.arange(1, 33, dtype=np.float32) + templates = np.zeros((3, 12, 4), dtype=np.float32) + for unit in range(3): + templates[unit, :, unit] = -np.arange(12, dtype=np.float32) + + for name, array in ( + ('spike_times.npy', samples), + ('spike_clusters.npy', units), + ('spike_templates.npy', units), + ('amplitudes.npy', amplitudes), + ('templates.npy', templates), + ('channel_map.npy', np.arange(4, dtype=np.int32)), + ('channel_positions.npy', np.array([[0, 0], [20, 0], [0, 20], [20, 20]])), + ): + np.save(full / name, array) + (full / 'params.py').write_text( + "sample_rate = 30000.0\ndat_path = None\nn_channels_dat = 4\ndtype = 'int16'\n", + encoding='utf-8', + ) + + upstream_ids = np.array([1, 2, 8, 13, 18, 21, 28, 31], dtype=np.int64) + channels = np.array( + [[0, 1, -1] if i % 2 == 0 else [2, 3, -1] for i in range(len(upstream_ids))], + dtype=np.int32, + ) + waveforms = np.zeros((len(upstream_ids), 12, 3), dtype=np.float32) + for row, spike_id in enumerate(upstream_ids): + waveforms[row, :, 0] = spike_id + np.arange(12) / 12 + waveforms[row, :, 1] = -(spike_id + np.arange(12) / 12) + stem = '_phy_spikes_subset' + np.save(full / f'{stem}.spikes.npy', upstream_ids) + np.save(full / f'{stem}.channels.npy', channels) + np.save(full / f'{stem}.waveforms.npy', waveforms) + + # Proposed, optional upstream window metadata: simple homogeneous arrays. + starts = np.minimum(samples[upstream_ids] - 10, num_samples - 30 * fs) + bounds = np.column_stack((starts, starts + 30 * fs)).astype(np.int64) + types = np.zeros(len(upstream_ids), dtype=np.uint8) + types[-1] = 1 # rescue + strata = (samples[upstream_ids] // (3600 * fs)).astype(np.int32) + strata[-1] = -1 # rescue has no survey stratum + np.save(full / f'{stem}.window_bounds.npy', bounds) + np.save(full / f'{stem}.window_types.npy', types) + np.save(full / f'{stem}.window_strata.npy', strata) + source_hash = _sha256(full / 'spike_times.npy') + (full / f'{stem}.manifest.json').write_text( + json.dumps( + { + 'format': 'phy_spikes_subset', + 'format_version': 1, + 'source_sorting': {'num_spikes': len(samples), 'spike_times_sha256': source_hash}, + 'recording': { + 'sampling_frequency_hz': fs, + 'num_samples': num_samples, + 'num_channels': 4, + 'num_segments': 1, + }, + 'selection': {'algorithm': 'synthetic_windows_v1', 'seed': 42}, + 'waveform_extraction': {'nbefore_samples': 4, 'nafter_samples': 8}, + 'window_types': {'0': 'survey', '1': 'rescue'}, + }, + indent=2, + ) + + '\n', + encoding='utf-8', + ) + + global_ids = np.array([0, 2, 5, 8, 13, 21, 25, 31], dtype=np.int64) + for name in ('spike_times.npy', 'spike_clusters.npy', 'spike_templates.npy', 'amplitudes.npy'): + np.save(bounded / name, np.load(full / name)[global_ids]) + for name in ('templates.npy', 'channel_map.npy', 'channel_positions.npy', 'params.py'): + shutil.copyfile(full / name, bounded / name) + np.save(bounded / '_phy_spike_global_ids.npy', global_ids) + + bounded_rows, waveform_rows = _reference_remap(global_ids, upstream_ids) + np.save(bounded / f'{stem}.spikes.npy', bounded_rows) + np.save(bounded / f'{stem}.channels.npy', channels[waveform_rows]) + np.save(bounded / f'{stem}.waveforms.npy', waveforms[waveform_rows]) + (bounded / '_phy_curation_manifest.json').write_text( + json.dumps( + { + 'format': 'phy_bounded_curation', + 'format_version': 1, + 'source': {'num_spikes': len(samples), 'spike_times_sha256': source_hash}, + 'global_spike_ids_file': '_phy_spike_global_ids.npy', + 'duration_seconds': 1000 * 3600, + }, + indent=2, + ) + + '\n', + encoding='utf-8', + ) + return LongRecordingFixture(full, bounded, global_ids, upstream_ids, bounded_rows) diff --git a/phy/apps/tests/test_large_recording_contract.py b/phy/apps/tests/test_large_recording_contract.py new file mode 100644 index 000000000..8e882afc1 --- /dev/null +++ b/phy/apps/tests/test_large_recording_contract.py @@ -0,0 +1,151 @@ +"""Reference contract tests for future prepare-large and waveform remapping. + +These tests deliberately do not exercise a production prepare-large command: +that command does not exist yet. Later implementations should reuse this +fixture and replace reference-only checks with command-level assertions. +""" + +import hashlib +import json + +import numpy as np +import pytest +from phylib.io.model import load_model + +from .large_recording_fixtures import _reference_remap, make_long_recording_fixture + + +@pytest.fixture +def long_recording(tmp_path): + return make_long_recording_fixture(tmp_path) + + +def test_long_recording_uses_exact_64_bit_sample_times(long_recording): + full = np.load(long_recording.full / 'spike_times.npy') + bounded = np.load(long_recording.bounded / 'spike_times.npy') + assert full.dtype == bounded.dtype == np.dtype('int64') + assert full[-1] > np.iinfo(np.int32).max + assert full[-1] / 30_000 > 999 * 3600 + np.testing.assert_array_equal(bounded, full[long_recording.global_ids]) + assert np.all(np.diff(long_recording.global_ids) > 0) + assert np.all(np.diff(bounded) >= 0) + assert bounded[2] == bounded[3] # equal times do not collapse distinct spikes + + +def test_two_stage_row_indices_and_waveform_data(long_recording): + full = long_recording.full + bounded = long_recording.bounded + global_ids = np.load(bounded / '_phy_spike_global_ids.npy') + full_waveform_ids = np.load(full / '_phy_spikes_subset.spikes.npy') + bounded_waveform_ids = np.load(bounded / '_phy_spikes_subset.spikes.npy') + assert global_ids.dtype == bounded_waveform_ids.dtype == np.dtype('int64') + np.testing.assert_array_equal( + global_ids[bounded_waveform_ids], np.intersect1d(full_waveform_ids, global_ids) + ) + upstream_rows = np.searchsorted(full_waveform_ids, global_ids[bounded_waveform_ids]) + for suffix in ('channels', 'waveforms'): + np.testing.assert_array_equal( + np.load(bounded / f'_phy_spikes_subset.{suffix}.npy'), + np.load(full / f'_phy_spikes_subset.{suffix}.npy')[upstream_rows], + ) + # A bounded row identifies one source row, and never a waveform tensor row. + assert bounded_waveform_ids.tolist() == [1, 3, 4, 5, 7] + assert global_ids[bounded_waveform_ids].tolist() == [2, 8, 13, 21, 31] + + +def test_ordinary_phy_waveform_files_are_aligned(long_recording): + bounded = long_recording.bounded + spike_ids = np.load(bounded / '_phy_spikes_subset.spikes.npy') + channels = np.load(bounded / '_phy_spikes_subset.channels.npy') + waveforms = np.load(bounded / '_phy_spikes_subset.waveforms.npy', mmap_mode='r') + assert spike_ids.ndim == 1 and np.all(np.diff(spike_ids) > 0) + assert channels.dtype == np.dtype('int32') + assert channels.shape == (len(spike_ids), waveforms.shape[2]) + assert waveforms.shape == (len(spike_ids), 12, 3) + assert np.all((channels == -1) | ((channels >= 0) & (channels < 4))) + assert np.all(waveforms[:, :, -1] == 0) # -1 means a zero-padded channel + assert spike_ids[-1] < len(np.load(bounded / 'spike_times.npy')) + + +def test_window_arrays_use_samples_and_simple_dtypes(long_recording): + full = long_recording.full + bounds = np.load(full / '_phy_spikes_subset.window_bounds.npy') + types = np.load(full / '_phy_spikes_subset.window_types.npy') + strata = np.load(full / '_phy_spikes_subset.window_strata.npy') + spike_samples = np.load(full / 'spike_times.npy') + assert bounds.shape == (len(types), 2) + assert len(types) == len(strata) + assert bounds.dtype == np.dtype('int64') + assert types.dtype == np.dtype('uint8') + assert strata.dtype == np.dtype('int32') + assert np.all(bounds[:, 1] > bounds[:, 0]) + assert np.all(bounds[:, 1] <= 1000 * 3600 * 30_000) + assert np.all((types == 0) | (types == 1)) + assert strata[-1] == -1 and types[-1] == 1 + source_waveform_times = spike_samples[long_recording.upstream_waveform_ids] + assert np.all((bounds[:, 0] <= source_waveform_times) & (source_waveform_times < bounds[:, 1])) + + +def test_reference_remap_handles_empty_and_disjoint_inputs(): + a = np.array([14, 19, 117], dtype=np.int64) + b = np.array([2, 4, 119], dtype=np.int64) + for left, right in ((a, b), (a, b[:0]), (a[:0], b), (a[:0], b[:0])): + bounded_rows, source_rows = _reference_remap(left, right) + assert bounded_rows.dtype == source_rows.dtype == np.dtype('int64') + assert bounded_rows.shape == source_rows.shape == (0,) + rows, upstream_rows = _reference_remap(a, np.array([14, 15, 117, 118])) + assert rows.tolist() == [0, 2] + assert upstream_rows.tolist() == [0, 2] + + +@pytest.mark.parametrize('ids', [[2, 1], [1, 1], [-1, 1]]) +def test_reference_remap_rejects_invalid_row_order(ids): + valid = np.array([1, 2], dtype=np.int64) + invalid = np.array(ids, dtype=np.int64) + for left, right in ((valid, invalid), (invalid, valid)): + with pytest.raises(ValueError, match='nonnegative, sorted, and unique'): + _reference_remap(left, right) + + +def test_manifests_bind_source_index_space_not_mutable_clusters(long_recording): + full = long_recording.full + bounded = long_recording.bounded + source_hash = hashlib.sha256((full / 'spike_times.npy').read_bytes()).hexdigest() + upstream = json.loads((full / '_phy_spikes_subset.manifest.json').read_text()) + downstream = json.loads((bounded / '_phy_curation_manifest.json').read_text()) + assert upstream['source_sorting']['spike_times_sha256'] == source_hash + assert downstream['source']['spike_times_sha256'] == source_hash + assert upstream['source_sorting']['num_spikes'] == len(np.load(full / 'spike_times.npy')) + assert downstream['global_spike_ids_file'] == '_phy_spike_global_ids.npy' + assert downstream['duration_seconds'] == 1000 * 3600 + assert hashlib.sha256((bounded / 'spike_times.npy').read_bytes()).hexdigest() != source_hash + + +def test_bounded_dataset_loads_with_existing_phylib(long_recording): + """The old Phy loader consumes bounded rows, not original global indices.""" + model = load_model(long_recording.bounded / 'params.py') + try: + np.testing.assert_array_equal( + model.spike_waveforms.spike_ids, long_recording.bounded_waveform_ids + ) + assert model.n_spikes == len(long_recording.global_ids) + assert model.n_samples_waveforms == 12 + assert model.spike_times[-1] > 999 * 3600 + np.testing.assert_array_equal( + model.get_waveforms(long_recording.bounded_waveform_ids, channel_ids=[0, 1, 2, 3]), + _expected_dense_waveforms(long_recording.bounded), + ) + finally: + model.close() + + +def _expected_dense_waveforms(path): + """Independent dense expansion of the fixture's sparse channel slots.""" + channels = np.load(path / '_phy_spikes_subset.channels.npy') + waveforms = np.load(path / '_phy_spikes_subset.waveforms.npy') + expected = np.zeros((len(channels), 12, 4), dtype=waveforms.dtype) + for row, channel_row in enumerate(channels): + for slot, channel in enumerate(channel_row): + if channel >= 0: + expected[row, :, channel] = waveforms[row, :, slot] + return expected diff --git a/phy/cluster/_propositions.py b/phy/cluster/_propositions.py index dfd1710a9..ffc8f07a6 100644 --- a/phy/cluster/_propositions.py +++ b/phy/cluster/_propositions.py @@ -182,6 +182,7 @@ class MergePropositionCatalog: reviews: Mapping[str, PropositionReview] = field(default_factory=dict) live_unit_ids: tuple[int, ...] | None = None source_mapping: Mapping = field(default_factory=dict) + retired_units: Mapping = field(default_factory=dict) def __post_init__(self): source_ids = _ids(self.source_unit_ids, minimum=0) @@ -208,6 +209,23 @@ def __post_init__(self): object.__setattr__(self, 'reviews', MappingProxyType(reviews)) object.__setattr__(self, 'live_unit_ids', live_ids) object.__setattr__(self, 'source_mapping', source_mapping) + retired = {} + for key, value in _mapping(self.retired_units, name='retired_units').items(): + if not isinstance(key, str) or not key.isdecimal() or str(int(key)) != key: + raise ValueError( + 'retired_units keys must be canonical non-negative integer strings.' + ) + value = _mapping(value, name='retired unit') + operation = value.get('operation') + if operation not in {'merge', 'threshold_split'}: + raise ValueError('Unknown retired-unit operation.') + replacements = _ids(value.get('replacements'), name='replacements', minimum=1) + if any(unit < 0 for unit in replacements) or int(key) in replacements: + raise ValueError( + 'Retired-unit replacements must be distinct non-negative new IDs.' + ) + retired[key] = {'operation': operation, 'replacements': replacements} + object.__setattr__(self, 'retired_units', _freeze(retired)) @property def propositions(self): @@ -259,11 +277,22 @@ def reason_for(self, key): for unit_id in entry.proposition.unit_ids if unit_id not in self.live_unit_ids ) + causes = [] + for unit_id in missing: + retirement = self.retired_units.get(str(unit_id)) + if retirement is not None: + verb = 'merged' if retirement['operation'] == 'merge' else 'split' + causes.append(f'{unit_id} was {verb}') + if causes: + return f'Source clusters no longer exist: {missing}; ' + '; '.join(causes) + '.' return f'One or more source clusters no longer exist: {missing}.' return None - def project_live_ids(self, live_unit_ids): - return self._replace(live_unit_ids=_ids(live_unit_ids, name='live_unit_ids')) + def project_live_ids(self, live_unit_ids, *, retired_units=None): + changes = {'live_unit_ids': _ids(live_unit_ids, name='live_unit_ids')} + if retired_units is not None: + changes['retired_units'] = retired_units + return self._replace(**changes) def snapshot(self): """Return an immutable snapshot suitable for history entries.""" @@ -323,6 +352,7 @@ def _replace(self, **changes): 'reviews': self.reviews, 'live_unit_ids': self.live_unit_ids, 'source_mapping': self.source_mapping, + 'retired_units': self.retired_units, } values.update(changes) return type(self)(**values) @@ -355,8 +385,10 @@ def restore(self, snapshot): self._catalog = self._require_catalog(snapshot) return None - def project_live_ids(self, live_unit_ids): - self._catalog = self._catalog.project_live_ids(tuple(map(int, live_unit_ids))) + def project_live_ids(self, live_unit_ids, *, retired_units=None): + self._catalog = self._catalog.project_live_ids( + tuple(map(int, live_unit_ids)), retired_units=retired_units + ) return None def reject(self, key): @@ -422,7 +454,7 @@ def _review_fingerprint(catalog): ) -def decode_curation_mapping(mapping) -> MergePropositionCatalog: +def decode_curation_mapping(mapping, *, retired_units=None) -> MergePropositionCatalog: """Decode a supported AIND/SpikeInterface v2 curation document. Invalid individual ``merges`` entries become visible invalid catalog entries; @@ -453,7 +485,12 @@ def decode_curation_mapping(mapping) -> MergePropositionCatalog: entries.append(PropositionEntry(index, proposition=proposition, raw_mapping=raw)) except (TypeError, ValueError) as e: entries.append(PropositionEntry(index, invalid_reason=str(e), raw_mapping=raw)) - return MergePropositionCatalog(source_ids, tuple(entries), source_mapping=mapping) + return MergePropositionCatalog( + source_ids, + tuple(entries), + source_mapping=mapping, + retired_units={} if retired_units is None else retired_units, + ) def encode_curation_mapping(catalog: MergePropositionCatalog): diff --git a/phy/cluster/clustering.py b/phy/cluster/clustering.py index 662ca7439..32fa7480e 100644 --- a/phy/cluster/clustering.py +++ b/phy/cluster/clustering.py @@ -7,7 +7,8 @@ import logging import numpy as np -from phylib.io.array import _spikes_in_clusters, _spikes_per_cluster, _unique +from phylib.io.array import _spikes_in_clusters, _spikes_per_cluster +from phylib.io.array import _unique as _unique_dense from phylib.utils._types import _as_array, _is_array_like from phylib.utils.event import emit @@ -22,6 +23,23 @@ # ------------------------------------------------------------------------------ +def _unique(values): + """Preserve dense-ID performance without allocating by sparse absolute IDs.""" + if values is None or len(values) == 0: + return np.array([], dtype=np.int64) + values = np.asarray(values) + if int(values.max()) > max(1024, 4 * len(values)): + return np.unique(values[values >= 0]).astype(np.int64) + return _unique_dense(values) + + +def _largest_cluster(values): + if int(values.max()) > max(1024, 4 * len(values)): + ids, counts = np.unique(values, return_counts=True) + return int(ids[counts.argmax()]) + return int(np.bincount(values).argmax()) + + def _extend_spikes(spike_ids, spike_clusters): """Return all spikes belonging to the clusters containing the specified spikes.""" @@ -81,7 +99,7 @@ def _extend_assignment(spike_ids, old_spike_clusters, spike_clusters_rel, new_cl def _assign_update_info(spike_ids, old_spike_clusters, new_spike_clusters): old_clusters = _unique(old_spike_clusters) new_clusters = _unique(new_spike_clusters) - largest_old_cluster = np.bincount(old_spike_clusters).argmax() + largest_old_cluster = _largest_cluster(old_spike_clusters) descendants = list(set(zip(old_spike_clusters, new_spike_clusters))) update_info = UpdateInfo( description='assign', @@ -293,7 +311,9 @@ def _do_assign(self, spike_ids, new_spike_clusters): def _do_merge(self, spike_ids, cluster_ids, to): # Create the UpdateInfo instance here. descendants = [(cluster, to) for cluster in cluster_ids] - largest_old_cluster = np.bincount(self.spike_clusters[spike_ids]).argmax() + largest_old_cluster = max( + cluster_ids, key=lambda cluster: (len(self._spikes_per_cluster[cluster]), -cluster) + ) up = UpdateInfo( description='merge', spike_ids=list(spike_ids), diff --git a/phy/cluster/supervisor.py b/phy/cluster/supervisor.py index 54c5988d1..5b0b4a3f9 100644 --- a/phy/cluster/supervisor.py +++ b/phy/cluster/supervisor.py @@ -755,6 +755,11 @@ class Supervisor: skip_masked_clusters : bool Whether automatic navigation and similar-cluster selection skip noise and MUA clusters. The default is True. + clustering_factory : callable, optional + Construct the clustering model. Defaults to Clustering; bounded curation + supplies a guarded implementation with the same constructor interface. + cluster_meta_factory : callable, optional + Construct metadata from groups and labels. Defaults to create_cluster_meta. Events ------ @@ -790,6 +795,8 @@ def __init__( n_similar_clusters_to_select=None, skip_masked_clusters=True, merge_propositions=None, + clustering_factory=None, + cluster_meta_factory=None, ): super().__init__() self.context = context @@ -833,7 +840,7 @@ def __init__( # Create Clustering and ClusterMeta. # Load the cached spikes_per_cluster array. spc = context.load('spikes_per_cluster') if context else None - self.clustering = Clustering( + self.clustering = (clustering_factory or Clustering)( spike_clusters, spikes_per_cluster=spc, new_cluster_id=new_cluster_id ) @@ -841,7 +848,9 @@ def __init__( self._save_spikes_per_cluster() # Create the ClusterMeta instance. - self.cluster_meta = create_cluster_meta(cluster_groups or {}, self.cluster_labels) + self.cluster_meta = (cluster_meta_factory or create_cluster_meta)( + cluster_groups or {}, self.cluster_labels + ) # Create the GlobalHistory instance. self._global_history = GlobalHistory( diff --git a/phy/cluster/tests/test_clustering.py b/phy/cluster/tests/test_clustering.py index 9557221d7..ef8d03beb 100644 --- a/phy/cluster/tests/test_clustering.py +++ b/phy/cluster/tests/test_clustering.py @@ -26,6 +26,28 @@ # ------------------------------------------------------------------------------ +def test_sparse_absolute_cluster_ids_do_not_allocate_dense_histograms(monkeypatch): + original = np.bincount + + def bounded_bincount(values, *args, **kwargs): + assert not len(values) or np.max(values) < 1024 + return original(values, *args, **kwargs) + + monkeypatch.setattr(np, 'bincount', bounded_bincount) + high = 2_000_000_000 + clustering = Clustering(np.array([0, 1, 1, high], dtype=np.int64)) + first = clustering.merge([0, 1]) + assert first.largest_old_cluster == 1 + assert first.added == [high + 1] + second = clustering.merge([high, high + 1]) + assert second.largest_old_cluster == high + 1 + clustering.undo() + ae(clustering.spike_clusters, [high + 1, high + 1, high + 1, high]) + clustering.split([0]) + clustering.undo() + ae(clustering.spike_clusters, [high + 1, high + 1, high + 1, high]) + + def test_extend_spikes_simple(): spike_clusters = np.array([3, 5, 2, 9, 5, 5, 2]) spike_ids = np.array([2, 4, 0]) diff --git a/phy/cluster/tests/test_propositions.py b/phy/cluster/tests/test_propositions.py index 22caf1194..2bd63ae9d 100644 --- a/phy/cluster/tests/test_propositions.py +++ b/phy/cluster/tests/test_propositions.py @@ -30,6 +30,62 @@ def _source(merges=None): } +def test_retired_units_explain_stale_without_retargeting_or_overwriting_reviews(): + source = _source([{'unit_ids': [41, 56]}, {'unit_ids': [56, 72]}]) + catalog = decode_curation_mapping(source) + accepted, pending = [entry.key for entry in catalog.entries] + catalog = catalog.accept(accepted, [41, 56], 100) + retired = { + '41': {'operation': 'merge', 'replacements': [100]}, + '56': {'operation': 'merge', 'replacements': [100]}, + '72': {'operation': 'threshold_split', 'replacements': [101, 102]}, + } + catalog = catalog.project_live_ids([100, 101, 102], retired_units=retired) + assert catalog.status_for(accepted) is PropositionStatus.ACCEPTED + assert catalog.reason_for(accepted) is None + assert catalog.status_for(pending) is PropositionStatus.STALE + assert '56 was merged' in catalog.reason_for(pending) + assert '72 was split' in catalog.reason_for(pending) + assert encode_curation_mapping(catalog) == source + retired['56']['operation'] = 'threshold_split' + assert '56 was merged' in catalog.reason_for(pending) + controller = MergePropositionController(catalog) + controller.project_live_ids([100, 101, 102]) + assert controller.catalog.reason_for(pending) == catalog.reason_for(pending) + + +def test_retired_unit_validation(): + for retired in ( + [], + False, + {'01': {'operation': 'merge', 'replacements': [100]}}, + {'41': {'operation': 'label', 'replacements': [100]}}, + {'41': {'operation': 'merge', 'replacements': [41]}}, + {'41': {'operation': 'merge', 'replacements': []}}, + {'41': {'operation': 'threshold_split', 'replacements': [-1]}}, + ): + with raises(ValueError): + decode_curation_mapping(_source(), retired_units=retired) + + +def test_retirement_projection_undo_and_durable_modified_review(): + catalog = decode_curation_mapping(_source()) + key = catalog.propositions[0].key + controller = MergePropositionController(catalog) + before = controller.snapshot() + retired = {'41': {'operation': 'threshold_split', 'replacements': [100, 101]}} + controller.project_live_ids([56, 72, 100, 101], retired_units=retired) + assert '41 was split' in controller.catalog.reason_for(key) + controller.restore(before) + assert controller.catalog.status_for(key) is PropositionStatus.PENDING + modified = catalog.accept(key, [41, 72], 100).project_live_ids( + [56, 100], retired_units={'41': {'operation': 'merge', 'replacements': [100]}} + ) + assert modified.status_for(key) is PropositionStatus.ACCEPTED_MODIFIED + rejected = catalog.reject(key).project_live_ids([56, 100], retired_units=retired) + assert rejected.status_for(key) is PropositionStatus.REJECTED + + def test_aind_v2_decode_preserves_source_and_proposition_order(): source = _source([{'unit_ids': [56, 41], 'new_unit_id': 1000}]) diff --git a/tools/benchmark_long_recording.py b/tools/benchmark_long_recording.py new file mode 100644 index 000000000..b6bfabeef --- /dev/null +++ b/tools/benchmark_long_recording.py @@ -0,0 +1,398 @@ +"""Reproducible, file-backed capacity benchmark for long-recording services. + +Run from a development environment, for example:: + + uv run python tools/benchmark_long_recording.py --spikes 1000000 + +The benchmark creates synthetic Phy arrays without raw traces, times preparation +and replay in separate child processes, and checks replay against an independent +streamed oracle. It is intentionally capped at two million source spikes. +""" + +import argparse +import json +import os +import platform +import subprocess +import sys +import tempfile +import threading +import time +from pathlib import Path + +import numpy as np + +REPOSITORY = Path(__file__).resolve().parents[1] +if str(REPOSITORY) not in sys.path: + sys.path.insert(0, str(REPOSITORY)) + + +def _file_bytes(path): + path = Path(path) + return sum(item.stat().st_size for item in path.rglob('*') if item.is_file()) + + +def _make_source(path, spikes, units, duration_hours, batch_size): + """Write canonical arrays in bounded chunks, without allocating full vectors.""" + path.mkdir(parents=True) + duration_seconds = duration_hours * 3600 + sample_rate = 30_000 + num_samples = round(duration_seconds * sample_rate) + if num_samples <= 0 or num_samples > np.iinfo(np.int64).max: + raise ValueError('duration-hours produces an invalid int64 sample axis.') + + arrays = { + 'spike_times.npy': np.lib.format.open_memmap( + path / 'spike_times.npy', mode='w+', dtype=np.int64, shape=(spikes,) + ), + 'spike_clusters.npy': np.lib.format.open_memmap( + path / 'spike_clusters.npy', mode='w+', dtype=np.int32, shape=(spikes,) + ), + 'spike_templates.npy': np.lib.format.open_memmap( + path / 'spike_templates.npy', mode='w+', dtype=np.int32, shape=(spikes,) + ), + } + for start in range(0, spikes, batch_size): + stop = min(start + batch_size, spikes) + rows = np.arange(start, stop, dtype=np.int64) + arrays['spike_times.npy'][start:stop] = rows * (num_samples - 1) // spikes + assignments = (rows % units).astype(np.int32) + arrays['spike_clusters.npy'][start:stop] = assignments + arrays['spike_templates.npy'][start:stop] = assignments + for array in arrays.values(): + array.flush() + array._mmap.close() + + templates = np.lib.format.open_memmap( + path / 'templates.npy', mode='w+', dtype=np.float32, shape=(units, 4, 2) + ) + templates[:] = 0 + templates.flush() + templates._mmap.close() + np.save(path / 'channel_map.npy', np.array([0, 1], dtype=np.int32)) + np.save(path / 'channel_positions.npy', np.array([[0.0, 0.0], [20.0, 0.0]], dtype=np.float32)) + (path / 'params.py').write_text( + f'sample_rate = {sample_rate}.0\n' + f'num_samples = {num_samples}\n' + 'n_channels_dat = 2\n' + "dtype = 'int16'\n" + 'dat_path = None\n', + encoding='utf-8', + ) + return duration_seconds + + +def _peak_rss_bytes(): + """Return process peak RSS, or an explicit unavailable result.""" + if os.name != 'nt': + try: + import resource + + value = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss + # Linux and the BSDs report KiB; macOS reports bytes. + return int(value if sys.platform == 'darwin' else value * 1024), 'resource.ru_maxrss' + except (ImportError, AttributeError, OSError): + pass + + try: + import psutil + except ImportError: + return None, 'unavailable (install psutil for sampled peak RSS on Windows)' + + process = psutil.Process() + peak = [process.memory_info().rss] + stopped = threading.Event() + + def sample(): + while not stopped.wait(0.02): + try: + peak[0] = max(peak[0], process.memory_info().rss) + except psutil.Error: + return + + sampler = threading.Thread(target=sample, daemon=True) + sampler.start() + return _SampledRSS(stopped, sampler, peak) + + +class _SampledRSS: + def __init__(self, stopped, sampler, peak): + self.stopped = stopped + self.sampler = sampler + self.peak = peak + + def finish(self): + self.stopped.set() + self.sampler.join() + return int(self.peak[0]), 'psutil sampled every 20 ms' + + +def _measure_rss_start(): + value = _peak_rss_bytes() + return value if isinstance(value, _SampledRSS) else value + + +def _rss_result(handle): + return handle.finish() if isinstance(handle, _SampledRSS) else handle + + +def _phase_prepare(args): + from phy.apps.large_recording.prepare import prepare_large + + rss = _measure_rss_start() + started = time.perf_counter() + bounded = prepare_large( + args.params, + args.bounded_path, + duration_seconds=args.duration_seconds, + max_spikes=args.max_spikes, + max_bytes=args.max_bytes, + window_seconds=args.window_seconds, + stratum_seconds=args.stratum_seconds, + seed=args.seed, + batch_size=args.batch_size, + bin_seconds=args.bin_seconds, + ) + elapsed = time.perf_counter() - started + peak, method = _rss_result(rss) + print( + json.dumps( + { + 'phase': 'prepare', + 'wall_seconds': elapsed, + 'peak_rss_bytes': peak, + 'peak_rss_method': method, + 'bounded_path': str(bounded), + } + ) + ) + + +def _phase_curate(args): + from phy.apps.large_recording.curation import BoundedSession + + session = BoundedSession(args.bounded_path) + try: + if 0 not in session.unit_ids or 1 not in session.unit_ids: + raise ValueError('Synthetic source must contain units 0 and 1.') + session.record_merge([0, 1], session.next_cluster_id) + session.save() + finally: + session.close() + print(json.dumps({'phase': 'curate', 'saved': True})) + + +def _phase_replay(args): + from phy.apps.large_recording.replay import apply_curation + + rss = _measure_rss_start() + started = time.perf_counter() + output = apply_curation( + args.params, args.bounded_path, args.curated_path, batch_size=args.batch_size + ) + elapsed = time.perf_counter() - started + peak, method = _rss_result(rss) + print( + json.dumps( + { + 'phase': 'replay', + 'wall_seconds': elapsed, + 'peak_rss_bytes': peak, + 'peak_rss_method': method, + 'curated_path': str(output), + } + ) + ) + + +def _oracle(source_path, curated_path, spikes, units, batch_size): + """Independent streamed oracle for the benchmark's one known merge.""" + source = np.load(source_path / 'spike_clusters.npy', mmap_mode='r', allow_pickle=False) + output = np.load(curated_path / 'spike_clusters.npy', mmap_mode='r', allow_pickle=False) + expected_target = units + checked = 0 + matches = source.shape == output.shape == (spikes,) + for start in range(0, spikes, batch_size): + stop = min(start + batch_size, spikes) + expected = np.array(source[start:stop], copy=True) + expected[(expected == 0) | (expected == 1)] = expected_target + matches &= np.array_equal(expected, output[start:stop]) + checked += stop - start + source._mmap.close() + output._mmap.close() + return {'matches': bool(matches), 'rows_checked': checked, 'merge_target': expected_target} + + +def _run_phase( + script, + phase, + params, + bounded, + curated, + duration_seconds, + max_spikes, + max_bytes, + window_seconds, + stratum_seconds, + seed, + batch_size, + bin_seconds, +): + command = [ + sys.executable, + str(script), + '--_phase', + phase, + '--params', + str(params), + '--bounded-path', + str(bounded), + '--curated-path', + str(curated), + '--duration-seconds', + str(duration_seconds), + '--max-spikes', + str(max_spikes), + '--max-bytes', + str(max_bytes), + '--window-seconds', + str(window_seconds), + '--stratum-seconds', + str(stratum_seconds), + '--seed', + str(seed), + '--batch-size', + str(batch_size), + '--bin-seconds', + str(bin_seconds), + ] + try: + completed = subprocess.run( + command, cwd=REPOSITORY, check=True, capture_output=True, text=True + ) + except subprocess.CalledProcessError as error: + raise RuntimeError(error.stderr.strip() or error.stdout.strip()) from error + return json.loads(completed.stdout.strip().splitlines()[-1]) + + +def _run(args): + if args.spikes <= 0 or args.spikes > 2_000_000: + raise ValueError('--spikes must be between 1 and 2,000,000.') + if args.units < 2 or args.units > args.spikes: + raise ValueError('--units must be at least 2 and no greater than --spikes.') + if args.batch_size <= 0 or args.duration_hours <= 0: + raise ValueError('--batch-size and --duration-hours must be positive.') + + temporary = None + if args.output_dir: + root = Path(args.output_dir).resolve() + root.mkdir(parents=True, exist_ok=True) + else: + temporary = tempfile.TemporaryDirectory(prefix='phy-long-recording-benchmark-') + root = Path(temporary.name) + source_dir, bounded_dir, curated_dir = ( + root / name for name in ('source', 'bounded', 'curated') + ) + duration_seconds = _make_source( + source_dir, args.spikes, args.units, args.duration_hours, args.batch_size + ) + source_bytes = _file_bytes(source_dir) + common = { + 'params': source_dir / 'params.py', + 'bounded': bounded_dir, + 'curated': curated_dir, + 'duration_seconds': duration_seconds, + 'max_spikes': args.max_spikes, + 'max_bytes': args.max_bytes, + 'window_seconds': args.window_seconds, + 'stratum_seconds': args.stratum_seconds, + 'seed': args.seed, + 'batch_size': args.batch_size, + 'bin_seconds': args.bin_seconds, + } + prepare = _run_phase(Path(__file__).resolve(), 'prepare', **common) + _run_phase(Path(__file__).resolve(), 'curate', **common) + replay = _run_phase(Path(__file__).resolve(), 'replay', **common) + oracle = _oracle(source_dir, curated_dir, args.spikes, args.units, args.batch_size) + if not oracle['matches']: + raise RuntimeError('Independent streamed replay oracle did not match curated assignments.') + + result = { + 'benchmark': 'phy_long_recording_v1', + 'machine': { + 'platform': platform.platform(), + 'machine': platform.machine(), + 'processor': platform.processor(), + 'python': platform.python_version(), + 'numpy': np.__version__, + 'cpu_count': os.cpu_count(), + }, + 'parameters': { + 'spikes': args.spikes, + 'units': args.units, + 'duration_hours': args.duration_hours, + 'sample_rate_hz': 30000, + 'batch_size': args.batch_size, + 'max_spikes': args.max_spikes, + 'max_bytes': args.max_bytes, + 'window_seconds': args.window_seconds, + 'stratum_seconds': args.stratum_seconds, + 'seed': args.seed, + 'bin_seconds': args.bin_seconds, + }, + 'source_file_bytes': source_bytes, + 'bounded_file_bytes': _file_bytes(bounded_dir), + 'curated_file_bytes': _file_bytes(curated_dir), + 'bounded_spikes': int( + json.loads((bounded_dir / '_phy_curation_manifest.json').read_text(encoding='utf-8'))[ + 'bounded_num_spikes' + ] + ), + 'preparation': prepare, + 'replay': replay, + 'independent_streamed_replay_oracle': oracle, + 'output_dir': str(root), + 'output_dir_retained': temporary is None, + } + print(json.dumps(result, indent=2, sort_keys=True)) + if temporary is not None: + temporary.cleanup() + + +def _parser(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--spikes', type=int, default=1_000_000) + parser.add_argument('--units', type=int, default=1000) + parser.add_argument('--duration-hours', type=float, default=1000) + parser.add_argument('--batch-size', type=int, default=100_000) + parser.add_argument('--max-spikes', type=int, default=100_000) + parser.add_argument('--max-bytes', type=int, default=536_870_912) + parser.add_argument('--window-seconds', type=float, default=30) + parser.add_argument('--stratum-seconds', type=float, default=3600) + parser.add_argument('--seed', type=int, default=0) + parser.add_argument('--bin-seconds', type=float, default=3600) + parser.add_argument('--output-dir', type=Path) + parser.add_argument( + '--_phase', choices=('prepare', 'curate', 'replay'), help=argparse.SUPPRESS + ) + parser.add_argument('--params', type=Path, help=argparse.SUPPRESS) + parser.add_argument('--bounded-path', type=Path, help=argparse.SUPPRESS) + parser.add_argument('--curated-path', type=Path, help=argparse.SUPPRESS) + parser.add_argument('--duration-seconds', type=float, help=argparse.SUPPRESS) + return parser + + +def main(): + args = _parser().parse_args() + if args._phase == 'prepare': + _phase_prepare(args) + elif args._phase == 'curate': + _phase_curate(args) + elif args._phase == 'replay': + _phase_replay(args) + else: + _run(args) + + +if __name__ == '__main__': + main() diff --git a/tools/capture_long_recording_views.py b/tools/capture_long_recording_views.py new file mode 100644 index 000000000..d05f821fa --- /dev/null +++ b/tools/capture_long_recording_views.py @@ -0,0 +1,575 @@ +"""Capture reproducible screenshots of the bounded-evidence Qt views. + +The script builds a temporary synthetic Phy sorting, prepares it with the real +long-recording workflow, opens the real GUI, and saves screenshots. Existing +bounded inputs are copied into a temporary directory before opening. + +Run with a display, or under a virtual X server:: + + QT_QPA_PLATFORM=xcb LIBGL_ALWAYS_SOFTWARE=1 \ + xvfb-run -a -s '-screen 0 2400x1600x24 +extension GLX' \ + uv run python tools/capture_long_recording_views.py --amplitude-split + +Use ``--amplitude-split --bounded-dir PATH --cluster-ids ID`` for an existing +fresh bounded package, optionally with ``--threshold SOURCE_VALUE``. The three +PNG captures and ``capture.json`` include the exact committed threshold and +retained child sizes; graphical threshold values reflect pixel quantization. +""" + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +import numpy as np +from PyQt5.QtCore import QCoreApplication, QEvent, QPoint, QPointF +from PyQt5.QtGui import QMouseEvent +from PyQt5.QtTest import QTest + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.source import hash_file +from phy.apps.template.gui import TemplateController +from phy.gui.qt import QApplication, Qt +from phy.plot.transform import NDC, Range + +REPOSITORY = Path(__file__).resolve().parents[1] +DEFAULT_OUTPUT = REPOSITORY / 'docs' / 'images' +SAMPLE_RATE = 30_000 +HOURS = 8 +CLUSTER_IDS = np.array([0, 2, 7, 99], dtype=np.int32) + + +def _make_source(path, hours=HOURS, unit_count=4): + """Write a varied but deterministic synthetic source sorting.""" + path.mkdir(parents=True) + rng = np.random.default_rng(20261013) + cluster_ids = CLUSTER_IDS if unit_count == 4 else np.arange(unit_count, dtype=np.int32) + samples = SAMPLE_RATE * 3600 * hours + intervals = np.array( + [[0, 3 * 3600 * SAMPLE_RATE], [int(3.5 * 3600 * SAMPLE_RATE), samples]], + dtype=np.int64, + ) + valid = intervals[:, 1] - intervals[:, 0] + cumulative = np.cumsum(valid) + offsets = rng.integers(0, cumulative[-1], size=120_000, dtype=np.int64) + interval_index = np.searchsorted(cumulative, offsets, side='right') + previous = np.where(interval_index == 0, 0, cumulative[np.maximum(interval_index - 1, 0)]) + spike_times = intervals[interval_index, 0] + offsets - previous + order = np.argsort(spike_times, kind='stable') + spike_times = spike_times[order] + + unit_index = np.arange(len(spike_times), dtype=np.int64) % len(cluster_ids) + unit_index = unit_index[order] + elapsed = spike_times / SAMPLE_RATE + hour_phase = elapsed / 3600 + amplitudes = ( + 35 + + unit_index * 9 + + 5 * np.sin(hour_phase * np.pi / 3) + + 2 * hour_phase + + rng.normal(0, 2.5, len(spike_times)) + ).astype(np.float32) + depths = ( + 250 + + unit_index * 180 + + 20 * np.sin(hour_phase * np.pi / 2) + + rng.normal(0, 8, len(spike_times)) + ).astype(np.float32) + templates_for_spike = unit_index.astype(np.int32) + + np.save(path / 'spike_times.npy', spike_times.astype(np.int64)) + np.save(path / 'spike_clusters.npy', cluster_ids[unit_index]) + np.save(path / 'spike_templates.npy', templates_for_spike) + np.save(path / 'amplitudes.npy', amplitudes) + np.save(path / 'spike_depths.npy', depths) + + channel_map = np.arange(4, dtype=np.int32) + channel_positions = np.array([[0, 0], [20, 100], [0, 200], [20, 300]], dtype=np.float32) + np.save(path / 'channel_map.npy', channel_map) + np.save(path / 'channel_positions.npy', channel_positions) + + templates = np.zeros((len(cluster_ids), 40, len(channel_map)), dtype=np.float32) + time = np.arange(40, dtype=np.float32) + for unit in range(len(cluster_ids)): + center = 13 + (unit % len(channel_map)) * 3 + pulse = -np.exp(-0.5 * ((time - center) / 3.5) ** 2) * (35 + 8 * unit) + for channel in range(len(channel_map)): + templates[unit, :, channel] = pulse * np.exp( + -0.7 * abs(channel - unit % len(channel_map)) + ) + np.save(path / 'templates.npy', templates) + + pc_feature_ind = np.array( + [ + [unit % len(channel_map), (unit + 1) % len(channel_map), (unit + 2) % len(channel_map)] + for unit in range(len(cluster_ids)) + ], + dtype=np.int32, + ) + np.save(path / 'pc_feature_ind.npy', pc_feature_ind) + features = np.empty((len(spike_times), 3, 3), dtype=np.float32) + features[:, 0, :] = unit_index[:, None] * 3.5 + rng.normal(0, 0.65, (len(spike_times), 3)) + features[:, 1, :] = np.sin(hour_phase[:, None] * np.pi / 2) + rng.normal( + 0, 0.7, (len(spike_times), 3) + ) + features[:, 2, :] = np.cos(hour_phase[:, None] * np.pi / 3) + rng.normal( + 0, 0.6, (len(spike_times), 3) + ) + np.save(path / 'pc_features.npy', features) + + # Stored waveforms are a display reservoir, aligned to sorted source rows. + waveform_ids = np.arange(0, len(spike_times), 8, dtype=np.int64) + waveform_channels = np.tile( + np.arange(len(channel_map), dtype=np.int32), (len(waveform_ids), 1) + ) + waveforms = templates[unit_index[waveform_ids]].copy() + waveforms += rng.normal(0, 2.2, waveforms.shape).astype(np.float32) + stem = '_phy_spikes_subset' + np.save(path / f'{stem}.spikes.npy', waveform_ids) + np.save(path / f'{stem}.channels.npy', waveform_channels) + np.save(path / f'{stem}.waveforms.npy', waveforms) + + with (path / 'cluster_group.tsv').open('w', encoding='utf-8') as stream: + stream.write('cluster_id\tgroup\n') + for unit in cluster_ids: + stream.write(f'{unit}\t{"mua" if unit == 7 else "good"}\n') + (path / 'params.py').write_text( + f'sample_rate = {SAMPLE_RATE}.0\n' + f'num_samples = {samples}\n' + 'n_channels_dat = 4\n' + "dtype = 'int16'\n" + 'dat_path = None\n' + f'valid_intervals = {intervals.tolist()!r}\n', + encoding='utf-8', + ) + return path / 'params.py', intervals + + +def _wait_for_render(): + QApplication.processEvents() + QTest.qWait(1200) + QApplication.processEvents() + + +def _capture(gui, destination): + _wait_for_render() + # Phy's OpenGL canvases are native child windows. QWidget.grab() omits + # those children on some Qt/X11 combinations, so grab the rendered screen + # and crop to the application window instead. + screen = QApplication.primaryScreen() + frame = gui.frameGeometry() + pixmap = screen.grabWindow(0).copy(frame.x(), frame.y(), frame.width(), frame.height()) + if not pixmap.save(str(destination), 'PNG'): + raise OSError(f'Could not save screenshot: {destination}') + + +def _close_capture(gui, controller): + gui.close() + controller.close() + gui.deleteLater() + QCoreApplication.sendPostedEvents(None, QEvent.DeferredDelete) + QApplication.processEvents() + + +def _capture_comparison(bounded, config_dir, output_dir, selected, dataset_label): + """Capture actual GUI comparisons on an isolated, disposable package.""" + controller = TemplateController( + dir_path=bounded, config_dir=config_dir, plugins=[], enable_threading=False + ) + gui = controller.create_gui(default_views=['UnitStabilityView'], do_prompt_save=False) + try: + gui.resize(2100, 1200) + gui.show() + _wait_for_render() + supervisor = controller.supervisor + started = time.perf_counter() + supervisor.select(selected) + supervisor.block() + selection_seconds = time.perf_counter() - started + view = gui.get_view('UnitStabilityView') + assert list(view.cluster_ids) == supervisor.selected + assert len(view.cluster_ids) >= 2, 'Select at least two retained clusters.' + assert all(ax.get_legend() is None for ax in view.canvas.iter_ax()) + _capture(gui, output_dir / 'after-overview.png') + started = time.perf_counter() + view.toggle_amplitude(False) + view.toggle_depth(False) + rate_only_seconds = time.perf_counter() - started + _capture(gui, output_dir / 'after-rate-only.png') + camera = view.screenshot(dir=output_dir / 'camera') + shutil.copyfile(camera, output_dir / 'camera-rate-only.png') + native = float(np.diff(view.get_data(selected[0]).edges)[0]) + view.set_display_bin_seconds(2 * native) + _capture(gui, output_dir / 'after-rate-coarsened.png') + supervisor.toggle_merge_mode() + supervisor.block() + assert supervisor.selection.state.is_merge_mode + assert list(view.cluster_ids) == supervisor.selected + _capture(gui, output_dir / 'after-merge.png') + manifest = controller.bounded_session.manifest + report = { + 'dataset_label': dataset_label, + 'capture_revision': subprocess.check_output( + ['git', 'rev-parse', 'HEAD'], cwd=REPOSITORY, text=True + ).strip(), + 'clock_hours': manifest['num_samples'] / manifest['sample_rate'] / 3600, + 'selected_cluster_ids': selected, + 'displayed_cluster_ids': list(map(int, view.displayed_ids)), + 'merge_cluster_ids': supervisor.selected_merge, + 'summary_bins': len(view.get_data(selected[0]).exposure), + 'full_counts': { + str(unit): int(view.get_data(unit).counts.sum()) for unit in view.displayed_ids + }, + 'timing_seconds': { + 'select_and_block': selection_seconds, + 'rate_only_redraws': rate_only_seconds, + }, + 'gui_size': [gui.width(), gui.height()], + 'display_bins': len(view.get_display_data(selected[0]).exposure), + 'rate_line_width': view.rate_line_width, + 'display_limit': view.display_limit, + 'display_bin_seconds': view.display_bin_seconds, + 'screenshots': [ + 'after-overview.png', + 'after-rate-only.png', + 'after-merge.png', + 'after-rate-coarsened.png', + 'camera-rate-only.png', + ], + 'source_arrays_modified': False, + } + (output_dir / 'capture.json').write_text(json.dumps(report, indent=2) + '\n') + print(json.dumps(report), flush=True) + finally: + _close_capture(gui, controller) + + +def _capture_amplitude_split(bounded, config_dir, output_dir, selected, threshold, dataset_label): + """Capture a real threshold drag and K commit on a disposable bounded copy.""" + controller = TemplateController( + dir_path=bounded, config_dir=config_dir, plugins=[], enable_threading=False + ) + gui = controller.create_gui( + default_views=['EvidenceAmplitudeView', 'WaveformView'], do_prompt_save=False + ) + try: + supervisor = controller.supervisor + if controller.bounded_session.events: + raise ValueError('Amplitude capture requires a fresh bounded package without edits.') + if len(selected) != 1: + raise ValueError('Amplitude capture requires exactly one cluster ID.') + unit = selected[0] + ids = supervisor.clustering.spikes_per_cluster.get(unit, np.array([], dtype=np.int64)) + values = controller.bounded_session.attributes.get('amplitudes.npy') + if values is None or not len(ids): + raise ValueError('Select a retained unit with source amplitude evidence.') + amplitudes = values.reshape(-1)[ids] + finite = amplitudes[np.isfinite(amplitudes)] + if not len(finite): + raise ValueError('The selected unit has no finite source amplitudes.') + threshold = float(np.median(finite)) if threshold is None else float(threshold) + if not np.isfinite(threshold): + raise ValueError('The threshold must be finite.') + if not np.any(np.isfinite(amplitudes) & (amplitudes < threshold)) or not np.any( + ~(np.isfinite(amplitudes) & (amplitudes < threshold)) + ): + raise ValueError('Both threshold children must have retained evidence.') + + gui.resize(2200, 1250) + gui.show() + _wait_for_render() + for name in ('ClusterView', 'SimilarityView', 'MergePropositionsView', 'PropositionView'): + for view in gui.list_views(name): + view.dock.hide() + amplitude = gui.get_view('EvidenceAmplitudeView') + waveform = gui.get_view('WaveformView') + gui.splitDockWidget(amplitude.dock, waveform.dock, Qt.Horizontal) + gui.resizeDocks([amplitude.dock, waveform.dock], [1450, 700], Qt.Horizontal) + controller.n_spikes_waveforms = 60 + controller.n_spikes_waveforms_total = 60 + supervisor.select(selected) + supervisor.block() + amplitude.amplitude_scope = 'overview' + waveform.waveforms_type = 'waveforms' + waveform.plot() + _wait_for_render() + _capture(gui, output_dir / 'amplitude-before.png') + + # Deliver actual Qt events to the native OpenGL window. Reuse the view's + # transform so the drag is expressed in unscaled source amplitude units. + canvas = amplitude.canvas + width, height = canvas.get_size() + ndc = Range(amplitude.data_bounds, NDC).apply([[amplitude.duration / 2, threshold]])[0] + screen = (ndc + np.asarray(canvas.panzoom.pan)) * canvas.panzoom._zoom_aspect() + point = QPoint(round((screen[0] + 1) * width / 2), round((1 - screen[1]) * height / 2)) + if not 0 <= point.y() < height: + raise ValueError('The threshold is outside the displayed amplitude range.') + start = QPoint(max(0, point.x() - 20), point.y()) + QTest.mousePress(canvas, Qt.RightButton, Qt.AltModifier, start) + move = QMouseEvent( + QEvent.MouseMove, QPointF(point), Qt.NoButton, Qt.RightButton, Qt.AltModifier + ) + QApplication.sendEvent(canvas, move) + QTest.mouseRelease(canvas, Qt.RightButton, Qt.AltModifier, point) + _wait_for_render() + actual_threshold = amplitude.split_threshold + if actual_threshold is None or not supervisor.actions.get('split').isEnabled(): + raise RuntimeError('The actual Alt-right drag did not create an eligible preview.') + _capture(gui, output_dir / 'amplitude-preview.png') + highlighted = int(len(waveform._highlighted_spike_ids)) + if hasattr(canvas, 'requestActivate'): + canvas.requestActivate() + elif hasattr(canvas, 'setFocus'): + canvas.setFocus() + _wait_for_render() + QTest.keyClick(canvas, Qt.Key_K) + supervisor.block() + _wait_for_render() + if not controller.bounded_session.events: + raise RuntimeError('The actual K key did not commit the threshold predicate.') + event = controller.bounded_session.events[-1] + if event['operation'] != 'threshold_split' or amplitude.split_threshold is not None: + raise RuntimeError('K did not finish the expected semantic threshold split.') + _capture(gui, output_dir / 'amplitude-after.png') + report = { + 'dataset_label': dataset_label, + 'workflow': 'amplitude_split', + 'selected_cluster_ids': selected, + 'amplitude_scope': amplitude.amplitude_scope, + 'requested_source_threshold': threshold, + 'exact_source_threshold': event['predicate']['threshold'], + 'predicate': event['predicate'], + 'retained_parent_count': int(len(ids)), + 'retained_split_sizes': { + str(child): int(supervisor.n_spikes(child)) for child in event['children'] + }, + 'highlighted_waveform_traces': highlighted, + 'gui_size': [gui.width(), gui.height()], + 'screenshots': [ + 'amplitude-before.png', + 'amplitude-preview.png', + 'amplitude-after.png', + ], + 'source_arrays_modified': False, + 'capture_revision': subprocess.check_output( + ['git', 'rev-parse', 'HEAD'], cwd=REPOSITORY, text=True + ).strip(), + } + (output_dir / 'capture.json').write_text(json.dumps(report, indent=2) + '\n') + print(json.dumps(report), flush=True) + finally: + _close_capture(gui, controller) + + +def _capture_views(bounded, config_dir, output_dir, stability): + views = ( + ['UnitStabilityView'] + if stability + else [ + 'EvidenceAmplitudeView', + 'FeatureView', + 'SampledISIView', + 'SampledCorrelogramView', + ] + ) + controller = TemplateController( + dir_path=bounded, + config_dir=config_dir, + plugins=[], + enable_threading=False, + ) + gui = controller.create_gui(default_views=views, do_prompt_save=False) + try: + gui.resize(2100, 1200) + gui.show() + _wait_for_render() + # Focus the captures on the plotted evidence rather than the GUI tables. + similarity = gui.get_view('SimilarityView') + if similarity is not None: + similarity.dock.hide() + cluster_table = gui.get_view('ClusterView') + if cluster_table is not None: + cluster_table.dock.hide() + if not stability: + amplitude, feature, isi, correlogram = ( + gui.get_view(name) + for name in ( + 'EvidenceAmplitudeView', + 'FeatureView', + 'SampledISIView', + 'SampledCorrelogramView', + ) + ) + gui.splitDockWidget(amplitude.dock, feature.dock, Qt.Vertical) + gui.splitDockWidget(amplitude.dock, isi.dock, Qt.Horizontal) + gui.splitDockWidget(isi.dock, correlogram.dock, Qt.Vertical) + gui.resizeDocks([amplitude.dock, isi.dock], [1000, 1000], Qt.Horizontal) + gui.resizeDocks([amplitude.dock, feature.dock], [550, 550], Qt.Vertical) + gui.resizeDocks([isi.dock, correlogram.dock], [550, 550], Qt.Vertical) + _wait_for_render() + windows = controller.evidence_windows + times = controller.model.spike_samples + counts = [ + int(np.searchsorted(times, end) - np.searchsorted(times, start)) + for start, end in windows + ] + best_window = int(np.argmax(counts)) + controller.inspect_evidence_window(best_window) + selected = [0] if stability else [0, 2] + controller.supervisor.select(selected) + controller.supervisor.block() + _wait_for_render() + if stability: + filename = 'long-recording-overview.png' + else: + filename = 'long-recording-evidence.png' + destination = output_dir / filename + _capture(gui, destination) + print( + f'{destination} ({gui.width()}x{gui.height()}, window {best_window}, ' + f'{counts[best_window]} evidence spikes)' + ) + finally: + _close_capture(gui, controller) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--output-dir', type=Path, default=DEFAULT_OUTPUT) + parser.add_argument( + '--bounded-dir', type=Path, help='Existing package; copied before opening.' + ) + modes = parser.add_mutually_exclusive_group() + modes.add_argument( + '--comparison', action='store_true', help='Capture cluster overlay controls.' + ) + modes.add_argument( + '--amplitude-split', + action='store_true', + help='Capture a source-amplitude drag and K split.', + ) + parser.add_argument( + '--threshold', type=float, help='Source amplitude threshold; default: median.' + ) + parser.add_argument( + '--cluster-ids', + type=int, + nargs='+', + help='Selected IDs; exactly one for amplitude splits.', + ) + parser.add_argument('--dataset-label', help='Dataset provenance label stored in capture.json.') + parser.add_argument('--synthetic-hours', type=int, default=HOURS) + parser.add_argument('--synthetic-units', type=int, default=4) + parser.add_argument('--summary-bin-seconds', type=float, default=1800) + args = parser.parse_args() + if args.synthetic_hours < 4: + parser.error('--synthetic-hours must be at least 4.') + if args.synthetic_units < 2: + parser.error('--synthetic-units must be at least 2.') + if args.bounded_dir and not (args.comparison or args.amplitude_split): + parser.error('--bounded-dir requires --comparison or --amplitude-split.') + if args.threshold is not None and not args.amplitude_split: + parser.error('--threshold requires --amplitude-split.') + if args.amplitude_split and args.cluster_ids and len(args.cluster_ids) != 1: + parser.error('--amplitude-split requires exactly one --cluster-ids ID.') + if args.bounded_dir: + source = args.bounded_dir.resolve() + if not (source / '_phy_curation_manifest.json').is_file(): + parser.error('--bounded-dir must contain a bounded curation manifest.') + if source == args.output_dir.resolve() or source in args.output_dir.resolve().parents: + parser.error('--output-dir must be outside the input bounded package.') + if args.amplitude_split: + if not args.cluster_ids or len(args.cluster_ids) != 1: + parser.error( + '--bounded-dir amplitude splits require exactly one --cluster-ids ID.' + ) + elif not args.cluster_ids or len(args.cluster_ids) < 2: + parser.error('--bounded-dir comparisons require at least two --cluster-ids.') + args.output_dir.mkdir(parents=True, exist_ok=True) + app = QApplication.instance() or QApplication(sys.argv[:1]) + + with tempfile.TemporaryDirectory(prefix='phy-long-recording-screenshots-') as temporary: + root = Path(temporary) + if args.bounded_dir: + bounded = root / 'bounded' + original_hashes = { + str(path.relative_to(source)): hash_file(path) + for path in source.rglob('*') + if path.is_file() + } + shutil.copytree(source, bounded, ignore=shutil.ignore_patterns('.phy', '*.log')) + capture = _capture_amplitude_split if args.amplitude_split else _capture_comparison + extra = [args.threshold] if args.amplitude_split else [] + capture( + bounded, + root / 'config', + args.output_dir, + args.cluster_ids, + *extra, + args.dataset_label + or 'Existing bounded package; acquisition exposure as declared.', + ) + checked_hashes = { + str(path.relative_to(source)): hash_file(path) + for path in source.rglob('*') + if path.is_file() + } + if original_hashes != checked_hashes: + raise RuntimeError('The input bounded package changed during capture.') + report_path = args.output_dir / 'capture.json' + report = json.loads(report_path.read_text()) + report['input_sha256'] = original_hashes + report['input_package_unchanged'] = True + report_path.write_text(json.dumps(report, indent=2) + '\n') + app.quit() + return + params, intervals = _make_source( + root / 'source', hours=args.synthetic_hours, unit_count=args.synthetic_units + ) + bounded = prepare_large( + params, + root / 'bounded', + duration_seconds=args.synthetic_hours * 3600, + valid_intervals=intervals, + max_spikes=20_000, + max_bytes=268_435_456, + window_seconds=30, + stratum_seconds=3600, + seed=0, + batch_size=50_000, + bin_seconds=args.summary_bin_seconds, + ) + if args.amplitude_split: + _capture_amplitude_split( + bounded, + root / 'config', + args.output_dir, + args.cluster_ids or [0], + args.threshold, + args.dataset_label or f'Synthetic {args.synthetic_hours}-hour fixture.', + ) + elif args.comparison: + _capture_comparison( + bounded, + root / 'config', + args.output_dir, + args.cluster_ids + or ( + list(map(int, CLUSTER_IDS[:3])) + if args.synthetic_units == 4 + else list(range(min(12, args.synthetic_units))) + ), + args.dataset_label or f'Synthetic {args.synthetic_hours}-hour fixture.', + ) + else: + _capture_views(bounded, root / 'config-stability', args.output_dir, stability=True) + _capture_views(bounded, root / 'config-evidence', args.output_dir, stability=False) + app.quit() + + +if __name__ == '__main__': + main() diff --git a/tools/compare_curation_modes.py b/tools/compare_curation_modes.py new file mode 100644 index 000000000..b81f98d12 --- /dev/null +++ b/tools/compare_curation_modes.py @@ -0,0 +1,562 @@ +"""Independent acceptance comparison for complete ordinary and bounded outputs. + +Manual protocol (same ledger on both branches): alias each original unit ID, +merge parents into ``as``, split all spikes of ``parent`` with finite amplitude +strictly below ``threshold`` into ``selected`` (equal/nonfinite => ``remainder``), +then explicitly label every final unit's group and notes. Undo/redo apply to +curation commands; save/reopen are history boundaries. Save, reopen, save again +without editing, and preserve both outputs. Compare COMPLETE replay arrays, +never bounded evidence arrays. Producer metrics (SNR etc.) are excluded: clearing +them on transformed units is an intentional bounded metadata policy. + +Ledger JSON: {"aliases": {"a": 0, "b": 1}, "actions": [ + {"op": "merge", "parents": ["a", "b"], "as": "ab"}, + {"op": "split", "parent": "ab", "threshold": 1.5, + "selected": "low", "remainder": "rest"}, + {"op": "label", "ids": ["low"], "field": "group", "value": "good"}, + {"op": "undo"}, {"op": "redo"}, {"op": "save"}, {"op": "reopen"}], + "expected_labels": {"low": {"group": "good"}}, "fields": ["group", "notes"]}. +Explicit final labels avoid comparing legacy ordinary metadata inheritance with +bounded transformed-unit clearing. Use each GUI's reported IDs to track aliases. +Amplitude predicates must use raw amplitudes.npy; TemplateModel sanitizes nonfinite +values to zero, so GUI lasso/predicate selection on displayed amplitudes alone is +not this protocol. The ordinary engine runs full Supervisor commands without attaching GUI views; +the streamed engine is a separate baseline, NOT full ordinary GUI evidence. +""" + +import argparse +import copy +import csv +import gc +import hashlib +import json +import shutil +import subprocess +from pathlib import Path + +import numpy as np + + +def digest(path): + h = hashlib.sha256() + with Path(path).open('rb') as stream: + while block := stream.read(1024 * 1024): + h.update(block) + return h.hexdigest() + + +def revision(): + return subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() + + +def labels(path, fields): + result = {field: {} for field in fields} + tables = sorted(Path(path).glob('cluster_*.tsv')) + sorted(Path(path).glob('cluster_*.csv')) + for table in tables: + # GUI-generated summary duplicates authoritative label tables. + if table.stem == 'cluster_info': + continue + with table.open(newline='') as stream: + reader = csv.DictReader(stream, delimiter=',' if table.suffix == '.csv' else '\t') + if 'cluster_id' not in (reader.fieldnames or []): + raise ValueError(f'Label table lacks cluster_id: {table}') + for row in reader: + unit = int(row['cluster_id']) + for field in fields: + if field in row: + if unit in result[field]: + raise ValueError(f'Duplicate {field} label for {unit}') + result[field][unit] = row[field] + return result + + +def write_labels(path, data): + for field, values in data.items(): + if not field.isidentifier(): + raise ValueError('Label fields must be identifiers') + with (Path(path) / f'cluster_{field}.tsv').open('w', newline='') as stream: + writer = csv.writer(stream, delimiter='\t') + writer.writerow(['cluster_id', field]) + writer.writerows(sorted(values.items())) + + +def compare_outputs( + left, + right, + fields=('group', 'notes'), + chunk_size=100000, + source=None, + expected_labels=None, + bounded_dir=None, +): + """Exact all-row partition equivalence, O(chunk_size + number of units) RAM. + + A two-way raw-ID mapping proves equivalence without partition hash collisions. + SHA256 is used only to identify saved inputs/coordinates, not as that proof. + """ + report = { + 'ok': False, + 'method': 'exact all-row two-way ID mapping', + 'revision': revision(), + 'left': str(Path(left).resolve()), + 'right': str(Path(right).resolve()), + } + try: + if chunk_size <= 0: + raise ValueError('chunk_size must be positive') + arrays = [ + np.load(Path(p) / 'spike_clusters.npy', mmap_mode='r', allow_pickle=False) + for p in (left, right) + ] + times = [ + np.load(Path(p) / 'spike_times.npy', mmap_mode='r', allow_pickle=False) + for p in (left, right) + ] + for a in arrays + times: + if a.ndim not in (1, 2) or (a.ndim == 2 and a.shape[1] != 1): + raise ValueError('Expected scalar per-spike arrays') + if a.dtype.kind not in 'iu': + raise ValueError('Assignments and source sample times must be integers, no NaNs') + arrays = [a.reshape(-1) for a in arrays] + times = [a.reshape(-1) for a in times] + if len({len(a) for a in arrays + times}) != 1: + raise ValueError('Unequal source/output row counts') + report['rows'] = len(arrays[0]) + report['assignment_dtypes'] = [str(a.dtype) for a in arrays] + if times[0].dtype != times[1].dtype: + raise ValueError('Unequal source time dtypes') + forward, reverse = {}, {} + mismatches, examples = 0, [] + for start in range(0, len(arrays[0]), chunk_size): + stop = start + chunk_size + if not np.array_equal(times[0][start:stop], times[1][start:stop]): + raise ValueError(f'Source time row alignment differs at chunk {start}') + aa, bb = arrays[0][start:stop], arrays[1][start:stop] + if np.any(aa < 0) or np.any(bb < 0): + raise ValueError('Negative cluster ID') + # Unique raw pairs retain exact integer comparison; process first occurrences + # in source row order so diagnostics are stable across chunk sizes. + pairs = np.empty(len(aa), dtype=[('left', aa.dtype), ('right', bb.dtype)]) + pairs['left'], pairs['right'] = aa, bb + unique, first, counts = np.unique(pairs, return_index=True, return_counts=True) + for pair_index in np.argsort(first): + a, b = int(unique[pair_index]['left']), int(unique[pair_index]['right']) + if forward.get(a, b) != b or reverse.get(b, a) != a: + mismatches += int(counts[pair_index]) + if len(examples) < 10: + examples.append( + {'row': start + int(first[pair_index]), 'left': a, 'right': b} + ) + else: + forward[a], reverse[b] = b, a + report.update( + partition_mismatch_rows=mismatches, + examples=examples, + id_mapping={str(k): v for k, v in sorted(forward.items())}, + ) + report['saved_provenance'] = { + side: { + name: json.loads((Path(p) / name).read_text()) + for name in ('comparison_reference.json', '_phy_curation_lineage.json') + if (Path(p) / name).exists() + } + for side, p in (('left', left), ('right', right)) + } + report['input_hashes'] = { + side: { + name: digest(Path(p) / name) for name in ('spike_times.npy', 'spike_clusters.npy') + } + for side, p in (('left', left), ('right', right)) + } + permitted = set() + if source is not None: + if __package__: + from tools.curation_output_contract import compare_file_contract + else: + from curation_output_contract import compare_file_contract + + contract, a_labels, b_labels = compare_file_contract( + left, right, source, forward, expected_labels, bounded_dir, chunk_size + ) + report['file_contract'] = contract + permitted = { + (d['field'], d['left_id'], d['right_id'], d['left'], d['right']) + for d in contract['permitted_metadata_differences'] + } + report['permitted_membership_policy_differences'] = contract[ + 'permitted_metadata_differences' + ] + else: + a_labels, b_labels = labels(left, fields), labels(right, fields) + differences = [] + for field in fields: + for a, b in forward.items(): + av, bv = a_labels.get(field, {}).get(a, ''), b_labels.get(field, {}).get(b, '') + if av != bv and (field, a, b, av, bv) not in permitted: + differences.append( + {'field': field, 'left_id': a, 'right_id': b, 'left': av, 'right': bv} + ) + unmapped = { + 'left': { + field: sorted(set(values) - set(forward)) for field, values in a_labels.items() + }, + 'right': { + field: sorted(set(values) - set(reverse)) for field, values in b_labels.items() + }, + } + report['unmapped_label_ids'] = unmapped + report['manual_label_mismatches'] = differences + report['compared_fields'] = list(fields) + report['ok'] = ( + mismatches == 0 + and not differences + and not any(ids for side in unmapped.values() for ids in side.values()) + ) + except (ValueError, OSError, KeyError, TypeError, AttributeError, IndexError) as error: + report['error'] = str(error) + return report + + +def run_reference(source, ledger, output, engine='ordinary', chunk_size=100000): + """Execute a common action ledger independently of bounded journal code. + + Streaming uses disk snapshots and direct masks. Ordinary uses full Supervisor + split with ALL matching source row IDs, never a bounded predicate projector. + Output is a comparison artifact (assignments, times, manual TSVs), not a full + sorting export. No source file is modified. + """ + source, output = Path(source).resolve(), Path(output).resolve() + if output.exists() or source == output or source in output.parents: + raise ValueError('Output must be new and outside the immutable source') + if engine not in ('ordinary', 'streamed') or chunk_size <= 0: + raise ValueError('Invalid reference engine or chunk size') + fields = ledger.get('fields', ['group', 'notes']) + data = labels(source, fields) + aliases = {k: int(v) for k, v in ledger['aliases'].items()} + amplitudes = np.load(source / 'amplitudes.npy', mmap_mode='r').reshape(-1) + output.mkdir(parents=True) + shutil.copyfile(source / 'spike_times.npy', output / 'spike_times.npy') + shutil.copyfile(source / 'spike_clusters.npy', output / 'spike_clusters.npy') + history, future = [], [] + snapshots = [] + state = None + current = None + allocator = 0 + callback_ids = set() + + def dispose(): + nonlocal state, current + from phylib.utils.event import _EVENT + + _EVENT._callbacks = [item for item in _EVENT._callbacks if id(item) not in callback_ids] + callback_ids.clear() + if engine == 'streamed' and state is not None: + state.flush() + state._mmap.close() + current = None + state = None + # Supervisor contains cycles through history and action creators. + gc.collect() + + def reopen(): + nonlocal state, allocator + dispose() + if engine == 'ordinary': + from phy.cluster.supervisor import Supervisor + + class ReferenceSupervisor(Supervisor): + # Presentation hooks only: native command/history/metadata code is unchanged. + def _apply_selection_change(self, *args, **kwargs): + pass + + def _set_merge_mode_ui(self, *args): + pass + + def _reveal_cluster_result(self, *args): + pass + + def _hide_merge_view(self): + pass + + def _workflow_context(self): + return {} + + def _reset_cluster_view(self): + pass + + from phylib.utils.event import _EVENT + + existing = {id(item) for item in _EVENT._callbacks} + try: + state = ReferenceSupervisor( + np.load(output / 'spike_clusters.npy').reshape(-1), + cluster_groups=data.get('group', {}), + cluster_labels={f: d for f, d in data.items() if f != 'group'}, + new_cluster_id=allocator or None, + ) + finally: + callback_ids.update( + id(item) for item in _EVENT._callbacks if id(item) not in existing + ) + state.cluster_view = state.similarity_view = state.select_actions = None + else: + state = np.load(output / 'spike_clusters.npy', mmap_mode='r+').reshape(-1) + + def assignments(): + return state.clustering.spike_clusters if engine == 'ordinary' else state + + def persist(): + nonlocal data + if engine == 'ordinary': + data = {field: state.get_labels(field) for field in fields} + np.save(output / 'spike_clusters.npy', assignments()) + else: + state.flush() + write_labels(output, data) + + try: + reopen() + for index, action in enumerate(ledger['actions']): + op = action['op'] + if op in ('save', 'reopen'): + persist() + if op == 'reopen': + reopen() + history.clear() + future.clear() + continue + if op in ('undo', 'redo'): + if engine == 'ordinary': + getattr(state, op)() + else: + origin, destination = (history, future) if op == 'undo' else (future, history) + if not origin: + raise ValueError(f'No {op} available') + snap = output / f'.history-{index}.npy' + state.flush() + shutil.copyfile(output / 'spike_clusters.npy', snap) + saved_array = np.load(snap, mmap_mode='r').reshape(-1) + snapshots.append(saved_array) + destination.append((saved_array, copy.deepcopy(data))) + saved, data = origin.pop() + for start in range(0, len(state), chunk_size): + state[start : start + chunk_size] = saved[start : start + chunk_size] + continue + if engine == 'streamed': + # Disk snapshots keep RAM bounded even for large reference runs. + snap = output / f'.history-{index}.npy' + state.flush() + shutil.copyfile(output / 'spike_clusters.npy', snap) + saved_array = np.load(snap, mmap_mode='r').reshape(-1) + snapshots.append(saved_array) + history.append((saved_array, copy.deepcopy(data))) + future.clear() + if op == 'label': + ids = [aliases[a] for a in action['ids']] + active = set() + for start in range(0, len(assignments()), chunk_size): + active.update(map(int, np.unique(assignments()[start : start + chunk_size]))) + if not ids or not set(ids) <= active: + raise ValueError('Label aliases must refer to active units') + if engine == 'ordinary': + state.label(action['field'], action['value'], ids) + else: + data.setdefault(action['field'], {}).update( + dict.fromkeys(ids, action['value']) + ) + continue + current = assignments() + next_id = max( + allocator, + ( + max( + ( + int(np.max(current[start : start + chunk_size])) + for start in range(0, len(current), chunk_size) + ), + default=-1, + ) + + 1 + ), + ) + if engine == 'ordinary': + next_id = int(state.clustering.new_cluster_id()) + active = set() + for start in range(0, len(current), chunk_size): + active.update(map(int, np.unique(current[start : start + chunk_size]))) + if op == 'merge': + parents = [aliases[a] for a in action['parents']] + if ( + len(parents) < 2 + or len(set(parents)) != len(parents) + or not set(parents) <= active + ): + raise ValueError('Merge requires distinct active parent aliases') + if engine == 'ordinary': + state.merge(parents) + else: + for start in range(0, len(current), chunk_size): + block = current[start : start + chunk_size] + block[np.isin(block, parents)] = next_id + if engine == 'streamed': + for values in data.values(): + for parent in parents: + values.pop(parent, None) + allocator = next_id + 1 + aliases[action['as']] = next_id + elif op == 'split': + parent, threshold = aliases[action['parent']], float(action['threshold']) + if parent not in active: + raise ValueError('Split parent alias is inactive') + if not np.isfinite(threshold): + raise ValueError('Threshold must be finite') + if engine == 'ordinary': + selected = np.flatnonzero( + (current == parent) & np.isfinite(amplitudes) & (amplitudes < threshold) + ) + if not len(selected) or len(selected) == np.count_nonzero(current == parent): + raise ValueError('Split must have both selected and remainder rows') + state.split(selected) + else: + selected_count, remainder_count = 0, 0 + for start in range(0, len(current), chunk_size): + mask = current[start : start + chunk_size] == parent + values = amplitudes[start : start + chunk_size] + below = np.isfinite(values) & (values < threshold) + selected_count += int(np.count_nonzero(mask & below)) + remainder_count += int(np.count_nonzero(mask & ~below)) + if not selected_count or not remainder_count: + raise ValueError('Split must have both selected and remainder rows') + for start in range(0, len(current), chunk_size): + block = current[start : start + chunk_size] + values = amplitudes[start : start + chunk_size] + mask = block == parent + below = np.isfinite(values) & (values < threshold) + block[mask & below] = next_id + block[mask & ~below] = next_id + 1 + if engine == 'streamed': + for values in data.values(): + values.pop(parent, None) + allocator = next_id + 2 + aliases[action['selected']], aliases[action['remainder']] = next_id, next_id + 1 + else: + raise ValueError(f'Unknown ledger operation: {op}') + persist() + for alias, expected in ledger.get('expected_labels', {}).items(): + for field, value in expected.items(): + if str(data[field].get(aliases[alias], '')) != str(value): + raise ValueError(f'Final label expectation failed: {alias}.{field}') + report = { + 'engine': engine, + 'evidence': ( + 'full ordinary Supervisor, no GUI views attached' + if engine == 'ordinary' + else 'streamed reference, NOT full ordinary GUI' + ), + 'revision': revision(), + 'source': str(source), + 'aliases': aliases, + 'ledger': ledger, + 'input_hashes': { + name: digest(source / name) + for name in ('spike_times.npy', 'spike_clusters.npy', 'amplitudes.npy') + }, + } + (output / 'comparison_reference.json').write_text(json.dumps(report, indent=2) + '\n') + return report + finally: + for saved_array in snapshots: + saved_array._mmap.close() + for snap in output.glob('.history-*.npy'): + snap.unlink() + dispose() + mapping = getattr(amplitudes, '_mmap', None) + if mapping is not None: + mapping.close() + + +def report_path(path, inputs): + """Reject existing paths and all locations inside either immutable input.""" + path = Path(path).resolve() + for root in inputs: + root = Path(root).resolve() + if path == root or root in path.parents: + raise ValueError('Report must be outside both input directories') + if path.exists(): + raise FileExistsError(f'Report already exists: {path}') + return path + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='command', required=True) + compare = sub.add_parser('compare') + compare.add_argument('ordinary') + compare.add_argument('bounded_replay') + compare.add_argument('--fields', nargs='+', default=['group', 'notes']) + compare.add_argument('--output', required=True) + compare.add_argument('--chunk-size', type=int, default=100000) + compare.add_argument( + '--source', help='Original immutable sorting: enable the whole-file contract.' + ) + compare.add_argument( + '--expectations', help='JSON map of replay unit IDs to explicit final field values.' + ) + compare.add_argument( + '--bounded-dir', help='Final round package, to verify preparation/parent binding.' + ) + reference = sub.add_parser('reference') + reference.add_argument('source') + reference.add_argument('ledger') + reference.add_argument('output') + reference.add_argument('--engine', choices=['ordinary', 'streamed'], default='ordinary') + reference.add_argument('--chunk-size', type=int, default=100000) + args = parser.parse_args() + if args.command == 'compare': + try: + if (args.expectations or args.bounded_dir) and not args.source: + raise ValueError('--expectations and --bounded-dir require --source') + inputs = [args.ordinary, args.bounded_replay] + if args.bounded_dir: + inputs.append(args.bounded_dir) + if args.source: + source_path = Path(args.source).resolve() + inputs.append(source_path.parent if source_path.is_file() else source_path) + destination = report_path(args.output, inputs) + result = compare_outputs( + args.ordinary, + args.bounded_replay, + args.fields, + args.chunk_size, + source=args.source, + expected_labels=json.loads(Path(args.expectations).read_text()) + if args.expectations + else None, + bounded_dir=args.bounded_dir, + ) + # Exclusive creation also rejects another writer winning after validation. + with destination.open('x') as stream: + json.dump(result, stream, indent=2, allow_nan=False) + stream.write('\n') + except (ValueError, OSError) as error: + print(json.dumps({'ok': False, 'error': str(error)})) + return 1 + print(json.dumps({'ok': result['ok'], 'report': args.output})) + return 0 if result['ok'] else 1 + try: + result = run_reference( + args.source, + json.loads(Path(args.ledger).read_text()), + args.output, + args.engine, + args.chunk_size, + ) + except (ValueError, OSError, KeyError) as error: + print(json.dumps({'ok': False, 'error': str(error)})) + return 1 + print(json.dumps({'ok': True, 'evidence': result['evidence'], 'output': args.output})) + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tools/curation_output_contract.py b/tools/curation_output_contract.py new file mode 100644 index 000000000..11442de05 --- /dev/null +++ b/tools/curation_output_contract.py @@ -0,0 +1,858 @@ +"""Source-grounded file contract for independent curation acceptance. + +This checker does not call the bounded assignment projector or metadata event +interpreter. It checks native files, full row identities, and cumulative action +semantics against the original source and explicit final curator expectations. +""" + +import ast +import csv +import hashlib +import json +import runpy +from pathlib import Path + +import numpy as np + +from phy.apps.large_recording.source import Source + + +def _json(path): + return json.loads(Path(path).read_text(encoding='utf-8')) + + +def _hash(path): + value = hashlib.sha256() + with Path(path).open('rb') as stream: + for block in iter(lambda: stream.read(1024 * 1024), b''): + value.update(block) + return value.hexdigest() + + +def _json_hash(value, newline=False): + text = json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False) + return hashlib.sha256((text + ('\n' if newline else '')).encode()).hexdigest() + + +def _cell(value): + if value is None or value == '': + return '' + if isinstance(value, (dict, list, bool)): + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False) + return str(value) + + +def _table(path): + with path.open(encoding='utf-8', newline='') as stream: + reader = csv.DictReader(stream, delimiter=',' if path.suffix == '.csv' else '\t') + columns = reader.fieldnames or [] + if ( + len(columns) < 2 + or columns.count('cluster_id') != 1 + or len(set(columns)) != len(columns) + ): + raise ValueError(f'Invalid metadata columns: {path.name}') + values = {field: {} for field in columns if field != 'cluster_id'} + seen = set() + for row in reader: + if None in row or any(v is None for v in row.values()): + raise ValueError(f'Malformed metadata row: {path.name}') + unit = int(row['cluster_id']) + if unit < 0 or unit in seen: + raise ValueError(f'Invalid or duplicate metadata unit: {path.name}') + seen.add(unit) + for field in values: + values[field][unit] = row[field] + return columns, values + + +def _native_metadata(path, source, fields, active_ids): + """Native canonical tables take precedence only over proven source aliases.""" + tables = { + p.name: _table(p) + for p in sorted([*path.glob('cluster_*.tsv'), *path.glob('cluster_*.csv')]) + if p.stem != 'cluster_info' + } + source_tables = {t['filename']: t for t in source.metadata_schema['tables']} + canonical, aliases = {}, [] + for field in fields: + name = f'cluster_{field}.tsv' + if name in tables and tables[name][0] == ['cluster_id', field]: + canonical[field] = tables[name][1][field] + result = {} + active_ids = set(active_ids) + + def current_rows(name, field, rows): + # Native loading omits completely blank imported columns, leaving their + # original files untouched even after some original IDs are retired. + # Recognize only an exact byte-pinned, entirely blank source column. + if ( + set(rows) - active_ids + and name in source_tables + and all(not _cell(v) for v in source.metadata.get(field, {}).values()) + and _hash(path / name) == _hash(source.path / name) + ): + aliases.append( + {'filename': name, 'field': field, 'policy': 'byte-pinned blank source column'} + ) + return {u: v for u, v in rows.items() if u in active_ids} + return rows + + for name, (columns, values) in tables.items(): + if len(values) == 1: + field = next(iter(values)) + if name == f'cluster_{field}.tsv' and field in canonical: + if field not in fields: + raise ValueError(f'Unexpected native metadata field: {field}') + result[field] = current_rows(name, field, canonical[field]) + continue + descriptor = source_tables.get(name) + if descriptor is None or columns != descriptor['columns']: + raise ValueError(f'Unsupported ordinary metadata layout: {name}') + if _hash(path / name) != _hash(source.path / name): + raise ValueError(f'Ordinary source alias changed: {name}') + for field, rows in values.items(): + if field in canonical: + aliases.append( + {'filename': name, 'field': field, 'policy': 'unchanged source alias'} + ) + elif field in result: + raise ValueError(f'Ambiguous ordinary metadata field: {field}') + else: + result[field] = current_rows(name, field, rows) + return result, aliases, set(tables) + + +def _parameters(path, source): + values = { + k: v for k, v in runpy.run_path(str(path / 'params.py')).items() if not k.startswith('__') + } + + def parameter_code(filename): + tree = ast.parse(filename.read_bytes()) + tree.body = [ + node + for node in tree.body + if not ( + isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in ('dat_path', 'num_samples', 'valid_intervals') + ) + ] + return ast.dump(tree, include_attributes=False) + + if parameter_code(path / 'params.py') != parameter_code(source.params_path): + raise ValueError('Changed parameter Python outside managed overrides') + allowed_added = {'num_samples', 'valid_intervals'} + if set(values) - set(source.params) - allowed_added: + raise ValueError('Unexpected added parameter keys') + if set(source.params) - set(values): + raise ValueError('Missing source parameter keys') + for key, value in source.params.items(): + if key in ('dat_path', 'num_samples', 'valid_intervals'): + continue + if callable(value) or type(value).__name__ == 'module': + continue # Unchanged helper/import syntax was checked above. + if not np.array_equal(values[key], value): + raise ValueError(f'Changed source parameter: {key}') + if values.get('num_samples', source.num_samples) != source.num_samples: + raise ValueError('Changed recording extent') + if not np.array_equal( + values.get('valid_intervals', source.valid_intervals), source.valid_intervals + ): + raise ValueError('Changed acquisition exposure') + + def raw_paths(value, base): + items = value if isinstance(value, (tuple, list)) else [value] + return [ + str((base / str(p)).resolve()) + for p in items + if p is not None and str(p).strip() not in ('', 'None') + ] + + expected = raw_paths(source.params.get('dat_path'), source.path) + actual = raw_paths(values.get('dat_path'), path) + if actual != expected: + raise ValueError('Changed raw data reference') + + +def _history_records(provenance): + if not isinstance(provenance, dict) or not isinstance(provenance.get('history'), list): + raise ValueError('Malformed metadata provenance/history') + if any( + not isinstance(record, dict) or not isinstance(record.get('operation'), str) + for record in provenance['history'] + ): + raise ValueError('Malformed metadata history record') + return provenance['history'] + + +def _predicate(value, arrays): + if not isinstance(value, dict) or set(value) != { + 'attribute', + 'operator', + 'threshold', + 'nonfinite', + }: + raise ValueError('Malformed threshold predicate') + threshold = value['threshold'] + if ( + value['attribute'] not in ('amplitudes.npy', 'spike_templates.npy') + or value['attribute'] not in arrays + or value['operator'] not in ('<', '<=', '>', '>=') + or value['nonfinite'] not in ('keep', 'reject') + or isinstance(threshold, bool) + or not isinstance(threshold, (int, float)) + or not np.isfinite(threshold) + ): + raise ValueError('Unsupported threshold predicate schema') + + +def _event_schema(event, record, arrays): + operation = event.get('operation') + keys = { + 'merge': {'version', 'operation', 'parents', 'target', 'metadata'}, + 'threshold_split': {'version', 'operation', 'parent', 'children', 'predicate', 'metadata'}, + 'label': {'version', 'operation', 'field', 'ids', 'value'}, + } + if ( + operation not in keys + or set(event) != keys[operation] + or type(event['version']) is not int + or event['version'] != 2 + ): + raise ValueError('Malformed current journal event schema') + if operation == 'label': + return + metadata = event['metadata'] + # New memberships also introduce the built-in unreviewed group field. + fields = set(record['parent_metadata']) | {'group'} + if not isinstance(metadata, dict) or set(metadata) != fields: + raise ValueError('Journal membership metadata field coverage disagrees') + if operation == 'merge': + valid = all(value is None for value in metadata.values()) + else: + _predicate(event['predicate'], arrays) + children = {str(u) for u in event['children']} + valid = all( + isinstance(values, dict) + and set(values) == children + and all(value is None for value in values.values()) + for values in metadata.values() + ) + if not valid: + raise ValueError('Journal membership metadata contradicts clearing policy') + + +def _history(source, curated, journal, manifest, expected_labels): + """Track only action ancestry and explicit label intent, never spike projection.""" + original = _history_records(source.metadata_provenance) + history = _history_records(curated.metadata_provenance) + if history[: len(original)] != original: + raise ValueError('Lost original metadata provenance') + if set(curated.metadata_provenance) != set(source.metadata_provenance) or any( + curated.metadata_provenance[k] != value + for k, value in source.metadata_provenance.items() + if k != 'history' + ): + raise ValueError('Changed non-history metadata provenance') + inherited = ( + _history_records(manifest.get('metadata_provenance', {'history': []})) + if manifest is not None + else original + ) + if history[: len(inherited)] != inherited or len(history) != len(inherited) + len(journal): + raise ValueError('Cumulative metadata history disagrees with preparation/journal') + for event_index, (event, record) in enumerate( + zip(journal, history[len(inherited) :], strict=True) + ): + _event_schema(event, record, source.arrays) + operation = event['operation'] + if operation != record['operation'] or event.get('version') != 2: + raise ValueError('Journal operation/version disagrees with history') + if operation == 'merge': + valid = record['parents'] == event['parents'] and record['children'] == [ + event['target'] + ] + elif operation == 'threshold_split': + valid = ( + record['parents'] == [event['parent']] + and record['children'] == event['children'] + and record['predicate'] == event['predicate'] + ) + elif operation == 'label': + valid = record == event + else: + raise ValueError('Unsupported metadata policy transition in contract') + if not valid: + raise ValueError('Journal details disagree with cumulative history') + if operation in ('merge', 'threshold_split'): + parent_revision = ( + manifest['source']['revision'] if manifest is not None else source.revision + ) + if record['event_index'] != event_index: + raise ValueError('Membership provenance has wrong event index') + if record['source_revision'] != parent_revision: + raise ValueError('Membership provenance has wrong source revision') + fields = set(source.metadata) + if any( + record['operation'] in ('merge', 'threshold_split') for record in history[len(original) :] + ): + fields.add('group') + fields.update( + record['field'] for record in history[len(original) :] if record['operation'] == 'label' + ) + if any(set(values) - fields for values in expected_labels.values()): + raise ValueError('Expected labels have no source field or recorded label intent') + if set(curated.metadata) != fields: + raise ValueError('Unexpected replay metadata fields without label intent') + candidates = { + int(u): {f: {_cell(source.metadata.get(f, {}).get(int(u)))} for f in fields} + for u in source.unit_ids + } + resolved = { + f: {int(u): _cell(v) for u, v in values.items()} for f, values in source.metadata.items() + } + active = set(candidates) + next_id = source.next_cluster_id + retired = dict(source.retired_units) + explicit = {} + for record in history[len(original) :]: + operation = record['operation'] + id_lists = ('ids',) if operation == 'label' else ('parents', 'children') + for key in id_lists: + ids = record.get(key) + if ( + not isinstance(ids, list) + or not ids + or any(type(u) is not int or u < 0 for u in ids) + or len(set(ids)) != len(ids) + ): + raise ValueError('Malformed unit IDs in metadata history') + if operation == 'label': + field, value = record['field'], _cell(record['value']) + if field not in fields or not set(record['ids']) <= active: + raise ValueError('History label has unknown field/unit') + for unit in record['ids']: + candidates[unit][field] = {value} + explicit.setdefault(unit, {})[field] = value + resolved.setdefault(field, {})[unit] = value + elif operation in ('merge', 'threshold_split'): + record_keys = { + 'operation', + 'source_revision', + 'event_index', + 'parents', + 'children', + 'parent_metadata', + 'policy', + } + if operation == 'threshold_split': + record_keys.add('predicate') + if set(record) != record_keys: + raise ValueError('Unexpected membership provenance fields') + parents, children = record['parents'], record['children'] + if (operation == 'merge' and len(parents) < 2) or ( + operation == 'threshold_split' and len(parents) != 1 + ): + raise ValueError('Invalid membership parent cardinality') + if operation == 'threshold_split': + _predicate(record['predicate'], source.arrays) + if ( + record.get('policy') != 'membership_v1' + or len(set(parents)) != len(parents) + or not set(parents) <= active + ): + raise ValueError('Unsupported or inactive membership history') + if ( + len(children) != (1 if operation == 'merge' else 2) + or len(set(children)) != len(children) + or min(children) < next_id + ): + raise ValueError('Invalid allocated history identities') + snapshot = { + f: {str(u): values.get(u, '') for u in parents} for f, values in resolved.items() + } + recorded = { + f: {str(u): _cell(v) for u, v in values.items()} + for f, values in record['parent_metadata'].items() + } + if snapshot != recorded: + raise ValueError('Parent metadata provenance disagrees with independent history') + inherited = {f: set().union(*(candidates[u][f] for u in parents)) for f in fields} + active.difference_update(parents) + for parent in parents: + retired[str(parent)] = {'operation': operation, 'replacements': children} + for parent in parents: + explicit.pop(parent, None) + for values in resolved.values(): + values.pop(parent, None) + resolved.setdefault('group', {}) + for child in children: + candidates[child] = {f: set(v) for f, v in inherited.items()} + for values in resolved.values(): + values[child] = '' + active.add(child) + next_id = max(children) + 1 + else: + raise ValueError(f'Unsupported cumulative history operation: {operation}') + if retired != curated.retired_units: + raise ValueError('Retired unit lineage disagrees with independent history') + if curated.next_cluster_id < next_id: + raise ValueError('Allocation floor disagrees with cumulative history') + if active != set(map(int, curated.unit_ids)): + raise ValueError('Final catalog disagrees with independent action ancestry') + if not set(expected_labels) <= active: + raise ValueError('Explicit expectations refer to inactive units') + for unit, values in explicit.items(): + for field, value in values.items(): + if expected_labels.get(unit, {}).get(field) != value: + raise ValueError( + f'Missing or wrong independent final label expectation: {unit}.{field}' + ) + for unit, values in expected_labels.items(): + for field, value in values.items(): + intended = explicit.get(unit, {}).get( + field, + _cell(source.metadata.get(field, {}).get(unit)) + if unit in set(map(int, source.unit_ids)) + else '', + ) + if value != intended: + raise ValueError('Independent expectation disagrees with recorded label intent') + return fields, candidates, explicit, active, history[len(original) :] + + +def _assignment_oracle(source, curated, history, chunk_size): + """Independently execute cumulative membership intent on every original row. + + Attribute values always come from the immutable original source. Only a + chunk of assignments is copied, including when earlier rounds split a unit + that is subsequently merged or split again. Labels have no row effect. + """ + original = source.arrays['spike_clusters.npy'].reshape(-1) + current = curated.arrays['spike_clusters.npy'].reshape(-1) + membership = [r for r in history if r['operation'] != 'label'] + attributes = {} + for record in membership: + if record['operation'] != 'threshold_split': + continue + name = record['predicate']['attribute'] + values = source.arrays[name] + if values.shape not in ((len(original),), (len(original), 1)): + raise ValueError('Predicate attribute must be a scalar per source spike') + attributes[name] = values.reshape(-1) + if len(current) != len(original): + raise ValueError('Assignment oracle requires complete source rows') + for start in range(0, len(original), chunk_size): + stop = start + chunk_size + expected = np.array(original[start:stop], dtype=np.int64, copy=True) + for record in membership: + parents, children = record['parents'], record['children'] + if record['operation'] == 'merge': + expected[np.isin(expected, parents)] = children[0] + continue + parent_rows = np.flatnonzero(expected == parents[0]) + predicate = record['predicate'] + values = attributes[predicate['attribute']][start:stop][parent_rows] + finite = np.isfinite(values) + if predicate['nonfinite'] == 'reject' and not finite.all(): + raise ValueError('Predicate rejects nonfinite values in source parent rows') + threshold = predicate['threshold'] + operator = predicate['operator'] + if operator == '<': + selected = values < threshold + elif operator == '<=': + selected = values <= threshold + elif operator == '>': + selected = values > threshold + else: # Schema has already restricted the fourth operator to >=. + selected = values >= threshold + # Nonfinite values always remain in the first (remainder) child. + expected[parent_rows] = np.where(finite & selected, children[1], children[0]) + mismatch = np.flatnonzero(expected != current[start:stop]) + if len(mismatch): + row = start + int(mismatch[0]) + raise ValueError( + f'Assignment disagrees with cumulative source predicates at spike {row}' + ) + return { + 'verified': True, + 'source_rows': len(original), + 'membership_events': len(membership), + 'threshold_splits': sum(r['operation'] == 'threshold_split' for r in membership), + } + + +def _local_file(root, name): + if ( + not isinstance(name, str) + or not name + or Path(name).name != name + or any(c in name for c in ('\\', '\0', '\n', '\r')) + ): + raise ValueError('Unsafe checkpoint asset name') + path = root / name + if path.is_symlink() or not path.is_file(): + raise ValueError('Missing or linked checkpoint asset') + return path + + +def _checkpoint(bounded, manifest, lineage, journal, right, curated, chunk_size): + """Bind the exact saved generation, including allocator and retained rows.""" + root = Path(bounded) + pointer = _json(root / '.phy-curation/current.json') + generation = pointer['generation'] + if ( + not isinstance(generation, str) + or len(generation) != 32 + or any(c not in '0123456789abcdef' for c in generation) + ): + raise ValueError('Invalid checkpoint generation') + directory = root / '.phy-curation/generations' / generation + descriptor = _json(_local_file(directory, 'checkpoint.json')) + if set(pointer) != {'generation', 'sha256'} or descriptor.get('version') != 1: + raise ValueError('Unsupported checkpoint pointer/descriptor') + if pointer['sha256'] != _json_hash(descriptor, newline=True) or descriptor[ + 'manifest_sha256' + ] != _json_hash(manifest, newline=True): + raise ValueError('Checkpoint pointer/manifest binding changed') + required = {'spike_clusters.npy', 'journal.json', 'metadata.json'} + inventory = descriptor['files'] + if not required <= set(inventory) or not set(descriptor['extra_documents']) <= set(inventory): + raise ValueError('Checkpoint inventory is incomplete') + if {p.name for p in directory.iterdir()} != set(inventory) | {'checkpoint.json'}: + raise ValueError('Unexpected checkpoint generation assets') + for name, digest in inventory.items(): + if _hash(_local_file(directory, name)) != digest: + raise ValueError('Checkpoint asset hash changed') + if _json(directory / 'journal.json') != journal or _json(directory / 'metadata.json') != _json( + right / '_phy_cluster_metadata.json' + ): + raise ValueError('Export differs from saved journal/metadata generation') + for key in ('next_cluster_id', 'metadata_policy', 'metadata_provenance'): + if descriptor[key] != lineage[key]: + raise ValueError(f'Export differs from saved checkpoint: {key}') + if descriptor['metadata_schema'] != curated.metadata_schema: + raise ValueError('Export metadata schema differs from saved checkpoint') + if 'curation_review.json' in inventory and _json(directory / 'curation_review.json') != _json( + right / 'curation_review.json' + ): + raise ValueError('Export review differs from saved checkpoint') + name = manifest['global_spike_ids_file'] + ids_path = _local_file(root, name) + if _hash(ids_path) != manifest['bounded_files'][name]: + raise ValueError('Retained row identity hash changed') + ids = saved = None + try: + ids = np.load(ids_path, mmap_mode='r', allow_pickle=False) + saved = np.load(directory / 'spike_clusters.npy', mmap_mode='r', allow_pickle=False) + if ( + ids.dtype != np.int64 + or ids.shape != (manifest['bounded_num_spikes'],) + or saved.shape != ids.shape + or saved.dtype.kind not in 'iu' + ): + raise ValueError('Invalid saved retained assignment alignment') + if np.any(ids < 0) or np.any(ids >= curated.n_spikes) or np.any(ids[1:] <= ids[:-1]): + raise ValueError('Invalid retained row coordinates') + full = curated.arrays['spike_clusters.npy'].reshape(-1) + for start in range(0, len(ids), chunk_size): + if not np.array_equal( + full[ids[start : start + chunk_size]], saved[start : start + chunk_size] + ): + raise ValueError('Full replay disagrees with saved retained assignments') + finally: + for array in (ids, saved): + if array is not None: + array._mmap.close() + return generation + + +def compare_file_contract( + left, right, source_path, mapping, expected_labels=None, bounded_dir=None, chunk_size=100000 +): + """Return file report plus normalized native/replay label maps. + + ``source_path`` is the original immutable source even across multiple rounds. + Current preparation binding is verified when ``bounded_dir`` is supplied; + older revisions are recorded ancestry, not independently fetched snapshots. + """ + left, right = Path(left), Path(right) + source_path = Path(source_path) + params = source_path if source_path.is_file() else source_path / 'params.py' + expected_labels = { + int(u): {f: _cell(v) for f, v in values.items()} + for u, values in (expected_labels or {}).items() + } + source = curated = None + try: + source = Source(params, batch_size=chunk_size) + curated = Source(right / 'params.py', batch_size=chunk_size) + if source.identity != curated.identity: + raise ValueError('Replay immutable coordinate identity differs from original source') + lineage = _json(right / '_phy_curation_lineage.json') + lineage_keys = { + 'format', + 'format_version', + 'coordinate_identity', + 'parent_revision', + 'journal_sha256', + 'revision', + 'preparation_sha256', + 'next_cluster_id', + 'unit_ids', + 'retired_units', + 'proposition_documents', + 'metadata_provenance', + 'metadata_policy', + } + if set(lineage) != lineage_keys: + raise ValueError('Unexpected lineage fields') + proposition_hashes = { + name: _hash(right / name) + for name in ('curation.json', 'curation_review.json') + if (right / name).exists() + } + if lineage['proposition_documents'] != proposition_hashes: + raise ValueError('Missing or wrong proposition lineage bindings') + journal = _json(right / '_phy_curation_journal.json') + if lineage.get('metadata_policy') != 'membership_v1': + raise ValueError('Strict file contract requires membership_v1 replay') + if lineage.get('journal_sha256') != _json_hash(journal, newline=True): + raise ValueError('Journal hash disagrees with lineage') + manifest = None + if bounded_dir is not None: + manifest = _json(Path(bounded_dir) / '_phy_curation_manifest.json') + if ( + manifest.get('format') != 'phy_bounded_curation' + or manifest.get('format_version') != 1 + ): + raise ValueError('Unsupported bounded preparation format') + if lineage.get('preparation_sha256') != _json_hash(manifest, newline=True): + raise ValueError('Preparation hash disagrees with lineage') + if ( + lineage.get('parent_revision') != manifest['source']['revision'] + or manifest['source']['identity'] != source.identity + ): + raise ValueError('Lineage parent/identity disagrees with preparation') + elif lineage.get('parent_revision') != source.revision: + raise ValueError('Later-round parent verification requires bounded_dir') + fields, candidates, explicit, active, history = _history( + source, curated, journal, manifest, expected_labels + ) + saved_generation = None + if bounded_dir is not None: + saved_generation = _checkpoint( + Path(bounded_dir), manifest, lineage, journal, right, curated, chunk_size + ) + native, aliases, native_tables = _native_metadata(left, source, fields, mapping.keys()) + if set(native) != fields: + raise ValueError('Missing native metadata field/table coverage') + replay_metadata = { + f: {int(u): _cell(v) for u, v in values.items()} + for f, values in curated.metadata.items() + } + mapped = {int(a): int(b) for a, b in mapping.items()} + reverse = {b: a for a, b in mapped.items()} + if len(reverse) != len(mapped) or not set(reverse) <= active: + raise ValueError('Partition mapping disagrees with final catalog') + untouched = set(map(int, source.unit_ids)) & active + assignment_oracle = _assignment_oracle(source, curated, history, chunk_size) + differences = [] + for field in fields: + for unit in active: + value = replay_metadata.get(field, {}).get(unit, '') + expected = expected_labels.get(unit, {}).get( + field, + _cell(source.metadata.get(field, {}).get(unit)) if unit in untouched else '', + ) + if value != expected: + raise ValueError( + f'Replay metadata violates label/clearing policy: {unit}.{field}' + ) + if unit not in reverse: + # Empty source units are deliberately not silently matched. + raise ValueError(f'Final metadata unit has no mapped source rows: {unit}') + left_id = reverse[unit] + native_value = native.get(field, {}).get(left_id, '') + allowed = ( + {expected} + if field in expected_labels.get(unit, {}) or unit in untouched + else candidates[unit][field] | {''} + ) + if native_value not in allowed: + raise ValueError( + f'Ordinary metadata has unsupported ancestor value: {left_id}.{field}' + ) + if native_value != value: + differences.append( + { + 'field': field, + 'left_id': left_id, + 'right_id': unit, + 'left': native_value, + 'right': value, + 'policy': 'native ancestor inheritance versus cleared new membership', + } + ) + if set(native.get(field, {})) - set(mapped): + raise ValueError(f'Unmapped ordinary metadata units: {field}') + for path in (left, right): + _parameters(path, source) + immutable = set(source.arrays) - {'spike_clusters.npy'} + immutable.update( + n + for n in ('_phy_spikes_subset.manifest.json', 'timestamps.json', 'curation.json') + if (source.path / n).exists() + ) + for name in sorted(immutable): + expected_hash = _hash(source.path / name) + for path in (left, right): + if not (path / name).is_file() or _hash(path / name) != expected_hash: + raise ValueError(f'Immutable asset missing or changed: {path.name}/{name}') + # Source layouts are authoritative for replay; only new label tables may be added. + source_schema = {t['filename']: t for t in source.metadata_schema['tables']} + right_schema = {t['filename']: t for t in curated.metadata_schema['tables']} + for name, table in source_schema.items(): + if right_schema.get(name) != table: + raise ValueError(f'Replay source metadata layout changed: {name}') + for name, table in right_schema.items(): + if name not in source_schema: + non_id = [f for f in table['columns'] if f != 'cluster_id'] + if ( + len(non_id) != 1 + or name != f'cluster_{non_id[0]}.tsv' + or table['columns'] != ['cluster_id', non_id[0]] + or non_id[0] in source.metadata + ): + raise ValueError(f'Unexpected replay metadata table: {name}') + for path in (left, right): + review = path / 'curation_review.json' + base_review = ( + _json(source.path / 'curation_review.json') + if (source.path / 'curation_review.json').exists() + else {'reviews': {}} + ) + if review.exists(): + from phy.cluster._propositions import decode_review_mapping + + doc = _json(review) + decode_review_mapping(doc) + if ( + doc.get('source', {}).get('filename') != 'curation.json' + or doc.get('source', {}).get('sha256') != _hash(source.path / 'curation.json') + or doc['reviews'] != base_review['reviews'] + ): + raise ValueError( + 'Review binding or inherited decisions changed without expectations' + ) + elif base_review['reviews'] or ( + path == right and (source.path / 'curation.json').exists() + ): + raise ValueError('Missing inherited or generated proposition reviews') + right_required = ( + immutable + | { + 'spike_clusters.npy', + 'params.py', + '_phy_cluster_metadata.json', + '_phy_cluster_metadata_schema.json', + '_phy_curation_lineage.json', + '_phy_curation_journal.json', + } + | set(right_schema) + ) + if (source.path / 'curation.json').exists(): + right_required.add('curation_review.json') + left_required = immutable | {'spike_clusters.npy', 'params.py'} | native_tables + derived_static = [] + inverse_name = 'whitening_mat_inv.npy' + if inverse_name not in source.arrays and (left / inverse_name).exists(): + matrix = source.arrays.get('whitening_mat.npy') + matrix = np.eye(len(source.arrays['channel_map.npy'])) if matrix is None else matrix + expected_inverse = np.linalg.inv(matrix) + actual_inverse = np.load(left / inverse_name, allow_pickle=False) + if actual_inverse.dtype != expected_inverse.dtype or not np.array_equal( + actual_inverse, expected_inverse + ): + raise ValueError('Invalid native derived whitening inverse') + left_required.add(inverse_name) + derived_static.append(inverse_name) + inventories = {} + for role, path, required in ( + ('ordinary', left, left_required), + ('replay', right, right_required), + ): + names = {p.name for p in path.iterdir()} + optional = {'phy.log', 'cluster_info.tsv', '.phy'} if role == 'ordinary' else set() + if role == 'ordinary': + optional.add('curation_review.json') + for name in ( + '_phy_cluster_metadata.json', + '_phy_cluster_metadata_schema.json', + '_phy_curation_lineage.json', + '_phy_curation_journal.json', + ): + if (source.path / name).exists() and (path / name).exists(): + if _hash(source.path / name) != _hash(path / name): + raise ValueError(f'Native source snapshot changed: {name}') + optional.add(name) + if required - names: + raise ValueError(f'Missing {role} assets: {sorted(required - names)}') + if names - required - optional: + raise ValueError(f'Unexpected {role} files: {sorted(names - required - optional)}') + for name in names: + item = path / name + if item.is_symlink() or (item.is_dir() and name != '.phy'): + raise ValueError(f'Unexpected non-file output asset: {name}') + if (path / 'cluster_info.tsv').exists(): + _, summary = _table(path / 'cluster_info.tsv') + summary_ids = set(next(iter(summary.values()))) + if summary_ids != set(mapped): + raise ValueError('Generated cluster_info catalog disagrees') + for field in {'group', 'notes'} | { + f for values in expected_labels.values() for f in values + }: + expected_map = ( + native.get(field, {}) + if role == 'ordinary' + else replay_metadata.get(field, {}) + ) + for unit, value in summary.get(field, {}).items(): + if unit not in mapped or value != expected_map.get(unit, ''): + raise ValueError('Generated cluster_info manual labels disagree') + inventories[role] = { + 'required': sorted(required), + 'state_artifacts': sorted(names - required), + 'sha256': {name: _hash(path / name) for name in sorted(required)}, + } + return ( + { + 'ok': True, + 'assignment_oracle': assignment_oracle, + 'immutable_files': sorted(immutable), + 'native_derived_static_files': derived_static, + 'all_metadata_fields': sorted(fields), + 'permitted_metadata_differences': differences, + 'ordinary_aliases': aliases, + 'inventories': inventories, + 'preparation_verified': manifest is not None, + 'saved_generation_verified': saved_generation, + 'historical_parents': ( + 'recorded cumulative ancestry; current parent bound to supplied manifest' + if manifest is not None + else 'current parent matches original source; preparation not supplied' + ), + }, + native, + replay_metadata, + ) + except (TypeError, AttributeError, IndexError) as error: + raise ValueError(f'Malformed contract document: {error}') from error + finally: + if curated is not None: + curated.close() + if source is not None: + source.close() diff --git a/tools/validate_aind_export.py b/tools/validate_aind_export.py new file mode 100644 index 000000000..122ddf0d3 --- /dev/null +++ b/tools/validate_aind_export.py @@ -0,0 +1,220 @@ +"""Validate an AIND-produced sorting against Phy's declared source contract. + +This checks software interoperability. It cannot establish acquisition truth; +the exporter revision and authoritative exposure/filler policy must be supplied +and reviewed by the producer. +""" + +import argparse +import json +import sys +import time +from pathlib import Path + +REPOSITORY = Path(__file__).resolve().parents[1] +if str(REPOSITORY) not in sys.path: + sys.path.insert(0, str(REPOSITORY)) + +from phy.apps.large_recording.prepare import prepare_large +from phy.apps.large_recording.source import Source, hash_file + +REPORT_FORMAT = 'phy_aind_export_validation' +REPORT_VERSION = 1 + + +def _failure(error): + return {'type': type(error).__name__, 'message': str(error)} + + +def _cli_failure(error, args): + report = { + 'format': REPORT_FORMAT, + 'format_version': REPORT_VERSION, + 'status': 'failed', + 'failures': [_failure(error)], + 'engineering_validation_only': True, + 'acquisition_truth_established': False, + 'phy_revision': args.phy_revision, + 'exporter_revision': args.exporter_revision, + } + sys.stdout.write(json.dumps(report, indent=2, sort_keys=True) + '\n') + return 1 + + +def _input_inventory(source_path): + """Hash regular files in the source without interpreting their contents.""" + return { + path.name: {'bytes': path.stat().st_size, 'sha256': hash_file(path)} + for path in sorted(source_path.iterdir()) + if path.is_file() + } + + +def validate_export( + params_path, + *, + phy_revision, + exporter_revision, + output_dir=None, + max_spikes=2_000_000, + max_bytes=536_870_912, + batch_size=100_000, +): + """Return a JSON-serializable validation report; never edits the source.""" + started = time.perf_counter() + params_path = Path(params_path).resolve() + source_path = params_path.parent + report = { + 'format': REPORT_FORMAT, + 'format_version': REPORT_VERSION, + 'status': 'failed', + 'failures': [], + 'engineering_validation_only': True, + 'acquisition_truth_established': False, + 'requires_authoritative_exposure_and_filler_policy': True, + 'phy_revision': phy_revision, + 'exporter_revision': exporter_revision, + 'source': {'params_path': str(params_path), 'path': str(source_path)}, + 'budgets': {'max_spikes': int(max_spikes), 'max_bytes': int(max_bytes)}, + 'timing_seconds': {}, + } + if not phy_revision or not exporter_revision: + report['failures'].append( + _failure(ValueError('Both Phy and exporter revisions must be supplied.')) + ) + report['timing_seconds']['total'] = time.perf_counter() - started + return report + try: + phase = time.perf_counter() + with Source(params_path, batch_size=batch_size) as source: + report['timing_seconds']['source_validation'] = time.perf_counter() - phase + report['source'].update( + { + 'sha256': source.identity['sha256'], + 'num_spikes': source.n_spikes, + 'num_units': len(source.unit_ids), + 'sample_rate_hz': source.sample_rate, + 'num_samples': source.num_samples, + 'time_base': source.time_base, + 'exposure': { + 'valid_intervals': source.valid_intervals.tolist(), + 'valid_samples': int( + (source.valid_intervals[:, 1] - source.valid_intervals[:, 0]).sum() + ), + 'valid_seconds': float( + (source.valid_intervals[:, 1] - source.valid_intervals[:, 0]).sum() + / source.sample_rate + ), + }, + 'array_counts': { + name: {'rows': int(len(array)), 'shape': list(array.shape)} + for name, array in sorted(source.arrays.items()) + }, + } + ) + source_root = source.path + declared_intervals = ( + 'valid_intervals' in source.params + or 'valid_intervals' in source.upstream_recording + ) + if source.time_base is None: + raise ValueError('AIND export requires an explicit time_base contract.') + if source.time_base['kind'] != 'elapsed_ticks': + raise ValueError('AIND export time_base.kind must be elapsed_ticks.') + if not declared_intervals: + raise ValueError( + 'AIND export requires explicit authoritative valid_intervals in params.py ' + 'or the waveform manifest.' + ) + phase = time.perf_counter() + report['source']['input_files'] = _input_inventory(source_root) + report['timing_seconds']['input_hashing'] = time.perf_counter() - phase + if output_dir is not None: + destination = Path(output_dir).resolve() + if destination == source_root or source_root in destination.parents: + raise ValueError('Prepared output must be outside the immutable source directory.') + if destination.exists(): + raise ValueError(f'Prepared output must be fresh; already exists: {destination}.') + phase = time.perf_counter() + prepared = prepare_large( + params_path, + destination, + max_spikes=max_spikes, + max_bytes=max_bytes, + batch_size=batch_size, + ) + report['timing_seconds']['prepare'] = time.perf_counter() - phase + manifest = json.loads((prepared / '_phy_curation_manifest.json').read_text()) + report['prepared_output'] = { + 'path': str(prepared), + 'num_spikes': manifest['bounded_num_spikes'], + 'num_units': len(manifest['unit_ids']), + 'waveform_coverage': manifest['waveform_coverage'], + 'budgets': manifest['budgets'], + 'output_bytes': sum(p.stat().st_size for p in prepared.rglob('*') if p.is_file()), + } + except Exception as error: + report['failures'].append(_failure(error)) + if not report['failures']: + report['status'] = 'passed' + report['timing_seconds']['total'] = time.perf_counter() - started + return report + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('params', type=Path, help='AIND sorting params.py') + parser.add_argument('--phy-revision', required=True) + parser.add_argument('--exporter-revision', required=True) + parser.add_argument( + '--output-dir', type=Path, help='Optional fresh bounded package destination' + ) + parser.add_argument('--max-spikes', type=int, default=2_000_000) + parser.add_argument('--max-bytes', type=int, default=536_870_912) + parser.add_argument('--batch-size', type=int, default=100_000) + parser.add_argument( + '--report', type=Path, help='Write JSON report to this path (default stdout)' + ) + args = parser.parse_args(argv) + if args.report: + source_root = args.params.resolve().parent + report_path = args.report.resolve() + protected_roots = [source_root] + if args.output_dir is not None: + protected_roots.append(args.output_dir.resolve()) + if any(report_path == root or root in report_path.parents for root in protected_roots): + failure = ValueError( + 'Report path must be outside the immutable source and prepared output directories.' + ) + elif args.report.exists() or args.report.is_symlink(): + failure = ValueError(f'Report path must be fresh; already exists: {args.report}.') + else: + failure = None + if failure is not None: + return _cli_failure(failure, args) + report = validate_export( + args.params, + phy_revision=args.phy_revision, + exporter_revision=args.exporter_revision, + output_dir=args.output_dir, + max_spikes=args.max_spikes, + max_bytes=args.max_bytes, + batch_size=args.batch_size, + ) + payload = json.dumps(report, indent=2, sort_keys=True) + '\n' + if args.report: + try: + with args.report.open('x', encoding='utf-8') as stream: + stream.write(payload) + except FileExistsError: + return _cli_failure( + FileExistsError(f'Report path must be fresh; already exists: {args.report}.'), + args, + ) + else: + sys.stdout.write(payload) + return 0 if report['status'] == 'passed' else 1 + + +if __name__ == '__main__': + raise SystemExit(main())