Support bounded curation and full-source replay for long recordings

design/README.md

@@ -57,3 +57,19 @@ and section 11 for integration and manual acceptance.
 
 The implementation and user documentation are complete on the unreleased phy
 2.2 branch. Final large-dataset smoke testing and save/reopen validation remain.
+
+## Long-recording curation (proposed)
+
+The [long-recording curation Phy-side design contract](large-recording-curation.md)
+describes bounded local datasets for ~1000-hour sorting datasets, the three spike-index
+spaces, optional upstream waveform evidence, full-duration metrics, replay safety,
+and the implementation stages. The accompanying synthetic 1000-hour
+fixture and reference tests are in `phy/apps/tests/large_recording_fixtures.py`
+and `phy/apps/tests/test_large_recording_contract.py`.
+
+This is a proposal and a reference test contract, **not yet** a `prepare-large`
+implementation or an upstream export-format agreement.
+
+The [integration audit and plan](large-recording-integration-plan.md) refines
+the contract with correctness findings, module boundaries, ordered commits in one PR and
+acceptance gates. Follow its gates before enabling editable bounded sessions.

design/aind-export-handoff.md

@@ -0,0 +1,90 @@
+# AIND export handoff for elapsed-time coordinates
+
+This note describes the producer work needed before an AIND export can use
+Phy's explicit elapsed-tick contract. It is a handoff, not evidence that an
+exporter change has been implemented or that a real export has passed
+acceptance. The exporter source repository was not available in this worktree;
+the steps below are proposed commit-sized changes, with no repository or branch
+assumed.
+
+## Proposed exporter changes
+
+1. **Establish authoritative acquisition truth.** Obtain acquisition logs or
+   equivalent source records for clock origin, piece boundaries, dropped or
+   invalid spans, filler, and the physical recording extent. Resolve the
+   reported interruptions and the 35 sorting spikes reported inside filler.
+   Do not infer validity solely from spike times or endpoint interpolation.
+2. **Emit a canonical event time axis.** For each sorting spike, emit a
+   nonnegative integer elapsed tick from the source time origin. Declare
+   `time_base = {'kind': 'elapsed_ticks', 'ticks_per_second': ...}`. Set
+   `num_samples` to the exclusive bound that includes the last representable
+   event tick; retain a physical frame count separately (for example as
+   `recording.num_frames`). Do not call a tick bound a frame count.
+3. **Emit authoritative exposure.** Emit ordered, disjoint integer
+   `valid_intervals` in the same elapsed-tick coordinates, excluding missing
+   acquisition time and unusable filler. Ensure every emitted sorting spike
+   belongs to a valid interval. If any sorting rows fall outside exposure,
+   resolve them using acquisition truth upstream; never silently drop, move, or
+   relabel rows.
+4. **Emit aligned waveform and window evidence.** Store
+   `_phy_spikes_subset.sample_index.npy` in canonical elapsed ticks, with each
+   value equal to `spike_times[subset_spikes[w]]`. Emit producer windows in
+   elapsed ticks, with boundaries that do not bridge unsupported intervals.
+   If the windows overlap, retain their original boundaries and ensure their
+   union accurately represents potential evidence. Add
+   `recording.sample_index_coordinates: "elapsed_ticks"` and
+   `recording.window_bounds_coordinates: "elapsed_ticks"` to the manifest when
+   those sidecars are present. Add `recording.time_base` and authoritative
+   `valid_intervals`; if `params.py` also declares them, keep the declarations
+   identical.
+5. **Keep raw voltage semantics explicit.** Do not put elapsed ticks into raw
+   frame-index fields or claim ordinary raw-trace loading works. An export with
+   a real raw-data path is outside this contract until an explicit validated
+   event-to-frame mapping is supported. Stored waveform data can be exported
+   without a raw path.
+
+## Report evidence separately from verification
+
+The supplied export report describes 420 frame/clock endpoints for 210 pieces,
+with strictly increasing frame endpoints, nondecreasing clock endpoints, and
+15 equal adjacent clock pairs. It also reports backward timestamp steps that
+are not represented by those endpoints. These are reported properties of the
+export metadata, not independent verification of the corresponding acquisition
+events.
+
+The reported `total_dropped_within_pieces_s` was not independently confirmed
+against acquisition truth. Endpoint interpolation does not establish the exact
+valid-time mask, a sample-accurate raw clock, or reversibility: equal clock
+endpoints make the endpoint map non-invertible. The reported 35 spikes inside
+filler likewise require independent confirmation and resolution. A 30 kHz
+label, monotonic spike times, or a plausible endpoint-derived extent cannot
+replace those checks.
+
+Changing Phy's sampling seed (including setting it to `42`) does not make an
+export interoperable. The seed only selects among windows already defined in
+Phy's coordinate system; it cannot recover producer acquisition boundaries,
+map clock coordinates to frames, or determine which spans contain valid data.
+Use compatible producer windows and an explicit coordinate and exposure
+contract.
+
+## Acceptance evidence to return with an export
+
+Provide the exporter commit or reviewable diff, a short provenance note naming
+the authoritative source for boundaries and filler, and machine-checkable
+checks showing:
+
+- every exported spike's elapsed tick and valid-interval membership;
+- the exclusive event-axis bound and separately named physical frame count;
+- waveform-row alignment of sample indices to `spike_times[subset_spikes]`;
+- producer windows' coordinate kind, valid exposure, and source waveform
+  coverage;
+- exact source, retained, and omitted waveform counts under Phy's unchanged
+  512 MiB package and 2,000,000-spike budgets.
+
+If the exporter removes spike rows, it must apply the identical row selection to
+every spike-aligned array and rewrite any row-reference arrays such as
+`subset_spikes` consistently. Silent removal leaves waveform and metadata rows
+misaligned and is not acceptable. Phy can validate the explicit contract and
+the exported arrays; only the exporter and acquisition evidence can establish
+that the chosen exposure is scientifically authoritative. A local older
+recording is not acceptance evidence for a new export.

design/aind-native-export-acceptance.md

@@ -0,0 +1,63 @@
+# Native AIND export acceptance
+
+This checklist accepts an export for Phy software integration only. Passing
+checks do not prove scientific validity. The exporter owner must provide
+authoritative acquisition evidence for the clock origin, physical extent,
+valid exposure, dropped spans, and filler policy. A Phy run cannot supply that
+evidence by inferring or converting timestamps or exposure.
+
+## Before validation
+
+- [ ] Record the exact Phy revision and native exporter revision in the report.
+- [ ] Attach the reviewable exporter change and provenance for acquisition
+  boundaries and filler classification.
+- [ ] Confirm the exporter declares one explicit time base and emits
+  `num_samples`, `valid_intervals`, waveform sample indices, and producer
+  windows in that same coordinate system.
+- [ ] Confirm every sorting spike is inside authoritative valid exposure and
+  every waveform sample index equals `spike_times[subset_spikes]`.
+- [ ] Resolve spikes inside filler from acquisition evidence upstream. Any
+  intentional row removal must use the same row mask for every spike-aligned
+  array and update row references consistently.
+- [ ] Keep elapsed ticks separate from physical frame counts. Do not route
+  elapsed ticks to raw frame-index fields or claim raw-trace loading is
+  supported by this contract.
+
+## Phy engineering validation
+
+1. Run the native export through `tools/validate_aind_export.py` with the two
+   revisions above, without an output directory. Retain its JSON report,
+   including input file hashes, time contract, exposure, array counts, and
+   timings. Resolve every source validation failure at the exporter.
+2. Run optional preparation first with Phy's default budgets: 2,000,000 spikes
+   and 512 MiB. Use a new destination outside the source directory. Record
+   retained spike and waveform counts, omitted evidence, output size, and
+   timings. Confirm the original source file hashes are unchanged.
+3. If the complete producer evidence exceeds either default cap, run larger
+   spike and byte caps as distinct, explicitly labeled runs, each to a fresh
+   destination. Report each cap and its result independently; never treat a
+   larger-cap success as a pass under the defaults.
+4. Open the default bounded output in the Phy GUI. Exercise navigation at the
+   beginning, middle, and end of retained windows, select units with and
+   without retained stored waveforms, and verify displayed waveform/channel
+   mapping and feature behavior.
+5. Exercise the three-round SLAY carryforward workflow on the bounded output:
+   save each round, carry assignments and labels forward, reopen the result,
+   and compare final assignments and metadata with an independent expected
+   result. Preserve the round-by-round artifacts and logs.
+6. Recheck the native source hashes after GUI and carryforward work. All curation
+   products must be separate from the source export.
+
+## Decision record
+
+- [ ] Phy contract validation passes with no timestamp or exposure inference.
+- [ ] Default-budget preparation and output counts are recorded.
+- [ ] Larger-cap runs, if needed, are separately reported.
+- [ ] GUI and three-round SLAY carryforward evidence is attached.
+- [ ] Acquisition authority has approved the clock, exposure, and filler
+  policy independently of the software checks.
+
+Record the outcome as **software integration accepted** only when the Phy
+checks pass. Record **acquisition truth accepted** separately and only when the
+authoritative producer evidence has been reviewed. The first outcome alone is
+not scientific acceptance of the export.

design/aind-real-data-validation.md

@@ -0,0 +1,137 @@
+# Real AIND software diagnostic
+
+## Conclusion
+
+The downloaded export does **not** work unchanged with PR #1428: its time
+coordinates and exposure declarations still need upstream correction. A
+separately labeled diagnostic derivative exercised the complete Phy workflow
+over 469,725,534 rows and exposed one additional Phy defect. That defect is
+fixed, with portable regression coverage. Preparation, GUI curation,
+save/reopen, full replay, independent verification, and repreparation now pass
+on the derivative.
+
+This is software integration evidence, not scientific acceptance of the
+original export. The derivative's exposure is inferred from the JSON endpoint
+map, and 33 filler-region spikes were explicitly quarantined. Those choices
+require acquisition evidence and producer review before use for curation.
+
+## Source and unchanged-export checks
+
+The supplied download script selected computation
+`b1b0782a-b3ea-4f34-81d7-b3c6e36452b3`, rather than the earlier reported
+`85640506-601d-4826-aa61-d3b822e53c84`. Its recording is
+`experiment1_AmplifierData_recording1_group0` from session
+`854393_2026-06-09_19-34-26`.
+
+All 67 downloaded files matched the server's listed sizes, totaling
+15,381,599,263 bytes. NumPy headers and payload sizes were checked. The export
+contains 469,725,567 spikes, 705 declared units, 140,240 stored waveforms and
+177 producer windows. Spike times are monotonic across the complete array.
+
+The unchanged source fails `prepare-large` with the intended early
+`timestamps.json requires an explicit time_base` diagnostic. It declares
+10,800,000,000 pipeline frames, while its last spike is elapsed tick
+12,188,787,421. Exactly 54,808,765 spikes are beyond the declared extent.
+`params.py` and the waveform manifest lack an explicit elapsed-time contract
+and valid acquisition intervals.
+
+All 140,240 waveform centers match the JSON's frame-to-clock conversion within
+0.5 tick, while their stored sample indices remain pipeline frames. This
+reproduces producer interpolation; it does not establish sample-accurate
+acquisition timestamps or exposure. Under that same map, filler regions contain
+23, 4 and 6 spikes. These 33 rows differ from the 35 reported for the other
+computation. Neither explicit large gap contains spikes.
+
+## Diagnostic derivative and limits
+
+The original export was retained unchanged. A separate source was constructed
+solely to isolate software behavior:
+
+- Declare the elapsed-tick contract at 30 kHz, an exclusive extent of
+  12,188,787,510 ticks, and canonical sidecar coordinates.
+- Copy exact `spike_times[waveform_ids]` into the waveform sample-index sidecar.
+  Convert each producer `[a, b)` window to
+  `[round(map(a)), round(map(b - 1)) + 1)` on the elapsed axis.
+- Infer diagnostic exposure by excluding only the mapped explicit large gaps
+  and filler regions. This does not locate unknown dropped frames within
+  pieces or establish acquisition truth.
+- Confirm that coordinate normalization alone still rejects the 33 filler
+  spikes through the strict exposure-membership check.
+- Quarantine those source rows explicitly, apply the identical row selection
+  to every spike-aligned array, and remap waveform row references. No stored
+  waveform references a quarantined row. Keep a provenance file listing every
+  excluded source row and spike tick.
+
+No conversion or filtering behavior was added to the production importer.
+The diagnostic source, outputs, SHA-256 inventory, assumptions and one-off
+reproduction scripts are retained beneath the downloaded computation's
+`diagnostic-validation/` directory on the validation workstation.
+
+## Observed Phy defect and fix
+
+With the actual bounded data, merge, undo/redo, amplitude split and undo/redo
+left NumPy integer child IDs as keys in split-event metadata. Projected session
+metadata had been normalized, but the retained event had not. Saving failed
+while serializing `journal.json` with `TypeError: keys must be str, int, float,
+bool or None, not numpy.int64`.
+
+Both split-event construction paths now normalize metadata before retaining
+the event. This preserves IDs and the remainder/selected child order. Two
+session API regressions reproduced the failure before the fix; a GUI regression
+also exercises merge undo/redo, split, save and reopen. The focused
+curation/lifecycle run passed 43 tests. Independent Sol-high review confirmed
+the root cause and semantics of the fix.
+
+## Full workflow results
+
+Default preparation retained 1,486,361 spikes in 36 windows and 27,865 of
+140,240 waveforms. Every unit has spike evidence; 702 of 705 have stored
+waveform evidence. The package occupies 532,625,401 bytes, within the
+536,870,912-byte cap. The complete producer-window plan is estimated at
+2,554,394,546 bytes and 7,298,318 spikes, exceeding both default budgets.
+
+All default bounded views opened, including waveform-derived features without
+exported PCs. Selection included units 574, 578 and 579, which have no retained
+stored waveforms. Navigation covered the beginning, middle and final retained
+windows. Artificial merge and amplitude-split actions passed undo/redo, an
+independent bounded assignment comparison, save and reopen, without logged
+application errors. These actions are test operations, not biological curation.
+
+Full replay wrote a separate output. An independent streamed oracle compared
+**all 469,725,534 assignments**, using explicit merge masks and amplitude
+comparisons rather than the production replay functions. Edge searches and
+weighted/unweighted histograms independently checked the original per-bin
+counts, finite amplitude counts and amplitude sums. Exposure arithmetic was
+checked against the declared diagnostic intervals, without claiming their
+scientific validity.
+
+Every bounded spike row was checked against its source row. All 27,865 bounded
+waveform rows, channel lists and center indices matched their mapped source
+rows exactly. Replay preserved immutable array bytes and timestamp provenance.
+Repreparation's complete unit counts matched the replay oracle. SHA-256 checks
+confirmed that every file in the original download remained unchanged.
+
+## Measurements and acceptance limits
+
+Single workstation runs used Linux, Python 3.12.3, phylib 2.7.1, 100,000-row
+batches, Xvfb/software OpenGL and Qt's offscreen platform. The workstation has
+125 GiB RAM; source files were warm in the filesystem cache. Some later phases
+ran concurrently with tests or another verification phase. These are observed
+integration costs, not controlled throughput benchmarks or laptop predictions.
+
+| Phase | Wall time | Peak process RSS |
+| --- | ---: | ---: |
+| Preparation | 50.1 s | 14.2 GiB |
+| GUI actions and reopen | 9.1 s | 0.68 GiB |
+| Full replay | 37.0 s | 25.0 GiB |
+| Repreparation | 51.0 s | 14.1 GiB |
+
+Offline RSS includes resident memory-mapped source/output pages. It is not
+limited by the bounded package's byte budget. These measurements do not prove a
+small process-RSS bound on a much longer source.
+
+The original export remains blocked until its producer supplies consistent
+time coordinates, authoritative exposure and a justified filler-spike policy.
+See the [exporter handoff](aind-export-handoff.md). No acquisition logs or
+verified exporter source repository were available, and this diagnostic does
+not resolve timestamp precision within pieces or chronic unit identity.

design/curation-main-gui-result.json

@@ -0,0 +1,76 @@
+{
+  "ok": true,
+  "method": "exact all-row two-way ID mapping",
+  "revision": "c79666c672ab045e356b31f0764efc2cd0e4bdb4",
+  "rows": 469725567,
+  "assignment_dtypes": [
+    "int32",
+    "int64"
+  ],
+  "partition_mismatch_rows": 0,
+  "examples": [],
+  "input_hashes": {
+    "left": {
+      "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c",
+      "spike_clusters.npy": "3cbef5e08dc915ddd57aac9c214370a34282d264695f53959c184d6f8f2b7148"
+    },
+    "right": {
+      "spike_times.npy": "fedfe8732bebf59a7446a14a3979fbacd322707b66a3506b07bbe0b3837b042c",
+      "spike_clusters.npy": "b0223191c2cd29d2167b975175d22428bfe444175e19dc68e67aa9a82aea5168"
+    }
+  },
+  "unmapped_label_ids": {
+    "left": {
+      "group": [],
+      "notes": []
+    },
+    "right": {
+      "group": [],
+      "notes": []
+    }
+  },
+  "manual_label_mismatches": [],
+  "compared_fields": [
+    "group",
+    "notes"
+  ],
+  "scope": "complete curation assignments and requested manual labels; baseline GUI rendering errors retained",
+  "main_revision": "ad64e9138a6556f4785a359c3ee1258864d60f1d",
+  "bounded_runtime_revision": "06637b24aeca9b0984d7928d459187df46406444",
+  "reviewed_pr_head": "c79666c672ab045e356b31f0764efc2cd0e4bdb4",
+  "mapped_units": 703,
+  "gui_rendering_error_free": false,
+  "baseline_feature_errors": {
+    "acceptance": 7,
+    "reopen": 5
+  },
+  "actual_restored_view_count": {
+    "AmplitudeView": 1,
+    "ClusterScatterView": 1,
+    "ClusterView": 1,
+    "CorrelogramView": 1,
+    "FeatureView": 1,
+    "FiringRateView": 1,
+    "ISIView": 1,
+    "MergePropositionsView": 1,
+    "ProbeView": 1,
+    "SimilarityView": 1,
+    "WaveformView": 1
+  },
+  "raw_source_parent_scan": {
+    "units": [
+      40,
+      44,
+      48
+    ],
+    "rows": 1156852,
+    "nonfinite": 0,
+    "equal_to_threshold": 0,
+    "independently_verified": true
+  },
+  "source_integrity": {
+    "ok": true,
+    "raw_files_verified": 67,
+    "immutable_main_working_files": 19
+  }
+}