Skip to content

Rewrite codex() and add phenocycler() for Akoya CODEX / PhenoCycler-Fusion data - #429

Draft
timtreis wants to merge 107 commits into
mainfrom
feat/akoya-readers
Draft

timtreis wants to merge 107 commits into
mainfrom
feat/akoya-readers

Conversation

@timtreis

@timtreis timtreis commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Rewrite codex() and add phenocycler() for Akoya data

codex() only read a 2019 CODEX Processor layout. It read files relative to the cwd, read only the first region, lost the marker names, and could not read any QPTIFF. Current Akoya instruments (PhenoCycler-Fusion) write only .qptiff.

Changes

  • New phenocycler() reads PhenoCycler-Fusion 1.x/2.x .qptiff / .er.qptiff scans, including Opal mode. It also reads other PerkinElmer/Akoya QPTIFFs: Vectra Polaris, PhenoImager HT (incl. unmixed uint16), brightfield RGB and inForm component data. Optional .xpd / *channels.csv sidecars go into the metadata, and masks are read via labels=.
  • codex() is rewritten for CODEX Processor per-marker TIFFs, Uploader/Processor hyperstacks, Instrument Manager .qptiff scans and the Processor's FCS/CSV cell tables. It also reads masks via labels= or <region>_mask.tif.
  • Both readers share one QPTIFF core:
    • Pages are classified by the XML <ImageType>.
    • Level 0 is read lazily through tifffile.aszarr and the pyramid is recomputed.
    • Channel names are de-duplicated.
    • Metadata goes into sdata.attrs["akoya"], both structured and raw.
    • The lazy arrays can be pickled.
  • Each region gets its own coordinate system unless the regions share a slide, in which case they are placed at their stage positions in "global".
  • The CLI extends codex and adds a phenocycler command.

Breaking (codex())

  • Element names change to {region}_image, {region}_cells, {region}_labels and table; they were images and {region}.
  • All regions are read; regions=[...] reads a subset.
  • var_names are marker names, not cycCCC_chHHH.
  • New parameters are keyword-only, and imread_kwargs is ignored with a warning.

Validation

  • Synthetic (CI): a QPTIFF, FCS and Processor-tree fixture writer feeds about 110 tests.
  • Real public data (tests/test_akoya_real_data.py): it runs with AKOYA_TEST_DATA set and is skipped otherwise. All 10 tests pass on an HPC node. Full-resolution tiles equal tifffile's page decoding, and the same QPTIFF read through codex() and phenocycler() gives the same result.
Data Generation
Figshare+ 25123538 AML2_183 Fusion 1.x
S-BIAD3467 Tonsil-ctrl Fusion 2.x, Opal mode
Zenodo 22107401 Arlene_Tonsil_9x9, Panel33C (unmixed) CIM 1.30; PhenoImager HT
S-BIAD859 reg001_montage.tif CODEX Uploader hyperstack
Zenodo 7742474 per-marker TIFFs CODEX Processor 1.7
OME Vectra samples H&E, inForm component data Vectra Polaris, inForm
phenomenalist tonsil CSV CODEX Processor table

A one-off check of 23 public datasets found no reader issue. It covered element validation, extents, round trips through sdata.write/read_zarr, spatialdata-plot renders and cell alignment.

Not covered

  • HuBMAP layouts (Globus is blocked on our cluster).
  • No real CODEX .fcs, .er.qptiff or .xpd files were available; those paths are tested only on synthetic data.

timtreis and others added 30 commits October 17, 2026 01:07
…ters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse PerkinElmer/Akoya QPTIFF page XML (channels by <ImageType>,
Biomarker > Name), calibrated pixel size from XResolution, stage
position, generation detection, and Fusion JSON / CIM ExperimentV4
scan-profile enrichment. Expose level 0 lazily via aszarr. Add
PhenocyclerKeys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Join profile records to pages positionally when the non-blank counts
match and no pair contradicts the names, else by normalised name only
for names with equal page and record counts. Skip blank and markerless
profile records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and CIM outputs

codex() now discovers every region of a CODEX Processor/Uploader tree
(per-marker stitched tifs, hyperstacks named via ImageJ Labels or
channelNames.txt) and CODEX Instrument Manager qptiffs, reads them lazily
and stores one `<region>_image` per region with metadata in
sdata.attrs["akoya"]. Paths are resolved against the input instead of the
cwd. Cell tables are discovered but not read yet, so codex() returns
images only for now.

BREAKING CHANGE: the image element is named `<region>_image` instead of
`images`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Memory-map uncompressed contiguous tiffs per chunk instead of reading one
zarr chunk per plane, accept a single region-less .tif, keep only
region-less CSVs that look like cell tables (error when ambiguous) and
warn when imread_kwargs is not used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…all regions

Marker names come from the FCS $PnS keywords or the Processor CSV header
'cycCCC_chHHH:<marker>'; tables of all regions are read relative to the
given path and annotate one circles element per region.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Region-less tables that cannot be read or assigned to a region are skipped
with a warning instead of aborting the read; next to a single stem-named
image they annotate it. Overlapping regions raise early, blank $PnS fall
back to $PnN, and attrs only record tables that contributed cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in Akoya readers

`cells_table`/`labels` now work in `codex()` and `phenocycler()`: a path or a
{region: path} mapping (CLI: JSON). Masks become lazy `<region>_labels` with the
region image's transformation and the table annotates them; without a mask cells
stay circles. A user `cells_table` replaces auto-discovered CODEX tables. A single
table for several regions needs a region column (codex `region`, QuPath `Image`,
or `cells_table_kwargs["region_column"]`). Different panels give per-region tables.
SPRM/inForm are detected but raise NotImplementedError until a later release.

The lazy tiff reader moves from codex.py to _akoya.py (now `aszarr(level=0)`, so
pyramidal OME masks work) and is shared by images and masks. A `str` passed as
`regions`/`channel_names` is wrapped into a list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…koya cell tables

Source columns that clash case-insensitively with `region`/`cell_id` are kept as
`source_<name>` instead of crashing table validation. Masks whose shape differs
from the region image, and masks paired with row-number instance ids, warn.
Mapping keys of existing but unselected regions are ignored. QuPath tables
without marker columns warn; malformed CLI mappings are a BadParameter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QuPath GeoJSON cell outlines (cells_table_kwargs['geojson'] or a sibling
<stem>.geojson / <stem>_cells.geojson) become <region>_labels annotated by
the table. Features match the table by Object ID (UUID -> row instance id),
else in row order. Chunk-wise map_blocks over an STRtree; higher id wins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Order or id matches are now checked: >5% of polygons more than 1 px
from their row's centroid raise (wrong order, file or geojson_units).
Duplicate feature ids raise; cells without a polygon warn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… coordinate systems

QuPath cell outlines are stored as {region}_polygons besides the rasterised
labels. Several scans of the same sample space are placed by stage position
in "global"; unrelated regions (and regions without an image) each get a
coordinate system named after the region; a single region stays in "global".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add CHANGELOG entries for phenocycler() and the breaking codex() rewrite,
mention PhenoCycler-Fusion in the README, complete the reader docstrings
(coordinate systems of image-less regions, kept polygons, source_<name>
columns, sanitised obs names, sprm/inform formats), name the kept polygons
in the mask-wins warning, and add tests on real public datasets that run
only when AKOYA_TEST_DATA is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SPACEc and the HuBMAP codex-pipeline write `channelnames.txt`; an exact
name match silently dropped the channel names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Columns named exactly like the region or instance key were dropped even when
they were not read as the regions or instance ids, contradicting the reader
docs. Only the column an adapter reads as the key is now dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`imread_kwargs` used to be the third positional parameter of `codex()`;
passing it positionally now raises a TypeError pointing to the keyword
instead of being treated as a region list. An empty `regions` raises a
ValueError and duplicate regions are read once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UTF-16 .xpd, wells or items that are not JSON objects, or a non-UTF-8
channels.csv aborted the whole read. Sidecars are metadata only: they are
now ignored with a UserWarning, byte order marks are accepted and an .xpd
without wells no longer warns about 0 markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dask arrays held tifffile's zarr store and its open file handle, so
they could not be pickled (dask.distributed, process schedulers) and left
unclosed files behind (ResourceWarning). A small array-like now keeps only
the path and the series or page, opens the file on the first chunk read in
each process and closes it with the object; native tile chunks are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`reg001_antibodies.csv` or a HuBMAP SPRM `reg001_...-cell_channel_mean.csv`
was discovered as the region's cell table and aborted the read. Discovery
now keeps only CSVs detected as CODEX Processor tables, as it already did
for CSVs without a region in their name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A QuPath (or inForm, generic) table for several scans with pixel sizes
that differ in the 4th decimal could not be read: it required one pixel
size for all regions. Each cell's centroid, area and GeoJSON outline are
now converted with the pixel size of the region its region column names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…enocycler

Discovery matched the suffix case-sensitively, and the acquisition-folder
check looked at the absolute path, so a dataset stored below a
`raw/images/phenocycler/` folder was ignored entirely. Only the folders
below the searched directory are now checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With pages [DAPI, CD3, CD4] and records [DAPI, CD3, DAPI], CD4 got the
cycle of the second DAPI. Positional pairing is now skipped when a record
names the marker of another page; the name join leaves such channels unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Duplicate instance ids within one region surfaced as TableModel's raw
'Instance key ... not unique' error. The adapters now raise a ValueError
naming the table, the region and the duplicated ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`codex()` and `phenocycler()` duplicated the images -> labels -> polygons
-> cells -> SpatialData -> attrs assembly; it is now one helper. Both
readers also name a qptiff without `regNNN` the same way (suffix and
trailing `_Scan<N>` stripped, `.er.qptiff` preferred). The discovery
helpers of the reader modules are private and no longer exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex(path, fcs, imread_kwargs) is restored; all newer parameters are
keyword-only, as is everything after path in phenocycler().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eads images and labels

Drop the QuPath, MCMICRO mcquant, HuBMAP SPRM, inForm and generic table
adapters with their format detection, cells_table_format and
cells_table_kwargs (also in the CLI), the µm/mask-pixel coordinate
conversions and mask channel selection. phenocycler() loses cells_table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants