BF: relax filename regex — non-numeric ses, optional run, extra BIDS entities - #14
BF: relax filename regex — non-numeric ses, optional run, extra BIDS entities#14yarikoptic wants to merge 1 commit into
Conversation
…entities Real OpenNeuro derivatives use filename shapes the previous regex could not match, causing summarize_confounds to hard-exit on the first confound file of many datasets. Observed patterns: - non-numeric session IDs, e.g. `ses-timepoint2`, `ses-test`, `ses-func11`, `ses-T1`, `ses-pre` - missing `_run-XX` entity entirely (single-run acquisitions), e.g. `sub-01_task-rest_desc-confounds_timeseries.tsv` - extra BIDS entities between `task-` and `run-`, e.g. `sub-13_task-MemorySpan_acq-multiband_run-01_...` The new regex uses named groups (`[^_]+` per entity) plus a non-capturing repeat of `_ENT-VAL` pairs between task and run with a negative-lookahead on `run-` so it doesn't swallow the run entity itself. The group extraction switched from `m.groups()` to `m.groupdict()` so column ordering follows `re_groups` instead of regex group order. Verified against 28 datasets from ReproNim/OpenNeuroDerivatives-NIDM that previously failed with `AttributeError: 'NoneType' object has no attribute 'groups'` (before the recent "unsupported filename" hard-exit was added) or the newer `unsupported filename: ...` sys.exit(1). Co-Authored-By: Claude Code 2.1.209 / Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request refactors the regular expression in summarize_confounds to use named groups and handle optional BIDS entities and runs, updating the dictionary merging logic accordingly. The review feedback suggests enhancing the regex to support alphanumeric BIDS entity keys for better robustness, and simplifying the dictionary update by directly using m.groupdict().
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| r'sub-(?P<subject_id>[^_]+)' | ||
| r'(?:_ses-(?P<ses>[^_]+))?' | ||
| r'_task-(?P<task>[^_]+)' | ||
| r'(?:_(?!run-)[a-z]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...) |
There was a problem hiding this comment.
To make the regex more robust against custom or non-standard BIDS entity keys (which may contain uppercase letters or numbers, e.g., Acq, rec2), we should allow alphanumeric characters [a-zA-Z0-9] instead of just lowercase letters [a-z] for the entity keys.
| r'(?:_(?!run-)[a-z]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...) | |
| r'(?:_(?!run-)[a-zA-Z0-9]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...) |
| ) | ||
| sys.exit(1) | ||
| c_sum.update(dict(zip(re_groups, m.groups()))) | ||
| c_sum.update({k: m.groupdict().get(k) for k in re_groups}) |
There was a problem hiding this comment.
Since fname_re defines exactly the named groups corresponding to re_groups, m.groupdict() will return a dictionary containing exactly those keys with their matched values (or None if they didn't match). We can simplify the update of c_sum by directly passing m.groupdict() instead of using a dictionary comprehension.
| c_sum.update({k: m.groupdict().get(k) for k in re_groups}) | |
| c_sum.update(m.groupdict()) |
@yarikoptic disclaimer -- this PR is solely AI driven but does relate also to
and on a brief review looks good to me.
Summary
The current
fname_reinsummarize_confoundsfails on many real BIDS/OpenNeuro derivative filenames, causing the script to hard-exit on the first confound file. Discovered while running the pipeline in ReproNim/OpenNeuroDerivatives-NIDM — 28 of 219 datasets on 2025-05-16 aborted at line 105 withunsupported filename: sub-...(see the tracking issue).Three real-world filename shapes the current
sub-(\w+)(_ses-\d+)?_task-(\w+)_run-(\d+)cannot handle:ses-timepoint2,ses-test,ses-func11,ses-T1,ses-pre(5 datasets)_run-entity — single-run acquisitions likesub-01_task-rest_desc-confounds_timeseries.tsv(17 datasets)task-andrun-— e.g.sub-13_task-MemorySpan_acq-multiband_run-01_...(ds000237)The fix
re_groupsand the regex stay in sync.[^_]+per entity — permissive on ID shape but still anchored to BIDS entity boundaries.(?:_(?!run-)[a-z]+-[^_]+)*middle segment allows any number of BIDS entities to sit betweentask-andrun-, with a negative lookahead onrun-so it doesn't swallow that entity itself.dict(zip(re_groups, m.groups()))to{k: m.groupdict().get(k) for k in re_groups}— column ordering now followsre_groups, not regex group order.Verification
Tested against the 28 filenames observed to fail in the OpenNeuroDerivatives-NIDM May-2025 run (
.ductlogs). All 28 now match and yield the expected(subject_id, ses, task, run)groups — including edge cases likesub-MSC10,sub-rid000041,sub-13_task-...acq-multiband_run-01,sub-01_task-rest(no run),sub-080_ses-T1.Commits
BF: relax filename regex — non-numeric ses, optional run, extra BIDS entitiesTest plan
summarize_confoundsCSV output still validates againstmetadata/summarize_confounds_fmriprep_dictionary.csvon a real dataset (reviewer to spot-check)