Skip to content

BF: relax filename regex — non-numeric ses, optional run, extra BIDS entities - #14

Open
yarikoptic wants to merge 1 commit into
mainfrom
fix-fname-regex-optional-run-and-entities
Open

BF: relax filename regex — non-numeric ses, optional run, extra BIDS entities#14
yarikoptic wants to merge 1 commit into
mainfrom
fix-fname-regex-optional-run-and-entities

Conversation

@yarikoptic

Copy link
Copy Markdown
Member

@yarikoptic disclaimer -- this PR is solely AI driven but does relate also to

and on a brief review looks good to me.

Summary

The current fname_re in summarize_confounds fails on many real BIDS/OpenNeuro derivative filenames, causing the script to hard-exit on the first confound file. Discovered while running the pipeline in ReproNim/OpenNeuroDerivatives-NIDM28 of 219 datasets on 2025-05-16 aborted at line 105 with unsupported filename: sub-... (see the tracking issue).

Three real-world filename shapes the current sub-(\w+)(_ses-\d+)?_task-(\w+)_run-(\d+) cannot handle:

  1. Non-numeric session IDsses-timepoint2, ses-test, ses-func11, ses-T1, ses-pre (5 datasets)
  2. No _run- entity — single-run acquisitions like sub-01_task-rest_desc-confounds_timeseries.tsv (17 datasets)
  3. Extra BIDS entities between task- and run- — e.g. sub-13_task-MemorySpan_acq-multiband_run-01_... (ds000237)

The fix

fname_re = re.compile(
    r'sub-(?P<subject_id>[^_]+)'
    r'(?:_ses-(?P<ses>[^_]+))?'
    r'_task-(?P<task>[^_]+)'
    r'(?:_(?!run-)[a-z]+-[^_]+)*'  # optional entities: acq, rec, dir, echo, space, ...
    r'(?:_run-(?P<run>[^_]+))?'
)
  • Named groups (rather than positional) so re_groups and the regex stay in sync.
  • [^_]+ per entity — permissive on ID shape but still anchored to BIDS entity boundaries.
  • The (?:_(?!run-)[a-z]+-[^_]+)* middle segment allows any number of BIDS entities to sit between task- and run-, with a negative lookahead on run- so it doesn't swallow that entity itself.
  • Extraction switched from dict(zip(re_groups, m.groups())) to {k: m.groupdict().get(k) for k in re_groups} — column ordering now follows re_groups, not regex group order.

Verification

Tested against the 28 filenames observed to fail in the OpenNeuroDerivatives-NIDM May-2025 run (.duct logs). All 28 now match and yield the expected (subject_id, ses, task, run) groups — including edge cases like sub-MSC10, sub-rid000041, sub-13_task-...acq-multiband_run-01, sub-01_task-rest (no run), sub-080_ses-T1.

Commits

  • b0010a1 BF: relax filename regex — non-numeric ses, optional run, extra BIDS entities

Test plan

  • All 28 previously-failing filenames now match with correct named groups
  • Numeric-only filenames (previously supported) still match
  • Downstream summarize_confounds CSV output still validates against metadata/summarize_confounds_fmriprep_dictionary.csv on a real dataset (reviewer to spot-check)

…entities

Real OpenNeuro derivatives use filename shapes the previous regex could
not match, causing summarize_confounds to hard-exit on the first
confound file of many datasets. Observed patterns:

- non-numeric session IDs, e.g. `ses-timepoint2`, `ses-test`,
  `ses-func11`, `ses-T1`, `ses-pre`
- missing `_run-XX` entity entirely (single-run acquisitions),
  e.g. `sub-01_task-rest_desc-confounds_timeseries.tsv`
- extra BIDS entities between `task-` and `run-`,
  e.g. `sub-13_task-MemorySpan_acq-multiband_run-01_...`

The new regex uses named groups (`[^_]+` per entity) plus a
non-capturing repeat of `_ENT-VAL` pairs between task and run
with a negative-lookahead on `run-` so it doesn't swallow the run
entity itself. The group extraction switched from `m.groups()` to
`m.groupdict()` so column ordering follows `re_groups` instead of
regex group order.

Verified against 28 datasets from ReproNim/OpenNeuroDerivatives-NIDM
that previously failed with `AttributeError: 'NoneType' object has no
attribute 'groups'` (before the recent "unsupported filename" hard-exit
was added) or the newer `unsupported filename: ...` sys.exit(1).

Co-Authored-By: Claude Code 2.1.209 / Claude Opus 4.7 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the regular expression in summarize_confounds to use named groups and handle optional BIDS entities and runs, updating the dictionary merging logic accordingly. The review feedback suggests enhancing the regex to support alphanumeric BIDS entity keys for better robustness, and simplifying the dictionary update by directly using m.groupdict().

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread summarize_confounds
r'sub-(?P<subject_id>[^_]+)'
r'(?:_ses-(?P<ses>[^_]+))?'
r'_task-(?P<task>[^_]+)'
r'(?:_(?!run-)[a-z]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make the regex more robust against custom or non-standard BIDS entity keys (which may contain uppercase letters or numbers, e.g., Acq, rec2), we should allow alphanumeric characters [a-zA-Z0-9] instead of just lowercase letters [a-z] for the entity keys.

Suggested change
r'(?:_(?!run-)[a-z]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...)
r'(?:_(?!run-)[a-zA-Z0-9]+-[^_]+)*' # optional BIDS entities between task and run (acq, rec, dir, echo, space, ...)

Comment thread summarize_confounds
)
sys.exit(1)
c_sum.update(dict(zip(re_groups, m.groups())))
c_sum.update({k: m.groupdict().get(k) for k in re_groups})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since fname_re defines exactly the named groups corresponding to re_groups, m.groupdict() will return a dictionary containing exactly those keys with their matched values (or None if they didn't match). We can simplify the update of c_sum by directly passing m.groupdict() instead of using a dictionary comprehension.

Suggested change
c_sum.update({k: m.groupdict().get(k) for k in re_groups})
c_sum.update(m.groupdict())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant