Add HTSLIB/BGZIPTABIX and deprecate redundant modules by itrujnara · Pull Request #11571 · nf-core/modules

itrujnara · 2026-05-08T13:28:58Z

⚠️ Heads up: This PR contains numerous significant changes. Proper review might take time.

Changes

Added HTSLIB/BGZIPTABIX module for bgzip compression, decompression, and indexing
Deprecated SAMTOOLS/BGZIP
Deprecated TABIX/BGZIP
Deprecated TABIX/BGZIPTABIX
Deprecated TABIX/TABIX
Replaced all deprecated modules with HTSLIB/BGZIPTABIX in subworkflows and CI setup blocks

⚠️ Breaking changes

HTSLIB/BGZIPTABIX is not an exact drop-in replacement for any of the deprecated modules; some channel tweaks and/or additional inputs are required
Automatic renaming from TABIX/TABIX was not ported; pipelines must control the naming of its outputs through configs

Rationale

This change was initially proposed by Maxime. bgzip-related operations are currently spread across 4 different modules with largely overlapping functionality. The modules are not consistent in their API nor behavior. This makes it hard for developers to quickly distinguish their purposes and choose the right tools. Additionally, it creates an unnecessary maintenance burden. Finally, none of the tools is placed in a correctly named subdirectory. A single, multi-purpose module will guarantee a predictable and consistent behavior, and facilitate maintenance in the future.

Design and expected behavior

Purpose

The new module can be used for:

compressing files to bgzip
converting other compression types to bgzip (currently supported: gz, bz2, xz)
decompressing bgzip and several other formats
indexing compressed tabular files with tabix

Inputs

The module accepts the following input channels:

meta, infile, index, regions: meta map, arbitrary input file, optional tabix index
action: explicit action input: compress or decompress
make_index: boolean, whether to generate a tabix index
out_ext: output file extension (Mahesh style, as discussed on Slack)

Intended behavior

If action = "compress", the module should:

compress an uncompressed file with bgzip
decompress a non-bgzip compressed file and recompress with bgzip
return a bgzip-compressed file as-is
if requested, index the resulting bgzip with tabix
if regions are passed, performs reindexing with tabix
If a tabix index is passed, it is returned as-is and no new index is generated (for consistency with prior modules).

If action = "decompress", the module should:

decompress a bgzip or other compressed file
return an uncompressed file as-is
not create any index

Any other action is invalid.

Error handling

If action is not compress or decompress, an error is raised
If action is decompress and make_index is true, a warning is emitted

Validation plan

New module

The following behaviors are covered by tests:

The following errors are tested:

Invalid action
Name clash in gzip → bgzip conversion

The following are not tested due to nf-test limitations:

The output of compress is valid bgzip
The output is bit-identical to the input in case of a no-op

Deprecated modules

All tests in deprecated modules have been replaced with one normal and one stub test, with assertions compliant with the deprecation instruction.

Other affected components

All module and subworkflow CI tests that relied on the deprecated modules have been refactored to use the new module. Snapshots have been updated where relevant, verifying that changes only affect file naming. All relevant CI tests pass.

Approval plan

REVIEWER: DO NOT MERGE. Due to the substantial scope of this PR, I want to get at least 3 positive reviews from core/maintainer reviewers before merging. I also want to receive feedback from certain key people (Maxime, Rike, Famke, Luisa).

PR checklist

…ession with tests

…oved compatibility and add new tests for name clash handling

…multiple test files

…ation workflow

- Updated tests across multiple modules to include empty input arrays for compatibility. - Introduced new test cases for compressing VCF files with regions and indexing. - Deprecated the tabix module, replacing its functionality with HTSLIB/BGZIPTABIX. - Adjusted workflow and test configurations to reflect the new module structure. - Updated metadata and tags in various subworkflows to align with the new HTSLIB module.

…metadata consistency

…h index

…port in bgziptabix module

…to tabixupdate

itrujnara · 2026-05-08T13:39:07Z

@nf-core-bot fix linting

…pressed files

…to tabixupdate

SPPearce · 2026-05-08T14:33:04Z

Automatic renaming from TABIX/TABIX was not ported; pipelines must control the naming of its outputs through configs
Can you elaborate on this?

itrujnara · 2026-05-08T14:41:35Z

Automatic renaming from TABIX/TABIX was not ported; pipelines must control the naming of its outputs through configs
Can you elaborate on this?

The latest version of TABIX/TABIX had a piece of code (main.nf line 24) that sometimes made it transplant additional parts of the compressed file name (e.g. if the input file is called test.annotated.vcf.gz, it would call the index ${prefix}.annotated.vcf.gz.tbi). I am not sure if this was intentional. Since this module has a broader functionality, I decided to use a more standardized procedure, where the pipeline determines the prefix through configuration and suffix through module input. This adds a tiny bit of extra work for pipeline maintainers, but should not cause major issues.

…st snapshots

famosab

My few cents, I like it generally. I think I need to take another look at the module logic but my main point is that I am not entirely convinced that having only one module is the way to go here, I would rather have one for the compress and one for the decompress direction but that might be personal preference.

famosab · 2026-05-11T15:34:06Z

+    val action
+    val make_index
+    val out_ext


what is the reason to have them all separated in channels? I feel like they could be in a tuple but thats not a strong opinion. in the pipeline i would control these values via the meta map anyway and set this then in the modules config I think

I feel like the practice with the most support in the community is to have this type of "setting" inputs is to have them as separate val channels. Some (e.g. Simon) would push much of this into the config, but it's not fully accepted as it stands.

famosab · 2026-05-11T15:34:35Z

+    tuple val(meta), path("${outfile}")          , emit: output
+    tuple val(meta), path("${outfile}.{tbi,csi}"), emit: index , optional: true
+    // all htslib tools have the same version, we use bgzip
+    tuple val("${task.process}"), val('htslib'), eval("bgzip --version | sed '1! d; s/bgzip (htslib) //'"), topic: versions, emit: versions_htslib


we should add the xz version here as well

famosab · 2026-05-11T15:35:28Z

I would use

nextflow lint -format -sort-declarations -spaces 4 -harshil-alignment

on this file to clean this up nicely. (mainly setting {}

Done, it broke Harshil alignment though

famosab · 2026-05-11T15:36:15Z

+    - infile_tbi:
+        type: file
+        description: Optional tabix index for the input file. Required when decompressing
+          and creating an index for the output file.


do we really need an index when we decompress? 🤔

My bad, too much tab-driven development, fixed it

famosab · 2026-05-11T15:37:32Z

+  - action:
+      type: string
+      description: Action to perform, either `compress` or `decompress`
+  - make_index:


I would argue that we want this index to always be created automatically and that its just either used then in the pipeline downstream with some kind of .join or ignored. But in most cases I think the index is needed rather than not needed.

I tried to remove this option, but it causes issues in cases where the format is legitimate bgzip input, but not supported by tabix (i.e. any non-tabular file). I know it's a minor use case, but I see no other way to make it a comprehensive bgzip module.

famosab · 2026-05-11T15:38:58Z

+        ontologies:
+          - edam: http://edamontology.org/format_3475 # TSV
+          - edam: http://edamontology.org/format_3003 # BED
+  - action:


I am not 100% sure if I like the approach that one module does both directions, i feel like having one compress and one decompress module would make it a bit easier to reuse but I see the maintenance burden again.

It's a philosophical question, we can discuss it with the others on Slack. It was Maxime's idea to merge everything into one module and I think it makes sense maintenance-wise (without putting that much burden on pipeline developers).

famosab · 2026-05-11T15:39:47Z

    touch ${prefix}.paraphase.bam
    touch ${prefix}.paraphase.bam.bai
-    echo '' | gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz
+    echo "" | gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz


Suggested change

echo "" | gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz

echo | gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz

why is there something added to the prefix here 🤔

No idea, I did not make this module, only changed the quotes to fix a linting failure

You suggestion fails in nf-core modules lint

yeah it should not, i already updated that in the docs und have an open issue with tools but then leave it as it was before

…iption for optional input file index

…to tabixupdate

maxulysse

It's all I ever dreamed it should be

itrujnara added 10 commits May 7, 2026 08:29

feat(htslib_bgziptabix): add module for bgzip compression and decompr…

934acb9

…ession with tests

feat(htslib_bgziptabix): update environment and main process for impr…

e22514b

…oved compatibility and add new tests for name clash handling

feat(tabix_bgzip): deprecate TABIX/BGZIP

10752d9

feat: update references from TABIX_BGZIP to HTSLIB_BGZIPTABIX across …

eab5fd4

…multiple test files

feat(tabix_bgziptabix): deprecate TABIX/BGZIPTABIX

1abe38a

feat: refactor input assignment in test processes for consistency

83b1596

feat: update test snapshots and fix timestamp formatting in VCF annot…

3b15095

…ation workflow

feat: deprecate SAMTOOLS_BGZIP

cb0ef77

feat: update VCF processing workflows to improve output handling and …

d9d8010

…metadata consistency

itrujnara requested a review from maxulysse as a code owner May 8, 2026 13:28

itrujnara changed the title ~~Tabixupdate~~ Add HTSLIB/BGZIPTABIX and deprecate redundant modules May 8, 2026

itrujnara and others added 4 commits May 8, 2026 15:33

Merge branch 'master' into tabixupdate

df9984d

feat: update tabix command logic and add test for VCF compression wit…

64170ab

…h index

feat: update input file definitions and add optional regions file sup…

7c60cfd

…port in bgziptabix module

Merge branch 'tabixupdate' of https://github.com/itrujnara/modules in…

e7dea71

…to tabixupdate

itrujnara added 2 commits May 8, 2026 13:45

feat: update output file definitions to use consistent naming for com…

74279c7

…pressed files

fix: Add missing module information in subworkflow meta.yml

4c4cf85

itrujnara added new module Adding a new module size/xl labels May 8, 2026

itrujnara and others added 4 commits May 8, 2026 16:12

Merge branch 'master' into tabixupdate

7cb1de4

fix: update samtools version to 1.23.1 in environment.yml

03f2444

Merge branch 'tabixupdate' of https://github.com/itrujnara/modules in…

6e29c48

…to tabixupdate

test: Update snapshot for bam_variant_calling_mpileup_bcftools

3ef74c5

fix: update container image and samtools version in PARAPHASE process

7db504a

itrujnara and others added 3 commits May 8, 2026 15:00

fix (multivcfanalyzer): update htslib version to 1.23.1 and adjust te…

cd93d55

…st snapshots

fix(paraphase): standardize quotes in gzip command for consistency

69f07c3

Merge branch 'master' into tabixupdate

115d7b7

famosab reviewed May 11, 2026

View reviewed changes

itrujnara and others added 9 commits May 12, 2026 10:01

fix(htslib_bgziptabix): improve string interpolation and update descr…

1747b37

…iption for optional input file index

Merge branch 'tabixupdate' of https://github.com/itrujnara/modules in…

5bfd88a

…to tabixupdate

fix: fix linting in htslib meta nd paraphase stub

fddf258

Merge branch 'master' into tabixupdate

f0e63d8

test: update snapshot for htslib/bgziptabix

a5a8ec9

Merge branch 'tabixupdate' of https://github.com/itrujnara/modules in…

8762946

…to tabixupdate

test: update snapshot for htslib/bgziptabix

a8f7621

Merge branch 'master' into tabixupdate

dfd06a9

Merge branch 'master' into tabixupdate

d15e7c7

maxulysse approved these changes May 13, 2026

View reviewed changes

	echo "" \| gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz
	echo \| gzip > ${prefix}_paraphase_vcfs/${prefix}_stub.vcf.gz

Conversation

itrujnara commented May 8, 2026

Changes

⚠️ Breaking changes

Rationale

Design and expected behavior

Purpose

Inputs

Intended behavior

Error handling

Validation plan

New module

Deprecated modules

Other affected components

Approval plan

PR checklist

Uh oh!

itrujnara commented May 8, 2026

Uh oh!

SPPearce commented May 8, 2026

Uh oh!

itrujnara commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

famosab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxulysse left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

itrujnara commented May 8, 2026 •

edited

Loading