Skip to content

ci: shard the Iceberg extensions test task - #6113

Open
ErikBPF wants to merge 1 commit into
apache:mainfrom
ErikBPF:ci/shard-iceberg-extensions
Open

ErikBPF wants to merge 1 commit into
apache:mainfrom
ErikBPF:ci/shard-iceberg-extensions

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Closes #6103.

Rationale

The iceberg-spark-extensions job ran unsharded and was the longest job in the
Iceberg workflow — 55m24s in a recent merge-queue run, against four shards for the
core iceberg-spark task. It now uses the same shard machinery as the core task.

What changes

  • iceberg-spark-extensions takes the shared shard matrix
    (fromJSON(needs.build-native.outputs.shard-matrix), fail-fast: false), gains
    the /shard-${{ matrix.shard }} name suffix, and passes the same Gradle wiring as
    the core task: --init-script ../dev/ci/iceberg-test-shards.gradle,
    -PcometShardTask=:iceberg-spark:iceberg-spark-extensions-...:test,
    -PcometShardIndex/-PcometShardCount.
  • Each shard uploads its candidate inventory and test reports.
  • A new iceberg-spark-extensions-shard-coverage job verifies the extensions
    task's inventories (check-iceberg-shards.py --manifests ... --task ...extensions...:test),
    mirroring the core coverage job.
  • dev/ci/check-ci-config.py gains check_iceberg_extensions_shards() so the job
    cannot silently lose its shard wiring.

Expected effect

~55m24s → projected ~15-20 min for the extensions job, roughly 35 queue-minutes
per run. This is the largest single saving identified in the CI review.

How it was tested

  • Guard is RED before the workflow change (four failures) and GREEN after:
    python3 dev/ci/check-ci-config.pyCI config checks passed.
  • python3 dev/ci/test-iceberg-shards.py → 15 tests OK.
  • Not yet exercised end-to-end: per-shard balance and a non-empty extensions
    unshardedCandidates inventory need one label run to confirm. The new coverage
    job fails if the union of shard candidates is not exactly the unsharded
    inventory, so a bad split cannot merge silently.

Constraint

SHARD_COUNT = 4 in dev/ci/check-iceberg-shards.py is global; the extensions
task reuses the same count.

iceberg-spark-extensions ran unsharded and was the longest job in a queue run (55m24s). Shard it exactly like the core iceberg-spark task: the shared shard matrix, --init-script, -PcometShardTask for its own task, and per-shard inventory uploads verified by a new iceberg-spark-extensions-shard-coverage job.

dev/ci/check-ci-config.py gains check_iceberg_extensions_shards() so the job cannot silently lose its shard wiring.
@github-actions github-actions Bot added build Build environment enhancement New feature or request area:ci CI/CD, GitHub Actions, build tooling area:Iceberg labels Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI/CD, GitHub Actions, build tooling area:Iceberg build Build environment enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: shard the Iceberg extensions test task

1 participant