Skip to content

Consider a pre-baked pgxn-tools-based container with common build deps pre-installed #96

Description

@jnasbyupgrade

Background

While investigating CI speed in Postgres-Extensions/cat_tools, pulled real job/step-level timing from several recent ci.yml runs (gh api .../actions/runs/<id>/jobs, per-step timestamps, not estimates). Findings specific to that repo aren't relevant here, but one pattern is generic to any pgxntool-based repo using the pgxn/pgxn-tools container across a PG-version matrix:

  • "Initialize containers" (pulling + booting pgxn/pgxn-tools) costs roughly 20-30s and recurs on essentially every container-based CI job -- in cat_tools's matrix that's ~30 jobs per run.
  • Several jobs additionally apt-get install build dependencies (e.g. flex bison libkrb5-dev for building pg_tle from source) on top of that.
  • None of this sits on the critical path in cat_tools's case (its matrix runs in parallel, and the true bottleneck there is a single long sequential job, unrelated to this) -- so this is a COST optimization (aggregate billable compute-minutes), not a latency one. Rough estimate for cat_tools alone: perhaps 3-7 minutes of aggregate compute-minutes per CI run.

Proposal

A pre-baked container image, built on top of (or alongside) pgxn/pgxn-tools, with common build-time dependencies already installed, so individual jobs across pgxntool-based repos skip re-installing them every single run. Candidates to pre-install, based on what's currently installed ad hoc in cat_tools's CI: rsync, postgresql-server-dev-* for the supported PG major range, and pg_tle's own build dependencies (flex, bison, libkrb5-dev) -- likely more once other pgxntool-based repos' CI is surveyed the same way.

This is deliberately scoped at the pgxntool level rather than per-repo: the value is in every repo built on pgxntool benefiting from one shared image, not in cat_tools maintaining its own.

Not in scope for this issue

  • Consolidating a PG-version test matrix into one container instead of one-container-per-version -- investigated separately for cat_tools and found to be a wash-to-negative on wall-clock (parallel matrix already hides the per-job container-boot cost from latency; consolidating would trade a modest compute-cost saving for a slower critical path). That's a per-repo CI-structure decision, not a pgxntool-level one, and isn't what this issue is proposing.
  • Actually building/maintaining the image -- this issue is to record the idea and the rough cost justification; scoping the image contents and where it'd be hosted/published is follow-up work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions