-
-
Notifications
You must be signed in to change notification settings - Fork 537
ci: Local bin/relock flow with mise #2980
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sl0thentr0py
wants to merge
4
commits into
master
Choose a base branch
from
neel/relock
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # Loaded only when MISE_ENV=devcontainer (set by .devcontainer/run). Pins Ruby to | ||
| # the single version baked into the image, overriding the relock matrix in .mise.toml. | ||
| [tools] | ||
| ruby = "{{ env.RUBY_VERSION | default(value='latest') }}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,267 @@ | ||
| #!/usr/bin/env ruby | ||
| # frozen_string_literal: true | ||
|
|
||
| # Regenerate the per-matrix CI lockfiles that pin our dependencies | ||
| # (supply-chain hardening). Each test-matrix cell gets a committed | ||
| # `<gem>/gemfiles/<cell>.gemfile` wrapper and a `.gemfile.lock`; this script | ||
| # generates both. | ||
| # | ||
| # Each gem's `test-matrix.json` IS the source of truth — we expand cells from | ||
| # those declarative matrices (the same files CI reads to build its job matrix), | ||
| # so the locks can never drift from the matrix CI actually runs. Edit the matrix | ||
| # to add/remove a cell, then run this to materialize the gemfiles and locks. | ||
| # | ||
| # Each cell must resolve against its own Ruby (gemspecs gate on | ||
| # required_ruby_version), so every cell runs under a matching Ruby provided by | ||
| # mise (https://mise.jdx.dev). The required Rubies are declared in .mise.toml; | ||
| # install them once with `mise install`. This script resolves against those | ||
| # already-installed Rubies and aborts if any are missing. | ||
| # | ||
| # bin/relock # refresh every cell | ||
| # bin/relock --gem sentry-ruby # one gem | ||
| # bin/relock --cell sentry-ruby/gemfiles/ruby-3.2_rack-3_redis-5.gemfile | ||
| # bin/relock -l # list every cell's lock path | ||
| # | ||
| # The committed locks are multi-platform (the PLATFORMS section spans | ||
| # linux/darwin/java); `bundle lock --update` preserves that list, so resolving | ||
| # natively on any host keeps the locks valid for CI's x86_64-linux runners. | ||
| # | ||
| # See --help for all options. | ||
|
|
||
| require "optparse" | ||
| require "shellwords" | ||
| require "json" | ||
|
|
||
| ROOT = File.expand_path("..", __dir__) | ||
|
|
||
| # The gem each matrix axis pins -> env var the gem's Gemfile reads. The gem name | ||
| # is also the filename segment (the part before the first "-"); the matrix key | ||
| # is "<gem>_version" (redis is the exception: redis_rb_version), so we recover | ||
| # the gem from a matrix key with key.split("_").first. | ||
| # | ||
| # Values pass through verbatim: the matrix already normalizes them the way | ||
| # GitHub Actions renders the matrix (rack-2 not rack-2.0), and the Gemfiles wrap | ||
| # them in Gem::Version.new(...), so "2" and "2.0" resolve identically — matching CI. | ||
| GEM_ENV_MAPPING = { | ||
| "rack" => "RACK_VERSION", | ||
| "redis" => "REDIS_RB_VERSION", | ||
| "rails" => "RAILS_VERSION", | ||
| "sidekiq" => "SIDEKIQ_VERSION" | ||
| }.freeze | ||
|
|
||
| Cell = Struct.new(:gem, :base, :ruby, :env, keyword_init: true) do | ||
| def wrapper | ||
| "#{gem}/gemfiles/#{base}.gemfile" | ||
| end | ||
|
|
||
| def lock | ||
| "#{wrapper}.lock" | ||
| end | ||
|
|
||
| def label | ||
| "#{gem} / #{base}" | ||
| end | ||
| end | ||
|
|
||
| # Expand one test-matrix.json entry into a cell. The entry's keys (in file | ||
| # order) become the filename segments and the env the Gemfile reads: | ||
| # {"ruby_version":"3.2","rack_version":"2","redis_rb_version":"4"} | ||
| # -> ruby-3.2_rack-2_redis-4, {RACK_VERSION=2, REDIS_RB_VERSION=4} | ||
| # "options" (e.g. rubyopt) is a test-time concern and doesn't affect resolution. | ||
| def cell_from_entry(gem, entry) | ||
| ruby = entry.fetch("ruby_version") | ||
| segments = ["ruby-#{ruby}"] | ||
| env = {} | ||
|
|
||
| entry.each do |key, value| | ||
| next if key == "ruby_version" || key == "options" | ||
|
|
||
| name = key.split("_").first | ||
| var = GEM_ENV_MAPPING[name] | ||
| abort "Unknown matrix key: '#{key}' in #{gem}/test-matrix.json" unless var | ||
| segments << "#{name}-#{value}" | ||
| env[var] = value | ||
| end | ||
|
|
||
| Cell.new(gem: gem, base: segments.join("_"), ruby: ruby, env: env) | ||
| end | ||
|
|
||
| # Parse a wrapper/lock path's base name like "ruby-3.2_rack-3_redis-5" back into | ||
| # a cell (used by --cell, which addresses a single cell by path). | ||
| def parse_cell(gem, base) | ||
| segments = base.split("_") | ||
| ruby = segments.shift.sub(/\Aruby-/, "") | ||
|
|
||
| env = {} | ||
| segments.each do |seg| | ||
| name, value = seg.split("-", 2) | ||
| var = GEM_ENV_MAPPING[name] | ||
| abort "Unknown matrix axis '#{name}' in #{gem}/gemfiles/#{base}" unless var | ||
| env[var] = value | ||
| end | ||
|
|
||
| Cell.new(gem: gem, base: base, ruby: ruby, env: env) | ||
| end | ||
|
|
||
| def matrix_path(gem) | ||
| File.join(ROOT, gem, "test-matrix.json") | ||
| end | ||
|
|
||
| def discover_cells(gems) | ||
| gems.flat_map do |gem| | ||
| path = matrix_path(gem) | ||
| abort "No test-matrix.json for gem '#{gem}'" unless File.exist?(path) | ||
| JSON.parse(File.read(path)).map { |entry| cell_from_entry(gem, entry) }.uniq(&:wrapper) | ||
| end | ||
|
sl0thentr0py marked this conversation as resolved.
|
||
| end | ||
|
|
||
| def all_gems | ||
| Dir.glob(File.join(ROOT, "*", "test-matrix.json")).map { |p| File.basename(File.dirname(p)) }.sort | ||
| end | ||
|
|
||
| # Absolute path to the mise binary. It's usually a shell function (so plain | ||
| # `mise` won't resolve via execvp); ask a login shell where the real binary is. | ||
| def mise_bin | ||
| @mise_bin ||= begin | ||
| found = `sh -lc 'command -v mise' 2>/dev/null`.strip | ||
| found = found.lines.last.to_s.strip if found.include?("\n") | ||
| candidates = [ENV["MISE_BIN"], found, "/opt/homebrew/bin/mise", File.expand_path("~/.local/bin/mise"), "/usr/local/bin/mise"] | ||
| candidates.compact.find { |c| File.executable?(c) } || | ||
| abort("mise not found. Install it: https://mise.jdx.dev") | ||
| end | ||
| end | ||
|
|
||
| # Shell run under the cell's Ruby. Writes the wrapper, re-resolves, and adds | ||
| # checksums where the bundler version supports them. | ||
| RESOLVE = <<~SH | ||
| set -euo pipefail | ||
| mkdir -p "$(dirname "$BUNDLE_GEMFILE")" | ||
| echo 'eval_gemfile "../Gemfile"' > "$BUNDLE_GEMFILE" | ||
| # --add-checksums needs Bundler >= 2.5; fall back to a plain update on older Rubies. | ||
| bundle lock --update --add-checksums || bundle lock --update | ||
|
dingsdax marked this conversation as resolved.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This produces confusing noise: I would actually recommend suppressing this because it's not immediately clear whether it's a problem or not. |
||
| SH | ||
|
|
||
| # Abort (don't auto-install) if any cell's Ruby is missing — the Rubies are | ||
| # declared in .mise.toml and provisioned once via `mise install`. | ||
| def ensure_installed(cells) | ||
| missing = cells.map(&:ruby).uniq.reject do |spec| | ||
| system(mise_bin, "where", "ruby@#{spec}", out: File::NULL, err: File::NULL) | ||
| end | ||
| return if missing.empty? | ||
|
|
||
| warn "Ruby not installed: #{missing.map { |s| "ruby@#{s}" }.join(', ')}." | ||
| abort "Run `mise install` first." | ||
| end | ||
|
|
||
| def cell_env(cell) | ||
| { "BUNDLE_GEMFILE" => File.join(ROOT, cell.wrapper) }.merge(cell.env) | ||
| end | ||
|
|
||
| def run_mise(cell) | ||
| # bash -c (not -lc): inherit the PATH/env mise just set; a login shell would | ||
| # re-source the profile and reset Ruby back to the host default. | ||
| argv = [mise_bin, "exec", "ruby@#{cell.ruby}", "--", "bash", "-c", RESOLVE] | ||
| [cell_env(cell), argv] | ||
| end | ||
|
|
||
| # ---- options ------------------------------------------------------------- | ||
|
|
||
| opts = { | ||
| gems: [], | ||
| cell: nil, | ||
| list: false | ||
| } | ||
|
|
||
| parser = OptionParser.new do |o| | ||
| o.banner = "Usage: bin/relock [options]" | ||
| o.on("--gem NAME", "Only one gem (repeatable). Default: all.") { |v| opts[:gems] << v } | ||
| o.on("--cell PATH", "Resolve exactly one cell by its wrapper/lock path.") { |v| opts[:cell] = v } | ||
| o.on("-l", "--list", "List every cell's lock path, one per line; do nothing.") { opts[:list] = true } | ||
| o.on("-h", "--help") { puts o; exit 0 } | ||
| end | ||
| parser.parse!(ARGV) | ||
|
|
||
| # ---- select cells -------------------------------------------------------- | ||
|
|
||
| if opts[:cell] | ||
| # Single explicit cell. Derive gem + base by position | ||
| # (<gem>/gemfiles/<base>.gemfile) so it works for absolute, relative, or | ||
| # symlinked paths. Accept either the .gemfile or .gemfile.lock form. | ||
| parts = opts[:cell].sub(/\.lock\z/, "").split("/") | ||
| gi = parts.rindex("gemfiles") | ||
| abort "--cell must point at a <gem>/gemfiles/<cell>.gemfile path" unless gi&.positive? | ||
| gem = parts[gi - 1] | ||
| base = File.basename(parts[gi + 1], ".gemfile") | ||
| cells = [parse_cell(gem, base)] | ||
|
sl0thentr0py marked this conversation as resolved.
|
||
| else | ||
| gems = opts[:gems].empty? ? all_gems : opts[:gems] | ||
| cells = discover_cells(gems) | ||
| end | ||
|
|
||
| if cells.empty? | ||
| warn "No matching lockfile cells found." | ||
| exit 1 | ||
| end | ||
|
|
||
| # ---- list ---------------------------------------------------------------- | ||
|
|
||
| # Bare lock paths, one per line — scriptable (e.g. piped to xargs). | ||
| if opts[:list] | ||
| cells.each { |c| puts c.lock } | ||
| exit 0 | ||
| end | ||
|
|
||
| # Resolve against already-installed Rubies (declared in .mise.toml) — never | ||
| # auto-installs. | ||
| ensure_installed(cells) | ||
|
|
||
| # ---- execute ------------------------------------------------------------- | ||
|
|
||
| # ANSI styling, but only when stdout is an interactive terminal (pipes/CI get | ||
| # plain text). $stdout.tty? guards every escape so logs stay grep-friendly. | ||
| TTY = $stdout.tty? | ||
| def color(text, code) | ||
| TTY ? "\e[#{code}m#{text}\e[0m" : text | ||
| end | ||
|
|
||
| def fmt_duration(seconds) | ||
| seconds < 60 ? format("%.1fs", seconds) : format("%dm%02ds", (seconds / 60).to_i, (seconds % 60).to_i) | ||
| end | ||
|
|
||
| # Serial: cells sharing a Ruby also share that Ruby's gem home, and git-sourced | ||
| # gems (e.g. debug) collide on .git locks when fetched concurrently. | ||
| failures = [] | ||
| width = cells.size.to_s.length | ||
| started = Time.now | ||
|
|
||
| cells.each_with_index do |cell, i| | ||
| counter = color("[#{(i + 1).to_s.rjust(width)}/#{cells.size}]", "1;34") | ||
| puts "#{counter} #{color('→', '36')} #{color(cell.label, '1')} #{color("(ruby #{cell.ruby})", '90')}" | ||
|
|
||
| env, argv = run_mise(cell) | ||
| cell_started = Time.now | ||
| # chdir into the gem dir to match CI's working-directory (avoids stray | ||
| # files and keeps any .bundle/ config local to the gem). | ||
| ok = system(env, *argv, chdir: File.join(ROOT, cell.gem)) | ||
| elapsed = fmt_duration(Time.now - cell_started) | ||
|
|
||
| if ok | ||
| puts "#{' ' * (width * 2 + 3)} #{color('✓', '32')} done #{color("in #{elapsed}", '90')}" | ||
| else | ||
| failures << cell | ||
| puts "#{' ' * (width * 2 + 3)} #{color('✗', '31')} #{color("failed after #{elapsed}", '31')}" | ||
| end | ||
| end | ||
|
|
||
| total_elapsed = fmt_duration(Time.now - started) | ||
| ok_count = cells.size - failures.size | ||
|
|
||
| puts | ||
| if failures.empty? | ||
| puts color("✓ Regenerated #{ok_count}/#{cells.size} cell(s) in #{total_elapsed}.", "1;32") | ||
| else | ||
| puts color("✗ Regenerated #{ok_count}/#{cells.size} cell(s) in #{total_elapsed}.", "1;31") | ||
| warn color("Failed:", "1;31") | ||
| failures.each { |c| warn color(" #{c.lock}", "31") } | ||
| exit 1 | ||
| end | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't think this is used.