[miguel-heygen]

What

The container image for an arm64 Lambda video function, and CI that builds it and runs it on a native ARM runner. Stacked on #6062, which adds the native ffmpeg build.

Nothing deploys this image yet: the release artifact and the lambda deploy option come next. The default x86 ZIP deploy is untouched.

Why

Lambda arm64 needs native binaries for ffmpeg (#6062) and Chrome. Everything else in the handler ZIP is architecture-neutral JavaScript. Building the image from that same ZIP means the x86 and arm64 functions run identical handler code.

How

• scripts/stage-arm64-image.ts unzips the handler ZIP into a staging /var/task. It replaces bin/ffmpeg and bin/ffprobe with the native tree from build(aws-lambda): pinned native ffmpeg build for an arm64 Lambda image #6062's artifact (binaries, their two bundled libraries, licenses), then asserts both are 64-bit arm64 ELF.
  • The npm @sparticuz/chromium package ships x64 Chrome only. The script downloads Sparticuz's arm64 pack for the exact installed version, checks it against a sha256 pinned per version, and unpacks it over the package's bin/. src/chromium.ts calls executablePath() unchanged.
  • Bumping @sparticuz/chromium fails the staging until its arm64 pack is pinned. A handler ZIP built with chrome-headless-shell is refused.
• image/Dockerfile gains a video stage that only copies the staged directory onto the same pinned Lambda Node 22 base the ffmpeg was built on. One ARG owns that pin, and any host can build the stage without emulation.
• The workflow (renamed lambda-arm64-image.yml) keeps build(aws-lambda): pinned native ffmpeg build for an arm64 Lambda image #6062's two ARM ffmpeg builds and byte-identical comparison. It adds two jobs:
  • one builds the handler ZIP the way the release does;
  • one stages and builds the image on an ARM runner, then:
    • runs the bundled ffmpeg and a real Chrome render inside it, launched with the same executablePath() and args as src/chromium.ts;
    • boots it under the Lambda runtime interface emulator and invokes the real handler, expecting its own "no recognised Action" error.

Test plan

• bun test scripts/stage-arm64-image.test.ts: 5 pass, 0 fail, exit 0, three runs in a row
• One mutation per behaviour, each failing only its own test: skip the arm64 ELF check, skip the pack sha256 check, skip the version-pin check, allow chrome-headless-shell, never swap Chrome
• This PR's Lambda arm64 image workflow (ARM ffmpeg x2 + match, handler ZIP, image smoke on ARM)
• Comments follow CONTRIBUTING.md "Comments"

Not exercised here: the image running in Lambda itself. That is proven with the deploy PR.

[vanceingalls]

Reviewed at dcdeefc2, only the one-commit delta on #6062's head f43fc696. The staging design is sound. The blocker is that the PR's own image smoke fails, and the reason is in the smoke setup, not the image.

What holds

• The Chrome pin is right. I downloaded chromium-v148.0.0-pack.arm64.tar from the Sparticuz release. Its sha256 is c18a5241…5f0e, the value pinned in CHROMIUM_ARM64_PACK_SHA256. It holds the same four files the npm package ships in bin/ (chromium.br, al2023.tar.br, fonts.tar.br, swiftshader.tar.br), so untarring over bin/ replaces every x64 file and leaves none behind.
• The ffmpeg swap lands where the rpath expects. #6062's tarball is hf-media/{bin,lib,share}. --strip-components=1 gives /var/task/bin + /var/task/lib, and $ORIGIN/../lib resolves. The ZIP stages both bin/ffmpeg and bin/ffprobe (build-zip.ts:434), so the non-forcing rmSyncs can't hit a missing file.
• Locally: bun test scripts/stage-arm64-image.test.ts passes 5/5 on Bun 1.4.4, run twice. My box has no zip, so I used a shim; the ubuntu runners have it, and the Windows suite skips aws-lambda (windows-render.yml:420).
• In CI, Handler ZIP, both ffmpeg builds and ffmpeg builds match are green, and the bundled ffmpeg prints ffmpeg version 7.0.2 inside the image.

Blocker: Chrome can't launch in the image smoke, because the smoke never inflates the AL2023 libraries
Image smoke fails at "ffmpeg and Chrome run in the image" (job 119910604065):

/tmp/chromium: error while loading shared libraries: libnspr4.so: cannot open shared object file

• Why: @sparticuz/chromium 148 inflates al2023.tar.br and puts /tmp/al2023/lib on LD_LIBRARY_PATH only when isRunningInAmazonLinux2023() sees AWS_EXECUTION_ENV (or AWS_LAMBDA_JS_RUNTIME / CODEBUILD_BUILD_IMAGE) naming node 20/22/24. Real Lambda sets that variable.
• The gap: the pinned base image's config doesn't set it. Its Env is only LANG, TZ, PATH, LD_LIBRARY_PATH, LAMBDA_TASK_ROOT and LAMBDA_RUNTIME_DIR. And docker run --entrypoint /var/lang/bin/node (lambda-arm64-image.yml:88) skips the runtime, so the smoke never inflates the libraries.
• What's missing: that pack holds libnspr4, libnss3, libexpat and the rest. None of them are in the base layers; I listed all six arm64 layers.
• Effect: the step doesn't yet prove "Chrome renders, launched the way src/chromium.ts launches it". The al2023 half of the arm64 pack is never exercised.
• Fix: add -e AWS_EXECUTION_ENV=AWS_Lambda_nodejs22.x to that docker run, so the smoke takes the same branch as Lambda.

Important (non-blocking)

• No workflow runs on a @sparticuz/chromium bump. The workflow's paths (:10-13) cover image/**, the stage script and the workflow. A version bump only touches packages/aws-lambda/package.json + bun.lock, so the "fails the staging until its arm64 pack is pinned" guard doesn't run on that PR. It first fails at the next image build, after the bump has merged.
• Same gap for ZIP layout changes. A build-zip.ts change to where ffmpeg, ffprobe or Chrome land isn't covered either.
• Fix: add packages/aws-lambda/package.json and packages/aws-lambda/scripts/build-zip.ts to paths. This matters once the deploy PR ships the image.

Nit

• /var/task/lib is on the base LD_LIBRARY_PATH, so the bundled gnutls/nettle libraries are visible to every process in the function, not just ffmpeg. They come from the same AL2023 release as the base, and /lib64 resolves first, so I see no conflict today. The $ORIGIN rpath means they could live under bin/../lib anywhere, if you'd rather keep them off the shared path.

CI: Lambda arm64 image is red at dcdeefc2 (Image smoke, above). As a stacked PR (base build/lambda-arm64-ffmpeg), ci.yml's required checks don't run here. Comments and regression are green.

Verdict: REQUEST CHANGES
Reasoning: The image smoke is this PR's main evidence, and it fails. The cause is a missing AWS_EXECUTION_ENV, so Chrome's AL2023 libraries never load. It's a one-line fix. The staging script, the pins and the ffmpeg swap all check out.

— Via

[miguel-heygen]

Thanks, both fixed at 04eef55 (rebased onto #6062's a51fd1d).

• Blocker: the smoke's docker run now sets AWS_EXECUTION_ENV=AWS_Lambda_nodejs22.x, so @sparticuz/chromium inflates al2023.tar.br and Chrome launches on the same branch it takes in Lambda. This run of Lambda arm64 image is the check.
• Paths: the workflow now also runs on packages/aws-lambda/package.json and packages/aws-lambda/scripts/build-zip.ts, so a @sparticuz/chromium bump hits the pin guard on its own PR.
• Nit, /var/task/lib on LD_LIBRARY_PATH: keeping it as is. The two libraries are GnuTLS and nettle from the same Amazon Linux 2023 release as the base, the base has neither, and /lib64 resolves first. Moving them off the shared path would need a separate directory plus symlinks in bin/ for no current conflict. If that ever changes, the $ORIGIN rpath means only the staging script has to move.

[miguel-heygen]

Follow-up at 003729a: the image is no longer smoked with an env the image doesn't carry.

• Where Lambda's value comes from. The pinned base image's own bootstrap sets it. /var/runtime/bootstrap line 63 is export AWS_EXECUTION_ENV=AWS_Lambda_nodejs22.x, unconditionally, and /lambda-entrypoint.sh execs that bootstrap under both real Lambda and the runtime interface emulator. The image config's Env doesn't carry it (LANG, TZ, PATH, LD_LIBRARY_PATH, LAMBDA_TASK_ROOT, LAMBDA_RUNTIME_DIR), which is why a bare docker run --entrypoint node missed it.
• What sparticuz needs. @sparticuz/chromium 148 accepts any AWS_EXECUTION_ENV containing 22.x (build/esm/index.mjs:47), so AWS_Lambda_nodejs22.x takes the AL2023 branch.
• Checked: a probe handler mounted into the base image and invoked through its entrypoint under the emulator returned {"executionEnv":"AWS_Lambda_nodejs22.x"}. That run was x86 on a devbox, but the bootstrap line is the same file in the arm64 image.
• The smoke now takes that same path. smoke-video.mjs is a handler, mounted into the image, started with the image's normal entrypoint (hf-video:ci smoke-video.handler) and invoked over the emulator's invoke API. No env is set by hand. It returns the AWS_EXECUTION_ENV it saw along with the arch, ffmpeg version, Chrome version and screenshot size, and the step asserts arm64, ffmpeg 7.0.2 and a non-empty screenshot.
• Both emulator steps now retry with --retry-all-errors, because the emulator can reset the first connection while it starts.

The live render on the dev account is the final proof that Chrome launches in a real Lambda invoke.

[vanceingalls]

Re-reviewed at 003729ae. It sits on #6062's new head a51fd1db, and #6064's own first commit is unchanged by the restack (git range-diff: dcdeefc2 = 661c98b8). Two commits are new. This resolves my change request (5616554077).

What's fixed

• The Chrome smoke now goes through the image's own Lambda bootstrap. smoke-video.mjs is now a handler, and the workflow invokes it as smoke-video.handler via the runtime interface emulator. No env vars are set by hand.
• The smoke passes in CI (Image smoke). The handler reports "executionEnv":"AWS_Lambda_nodejs22.x","arch":"arm64","ffmpeg":"ffmpeg version 7.0.2 …","chrome":"HeadlessChrome/148.0.7778.0","screenshotBytes":182. So the bootstrap sets AWS_EXECUTION_ENV as Lambda does, @sparticuz/chromium inflates the arm64 al2023 libraries, and Chrome renders.
• The jq -e gate checks arch, a non-empty screenshot and the ffmpeg version, so a smoke that silently falls back can't pass. The handler-load step still answers no recognised Action.
• The workflow paths now include packages/aws-lambda/package.json and scripts/build-zip.ts. That closes the important note from my last review: a @sparticuz/chromium bump now re-stages the image on its own PR.

CI: Lambda arm64 image is green at 003729ae: both ffmpeg builds, ffmpeg builds match, Handler ZIP and Image smoke. Comments and regression are green too. As a stacked PR (base build/lambda-arm64-ffmpeg), ci.yml's required checks don't run here. #6062's new 10-bit x265 commit is the base, so I didn't review it here.

The earlier nit (bundled gnutls/nettle on the shared /var/task/lib path) still stands as a taste call, not a blocker.

Verdict: APPROVE
Reasoning: The image smoke now launches Chrome through Lambda's own bootstrap and renders on arm64 with the pinned pack. That was the evidence missing at dcdeefc2, and the trigger paths now cover a Chromium bump.

— Via

The base branch was changed.

[heygengenesis]

Verdict: APPROVE · 0 blocker · 0 important · 0 nits

Re-reviewed at 0a6af61f. The only change since my approval at c4889c0c is one commit to scripts/stage-arm64-image.ts (+12/−10).

• The URL comes from the pin table. It is built from the matched pin key, which equals the installed version, so the request goes to the same URL as before. A version with no pin still throws before any fetch.
• Only checksum-matched bytes reach tar. The pack is hashed first and then piped to tar -xf - over stdin, so no temp file is written. A mismatch throws before run. input with a piped stdin is correct for spawnSync, and maxBuffer limits only stdout and stderr.
• Both CodeQL threads are resolved, and CodeQL is green at this head. bun test scripts/stage-arm64-image.test.ts passes 5/5 locally (Bun 1.4.4). The "swaps Chrome" test exercises the stdin extraction path.

CI

Lambda arm64 image, CI, CodeQL, regression, preview-regression, PR captures, Player perf and Comments are green at 0a6af61f. Windows render verification was still running when I posted, and it skips aws-lambda. The earlier /var/task/lib nit is still a taste call.

— Via