[miguel-heygen]

What

A pinned, reproducible build of ffmpeg + ffprobe for an arm64 Lambda video function, under packages/aws-lambda/image/, plus a CI workflow that builds it natively on GitHub's ARM runners.

Nothing ships it yet: the image and the lambda deploy option that use it come in follow-up PRs. The default x86 ZIP deploy is untouched.

Why

The x86 Lambda handler uses the ffmpeg-static binary. Its arm64 sibling is a static-glibc build that crashes on Lambda during decode, and the stock dynamic arm64 builds are too large for a ZIP function. An arm64 video function therefore ships as a container image, and that image needs an ffmpeg built against the glibc of the Lambda Node 22 base it runs on, with every codec the engine calls:

Library	Engine caller
libx264 (r3191, the revision the x86 ffmpeg-static uses), libx265 with 8-bit and 10-bit	engine/src/services/streamingEncoder.ts (h264/h265 encode), producer/src/services/distributed/plan.ts (mp4 h265), engine/src/services/chunkEncoder.ts (HDR yuv420p10le)
libvpx (VP9 encode, VP9/VP8 alpha decode)	engine/src/services/chunkEncoder.ts (webm), parsers/src/mediaAlpha.ts alphaDecoderFor
libopus	engine/src/services/chunkEncoder.ts muxVideoWithAudio (webm audio)
libzimg (zscale)	engine/src/services/videoFrameExtractor.ts toneMapsWithZscale (assumed present on Linux), core/src/hdrToneMap.ts
libdav1d	AV1 source decode, same as the x86 ffmpeg-static build
zlib, GnuTLS	PNG frames; HTTPS inputs

How

• sources.txt pins every source by sha256 (x264 by git commit, since it has no stable tarballs).
• Dockerfile builds on the Lambda Node 22 image itself (pinned by digest), so the binaries link against the exact glibc they run on. Build packages come from the base's pinned Amazon Linux 2023 release.
• build-ffmpeg.sh links the codec libraries statically into a glibc-dynamic ffmpeg/ffprobe. Shared libraries the bare base lacks (GnuTLS, nettle) are copied beside the binaries and found through an $ORIGIN rpath, so the runtime image needs no package install. License files and the source pins go into share/licenses/. The tarball is packed deterministically (sorted, fixed mtime and owner, gzip -n).
• check-ffmpeg.sh runs in a check stage: the bare Lambda base plus only the built tree. It asserts the encoders, decoders, filters and protocols the engine uses. It also round-trips h264, h265, ProRes 4444, Opus, AAC and PNG; checks that VP9 alpha survives encode and libvpx decode; asserts a 10-bit HEVC encode really comes out yuv420p10le and runs the engine's HDR-to-SDR zscale chain on it; and checks HTTPS against the system trust store, where a trusted host passes and a wrong host name is refused. The artifact stage copies the check's summary, so no tarball can come out of a build whose check failed.
• .github/workflows/lambda-ffmpeg.yml builds on two separate ubuntu-24.04-arm runners and fails unless both tarballs are byte-identical. It runs on PRs that touch the recipe, and by hand.

Test plan

• The same recipe built and checked natively on x86_64 (Lambda Node 22 amd64 base): the check stage passed every assertion; the tarball is 29.6 MB, and only GnuTLS and nettle are bundled beside the binaries
• x86_64 reproducibility: a second build with --no-cache produced a byte-identical tarball
• The check is not vacuous: each of these makes the check stage fail, and the unmodified recipe passes: the test CA not trusted (TLS fails), alpha decoded with ffmpeg's native VP9 decoder instead of libvpx (alpha frames rejected), the bundled libgnutls.so.30 removed (ffmpeg cannot load), the wrong-host probe pointed at the right host ("accepted a wrong host"), and an earlier 8-bit-only x265 build (ffprobe reports yuv420p for a yuv420p10le encode)
• arm64: this PR's Lambda arm64 ffmpeg workflow (two native ARM builds, check stage, byte-identical comparison) passed at f43fc69; rerunning at this head
• Documentation updated (not yet: nothing user-facing ships in this PR)
• Comments follow CONTRIBUTING.md "Comments"

Not exercised here: running these binaries inside Lambda itself. That is proven with the image and deploy PRs that use them.

[jrusso1020]

Reviewed at f7ec675. The artifact stage depends on the check stage, the source archives are pinned by hash, and the new workflow compares two independent native ARM builds. This is still build-only: none of the five changed files touch deploy or anything that consumes the build at runtime.

P2: build 10-bit x265 and check the encoded pixel format (packages/aws-lambda/image/build-ffmpeg.sh:33-35, packages/aws-lambda/image/check-ffmpeg.sh:43-47).

• The recipe builds 8-bit only. In the pinned x265 4.1, CMake defaults HIGH_BIT_DEPTH and LINKED_10BIT to OFF, and the recipe doesn't override either.
• FFmpeg falls back without failing. With no 10/12-bit x265, the pinned FFmpeg 7.0.2 offers only 8-bit formats (libavcodec/libx265.c:897-904). fftools/ffmpeg_mux_init.c:527-534 then picks a supported format and only logs a warning.
• The check can't see it. ff() runs at -loglevel error, so asking for yuv420p10le can silently produce 8-bit HEVC and still pass the tonemap check.
• The engine needs 10-bit. Its HDR encode requires yuv420p10le (packages/engine/src/services/chunkEncoder.ts:163-169).

Fix: build x265 with 8-bit and 10-bit support, and assert hdr.mp4's actual pix_fmt with ffprobe. A command that succeeds doesn't prove 10-bit.

Since the first requested head (6f55484), the only changes are the renamed recipe and scripts and a change to GnuTLS linkage. The x265 configuration and the HDR check are unchanged. The downloaded x265 and FFmpeg archives match the sources.txt hashes, and the bash syntax checks pass. The base-image digest has both ARM64 and AMD64 manifests.

Not run: a local Docker compile, native ARM codec execution, a Lambda invocation, or the two-build reproducibility check. Whether x264 matches the engine's build is a separate open question I haven't verified.

Review by Rames (hf-review)

[miguel-heygen]

Thanks, fixed at a51fd1d.

• 10-bit x265. build-ffmpeg.sh now builds x265's 10-bit library (HIGH_BIT_DEPTH=ON, EXPORT_C_API=OFF), links it into the 8-bit build (LINKED_10BIT=ON), and merges both archives into the one libx265.a ffmpeg links (x265's multilib layout).
• The check asserts the output, not the command. check-ffmpeg.sh now reads hdr.mp4's pix_fmt with ffprobe and requires yuv420p10le.
• Proof on x86: the new build passes the whole check. The same check run against the previous 8-bit-only build fails at exactly that line: ffprobe reports yuv420p for an encode that asked for yuv420p10le, the silent fallback you described.
• x264 parity, your open question: the x86 path's ffmpeg-static 7.0.2 writes x264 - core 164 r3191 4613ac3 into its own encoded output. The recipe now pins that commit, 4613ac3c15fd, r3191 on x264's stable branch.
• Also in this head: the check's ffprobe calls now end their options with --, which the producer's ffprobe argv contract test requires. That was this PR's red producer unit check.

[jrusso1020]

Re-reviewed at a51fd1d against my f7ec675 request for changes. The blocker is fixed: x265 is no longer 8-bit only, and the HDR output format is now checked.

• Build (packages/aws-lambda/image/build-ffmpeg.sh:33-43): builds the 10-bit library separately (HIGH_BIT_DEPTH=ON, EXPORT_C_API=OFF), links it into the 8-bit build (LINKED_10BIT=ON), and merges both static archives with ar -M. This is x265's upstream multilib layout. ENABLE_SVE2=OFF leaves out kernels the base image's gcc 11 can't build.
• Check (check-ffmpeg.sh:46): now requires the probed pix_fmt of the encoded HDR fixture to be yuv420p10le.

What was verified

• Both scripts pass shell syntax checks.
• The new check, run against system-FFmpeg fixtures, rejects 8-bit output and accepts 10-bit. That validates the check itself, not the pinned artifact.

What was not run

• A full Docker, x265 or FFmpeg compile.
• Native ARM execution, a Lambda invocation, or the two-build reproducibility check.
• The ARM and x86 build evidence in the PR is the author's, and CI was not independently confirmed here.
• x264 is now pinned to r3191, but byte parity with the existing x86 binary is still unverified.

This approves the build recipe only. It is not acceptance for release or deployment.

Review by Rames (hf-review)