Repository navigation
feat(aws-lambda): run chunks on an optional arm64 image function - #6065
Conversation
vanceingalls
left a comment
There was a problem hiding this comment.
Reviewed at f3dbc6be (one commit on main 2ef60f3f). The head then moved to 3502b082, which only rewrites the Globals tags comment in template.yaml; the test file's blob is unchanged.
What holds
- Audio parity holds even where chunks carry audio.
renderChunkdoes mux audio into chunks whenencoder.chunksCarryAudiois set (renderChunk.ts:1011-1019). ButmuxChunkWithAudiostream-copies (chunkAudio.ts:234,-c copy), so the audio packets in an arm64 chunk are the plan's x86 encodes, byte for byte. The only encodes are in plan, audio and assemble, and all of them stay onRenderFunction. - Every chunk renders on one architecture. Both
RenderChunktasks (v1:464, v2 Distributed Map:583) take the same!If, and nothing else in the handler invokesrenderChunk. So a render never concats x86-encoded chunks with arm64 ones. - The concurrency sizing still lines up. The plan reads
HYPERFRAMES_LAMBDA_RESERVED_CONCURRENCYfrom its own environment (handler.ts:376), which equals the video function's reservation. With no reservation, it sizes fromGetAccountSettings, which onlyRenderFunction's role may call. That's correct, because only the plan calls it. - Moving
RuntimeandArchitecturesoffGlobalsis required, not cosmetic. AnImagefunction rejectsRuntime.MemorySize,Timeout,Tracing,Tagsand the globalEnvironmentstill reach both functions. - The role and alarm wiring holds. The
!If … AWS::NoValuelist element on the role drops cleanly when the parameter is empty, and the chunk-invocation alarm follows the chunks. - Locally:
cfn-lint1.58.0 passes on this template, as it does on main's.
Nits (non-blocking)
ChromeSourcereaches the video function throughGlobals.HYPERFRAMES_LAMBDA_CHROME_SOURCE: !Ref ChromeSourcemerges into the image function's environment. WithChromeSource=chrome-headless-shelland aVideoImageUri, every chunk would look for a shell binary the image never has (#6064's staging refuses shell ZIPs). SettingHYPERFRAMES_LAMBDA_CHROME_SOURCE: sparticuzonVideoFunction(function-level keys win), or adding aRulesassertion, would close that.- For the CLI-option PR:
packages/cli/src/commands/lambda/policies.tshas noecr:actions. Deploying an image function needsecr:BatchGetImage/ecr:GetDownloadUrlForLayeron the repository (plusecr:SetRepositoryPolicywhen Lambda adds its own pull policy) for whoever runssam deploy. - The CDK stack has no video-function option yet (
HyperframesRenderStack.ts:3says it mirrors this template). That's fine as a follow-up.
CI: green at 3502b082. Seven of the 8 required checks show success (Build, Typecheck, Test: runtime contract, Tests on windows-latest, Render on windows-latest, Semantic PR title, regression), and Comments passes too. The Test aggregate check-run still reads "in_progress", but its job's steps all completed and the CI run concluded success at 15:33Z. Its shards (Test (runtime), (studio), (support)) are green.
Verdict: APPROVE
Reasoning: An empty parameter leaves the deploy unchanged except for where Runtime and Architectures are declared. With the parameter set, only renderChunk moves to arm64, and chunk audio is stream-copied, so the audio-bytes claim holds.
— Via
A new VideoImageUri parameter adds an arm64 container-image function that only renders chunks. Plan, audio and assemble stay on the x86 ZIP function, so audio encodes are unchanged. Empty (the default) deploys exactly as before. The reserved concurrency moves to whichever function renders the chunks.
3502b08 to
f901661
Compare
vanceingalls
left a comment
There was a problem hiding this comment.
Changing my approval at f901661f to request changes. I missed this in my earlier review, and it was found while reviewing #6073, the CLI PR that deploys this template.
Blocker: sam deploy refuses this template, so hyperframes lambda deploy stops working for everyone
VideoFunction (examples/aws-lambda/template.yaml:193-201) is an AWS::Serverless::Function with PackageType: Image. SAM CLI checks every deploy for image functions before it evaluates any Condition. image_repository_validation sets required whenever any resource with an ImageUri has PackageType: Image. So when --image-repository, --image-repositories and --resolve-image-repos are all absent, sam deploy exits 2. That holds even when VideoImageUri is empty and HasVideoFunction is false.
I reproduced this with SAM CLI 1.169.0, running the argv the CLI's samDeploy builds:
sam deploy --stack-name s --region us-east-1 --resolve-s3 --capabilities CAPABILITY_IAM \
--no-confirm-changeset --no-fail-on-empty-changeset --parameter-overrides ChromeSource=sparticuz
Error: Missing option '--image-repositories', '--image-repository', '--resolve-image-repos'
The same command against main's template gets past validation and only fails at CreateChangeSet on my fake credentials. The released CLI deploys this template, because prebuilt-template.mjs copies it into the package. So if this merges and ships, the default x86 lambda deploy breaks too. cfn-lint, which I ran last time, doesn't check this; only SAM CLI does.
Everything else in my earlier review still holds. The fix may live in #6073, since the CLI is what calls sam deploy. Either way, this template shouldn't ship without a deploy path that passes SAM's check.
Verdict: REQUEST CHANGES
Reasoning: With this template, sam deploy fails SAM's image-repository check for every stack, which breaks the CLI's default deploy. I reproduced it on SAM CLI 1.169.0.
— Via
… image repository sam deploy refuses a template with any top-level image function unless it is given an image repository, even when that function's Condition is false, so every deploy of the template failed. The video function now lives in video-function.yaml, a nested stack created only when VideoImageUri is set. The release ships it beside template.yaml and the CLI stages both. A new workflow runs the CLI's sam deploy argv against the template with fake credentials and requires it to reach the first upload.
|
Thanks, this was a real blocker, and the live deploy would have hit it. Fixed in f9bfde8, at the template rather than in one caller, because the CLI,
The live deploy and render with #6073 come next. |
vanceingalls
left a comment
There was a problem hiding this comment.
Re-reviewed at ede453d1: two new commits, f901661f..ede453d1. f9bfde86 nests the video function, and ede453d1 runs the plan on it. This resolves my change request (5617532846).
What's fixed
-
sam deploygets past SAM's image-repository check again. The image function now lives invideo-function.yamlas a plainAWS::Lambda::Function, nested through a conditionalAWS::CloudFormation::Stack.get_template_artifacts_formatreads only the parent's resources, so nothing there needs an image repository. -
I ran the full packaging step locally, not just validation. I ran SAM CLI 1.169.0 against a local S3 and CloudFormation mock (moto), with the CLI's argv and
--s3-bucket, for bothVideoImageUri=""and a digest-pinned ECR URI.- Both runs packaged and reached
Changeset created successfully. Execution then failed only inside the mock. - The nested template was uploaded and
TemplateURLrewritten to its S3 URL. Code.ImageUristayed aRef, so SAM didn't try to push an image.
- Both runs packaged and reached
-
The new
sam deploy validationjob is a real check. I rancheck-sam-deploy.shwithf901661f's template, and it exits 1 onMissing option '--image-repositories'…. It's green at this head. -
The nested function keeps what
Globalsused to give it.Globalsis unchanged, and the nested function sets each of its keys itself: memory, timeout, active tracing, the two tags,NODE_OPTIONSandHYPERFRAMES_LAMBDA_CHROME_SOURCE. Its role spells outS3CrudPolicy's actions plusAbortMultipartUpload, as before. -
Every reference moved to the nested stack's outputs. That covers both
RenderChunktasks, the state-machine invoke policy, both alarms and theVideoFunctionArnoutput. -
The packaged CLI ships the nested template.
prebuilt-template.mjscopiesvideo-function.yaml, andstagePrebuiltHandlerstages every.yamlbeside the ZIP. -
The deploy policy already covers the nested stack. The actions it needs,
cloudformation:CreateStack/UpdateStack/DeleteStackandiam:CreateRole/PutRolePolicy/AttachRolePolicy/TagRole, are all inpolicies.ts.CAPABILITY_IAMcovers the unnamed role. -
Running the plan on the video function (
ede453d1) is needed, not optional.renderChunk.ts:554refuses a chunk whoseffmpeg -versionfirst line differs from the one the plan recorded. x86'sffmpeg-staticsaysffmpeg version 7.0.2-static https://johnvansickle.com/ffmpeg/ …, and the arm64 image saysffmpeg version 7.0.2 Copyright …. So with an x86 plan, every arm64 chunk fails withFFMPEG_VERSION_MISMATCH. I missed this atf901661f. BothPlantasks (v1, v2) now take the same!Ifas the chunks. The nested role gainslambda:GetAccountSettings, which is the only AWS call the plan makes beyond S3. Audio (v2) and bothAssembletasks stay onRenderFunction.
Important (please fix, not blocking):
- The PR body's audio claim is stale. "Plan, audio and assemble … Keeping them on the x86 function means a mixed deploy produces the same audio bytes as an x86-only one" no longer holds. The plan's mix (
mixPlanAudio,plan.ts:1337) now runs on the arm64 build, and v1 chunks stream-copy slices of it. It's the same ffmpeg version, but a different build and SIMD path, so byte-identical audio across deploy types isn't guaranteed. I don't see a user-facing defect, but the body (and the squash message, if it carries the body) should say what actually ships.
Nits (non-blocking)
- The
RenderChunkInvocationAlarmnow also counts plan invocations, one per render, because it watches the video function. - The CI check never reaches the nested stack. It stops at the first upload, which is
RenderFunction's ZIP and comes before the nested stack in the template. So a regression in packaging the nested stack would still pass. The script also hard-codes the CLI's argv, and the workflow only triggers onexamples/aws-lambda/**, so a change tosam.tswon't re-run it.
#6073's restack onto this head (3cfaadd4) already prices Plan/PlanV2 at the video function's architecture and keeps the stage-every-.yaml loop, so I've dropped those two cross-PR notes from my last comment.
CI: green at ede453d1, with all 8 required checks passing (Build, Typecheck, Test, Test: runtime contract, Tests on windows-latest, Render on windows-latest, Semantic PR title, regression). The new sam deploy validation job is green. Comments (not required) is red on the comment-share ratchet.
Verdict: APPROVE
Reasoning: Nesting the image function clears SAM's check for both the x86 and arm64 values, which I confirmed through packaging and change-set creation. Running the plan where the chunks run fixes an ffmpeg-version mismatch that would have failed every arm64 chunk. The audio-parity sentence in the body needs updating, but it doesn't block.
— Via
vanceingalls
left a comment
There was a problem hiding this comment.
Re-reviewed at 3ba71a33, two commits on ede453d1, which I approved (5618919196). I've reviewed both commits.
What changed
ebea3594is a rename only. InstagePrebuiltHandler(prebuiltHandler.ts:94),templatesbecomestemplateDir. It still stages every.yamlbeside the pinned template.3ba71a33is docs only.VideoImageUri's description now says the image runs the plan and the chunks, which matches the!Ifon bothPlantasks (template.yaml:278,:342).- The PR body now matches the code. It covers the plan on the video function,
lambda:GetAccountSettingson its role (video-function.yaml:63), and the contract test that pins plan and chunks to one function (HyperframesRenderStack.snapshot.test.ts:348). That resolves the important note from my last review.
Nit (non-blocking): the audio sentence is a little broad. "Audio and assemble stay on x86, so an MP4's audio, which the separate audio task encodes, is byte-identical to an x86-only deploy". The separate audio task only runs for plan v2 with workerConcurrency > 1, MP4 and non-CFR (handler.ts:513, plan.ts:1094). A CFR MP4, a single-worker render, or a v1 plan still mixes its audio in the plan (plan.ts:1536), which now runs on arm64. Scoping it to "the default v2 render of a VFR MP4" would make it exact.
Nits from earlier that still stand: the chunk-invocation alarm counts plan invocations, and the sam deploy check stops before packaging the nested stack.
CI: green at 3ba71a33, with all 8 required checks passing. Build, Typecheck, Test and Test: runtime contract passed in CI run 39845787362; a second CI run on the same head skipped those jobs. Tests on windows-latest first failed on its engine-cli shard: chunkEncoder.unreadableFrame.test.ts took 7.8s against a 5s limit, and parallelWorkerBrowsers.browser.test.ts timed out at 45s. This PR touches neither, and the re-run passed. sam deploy validation and Comments are green.
Verdict: APPROVE
Reasoning: Since my approval at ede453d1, the only changes are a local-variable rename and a parameter description. The PR body now describes what ships, apart from one sentence that is a little broad.
— Via
What
The SAM template (
examples/aws-lambda/template.yaml, also the templatehyperframes lambda deployuses) gets an optionalVideoImageUriparameter. When it is set, an arm64 container-image function,<project>-video, runs the plan and renders the chunks. The existing x86 ZIP function keeps audio and assemble.Empty, the default, deploys exactly as before: one x86 function does everything. Nothing passes the parameter yet; the CLI option comes next (#6073).
Why
Chunk rendering is the bulk of a render's compute and runs well on arm64. The plan must run there too: a chunk refuses a plan recorded with a different ffmpeg build (
renderChunk.ts), so plan and chunks share one function. Audio and assemble stay on x86, so an MP4's audio, which the separate audio task encodes, is byte-identical to an x86-only deploy.How
examples/aws-lambda/video-function.yaml, a nested stack (VideoFunctionStack) created only whenVideoImageUriis set.sam deployrefuses a template with any top-level image function unless it is given an image repository, even when that function'sConditionis false. Inline, it broke every deploy.AWS::Lambda::Function(PackageType: Image, arm64) and its role.lambda:GetAccountSettings, which the plan calls to size the render.RenderChunktasks (plan v1 and v2) invoke!If [HasVideoFunction, <video function>, RenderFunction]. Every other task is unchanged, and the state machine role may invoke both functions.RuntimeandArchitecturesmove fromGlobalsonto the ZIP function.VideoFunctionArnoutput.video-function.yamlinto the package besidetemplate.yaml, andlambda deploywithout a checkout stages both.Lambda SAM template:examples/aws-lambda/scripts/check-sam-deploy.shrunssam validate --linton both templates. It then runs the CLI'ssam deployargv with fake credentials, with and without a video image, and requires each run to get past validation and stop at the first upload.Test plan
bash examples/aws-lambda/scripts/check-sam-deploy.shwith SAM CLI 1.169.0: both templates lint clean, and both deploys get past validation. The same script on this PR's previous template fails withMissing option '--image-repositories', '--image-repository', '--resolve-image-repos'.sam packagewith real credentials uploads the nested template and leaves the child'sImageUrias the parameter referencebun test src/cdk/HyperframesRenderStack.snapshot.test.ts scripts/pin-handler-zip.test.ts: 29 pass, three runs in a row. CLIvitest run src/commands/lambda: 65 pass, three runs in a row. Typecheck, oxlint, oxfmt and fallow are clean.template.yaml. The contract test pins the plan and the chunks to the same function: routing either one back to x86 fails it. (Earlier: v2 chunks left on x86; role unable to invoke the video function.)