Skip to content

feat: opt-in 4:4:4 chroma for SDR MP4 masters - #5820

Open
PaprikaChen wants to merge 9 commits into
heygen-com:mainfrom
PaprikaChen:feat/chroma-subsampling-444
Open

PaprikaChen wants to merge 9 commits into
heygen-com:mainfrom
PaprikaChen:feat/chroma-subsampling-444

Conversation

@PaprikaChen

@PaprikaChen PaprikaChen commented Oct 18, 2026 •

Copy link
Copy Markdown

What

Adds an opt-in chromaSubsampling: "420" | "444" render option (--chroma-subsampling on hyperframes render, a RenderConfig field, and the Studio / producer HTTP render routes). "444" renders an SDR MP4 master with full-resolution colour: libx264 yuv444p (High 4:4:4 Predictive) encoded from lossless PNG capture. The default stays "420" and the default render path is unchanged.

Why

Every SDR MP4 is 4:2:0, which softens or fringes thin coloured text, UI strokes and saturated edges. Motion-graphics masters are mostly that. 4:4:4 keeps the colour, but many players cannot decode it, so it is opt-in only.

Related work

Fixes #5555. The approach was posted on the issue before implementing.

Measurement (issue step 1)

Each clip was encoded from a lossless RGB master with libx264 medium at the same CRF, using the engine's SDR colour conversion (BT.709, aq-mode=3). PSNR and SSIM are computed per plane against the master. Windows 11, ffmpeg 9.0.2, 1920x1080, 30 fps, 5 s.

Clip (CRF 18) Chroma Size (KB) Encode (s) PSNR U / V (dB) SSIM U / V
Coloured text on dark (rendered by HyperFrames, PNG capture) 4:2:0 1756 0.83 38.28 / 37.17 0.9880 / 0.9881
4:4:4 757 1.17 54.44 / 54.01 0.9996 / 0.9997
UI screen recording (Studio window) 4:2:0 1200 0.92 40.76 / 38.20 0.9900 / 0.9874
4:4:4 1064 1.23 53.37 / 53.12 0.9987 / 0.9990
Footage (NASA RED 4K clip, public domain, downscaled) 4:2:0 2496 1.92 51.33 / 51.93 0.9949 / 0.9956
4:4:4 2566 2.45 51.76 / 52.53 0.9950 / 0.9957
Gradient 4:2:0 952 1.43 54.54 / 54.01 0.9971 / 0.9969
4:4:4 984 1.69 53.82 / 53.53 0.9968 / 0.9965

CRF 23 shows the same pattern. Large gain for text and UI (about +13 to +16 dB chroma, visibly cleaner edges, smaller files at the same CRF); none for footage or gradients.

4x nearest-neighbour crops (master | 4:2:0 | 4:2:0 lanczos | 4:4:4):

text-red-on-blue ui-preview-text footage-sphere-handle gradient-edge

End to end through this PR (hyperframes render vs hyperframes render --chroma-subsampling 444, compared with a PNG-capture reference): chroma PSNR 21.9 / 22.9 dB vs 44.6 / 45.4 dB on a 1 px red/blue stripe test.

How

  • Capture is the catch. At JPEG quality 100, Chrome's beginFrame (the Linux default) and CDP screenshots are already 4:2:0; only drawElement's canvas JPEG is 4:4:4, and drawElement can fall back to screenshot mid-render. Verified with Chrome 152 on Windows and on Ubuntu 24.04 (GitHub Actions). So "444" captures PNG on every path and turns drawElement off (compile gate chroma_444).
  • engine: CaptureOptions.transparentBackground separates "PNG capture" from "transparent background". Absent, it follows format === "png" as before, so alpha output and motion blur are unchanged; false keeps the page background so an opaque MP4 does not turn black. Opaque PNG skips drawElement (JPEG-only verification and worker encode). getEncoderPreset takes an optional chromaSubsampling that only changes the SDR MP4 branch to yuv444p; x264 picks High 4:4:4 Predictive and odd sizes need no padding.
  • producer: RenderConfig.chromaSubsampling; resolveCaptureSurface gives the probe and main capture the same format, quality and background; the preset gets the chroma value, so every encode path and the segment-resume hash see yuv444p.
  • Validation: "444" with a non-mp4 format, GPU encoding or HDR output is rejected before any work starts; auto-detected HDR sources fail before capture with a --sdr hint; distributed / Lambda / Cloud Run requests refuse it (chunk workers and the CFR assemble are 4:2:0). The CLI, the Studio route (400) and the producer HTTP server reject the same combinations.
  • GPU encoders: only NVENC lists yuv444p (QSV and VAAPI are NV12-only here, AMF has no 4:4:4 H.264, VideoToolbox H.264 has no 4:4:4 profile), so GPU encoding is rejected rather than silently falling back to 4:2:0 on some hosts. NVENC support could be a follow-up.

Player compatibility (issue step 3)

Also in the rendering guide.

Player 4:4:4 MP4 How tested
Chrome 155 (Windows) Plays Automated: load / play / seek, decoded frame vs ffmpeg 73.9 dB
Edge 155 (Windows) Plays Automated (73.9 dB) and manual
Firefox 157 (Windows) Does not play MediaError 3: no YUV 4:4:4 decoder; canPlayType empty
VLC 3.0.24 (Windows) Plays Automated, frames inspected
Windows Media Player / Media Player app Does not play Manual: "unsupported encoding settings"
Safari, iOS 26.6 Plays Manual
Safari 26.6.2 (macOS 26.6.2) Plays Automated on a GitHub macOS runner (safaridriver): load / play / seek, frame matches visually
QuickTime / AVFoundation (macOS 26.6.2) Plays Automated through AVFoundation (QuickTime Player's playback framework): playable, profile 244, all frames decoded, frame vs ffmpeg 65.5 dB

Every player above plays the default 4:2:0 output.

Cost

Median of 3, 1 capture worker, 1920x1080, 300 frames, Windows 11 with a hardware GPU.

Composition Default Default, drawElement off 444
Coloured text (capture / total) 11.2 s / 16.2 s 20.5 s / 24.9 s 26.8 s / 31.5 s
Full-frame footage + title (capture / total) 19.6 s / 25.9 s 26.0 s / 30.7 s 85.1 s / 91.2 s

About 2x the default render time for text and UI, about 3.5x for full-frame footage: PNG encoding cost grows with image detail. Documented in the guide.

Out of scope (noted, not changed)

  • The default 4:2:0 path downsamples chroma with flags=neighbor; lanczos gains about 1.5 to 2 dB chroma on text and UI with no compatibility cost. Happy to open a separate issue.
  • Motion-blur MP4 also captures PNG and so clears the page background; transparentBackground: false would fix it in one line.
  • Opaque PNG capture keeps Chrome's optimizeForSpeed off (it protects alpha); enabling it for opaque PNG may cut the 444 capture cost.

Test plan

  • Unit tests added/updated: engine (getEncoderPreset, encoder args at odd sizes, wantsTransparentBackground, verification-page background), producer (chromaSubsampling gates, capture surface, compile gate, render request, HTTP server), studio-server (encoder options, render route), CLI (plan, Docker argv, local render forwarding)
  • Manual testing performed: real renders with and without --chroma-subsampling 444 (yuv444p / High 4:4:4 Predictive, background unchanged, PNG capture logged), 4 workers, odd 1281x721 size, every rejection case; player tests above
  • Documentation updated: rendering guide (4:4:4 masters, compatibility, cost), CLI reference, producer docs and README, CLI built-in docs, hyperframes-cli skill reference and manifest
  • Comments follow CONTRIBUTING.md "Comments"; check-comment-citations and comment-ratchet pass

Developed on Windows: engine, producer, studio-server and CLI suites show no new failures against main; the remaining failures there are Windows-only and also fail on main (symlink permissions, POSIX sleep/echo in tests, path separators, ffmpeg 8+ dropping -vsync).

🤖 Generated with Claude Code

PaprikaChen and others added 9 commits October 17, 2026 20:20
PNG capture used to imply a cleared page background. A new
CaptureOptions.transparentBackground lets lossless opaque PNG capture
(4:4:4 MP4 output) keep the page background. Absent, it still follows
format === "png", so alpha output and motion blur are unchanged.

Opaque PNG skips drawElement, whose verification and worker-encode
paths are JPEG-only.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getEncoderPreset takes an optional chromaSubsampling ("420" by
default). "444" returns yuv444p for SDR MP4, so libx264 encodes the
High 4:4:4 Predictive profile and odd sizes need no even padding.
WebM, MOV and HDR presets ignore it. The ChromaSubsampling type is
exported for the producer.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture

RenderConfig.chromaSubsampling ("420" by default) opts an SDR MP4
render into 4:4:4. "444" encodes libx264 yuv444p and captures every
frame as opaque PNG: Chrome's quality-100 JPEG is already 4:2:0 on
beginFrame and screenshot capture, and drawElement can fall back to
screenshot mid-render, so PNG is the only capture that keeps the
colour. drawElement is turned off for these renders.

The probe and main capture options share resolveCaptureSurface, so
they cannot disagree on format, quality or background.

Formats other than mp4, GPU encoding and HDR output are rejected
before any work starts, auto-detected HDR sources fail before capture,
and distributed requests refuse 444. The default render is unchanged.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared parseEncoderOptions validates chromaSubsampling and answers
400 for "444" with a non-mp4 format, gpu, or force-hdr, so a request
that would come back 4:2:0 never starts a render. The Studio render
route, both Studio adapters and the producer HTTP server forward it.
The producer server also refuses "444" with outputDynamicRange "hdr",
which parseEncoderOptions cannot see.

No Studio UI control; the option is route-level like crf and hdrMode.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`hyperframes render --chroma-subsampling 444` opts an SDR MP4 render
into full-resolution chroma (yuv444p with lossless PNG capture). The
default stays 420. The plan rejects 444 with a non-mp4 format, --gpu
or --hdr before anything renders, with a usage error that names the
conflicting flag.

The value reaches the local render request and, only when it is 444,
the Docker argv, so default Docker invocations are unchanged.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a "4:4:4 masters" section to the rendering guide: when to use
--chroma-subsampling 444, why it captures PNG, the measured chroma gain
(large for text and UI, none for footage and gradients), its limits,
and a player compatibility table to be filled in from testing.

Document the flag in the CLI reference, the built-in rendering docs and
the hyperframes-cli skill reference (with the regenerated skills
manifest), and the chromaSubsampling option in the producer docs.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fill the 4:4:4 compatibility table from testing H.264 High 4:4:4
Predictive output. Chrome 155, Edge 155, VLC 3.0, Safari on iOS 26.6
and Safari 26.6 on macOS play it, and so does AVFoundation on macOS
26.6, the framework QuickTime Player plays through. Firefox 157 on
Windows and both Windows media players do not.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the vague "slower capture" note with measured numbers: a text
and UI composition rendered in about twice the default time, full-frame
footage in about three and a half times, because PNG encoding cost grows
with image detail.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling-444

# Conflicts:
#	packages/producer/src/renderRequest.ts
#	packages/producer/src/services/renderOrchestrator.ts
#	skills-manifest.json
@mintlify

mintlify Bot commented Oct 22, 2026

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 90da21a.

Approve preview

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opt-in 4:4:4 chroma for MP4 masters (measure first)

1 participant