Repository navigation
fix(cli): long projects check to completion: boxed resumable copies, --prepare-media, idle deadline - #6104
fix(cli): long projects check to completion: boxed resumable copies, --prepare-media, idle deadline#6104miguel-heygen wants to merge 12 commits into
Conversation
vanceingalls
left a comment
There was a problem hiding this comment.
Reviewed at fdd986f7.
Requesting changes for one blocker: the check page never fetches the boxed copy this PR makes up front.
What works: the producer side is fine. resolveProxies passes box through to resolveProxy (proxyTranscoder.ts:681), and proxyCached checks the same box, so a repeat check skips the up-front copy. Making the copy at the composition's box is the right idea.
Blocker: the check server drops hf-proxy-box, so the page's request misses the boxed copy
preResolveHostileMediaProxies now makes the copy under the boxed cache name (checkBrowser.ts:170-178). But the server the check page talks to is serveStaticProjectHtml, and its proxy handler always resolves the copy without a box:
// packages/cli/src/utils/staticProjectServer.ts:239
const proxyPath = await resolveProxy(projectDir, filePath, variant, undefined, { signal });The check page is a top-level page, so the runtime sends the box: displayScale = 1 at mediaProxy.ts:68, shownBox at mediaProxy.ts:327-330. The request is clip.mp4?hf-proxy=h264&hf-proxy-box=2048x1448, and the server reads only hf-proxy. Studio's preview route reads the box (routes/preview.ts:652); this server never did.
I confirmed it with a throwaway test in staticProjectServer.test.ts, using the file's existing mocks:
- Request:
fetch(\${server.url}clip.mp4?hf-proxy=h264&hf-proxy-box=2048x1448`)`. - Expected:
resolveProxy(projectDir, clip.mp4, "h264", { width: 2048, height: 1448 }, …). - Received: the box argument is
undefined.
What that means:
- On main: the up-front copy and the page's request both use the source-size name, so the page gets a cache hit.
- On this branch: the up-front copy is boxed, but the page's request goes to the source-size name. It misses and starts the full-size 4K transcode, the work this PR is meant to avoid.
- Effect on the check: either the check waits for both copies (slower than main), or it samples videos the browser can't decode. The second case would happen if that transcode is cancelled when the server closes, which would also explain "1 copy in the cache after the check" in your table. That table measures only the up-front step, not the page's request.
Suggested fix:
- Parse the box in
serveProxyRequestthe same waypreview.tsdoes:parsePreviewProxyBox(params.get(PREVIEW_PROXY_BOX_PARAM)). Pass it toresolveProxy. - Add a test that links the two halves. For example, after
preResolveHostileMediaProxies, the page's?hf-proxy=…&hf-proxy-box=…request should resolve to the samegetProxyCachePath(…, box). Today's tests mock each side separately, which is why this passes. - This changes all seven callers of
serveStaticProjectHtml, so it's worth a look that snapshot, validate and compare are fine getting boxed copies.
Nits
proxyTranscoder.ts:668-673: passingDEAD_TEMP_MSexplicitly overrides theHYPERFRAMES_PROXY_CACHE_STALE_TEMP_MSenv knob on this path (proxyCache.ts:51,:153). If the knob is meant to apply here, usemin(env, 10m)or leave it unset.- The 10-minute "untouched means dead" rule can delete a live temp file. Example: Studio's ffmpeg is suspended while a laptop sleeps, and a check then runs on the same project. ffmpeg's rename then fails, and the next request makes the copy again. That's rare, and the fallback is acceptable, but it's worth a comment.
Verdict: REQUEST CHANGES
Reasoning: The check server ignores hf-proxy-box, so the page's request misses the boxed copy made up front and starts the full source-size transcode. Shown by a unit probe against serveStaticProjectHtml.
— Via
Edit accuracy: accurate 271 (base branch 271), smooth 150 of thoseThe gate passes. Quarantined, measured but not gated (0) |
vanceingalls
left a comment
There was a problem hiding this comment.
Re-reviewed at c78aa218 (delta since fdd986f7).
Fixed: the first blocker. serveStaticProjectHtml now reads hf-proxy-box and passes it to resolveProxy (staticProjectServer.ts:286-292, :245). It returns 404 for a box that isn't a rung, or a box without hf-proxy, matching Studio. The new tests pin both. The cleanup threshold now respects the env knob, using min(env, 10m). Thanks.
Blocker: the server's own pre-warm still starts the source-size transcode on every check
serveStaticProjectHtml still pre-warms without a box:
staticProjectServer.ts:272callsinjectMediaCodecMap(html, projectDir, [{ html }], undefined, /* prewarm */ true, copies.signal).- That function (
studio-server/src/helpers/mediaProxyPreview.ts:111-118) callsresolveProxy(projectDir, src, proxyVariantFor(facts), undefined, { signal })for every asset whereshouldPrewarmProxy. That includes HEVC and ProRes (mediaCodecMap.ts:59-60).
On main, the check makes the source-size copy up front, so this pre-warm finds it in the cache and does nothing.
On this branch, the check only makes the boxed copy up front. So when runBrowserCheck opens the server (checkBrowser.ts:258), the pre-warm misses the cache (proxyTranscoder.ts resolveProxy: existsSync(cachePath) is false) and starts a full source-size 4K transcode in the background. It runs alongside Chrome for the whole check. copies.abort() stops it when the server closes (staticProjectServer.ts:351/363), so it never finishes or gets cached, and it starts again on every check. Main's second check runs no transcode at all.
That brings back the cost this PR removes: a full-resolution 4K HEVC encode competing for the runner's CPU during sampling, which is the runner time-limit case in the PR body. The "second check (copy reused) 9 s" row was measured with this background encode running and then killed.
The same mismatch now reaches the other six callers of serveStaticProjectHtml (snapshot, validate, compare, …). Their pages now get the boxed copy, but their pre-warm still makes the source-size one, which nothing will ask for.
Suggested fix: pre-warm the copy the page will ask for. Pass the stage box (quantizePreviewProxyBox of findCompositionDimensions(html)) through injectMediaCodecMapIntoHtml to its resolveProxy call. Alternatively, let runBrowserCheck turn the pre-warm off, since it has already made the copies.
Please add a test. For example: after preResolveHostileMediaProxies has made the boxed copy, opening serveStaticProjectHtml should not call resolveProxy with an undefined box.
CI
Tests on windows-latest: engine-cli is red at this head. The @hyperframes/cli test process Exited with code 1, but the job log shows no failing test and no summary, so I couldn't tell whether this change caused it. Worth checking the test-step-log-engine-cli artifact, or re-running.
Verdict: REQUEST CHANGES
Reasoning: The check page now gets the boxed copy, but the check server's pre-warm still starts the source-size transcode in the background on every check. That repeats the cost the PR removes, and on repeat checks it is new work main doesn't do.
— Via
…stopped copy resumes A long source is copied in one-minute pieces that stay in the cache, so a copy that is killed or runs out of time continues from its finished pieces. check --prepare-media runs only that step: no browser, no report, a per-file result as JSON, and exit 1 when a copy failed.
The short limit for dead partial copies would delete the first pieces of a long copy that is still encoding. Finished pieces now follow their own one-hour limit and are never swept while their copy is in flight.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
The comment gate limits a comment line to 120 characters and a run to four lines.
There was a problem hiding this comment.
Verdict: REQUEST CHANGES · 1 blocker · 1 important · 0 nits — reviewed at 90557abe.
Blocker, still open from @vanceingalls's review at c78aa218: the check server still pre-warms the source-size copy. staticProjectServer.ts:272 → injectMediaCodecMapIntoHtml (mediaProxyPreview.ts:113-118) calls resolveProxy(..., undefined /* box */, ...) for every HEVC/ProRes file, so every check starts a full 4K encode next to Chrome. The resumable pieces make this worse. Each check's aborted pre-warm now leaves finished source-size .part-* pieces, so repeat checks keep moving that unwanted encode forward. Those pieces stay out of the size cap until they've gone an hour untouched (readCacheInventory skips live leftovers). Fix: pre-warm at the stage box, or let runBrowserCheck/--prepare-media callers turn the pre-warm off. Add a test that opening the server after preResolveHostileMediaProxies makes no resolveProxy call without a box.
Important (CI): the required Comments check fails at this head. comment-ratchet reports that the comment share went up in mediaMetadata.ts (+1 line) and proxyCache.ts (+3 lines).
Scope
Read the whole diff: check.ts --prepare-media, checkBrowser box/files, staticProjectServer box parsing, proxyCache piece sweep, proxyTranscoder chunking/concat. The box fix from c78aa218 holds.
Not blocking, worth a thought: two processes (Studio + check) chunking the same cache key can race. Whichever finishes first unlinks the pieces (encodeInChunks) while the other is still concatenating them. On Linux the second copy fails once and the next request hits the cache. On Windows unlinkSync of a piece that is still open throws EBUSY and drops the finished copy. Other CI runs were still in progress when I reviewed.
The comment ratchet lets a file's comment share only go down.
There was a problem hiding this comment.
Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits — 1 resolved, 1 open, 0 new · 90557abe → 415bf3db
✅ The Comments check passes now (the commit only removes comments). The blocker from my last review is still open: staticProjectServer.ts:272 still calls injectMediaCodecMap(..., true /* prewarm */, ...), and mediaProxyPreview.ts:114-118 still calls resolveProxy with no box.
…the source-size one The server started a source-size copy of every hostile video next to Chrome on each check, because its pre-warm passed no box while the page and the up-front copy use the composition's box. The pre-warm now takes the same box, found by the one helper the up-front copy uses.
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 0 nits — 1 resolved, 0 open, 0 new · 415bf3db → 72ae3495
✅ Pre-warm blocker fixed. staticProjectServer.ts:280 now passes findStageProxyBox(html), and mediaProxyPreview.ts:119 forwards it to resolveProxy. The check server warms the same boxed copy that preResolveHostileMediaProxies made, so it no longer starts a source-size encode. Both new tests pin the box, and CI is green at this head.
The check ended 120 s after Chrome was ready no matter what it was doing, so a long project whose sampling and overlap pass each kept working still failed (the overlap pass was one page call that reported nothing for its whole run). The deadline is now an idle clock: every phase entered, every sample finished, every overlap batch of eight times and every operation that settles restarts it, and the check ends only after 120 s with none. The overlap grid is asked for in batches so it can report between them.
The comment gate allows 120 characters on a comment line.
There was a problem hiding this comment.
Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits — 0 resolved, 0 open, 1 new · 72ae3495 → b501527e
The idle deadline works. Progress only comes from enter() and the two coarse deadline.run() calls, so a hung page or a hung collectSampleAt still expires after 120 s, and rearm is a no-op before start and after close. But required Test (runtime) fails at this head (↓ inline).
| } | ||
|
|
||
| /** Reruns content_overlap on a fine grid, unconditionally rather than gated on fingerprint change, since an animation aliased to the sparse grid collides between identical-fingerprint samples. */ | ||
| async function collectOverlapGridInBatches( |
There was a problem hiding this comment.
blocker: batching the dense overlap grid in groups of 8 breaks checkBrowser.test.ts:344 ("runs the dense overlap grid in one page task…"). It now sees 10 page tasks instead of 1 (expected 10 to be 1), and that fails required Test (runtime). That test pins one page task for the whole grid. Either update it to the new contract (one task per OVERLAP_BATCH, sample times kept on what each task logs) or keep one task and report progress another way.
There was a problem hiding this comment.
Verdict: REQUEST CHANGES · 2 blockers · 0 important · 0 nits — 1 resolved, 0 open, 2 new · b501527e → 611175a0
✅ The runtime test now states the batched contract: at most 8 per task, ceil(n/8) tasks, and each queued message carries its own batch's range. Test (runtime) passes. Two required checks fail at this head:
- blocker (Typecheck):
checkBrowser.test.ts:344. The clilibhas noObject.groupBy, so tsc reports TS2550, then TS7006 forcallsand TS2339 for.lengthat 344–345. Count per task with aMaporreduceinstead. - blocker (Build):
verify:packed-manifestsreports@hyperframes/studio-serverpacked at 142043 bytes, over its 141718 budget.Buildpassed atb501527eand this commit only touches a test, so the merge with the currentmainis what crosses the line. Rebase, then trim, or raise the budget inscripts/package-size-budgets.jsonwith the reason in the PR body.
… for resumable copies
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 0 nits — 2 resolved, 0 open, 0 new · 611175a0 → fe4b2e54
✅ Typecheck: the test counts per task with a Map, with no Object.groupBy. ✅ Build: the @hyperframes/studio-server packed budget goes from 141718 to 142100 (+382 B), with the reason in the commit and body. With main merged, every non-test source file carries the same patch as at 611175a0. Lint, Typecheck, Build and every Test job passed at this head.
Addressed: the pre-warm blocker was fixed in a later commit and re-reviewed at the current head.
What
hyperframes checknow makes the browser-playable copy of an HEVC/ProRes video at the size that fills the composition, instead of at the source size. A 4K clip in a 1920x1080 project is copied at 2573x1448 (the composition's box on the preview size ladder) rather than 3840x2160. Copies for preview and publish are unchanged. It also clears partial copies that a stopped process left behind, once they have been untouched for 10 minutes (the old limit was an hour).Two more changes cover a clip too long to copy inside a caller's time limit: a source over 90 seconds is copied in one-minute pieces that stay in the cache, so a stopped copy continues from its finished pieces instead of starting over; and
hyperframes check --prepare-mediamakes only the copies and exits (no browser, no report), so a caller can run that step first without the check's time limit.Why
A check on a long 4K60 HEVC project spent its time in "making browser-playable copies". The copy re-encodes every frame at the source size, and a stopped run (a runner time limit, a closed terminal) throws the partial copy away, so the next run starts again. The page the check opens asks for a copy that fills the composition, so the 4K pixels were never needed there.
On the devbox (8 cores) a 4K60 HEVC copy runs at about 0.6x realtime, so 400 s of footage takes about 11 minutes at the source size.
How
serveStaticProjectHtml) now readshf-proxy-boxthe way Studio's preview route does and asksresolveProxyfor that copy. Before, it always asked for the source-size copy, so the page's request would have missed the boxed copy made up front. An invalid box, or a box withouthf-proxy, answers 404 as in the preview route. This is the one place the page's request and the up-front copy now share a cache key. The seven callers of the server (check, snapshot, validate, compare and the others) now get boxed copies for the box their page asks for, the same copies Studio makes.preResolveHostileMediaProxiesfinds the composition's size (findCompositionDimensions), rounds it up the existing preview size ladder (quantizePreviewProxyBox, now exported from@hyperframes/core) and asks for that box. The cache-hit check uses the same box, so a repeat check finds the copy.resolveProxiestakes an optionalboxper source and passes it toresolveProxy, which already names and sizes a boxed copy.resolveProxiesfirst sweeps its cache folder for dead partial copies (.tmp-*untouched for 10 minutes, or less ifHYPERFRAMES_PROXY_CACHE_STALE_TEMP_MSis set lower). A copy being written keeps growing its file, so a live one is kept; a copy whose ffmpeg is suspended for 10 minutes (a laptop asleep) loses its partial and is made again.serveStaticProjectHtmlpre-warms copies when it opens; it now asks for the page's box (findStageProxyBox, the same helper the up-front copy uses) instead of the source-size copy, so a check no longer starts a full-size encode next to Chrome. The other six callers of the server get the same.CheckDeadlineis an idle clock: every phase entered, every sample finished, every operation that settles and every overlap batch restarts the 120 s window, and the check ends only after 120 s with none. The overlap pass was one page call over every time (up to ~240) with no way to report progress, which is where a ~70 s 1080x1920 project with 8 sub-compositions ran out at ~104 s while three--rangeslices passed; it now asks the driver for batches of eight and reports between them. A hung page still ends 120 s after its last progress, and the message reads "no progress for 120 s after Chrome was ready".-ss/-t), each written under a temp name and renamed to a stable.part-<key>-<n>name when finished, then joined with the concat demuxer without re-encoding. A copy that is killed, aborted or out of its 15 minute budget loses at most the piece in flight, and the next run (check, preview or publish, since they share the one transcoder) continues from the first missing piece. Shorter sources keep the single pass. Finished pieces are swept after an hour untouched and never while their copy is running (the 10 minute limit above applies to temp files only, otherwise it would delete the first pieces of a long copy still encoding).check --prepare-media [--json]. Makes only the copies, bounded per file by the existing 15 minute limit, then exits: no browser, no report, same project lock as a check, honors--proxy/--no-proxy.--jsonprints{ ok, prepared: [{ file, state: "ready" | "cached" | "failed", reason? }], total, ready, failed }; progress stays on stderr. Exit 0 when every copy is ready, 1 when any failed. Documented indocs/packages/cli.mdx.Evidence
Synthetic 30 s 4K60 HEVC clip with audio in a 1920x1080 project;
hyperframes checkbuilt from main and from this branch on the same 8-core machine (the branch run was on a busier machine):About 1.5 to 1.9x less time for a 1080p composition (the ladder rounds 1080p up to 2573x1448, not 1080p); a 4K composition gets no change. For 400 s of 4K60 that is roughly 6 minutes instead of 11 on this machine, so a slower machine can still pass a runner's limit. Resumable pieces and
--prepare-mediaare covered below.Resumable pieces, synthetic 200 s 1080p HEVC with audio, Linux, ffmpeg 7.0:
check --prepare-media --json, stopped at 25 s, rerunstate: ready, exit 0; a third run reportscachedin 3 msTest plan
Unit tests added/updated
Manual testing performed
Documentation updated (if applicable)
Comments follow CONTRIBUTING.md "Comments": they say why, not what, and a bug fix says what the code must do and how to reproduce the bug
checkBrowser.test.ts: the copy is requested at the composition's box (2048x1448 for 1920x1080) and the cache check uses the same box. Fails without the fix.proxyTranscoder.test.ts:resolveProxiesmakes a batch's copy at the box it is given, under that box's own cache name; and it removes an 11-minute-old partial while keeping a 1-minute-old one. Both fail without the fix.staticProjectServer.test.ts: a page request withhf-proxy-box=2048x1448resolves that box; an invalid box and a box withouthf-proxyanswer 404. Fail without the change.Each of the five failed without the change and passes with it; studio-server (53 tests),
checkBrowserandstaticProjectServer(82) passed 3 runs in a row.tscfor cli, studio-server and core, oxlint, oxfmt, fallow and the commit hooks pass.Pieces: the rerun encodes only the missing pieces, then joins and leaves only the final file; a short source stays one pass; stale pieces are swept after an hour but not while their copy runs;
--prepare-mediaenters throughcreateCheckCommand(no check pipeline call, JSON shape, exit 1 on a failed copy). Removing the "skip a finished piece" line turns the resume test red (expected [ '0', '60', '0', '60', '120' ] to deeply equal [ '0', '60', '60', '120' ]); removing the in-flight guard turns the sweep test red. cli check tests (170) and studio-server helpers (523) passed 3 runs in a row; typecheck, oxlint, oxfmt and fallow pass.Pre-warm:
mediaProxyPreview.test.ts(the real pre-warm passes the box toresolveProxy) andstaticProjectServer.test.ts(opening the server passes the composition's box to the injection); putting the unboxed call back turns both red. Idle deadline: new tests incheckDeadline.test.ts(keeps going while steps make progress then ends; a settled operation restarts the clock; the message names the phase) andcheck.test.ts(a 10-minute-class overlap pass and slow samples that each finish complete withoutcheck_runtime_failure). Each fails when its piece is removed: enter() no longer restarting the clock (8 red), overlap grid in one call (1 red), no per-sample progress (1 red). 136 tests green x3 including the real-Chrome process tests (checkDeadline.process.test.ts), tsc clean. Not exercised: a 4K60 source through the pieces, audio by ear at piece joins (frame count and duration only; the audio ends 21 ms long over 3 joins), a caller that SIGKILLs only the CLI (its ffmpeg runs to the end of the current piece and its temp file is swept after the limit), a 4K composition, a project whose videos are larger than the stage, and any Windows run.Size budget
@hyperframes/studio-serverpacked budget 141718 -> 142100 bytes (+382). The resumable 60 s copy pieces (proxyTranscoder.ts,proxyCache.ts) are real shipped code; packed size was 142043 (+325 over).