[xuanruli]

What

hyperframes audio <file> --read reads one whole recording — an episode, a reference video — as time-stamped lines an agent can read or grep, instead of measurements of one sound:

thailand.mp4 · 574.4 s · stems: bass, drums, guitar, other, piano, vocals
voice: 51 lines, 1 pauses, mean 5.1 chars/s, median pitch 190 Hz
effects: 320 in 70 sounds (S1 thump ×39, S2 thump ×32, S3 whoosh ×28, …)
music: 21 pieces, 38 breaks

22.7–34.0   voice   "1997年那场…"  5.2 chars/s (mean 5.1)  rising end
4.3         effect  riser (S24): high, attack 1620 ms, tail 0.8 s, +8 dB over music
31.7        effect  thump (S1): low, attack 70 ms, tail 0.1 s, while the music is out
39.9–59.8   music   piece 2 (bass · 100 BPM), first appearance after a break, fades from 58.3
30.9–32.1   music   break 1.2 s

• voice: sentences from a transcript (--transcript, or <file>.transcript.json beside it) with speaking rate, pitch when it is 3+ semitones off the median, and a rising or falling end; pauses of 0.4 s or more.
• effects: onsets in the drums stem, named by rule (riser, boom, thump, whoosh, click, tick, ding, hit), grouped by identical sound (S1 most used), quick runs folded into one line. Hits on the music's beat and sounds more than 20 dB under the recording are left out.
• music: the four other non-voice stems, split at real cut-outs (each checked against the 2 s around it), with pieces matched by chroma, stem mix and tempo — resumes after a short break, returns later, or new.

--range a,b picks what is shown; the whole recording's text is cached under .hyperframes/read-audio per file content, with its stems.

Stems

The default is --quality local, which never uploads; at --quality best the recording is uploaded through the asset direct-upload routes and separated by HeyGen's /v3/audio/stems with the CLI's HeyGen credential (HEYGEN_API_KEY, HYPERFRAMES_API_KEY, HEYGEN_ACCESS_TOKEN or a saved hyperframes auth login credential; HYPERFRAMES_AUDIO_STEMS_URL + HYPERFRAMES_AUDIO_API_KEY point at another service instead, and the HeyGen credential is never sent there) (POST waits for the stems; in-progress or dropped requests retry the same body and idempotency key; contract in packages/cli/src/audio/read/stems.ts). The backend half of that route is a separate change. The api analysis provider now finds its credential the same way, on /v3/audio/analysis. Without stems, effects are read from the mix outside speech and the music is not read; with neither stems nor a transcript, voice and effects are not read at all, because the mix alone does not tell speech from music. Every case says so in a warning.

analysis/providers/shared.ts gains serviceAccess and serviceFailure, now shared by the analysis API provider and the stems client.

The configured separation deadline aborts encoding, upload, job requests, retry waits and downloads. Stem files use fixed names for the six documented stems and the existing atomic streaming downloader. An explicitly empty service URL disables that service. Fractional sample hops stay aligned over long recordings; cached read analyses are versioned so older timestamps are recomputed.

Incomplete responses are not cached as complete: all six documented stems must be present. Missing stems produce a fallback warning, and an incomplete cached manifest causes a later --quality best read to retry.

Verification

• Followthrough baseline validation: 103 tests pass across 14 audio, upload and download test files. Includes a signal at 600 seconds, stalled upload/POST cancellation, and ignoring unknown stem names. Synthetic signals and local mock servers only; no live service calls.
• After the partial-stems repair and author protocol integration: all 6 readAudio integration tests pass, including partial-response retry and incomplete cached-manifest recovery.
• CLI typecheck, pre-commit core/studio/scripts typechecks, oxlint, oxfmt, comment checks and fallow pass using existing dependencies. Fresh GitHub CI and Docs validation pending; local Mintlify CLI was unavailable.
• The inherited support-test release-age parsing failure is repaired with the focused fix from fix(cli): keep whisper CJK tokens as separate timed words #6058 (f9c6ba3), tested here with a mocked installer; no local install executed.
• Author-reported experiment, not rerun during followthrough: a 574 s Chinese episode with six local htdemucs_6s stems and a Whisper transcript took 9.4 s and about 1 GB RSS. The author checked reported breaks against per-stem levels. That episode is not in the repository.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Oct 22, 2026, 11:55 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[jerrai-bot-heygen]

Requesting a partial-stems repair at 08f349114aaed6a531adc79acdb44de168e837d6: a service response containing only the required vocals stem is accepted and cached as a complete separation, although effects were not measured. The inline comment describes the false zero and durable cache behavior. I checked the local-by-default path and the explicit empty-URL service disablement in source; I found no separate upload-boundary defect. Exact-head CLI smoke, lint, and fallow passed; Test (support), typecheck, and other CI were still running at this review. No live stems service or long-recording validation here. — Jerrai

[jerrai-bot-heygen]

A completed stems job may contain only vocals: fetchStems requires that one name and writes a manifest for whatever other supported stems happen to be returned. Here, with no drums and no pitched bed, source is an empty array, so findEffects produces a non-null effects: { events: [], groups: [] } and the report claims zero effects, with no warning that effects/music were not measured. With a pitched stem but no drums, it instead searches the music bed itself for effects. The separation: "stems" result is then cached and never upgraded on subsequent --quality best reads, even if the service later supplies the missing stems. Please distinguish missing layers from measured silence, warn/fall back appropriately, and add a partial-stems response + cache-retry regression.

[xuanruli]

Fixed in 99048fe, included in 4ce2f62. A separation is cached only when all six documented stems exist; incomplete responses fall back to mix with a warning naming missing stems. cachedStems rejects partial manifests, and readAudio invalidates a previously cached stems analysis when its manifest is incomplete, allowing best to retry. Added a vocals-only response → complete response regression plus a stale partial-manifest retry regression. All 6 readAudio integration tests and CLI typecheck pass. The author’s e22a831 same-key POST retry contract is preserved in this head.

[jerrai-bot-heygen]

Re-reviewed 4ce2f62d1e310e5faef018cd742c8e02e66148ab against my partial-stems finding at 08f349114aaed6a531adc79acdb44de168e837d6. fetchStems now requires all six documented stem names before writing a complete manifest; a partial response falls back to mix with a missing-stems warning, rather than presenting unmeasured effects as zero. cachedStems rejects incomplete manifests and readAudio invalidates an existing stems analysis if its manifest is incomplete, so a later best-quality read retries. The integration regression covers vocals-only response → complete response and stale partial-manifest retry. The same-key POST path builds its body/key once and resends for 409 request_in_progress or selected connection drops, with the separation timeout signal; the test checks key reuse. The live stems backend protocol and long-recording behavior were not independently exercised or confirmed. Exact-head Test (support), lint, fallow, and CLI npx shim checks had executed successfully; typecheck, CLI smoke, and other checks were still in progress at verdict. No live service call, local test, merge or release by me. — Jerrai

[miguel-heygen]

Additive review at 4ce2f62d1e310e5faef018cd742c8e02e66148ab: source-clear in the audited scope, with COMMENT while the current CodeQL gate is red.

I corroborated Jerrai's repaired partial-stems finding: the six-layer guard (packages/cli/src/audio/read/stems.ts:157) and cached-analysis invalidation (read.ts:197) prevent missing measurements becoming cached silence. The POST loop retains its key/body and deadline; local/default and empty-URL boundaries remain intact.

Audited: the changed audio reader, analysis/provider, command/upload paths and affected consumers; repair delta, focused tests, docs and packed-manifest parser change. Existing reviews/security comments checked for overlap.
Trusting: live backend idempotency/availability, real separation quality and the author's episode/performance evidence. No live upload/model call.

Independent exact-head local verification: 74 tests, nine files, zero skipped with existing Node22/FFmpeg. A separate simulated dropped-socket control preserved the POST key/body; weakening the missing-stems guard failed the warning regression. Snapshots restored and test servers closed.

At 11:19:42 UTC, head still matches and hosted typecheck/support/lint/smoke checks passed, but CodeQL119789418071 is failure: “Unvalidated dynamic method call” at packages/cli/src/audio/read/read.test.ts:104. I read its annotation and the guarded Map.get handler dispatch; this is a test-fixture alert, not a separately demonstrated production exploit. The current PR body does not explain that failing result. It needs repair or an accepted disposition through normal gates. Three Windows test jobs were still running. No merge or rerun by me.

— Magi Home

Verdict: COMMENT
Reasoning: No additional source blocker found; independent controls support the repair. The observed CodeQL failure prevents an approval at this snapshot.

[jerrai-bot-heygen]

Reviewed the successor commit against the previously approved 4ce2f62d head. The only delta is the loopback test fixture's explicit typeof route === "function" check before dispatching a handler from the route Map. It does not change production audio behavior or the partial-stems/cache-retry repair. The prior test-fixture CodeQL alert motivated this guard; fresh JavaScript/TypeScript CodeQL analysis is still running, so I have not represented it as cleared. Other CI remains independently gated. No live stems-service or paid-media validation in this review.

— Jerrai

[jerrai-bot-heygen]

Reviewed the merge-forward exact head. Its parents are the previously approved 2f7c4702 audio head and new main 2faf660a. I compared nontruncated exact-ref Git trees for all 23 net PR-owned files: every blob is unchanged from the approved audio head, including stems, cache, service retry, upload and tests. The two consumer-install helper blobs at the merge head are byte-identical to the new main parent, rather than an unreviewed mix of the older branch helper and main's newer parser. The first-parent delta consists of inherited main changes outside the audio feature. Fresh-head lint executed successfully; support tests, Windows lane and JavaScript CodeQL were still running at review. I did not independently rerun the reported scoped Windows diagnostic or verify a live stems backend/media job. This is a code review of the integrated head, not a merge/release action.

— Jerrai

[miguel-heygen]

Reviewed the merge-forward head a9b6db396eeb044f94e2c51e736620efebf365aa. All 23 net PR-owned files have the same Git blobs as reviewed 2f7c4702; the consumer-install helper and its tests are byte-identical to main parent 2faf660a. Independently ran the two focused ageBlockedPackages parser tests: both pass, zero skipped. No new source finding in the integration.

Initial CI snapshot: CLI smoke (required) job119808974463 fails during npm install -g because npm returns HTTP404 for puppeteer-core-25.16.0.tgz. Correction: the earlier build, render and telemetry smoke steps completed successfully; the failure occurs in the later global-install phase. The full required job did not pass at that snapshot. Current-head Windows, runtime/build and JavaScript CodeQL were still running in the snapshot. I did not rerun CI or claim the registry failure is transient.

The previous-head scoped Windows diagnostic job119801477742 succeeded: 2316 engine/5315 CLI tests, plus runtime contracts. Its full log is retained, alongside the original unexplained exit1; that original failure is not claimed fixed and the prior-head success does not replace new-head gates.

Audited: the exact merge parents, complete first-parent delta, blob parity for all net PR-owned files, newer consumer-install parser and its two tests. Audio/cache/service-boundary source remains as in my prior bounded review, with no live service call. Trusting: inherited main changes outside the integration conflict surface and hosted broader suites at their actual head. This is a bounded review of the larger audio feature, not real-recording/backend verification. No merge or deploy.

— Magi Home

Verdict: COMMENT
Reasoning: The integration is source-clear. The installation failure needed diagnosis before an approval refresh; fresh required checks remain independent.

Update 2026-10-22 12:14 UTC: scoped required CLI-smoke job119813422506 now SUCCESS (full log inspected). Current Windows engine-cli job119809364796 again ends exit1 without failed assertion or Vitest summary; required Windows aggregate119814492732 fails. This remains undiagnosed; no rerun by this reviewer.

[jerrai-bot-heygen]

Exact final tree matches the reviewed integration, with temporary diagnostics absent. Earlier Windows exits remain unexplained; fresh required checks independently gate merge. — Jerrai

[miguel-heygen]

Reviewed cleanup head e45dd4e6ad42905335be37de80d9139f9adb45e5. Its complete Git tree is identical to reviewed a9b6db396 (9989d3f0ae0b160854cf44149b61df1caafe0756); the net delta is empty. The cleanup removes all ten temporary Windows diagnostic lines in packages/cli/vitest.config.ts, preserving the existing Fontconfig cleanup. The audio/cache/service-boundary repairs and main integration are unchanged.

I inspected the complete diagnostic Windows log for job119817507264, checked out at merge c2022f3 containing diagnostic head 596a664: engine 2316 passed, CLI 5315 passed / 223 skipped, and runtime contract 2311 passed. The instrumented CLI runner explicitly reports exit 0. This confirms that run completed; it does not diagnose or fix the earlier premature exits, and it does not substitute for final-head CI.

Audited: complete cleanup commit, exact Git-tree parity, authoritative final PR stat (23 files), prior review/inline discussions, and the diagnostic log. This continues my earlier bounded source review and 74-test local verification; I did not rerun those unchanged tests. Trusting: live backend protocol, separation quality, real-recording/performance claims, and final-head hosted suites as they complete. No live service call, source change, rerun, merge or deployment.

At the current e45dd4e6 snapshot, required smoke and other CI are still running; no failed check is present. JavaScript CodeQL is also running, so the aggregate neutral signal is not a claim that analysis has cleared. The earlier unexplained Windows failures remain recorded. Normal required checks remain merge gates.

Verdict: APPROVE
Reasoning: The final source exactly restores the reviewed tree and introduces no new source blocker. The diagnostic run now has complete passing summaries; final-head CI must finish through the normal gates.

— Magi Home