Skip to content

[Feat] feature-demo skill: polished, narrated demo videos - #1177

Draft
mrubens wants to merge 36 commits into
developfrom
feat/feature-demo-skill
Draft

[Feat] feature-demo skill: polished, narrated demo videos#1177
mrubens wants to merge 36 commits into
developfrom
feat/feature-demo-skill

Conversation

@mrubens

@mrubens mrubens commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What

A new feature-demo standard skill that produces polished, Screen Studio-style demo videos of product features: a real recording driven live in the sandbox browser, post-produced with smooth zoom-to-element moves, a synthetic cursor, click ripples, captions, and optional voice-over narration.

Invoke with $feature-demo or by asking for a demo video in a task.

How it works

One declarative demo script drives everything. The capture runner performs the real browser interactions (scroll, hover, click, type) via the agent-browser CLI and records, on the same clock, where the cursor is, when clicks land, and the resolved rectangle of every focused element. The bundled Remotion composition consumes that timeline, so a zoom can never drift from the element it targets.

Pipeline: author script → capture (delegated to the proof-runner subagent, keeping browser containment intact) → narrate (optional) → fit timing → render → verify → upload via manage_artifacts.

The bundled Remotion project is a reference template, not a fixed pipeline stage: the agent copies it into the work dir and adapts the copy (branding, caption styling, layout, presets) when a demo calls for it, installing Remotion's official agent skills (remotion-dev/skills — their sanctioned successor to the deprecated Remotion MCP) for API guidance. The timeline JSON is the stable contract between capture and render, and a render/README.md documents the hard-won invariants worth preserving (counter-scaled cursor, edge-clamp guard, timeline-driven interpolation).

The timing fit encodes what makes these demos feel produced rather than recorded: the dead opening hold is trimmed, the voice-over is paced into a conversational band (pitch-preserving atempo), and the recording's playback rate is solved so each spoken line finishes just before its zoom lands.

Narration without the key in the sandbox

Narration uses ElevenLabs, but the provider key never enters a task sandbox:

  • New control-plane endpoint POST /api/tts/narration (route policy task-token, rate-limited, streamed body cap). The sandbox posts plain caption text with its run-scoped token; the control plane holds R_ELEVENLABS_API_KEY / R_ELEVENLABS_VOICE_ID and returns audio. Same posture as the inference gateway and custom MCP proxy.
  • R_ELEVENLABS_* are registered as control-plane env names, so they are reserved from the generic env editor and stripped from sandbox env injection.
  • Deployments without the vars get a 404 and the skill degrades cleanly to captions-only.

Worker image

Remotion cannot render with the bundled agent-browser Chrome (Chrome 143 removed old-headless), so the image now bakes Remotion's chrome-headless-shell at /opt/remotion/headless-shell (both architectures, pinned, arch-independent symlink). Older snapshots fall back to npx remotion browser ensure at render time — the smoke run below exercised exactly that fallback.

Testing

  • pnpm lint, pnpm check-types, pnpm knip green across workspaces.
  • New unit tests: TTS endpoint (auth, config-off 404, body caps including a lying/endless stream, upstream failure opacity, key injection), skill structure/policy assertions, control-plane env membership, Dockerfile parity.
  • End-to-end smoke inside the real worker sandbox image using the shipped skill files: capture of a live docs page → captions-only timing fit (trimmed 1.21s of dead opening) → npm install + headless-shell fallback → rendered a clean 7.6s 1080p mp4, verified with ffprobe and frame inspection.

Docs

  • Cookbook recipe: Record feature demo videos (setup, narration env vars, tips).
  • environment-variables reference for the two new optional vars, including the recommendation to scope the ElevenLabs key to text-to-speech only.

A new standard skill that produces Screen Studio-style feature demos:
the demo script drives real browser interactions (delegated to the
proof-runner subagent via agent-browser) while logging cursor, click,
and element-rect timelines on the same clock, then a bundled Remotion
composition post-produces the recording with zoom-to-element moves,
click ripples, captions, and optional voice-over narration.

- packages/cloud-agents .../feature-demo: SKILL.md orchestration,
  capture runner, narration + timing-fit scripts, render project
  (excluded from tsc/knip/oxlint like agent-browser; copied verbatim
  into the sandbox skills dir at activation)
- apps/api /api/tts/narration: control-plane TTS endpoint (task-token
  policy + rate limit). ElevenLabs credentials live only on the control
  plane; R_ELEVENLABS_* are control-plane env names, stripped from
  sandbox env injection. Unconfigured deployments 404 and the skill
  degrades to captions-only.
- apps/worker/Dockerfile: bake Remotion's chrome-headless-shell at
  /opt/remotion/headless-shell (the bundled Chrome removed old
  headless); runtime fallback via remotion browser ensure for older
  snapshots.
- docs: cookbook recipe + environment variable reference.
@roomote-community

roomote-community Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

No code issues found. See task

  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/scripts/fit-timing.mjs:164-169 Narrated clips are scheduled to end after their focus zooms. — dismissed: narration is intentionally set to continue through the zoom hold.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/capture/capture.mjs:112-116 The synthetic cursor moves during intervals where the recorded mouse is still stationary.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/scripts/fit-timing.mjs:146 Narration uses caption display times instead of focus-zoom landing times, so it begins before the zoom starts.
  • apps/worker/Dockerfile:242-246 Comment lines break the continued RUN command and prevent the worker image from building.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md:89 The timing instructions say narration ends before its zoom, while the fitter intentionally starts it shortly before landing and plays it through the hold.
  • apps/worker/src/run-task/proof-runner-prompt.ts:40-46 The verified copy remains parent-writable in shared /tmp, so it can still be replaced after hashing and before node opens it.
  • packages/types/src/mcp-oauth.ts:538 The credential-only ElevenLabs catalog entry is mounted as an MCP proxy with no upstream URL, preventing the API from starting.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/capture/capture.mjs:49 The worker's agent-browser wrapper strips --headed and forces headless mode, so GPU-backed canvases still stall during capture.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md:61 Mandatory source-file selector verification has no viable path for supported external public-page demos.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md:63 External pages cannot correct a named selector failure because the prescribed re-capture requires a second proof-runner brief, which the workflow does not permit.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/capture/capture.mjs:83 Headed capture is preempted by the wrapper's headless preview-cookie seeding, so the WebGL fallback always aborts.
  • packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/render/src/DemoStage.tsx:134-135 The coverage-floor ramp still creates a vertically cropped, narrower-than-canvas window while focus zooms animate, exposing side gutters.

Reviewed 4597bcb

Comment on lines +52 to +54
const js = `(function(){var e=document.querySelector(${JSON.stringify(
sel,
)});if(!e)return null;var r=e.getBoundingClientRect();return{x:r.x,y:r.y,w:r.width,h:r.height};})()`;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b995257: the selector now travels into the page-eval as base64 with a strict alphabet check before interpolation, so no selector bytes can reach the constructed code.

mrubens added 5 commits August 8, 2026 00:55
eleven_v3 reads slowly and needs the 1.12 atempo speed-up; v2-family
models pace naturally and should not be sped up. The narration endpoint
now returns its modelId, build-narration records it in the manifest,
and fit-timing derives the atempo default from it (argv still wins).
Per review direction: the bundled Remotion project is a starting point
the agent copies and adapts (branding, captions, layout, presets), not
a fixed pipeline stage. The timeline JSON stays the stable contract
between capture and render. When adapting, the skill installs Remotion's
official agent skills (remotion-dev/skills — their sanctioned successor
to the deprecated Remotion MCP) for current API guidance. A render/
README documents the bug classes worth preserving (counter-scaled
cursor, edge-clamp guard, timeline-driven interpolation).
Simplify per review: one model, one settings profile, no
R_ELEVENLABS_MODEL knob and no model-aware atempo plumbing.
multilingual_v2 reads voice clones naturally at a natural pace; the
timing fit's atempo stays available as an explicit argv override.
- capture: bracket cursor motion with a hold key at motion start so the
  synthetic cursor no longer drifts toward its next target through
  waits, holds, and scrolls (and clicks on a fresh element get a short
  bracketed hop instead of a slow glide).
- capture: pass selectors into the page-eval as alphabet-checked base64
  so no selector bytes are interpolated into code (CodeQL).
- fit-timing: solve the playback rate by simulating the schedule
  instead of a closed-form cumulative estimate; a line that waits for a
  late anchor ends later than the estimate, which could push later
  lines past their anchors. Lines intentionally START just before their
  zoom and play over its hold; the docs now say so explicitly.
The cookbook index is generated from recipe frontmatter; run the
generator instead of hand-editing between the markers.
The binary name differs per platform: headless_shell on linux-arm64,
chrome-headless-shell on linux64, so the hardcoded name failed the
amd64 image build. Resolve it with a find over both names (validated
under amd64 emulation and in the arm64 sandbox image).
Comment thread apps/worker/Dockerfile Outdated
Capture emits an explicit per-caption anchor (the zoom's landing time);
the timing fit schedules against it. caption.start begins with the
cursor glide ~moveMs earlier, so anchoring there started narration
before the zoom even began moving. Also hoist the Dockerfile block
comment above the RUN chain.
Comment thread packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md Outdated
Field failure from a real game demo: CDP screencast only emits frames
on visual damage and stamps them without wall-clock gaps, so a 52s
interaction over a static-rendering surface collapsed into a sub-second
video. The capture runner now injects an imperceptible 2px rAF ticker
after record start so compositor damage flows at wall-clock rate
(static-page regression: 7.93s timeline -> 8.0s recording), closes any
pre-existing pages before recording (record start creates a fresh
browser context; a page left over from inspection would split beats
and recording across contexts), and fails loudly when the recording
comes back much shorter than the interaction instead of letting a
garbage video flow downstream. SKILL.md routes that failure to a
'stale sandbox runtime' blocker (old snapshots carry a 2018 legacy
ffmpeg that crashes record stop under real frame volume).
@mrubens

mrubens commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Field-failure fix: static surfaces recorded as sub-second videos

A real dogfood run (game demo) surfaced a capture failure mode none of the development surfaces hit: CDP screencast only emits frames on visual damage, so a statically-rendered app (games between state changes, idle dashboards) collapsed a 52-second interaction into a 0.8s video with no wall-clock mapping. Docs pages and fixtures masked this because scrolling and hover repaint constantly.

Fixed in 8dc12b2, all reproduced/verified against live sandboxes:

  • Capture injects an imperceptible 2px requestAnimationFrame ticker after record start, keeping compositor damage — and therefore frames — flowing at wall-clock rate. Static-page regression test in the worker image: 7.93s timeline → 8.0s recording (previously ~0.2s).
  • record start creates a fresh browser context, so capture now closes pre-existing pages first — otherwise a page left open by an earlier inspection step leaves the beats driving one context while the recorder watches another.
  • Honest-state gate: capture fails loudly when the recording is much shorter than the interaction, instead of letting a garbage video reach render/upload. SKILL.md routes the related record stop ffmpeg crash (old snapshots carry a 2018 legacy-installer ffmpeg) to a stale sandbox runtime blocker.

Second dogfood run failed at the delegation seam in two ways: the
brief's home-relative runner path did not resolve in the delegated
runtime, and the proof-runner refused the capture script as a
disallowed browser path (its instructions allow only the agent-browser
CLI, and a per-brief assurance is not reliable authority).

- The skill now stages the runner at /tmp/feature-demo/capture.mjs
  before delegating (no home-relative paths across the boundary) and
  parent-side paths use $HOME explicitly.
- The proof-runner prompt carries a narrow standing exception: the
  staged runner is an agent-browser orchestrator and running it as the
  brief specifies is compliant; the exception names that one path and
  no other script.
@mrubens

mrubens commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Second dogfood failure: the delegation seam

A second live run failed before capture in two ways, both at the parent→proof-runner boundary:

  1. Home-relative paths don't survive delegation — the brief pointed at ~/.agents/skills/feature-demo/..., which didn't resolve in the delegated runtime. The skill now stages the runner at /tmp/feature-demo/capture.mjs before delegating, and parent-side commands use $HOME explicitly.
  2. The proof-runner refused the runner as a disallowed browser path. Its instructions allow only the agent-browser CLI, and a per-brief assurance turned out to be unreliable authority — correctly so, arguably. The proof-runner prompt now carries a narrow standing exception: the staged runner at exactly that path is a sanctioned agent-browser orchestrator; no other script is covered.

Fixed in b17b6c4 with prompt/skill test coverage for both.

Comment thread apps/worker/src/run-task/proof-runner-prompt.ts Outdated
mrubens added 3 commits August 8, 2026 02:24
Live-sandbox controls showed the sparse-frames failure is page-specific:
a WebGL game stalls headless frame production for the whole page while
plain pages on the same sandbox record at full wall-clock rate. SKILL.md
now names that blocker (webgl surface stalls headless recording) so
agents diagnose by target instead of burning retries; the stale-runtime
ffmpeg hint stays scoped to actual record-stop ffmpeg errors.
Review caught that the sanction trusted a parent-writable /tmp path:
anything staged there would inherit the narrow agent-browser allowance.
The worker now computes the sha256 of the activated feature-demo
capture runner at config-generation time and pins it into the
proof-runner prompt; the runner verifies the staged file's digest
before executing and reports 'capture runner integrity mismatch'
otherwise. Without a digest (skill absent) the sanction is omitted
entirely.
Admins configure narration TTS under Settings → Integrations → ElevenLabs
(API key + voice ID) instead of deployment env vars. Encrypted at rest,
deployment-scoped, admin-only — same storage as Granola.

New 'credential_only' serverMode: the integration exposes no MCP tools
to agents and is explicitly excluded from getMcpServerConfigs, so the
key is consumed only by the control-plane /api/tts endpoint and never
travels toward a task sandbox (proven by an inverse delivery test).

The TTS endpoint now resolves credentials from the connection first,
falling back to R_ELEVENLABS_* env vars for self-hosters. UI, tRPC
commands/procedures, hooks, icon, self-setup catalog metadata, and docs
updated; env-var mocks in the TTS test replaced with a resolver mock.
Comment thread packages/types/src/mcp-oauth.ts
mrubens added 4 commits August 8, 2026 11:51
The API server registers a proxy route for every non-native catalog
entry, and route construction throws for an integration without an
upstream URL — so adding credential-only ElevenLabs broke server
construction (caught by the live-server integration suite). Add
isCredentialOnlyMcpIntegration and skip those entries in the proxy
registration loop, alongside native and Linear.
Completes the review point on bbcbb92: the worker's curated proxy-path
map no longer carries a dead entry for integrations that are never
delivered to sandboxes.
Review caught a TOCTOU in the digest sanction: sha256sum then node as
separate commands over the parent-writable staging path leaves a swap
window. The proof-runner now copies the staged file to a fresh private
path, digests the copy, and executes exactly that verified copy —
checked bytes are executed bytes, and the staging path is never
re-read after verification.
Follow-up to 6b206ef: the first assertion still matched the old
'runner at' phrasing; the sanction now reads 'runner staged by the
parent at'.
mrubens added 2 commits August 8, 2026 12:13
The prior mktemp copy still lived in shared /tmp, so a same-uid parent
could swap it between sha256sum and node opening it — copying does not
close a TOCTOU. The proof-runner now runs a single node process that
reads capture.mjs exactly once, hashes those bytes, and executes the
same bytes via a data: import only on an exact match. The file is
opened once, so there is no post-hash window; the digest is still
baked in at config-generation time. SKILL.md hands off paths and lets
the proof-runner own the verified invocation.
A live dogfood run collapsed a 47s game interaction into a 0.7s clip:
GPU-backed canvases (games, 3D, WebGL/WebGPU) never present to the
headless compositor, starving the screencast. agent-browser's own docs
prescribe `--headed` for exactly this, and on displayless Linux it
auto-starts a private Xvfb (present in the worker image). The capture
runner now prefixes --headed on every agent-browser command by default
(HEADED=0 opts out); harmless on ordinary pages. Verified end-to-end in
roomote-worker:arm64-real against an animated WebGL canvas: full
pipeline, real-duration recording, auto-Xvfb, exit 0. SKILL.md updated;
this supersedes the earlier 'file an upstream issue' plan — the flag
already exists.
mrubens added 4 commits August 8, 2026 12:45
The generated wrapper stripped --headed and force-exported
AGENT_BROWSER_HEADED=false, so the capture runner's --headed was a
no-op (review catch). The wrapper now forwards --headed and exports
AGENT_BROWSER_HEADED=true only when a caller explicitly passes it;
default behavior is byte-identical (headless), preview-cookie seeding
stays headless, and --headed false/=false explicitly opts out. Verified
the arg handling in isolation across the command set.
… capture

Forensics on a stalled demo capture (live worker container) traced the
real root cause past the misleading agent narrative: worker containers
run with no init process and a 512 pids-limit, so agent-browser daemon
restarts leak Chrome process trees whose defunct children accumulate
(observed 174 defunct, pids.current 494/512) until fresh Chrome dies
with pthread_create: Resource temporarily unavailable — surfaced as
Chrome exiting early and misread as a display problem.

- Run docker workers and task daemons with --init (tini) so orphaned
  children are reaped.
- Raise DOCKER_WORKER_PIDS_LIMIT default 512 -> 2048 (still overridable).
- feature-demo capture: stop any existing agent-browser daemon
  (agent-browser close, not just close --all) before recording so
  --headed actually applies, and hard-fail when record start reports
  '--headed ignored: daemon already running' instead of silently
  recording a headless (stalling) session.

Diagnosis from the Roomote-Evals forensic session; verified the code
gaps directly. Capture path re-smoked end-to-end in the worker image
including the daemon-already-running trap.
0.27 does not auto-start a virtual display for --headed, so headed
recording died with 'Missing X server or $DISPLAY' (and, against an
already-running daemon, '--headed ignored'). 0.33.2 auto-starts Xvfb on
displayless Linux, which is what the feature-demo capture path needs.
Verified in the worker image: headed recording of an ordinary page now
succeeds end-to-end where 0.27 failed. (WebGL-heavy game surfaces can
still hang the headed browser — a separate, known limitation the skill
reports as a blocker.)
The demo's story beats, captions, and selector candidates are creative
planning work, so the skill now delegates that step to the existing
advisor subagent (R_PLANNING_MODEL when configured, else the coding
model at the advisor reasoning level) with a complete brief. The parent
keeps ownership of everything the advisor cannot do: verifying proposed
selectors against real page source, writing demo-script.json, and the
capture/render pipeline. Follows the harness advisor contract: internal
guidance only, one retry on empty output, then plan directly.
Comment thread packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md Outdated
Source-file selector verification is impossible for a public page the
user names: no local source to grep, and the parent must not drive the
browser. Split verification by surface — repository-backed surfaces
grep source; external pages delegate a read-only resolve-only pre-flight
to the proof-runner (open URL, snapshot/get box, confirm or correct each
selector) before authoring the script.
Comment thread packages/cloud-agents/src/server/workflows/skills/standard/feature-demo/SKILL.md Outdated
mrubens added 3 commits August 8, 2026 17:48
Baking the chrome-headless-shell but not the render project's deps left
every demo paying a multi-minute Remotion + React npm install. Bake
node_modules at /opt/feature-demo/render from the skill's own pinned
package.json (single source of truth, no drift), and have the render
step reuse them (cp the baked node_modules into the work-dir copy); npm
install is now only the fallback for an adapted composition that adds
deps or an older snapshot. Validated in the worker image: rendered fully
offline (npx --offline) reusing the baked modules, no install.
The previous pre-flight conflicted with the proof-runner contract (fixed
browser target, one brief per task), so it was not actually usable. For
an external public page the browser is reachable only through the single
capture delegation, and the capture runner already resolves every
selector live and fails loudly naming any that miss. Author best-effort
resilient selectors and correct them from that named failure within the
allowed retry, rather than inventing a second browser trip.
…tion

Adapting an existing composition needs markup, render, and doc-lookup
guidance — not the full remotion-dev/skills bundle (which also pulls
create/maps/saas/upgrade/interactivity). Pin the install to
--skill remotion-markup --skill remotion-render --skill remotion-docs
--yes. Verified the exact CLI syntax in the worker image: installs
exactly those three, non-interactively, project scope, no warnings.
@mrubens
mrubens marked this pull request as draft August 8, 2026 22:09
Inspired by Mux's agent-video two-pass design, inverted to keep our
post-production polish: narration is synthesized BEFORE capture, and
the capture runner conducts the browser to it — each captioned beat
holds for exactly as long as its line takes to speak (real clip
duration when narrated; estimated speaking time from the caption text
when the deployment has no TTS configured), the voice starts 0.4s
before its zoom lands, and clip start times are stamped at the moment
each zoom actually lands. Captions-only demos therefore read at
narrative pace too, and fit-timing collapses from a reconciliation
solver (atempo, playback-rate search, rescheduling) to a trim-only
pass — that whole problem class is gone.

Also settles the recording mode on evidence from repeated in-image
runs: headless + frame ticker is deterministic at wall-clock rate
(3/3 identical), while agent-browser 0.33.2 headed recording wedges
its daemon on longer sessions regardless of ticker (3/3 failures,
failure point moves). Default is now headless; '\''headed'\'': true in
the demo script opts a GPU/WebGL surface into headed capture with an
honest failure path. Validated end-to-end narrated in the worker
image: clip starts land exactly at zoom-land minus the voice lead and
the rendered mp4 carries the audio track at the stamped times.
// headless compositor — opt in per demo with `"headed": true` in the script
// (or HEADED=1). Note: current agent-browser headed recording can wedge the
// daemon on longer sessions; use it only when the surface requires it.
const HEADED = script.headed === true || process.env.HEADED === '1';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting "headed": true still cannot start a headed capture when preview cookie seeding is active. The wrapper treats record start as a seedable command and invokes the real CLI with AGENT_BROWSER_HEADED=false to set cookies in the default session before forwarding this command. That creates a headless daemon, so the following --headed record start is ignored and this runner aborts at the daemon-warning check. Seed cookies through the requested launch mode or isolate the seeding session so the advertised WebGL fallback can run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — the seeding calls hardcoded AGENT_BROWSER_HEADED=false, so whenever seeding was what launched the daemon, a later --headed record start was silently headless (and my warning-string check then aborted, or worse, false-positived when the runner's own viewport call had already launched the daemon headed). Fixed in 3aa8cc9 on both layers: seed_preview_cookies now inherits the wrapped command's requested mode (AGENT_BROWSER_WANT_HEADED), and the runner replaced warning parsing with a positive mode check — after record start it reads navigator.userAgent and hard-fails a headed request only when the browser actually reports HeadlessChrome. Validated in the worker image with seeding active (ROOMOTE_WEB_HOST + bypass value set): headed opt-in records headed and passes the UA check; the headless default is unaffected.

mrubens added 5 commits August 8, 2026 21:27
…by UA

Review caught that the wrapper's preview-cookie seeding always invoked
the CLI with AGENT_BROWSER_HEADED=false before seedable commands, so
when seeding was what launched the daemon, a later --headed record was
silently headless. Seeding now inherits the wrapped command's requested
mode. The capture runner also stops parsing the ambiguous
'daemon already running' warning (a false positive when the runner's
own viewport call pre-launched the daemon headed) and instead verifies
the browser's actual mode positively via its user agent — headless
Chrome self-identifies as HeadlessChrome. Validated in the worker image
with seeding active (preview host + bypass value set): headed opt-in
records headed and passes the UA check; headless default unaffected.
Feedback from watching a produced demo: the constant zoom-to-element
reads as artificial, and the story is too sparse. New beat grammar:
'show' (scroll the subject into view and narrate over it, camera wide)
is the default narrated move; 'focus' zooms are reserved as the wind-up
for an interaction the demo actually performs, so a zoom communicates
'watch this click'. The advisor now writes the narration first as a
flowing 4-7 line spoken story (8-20 words per line, why-it-matters to
payoff) instead of 3-9 word labels, and the speaking-time estimate
clamp rises to fit. Validated in the worker image: a three-line demo
runs 20.5s with exactly one zoom (the click wind-up), camera wide and
scrolling everywhere else.
Narrative-length cuts (25-45s, 750+ frames, software rendering on a
small container) routinely exceed the harness's default two-minute
command window; agents were burning a first attempt on the timeout and
retrying. Tell them to allocate ~10 minutes up front.
Two rulings from watching produced demos: on-screen captions must be
the spoken lines verbatim (the lines.json divergence mechanism is
removed), and narrated captions now highlight the word being spoken.
The TTS endpoint uses ElevenLabs' with-timestamps variant and passes
character alignment through; build-narration rolls it into per-word
timings (with a sanity check that skips highlighting if alignment ever
disagrees with the text); the renderer draws captions as word spans —
active word glowing, spoken words bright, upcoming words dimmed. Word
times are relative to clip starts, so audio-first stamping keeps the
highlight accurate with no extra sync machinery. Captions-only demos
render plain. Verified by rendering a timed line in the worker image
and inspecting frames mid-line: the highlight sweeps with the audio.
Captions no longer overlay the recorded content: the canvas splits into
a stage (where the window lays out and zooms) and a reserved caption
band on the backdrop beneath it — the window is inset anyway, so the
space was already paid for. Zoom translation clamps to the stage and a
stage clip guarantees even a deep zoom never bleeds into the band.
Deriving the stage-aware clamp also fixed a latent bug in the original
edge-clamp bounds that could show backdrop gaps when zooming targets
near the window edge.

The demo script also gains a declarative captionStyle object —
position top/bottom, active-word accent color, pill on/off (off = bare
text with drop shadow), and a font-size multiplier — for brand-fit
tweaks without touching the render template; deeper redesigns still go
through the adaptation path. Verified by rendering wide and
deep-edge-zoom frames in the worker image: band stays clean in both.

Also: lint-staged oxfmt now passes --no-error-on-unmatched-pattern so a
commit touching only formatter-ignored paths (like this skill's render
template) is not rejected by the pre-commit hook.

const SHOT_W = timeline.video.width;
const SHOT_H = timeline.video.height;
const usableH = captionBand > 0 ? stageH - pad : canvasH - 2 * pad;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reserved band changes the wide preset's base window from 1344px to about 1205px. At the documented/default 1.5x focus scale, the window is then only about 1807px wide on a 1920px canvas, so the BASE_W * S >= canvasW clamp never runs and backdrop shows at the left/right during ordinary focus zooms. Previously the 2016px scaled window covered the frame. Preserve enough stage width (or ensure the zoom/clamp covers the stage) so the normal 1.5x focus does not regress into visible side gutters.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c02c34f. The band-shrunk window (1205px wide) at the default 1.5x scale covered only ~1807px, exposing side gutters — as you said. The fix floors the effective scale at full-width coverage (canvasW / BASE_W) whenever a zoom passes the vertical-cropping threshold (stageH / BASE_H), ramping in smoothly over 0.15 scale units, so the view is always either a floating uncropped window or a full-bleed zoom, never a cropped hybrid. Show-beat glides drop to 1.12x to stay below the cropping threshold and keep floating.

One subtlety the fix surfaced: the floor lands exactly on canvasW / BASE_W, and in floating point BASE_W * S evaluates one ulp short of canvasW (1205 * (1920/1205) = 1919.9999999999998), which silently skipped the edge-clamp gate and let the raw center-pull expose a gutter anyway. The floor now overshoots by 0.002 and the clamp gates get a half-pixel epsilon.

Validated in the worker image with the reviewer scenario (default 1.5x focus + a show glide, captions-on): frames early/mid/late in the zoom hold are all full-bleed edge to edge, and the glide floats uncropped.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ramp still leaves a cropped-but-not-full-width interval during every normal focus animation. In the wide captioned preset, sRaw = 1.2 yields S ≈ 1.307: the window is 984px tall (past the 873px stage, so it is clipped) but only 1575px wide (short of the 1920px canvas). The side gutters therefore remain visible while the zoom crosses this range, despite the comment claiming the view is never a cropped hybrid. Snap to sCover once sRaw reaches sCrop, or delay clipping until horizontal coverage is reached.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right that the ramp still animated through the bad state, and chasing it turned up why the floor was the wrong shape of fix entirely.

In the wide preset the window starts cropping vertically at 1.16x (with a caption band reserved; 1.29x without one) but does not reach the canvas edges until 1.59x. Every scale in between is cropped AND narrower than the canvas, so a zoom cannot get from rest to full-bleed without crossing it — no ramp avoids that. Full-bleed is not reachable cleanly in that geometry at all, and this predates the caption band; the band only widened the gap enough to put the resting default 1.5x inside it.

So 4597bcb caps the zoom at the largest scale that keeps the whole window on the stage, less an 8px inset so a sliver of backdrop always remains. The window is then always a complete rounded card, never a crop with backdrop down one side. Presets where coverage comes first are untouched: vertical covers the width at 1.20x and does not crop until 2.58x, so it still goes full-bleed. The tradeoff is that a focus zoom is now a gentle push-in rather than a punch to full screen, which suits a skill that already treats zooms as rare wind-ups; glides drop to 1.06 so a focus beat still reads as distinct, and SKILL.md documents the cap.

The edge clamps also lost their gates. The same pair of bounds means 'keep covering the stage' when the scaled window is bigger than it and 'stay inside the stage' when it is smaller — they just swap order — so clamping unconditionally is right in both regimes, and it fixes a latent case where a nearly-stage-height window could drift into the caption band on the center pull. The floating-point epsilon from the previous commit is gone with the floor.

Verified over every frame rather than samples: a render exercising both documented zoom extremes (1.5 and 1.9) is probed at the stage's left and top edge strips and each frame classified. 339/339 frames floating, 0 hybrid. As a control, the same probe finds 143 hybrid frames in the previous build, beginning at frame 43 — right inside the ease, which is exactly what you flagged.

mrubens added 3 commits August 9, 2026 00:05
The reserved caption band shrinks the window enough that a default 1.5x
focus zoom cropped vertically without covering the canvas width, exposing
backdrop side gutters. Floor the effective scale at full-width coverage
once a zoom passes the cropping threshold, so the view is always either a
floating uncropped window or a full-bleed zoom. The floor lands exactly on
canvasW / BASE_W, where floating point can leave BASE_W * S one ulp short
of the clamp gate — overshoot the floor slightly and give the gates a
half-pixel epsilon. Show-beat glides drop to 1.12x to stay below the
cropping threshold. The window's drop shadow moves outside the stage clip
so it falls on the backdrop instead of shearing at the stage edge.
Replaces the navy gradient with the brand color from the web app theme
(--background-new, ~#c9f24d) with a subtle radial falloff.
The coverage floor fixed the resting frame but not the animation: in the
wide preset the window starts cropping vertically at 1.16x (with a caption
band; 1.29x without) and does not reach the canvas edges until 1.59x, so
every scale in between is cropped AND narrower than the canvas, showing
backdrop down one side. A zoom cannot get from rest to full-bleed without
crossing that range, so no ramp shape avoids it — full-bleed is simply not
reachable cleanly in that geometry.

Cap the zoom at the largest scale that keeps the whole window on the stage,
less an 8px inset so a sliver of backdrop always remains. Presets where
coverage comes first (vertical covers the width at 1.20x and does not crop
until 2.58x) are unaffected and still go full-bleed.

The edge clamps drop their gates: the same pair of bounds means keep
covering when the window exceeds the stage and stay inside it when it does
not, so clamping unconditionally is correct in both regimes and also stops
a nearly-stage-height window from drifting into the caption band.

Glides drop to 1.06 so a focus beat still reads as a distinct push-in.

Verified in the worker image over every frame of a render exercising both
documented zoom extremes (1.5 and 1.9): 339/339 frames floating, 0 hybrid.
The same probe finds 143 hybrid frames in the previous build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants