Skip to content

feat(dgw): analyze any recording in the AI crate - #2043

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 18 commits into
build/cadeau-0.5.3from
feat/ai-analysis-pipeline
Open

feat(dgw): analyze any recording in the AI crate#2043
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 18 commits into
build/cadeau-0.5.3from
feat/ai-analysis-pipeline

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

AI analysis of any recorded session, as one call in the AI crate: AiClient::analyze_recording(&manifest, &manifest_path) takes the parsed session manifest and its path, and does the rest.

  • Terminal (.cast, .trp): streamed into timed transcript chunks, described with describe_session_actions.
  • Video (.webm, VP8/VP9): decoded with XMF, sampled twice a second, and a screenshot is kept where the screen changed (once it settles, and every 3 s during a long change; tiles that keep changing, like a clock, are ignored). Screenshots go out at most 1280x720, 40 per request, described by the new describe_screen_actions purpose. This is the frames-first design from the eval (no OCR).
  • Times come from the input, never from the model: transcript lines get ids (L12) and screenshots get ids (F00012); the model cites one, and the action takes its time.
  • A truncated answer is asked again in two halves; each chunk's answer is a checkpoint, so a retry only asks about the chunks left.
  • Working files live in .ai-analysis/ next to the manifest (not reused across models or prompts); the finished log is left as .ai-analysis.slog for the caller to move. The crate never writes recording.json.

The manifest type is the crate's own (RecordingManifest), so Gateway's JrecManifest stays out of the crate's API.

Also in this PR

Where to look hardest

  • recording_analysis/video.rs: frame selection and the decode loop (one next_frame() per decoded frame)
  • recording_analysis/mod.rs: prepare (plan, reuse of working files) and send (resume, cancellation)

Tested

  • devolutions-gateway-ai 91 + 28: frame selector on synthetic screens, screenshot requests for OpenAI and Anthropic (image blocks, labels, context not citable), line citation, resume, discard, cancellation.
  • Real recordings against real providers with the analyze_recording example:
    • S1 (ADUC password reset, WebM 2:12) with GPT-6 Luna: 32 screenshots, 13 actions, about $0.004. In video time, the password reset is placed at 89.5 s and the group add at 107.5 s, vs 88.1 s and 106.3 s in the system events (1.4 s and 1.2 s late, within the eval's ┬▒2 s; the kept screenshot comes after the screen settles). Claude Sonnet 5.5: 24 actions, same two times.
    • a Gateway .cast session (3 min of top) and a .trp session (6 min of htop): one action each, correct.

🤖 Generated with Claude Code

@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2045 October 6, 2026 21:26
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/ai-analysis-pipeline branch 2 times, most recently from f546209 to e1d6488 Compare October 7, 2026 17:50
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2046 October 7, 2026 20:36
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) changed the base branch from feat/ai-analysis-plumbing to feat/provisioner-task October 7, 2026 20:36
Closed
@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title feat(dgw): turn a finished recording into transcript chunks and a session log feat(dgw): analyze a recording in the AI crate Oct 7, 2026
Open
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2049 October 7, 2026 22:19
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) changed the base branch from feat/provisioner-task to feat/ai-analysis-plumbing October 7, 2026 22:19
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2051 October 7, 2026 22:20
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) changed the base branch from feat/ai-analysis-plumbing to feat/provisioner-task October 7, 2026 22:20
@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title feat(dgw): analyze a recording in the AI crate feat(dgw): analyze any recording in the AI crate Oct 7, 2026
Closed
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2052 October 8, 2026 19:35
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/ai-analysis-pipeline branch 3 times, most recently from afdd351 to 2c30aa5 Compare October 13, 2026 16:06
irvingouj@Devolutions (irvingoujAtDevolution) added a commit that referenced this pull request Oct 13, 2026
@irvingoujAtDevolution @claude
feat(dgw): run AI analysis as a provisioner task
d53ce20 
Gateway can now describe what the user did in a recorded session with
an AI provider, and add the result to the session as a new log. The
analysis itself is the AI crate's `analyze_recording`; Gateway gives it
the session manifest once the recording is finished, then adds the log
it returns to the session through the recording manager.

Provisioner tasks run through a ProvisionerTaskRunner, the one place
that keeps a task's record in gateway.db and the job running it on the
job queue in step. Each kind of work is a ProvisionerTask: what one
attempt does, and what to drop once its record ends. The runner records the Task and queues its
job, starts each attempt, cancels and awaits an attempt past its
timeout, retries transient failures until the job queue's attempts are
used up, and records how the Task ends. The Task is the source of
truth and its job only wakes it up: each Task names the one job
allowed to run it, so a stale or duplicate job stops without running
anything. At startup, before the job runner claims anything, every
unfinished Task is handed to a new job, which also covers a Task
recorded just before a crash, before its job was queued.

AI analysis (`recording-ai-analysis`) is the first kind. A repeated
request with the same Task ID gets its Task back, even while the
session records again. A new request is refused while the session is
recording, or when it has no recording.

The API key and the base URL stay in memory only, encrypted, as the
Task's secret in the provisioning store, until the Task ends or reaches
its deadline. After a restart the attempt finds no key and
fails its Task. Neither the key nor the URL appears in errors, logs or
stored results: the AI client (#2043) leaves the URL out of its
transport errors.

Analyses of one session run one at a time, so two attempts never
share the working files the AI crate keeps next to the manifest. Those
files are removed when the Task ends.

Starting an AI analysis is an internal function for now; the HTTP
endpoint comes next.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
feat(terminal-streamer): decode a complete TRP recording
cfa7bbc 
A finished recording can now be read as asciicast lines without
tailing it. The lines come out one packet at a time, so memory use
does not grow with the recording. The packet decoding is shared with
the live stream decoder.

Events written before the terminal setup are still held until the
setup arrives, but no more than 1 MiB of them: past that, the header
goes out with the terminal size known so far, so a recording that
never sends its setup is not held whole in memory.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
feat(dgw): let the AI cite the input instead of writing times
2c33362 
Action times now come from the input, never from the model. Each
transcript line sent to `describe_session_actions` gets an id (`L12`),
the model cites the line where an action starts, and the action takes
the time of that line. Actions citing a line that is not in the
transcript are skipped. The prompt version is now `session-actions-3`.

A new purpose, `describe_screen_actions`, lists what the user did in
screenshots of a session: each PNG is sent after a label with its id
and time, the model cites the screenshot that shows each action, and
the action takes its time. Screenshots marked as context show where
the session was and are never cited. Requests can now carry images,
for OpenAI chat completions and Anthropic Messages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
feat(dgw): analyze any recording in the AI crate
5e51eee 
`AiClient::analyze_recording` takes the parsed session manifest and
its path and does the rest. A session is recorded either as a terminal
or as a video, and the types hold it: `Recordings` is all terminal or
all video, and each recording is accepted only with the extension of
its kind (`Recordings::from_files` refuses a manifest that mixes them
or has none).

- terminal recordings (asciicast and TRP) are read as a stream into
  timed transcript chunks, described with `describe_session_actions`;
- video recordings (WebM, VP8 or VP9) are decoded with XMF and sampled
  twice a second; a screenshot is kept when the screen changed, once it
  settles, and every 3 seconds during a long change (16x16 tiles of a
  half-resolution gray image; tiles that keep changing, such as a
  clock, are ignored). The screenshots are sent at most 1280x720, 40 per
  request, with the previous one as context, and described with
  `describe_screen_actions`.

A truncated answer is asked again in two halves. Each chunk's answer is
saved as a checkpoint, so analysing again after a failure only asks
about the chunks left. The actions of all chunks are merged in time
order into a Session Recording Log, left next to the manifest.

The working files live in `.ai-analysis/` next to the manifest; files
made with another model or prompt are not reused. A successful analysis
removes them, and `recording_analysis::discard` removes them otherwise.
The crate never writes `recording.json`.

`start` runs the analysis on the Tokio runtime and returns a channel of
events: `Progress` (reading the recordings, with the bytes read;
asking the AI, chunk by chunk; writing the log), then `Complete` or
`Failure`, last. Each `Progress` has a stable stage name, a percent
and a short sentence, to be stored with a task or shown to whoever asks
about it. Dropping the receiver cancels the analysis.

The `analyze_recording` example runs it against a real provider and
shows the progress: `cargo run -p devolutions-gateway-ai --example
analyze_recording -- --recording-path <path>`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution
docs(dgw): describe recording analysis in the AI crate intent
b5653d5
@irvingoujAtDevolution
fix(dgw): read only the video track of a WebM recording
0715842 
Every CodecID was taken as the video codec, so a WebM with an audio
track, such as VP8 video with Opus audio, was refused as an
unsupported video codec, and blocks were decoded whatever their
track. The first video track is now picked from the track entries,
and only its blocks are decoded; audio and other tracks are skipped.

Checked locally with XMF on a real session remuxed with an Opus track
before and after the video: the same 32 screenshots as the original.
@irvingoujAtDevolution
fix(dgw): always report the reading progress first
32dcab8 
The first reading progress came from an interval tick racing the end
of the reading, so a small recording could be read before any
progress was sent. The start of the reading is now reported before
the files are read.
@irvingoujAtDevolution
docs(dgw): remove a control character from a doc comment
57e7189
@irvingoujAtDevolution
docs(dgw): describe how terminal recordings are prepared in the AI cr…
56b01c1 
…ate intent
@irvingoujAtDevolution
fix(dgw): keep the provider URL out of AI transport errors
f38a8d5 
reqwest errors repeat the URL of the request, so a transport error
named the provider's endpoint, which callers store in Task results and
logs. Transport errors now drop the URL, and the request debug log no
longer names the base URL.
@irvingoujAtDevolution @claude
fix(dgw): bound WebM tag and frame sizes in the AI crate
00560de 
The WebM parser zero-fills the size an element claims before reading it,
up to 4 GB by default, so a small hostile recording could claim gigabytes.
Elements larger than the file are now refused. The cap is the file length,
not a fixed constant, because remuxed recordings have a known Segment size
as large as the file and the cap applies to every element.

Decoded frames larger than 8192x8192 are refused before they are copied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
fix(dgw): check cancellation while sampling a long video frame
7c977bf 
A frame that stays on screen for a long time, as in an idle session, is
sampled many times. Each sample now checks the cancellation token, and the
half-resolution gray image of a frame is computed on its first sample
only, instead of on every sample.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
fix(terminal-streamer): count TRP packet headers toward the setup cap
182d841 
Events held until the terminal setup were counted by payload size only, so
a flood of empty packets was held whole in memory. Each packet now counts
its 8-byte header too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
refactor(dgw): drop the casts from the video sampling constants
5c57c3f 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
refactor(dgw): show video decoder errors with their cause
134dffa 
VpxError displays only "decoder error"; the libvpx code is in its source.
The messages now show the Display chain instead of the Debug output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
refactor(dgw): log AI analysis steps at debug level
a016fc8 
The library logs its steps at debug; the caller decides what is worth an
info line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
refactor(dgw): keep blocking file checks off the async AI analysis task
74f78f8 
The size of the recordings is read on a blocking thread, and the check for
a chunk already described uses tokio::fs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
fix(dgw): close the AI analysis events on a panic and bound long vide…
0079cc5 
…o pauses

- A panic in the analysis left the watcher holding a sender, so the events
  channel never ended and the caller waited forever. The watcher now stops
  through a drop guard, so the channel ends as documented.
- Dropping the receiver cancelled the caller's own token; it now cancels a
  child token, so a shared token is never cancelled for other work.
- A pause between two video frames was sampled every half second, so a
  corrupt timestamp far ahead spun until the attempt timeout. Past the
  animation window one picture no longer changes the frame selector, so the
  rest of the pause is skipped and counted, with the same frame IDs.
- PNG frames are finished explicitly, so a write or flush error is reported
  instead of dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution @claude
feat(dgw): give the AI analysis working files an owner
cfa0e8c 
The working files next to a recording were reused by any later analysis
with the same model and prompt version, and `discard` removed them whoever
made them. A caller that runs several analyses of one recording, such as
one Task after another, could resume from another one's checkpoints or
delete the checkpoints of one that is still retrying.

`AnalyzeRecording::owner` records whose the files are: only an analysis of
the same owner resumes from them, and another owner starts fresh.
`discard(manifest_path, Some(owner))` leaves another owner's files in place;
`discard(manifest_path, None)` removes them whoever made them, for a caller
that starts over from scratch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2059 October 13, 2026 17:32
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) changed the base branch from feat/provisioner-task to build/cadeau-0.5.3 October 13, 2026 17:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@CBenoit Benoît Cortier (CBenoit) Awaiting requested review from CBenoit

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@irvingoujAtDevolution