Skip to content

feat(dgw): run AI analysis of a recording as a provisioner task - #2044

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 21 commits into
feat/provisioning-task-secretsfrom
feat/recording-ai-analysis
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 21 commits into
feat/provisioning-task-secretsfrom
feat/recording-ai-analysis

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

AI analysis of a recording, started by DVLS with a plain HTTP request (no new token type), tracked as a provisioner Task and run as a job on the existing job queue.

Provisioner tasks run through a ProvisionerTaskRunner, the one place that keeps a task's record (#2041's ProvisionerTaskRecord) and its job in step. Each kind of work is a ProvisionerTask (what one attempt does, what to drop once its record ends); the runner creates the record, queues its job, runs each attempt with its timeout, retries transient failures and records how the Task ends. The Task is the source of truth and the job only wakes it up: each Task names the one job allowed to run it (#2041), so a stale or duplicate job does nothing. A Task is reported created only once its job is stored in the job queue. At startup, before the job runner claims anything, an unfinished Task whose job is no longer stored (such as after a crash between recording the Task and storing its job) gets a new one; a Task whose job is still stored keeps it. The analysis itself is the AI crate's analyze_recording (#2043); Gateway hands it the manifest of the finished recording and adds the log it returns to the session.

API (stable, no enable_unstable)

  • POST /jet/jrec/{session_id}/ai-analysis, scope gateway.tasks.recording-ai-analysis.start. Body: taskId, provider, model, baseUrl (needed for openai-compatible), maxOutputTokens, apiKey.
    • 202 + Task when created, 200 + Task for the same request again
    • 400 bad settings or body, 404 no recording
    • 409 task_id_conflict / recording_active / analysis_in_progress (+ the running taskId)
  • GET /jet/jrec/{session_id}/ai-analysis/{task_id}, scope gateway.tasks.recording-ai-analysis.read: the Task (state, payload, attempts, times), or 404. The 202 answer of the POST names it in its Location header. A Task of another kind, or the analysis of another session, is not found under that path.

Each provisioner task declares its own scopes, gateway.tasks.<kind>.start and gateway.tasks.<kind>.read, so a token for one kind of Task can neither start nor read another kind. The kind of AI analysis is recording-ai-analysis.

The two new scopes in Devolutions.Gateway.Utils and the regenerated OpenAPI clients are in #2054, stacked on this PR.

The first commit hands the provisioner task store (#2041) to DgwState on its own gateway.db connection; DgwState::mock becomes async for that, hence the .await in a few tests. It also passes job_queue::MAX_ATTEMPTS (5, same as the current default) to the job queue explicitly, so the retry check in the AI analysis job can't drift from it.

How it behaves

  • The API key and base URL live in memory only, encrypted, as the Task's secret in the provisioning store (feat(dgw): keep provisioner task secrets in the provisioning store #2058); never in the job, gateway.db or our logs. After a Gateway restart the Task fails with "the API key is no longer available".
  • Transient provider errors are retried by the job queue (up to 5 attempts); permanent ones fail the Task. 30 min per attempt, 2 h per Task.
  • Terminal (.cast, .trp) and video (.webm) recordings are analyzed (feat(dgw): analyze any recording in the AI crate #2043); video needs XMF v2026.9.25.0 or later. The result is a new ai-analysis-N.slog in the session's artifacts, and the Task payload says which file, the model, the prompt versions and the token usage.

Where to look hardest

  • provisioner_tasks/runner/mod.rs: run_attempt (cancel on timeout, retry vs. fail, on_end on every end) and resume (called from service.rs before the job runner starts)
  • provisioner_tasks/recording_ai_analysis/mod.rs: the AI ProvisionerTask, and lock_session: the working files belong to the session, so analyses of one session run one at a time (a Task past its deadline may still have an attempt running when the next Task starts)
  • start() in the same file: the order of the checks

Known, accepted

  • At VerbosityProfile All, reqwest/hyper's own TRACE lines show the provider's address (not the key). Our code never logs it.
  • If Gateway crashes right between adding the artifact and moving the .slog in, the manifest can point at an empty file. Closing that needs a change to add_artifact.

Tested: devolutions-gateway lib 218 (incl. the runner against a test task: stale jobs after resume, a job that cannot be queued, panics, invalid parameters and unknown kinds; and the AI analysis job against a mock provider: retry resumes at the first chunk left, permanent and exhausted errors, timeout, deadline, lost key, one analysis per session, leftover files removed), tests/ai_analysis.rs 8/8. End to end on the real Gateway binary (127.0.0.1), before the move to ProvisionerTaskRunner and not re-run since with XMF and GPT-6 Luna on a real WebM session: POST → the first attempt hit a connection reset and was retried 30 s later, resuming from the saved screenshots → succeeded, ai-analysis-0.slog in the manifest, working files gone, the API key nowhere in Gateway's logs or data folder.

🤖 Generated with Claude Code

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title feat/recording ai analysis feat(dgw): run AI analysis of a recording as a provisioner task Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Let maintainers know that an action is required on their side

  • Add the label release-required Please cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module) when you request a maintainer to cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module)

  • Add the label release-blocker Follow-up is required before cutting a new release if a follow-up is required before cutting a new release

  • Add the label publish-required Please publish libraries (`Devolutions.Gateway.Utils`, OpenAPI clients, etc) when you request a maintainer to publish libraries (Devolutions.Gateway.Utils, OpenAPI clients, etc.)

  • Add the label publish-blocker Follow-up is required before publishing libraries if a follow-up is required before publishing libraries

@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2045 October 6, 2026 21:26
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2046 October 7, 2026 20:36
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2049 October 7, 2026 22:19
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2051 October 7, 2026 22:20
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/recording-ai-analysis branch 3 times, most recently from a5967ae to 0dfb9a1 Compare October 13, 2026 17:01
The Gateway state now holds the provisioner task store, on its own
gateway.db connection, so the work it records can reach it. Building
the mock state opens an in-memory database, so `DgwState::mock` is now
async.

The job queue attempt limit is now the named constant `MAX_ATTEMPTS`,
still 5, so a job can refer to it.

No behavior change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gateway can now describe what the user did in a recorded session with
an AI provider, and add the result to the session as a new log. The
analysis itself is the AI crate's `analyze_recording`; Gateway gives it
the session manifest once the recording is finished, then adds the log
it returns to the session through the recording manager.

Provisioner tasks run through a ProvisionerTaskRunner, the one place
that keeps a task's record in gateway.db and the job running it on the
job queue in step. Each kind of work is a ProvisionerTask: what one
attempt does, and what to drop once its record ends. The runner records the Task and queues its
job, starts each attempt, cancels and awaits an attempt past its
timeout, retries transient failures until the job queue's attempts are
used up, and records how the Task ends. The Task is the source of
truth and its job only wakes it up: each Task names the one job
allowed to run it, so a stale or duplicate job stops without running
anything. At startup, before the job runner claims anything, every
unfinished Task is handed to a new job, which also covers a Task
recorded just before a crash, before its job was queued.

AI analysis (`recording-ai-analysis`) is the first kind. A repeated
request with the same Task ID gets its Task back, even while the
session records again. A new request is refused while the session is
recording, or when it has no recording.

The API key and the base URL stay in memory only, encrypted, as the
Task's secret in the provisioning store, until the Task ends or reaches
its deadline. After a restart the attempt finds no key and
fails its Task. Neither the key nor the URL appears in errors, logs or
stored results: the AI client (#2043) leaves the URL out of its
transport errors.

Analyses of one session run one at a time, so two attempts never
share the working files the AI crate keeps next to the manifest. Those
files are removed when the Task ends.

Starting an AI analysis is an internal function for now; the HTTP
endpoint comes next.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The provisioner can now start an AI analysis of a recorded session with
`POST /jet/jrec/{session_id}/ai-analysis`, and follow its Task with
`GET /jet/jrec/{session_id}/ai-analysis/{task_id}`, which the
Location header of a 202 answer names. A Task of another kind, or the
analysis of another session, is not found under that path.

Each provisioner task declares its own scopes, named after its kind:
`gateway.tasks.<kind>.start` and `gateway.tasks.<kind>.read`. AI
analysis uses `gateway.tasks.recording-ai-analysis.start` and
`gateway.tasks.recording-ai-analysis.read`.

The request body holds the Task ID chosen by the caller, the AI provider
and model, an optional base URL and output token limit, and the API key.
A new Task answers 202, the same request again answers 200 with the same
Task. Gateway answers 409 with a code when the Task ID is used for other
parameters, when the session still records, or when another AI analysis
of the session is not finished; the last one names the Task ID. A
session without recording answers 404, invalid settings or an unknown
field answer 400.

The API key and the base URL stay in memory only: they are never logged,
returned or written to gateway.db.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Task was reported created once its job reached the channel of the
job queue task, which gives up after a few failed writes without
telling anyone, so a Task could be created without a job. The runner
now writes jobs to the job queue database itself, through
ProvisionerTaskJobQueue, and only reports a Task created once its job
is stored. At startup, a Task whose new job cannot be stored is failed
as not queued, instead of waiting for its deadline with no job.

A Task is only cleaned up once its end is recorded: when recording a
failure fails, the job queue tries the attempt again; when recording a
success fails, the Task keeps its key and working files and fails at
its deadline, since running the work again would add a second log.

Waiting for an earlier AI analysis of the same session now stops when
the attempt is cancelled, so the attempt timeout also covers the wait.
At startup, every unfinished Task got a new job, even when the job it
had was still stored in the job queue and would run anyway, leaving a
stale job behind. The runner now reads the provisioner task jobs the
job queue still holds, and only gives a new job to a Task left without
one, such as a Task created just before a crash or whose job was
dropped. The job queue gains `job_definitions` to list the stored jobs
of a name that may still run.
`as_of_v2022_3_0_0` lists what DVLS 2022.3 sends; later scopes such as
`gateway.recording.delete` were never added there. The AI analysis scopes
are covered by the `ai_analysis` tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`AiSettings` and `AiAccess` (and so `StartRequest`) derived `Debug`, which
prints the base URL that must never reach the logs. Nothing logs them today;
this keeps it that way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The server only ever sends 0 to 100, as documented.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failure to list or hand over the unfinished Tasks stopped the whole
service. Unfinished Tasks still fail at their deadline, so the error is
logged and startup goes on; one Task that cannot get a new job no longer
stops the others.

Also follows the provisioner task store changes below this PR:
`update_running` takes `now` and `job_token` is no longer optional.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A newer DVLS may send a new optional field to an older Gateway. With
deny_unknown_fields the older Gateway refused the whole request with 400,
so the analysis never ran. Every other Gateway endpoint ignores unknown
fields; this one now does too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…deadline

An attempt started near the 2-hour deadline could keep calling the AI for
30 more minutes, add its log to the recording, and still leave the Task
failed as timed out.

Attempts now start only during the first 2 hours, and the deadline is one
attempt timeout after that, so an attempt started in time can finish and
its result counts. The runner fails a Task that has no time left for a
full attempt as timed out, without running it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ress

`percent` is `done / total`, and `message` is English that clients
translate anyway, so neither belongs in the API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ads it

A recording is remuxed 60 seconds after it ends, and the remux replaces
the video file. An analysis started right after the session ended read
the video before it was remuxed.

The analysis now waits until the job queue stores no remux job for a file
of the session, checking every second, before it reads the recordings.
The job queue lists remux jobs that are scheduled or running, including
those stored before a Gateway restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The working files of a session were reused by any later Task with the
  same model, and the late job of a Task that ended without it, such as one
  failed past its deadline on a status read, could delete the files of a
  newer Task, which then paid for every chunk again. The analysis now names
  its Task as the owner of the files: the attempts of one Task resume from
  them, a new Task starts fresh, and a Task's cleanup leaves another Task's
  files in place.
- An attempt starts only with one minute to spare past its timeout: it ends a
  little later while it stops, and stored deadlines lose their fraction of a
  second. This also keeps the runner test clear of rounding.
- While describing, progress is written whenever another chunk is done, not
  when the percent changes, since the payload no longer has a percent.
- `deadlineAt` says a Task can time out earlier, and the failure reasons are
  listed in full.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Analysing a recording again kept the earlier results: each success added
another `ai-analysis-N.slog` to the manifest, so one recording could hold
several analyses.

A new Task is now a hard restart. Its first attempt removes every earlier
AI analysis result of the session, the file and its manifest entry, even a
succeeded one, and the working files whoever left them. Its later attempts
are job retries and resume from the working files. A recording holds at
most one AI analysis.

The recording manager gains `ClearArtifacts`, which drops the manifest
entries first, so the manifest never lists a file that is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants