Repository navigation
feat(dgw): run AI analysis of a recording as a provisioner task - #2044
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 21 commits into
Conversation
Let maintainers know that an action is required on their side
|
c81d3f1 to
5b16e3a
Compare
5b16e3a to
faf9419
Compare
faf9419 to
55fef54
Compare
55fef54 to
5ff453b
Compare
5ff453b to
f2948c9
Compare
f2948c9 to
633c904
Compare
633c904 to
6083262
Compare
6083262 to
ed522eb
Compare
a5967ae to
0dfb9a1
Compare
The Gateway state now holds the provisioner task store, on its own gateway.db connection, so the work it records can reach it. Building the mock state opens an in-memory database, so `DgwState::mock` is now async. The job queue attempt limit is now the named constant `MAX_ATTEMPTS`, still 5, so a job can refer to it. No behavior change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gateway can now describe what the user did in a recorded session with an AI provider, and add the result to the session as a new log. The analysis itself is the AI crate's `analyze_recording`; Gateway gives it the session manifest once the recording is finished, then adds the log it returns to the session through the recording manager. Provisioner tasks run through a ProvisionerTaskRunner, the one place that keeps a task's record in gateway.db and the job running it on the job queue in step. Each kind of work is a ProvisionerTask: what one attempt does, and what to drop once its record ends. The runner records the Task and queues its job, starts each attempt, cancels and awaits an attempt past its timeout, retries transient failures until the job queue's attempts are used up, and records how the Task ends. The Task is the source of truth and its job only wakes it up: each Task names the one job allowed to run it, so a stale or duplicate job stops without running anything. At startup, before the job runner claims anything, every unfinished Task is handed to a new job, which also covers a Task recorded just before a crash, before its job was queued. AI analysis (`recording-ai-analysis`) is the first kind. A repeated request with the same Task ID gets its Task back, even while the session records again. A new request is refused while the session is recording, or when it has no recording. The API key and the base URL stay in memory only, encrypted, as the Task's secret in the provisioning store, until the Task ends or reaches its deadline. After a restart the attempt finds no key and fails its Task. Neither the key nor the URL appears in errors, logs or stored results: the AI client (#2043) leaves the URL out of its transport errors. Analyses of one session run one at a time, so two attempts never share the working files the AI crate keeps next to the manifest. Those files are removed when the Task ends. Starting an AI analysis is an internal function for now; the HTTP endpoint comes next. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The provisioner can now start an AI analysis of a recorded session with
`POST /jet/jrec/{session_id}/ai-analysis`, and follow its Task with
`GET /jet/jrec/{session_id}/ai-analysis/{task_id}`, which the
Location header of a 202 answer names. A Task of another kind, or the
analysis of another session, is not found under that path.
Each provisioner task declares its own scopes, named after its kind:
`gateway.tasks.<kind>.start` and `gateway.tasks.<kind>.read`. AI
analysis uses `gateway.tasks.recording-ai-analysis.start` and
`gateway.tasks.recording-ai-analysis.read`.
The request body holds the Task ID chosen by the caller, the AI provider
and model, an optional base URL and output token limit, and the API key.
A new Task answers 202, the same request again answers 200 with the same
Task. Gateway answers 409 with a code when the Task ID is used for other
parameters, when the session still records, or when another AI analysis
of the session is not finished; the last one names the Task ID. A
session without recording answers 404, invalid settings or an unknown
field answer 400.
The API key and the base URL stay in memory only: they are never logged,
returned or written to gateway.db.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Task was reported created once its job reached the channel of the job queue task, which gives up after a few failed writes without telling anyone, so a Task could be created without a job. The runner now writes jobs to the job queue database itself, through ProvisionerTaskJobQueue, and only reports a Task created once its job is stored. At startup, a Task whose new job cannot be stored is failed as not queued, instead of waiting for its deadline with no job. A Task is only cleaned up once its end is recorded: when recording a failure fails, the job queue tries the attempt again; when recording a success fails, the Task keeps its key and working files and fails at its deadline, since running the work again would add a second log. Waiting for an earlier AI analysis of the same session now stops when the attempt is cancelled, so the attempt timeout also covers the wait.
At startup, every unfinished Task got a new job, even when the job it had was still stored in the job queue and would run anyway, leaving a stale job behind. The runner now reads the provisioner task jobs the job queue still holds, and only gives a new job to a Task left without one, such as a Task created just before a crash or whose job was dropped. The job queue gains `job_definitions` to list the stored jobs of a name that may still run.
`as_of_v2022_3_0_0` lists what DVLS 2022.3 sends; later scopes such as `gateway.recording.delete` were never added there. The AI analysis scopes are covered by the `ai_analysis` tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`AiSettings` and `AiAccess` (and so `StartRequest`) derived `Debug`, which prints the base URL that must never reach the logs. Nothing logs them today; this keeps it that way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The server only ever sends 0 to 100, as documented. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failure to list or hand over the unfinished Tasks stopped the whole service. Unfinished Tasks still fail at their deadline, so the error is logged and startup goes on; one Task that cannot get a new job no longer stops the others. Also follows the provisioner task store changes below this PR: `update_running` takes `now` and `job_token` is no longer optional. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A newer DVLS may send a new optional field to an older Gateway. With deny_unknown_fields the older Gateway refused the whole request with 400, so the analysis never ran. Every other Gateway endpoint ignores unknown fields; this one now does too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…deadline An attempt started near the 2-hour deadline could keep calling the AI for 30 more minutes, add its log to the recording, and still leave the Task failed as timed out. Attempts now start only during the first 2 hours, and the deadline is one attempt timeout after that, so an attempt started in time can finish and its result counts. The runner fails a Task that has no time left for a full attempt as timed out, without running it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ress `percent` is `done / total`, and `message` is English that clients translate anyway, so neither belongs in the API. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ads it A recording is remuxed 60 seconds after it ends, and the remux replaces the video file. An analysis started right after the session ended read the video before it was remuxed. The analysis now waits until the job queue stores no remux job for a file of the session, checking every second, before it reads the recordings. The job queue lists remux jobs that are scheduled or running, including those stored before a Gateway restart. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The working files of a session were reused by any later Task with the same model, and the late job of a Task that ended without it, such as one failed past its deadline on a status read, could delete the files of a newer Task, which then paid for every chunk again. The analysis now names its Task as the owner of the files: the attempts of one Task resume from them, a new Task starts fresh, and a Task's cleanup leaves another Task's files in place. - An attempt starts only with one minute to spare past its timeout: it ends a little later while it stops, and stored deadlines lose their fraction of a second. This also keeps the runner test clear of rounding. - While describing, progress is written whenever another chunk is done, not when the percent changes, since the payload no longer has a percent. - `deadlineAt` says a Task can time out earlier, and the failure reasons are listed in full. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Analysing a recording again kept the earlier results: each success added another `ai-analysis-N.slog` to the manifest, so one recording could hold several analyses. A new Task is now a hard restart. Its first attempt removes every earlier AI analysis result of the session, the file and its manifest entry, even a succeeded one, and the working files whoever left them. Its later attempts are job retries and resume from the working files. A recording holds at most one AI analysis. The recording manager gains `ClearArtifacts`, which drops the manifest entries first, so the manifest never lists a file that is gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
0dfb9a1 to
1803230
Compare
AI analysis of a recording, started by DVLS with a plain HTTP request (no new token type), tracked as a provisioner Task and run as a job on the existing job queue.
Provisioner tasks run through a
ProvisionerTaskRunner, the one place that keeps a task's record (#2041'sProvisionerTaskRecord) and its job in step. Each kind of work is aProvisionerTask(what one attempt does, what to drop once its record ends); the runner creates the record, queues its job, runs each attempt with its timeout, retries transient failures and records how the Task ends. The Task is the source of truth and the job only wakes it up: each Task names the one job allowed to run it (#2041), so a stale or duplicate job does nothing. A Task is reported created only once its job is stored in the job queue. At startup, before the job runner claims anything, an unfinished Task whose job is no longer stored (such as after a crash between recording the Task and storing its job) gets a new one; a Task whose job is still stored keeps it. The analysis itself is the AI crate'sanalyze_recording(#2043); Gateway hands it the manifest of the finished recording and adds the log it returns to the session.API (stable, no
enable_unstable)POST /jet/jrec/{session_id}/ai-analysis, scopegateway.tasks.recording-ai-analysis.start. Body:taskId,provider,model,baseUrl(needed for openai-compatible),maxOutputTokens,apiKey.task_id_conflict/recording_active/analysis_in_progress(+ the runningtaskId)GET /jet/jrec/{session_id}/ai-analysis/{task_id}, scopegateway.tasks.recording-ai-analysis.read: the Task (state, payload, attempts, times), or 404. The 202 answer of thePOSTnames it in itsLocationheader. A Task of another kind, or the analysis of another session, is not found under that path.Each provisioner task declares its own scopes,
gateway.tasks.<kind>.startandgateway.tasks.<kind>.read, so a token for one kind of Task can neither start nor read another kind. The kind of AI analysis isrecording-ai-analysis.The two new scopes in
Devolutions.Gateway.Utilsand the regenerated OpenAPI clients are in #2054, stacked on this PR.The first commit hands the provisioner task store (#2041) to
DgwStateon its owngateway.dbconnection;DgwState::mockbecomes async for that, hence the.awaitin a few tests. It also passesjob_queue::MAX_ATTEMPTS(5, same as the current default) to the job queue explicitly, so the retry check in the AI analysis job can't drift from it.How it behaves
gateway.dbor our logs. After a Gateway restart the Task fails with "the API key is no longer available"..cast,.trp) and video (.webm) recordings are analyzed (feat(dgw): analyze any recording in the AI crate #2043); video needs XMFv2026.9.25.0or later. The result is a newai-analysis-N.slogin the session'sartifacts, and the Task payload says which file, the model, the prompt versions and the token usage.Where to look hardest
provisioner_tasks/runner/mod.rs:run_attempt(cancel on timeout, retry vs. fail,on_endon every end) andresume(called fromservice.rsbefore the job runner starts)provisioner_tasks/recording_ai_analysis/mod.rs: the AIProvisionerTask, andlock_session: the working files belong to the session, so analyses of one session run one at a time (a Task past its deadline may still have an attempt running when the next Task starts)start()in the same file: the order of the checksKnown, accepted
All, reqwest/hyper's own TRACE lines show the provider's address (not the key). Our code never logs it..slogin, the manifest can point at an empty file. Closing that needs a change toadd_artifact.Tested: devolutions-gateway lib 218 (incl. the runner against a test task: stale jobs after resume, a job that cannot be queued, panics, invalid parameters and unknown kinds; and the AI analysis job against a mock provider: retry resumes at the first chunk left, permanent and exhausted errors, timeout, deadline, lost key, one analysis per session, leftover files removed),
tests/ai_analysis.rs8/8. End to end on the real Gateway binary (127.0.0.1), before the move toProvisionerTaskRunnerand not re-run since with XMF and GPT-6 Luna on a real WebM session:POST→ the first attempt hit a connection reset and was retried 30 s later, resuming from the saved screenshots →succeeded,ai-analysis-0.slogin the manifest, working files gone, the API key nowhere in Gateway's logs or data folder.🤖 Generated with Claude Code