feat(mobile): transcribe voice input through the Kilo gateway by iscekic · Pull Request #6032 · Kilo-Org/cloud

iscekic · 2026-09-10T00:43:18Z

Changelog for users

Voice input gains a second engine through the Kilo gateway; a Gateway transcription switch in Preferences is off by default, so voice input behaves exactly as today and never calls the gateway.
The switch is a two-way choice, not a fallback: off runs the operating system's speech recogniser, on runs the Kilo gateway. The chosen engine owns the whole dictation and the other engine is never called.
The mode applies to every voice dictation in the app.
With the switch on, the gateway transcribes with your chosen model; if you have not chosen one, the first model the gateway offers is the default.
A picker lists the gateway's actual models, and the chosen model persists across launches.
An unreachable gateway, an unavailable model, no speech, and a timeout each get their own message; an unavailable model and no model open the transcription settings.
A gateway dictation that times out shows the retryable timeout instead of an unexpected-response error, keeps the sealed recording, and shows the retry message.
While the gateway transcribes, the composer shows Transcribing... and the microphone button cancels the upload; a transcript that lands right after you tap stop fills the composer.
Multipart voice transcriptions enforce the organization's provider allow-list and data-collection policy.
Toasts now appear at the bottom of the screen, and tab-bar navigation away from an unsent draft no longer triggers the discard prompt.

Changelog for maintainers

apps/mobile/src/lib/voice-input/voice-input-engine-select.ts — the selector composes the OS recogniser and the gateway engine behind the single VoiceInputNative the controller consumes. It reads the chosen engine at start(), routes start/stop/abort and every capability probe to that one engine, and drops events from the idle engine. No engine ever hands a session to the other. Review this file first.
apps/mobile/src/lib/voice-input/voice-input-engine-mode.ts — resolveVoiceInputEngineName(enabled) maps the switch to os or gateway. The former gateway-primary/device-primary modes and the VoiceInputEngineMode union are gone.
apps/mobile/src/lib/voice-input/native-voice-input.ts — createSelectingVoiceInputNative replaces the two-engine dispatcher; the gateway-enabled subscription still refreshes availability so enabling the switch surfaces the mic without a restart.
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-preference.ts — two SecureStore keys remain (enabled default off, chosen model). The gateway-transcription-primary key and its hooks are removed.
apps/mobile/src/lib/voice-input/gateway/native-gateway-voice-input.ts:22 — resolveGatewayTranscriptionModelId() returns the stored model, else the first catalogue entry, else null. An unreachable or empty catalogue reads as no model, which surfaces the actionable picker message instead of an upload that must fail. The engine's readModelId dependency is now async.
apps/mobile/src/lib/voice-input/use-voice-input-actions.ts — the start path branches once: switch on starts the gateway engine with no OS consent disclosure (the switch is the consent); switch off runs the existing OS consent flow. The model precondition and the picker alert on start are gone because the engine now resolves the default. VOICE_INPUT_TOAST_ID is a module-private constant shared by every voice-input toast, so sonner replaces messages in place; check:unused reports no unused exports, no checker ignore.
Removed with the fallback: the engine-fell-back controller event and its listener, classifyVoiceInputEngineFallback, the voice-engines-failed error case and its copy, the tone: 'info' feedback field, the Preferences "Use as primary" row, and the voice-input-engine-dispatch* module plus its five suites. New coverage is voice-input-engine-select.test.ts.
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-client.ts:133 — a cancelled upload that resolves with no response is classified by checking the timeout signal first: a fired timeout returns the retryable timeout, while a caller abort stays an invalid response.
apps/web/src/app/api/openrouter/audio/transcriptions/route.ts:278 — the resolved organization provider policy is computed once and appended to the multipart form as a JSON provider field (the JSON branch merges it as before); multipart requests also carry safety_identifier and user, both the provider-specific hash.
Wire contract: recordings upload as multipart with a 30 s timeout; responses validate as { text: string }; failures classify into distinct codes that map to one copy each. The gateway route accepts multipart uploads and exempts free models from the zero-balance gate; the catalogue fetch is uncached; the feature value is mobile-voice-input. The fake gateway keeps exactly its two committed transcription models and a request counter; no fixture, test hook, or test-only flag is committed. The feature-flags mounted test mocks the gateway transcription preference module, whose import-time Sentry chain cannot resolve under vitest.

Owner manual verification

The acoustic microphone-to-transcript checks are skipped, pending owner verification on a real device. No sound was generated or played for microphone pickup. No spoken-utterance transcription through either engine was automated or asserted.

E2E proof

The device captures for the Preferences rows, persistence across launch, the fake-gateway round trip, and the switch-off no-call check remain valid. The prior "use as primary" and fallback hand-off captures are superseded by this change and are not part of the current proof.

Owner request

Support transcribing voice input through the Kilo gateway as well as the operating system's speech recognition.

Today voice input is on-device only. The gateway adds a second engine.

Add a setting with two parts:

A switch that enables gateway transcription. It is OFF by default.

A model picker for the transcription model.

The switch picks the engine for every dictation in the app: off keeps the operating system's recogniser, on sends the recording to the Kilo gateway. There is no fallback between the two engines — the chosen engine owns the whole dictation.

When gateway transcription is off, behaviour is exactly what it is today: on-device only, and no gateway call is made.

If the user has not chosen a model, the first model the gateway offers is the default.

Cover the failure cases the user can hit: the gateway is unreachable, the request times out, the chosen model is unavailable, and the recording holds no speech. Each gets its own message and its own remedy.

The switch and the chosen model must persist across launches, and the model picker must list the models the gateway actually offers, not a hardcoded list.

Every verification and publication gate must pass. The acoustic microphone-to-transcript conversion — a real utterance becoming a transcript through either engine — awaits owner manual verification by Igor on a real device: automated proof never generates or plays sound for microphone pickup, never depends on the acoustic path, and never uses simulator speech recognition diagnosis that serves only that proof; no acoustic scenario blocks the section and no repair task chases an acoustic failure; the PR body states that the acoustic microphone-to-transcript check awaits owner manual verification and does not claim it passed.

Everything else keeps its automated proof and runs as normal: the recording → upload → response round trip against the fake gateway, whose fixed transcript needs no spoken audio; the failure UX messages; the switch-off path, which starts and ends a device session and makes no gateway call, asserted on the fake server's request counter; and the unit suites covering the engine selector, the request timeout, and persistence.

The no-speech state is proved by the same automated device run, but its empty-transcript engine is temporary harness support that lives outside the product diff, per the owner rule of 2026-09-08: no fixture, E2E-only code, test hook, or test-only runtime flag for it is committed to the product repo, and there is no owner permission for a committed fake-transcribe-silent model. The harness seeds the empty-transcript behaviour at run time in front of the delivered fake gateway, which keeps exactly its two committed transcription models (the fixed transcript and the broken one); ordinary unit tests are unaffected.

The mobile static gate is part of the pass set: check:unused (knip) must report no unused exports, repaired at the source with runtime behavior preserved, never suppressed through a checker ignore comment or config exemption.

E2E proof — log excerpts

[p1] Dictate a voice message while the gateway stalls past the 30s timeout: mic  -> pass :: owner-manual: acoustic spoken-utterance -> transcript conversion through either engine (not exercised; no sound generated); the automatic timeout classification passed live — p1-timeout.log carries 'fault.sh: stalled pids [59771] for 120s' then 'SCENE p1 OK', i.e. the final assert 'Transcription took too long. Try again.' (voiceInput.gatewayTimeout) rendered, not voiceInput.stopped; supporting focused tests in p1-focused-tests.log (26 passed).
[p2] Enterprise org member multipart transcription through /api/gateway/audio/tr -> pass :: p2-upstream-fields.log carries 'gateway response: HTTP 200 {"text":"Gateway transcription online"}', 'upstream provider = {"only":["openai"],"data_collection":"deny"}' and 'upstream safety_identifier = ...' equal to 'upstream user'; focused route tests 'forwards the organization provider policy on multipart requests' and 'attaches the safety identifier to multipart upstream requests' pass in p2-focused-tests.log.

.../e2e-mobile-app/p1-timeout.log

fault.sh: stalled pids [59771] for 120s
SCENE p1 OK
scenario steps (file p1.json, packed tree HEAD f202d33bc):
  ["tap Start voice input","assert Listening... (10s)","tap Stop voice input","assert Transcribing... (10s)","assert Transcription took too long. Try again. (50s)"]
setup while the composer is open: fault.sh <worktree> stall nextjs --hold 120 (nextjs suspended; port 3300 bound but not answering).
observed: "SCENE p1 OK" -> every assert passed, including the final one, so the mic sheet rendered the timeout copy.
expected state: retryable timeout. voiceInput.gatewayTimeout = "Transcription took too long. Try again."
contrast: gateway-invalid-response maps to voiceInput.stopped = "Voice input stopped. Tap the microphone to try again." (not observed).
owner-manual (not exercised): the acoustic spoken-utterance -> transcript conversion through either engine; no sound was generated or played.

.../e2e-mobile-app/p2-upstream-fields.log

p2 live check: enterprise org member multipart transcription through /api/gateway/audio/transcriptions
platform: ios (device 2E687210-E477-4D23-94F1-9736C860ACD8); request issued against the local stack (nextjs :3300)
org: 0605a00e-dfb4-4361-946d-b81cca2e02bd plan=enterprise provider_allow_list=["openai"] data_collection=deny
user: fe46cbb5-c197-42c4-ae0f-02a1dd8fc05c (e2e-mobile-kwf-fix-review-ae2c-ios@example.com, role=member)
gateway response: HTTP 200 {"text":"Gateway transcription online"}
upstream request: POST /api/openrouter/audio/transcriptions multipart
upstream model = fake-transcribe
upstream provider = {"only":["openai"],"data_collection":"deny"}
upstream safety_identifier = QJyylBTUbBUKZ2cAKnCpqZAXmtkmZP741isDYxuRLGg=
upstream user = QJyylBTUbBUKZ2cAKnCpqZAXmtkmZP741isDYxuRLGg=

…vice fallback #6032

kilo-code-bot · 2026-09-10T05:53:34Z

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Incremental review of the changes since 4a9ff31 (engine-selection rewrite plus the recording-cleanup, org-scoped-catalogue, and malformed-multipart fixes). All previously raised findings are addressed in current HEAD; no new issues found in the changed code.

Files Reviewed (18 files)

apps/mobile/src/components/app-root-providers.tsx
apps/mobile/src/components/preferences-screen.tsx
apps/mobile/src/components/transcription-model-picker-sheet.tsx
apps/mobile/src/lib/storage-keys.ts
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-preference.ts
apps/mobile/src/lib/voice-input/gateway/gateway-voice-input-engine.ts
apps/mobile/src/lib/voice-input/gateway/native-gateway-voice-input.ts
apps/mobile/src/lib/voice-input/native-voice-input.ts
apps/mobile/src/lib/voice-input/use-voice-input-actions.ts
apps/mobile/src/lib/voice-input/voice-input-controller.ts
apps/mobile/src/lib/voice-input/voice-input-engine-mode.ts
apps/mobile/src/lib/voice-input/voice-input-engine-select.ts
apps/mobile/src/lib/voice-input/voice-input-feedback.ts
apps/mobile/src/lib/voice-input/voice-input-listeners.ts
apps/mobile/src/lib/voice-input/voice-input-state.ts
apps/web/src/app/api/openrouter/audio/transcriptions/route.ts
apps/mobile/vitest.mounted.config.ts
apps/mobile/vitest.pure.config.ts

Previous Review Summaries (2 snapshots, latest commit 4a9ff31)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit `4a9ff31`)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (38 files)

apps/mobile/package.json
apps/mobile/src/app/(app)/_layout.tsx
apps/mobile/src/app/(app)/transcription-model-picker.tsx
apps/mobile/src/components/agents/chat-composer.tsx
apps/mobile/src/components/agents/use-new-session-discard-guard.ts
apps/mobile/src/components/app-root-providers.tsx
apps/mobile/src/components/kilo-pass/kilo-pass-subscription-card.tsx
apps/mobile/src/components/preferences-screen.tsx
apps/mobile/src/components/profile-credits-card.tsx
apps/mobile/src/components/transcription-model-picker-sheet.tsx
apps/mobile/src/components/ui/preference-row.tsx
apps/mobile/src/components/voice-input-control.tsx
apps/mobile/src/i18n/locales/en.json
apps/mobile/src/lib/hooks/use-transcription-models.ts
apps/mobile/src/lib/storage-keys.ts
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-client.ts
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-preference.ts
apps/mobile/src/lib/voice-input/gateway/gateway-voice-input-engine.ts
apps/mobile/src/lib/voice-input/gateway/native-gateway-voice-input.ts
apps/mobile/src/lib/voice-input/native-voice-input.ts
apps/mobile/src/lib/voice-input/use-voice-input-actions.ts
apps/mobile/src/lib/voice-input/voice-input-controller.ts
apps/mobile/src/lib/voice-input/voice-input-engine-dispatch.ts
apps/mobile/src/lib/voice-input/voice-input-engine-mode.ts
apps/mobile/src/lib/voice-input/voice-input-feedback.ts
apps/mobile/src/lib/voice-input/voice-input-listeners.ts
apps/mobile/src/lib/voice-input/voice-input-state.ts
apps/mobile/src/lib/voice-input/voice-input-view-state.ts
apps/web/src/app/api/openrouter/audio/transcriptions/route.ts
apps/web/src/lib/ai-gateway/is-free-model.ts
apps/web/src/lib/ai-gateway/local-fake-llm.ts
apps/web/src/lib/ai-gateway/providers/get-provider.ts
apps/web/src/lib/ai-gateway/providers/openrouter-definition.ts
apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
apps/web/src/lib/ai-gateway/providers/provider-definitions.ts
apps/web/src/lib/feature-detection.ts
services/cloud-agent-next/test/e2e/fake-llm-server.ts
tools/i18n/check-catalogs.mjs

Previous review (commit `8ddd98a`)

Status: 3 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	3
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
`apps/mobile/src/lib/voice-input/gateway/gateway-transcription-client.ts`	133	Timeout cancel classified as invalid-response
`apps/web/src/app/api/openrouter/audio/transcriptions/route.ts`	278	Org provider policy dropped on multipart
`apps/web/src/app/api/openrouter/audio/transcriptions/route.ts`	294	Multipart upstream omits safety_identifier

Files Reviewed (24 files)

apps/mobile/src/lib/voice-input/gateway/gateway-transcription-client.ts - 1 issue
apps/web/src/app/api/openrouter/audio/transcriptions/route.ts - 2 issues
apps/mobile/src/lib/voice-input/voice-input-engine-dispatch.ts
apps/mobile/src/lib/voice-input/gateway/gateway-voice-input-engine.ts
apps/mobile/src/lib/voice-input/gateway/native-gateway-voice-input.ts
apps/mobile/src/lib/voice-input/gateway/gateway-transcription-preference.ts
apps/mobile/src/lib/voice-input/use-voice-input-actions.ts
apps/mobile/src/lib/voice-input/voice-input-controller.ts
apps/mobile/src/lib/voice-input/voice-input-state.ts
apps/mobile/src/lib/voice-input/voice-input-listeners.ts
apps/mobile/src/lib/voice-input/voice-input-view-state.ts
apps/mobile/src/lib/voice-input/voice-input-engine-mode.ts
apps/mobile/src/lib/voice-input/native-voice-input.ts
apps/mobile/src/components/preferences-screen.tsx
apps/mobile/src/components/agents/chat-composer.tsx
apps/mobile/src/components/agents/use-new-session-discard-guard.ts
apps/mobile/src/components/voice-input-control.tsx
apps/mobile/src/lib/hooks/use-transcription-models.ts
apps/web/src/lib/ai-gateway/is-free-model.ts
apps/web/src/lib/ai-gateway/local-fake-llm.ts
apps/web/src/lib/ai-gateway/providers/get-provider.ts
apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
services/cloud-agent-next/test/e2e/fake-llm-server.ts
apps/mobile/src/i18n/locales/en.json

Fix these issues in Kilo Cloud

_{Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0}

_{Review guidance: REVIEW.md from base branch main}

…vice fallback #6032

Replace the primary/fallback dispatcher with a selector. The gateway switch now chooses the only engine that runs: off is the OS recogniser, on is the Kilo gateway. No engine ever hands a session to the other. - Default the gateway model to the first catalogue entry when the user has not stored a choice. - Drop the "Use as primary" preference, the engine-fell-back event, the both-engines copy, and the transcribing hand-off tone. - Remove the dispatch machine and its fallback suites; cover the selector instead.

…ing-voice-i-6a1b # Conflicts: # apps/mobile/src/components/agents/chat-composer.tsx

request.formData() rejects on a malformed body or missing boundary. The rejection escaped the route as an unhandled 500. Catch it and return null so POST answers the controlled 400 invalid-request response.

The default model and the picker read the catalogue unscoped, then the upload carried the organization header. The first global model can be unavailable to the organization, so the default could pick a model the scoped upload rejects. Read the stored organization id and pass it to fetchTranscriptionModels, and pass it from the picker into useTranscriptionModels.

Recorder.release() frees the native object, not the recording file, so every gateway dictation left an m4a behind. Inject a best-effort deleteRecording dependency and run it once the URI is known on every terminal path: upload success, classified failure, model/auth short-circuit, stale session, and abort during recording or upload. Tests cover success, failure, and both aborts.

iscekic marked this pull request as draft September 10, 2026 00:43

iscekic added a commit that referenced this pull request Sep 10, 2026

feat(mobile): transcribe voice input through the Kilo gateway with de…

d82ce82

…vice fallback #6032

iscekic force-pushed the kwf/support-transcribing-voice-i-6a1b branch from 54d7121 to d82ce82 Compare September 10, 2026 01:31

iscekic added a commit that referenced this pull request Sep 10, 2026

feat(mobile): transcribe voice input through the Kilo gateway with de…

34a38ad

…vice fallback #6032

iscekic force-pushed the kwf/support-transcribing-voice-i-6a1b branch from d82ce82 to 34a38ad Compare September 10, 2026 03:10

iscekic added a commit that referenced this pull request Sep 10, 2026

feat(mobile): transcribe voice input through the Kilo gateway with de…

8ddd98a

…vice fallback #6032

iscekic force-pushed the kwf/support-transcribing-voice-i-6a1b branch from 34a38ad to 8ddd98a Compare September 10, 2026 05:16

iscekic marked this pull request as ready for review September 10, 2026 05:27

kilo-code-bot Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread apps/mobile/src/lib/voice-input/gateway/gateway-transcription-client.ts Outdated

Comment thread apps/web/src/app/api/openrouter/audio/transcriptions/route.ts Outdated

Comment thread apps/web/src/app/api/openrouter/audio/transcriptions/route.ts

iscekic marked this pull request as draft September 10, 2026 05:54

feat(mobile): transcribe voice input through the Kilo gateway with de…

4a9ff31

…vice fallback #6032

iscekic force-pushed the kwf/support-transcribing-voice-i-6a1b branch from f202d33 to 4a9ff31 Compare September 10, 2026 11:48

iscekic marked this pull request as ready for review September 10, 2026 12:03

iscekic added the human-ready The PR is ready for human review. label Sep 10, 2026

iscekic requested a review from eshurakov September 10, 2026 12:45

iscekic self-assigned this Sep 10, 2026

iscekic added 2 commits September 10, 2026 15:25

Merge remote-tracking branch 'origin/main' into kwf/support-transcrib…

db17a27

…ing-voice-i-6a1b # Conflicts: # apps/mobile/src/components/agents/chat-composer.tsx

iscekic changed the title ~~feat(mobile): transcribe voice input through the Kilo gateway with device fallback~~ feat(mobile): transcribe voice input through the Kilo gateway Sep 10, 2026

Merge branch 'main' into kwf/support-transcribing-voice-i-6a1b

9f3387d

pandemicsyn approved these changes Sep 10, 2026

View reviewed changes

Comment thread apps/mobile/src/lib/voice-input/gateway/gateway-voice-input-engine.ts

Comment thread apps/mobile/src/lib/voice-input/gateway/native-gateway-voice-input.ts Outdated

Comment thread apps/web/src/app/api/openrouter/audio/transcriptions/route.ts Outdated

iscekic added 3 commits September 10, 2026 19:39

fix(web): answer 400 for a malformed multipart transcription body

7a60c11

request.formData() rejects on a malformed body or missing boundary. The rejection escaped the route as an unhandled 500. Catch it and return null so POST answers the controlled 400 invalid-request response.

Navigation Menu

Uh oh!

feat(mobile): transcribe voice input through the Kilo gateway - #6032

feat(mobile): transcribe voice input through the Kilo gateway#6032
iscekic wants to merge 7 commits into
mainfrom
kwf/support-transcribing-voice-i-6a1b

iscekic commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kilo-code-bot Bot commented Sep 10, 2026 •

edited

Loading

Previous review (commit `4a9ff31`)

Previous review (commit `8ddd98a`)

Overview

WARNING

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

iscekic commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog for users

Changelog for maintainers

Owner manual verification

E2E proof

E2E proof — log excerpts

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kilo-code-bot Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Previous review (commit 4a9ff31)

Previous review (commit 8ddd98a)

Overview

WARNING

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

iscekic commented Sep 10, 2026 •

edited

Loading

kilo-code-bot Bot commented Sep 10, 2026 •

edited

Loading

Previous review (commit `4a9ff31`)

Previous review (commit `8ddd98a`)