Fix transcription reliability: whisper deploy, Parakeet batch resilience, and FluidAudio build

[turtleisaac]

Fix transcription reliability: whisper deploy, Parakeet batch resilience, and FluidAudio build

Summary

Three related fixes that make on-device transcription work end-to-end again. Each is an independent, reviewable commit.

1. Patcher never deployed whisper-transcriber → the social-captions Whisper engines report "not found".
2. Parakeet aborts the entire batch when one input has no decodable audio (issue Splicekit ERROR COREAUDIO - M3MAX #78) → timelines that mix stills/graphics/PDFs with audio produce zero words.
3. parakeet-transcriber no longer compiles against any published FluidAudio release → the tool can't be built from a clean checkout.

All three were verified locally on FCP 11.1 / SpliceKit 3.3.8 (Apple Silicon, macOS 15.6).

---

1. Patcher deploys whisper-transcriber (104fd0d)

patcher/SpliceKit/Models/PatcherModel.swift staged and deployed only silence-detector and parakeet-transcriber; deployTools() had no whisperBin parameter. whisper-transcriber ships in the patcher app bundle and the v3.3.8 notes say it's included so the Whisper large-v3 / large-v3-turbo caption engines "actually work" — but the deploy path never copied it into ~/Applications/SpliceKit/tools/.

Result: a clean patch reports success, yet the captions panel fails with:

Whisper large-v3 transcriber not found. Re-run the SpliceKit patcher, or pick a different engine.

Re-running never helps — every run repeats the same incomplete tool set.

Fix: stage whisper-transcriber in both the full-patch and update paths (mirroring the existing parakeet/silence staging), add a whisperBin parameter to deployTools(), and emit a loud WARNING when it's absent so the failure is visible in the patch log instead of silently degrading transcription.

Note: the separate patcher/SpliceKitPatcher/main.swift target already copies whisper correctly — only the shipped SwiftUI patcher (PatcherModel.swift) was missing it.

2. Parakeet batch resilience (d0900d1) — fixes #78

In tools/parakeet-transcriber/Sources/main.swift, the Phase-1 batch loop transcribed each file with a bare try await manager.transcribe(...) and no per-file error handling. When one entry can't be decoded as audio — a still image, a PDF, or a clip with no audio track — AVFoundation throws kAudioFileInvalidFileError / kAudioFileUnsupportedFileTypeError (surfaced as com.apple.coreaudio.avfaudio error 1685348671 / 1954115647). That error propagated out of the loop to the outer catch, set exitCode = 1, and produced zero words for the entire batch.

Timelines routinely mix stills/graphics with audio clips, so a single non-audio item silently broke transcription for the whole timeline. This is issue #78 — both avfaudio subcodes are the same root cause.

Fix: wrap the per-file transcribe in do/catch (mirroring the diarization phase, which already tolerates per-file failures). A failing file is recorded, logged (Skipping <file>: <reason>), and skipped so the remaining files still transcribe. Batch results report each skipped file's real error instead of a blanket "File not found".

3. Build against current FluidAudio (57b1ef0)

parakeet-transcriber did not compile against any published FluidAudio release: main.swift uses .tdtCtc110m (added in FluidAudio 0.13.7) together with AsrManager.transcribe(_:source:) (removed after 0.13.0), so no single version satisfies both. The floating from: "0.12.0" masked this by resolving to whatever 0.x was newest.

Fix:

• Package.swift: pin FluidAudio to .upToNextMinor(from: "0.15.6") so the pre-1.0 dependency can't silently pull a version with breaking API changes.
• main.swift: migrate the one incompatible call from transcribe(fileURL, source: .system) to the current transcribe(fileURL, decoderState: &state) API, using a fresh TdtDecoderState per batch entry (independent clips, no streaming context).

(0.15.6 is compatible with the entire file — .tdtCtc110m, .v2/.v3, model loading, diarizer — except that single call. No features dropped.)

---

Verification

• Patcher (Injecting as an internal plugin? #1): swiftc -typecheck of all patcher app sources against the built Sentry/Sparkle frameworks → 0 errors.
• Parakeet (Timeline Batch Export? #2, Signing failed #3): swift build -c release against FluidAudio 0.15.6 → succeeds. Running a batch of [non-audio .png, spoken .aiff]:
  • the .png throws avfaudio error 1954115647 and is skipped (logged, recorded), instead of aborting the batch;
  • the audio transcribes to accurate word-level output — "The quick brown fox jumps over the lazy dog." with per-word timings.

Notes for reviewers

• Commits are independent; Injecting as an internal plugin? #1 (patcher) is unrelated to Timeline Batch Export? #2/Signing failed #3 (the transcriber tool) and can be taken separately. Timeline Batch Export? #2 and Signing failed #3 are coupled — Signing failed #3 is what lets the tool (and therefore Timeline Batch Export? #2) build at all.
• Package.resolved is gitignored per repo convention, so it's not included.

---

Summary by cubic

Restores end-to-end on-device transcription by deploying Whisper, making Parakeet batch-safe, and pinning FluidAudio for a reliable build. Previously the patcher omitted whisper-transcriber, Parakeet aborted on a single non-audio file, and parakeet-transcriber could not compile; now Whisper is deployed with a clear warning if missing, Parakeet skips and logs bad files, and the tool builds against FluidAudio 0.15.6 using the decoder-state API.

• Patcher: stage and deploy whisper-transcriber in both full and update paths; add whisperBin to deployTools(); log a WARNING when Whisper is absent.

• Parakeet: wrap per-file transcribe in do/catch, log “Skipping : ”, and include the real per-file error in batch results instead of aborting the batch (fixes Splicekit ERROR COREAUDIO - M3MAX #78).

• Build: pin FluidAudio to .upToNextMinor(from: "0.15.6") and migrate to transcribe(fileURL, decoderState:&state) with a fresh TdtDecoderState per file.

• Rollout: Users must re-run the SwiftUI patcher once to deploy whisper-transcriber into ~/Applications/SpliceKit/tools/. No config or data migrations.

^{Written for commit 57b1ef0. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tools/parakeet-transcriber/Sources/main.swift">

<violation number="1" location="tools/parakeet-transcriber/Sources/main.swift:310">
P1: When single-file transcription fails, this catch swallows the error and the process exits successfully with an empty JSON array. Re-throw failures when `batchMode` is false so the existing outer handler reports the error and returns a nonzero exit status.</violation>
</file>

<file name="patcher/SpliceKit/Models/PatcherModel.swift">

<violation number="1" location="patcher/SpliceKit/Models/PatcherModel.swift:903">
P2: After a previous run leaves `whisper-transcriber` in the shared temporary build directory, a later run with no bundled Whisper silently redeploys that obsolete binary. Remove the Whisper artifact before the conditional copy in both full-patch and update paths.</violation>

<violation number="2" location="patcher/SpliceKit/Models/PatcherModel.swift:904">
P1: When the app bundle contains the source fallback, this `cp` fails because `bundledWhisper` is a directory, so full installs and updates do not stage Whisper. Build the binary before copying it, or package only an executable for this patcher path.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When single-file transcription fails, this catch swallows the error and the process exits successfully with an empty JSON array. Re-throw failures when batchMode is false so the existing outer handler reports the error and returns a nonzero exit status.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tools/parakeet-transcriber/Sources/main.swift, line 310:

<comment>When single-file transcription fails, this catch swallows the error and the process exits successfully with an empty JSON array. Re-throw failures when `batchMode` is false so the existing outer handler reports the error and returns a nonzero exit status.</comment>

<file context>
@@ -299,8 +300,22 @@ Task {
+                var decoderState = try TdtDecoderState()
+                let result = try await manager.transcribe(fileURL, decoderState: &decoderState)
+                asrResults.append((index: index, file: entry.file, result: result))
+            } catch {
+                // A single undecodable input (a still image, PDF, or a clip with no
+                // audio track — e.g. AVFoundation kAudioFileInvalidFileError /
</file context>

Suggested change

	} catch {
	} catch {
	if !batchMode { throw error }

[cubic-dev-ai]

P1: When the app bundle contains the source fallback, this cp fails because bundledWhisper is a directory, so full installs and updates do not stage Whisper. Build the binary before copying it, or package only an executable for this patcher path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At patcher/SpliceKit/Models/PatcherModel.swift, line 904:

<comment>When the app bundle contains the source fallback, this `cp` fails because `bundledWhisper` is a directory, so full installs and updates do not stage Whisper. Build the binary before copying it, or package only an executable for this patcher path.</comment>

<file context>
@@ -889,6 +898,12 @@ class PatcherModel: ObservableObject {
+        let whisperBin = buildDir + "/whisper-transcriber"
+        let bundledWhisper = (Bundle.main.resourcePath ?? "") + "/tools/whisper-transcriber"
+        if FileManager.default.fileExists(atPath: bundledWhisper) {
+            shell("cp '\(bundledWhisper)' '\(whisperBin)'")
+        }
+
</file context>

[cubic-dev-ai]

P2: After a previous run leaves whisper-transcriber in the shared temporary build directory, a later run with no bundled Whisper silently redeploys that obsolete binary. Remove the Whisper artifact before the conditional copy in both full-patch and update paths.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At patcher/SpliceKit/Models/PatcherModel.swift, line 903:

<comment>After a previous run leaves `whisper-transcriber` in the shared temporary build directory, a later run with no bundled Whisper silently redeploys that obsolete binary. Remove the Whisper artifact before the conditional copy in both full-patch and update paths.</comment>

<file context>
@@ -889,6 +898,12 @@ class PatcherModel: ObservableObject {
 
+        let whisperBin = buildDir + "/whisper-transcriber"
+        let bundledWhisper = (Bundle.main.resourcePath ?? "") + "/tools/whisper-transcriber"
+        if FileManager.default.fileExists(atPath: bundledWhisper) {
+            shell("cp '\(bundledWhisper)' '\(whisperBin)'")
+        }
</file context>