Add a Core AI backend and a Private Cloud Compute model - #218
Conversation
LanguageModelSession appends the prompt entry to its transcript before invoking the model. The Foundation Models adapter then passes that transcript to FoundationModels.LanguageModelSession and also sends the prompt through respond/streamResponse, which appends it again, so every FM-backed turn reaches the model with the final user message duplicated. The MLX adapter shows the intended contract: it renders the chat from session.transcript alone and treats the prompt parameter as a fallback. FM's API requires passing the prompt separately, so the fix on this adapter is to drop a trailing transcript prompt that matches the outgoing prompt before converting. Verified by decoding the rendered prompt of a custom LanguageModel backend on OS 27, where the doubled turn is directly observable; the same transcript construction applies to SystemLanguageModel on OS 26. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds two Foundation Models protocol backends behind the OS 27 session machinery: - CoreAILanguageModel (new CoreAI trait): runs Core AI .aimodel bundles exported with apple/coreai-models recipes through the new LanguageModelSession(model:) construction, mirroring the MLX and Llama trait wiring. Verified on device (iPhone 17 Pro, iOS 27 beta) with Qwen3 and SmolLM2 exports: streaming, multi-turn transcripts, and the executor's tool-calling path. - PrivateCloudComputeLanguageModel: wraps the OS 27 FoundationModels.PrivateCloudComputeLanguageModel (availability, quota, vision/tool capabilities). Requires the com.apple.developer.private-cloud-compute entitlement, granted on request for apps under two million users. Verified on device including tool calling. To share the session plumbing, the Foundation Models adapter's respond/streamResponse bodies move verbatim into file-scope fmRespond/fmStreamResponse helpers parameterized by an async session factory; SystemLanguageModel, the Core AI backend, and the PCC backend all delegate to them. Streamed and returned text also drops leading newlines, which OS 27 executors emit when they strip an empty reasoning block from the output. The CoreAI trait depends on a fork of apple/coreai-models because the upstream package cannot be consumed yet by lower-deployment-target apps or built for the simulator, and its marker detection breaks unk-token tokenizers. All three fixes are offered upstream; this PR stays a draft until they land and the dependency can point at apple/coreai-models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks, @james-333i. I've marked this as a draft until the two apple/coreai-models fixes land, as your description says. I'll read through the shape in the meantime. |
|
@mattt Since filing this I have found a method that removes the dependency on apple/coreai-models entirely, so I would like to restructure the PR before you spend time on the current one. iOS 27's FoundationModels defines a public LanguageModel protocol, and LanguageModelSession.init(model:) accepts any conformer. The Core AI models in apple/coreai-models already conform to it and so does Private Cloud Compute. Reading this PR's adapter with that in mind, the only Core AI specific lines are the construction of the model. Everything else hands it to FoundationModels.LanguageModelSession(model:) through the shared bridge. Proposed integration: a generic wrapper for any FoundationModels.LanguageModel conformer, taking an async factory so heavy loading stays lazy and the consumer owns load and unload. It depends only on the FoundationModels SDK framework behind @available(iOS 27, macOS 27, *). No CoreAI trait, no coreai-models dependency, no fork branch, nothing hosted. Core AI becomes a documented recipe: add apple/coreai-models to your app (the source package if your deployment floor is 27, or a self-built xcframework if you support earlier OS versions, which I have verified works with library evolution and CoreAI weak-linked), construct the model, and hand it to the wrapper. The PCC adapter and the shared bridge refactor stay as they are. That makes this mergeable without waiting on anything from Apple, and it covers any future model that adopts Apple's protocol. If that works for you I will push the restructured branch. |
Follow-up to the discussion in #210. This adds two Foundation Models protocol backends behind the OS 27
LanguageModelSession(model:)construction.CoreAILanguageModel(newCoreAItrait, wired like the MLX and Llama traits) runs Core AI.aimodelbundles exported with apple/coreai-models recipes. Verified on device (iPhone 17 Pro, iOS 27 beta) with Qwen3 and SmolLM2 exports: streaming, multi-turn transcripts, and the executor's tool-calling path.PrivateCloudComputeLanguageModelwraps the OS 27FoundationModels.PrivateCloudComputeLanguageModel, exposing availability, quota, and vision and tool capabilities. It requires thecom.apple.developer.private-cloud-computeentitlement, which Apple grants on request for apps under two million users. Verified on device, including tool calling.To share the session plumbing, the FM adapter's
respondandstreamResponsebodies move verbatim into file-scopefmRespondandfmStreamResponsehelpers parameterized by an async session factory. All three backends delegate to them. Streamed and returned text also drops leading newlines, which OS 27 executors emit when they strip an empty reasoning block from the output.Draft until the dependency is upstream. The
CoreAItrait currently depends on a fork of apple/coreai-models that carries two fixes offered upstream. Links are tracked in #210.Assuming those get approved (or similar fixes are issued by Apple), the dependency flips to apple/coreai-models and this becomes mergeable. This branch includes the fix from the standalone duplicated-trailing-prompt PR and will be rebased once that merges.