Skip to content

feat(api): support multimodal structured generation#218

Open
svonava wants to merge 3 commits into
mainfrom
agent/public-sdk-multimodal-generate
Open

feat(api): support multimodal structured generation#218
svonava wants to merge 3 commits into
mainfrom
agent/public-sdk-multimodal-generate

Conversation

@svonava

@svonava svonava commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What changed

  • extend Python and TypeScript generate and streaming helpers with native image inputs
  • add typed, mutually exclusive json_schema, regex, and ebnf grammar contracts
  • share image encoding, format detection, size/count validation, and request serialization while preserving text-only request bytes
  • synchronize the public gateway's bounded native image and grammar request contract with the managed implementation
  • add matching local sie_server support, including capability gates, bounded grammar parsing, internal JSON Schema $ref handling, and blocking/SSE adapter forwarding
  • update generated OpenAPI artifacts and add an approachable SDK-only vision example

Why

Managed native multimodal generation can now accept images and structured-output grammar, but the public SDK and local/self-hosted server could not express or accept that request. Console playground code would therefore be impossible to reproduce through the SDK, and the same snippet would behave differently against managed and local endpoints.

User impact

The same typed SIEClient.generate(...) call can target local/self-hosted SIE or the managed service. Existing text-only callers retain their current request shape.

Validation

  • Python SDK focused tests: 69 passed
  • TypeScript SDK focused tests: 66 passed
  • local server/adapter/streaming focused tests: 263 passed
  • gateway native-image tests: 6 passed for library and binary targets
  • gateway grammar tests: 50 passed for library and binary targets
  • gateway generate OpenAPI contract: passed
  • generated gateway OpenAPI is byte-identical to the checked-in artifact
  • Python Ruff, TypeScript Biome/typecheck/build, and Rust formatting: passed

The filtered public checkout omits an existing compile-time conformance fixture. Focused Rust tests used a temporary untracked link to the corresponding internal fixture; that link was removed and is not part of this PR.

@svonava
svonava marked this pull request as ready for review July 23, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant