Repository navigation
Add diffusers serve to CLI #15017
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add diffusers serve to CLI #15017
Changes from all commits
260ae4a
3e4aa57
5bc3948
fc92295
c964faf
c8967b0
a5781d1
a60093d
3df7581
b81768c
a7e8ea0
ccdcc36
2fd5dcb
8e8f6c3
c746c03
7179858
8830305
29ad358
57c1686
e66798c
196fb0f
aa2a41c
52f2be3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,7 +12,8 @@ specific language governing permissions and limitations under the License. | |
|
|
||
| # Command line interface | ||
|
|
||
| `diffusers-cli` is a command line client for running, inspecting, and packaging Diffusers pipelines. | ||
| `diffusers-cli` is a command line client for running, inspecting, and packaging Diffusers pipelines. It is also | ||
| installed as `diffusers`, so `diffusers serve` and `diffusers-cli serve` are the same command. | ||
|
|
||
| ## Available commands | ||
|
|
||
|
|
@@ -21,6 +22,7 @@ specific language governing permissions and limitations under the License. | |
| | [`env`](#env) | Print environment info for bug reports. | | ||
| | [`schema`](#schema) | Inspect a pipeline's `__call__` signature without downloading weights. | | ||
| | [`run`](#run) | Run a pipeline locally or in a Hugging Face Sandbox. | | ||
| | [`serve`](#serve) | Serve a pipeline over HTTP with the Generative Media Spec API. | | ||
| | [`custom_blocks`](#customblocks) | Package a local `ModularPipelineBlocks` subclass for the Hub. | | ||
| | [`fp16_safetensors`](#fp16safetensors) | Convert a checkpoint to fp16 `.safetensors`. | | ||
| | [`skills`](#skills) | Install pre-authored skill bundles into your AI coding agent. | | ||
|
|
@@ -287,6 +289,15 @@ diffusers-cli run -m black-forest-labs/FLUX.1-dev --dtype bf16 \ | |
| hf sandbox kill <id> | ||
| ``` | ||
|
|
||
| ## `serve` | ||
|
|
||
| Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think people know about GMS as it's not a standard. Can we hyperlink?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge. |
||
|
|
||
| ```bash | ||
| pip install "diffusers[serve]" | ||
| diffusers-cli serve black-forest-labs/FLUX.1-dev | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Redacted outputs would be nice here. |
||
| ``` | ||
|
|
||
| ## `custom_blocks` | ||
|
|
||
| Package a local `ModularPipelineBlocks` subclass for upload to the Hub. Reads a Python file, AST-scans it for | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,207 @@ | ||||||
| <!--Copyright 2026 The HuggingFace Team. All rights reserved. | ||||||
|
|
||||||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||||||
| the License. You may obtain a copy of the License at | ||||||
|
|
||||||
| http://www.apache.org/licenses/LICENSE-2.0 | ||||||
|
|
||||||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||||||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||||||
| specific language governing permissions and limitations under the License. | ||||||
| --> | ||||||
|
|
||||||
| # Serving | ||||||
|
|
||||||
| `diffusers serve` serves a pipeline over HTTP with the Generative Media Spec (GMS) API: clients discover what the | ||||||
| model accepts, create a generation, follow it while it runs, and download the results. | ||||||
|
|
||||||
| ```bash | ||||||
| pip install "diffusers[serve]" | ||||||
| diffusers serve black-forest-labs/FLUX.1-dev | ||||||
| ``` | ||||||
|
|
||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we show the output here? |
||||||
| `serve` takes the same loading and optimization flags as [`diffusers-cli run`](./cli#run). The server listens on | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would also add a line saying that remote inference is also supported. |
||||||
| `127.0.0.1:8000` and has no authentication, so keep it on a trusted network. Run `diffusers serve --help` for every | ||||||
| option. | ||||||
|
|
||||||
| ## Generations | ||||||
|
|
||||||
| Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here (the server caps it at 300): | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If the output is of type |
||||||
|
|
||||||
| ```bash | ||||||
| curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{ | ||||||
| "model": "black-forest-labs/FLUX.1-dev", | ||||||
| "inputs": {"prompt": "an astronaut riding a horse"}, | ||||||
| "parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42} | ||||||
| }' | ||||||
|
Comment on lines
+32
to
+36
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps also show how to do it in Python using |
||||||
| ``` | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "id": "gen_3f9a1c2b7d4e", | ||||||
| "model": "black-forest-labs/FLUX.1-dev", | ||||||
| "revision": "<commit hash of the model repo>", | ||||||
| "status": "complete", | ||||||
| "created_at": "2026-11-05T10:00:00Z", | ||||||
| "progress": null, | ||||||
| "outputs": { | ||||||
| "image": [ | ||||||
| { | ||||||
| "media_type": "image/png", | ||||||
| "width": 1024, | ||||||
| "height": 1024, | ||||||
| "url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png" | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For local servers, can the users specify the results directory?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, on server start up you can set the directory where the outputs are saved |
||||||
| } | ||||||
| ] | ||||||
| }, | ||||||
| "expires_at": "2026-11-05T11:00:12Z", | ||||||
| "seed": 42, | ||||||
| "usage": {"compute_seconds": 11.84} | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| Query the server from Python with [httpx](https://www.python-httpx.org/): | ||||||
|
|
||||||
| ```py | ||||||
| import httpx | ||||||
|
|
||||||
| response = httpx.post( | ||||||
| "http://localhost:8000/v1/generations", | ||||||
| headers={"Prefer": "wait=120"}, | ||||||
| json={"model": "black-forest-labs/FLUX.1-dev", "inputs": {"prompt": "an astronaut riding a horse"}}, | ||||||
| timeout=130, | ||||||
| ) | ||||||
| image_url = response.json()["outputs"]["image"][0]["url"] | ||||||
| with open("image.png", "wb") as f: | ||||||
| f.write(httpx.get(image_url).content) | ||||||
| ``` | ||||||
|
|
||||||
| Without `Prefer: wait`, the response comes back at once with `"status": "queued"`. Follow its progress as | ||||||
| server-sent events, or poll it: | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
can we move |
||||||
|
|
||||||
| ```bash | ||||||
| curl -N localhost:8000/v1/generations/gen_3f9a1c2b7d4e/events | ||||||
| ``` | ||||||
|
|
||||||
| ```text | ||||||
| id: 1 | ||||||
| event: progress | ||||||
| data: {"step": 1, "total_steps": 28} | ||||||
|
|
||||||
| ... | ||||||
|
|
||||||
| id: 29 | ||||||
| event: complete | ||||||
| data: {"id": "gen_3f9a1c2b7d4e", "status": "complete", "outputs": {...}, ...} | ||||||
| ``` | ||||||
|
|
||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| ```bash | ||||||
| curl localhost:8000/v1/generations/gen_3f9a1c2b7d4e # poll | ||||||
| curl -X POST localhost:8000/v1/generations/gen_3f9a1c2b7d4e/cancel # cancel | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also, let's not bundle cancel/delete/download into the poll example |
||||||
| curl -X DELETE localhost:8000/v1/generations/gen_3f9a1c2b7d4e # delete it and its files | ||||||
| curl -O http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png | ||||||
| ``` | ||||||
|
|
||||||
| `inputs` holds the prompt and media (`{"type": "image", "url": "https://..."}`), and `parameters` holds settings. | ||||||
| Both use the names the model's manifest declares. A completed generation lists a download URL for each output, or | ||||||
| inline base64 with `"response_format": "b64_json"`. Outputs expire after one hour. | ||||||
|
|
||||||
| Invalid requests are rejected with a pointer to the offending field: | ||||||
|
|
||||||
| ```json | ||||||
| {"error": {"code": "invalid_request", "message": "task 'text_to_image' declares no parameter 'cfg'", "pointer": "/parameters/cfg"}} | ||||||
| ``` | ||||||
|
|
||||||
| A request can also apply LoRAs for that generation only: | ||||||
|
|
||||||
| ```bash | ||||||
| curl localhost:8000/v1/generations -d '{ | ||||||
| "model": "black-forest-labs/FLUX.1-dev", | ||||||
| "inputs": {"prompt": "a tiny cat"}, | ||||||
| "adapters": [{"type": "lora", "path": "hf:alvdansen/littletinies", "scale": 0.8}] | ||||||
| }' | ||||||
| ``` | ||||||
|
|
||||||
| ## Manifests | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should this go before generations section? I think it reads better this way since you need to build request from it |
||||||
|
|
||||||
| `GET /v1/models/<model>` returns the model's manifest: its tasks, inputs, parameters and capabilities. | ||||||
|
|
||||||
| ```bash | ||||||
| curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev | ||||||
| ``` | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "spec": "gms/0.1", | ||||||
| "manifest": { | ||||||
| "id": "black-forest-labs/FLUX.1-dev", | ||||||
| "modalities": {"in": ["image"], "out": ["image"]}, | ||||||
| "tasks": { | ||||||
| "text_to_image": { | ||||||
| "inputs": {"prompt": {"type": "text", "required": true}, "prompt_2": {"type": "text"}}, | ||||||
| "outputs": {"image": {"type": "image"}}, | ||||||
| "parameters": { | ||||||
| "height": {"type": "integer"}, | ||||||
| "width": {"type": "integer"}, | ||||||
| "steps": {"type": "integer", "default": 28}, | ||||||
| "guidance_scale": {"type": "number", "default": 3.5}, | ||||||
| "seed": {"type": "integer"} | ||||||
| }, | ||||||
| "x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}} | ||||||
| } | ||||||
| }, | ||||||
| "capabilities": ["sse", "wait", "adapters"], | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What are Also, what is
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a If no
ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.
The
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant. |
||||||
| "adapters": {"lora": {"targets": ["transformer", "text_encoder"]}} | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| `serve` uses the file passed with `--manifest`, else a `gms.json` in the model repo, else a manifest derived from the | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where are these manifest specs mapped inside the server? |
||||||
| pipeline's `__call__` signature. Publish a `gms.json` to add value ranges, presets, or a duration in seconds. The | ||||||
| server checks the manifest at startup and refuses to start if it declares something the pipeline can't do. | ||||||
|
|
||||||
| `x-diffusers` maps a manifest's names to pipeline arguments, and `capabilities` lists what the server supports: | ||||||
| `sse` (event streams), `wait` (`Prefer: wait`) and `adapters` (per-request LoRAs). The manifest above is abridged. | ||||||
|
|
||||||
| ## Timeouts | ||||||
|
|
||||||
| - `--generation-timeout 10m` fails a generation that runs longer than 10 minutes. | ||||||
| - `--idle-timeout 30m` shuts the server down after 30 minutes without requests. | ||||||
|
|
||||||
| ## Security | ||||||
|
|
||||||
| The server has no authentication and listens on `127.0.0.1`. Before exposing it, put it behind a proxy that | ||||||
| authenticates requests. With `--remote`, requests already need your Hugging Face token. | ||||||
|
|
||||||
| The server downloads media inputs given as URLs. By default it only fetches public addresses, up to 256 MB, and doesn't | ||||||
| follow redirects. To allow only specific hosts, list them: | ||||||
|
|
||||||
| ```bash | ||||||
| diffusers serve black-forest-labs/FLUX.1-dev --allowed-media-domains upload.wikimedia.org | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need to enable this restriction right away? Seems like removing it would simplify the codebase a bit. |
||||||
| ``` | ||||||
|
|
||||||
| Listed hosts are fetched even when they're private, such as `localhost` while testing. | ||||||
|
|
||||||
| ## Multiple GPUs | ||||||
|
|
||||||
| `--data-parallel N` loads a copy of the pipeline on each of N GPUs and runs up to N generations at once. | ||||||
| `--context-parallel` is not supported by `serve` yet. | ||||||
|
|
||||||
| ```bash | ||||||
| diffusers serve black-forest-labs/FLUX.1-dev --data-parallel 4 | ||||||
| ``` | ||||||
|
|
||||||
| ## Remote serving | ||||||
|
|
||||||
| `--remote` runs the server in a [Hugging Face Job](https://huggingface.co/docs/huggingface_hub/guides/jobs) and | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does remote serving follow the same settings as |
||||||
| streams its logs. Once the model loads, it prints the server URL. Requests authenticate with your Hugging Face token. | ||||||
|
|
||||||
| ```bash | ||||||
| diffusers serve black-forest-labs/FLUX.1-dev --remote --flavor a10g-large | ||||||
|
|
||||||
| curl <url>/v1/generations -H 'Prefer: wait=120' -H "Authorization: Bearer $HF_TOKEN" \ | ||||||
| -d '{"model": "black-forest-labs/FLUX.1-dev", "inputs": {"prompt": "a cat on the moon"}}' | ||||||
| ``` | ||||||
|
|
||||||
| The Job stops when the server goes idle (`--idle-timeout` defaults to `30m`), and `--flavor`, `--image`, | ||||||
| `--dependencies` and `--namespace` work as in [`run --remote`](./cli#remote-execution---remote). | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -148,8 +148,10 @@ | |
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def _add_loading_arguments(parser: ArgumentParser) -> None: | ||
| parser.add_argument("--model", "-m", required=True, help="Model id on the Hugging Face Hub or local path.") | ||
| def _add_loading_arguments(parser: ArgumentParser, require_model: bool = True) -> None: | ||
| parser.add_argument( | ||
| "--model", "-m", required=require_model, help="Model id on the Hugging Face Hub or local path." | ||
| ) | ||
|
Comment on lines
+152
to
+154
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When would it be not required? |
||
| parser.add_argument( | ||
| "--device-map", | ||
| default=None, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The remote worker doesn’t inherit the local PR revision. Running the PR’s CLI locally with
--remotelaunched a stock worker whose installed Diffusers release lackedserve. Maybe we should clarify this?