Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
260ae4a
add diffusers serve and generate commands
DN6 Oct 9, 2026
3e4aa57
add remote serve
DN6 Oct 9, 2026
5bc3948
update
DN6 Oct 9, 2026
fc92295
update
DN6 Oct 9, 2026
c964faf
Merge remote-tracking branch 'origin/main' into diffusers-serve
DN6 Oct 14, 2026
c8967b0
Address serve review feedback and run serve --remote on HF Jobs
DN6 Nov 3, 2026
a5781d1
Add context-parallel serving and allow --compile with --context-parallel
DN6 Nov 4, 2026
a60093d
Restore --compile with --context-parallel guard
DN6 Nov 4, 2026
3df7581
clean up
DN6 Nov 4, 2026
b81768c
Remove the SGLang and vLLM-Omni serve backends and clean up the serve…
DN6 Nov 4, 2026
a7e8ea0
Reorganize serve modules and validate manifests at startup
DN6 Nov 4, 2026
ccdcc36
Flatten serve response formatting into _format_job_response and _form…
DN6 Nov 5, 2026
2fd5dcb
Derive serve fps and sample rate from the model and merge parallel.py…
DN6 Nov 5, 2026
8e8f6c3
Seed request seeds from the global RNG and build job responses inside…
DN6 Nov 5, 2026
c746c03
Resolve outputs, fps and adapter weights in the manifest so the backe…
DN6 Nov 5, 2026
7179858
Address review: validate adapter repo ids, add --generation-timeout
DN6 Nov 5, 2026
8830305
Wait on the event loop, time out by deadline, and drop low-value serv…
DN6 Nov 5, 2026
29ad358
Simplify serve: inline single-use helpers, drop 501 stubs, shorten docs
DN6 Nov 5, 2026
57c1686
Merge branch 'main' into diffusers-serve
DN6 Nov 5, 2026
e66798c
Remove ParallelBackend
DN6 Nov 17, 2026
196fb0f
Harden serve media inputs, support modular step callbacks, and simpli…
DN6 Nov 18, 2026
aa2a41c
Drop fraction from progress events
DN6 Nov 18, 2026
52f2be3
Clean up
DN6 Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .ai/skills/diffusers-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ description: >
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `run` | Run any `DiffusionPipeline` or `ModularPipeline`. Forwards `--pipeline-kwargs` verbatim, saves output by detecting its runtime type, optionally runs on HF Jobs via `--remote`. |
| `schema` | Print the input schema for a pipeline repo (kwarg names, types, defaults, descriptions). **No weights downloaded** — only the small index file. |
| `serve` | Serve a pipeline over HTTP with the Generative Media Spec API: model discovery, queued generations, polling or SSE progress. |
| `custom_blocks` | Package a local `ModularPipelineBlocks` subclass for the Hub. |
| `env` | Print versions of diffusers + torch + transformers + accelerate + safetensors + CUDA + GPU info. Use when investigating environment issues, dtype/precision support, or building bug reports. |

Expand All @@ -33,6 +34,7 @@ The other commands are small enough that `diffusers-cli <command> --help` is the

```bash
diffusers-cli schema --help
diffusers-cli serve --help
diffusers-cli custom_blocks --help
diffusers-cli env --help
```
Expand Down
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
title: Quickstart
- local: using-diffusers/cli
title: Command line interface
- local: using-diffusers/serve
title: Serving
title: Get started
- isExpanded: true
sections:
Expand Down
13 changes: 12 additions & 1 deletion docs/source/en/using-diffusers/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ specific language governing permissions and limitations under the License.

# Command line interface

`diffusers-cli` is a command line client for running, inspecting, and packaging Diffusers pipelines.
`diffusers-cli` is a command line client for running, inspecting, and packaging Diffusers pipelines. It is also

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remote worker doesn’t inherit the local PR revision. Running the PR’s CLI locally with --remote launched a stock worker whose installed Diffusers release lacked serve. Maybe we should clarify this?

installed as `diffusers`, so `diffusers serve` and `diffusers-cli serve` are the same command.

## Available commands

Expand All @@ -21,6 +22,7 @@ specific language governing permissions and limitations under the License.
| [`env`](#env) | Print environment info for bug reports. |
| [`schema`](#schema) | Inspect a pipeline's `__call__` signature without downloading weights. |
| [`run`](#run) | Run a pipeline locally or in a Hugging Face Sandbox. |
| [`serve`](#serve) | Serve a pipeline over HTTP with the Generative Media Spec API. |
| [`custom_blocks`](#customblocks) | Package a local `ModularPipelineBlocks` subclass for the Hub. |
| [`fp16_safetensors`](#fp16safetensors) | Convert a checkpoint to fp16 `.safetensors`. |
| [`skills`](#skills) | Install pre-authored skill bundles into your AI coding agent. |
Expand Down Expand Up @@ -287,6 +289,15 @@ diffusers-cli run -m black-forest-labs/FLUX.1-dev --dtype bf16 \
hf sandbox kill <id>
```

## `serve`

Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think people know about GMS as it's not a standard. Can we hyperlink?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge.


```bash
pip install "diffusers[serve]"
diffusers-cli serve black-forest-labs/FLUX.1-dev

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redacted outputs would be nice here.

```

## `custom_blocks`

Package a local `ModularPipelineBlocks` subclass for upload to the Hub. Reads a Python file, AST-scans it for
Expand Down
207 changes: 207 additions & 0 deletions docs/source/en/using-diffusers/serve.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
<!--Copyright 2026 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# Serving

`diffusers serve` serves a pipeline over HTTP with the Generative Media Spec (GMS) API: clients discover what the
model accepts, create a generation, follow it while it runs, and download the results.

```bash
pip install "diffusers[serve]"
diffusers serve black-forest-labs/FLUX.1-dev
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we show the output here?

`serve` takes the same loading and optimization flags as [`diffusers-cli run`](./cli#run). The server listens on

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also add a line saying that remote inference is also supported.

`127.0.0.1:8000` and has no authentication, so keep it on a trusted network. Run `diffusers serve --help` for every
option.

## Generations

Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here (the server caps it at 300):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the output is of type Generation object, I would make that clear here.


```bash
curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{
"model": "black-forest-labs/FLUX.1-dev",
"inputs": {"prompt": "an astronaut riding a horse"},
"parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42}
}'
Comment on lines +32 to +36

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps also show how to do it in Python using requests?

```

```json
{
"id": "gen_3f9a1c2b7d4e",
"model": "black-forest-labs/FLUX.1-dev",
"revision": "<commit hash of the model repo>",
"status": "complete",
"created_at": "2026-11-05T10:00:00Z",
"progress": null,
"outputs": {
"image": [
{
"media_type": "image/png",
"width": 1024,
"height": 1024,
"url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local servers, can the users specify the results directory?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, on server start up you can set the directory where the outputs are saved

diffusers-cli serve --output-dir <my folder>

}
]
},
"expires_at": "2026-11-05T11:00:12Z",
"seed": 42,
"usage": {"compute_seconds": 11.84}
}
```

Query the server from Python with [httpx](https://www.python-httpx.org/):

```py
import httpx

response = httpx.post(
"http://localhost:8000/v1/generations",
headers={"Prefer": "wait=120"},
json={"model": "black-forest-labs/FLUX.1-dev", "inputs": {"prompt": "an astronaut riding a horse"}},
timeout=130,
)
image_url = response.json()["outputs"]["image"][0]["url"]
with open("image.png", "wb") as f:
f.write(httpx.get(image_url).content)
```

Without `Prefer: wait`, the response comes back at once with `"status": "queued"`. Follow its progress as
server-sent events, or poll it:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server-sent events, or poll it:
server-sent events

can we move or poll it to where the example are? so it's a bit more clear what this example is


```bash
curl -N localhost:8000/v1/generations/gen_3f9a1c2b7d4e/events
```

```text
id: 1
event: progress
data: {"step": 1, "total_steps": 28}

...

id: 29
event: complete
data: {"id": "gen_3f9a1c2b7d4e", "status": "complete", "outputs": {...}, ...}
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Alternatively, poll it until `status` is `complete`

```bash
curl localhost:8000/v1/generations/gen_3f9a1c2b7d4e # poll
curl -X POST localhost:8000/v1/generations/gen_3f9a1c2b7d4e/cancel # cancel

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, let's not bundle cancel/delete/download into the poll example

curl -X DELETE localhost:8000/v1/generations/gen_3f9a1c2b7d4e # delete it and its files
curl -O http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png
```

`inputs` holds the prompt and media (`{"type": "image", "url": "https://..."}`), and `parameters` holds settings.
Both use the names the model's manifest declares. A completed generation lists a download URL for each output, or
inline base64 with `"response_format": "b64_json"`. Outputs expire after one hour.

Invalid requests are rejected with a pointer to the offending field:

```json
{"error": {"code": "invalid_request", "message": "task 'text_to_image' declares no parameter 'cfg'", "pointer": "/parameters/cfg"}}
```

A request can also apply LoRAs for that generation only:

```bash
curl localhost:8000/v1/generations -d '{
"model": "black-forest-labs/FLUX.1-dev",
"inputs": {"prompt": "a tiny cat"},
"adapters": [{"type": "lora", "path": "hf:alvdansen/littletinies", "scale": 0.8}]
}'
```

## Manifests

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this go before generations section? I think it reads better this way since you need to build request from it


`GET /v1/models/<model>` returns the model's manifest: its tasks, inputs, parameters and capabilities.

```bash
curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev
```

```json
{
"spec": "gms/0.1",
"manifest": {
"id": "black-forest-labs/FLUX.1-dev",
"modalities": {"in": ["image"], "out": ["image"]},
"tasks": {
"text_to_image": {
"inputs": {"prompt": {"type": "text", "required": true}, "prompt_2": {"type": "text"}},
"outputs": {"image": {"type": "image"}},
"parameters": {
"height": {"type": "integer"},
"width": {"type": "integer"},
"steps": {"type": "integer", "default": 28},
"guidance_scale": {"type": "number", "default": 3.5},
"seed": {"type": "integer"}
},
"x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}}
}
},
"capabilities": ["sse", "wait", "adapters"],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are sse and wait? Do we treat ControlNets as adapters as well? How do we decide the capabilities based on a model id?

Also, what is x-diffusers?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a gms.json file inside the model repo.
https://huggingface.co/diffusers-internal-dev/flux2-klein-4b-gms/blob/main/gms.json

If no gms.json file is present in the repo, we list these capabilties: ["sse", "wait"] and include adapters if the pipeline can load loras.

sse - Server Side Events. This is whether the server supports sending current state of the request (queued, generating etc) to the client side. Useful for cases where the UI or frontend needs to render something while the user waits

wait - Whether the server allows Prefer: wait HTTP headers which block the request until it finishes. By default, when the server gets a request, we return a 201 status code with a request id that the client can then use to query the server to know if a generation is complete or not.

ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.

x-diffusers is meant to be a field in the GMS spec that maps parameter names from a user request to diffusers pipeline kwargs. Since GMS is meant to be framework agnostic, we need some way to route parameters that might not share our nomenclature. e.g. if I were to serve Qwen-Image but I only want to offer true CFG, I would need a way to remap the meaning of guidance scale on my service because Diffusers calls it true_cfg_scale

"x-diffusers": {"parameters": {"guidance_scale": "true_cfg_scale"}}

The x- convention is from the OpenAPI extensions. So what it is communicating here is that we are extending GMS to serve diffusers models and we need to remap these parameters in order to do that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant.

"adapters": {"lora": {"targets": ["transformer", "text_encoder"]}}
}
}
```

`serve` uses the file passed with `--manifest`, else a `gms.json` in the model repo, else a manifest derived from the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are these manifest specs mapped inside the server?

pipeline's `__call__` signature. Publish a `gms.json` to add value ranges, presets, or a duration in seconds. The
server checks the manifest at startup and refuses to start if it declares something the pipeline can't do.

`x-diffusers` maps a manifest's names to pipeline arguments, and `capabilities` lists what the server supports:
`sse` (event streams), `wait` (`Prefer: wait`) and `adapters` (per-request LoRAs). The manifest above is abridged.

## Timeouts

- `--generation-timeout 10m` fails a generation that runs longer than 10 minutes.
- `--idle-timeout 30m` shuts the server down after 30 minutes without requests.

## Security

The server has no authentication and listens on `127.0.0.1`. Before exposing it, put it behind a proxy that
authenticates requests. With `--remote`, requests already need your Hugging Face token.

The server downloads media inputs given as URLs. By default it only fetches public addresses, up to 256 MB, and doesn't
follow redirects. To allow only specific hosts, list them:

```bash
diffusers serve black-forest-labs/FLUX.1-dev --allowed-media-domains upload.wikimedia.org

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to enable this restriction right away? Seems like removing it would simplify the codebase a bit.

```

Listed hosts are fetched even when they're private, such as `localhost` while testing.

## Multiple GPUs

`--data-parallel N` loads a copy of the pipeline on each of N GPUs and runs up to N generations at once.
`--context-parallel` is not supported by `serve` yet.

```bash
diffusers serve black-forest-labs/FLUX.1-dev --data-parallel 4
```

## Remote serving

`--remote` runs the server in a [Hugging Face Job](https://huggingface.co/docs/huggingface_hub/guides/jobs) and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does remote serving follow the same settings as run? If not, why not?

streams its logs. Once the model loads, it prints the server URL. Requests authenticate with your Hugging Face token.

```bash
diffusers serve black-forest-labs/FLUX.1-dev --remote --flavor a10g-large

curl <url>/v1/generations -H 'Prefer: wait=120' -H "Authorization: Bearer $HF_TOKEN" \
-d '{"model": "black-forest-labs/FLUX.1-dev", "inputs": {"prompt": "a cat on the moon"}}'
```

The Job stops when the server goes idle (`--idle-timeout` defaults to `30m`), and `--flavor`, `--image`,
`--dependencies` and `--namespace` work as in [`run --remote`](./cli#remote-execution---remote).
10 changes: 9 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,8 @@
"opencv-python",
"timm",
"flashpack",
"fastapi",
"uvicorn",
]

# this is a lookup table with items like:
Expand Down Expand Up @@ -253,6 +255,7 @@ def run(self):
extras["nvidia_modelopt"] = deps_list("nvidia_modelopt[hf]")
extras["sdnq"] = deps_list("sdnq")
extras["flashpack"] = deps_list("flashpack")
extras["serve"] = deps_list("fastapi", "uvicorn")

extras["dev"] = extras["quality"] + extras["test"] + extras["training"] + extras["docs"] + extras["torch"]

Expand Down Expand Up @@ -287,7 +290,12 @@ def run(self):
python_requires=">=3.11.0",
install_requires=list(install_requires),
extras_require=extras,
entry_points={"console_scripts": ["diffusers-cli=diffusers.commands.diffusers_cli:main"]},
entry_points={
"console_scripts": [
"diffusers-cli=diffusers.commands.diffusers_cli:main",
"diffusers=diffusers.commands.diffusers_cli:main",
]
},
classifiers=[
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
Expand Down
2 changes: 2 additions & 0 deletions src/diffusers/commands/diffusers_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
from .fp16_safetensors import FP16SafetensorsCommand
from .run import RunCommand
from .schema import SchemaCommand
from .serve import ServeCommand
from .skills import SkillsCommand


Expand Down Expand Up @@ -49,6 +50,7 @@ def main():
CustomBlocksCommand.register_subcommand(commands_parser)
RunCommand.register_subcommand(commands_parser)
SchemaCommand.register_subcommand(commands_parser)
ServeCommand.register_subcommand(commands_parser)
SkillsCommand.register_subcommand(commands_parser)

# Let's go
Expand Down
6 changes: 4 additions & 2 deletions src/diffusers/commands/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,10 @@
# ---------------------------------------------------------------------------


def _add_loading_arguments(parser: ArgumentParser) -> None:
parser.add_argument("--model", "-m", required=True, help="Model id on the Hugging Face Hub or local path.")
def _add_loading_arguments(parser: ArgumentParser, require_model: bool = True) -> None:
parser.add_argument(
"--model", "-m", required=require_model, help="Model id on the Hugging Face Hub or local path."
)
Comment on lines +152 to +154

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would it be not required?

parser.add_argument(
"--device-map",
default=None,
Expand Down
Loading
Loading