Skip to content

Add diffusers serve to CLI - #15017

Open
DN6 wants to merge 23 commits into
mainfrom
diffusers-serve
Open

DN6 wants to merge 23 commits into
mainfrom
diffusers-serve

Conversation

@DN6

@DN6 DN6 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

Introduce diffusers-cli serve to serve a diffusers compatible model/pipeline based on the GMS spec.

pip install "diffusers[serve]"
diffusers serve black-forest-labs/FLUX.1-dev
curl localhost:8000/v1/generations -H 'Prefer: wait=120' \
  -d '{"model": "black-forest-labs/FLUX.1-dev", "inputs": {"prompt": "a cat on the moon"}}'

Features

  • Supports the same optimisation flags as diffusers-cli run
  • Multi-GPU inference via --data-parallel (1 pipeline per GPU) and --context-parallel (tensor parallel will be added in a follow up)
  • Launch servers on remote infra via HF Jobs

Fixes # (issue)

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? (important for complex PRs)
  • Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests utils size/L PR with diff > 200 LOC labels Oct 12, 2026
# Conflicts:
#	docs/source/en/using-diffusers/cli.md

@SunMarc SunMarc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excited to have this ! Just did a pass over the doc for now

Comment thread docs/source/en/using-diffusers/cli.md Outdated

```bash
pip install "diffusers[serve]"
diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep --model if you want but i think it is better is it is optional. makes command shorter and most users will expect the model to follow.

As Lysandre pointed, diffusers serve would be nice also.

Lastly, for this specific model, do we need to specify the dtype ? In general, I expect that the model in its original dtype which is probably bf16.

Suggested change
diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16
diffusers-cli serve black-forest-labs/FLUX.1-dev --dtype bf16

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +308 to +309
`serve` takes the same loading and optimization flags as [`run`](#run) (`--dtype`, `--device-map`,
`--cpu-offload`, `--compile`, `--lora`, ...), plus `--host` (default `127.0.0.1`) and `--port` (default `8000`).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe restrict some the optimization flags unless we really want them ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DN6 I think this is following run so those flags are already supported?

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +312 to +320
[Modular repos](../modular_diffusers/overview) are detected the same way as in `run` and served in-process. They
report no step progress, and a cancel request only takes effect once the pipeline call returns.

| Flag | Effect |
|---|---|
| `--default-seed` | Seed used for requests that do not set one. Without it, each such request gets a random seed. |
| `--enable-cors` | Allow cross-origin requests from any origin, so a browser app on another host can call the server. |
| `--public-url` | Base URL clients reach the server at, used to build output file links. Set it behind a reverse proxy. |
| `--log-level` | Logging level for diffusers and the HTTP server (`debug`, `info`, `warning`, `error`, `critical`). |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should have serving its own dedicated section in the docs ? The doc is really dense so I think ppl will have a hard time finding the info they want. Maybe something like in transformers would be prefered ? https://huggingface.co/docs/transformers/v5.19.0/en/serve-cli/serving

Btw, I think it is fine to not explain all options of the server that users will never really set them. The agent will be able to get them nevertheless

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good 👍🏽 Yeah, we can move to separate section and have options under --help

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +328 to +346
```bash
# What the model accepts: tasks, inputs, parameters with types and defaults.
curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev

# Create a generation and wait up to 120 seconds for it.
curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{
"model": "black-forest-labs/FLUX.1-dev",
"inputs": {"prompt": "an astronaut riding a horse"},
"parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42}
}'

# Follow step progress as server-sent events, or poll the generation.
curl -N localhost:8000/v1/generations/<id>/events
curl localhost:8000/v1/generations/<id>

# Cancel a queued or running generation, or delete it together with its files.
curl -X POST localhost:8000/v1/generations/<id>/cancel
curl -X DELETE localhost:8000/v1/generations/<id>
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would do a section dedicated to v1/generations

Comment thread docs/source/en/using-diffusers/cli.md Outdated

```bash
diffusers-cli generate --sandbox-id <id> --inputs '{"prompt": "a cat on the moon"}'
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can't send directly an http request directly to the sandbox ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can. This was more of a convenience option to construct a GMS spec compatible request. But perhaps this is overkill and we can stick to just HTTP requests.

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +361 to +363
### Remote serving (`--remote`)

`--remote` starts the server in a [Hugging Face Sandbox](https://huggingface.co/docs/huggingface_hub/guides/sandbox)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool feature !

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +383 to +389
With `--backend sglang` or `--backend vllm`, the sandbox starts from the engine's own image
(`lmsysorg/sglang:latest` or `vllm/vllm-omni:latest`) unless `--image` names another, so the engine does not need
installing:

```bash
diffusers-cli serve --model black-forest-labs/FLUX.2-klein-4B --backend sglang --remote --flavor a100-large
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about adding right now the slgang and vllm backend in diffusers serve

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the motivation was to allow users to take advantage of features that exist in these libs that diffusers doesn't have. Any strong reason to exclude?

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +430 to +434
`--backend sglang` and `--backend vllm` run the model in [SGLang Diffusion](https://docs.sglang.io/) or
[vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni) instead of in-process. `serve` starts `sglang serve` or
`vllm serve --omni` as a child process on a local port, waits for it to become healthy, and translates GMS
requests to the engine's OpenAI-compatible image and video routes. Clients only ever talk to the GMS endpoints
of `serve`. The engine must be installed separately.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we can add a new endpoint on their side with GMS, that would be better instead no ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This is the ideal outcome. I saw this as a sort of intermediate step towards that goal.

Comment thread docs/source/en/using-diffusers/cli.md Outdated
Comment on lines +447 to +459
## `generate`

Send a generation to a server started with [`serve`](#serve), or to any other Generative Media Spec server, and
save its outputs. Unlike [`run`](#run), `generate` loads no pipeline: it reads the model's discovery document,
submits the request, polls the generation until it finishes, and downloads the files.

```bash
diffusers-cli generate --url http://localhost:8000 --inputs '{"prompt": "a cat on the moon"}'

diffusers-cli generate --url http://localhost:8000 \
--inputs '{"prompt": "make the fur grey", "image": "cat.png"}' \
--parameters '{"steps": 8, "seed": 0}' --output outputs/
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about generate, i don't think we should encourage users to use this but just rely on sending directly http request no ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I added as a convenience function. But plain HTTP is cleaner.

@DN6 DN6 changed the title [WIP] Add diffusers serve to CLI Add diffusers serve to CLI Nov 5, 2026
@DN6
DN6 marked this pull request as ready for review November 5, 2026 16:39
@DN6
DN6 requested review from SunMarc, sayakpaul and yiyixuxu November 5, 2026 16:40
parameters = blocks_signature(self.pipeline.blocks).parameters
else:
parameters = inspect.signature(type(self.pipeline).__call__).parameters
self._reports_steps = "callback_on_step_end" in parameters

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i think util the first class streaming PR is merged on modular side, we could add callbacks to modular pipelines programmatically, I included an example/guide here https://huggingface.co/docs/diffusers/main/en/using-diffusers/callback#modular-pipelines

@sayakpaul sayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed part of the PR. This is a large PR without any commentary of how some things are implemented.

I think it would be helpful to have some explanation on the design decisions that were taken to do the server work. For example, I don't understand the separation between DiffusersBackend and ParallelBackend (and similar others).

Additionally, it's not clear to what extent we want the users to customize the server itself. For example, do we want to expose wny flags to control concurrency?


## `serve`

Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think people know about GMS as it's not a standard. Can we hyperlink?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge.

Comment thread docs/source/en/using-diffusers/serve.md Outdated

## Generations

Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can users override this limit of 120s?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

120s here is referring to the example snippet. Prefer:wait is controlled by the user. The default is 300s

Comment on lines +32 to +36
curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{
"model": "black-forest-labs/FLUX.1-dev",
"inputs": {"prompt": "an astronaut riding a horse"},
"parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42}
}'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps also show how to do it in Python using requests?

"media_type": "image/png",
"width": 1024,
"height": 1024,
"url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local servers, can the users specify the results directory?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, on server start up you can set the directory where the outputs are saved

diffusers-cli serve --output-dir <my folder>

Comment thread docs/source/en/using-diffusers/serve.md Outdated
```text
id: 1
event: progress
data: {"step": 1, "total_steps": 28, "fraction": 0.0357}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does fraction mean?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fraction of the total steps. 1/28 == 0.0357

"x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}}
}
},
"capabilities": ["sse", "wait", "adapters"],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are sse and wait? Do we treat ControlNets as adapters as well? How do we decide the capabilities based on a model id?

Also, what is x-diffusers?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a gms.json file inside the model repo.
https://huggingface.co/diffusers-internal-dev/flux2-klein-4b-gms/blob/main/gms.json

If no gms.json file is present in the repo, we list these capabilties: ["sse", "wait"] and include adapters if the pipeline can load loras.

sse - Server Side Events. This is whether the server supports sending current state of the request (queued, generating etc) to the client side. Useful for cases where the UI or frontend needs to render something while the user waits

wait - Whether the server allows Prefer: wait HTTP headers which block the request until it finishes. By default, when the server gets a request, we return a 201 status code with a request id that the client can then use to query the server to know if a generation is complete or not.

ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.

x-diffusers is meant to be a field in the GMS spec that maps parameter names from a user request to diffusers pipeline kwargs. Since GMS is meant to be framework agnostic, we need some way to route parameters that might not share our nomenclature. e.g. if I were to serve Qwen-Image but I only want to offer true CFG, I would need a way to remap the meaning of guidance scale on my service because Diffusers calls it true_cfg_scale

"x-diffusers": {"parameters": {"guidance_scale": "true_cfg_scale"}}

The x- convention is from the OpenAPI extensions. So what it is communicating here is that we are extending GMS to serve diffusers models and we need to remap these parameters in order to do that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant.


## Remote serving

`--remote` runs the server in a [Hugging Face Job](https://huggingface.co/docs/huggingface_hub/guides/jobs) and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does remote serving follow the same settings as run? If not, why not?

Comment thread src/diffusers/commands/serve/app.py Resolved
) from e

from .app import create_app
from .backends import DiffusersBackend, ParallelBackend, backend_capabilities

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the advantage of doing it like this?

@DN6 DN6 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayakpaul Reduced the scope of the PR here to make review more manageable. Parallelism support will be added in a follow up. Sharing an overview of how the server works

Since a pipeline calls takes much longer than a normal HTTP request normally stays open, the server doesn't run the pipeline inside a request. Each request to the server becomes a job in a queue and we return a generation id to the user. Background workers pull from the request queue and run the jobs on available GPUs. The client/user can either wait on the request (Prefer: wait), periodically poll the server for an update on the job using the generation id, or follow progress as server-sent events.

Responsibilities of individual modules

  • serve/__init__.py: the serve command. Parses flags, loads the pipeline (one copy per GPU with --data-parallel), and starts the HTTP server.
  • serve/manifest.py: the manifest is the JSON document that tells clients what the model can do. This module is responsible for parsing it and setting up the server accordingly. GMS is meant to be framework agnostic and so uses generic names, an optional x-diffusers block maps them to pipeline kwargs where they differ (steps → num_inference_steps, seed → generator). The manifest comes from a gms.json file passed with the --manifest arg, or from a gms.json in the model repo, or is derived from the pipeline's __call__ signature. At startup the server checks that every input and parameter maps to a pipeline kwarg and that the declared capabilities and outputs are supported, and refuses to start otherwise.
  • serve/app.py: the HTTP routes (FastAPI). Defines the actual endpoints of the server
  • serve/scheduler.py: Contains GenerationJob that tracks a job's status, progress and outputs. Scheduler manages the request queue and the worker threads and cleaning up stale media assets.
  • serve/backends.py: DiffusersBackend runs a job: loads the input media and LoRAs, calls the pipeline, and saves the outputs. Its step callback reports progress and stops the pipeline on cancel or --generation-timeout; modular pipelines get a block in their denoising loops that calls it.
  • serve/remote.py: --remote starts the server in an HF Job and prints its URL once it is ready.

Comment thread docs/source/en/using-diffusers/serve.md Outdated

## Generations

Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

120s here is referring to the example snippet. Prefer:wait is controlled by the user. The default is 300s


## `serve`

Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge.

Comment thread src/diffusers/commands/serve/app.py Resolved
Comment thread docs/source/en/using-diffusers/serve.md Outdated
```text
id: 1
event: progress
data: {"step": 1, "total_steps": 28, "fraction": 0.0357}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fraction of the total steps. 1/28 == 0.0357

"media_type": "image/png",
"width": 1024,
"height": 1024,
"url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, on server start up you can set the directory where the outputs are saved

diffusers-cli serve --output-dir <my folder>

"x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}}
}
},
"capabilities": ["sse", "wait", "adapters"],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a gms.json file inside the model repo.
https://huggingface.co/diffusers-internal-dev/flux2-klein-4b-gms/blob/main/gms.json

If no gms.json file is present in the repo, we list these capabilties: ["sse", "wait"] and include adapters if the pipeline can load loras.

sse - Server Side Events. This is whether the server supports sending current state of the request (queued, generating etc) to the client side. Useful for cases where the UI or frontend needs to render something while the user waits

wait - Whether the server allows Prefer: wait HTTP headers which block the request until it finishes. By default, when the server gets a request, we return a 201 status code with a request id that the client can then use to query the server to know if a generation is complete or not.

ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.

x-diffusers is meant to be a field in the GMS spec that maps parameter names from a user request to diffusers pipeline kwargs. Since GMS is meant to be framework agnostic, we need some way to route parameters that might not share our nomenclature. e.g. if I were to serve Qwen-Image but I only want to offer true CFG, I would need a way to remap the meaning of guidance scale on my service because Diffusers calls it true_cfg_scale

"x-diffusers": {"parameters": {"guidance_scale": "true_cfg_scale"}}

The x- convention is from the OpenAPI extensions. So what it is communicating here is that we are extending GMS to serve diffusers models and we need to remap these parameters in order to do that.

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot for the PR! i left some comments

pip install "diffusers[serve]"
diffusers serve black-forest-labs/FLUX.1-dev
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we show the output here?

}'
```

## Manifests

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this go before generations section? I think it reads better this way since you need to build request from it

```

Without `Prefer: wait`, the response comes back at once with `"status": "queued"`. Follow its progress as
server-sent events, or poll it:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server-sent events, or poll it:
server-sent events

can we move or poll it to where the example are? so it's a bit more clear what this example is

event: complete
data: {"id": "gen_3f9a1c2b7d4e", "status": "complete", "outputs": {...}, ...}
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Alternatively, poll it until `status` is `complete`


```bash
curl localhost:8000/v1/generations/gen_3f9a1c2b7d4e # poll
curl -X POST localhost:8000/v1/generations/gen_3f9a1c2b7d4e/cancel # cancel

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, let's not bundle cancel/delete/download into the poll example

if schema is None and kwarg == "prompt" and annotation is inspect.Parameter.empty:
schema = {"type": "string"}
if schema is None:
continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we log a debug here?


name = kwarg_parameters.get(kwarg, kwarg)
if name in parameters:
continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we log a debug here?

else:
output = "image"

def task(required: tuple[str, ...], excluded: set[str]) -> dict[str, Any]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we only generate the registered for standard pipeline? for modular, could we map them from the _workflow_map?

# ---------------------------------------------------------------------------


class Manifest:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me as POC, i wrote down some thoughts for modular side as a follow-up #15351 not merge blocking

registered = [entry for entry in _AUTO_TASKS if pipeline_cls in entry[1].values()]
returned = [kind for kind, fields in OUTPUT_FIELDS.items() if output_fields and set(fields) & output_fields]
if returned:
output = returned[0]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh this won't work for pipeline that returns audio and video together no?

Comment on lines +152 to +154
parser.add_argument(
"--model", "-m", required=require_model, help="Model id on the Hugging Face Hub or local path."
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would it be not required?


```bash
pip install "diffusers[serve]"
diffusers-cli serve black-forest-labs/FLUX.1-dev

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redacted outputs would be nice here.

diffusers serve black-forest-labs/FLUX.1-dev
```

`serve` takes the same loading and optimization flags as [`diffusers-cli run`](./cli#run). The server listens on

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also add a line saying that remote inference is also supported.


## Generations

Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here (the server caps it at 300):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the output is of type Generation object, I would make that clear here.

"x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}}
}
},
"capabilities": ["sse", "wait", "adapters"],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant.

device = self.pipeline.device if hasattr(self.pipeline, "device") else torch.device("cpu")
if request.seed is not None:
# A CUDA generator must sit on the pipeline's own GPU, which is `cuda:1` and up with --data-parallel.
kwargs["generator"] = _get_generator(request.seed, str(device) if device.type == "cuda" else device.type)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be also conditioned with --data-parallel then? Or we use torch.accelerator.current_device? Or always use CPU as the generator device?

) from e
if names:
self.pipeline.set_adapters(names, adapter_weights=[adapter["weight"] for adapter in request.adapters])
# Worker threads start on GPU 0, and with --data-parallel this pipeline may sit on another one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same. Should this be explicitly conditioned on --data-parallel?

arrays = _as_audio_arrays(value)
if arrays is None:
raise GMSError("internal", f"cannot encode output {name!r} as {media_type}: not an audio array")
# TODO(DN6): write the files with `export_to_wav` once it is added to `diffusers.utils`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #15360.

if "duration" in parameters and _frame_lattice(parameters["duration"]) is None:
raise SystemExit(
f"The GMS manifest for {self.id!r} declares `duration` on task {name!r} without a frame lattice, so "
'seconds cannot be converted to frames. Add one, e.g. "lattice": {"frames": "8k+1", "fps": 24}.'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But do we allow this kind of phrasing when specifying frames? Isn't it better to strictly keep it an integer? What am I missing?

schema = {"type": "string"}
if schema is None:
continue
if schema["type"] == "string" and re.fullmatch(r"(.*_)?prompt(_\d+)?", kwarg):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I also think these regexes are a bit fragile. So, would prefer explicit rules / conditionals.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/L PR with diff > 200 LOC tests utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants