Repository navigation
Conversation
# Conflicts: # docs/source/en/using-diffusers/cli.md
SunMarc
left a comment
There was a problem hiding this comment.
Excited to have this ! Just did a pass over the doc for now
|
|
||
| ```bash | ||
| pip install "diffusers[serve]" | ||
| diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16 |
There was a problem hiding this comment.
We can keep --model if you want but i think it is better is it is optional. makes command shorter and most users will expect the model to follow.
As Lysandre pointed, diffusers serve would be nice also.
Lastly, for this specific model, do we need to specify the dtype ? In general, I expect that the model in its original dtype which is probably bf16.
| diffusers-cli serve --model black-forest-labs/FLUX.1-dev --dtype bf16 | |
| diffusers-cli serve black-forest-labs/FLUX.1-dev --dtype bf16 |
| `serve` takes the same loading and optimization flags as [`run`](#run) (`--dtype`, `--device-map`, | ||
| `--cpu-offload`, `--compile`, `--lora`, ...), plus `--host` (default `127.0.0.1`) and `--port` (default `8000`). |
There was a problem hiding this comment.
maybe restrict some the optimization flags unless we really want them ?
There was a problem hiding this comment.
@DN6 I think this is following run so those flags are already supported?
| [Modular repos](../modular_diffusers/overview) are detected the same way as in `run` and served in-process. They | ||
| report no step progress, and a cancel request only takes effect once the pipeline call returns. | ||
|
|
||
| | Flag | Effect | | ||
| |---|---| | ||
| | `--default-seed` | Seed used for requests that do not set one. Without it, each such request gets a random seed. | | ||
| | `--enable-cors` | Allow cross-origin requests from any origin, so a browser app on another host can call the server. | | ||
| | `--public-url` | Base URL clients reach the server at, used to build output file links. Set it behind a reverse proxy. | | ||
| | `--log-level` | Logging level for diffusers and the HTTP server (`debug`, `info`, `warning`, `error`, `critical`). | |
There was a problem hiding this comment.
Maybe we should have serving its own dedicated section in the docs ? The doc is really dense so I think ppl will have a hard time finding the info they want. Maybe something like in transformers would be prefered ? https://huggingface.co/docs/transformers/v5.19.0/en/serve-cli/serving
Btw, I think it is fine to not explain all options of the server that users will never really set them. The agent will be able to get them nevertheless
There was a problem hiding this comment.
Sounds good 👍🏽 Yeah, we can move to separate section and have options under --help
| ```bash | ||
| # What the model accepts: tasks, inputs, parameters with types and defaults. | ||
| curl localhost:8000/v1/models/black-forest-labs/FLUX.1-dev | ||
|
|
||
| # Create a generation and wait up to 120 seconds for it. | ||
| curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{ | ||
| "model": "black-forest-labs/FLUX.1-dev", | ||
| "inputs": {"prompt": "an astronaut riding a horse"}, | ||
| "parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42} | ||
| }' | ||
|
|
||
| # Follow step progress as server-sent events, or poll the generation. | ||
| curl -N localhost:8000/v1/generations/<id>/events | ||
| curl localhost:8000/v1/generations/<id> | ||
|
|
||
| # Cancel a queued or running generation, or delete it together with its files. | ||
| curl -X POST localhost:8000/v1/generations/<id>/cancel | ||
| curl -X DELETE localhost:8000/v1/generations/<id> | ||
| ``` |
There was a problem hiding this comment.
it would do a section dedicated to v1/generations
|
|
||
| ```bash | ||
| diffusers-cli generate --sandbox-id <id> --inputs '{"prompt": "a cat on the moon"}' | ||
| ``` |
There was a problem hiding this comment.
we can't send directly an http request directly to the sandbox ?
There was a problem hiding this comment.
We can. This was more of a convenience option to construct a GMS spec compatible request. But perhaps this is overkill and we can stick to just HTTP requests.
| ### Remote serving (`--remote`) | ||
|
|
||
| `--remote` starts the server in a [Hugging Face Sandbox](https://huggingface.co/docs/huggingface_hub/guides/sandbox) |
| With `--backend sglang` or `--backend vllm`, the sandbox starts from the engine's own image | ||
| (`lmsysorg/sglang:latest` or `vllm/vllm-omni:latest`) unless `--image` names another, so the engine does not need | ||
| installing: | ||
|
|
||
| ```bash | ||
| diffusers-cli serve --model black-forest-labs/FLUX.2-klein-4B --backend sglang --remote --flavor a100-large | ||
| ``` |
There was a problem hiding this comment.
Not sure about adding right now the slgang and vllm backend in diffusers serve
There was a problem hiding this comment.
So the motivation was to allow users to take advantage of features that exist in these libs that diffusers doesn't have. Any strong reason to exclude?
| `--backend sglang` and `--backend vllm` run the model in [SGLang Diffusion](https://docs.sglang.io/) or | ||
| [vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni) instead of in-process. `serve` starts `sglang serve` or | ||
| `vllm serve --omni` as a child process on a local port, waits for it to become healthy, and translates GMS | ||
| requests to the engine's OpenAI-compatible image and video routes. Clients only ever talk to the GMS endpoints | ||
| of `serve`. The engine must be installed separately. |
There was a problem hiding this comment.
if we can add a new endpoint on their side with GMS, that would be better instead no ?
There was a problem hiding this comment.
Yes. This is the ideal outcome. I saw this as a sort of intermediate step towards that goal.
| ## `generate` | ||
|
|
||
| Send a generation to a server started with [`serve`](#serve), or to any other Generative Media Spec server, and | ||
| save its outputs. Unlike [`run`](#run), `generate` loads no pipeline: it reads the model's discovery document, | ||
| submits the request, polls the generation until it finishes, and downloads the files. | ||
|
|
||
| ```bash | ||
| diffusers-cli generate --url http://localhost:8000 --inputs '{"prompt": "a cat on the moon"}' | ||
|
|
||
| diffusers-cli generate --url http://localhost:8000 \ | ||
| --inputs '{"prompt": "make the fur grey", "image": "cat.png"}' \ | ||
| --parameters '{"steps": 8, "seed": 0}' --output outputs/ | ||
| ``` |
There was a problem hiding this comment.
Not sure about generate, i don't think we should encourage users to use this but just rely on sending directly http request no ?
There was a problem hiding this comment.
Yeah, I added as a convenience function. But plain HTTP is cleaner.
… into backends.py
…nd no longer reads it per request
| parameters = blocks_signature(self.pipeline.blocks).parameters | ||
| else: | ||
| parameters = inspect.signature(type(self.pipeline).__call__).parameters | ||
| self._reports_steps = "callback_on_step_end" in parameters |
There was a problem hiding this comment.
so i think util the first class streaming PR is merged on modular side, we could add callbacks to modular pipelines programmatically, I included an example/guide here https://huggingface.co/docs/diffusers/main/en/using-diffusers/callback#modular-pipelines
sayakpaul
left a comment
There was a problem hiding this comment.
Reviewed part of the PR. This is a large PR without any commentary of how some things are implemented.
I think it would be helpful to have some explanation on the design decisions that were taken to do the server work. For example, I don't understand the separation between DiffusersBackend and ParallelBackend (and similar others).
Additionally, it's not clear to what extent we want the users to customize the server itself. For example, do we want to expose wny flags to control concurrency?
|
|
||
| ## `serve` | ||
|
|
||
| Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide. |
There was a problem hiding this comment.
I don't think people know about GMS as it's not a standard. Can we hyperlink?
There was a problem hiding this comment.
Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge.
|
|
||
| ## Generations | ||
|
|
||
| Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here: |
There was a problem hiding this comment.
Can users override this limit of 120s?
There was a problem hiding this comment.
120s here is referring to the example snippet. Prefer:wait is controlled by the user. The default is 300s
| curl localhost:8000/v1/generations -H 'Prefer: wait=120' -d '{ | ||
| "model": "black-forest-labs/FLUX.1-dev", | ||
| "inputs": {"prompt": "an astronaut riding a horse"}, | ||
| "parameters": {"width": 1024, "height": 1024, "steps": 28, "seed": 42} | ||
| }' |
There was a problem hiding this comment.
Perhaps also show how to do it in Python using requests?
| "media_type": "image/png", | ||
| "width": 1024, | ||
| "height": 1024, | ||
| "url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png" |
There was a problem hiding this comment.
For local servers, can the users specify the results directory?
There was a problem hiding this comment.
Yes, on server start up you can set the directory where the outputs are saved
diffusers-cli serve --output-dir <my folder>
| ```text | ||
| id: 1 | ||
| event: progress | ||
| data: {"step": 1, "total_steps": 28, "fraction": 0.0357} |
There was a problem hiding this comment.
fraction of the total steps. 1/28 == 0.0357
| "x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}} | ||
| } | ||
| }, | ||
| "capabilities": ["sse", "wait", "adapters"], |
There was a problem hiding this comment.
What are sse and wait? Do we treat ControlNets as adapters as well? How do we decide the capabilities based on a model id?
Also, what is x-diffusers?
There was a problem hiding this comment.
Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a gms.json file inside the model repo.
https://huggingface.co/diffusers-internal-dev/flux2-klein-4b-gms/blob/main/gms.json
If no gms.json file is present in the repo, we list these capabilties: ["sse", "wait"] and include adapters if the pipeline can load loras.
sse - Server Side Events. This is whether the server supports sending current state of the request (queued, generating etc) to the client side. Useful for cases where the UI or frontend needs to render something while the user waits
wait - Whether the server allows Prefer: wait HTTP headers which block the request until it finishes. By default, when the server gets a request, we return a 201 status code with a request id that the client can then use to query the server to know if a generation is complete or not.
ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.
x-diffusers is meant to be a field in the GMS spec that maps parameter names from a user request to diffusers pipeline kwargs. Since GMS is meant to be framework agnostic, we need some way to route parameters that might not share our nomenclature. e.g. if I were to serve Qwen-Image but I only want to offer true CFG, I would need a way to remap the meaning of guidance scale on my service because Diffusers calls it true_cfg_scale
"x-diffusers": {"parameters": {"guidance_scale": "true_cfg_scale"}}
The x- convention is from the OpenAPI extensions. So what it is communicating here is that we are extending GMS to serve diffusers models and we need to remap these parameters in order to do that.
There was a problem hiding this comment.
Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant.
|
|
||
| ## Remote serving | ||
|
|
||
| `--remote` runs the server in a [Hugging Face Job](https://huggingface.co/docs/huggingface_hub/guides/jobs) and |
There was a problem hiding this comment.
Does remote serving follow the same settings as run? If not, why not?
| ) from e | ||
|
|
||
| from .app import create_app | ||
| from .backends import DiffusersBackend, ParallelBackend, backend_capabilities |
There was a problem hiding this comment.
What is the advantage of doing it like this?
DN6
left a comment
There was a problem hiding this comment.
@sayakpaul Reduced the scope of the PR here to make review more manageable. Parallelism support will be added in a follow up. Sharing an overview of how the server works
Since a pipeline calls takes much longer than a normal HTTP request normally stays open, the server doesn't run the pipeline inside a request. Each request to the server becomes a job in a queue and we return a generation id to the user. Background workers pull from the request queue and run the jobs on available GPUs. The client/user can either wait on the request (Prefer: wait), periodically poll the server for an update on the job using the generation id, or follow progress as server-sent events.
Responsibilities of individual modules
serve/__init__.py: theservecommand. Parses flags, loads the pipeline (one copy per GPU with--data-parallel), and starts the HTTP server.serve/manifest.py: the manifest is the JSON document that tells clients what the model can do. This module is responsible for parsing it and setting up the server accordingly. GMS is meant to be framework agnostic and so uses generic names, an optionalx-diffusersblock maps them to pipeline kwargs where they differ (steps→num_inference_steps,seed→generator). The manifest comes from agms.jsonfile passed with the--manifestarg, or from agms.jsonin the model repo, or is derived from the pipeline's__call__signature. At startup the server checks that every input and parameter maps to a pipeline kwarg and that the declared capabilities and outputs are supported, and refuses to start otherwise.serve/app.py: the HTTP routes (FastAPI). Defines the actual endpoints of the serverserve/scheduler.py: ContainsGenerationJobthat tracks a job's status, progress and outputs.Schedulermanages the request queue and the worker threads and cleaning up stale media assets.serve/backends.py:DiffusersBackendruns a job: loads the input media and LoRAs, calls the pipeline, and saves the outputs. Its step callback reports progress and stops the pipeline on cancel or--generation-timeout; modular pipelines get a block in their denoising loops that calls it.serve/remote.py:--remotestarts the server in an HF Job and prints its URL once it is ready.
|
|
||
| ## Generations | ||
|
|
||
| Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here: |
There was a problem hiding this comment.
120s here is referring to the example snippet. Prefer:wait is controlled by the user. The default is 300s
|
|
||
| ## `serve` | ||
|
|
||
| Serve a pipeline over HTTP with the Generative Media Spec API. See [Serving](./serve) for the full guide. |
There was a problem hiding this comment.
Yes. The GMS spec repo isn't public yet, but it will be linked once when this PR is ready to merge.
| ```text | ||
| id: 1 | ||
| event: progress | ||
| data: {"step": 1, "total_steps": 28, "fraction": 0.0357} |
There was a problem hiding this comment.
fraction of the total steps. 1/28 == 0.0357
| "media_type": "image/png", | ||
| "width": 1024, | ||
| "height": 1024, | ||
| "url": "http://localhost:8000/v1/generations/gen_3f9a1c2b7d4e/files/image-0.png" |
There was a problem hiding this comment.
Yes, on server start up you can set the directory where the outputs are saved
diffusers-cli serve --output-dir <my folder>
| "x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}} | ||
| } | ||
| }, | ||
| "capabilities": ["sse", "wait", "adapters"], |
There was a problem hiding this comment.
Capabilities here refer to server capabilities not model capabilities. So we do not determine these from model ids. It needs to be defined before hand in a gms.json file inside the model repo.
https://huggingface.co/diffusers-internal-dev/flux2-klein-4b-gms/blob/main/gms.json
If no gms.json file is present in the repo, we list these capabilties: ["sse", "wait"] and include adapters if the pipeline can load loras.
sse - Server Side Events. This is whether the server supports sending current state of the request (queued, generating etc) to the client side. Useful for cases where the UI or frontend needs to render something while the user waits
wait - Whether the server allows Prefer: wait HTTP headers which block the request until it finishes. By default, when the server gets a request, we return a 201 status code with a request id that the client can then use to query the server to know if a generation is complete or not.
ControlNets are not treated as adapters. Currently only LoRAs are treated as adapters since they can be loaded per request.
x-diffusers is meant to be a field in the GMS spec that maps parameter names from a user request to diffusers pipeline kwargs. Since GMS is meant to be framework agnostic, we need some way to route parameters that might not share our nomenclature. e.g. if I were to serve Qwen-Image but I only want to offer true CFG, I would need a way to remap the meaning of guidance scale on my service because Diffusers calls it true_cfg_scale
"x-diffusers": {"parameters": {"guidance_scale": "true_cfg_scale"}}
The x- convention is from the OpenAPI extensions. So what it is communicating here is that we are extending GMS to serve diffusers models and we need to remap these parameters in order to do that.
yiyixuxu
left a comment
There was a problem hiding this comment.
thanks a lot for the PR! i left some comments
| pip install "diffusers[serve]" | ||
| diffusers serve black-forest-labs/FLUX.1-dev | ||
| ``` | ||
|
|
There was a problem hiding this comment.
can we show the output here?
| }' | ||
| ``` | ||
|
|
||
| ## Manifests |
There was a problem hiding this comment.
should this go before generations section? I think it reads better this way since you need to build request from it
| ``` | ||
|
|
||
| Without `Prefer: wait`, the response comes back at once with `"status": "queued"`. Follow its progress as | ||
| server-sent events, or poll it: |
There was a problem hiding this comment.
| server-sent events, or poll it: | |
| server-sent events |
can we move or poll it to where the example are? so it's a bit more clear what this example is
| event: complete | ||
| data: {"id": "gen_3f9a1c2b7d4e", "status": "complete", "outputs": {...}, ...} | ||
| ``` | ||
|
|
There was a problem hiding this comment.
| Alternatively, poll it until `status` is `complete` |
|
|
||
| ```bash | ||
| curl localhost:8000/v1/generations/gen_3f9a1c2b7d4e # poll | ||
| curl -X POST localhost:8000/v1/generations/gen_3f9a1c2b7d4e/cancel # cancel |
There was a problem hiding this comment.
also, let's not bundle cancel/delete/download into the poll example
| if schema is None and kwarg == "prompt" and annotation is inspect.Parameter.empty: | ||
| schema = {"type": "string"} | ||
| if schema is None: | ||
| continue |
There was a problem hiding this comment.
can we log a debug here?
|
|
||
| name = kwarg_parameters.get(kwarg, kwarg) | ||
| if name in parameters: | ||
| continue |
There was a problem hiding this comment.
can we log a debug here?
| else: | ||
| output = "image" | ||
|
|
||
| def task(required: tuple[str, ...], excluded: set[str]) -> dict[str, Any]: |
There was a problem hiding this comment.
are we only generate the registered for standard pipeline? for modular, could we map them from the _workflow_map?
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| class Manifest: |
There was a problem hiding this comment.
looks good to me as POC, i wrote down some thoughts for modular side as a follow-up #15351 not merge blocking
| registered = [entry for entry in _AUTO_TASKS if pipeline_cls in entry[1].values()] | ||
| returned = [kind for kind, fields in OUTPUT_FIELDS.items() if output_fields and set(fields) & output_fields] | ||
| if returned: | ||
| output = returned[0] |
There was a problem hiding this comment.
ohh this won't work for pipeline that returns audio and video together no?
| parser.add_argument( | ||
| "--model", "-m", required=require_model, help="Model id on the Hugging Face Hub or local path." | ||
| ) |
There was a problem hiding this comment.
When would it be not required?
|
|
||
| ```bash | ||
| pip install "diffusers[serve]" | ||
| diffusers-cli serve black-forest-labs/FLUX.1-dev |
There was a problem hiding this comment.
Redacted outputs would be nice here.
| diffusers serve black-forest-labs/FLUX.1-dev | ||
| ``` | ||
|
|
||
| `serve` takes the same loading and optimization flags as [`diffusers-cli run`](./cli#run). The server listens on |
There was a problem hiding this comment.
I would also add a line saying that remote inference is also supported.
|
|
||
| ## Generations | ||
|
|
||
| Create a generation. `Prefer: wait` holds the response until it finishes, for up to 120 seconds here (the server caps it at 300): |
There was a problem hiding this comment.
If the output is of type Generation object, I would make that clear here.
| "x-diffusers": {"parameters": {"steps": "num_inference_steps", "seed": "generator"}} | ||
| } | ||
| }, | ||
| "capabilities": ["sse", "wait", "adapters"], |
There was a problem hiding this comment.
Thanks! Are we supposed to document these bits here or under GMS? Seems to be very relevant.
| device = self.pipeline.device if hasattr(self.pipeline, "device") else torch.device("cpu") | ||
| if request.seed is not None: | ||
| # A CUDA generator must sit on the pipeline's own GPU, which is `cuda:1` and up with --data-parallel. | ||
| kwargs["generator"] = _get_generator(request.seed, str(device) if device.type == "cuda" else device.type) |
There was a problem hiding this comment.
Should this be also conditioned with --data-parallel then? Or we use torch.accelerator.current_device? Or always use CPU as the generator device?
| ) from e | ||
| if names: | ||
| self.pipeline.set_adapters(names, adapter_weights=[adapter["weight"] for adapter in request.adapters]) | ||
| # Worker threads start on GPU 0, and with --data-parallel this pipeline may sit on another one. |
There was a problem hiding this comment.
Same. Should this be explicitly conditioned on --data-parallel?
| arrays = _as_audio_arrays(value) | ||
| if arrays is None: | ||
| raise GMSError("internal", f"cannot encode output {name!r} as {media_type}: not an audio array") | ||
| # TODO(DN6): write the files with `export_to_wav` once it is added to `diffusers.utils`. |
| if "duration" in parameters and _frame_lattice(parameters["duration"]) is None: | ||
| raise SystemExit( | ||
| f"The GMS manifest for {self.id!r} declares `duration` on task {name!r} without a frame lattice, so " | ||
| 'seconds cannot be converted to frames. Add one, e.g. "lattice": {"frames": "8k+1", "fps": 24}.' |
There was a problem hiding this comment.
But do we allow this kind of phrasing when specifying frames? Isn't it better to strictly keep it an integer? What am I missing?
| schema = {"type": "string"} | ||
| if schema is None: | ||
| continue | ||
| if schema["type"] == "string" and re.fullmatch(r"(.*_)?prompt(_\d+)?", kwarg): |
There was a problem hiding this comment.
Agreed. I also think these regexes are a bit fragile. So, would prefer explicit rules / conditionals.
What does this PR do?
Introduce
diffusers-cli serveto serve a diffusers compatible model/pipeline based on the GMS spec.pip install "diffusers[serve]" diffusers serve black-forest-labs/FLUX.1-devFeatures
diffusers-cli run--data-parallel(1 pipeline per GPU) and--context-parallel(tensor parallel will be added in a follow up)Fixes # (issue)
Before submitting
self-reviewskill on the diff?documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.