Repository navigation
Conversation
Cosmos3OmniTransformer ran the und pathway (the causal text prefix) of every decoder layer on every transformer call. The und tokens attend only to und tokens and start from embed_tokens(input_ids), and Cosmos3OmniPipeline passes the same input_ids / position_ids on every step, so every layer's und keys and values are the same on every step of one pipeline call. The gen pathway reads them only through the key/value concatenation. Store each layer's post-RoPE und keys and values on the first transformer call of each CFG branch and skip the und pathway on the later calls. The pipeline enables this for one __call__ through a private transformer attribute, cleared in the existing finally; it stays off with mixed precision, cache hooks, custom attention processors, context parallelism and gradients. Add tests that check the transformer outputs with the reused keys and values, bit for bit, against the full pass (with and without CFG, with and without the und key norm for gen), that the und pathway runs once per CFG branch, and that the pipeline output over two calls with different prompts equals the output with the reuse disabled. Bit-identical nvidia/Cosmos3-Edge pipeline output in eager, bf16, on RTX 5090 at the model card settings (480x832, 121 frames, 20 steps, guidance 6.0). Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yiyixuxu
left a comment
There was a problem hiding this comment.
thanks for the PR! the optimization makes sense to me
cc cosmos team here @yzhautouskay, let me know what you think as well
design wise, does it make sense to follow the pattern we do for other models with Kv cache? (e.g. Flux2KvCache https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py#L88)
Follow the QwenImage21KVCache / Flux2KVCache pattern instead of the
private _und_kv_cache attribute keyed on tensor ids.
Add Cosmos3KVLayerCache / Cosmos3KVCache. Cosmos3OmniTransformer.forward
takes kv_cache and kv_cache_mode ("extract" stores each layer's
post-RoPE und keys and values, "cached" reads them and skips the und
pathway). Cosmos3OmniPipeline builds one cache per CFG branch behind a
new use_kv_cache argument (default True), extracting on the first step;
it stays off with mixed precision, cache / layer-skip hooks (also when
added with apply_*), context parallelism and custom attention
processors. Without a cache, _run_decoder_stack is called with three
arguments as before, so cache hooks that replace it keep working.
Tests drive kv_cache / kv_cache_mode directly, check that the pipeline
uses the cache (text MLP call count) and matches use_kv_cache=False bit
for bit, and that it falls back under apply_sea_cache.
Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
Drop the cond / uncond caches when the denoising loop ends so they do not raise the peak memory of the VAE decode. Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
If a step-end callback turns CFG on mid-loop, the uncond pass has no cache; run it uncached instead of raising. Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
|
Thanks @yiyixuxu, switched to that pattern, modeled on |
|
Follow-up: I'm aligning the cache cleanup with the Flux2 / GlmImage pattern ( |
Follow Flux2 Klein KV / GlmImage: add clear() to Cosmos3KVLayerCache / Cosmos3KVCache and call it after the loop, before decoding, instead of dropping the references in the finally block. Test that both caches are cleared. Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
|
Updated in a1ed7a5: |
What does this PR do?
Fixes #15286
Cosmos3OmniTransformerran the text prefix (und pathway) of every decoder layer on every transformer call, although its keys and values are fixed within one pipeline call. This PR caches each layer's und keys and values on the first call of each CFG branch in aCosmos3KVCache(same pattern asQwenImage21KVCache) and skips the und pathway afterwards;Cosmos3OmniPipelineturns this on with a newuse_kv_cache=Trueargument (off where reuse is not safe, e.g. cache hooks or mixed precision).Benchmark (
nvidia/Cosmos3-Edge, model card i2v call, 480x832, 121 frames, 20 steps, bf16, safety checker on, eager; main e84bda1 vs this PR at c1fe31c; 8 alternating pairs, one process per run, median):Output bit-identical to main on both GPUs (all 8 pairs).
New tests check the reuse bit for bit against the full pass (transformer with and without CFG, pipeline over two calls) and that the und pathway runs once per CFG branch. Existing Cosmos3 tests: no new failures on CUDA or CPU.
Self-review notes (
self-reviewskill)READY (run on c1fe31c; f124425 fixes its one-line finding; a1ed7a5 clears the caches like Flux2 / GlmImage).
ruff check/ruff format --checkpass. Left for your call:QwenImage21Pipeline; unsupported setups turn it off silently, as QwenImage21 does forcausal_condition.kv_cache,forwardre-checks hooks and context parallelism on every call._has_kv_cache_incompatible_hooks(),_cp_shard_fn).use_kv_cacheyet.torch.compile(compiled output is not bit-equal to eager on main either), other Cosmos3 checkpoints; the modular pipeline does not enable reuse.Before submitting
self-reviewskill on the diff?documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
cc @yiyixuxu @dg845