Skip to content

Cosmos3: reuse the text-prefix keys and values across denoising steps - #15287

Open
chsu-amd wants to merge 6 commits into
huggingface:mainfrom
chsu-amd:perf/cosmos3-text-kv-reuse
Open

chsu-amd wants to merge 6 commits into
huggingface:mainfrom
chsu-amd:perf/cosmos3-text-kv-reuse

Conversation

@chsu-amd

@chsu-amd chsu-amd commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #15286

Cosmos3OmniTransformer ran the text prefix (und pathway) of every decoder layer on every transformer call, although its keys and values are fixed within one pipeline call. This PR caches each layer's und keys and values on the first call of each CFG branch in a Cosmos3KVCache (same pattern as QwenImage21KVCache) and skips the und pathway afterwards; Cosmos3OmniPipeline turns this on with a new use_kv_cache=True argument (off where reuse is not safe, e.g. cache hooks or mixed precision).

Benchmark (nvidia/Cosmos3-Edge, model card i2v call, 480x832, 121 frames, 20 steps, bf16, safety checker on, eager; main e84bda1 vs this PR at c1fe31c; 8 alternating pairs, one process per run, median):

Platform PyTorch main this PR Speedup
NVIDIA RTX 5090 2.14.1+cu130 55.461 s 53.069 s +4.5%
AMD Radeon AI PRO R9700 (gfx1201) 2.16.0a0+rocm10.3 77.422 s 73.215 s +5.7%

Output bit-identical to main on both GPUs (all 8 pairs).

New tests check the reuse bit for bit against the full pass (transformer with and without CFG, pipeline over two calls) and that the und pathway runs once per CFG branch. Existing Cosmos3 tests: no new failures on CUDA or CPU.

Self-review notes (self-review skill)

READY (run on c1fe31c; f124425 fixes its one-line finding; a1ed7a5 clears the caches like Flux2 / GlmImage). ruff check / ruff format --check pass. Left for your call:

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? (important for complex PRs)
  • Was this discussed/approved via a GitHub issue or GitHub Discussions? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team we work with) of the model/pipeline this PR relates to?

Who can review?

cc @yiyixuxu @dg845

Cosmos3OmniTransformer ran the und pathway (the causal text prefix) of
every decoder layer on every transformer call. The und tokens attend
only to und tokens and start from embed_tokens(input_ids), and
Cosmos3OmniPipeline passes the same input_ids / position_ids on every
step, so every layer's und keys and values are the same on every step
of one pipeline call. The gen pathway reads them only through the
key/value concatenation.

Store each layer's post-RoPE und keys and values on the first
transformer call of each CFG branch and skip the und pathway on the
later calls. The pipeline enables this for one __call__ through a
private transformer attribute, cleared in the existing finally; it
stays off with mixed precision, cache hooks, custom attention
processors, context parallelism and gradients.

Add tests that check the transformer outputs with the reused keys and
values, bit for bit, against the full pass (with and without CFG, with
and without the und key norm for gen), that the und pathway runs once
per CFG branch, and that the pipeline output over two calls with
different prompts equals the output with the reuse disabled.

Bit-identical nvidia/Cosmos3-Edge pipeline output in eager, bf16, on
RTX 5090 at the model card settings (480x832, 121 frames, 20 steps,
guidance 6.0).

Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
@chsu-amd
chsu-amd marked this pull request as ready for review November 14, 2026 07:57
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR! the optimization makes sense to me
cc cosmos team here @yzhautouskay, let me know what you think as well

design wise, does it make sense to follow the pattern we do for other models with Kv cache? (e.g. Flux2KvCache https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux2.py#L88)

Follow the QwenImage21KVCache / Flux2KVCache pattern instead of the
private _und_kv_cache attribute keyed on tensor ids.

Add Cosmos3KVLayerCache / Cosmos3KVCache. Cosmos3OmniTransformer.forward
takes kv_cache and kv_cache_mode ("extract" stores each layer's
post-RoPE und keys and values, "cached" reads them and skips the und
pathway). Cosmos3OmniPipeline builds one cache per CFG branch behind a
new use_kv_cache argument (default True), extracting on the first step;
it stays off with mixed precision, cache / layer-skip hooks (also when
added with apply_*), context parallelism and custom attention
processors. Without a cache, _run_decoder_stack is called with three
arguments as before, so cache hooks that replace it keep working.

Tests drive kv_cache / kv_cache_mode directly, check that the pipeline
uses the cache (text MLP call count) and matches use_kv_cache=False bit
for bit, and that it falls back under apply_sea_cache.

Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
Drop the cond / uncond caches when the denoising loop ends so they do
not raise the peak memory of the VAE decode.

Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
If a step-end callback turns CFG on mid-loop, the uncond pass has no
cache; run it uncached instead of raising.

Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
@chsu-amd

Copy link
Copy Markdown
Contributor Author

Thanks @yiyixuxu, switched to that pattern, modeled on QwenImage21KVCache: Cosmos3KVCache with kv_cache / kv_cache_mode on the transformer, and use_kv_cache=True on the pipeline. At c1fe31c the model card i2v output is still bit-identical to main on RTX 5090 and AMD Radeon AI PRO R9700 (8 pairs each); the description has the updated numbers. The last commit, f124425, only runs the uncond pass without the cache when a step-end callback turns CFG on mid-loop, so the timed path is unchanged.

@chsu-amd

Copy link
Copy Markdown
Contributor Author

Follow-up: I'm aligning the cache cleanup with the Flux2 / GlmImage pattern (clear() after the denoising loop) and re-running the checks. I'll post here once it's updated.

Follow Flux2 Klein KV / GlmImage: add clear() to Cosmos3KVLayerCache /
Cosmos3KVCache and call it after the loop, before decoding, instead of
dropping the references in the finally block. Test that both caches
are cleared.

Signed-off-by: Hsu, Chan Fu <ChanFu.Hsu@amd.com>
@chsu-amd

Copy link
Copy Markdown
Contributor Author

Updated in a1ed7a5: Cosmos3KVLayerCache / Cosmos3KVCache now have clear(), and the pipeline calls it after the denoising loop, before decoding, as Flux2 Klein KV / GlmImage do. On RTX 5090 the model card i2v output is still bit-identical to main and the peak memory is unchanged (12.39 GiB allocated in both); the Cosmos3 tests show no new failures vs main on CUDA or CPU, and ruff passes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cosmos3 Omni transformer recomputes the text-prefix keys and values on every denoising step

3 participants