`Cosmos2VideoToWorldPipeline` applies FPS RoPE modulation that the released Cosmos-Predict2 checkpoints disable natively (default fps=16 → temporal RoPE ×1.5)

[rebel-seinpark]

Describe the bug

CosmosRotaryPosEmbed scales its temporal frequencies by FPS whenever fps is not None:

https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_cosmos.py#L505-L512

if fps is None:
    emb_t = torch.outer(seq[: pe_size[0]], temporal_freqs)
else:
    emb_t = torch.outer(seq[: pe_size[0]] / fps * self.base_fps, temporal_freqs)  # base_fps = 24

and Cosmos2VideoToWorldPipeline always feeds it — fps: int = 16 by default:

https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/cosmos/pipeline_cosmos2_video2world.py#L498
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/cosmos/pipeline_cosmos2_video2world.py#L713

However, in the reference implementation this modulation sits behind an enable_fps_modulation
flag, and the released Cosmos-Predict2 checkpoints ship with it turned off:

• rope implementation (guarded): https://github.com/nvidia-cosmos/cosmos-predict2/blob/main/cosmos_predict2/models/text2image_dit.py#L541-L552
• 2B Video2World net config: https://github.com/nvidia-cosmos/cosmos-predict2/blob/main/cosmos_predict2/configs/base/config_video2world.py#L117 (rope_enable_fps_modulation=False)
• 14B Video2World net config: same file, L207
• Text2Image nets: https://github.com/nvidia-cosmos/cosmos-predict2/blob/main/cosmos_predict2/configs/base/config_text2image.py#L110 (also L231, L309)

So natively these models use raw integer temporal RoPE positions and ignore FPS entirely,
while the diffusers port has no such flag and, with the pipeline default fps=16 against
base_fps=24, runs every denoising step with temporal RoPE positions stretched by
24/16 = 1.5× relative to how the checkpoints were trained. We traced this while
investigating consistently odd motion in default Video2World outputs.

Consequences with the current defaults:

• Every default Cosmos2VideoToWorldPipeline run diverges from the reference implementation.
• The fps argument changes the generated content in diffusers, whereas natively it only
  affects the exported container framerate.

Not affected:

• Cosmos2TextToImagePipeline — single frame, so the temporal position is 0 either way.
• Cosmos2_5_PredictBasePipeline / Cosmos2_5_TransferPipeline — they don't take an fps
  argument, so the transformer sees fps=None and uses raw positions, matching the native
  configs (which also disable the modulation).
• The Cosmos-Predict1 pipelines are out of scope here; this report is only about the
  Cosmos-Predict2 checkpoints.

Suggested fix

Either of:

1. Port enable_fps_modulation into CosmosTransformer3DModel's config (default False for
   the converted Cosmos-Predict2 checkpoints), mirroring the reference nets — the faithful fix; or
2. Stop passing fps to the transformer in Cosmos2VideoToWorldPipeline (or change the
   default to fps=24, which makes the scale factor 1), keeping fps for video export only.

Reproduction

import torch
from diffusers import Cosmos2VideoToWorldPipeline
from diffusers.utils import load_image

pipe = Cosmos2VideoToWorldPipeline.from_pretrained(
    "nvidia/Cosmos-Predict2-2B-Video2World", torch_dtype=torch.bfloat16
).to("cuda")

image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/yellow-scrubber.png"
)

# identical seeds, only fps differs — natively these would be identical, here they are not
common = dict(image=image, prompt="a yellow scrubber cleaning a plate")
video_16 = pipe(**common, fps=16, generator=torch.Generator().manual_seed(1)).frames[0]
video_24 = pipe(**common, fps=24, generator=torch.Generator().manual_seed(1)).frames[0]
# video_24 corresponds to the un-modulated (native) temporal RoPE; video_16 is the default

Logs

System Info

• diffusers: main (also reproduces on 0.38.0)
• the referenced code is identical on current main (line links above)

Who can help?

@a-r-r-o-w @yiyixuxu