Skip to content

Commit 4e91230

Browse files
[LTX-2.5] Match the reference slot rule for frames before the first keyframe plane
1 parent 28fb220 commit 4e91230

3 files changed

Lines changed: 38 additions & 12 deletions

File tree

‎docs/source/en/api/models/ltx2_diffusion_decoder.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -98,9 +98,9 @@ video = decoder.decode(
9898
```
9999

100100
The planes go through the same weights as the video. In every attention layer each video position also attends to the
101-
same spatial window on its two nearest planes, and each plane to the same window on its two nearest frames. This joint
102-
attention runs on a built-in PyTorch implementation whatever attention processor is set, since neither FlexAttention's
103-
block mask nor NATTEN expresses it.
101+
same spatial window on its two nearest planes (frames before the first plane attend to that plane alone), and each
102+
plane to the same window on its two nearest frames. This joint attention runs on a built-in PyTorch implementation
103+
whatever attention processor is set, since neither FlexAttention's block mask nor NATTEN expresses it.
104104

105105
Checkpoints trained for keyframe decoding also carry a learned tag added to the plane latents, which
106106
`decoder_keyframe_type_embedding=True` creates as `decoder.type_emb`. Decoding with keyframe planes on a checkpoint that

‎src/diffusers/models/autoencoders/ltx2_diffusion_decoder.py‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,8 @@ def mask_mod(batch_idx, head_idx, q_idx, kv_idx):
9696
# A keyframe decode carries a second stream through the decoder: a stack of single-frame latent *planes* `(B, P, H,
9797
# W, C)` whose plane axis sits in the video's temporal slot. Every weight is shared between the two streams, and they
9898
# only mix inside one joint neighborhood-attention softmax, where each video query also sees the same spatial window
99-
# on its two nearest planes and each plane query also sees its two nearest video frames.
99+
# on its two nearest planes (only the first plane, for frames before it) and each plane query also sees its two nearest
100+
# video frames.
100101
# --------------------------------------------------------------------------------------------------------------------
101102

102103
# Keyframe planes visible to one video query, and video frames visible to one plane query.
@@ -554,9 +555,9 @@ def _joint_neighborhood_attention(
554555
"""Joint neighborhood attention over a video volume and a stack of keyframe planes, one softmax per query.
555556
556557
A video query at `(t, h, w)` attends to its `Kt x Kh x Kw` video window plus the `Kh x Kw` window at the same `(h,
557-
w)` on each of its `_KEYFRAME_CONTEXT_SLOTS` nearest planes, ranked by `|t - keyframe_times|` whatever `Kt` is. A
558-
plane query attends to the `Kh x Kw` window on its own plane (there is no plane-to-plane attention) plus the same
559-
window on each of its nearest video frames.
558+
w)` on each of its `_KEYFRAME_CONTEXT_SLOTS` nearest planes, ranked by `|t - keyframe_times|` whatever `Kt` is; a
559+
frame before the first plane sees that plane only. A plane query attends to the `Kh x Kw` window on its own plane
560+
(there is no plane-to-plane attention) plus the same window on each of its nearest video frames.
560561
561562
Unlike [`LTX2VideoVaeNeighborhoodAttnProcessor`] and NATTEN, whose windows shift inward at the volume border, every
562563
window here is centered and *clamped*: offsets that fall outside the volume are masked out. That is also why no
@@ -575,6 +576,8 @@ def _joint_neighborhood_attention(
575576
keyframe_times = keyframe_times.to(device=query.device, dtype=torch.float32)
576577
frame_times = torch.arange(num_frames, dtype=torch.float32, device=query.device)
577578
video_slots = _nearest_slots(frame_times, keyframe_times, _KEYFRAME_CONTEXT_SLOTS)
579+
# The reference decoder gives frames before the first plane that plane alone, not their two nearest.
580+
video_slots[frame_times < keyframe_times.min(), 1:] = -1
578581
keyframe_slots = _nearest_slots(keyframe_times, frame_times, _KEYFRAME_CONTEXT_SLOTS)
579582

580583
side = max(1, round(math.sqrt(_JOINT_BRICK_QUERIES)))
@@ -1488,9 +1491,10 @@ class LTX2VideoDiffusionDecoderModel(ModelMixin, AttentionMixin, ConfigMixin):
14881491
14891492
Decoding can be anchored on *keyframe planes*: single-frame latents at known pixel frames, each encoded as a
14901493
standalone one-frame clip, passed as `keyframe_latents` / `keyframe_frame_indices` to [`decode`]. Every video
1491-
position then also attends to the same spatial window on its two nearest planes. Checkpoints trained for it carry a
1492-
learned tag added to the plane latents, `decoder.type_emb`, which `decoder_keyframe_type_embedding=True` creates.
1493-
Decoding with keyframe planes on a checkpoint that lacks that tag raises rather than decoding them untagged.
1494+
position then also attends to the same spatial window on its two nearest planes, or on the first plane alone for
1495+
frames before it. Checkpoints trained for it carry a learned tag added to the plane latents, `decoder.type_emb`,
1496+
which `decoder_keyframe_type_embedding=True` creates. Decoding with keyframe planes on a checkpoint that lacks that
1497+
tag raises rather than decoding them untagged.
14941498
14951499
This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented
14961500
for all models (such as downloading or saving).
@@ -1921,7 +1925,7 @@ def forward(
19211925
keyframe_latents (`torch.Tensor`, *optional*):
19221926
Keyframe planes of shape `(B, C, P, H, W)`, denormalized like `z` and on its latent grid, one latent
19231927
frame per plane, each encoded as a standalone one-frame clip. Every video position also attends to the
1924-
same spatial window on its two nearest planes.
1928+
same spatial window on its two nearest planes, or on the first plane alone for frames before it.
19251929
keyframe_frame_indices (`torch.Tensor`, *optional*):
19261930
The `(P,)` pixel frame of each plane in the decoded video. Required with `keyframe_latents`.
19271931

‎tests/models/autoencoders/test_models_ltx2_diffusion_decoder.py‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,14 +247,15 @@ def _dense_joint_attention(query, key, value, keyframe_query, keyframe_key, keyf
247247
248248
An independent statement of the visibility rule `_joint_neighborhood_attention` implements with query bricks: a
249249
centered window clamped (not shifted) at the volume border, the same spatial window on the two planes nearest a
250-
frame, and the same spatial window on the two frames nearest a plane.
250+
frame (only the first plane for frames before it), and the same spatial window on the two frames nearest a plane.
251251
"""
252252
batch_size, num_frames, height, width, heads, head_dim = query.shape
253253
num_planes = keyframe_query.shape[1]
254254
lo = [k // 2 for k in kernel]
255255
hi = [k - l - 1 for k, l in zip(kernel, lo)]
256256
frame_times = torch.arange(num_frames, dtype=torch.float32)
257257
video_slots = torch.argsort((frame_times[:, None] - keyframe_times[None]).abs(), dim=-1, stable=True)[:, :2]
258+
video_slots[frame_times < keyframe_times.min(), 1:] = -1
258259
plane_slots = torch.argsort((keyframe_times[:, None] - frame_times[None]).abs(), dim=-1, stable=True)[:, :2]
259260

260261
def grid(length):
@@ -417,6 +418,27 @@ def test_joint_attention_matches_dense_softmax(self, num_frames, num_planes, ker
417418
for got, expected in zip(ours, dense):
418419
assert torch.allclose(got, expected, atol=1e-5, rtol=1e-5), (got - expected).abs().max()
419420

421+
def test_frames_before_the_first_plane_see_only_that_plane(self):
422+
"""A frame before the first plane attends to it alone, so the second plane cannot reach it."""
423+
generator = torch.Generator().manual_seed(0)
424+
query, key, value = (torch.randn((1, 6, 5, 5, 2, 16), generator=generator) for _ in range(3))
425+
keyframe_query, keyframe_key, keyframe_value = (
426+
torch.randn((1, 2, 5, 5, 2, 16), generator=generator) for _ in range(3)
427+
)
428+
keyframe_times = torch.tensor([3.0, 5.0])
429+
video, _ = ltx2_diffusion_decoder._joint_neighborhood_attention(
430+
query, key, value, keyframe_query, keyframe_key, keyframe_value, keyframe_times, (3, 3, 3)
431+
)
432+
433+
keyframe_key, keyframe_value = keyframe_key.clone(), keyframe_value.clone()
434+
keyframe_key[:, 1] = torch.randn(keyframe_key[:, 1].shape, generator=generator)
435+
keyframe_value[:, 1] = torch.randn(keyframe_value[:, 1].shape, generator=generator)
436+
changed, _ = ltx2_diffusion_decoder._joint_neighborhood_attention(
437+
query, key, value, keyframe_query, keyframe_key, keyframe_value, keyframe_times, (3, 3, 3)
438+
)
439+
assert torch.equal(changed[:, :3], video[:, :3])
440+
assert not torch.allclose(changed[:, 3:], video[:, 3:])
441+
420442
def test_keyframe_geometry(self):
421443
indices = torch.tensor([0, 8, 16])
422444
# t(0) = 0 and t(f) = (f + (r - 1) / 2) / r: the center of the cell holding frame f.

0 commit comments

Comments
 (0)