diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index adb0f0bf991b..e1a4a9247894 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -685,8 +685,6 @@ title: Sana - local: api/pipelines/sana_sprint title: Sana Sprint - - local: api/pipelines/shap_e - title: Shap-E - local: api/pipelines/stable_cascade title: Stable Cascade - sections: diff --git a/docs/source/en/api/pipelines/shap_e.md b/docs/source/en/api/pipelines/shap_e.md deleted file mode 100644 index c683a53da7ba..000000000000 --- a/docs/source/en/api/pipelines/shap_e.md +++ /dev/null @@ -1,201 +0,0 @@ - - -# Shap-E - -The Shap-E model was proposed in [Shap-E: Generating Conditional 3D Implicit Functions](https://huggingface.co/papers/2305.02463) by Alex Nichol and Heewoo Jun from [OpenAI](https://github.com/openai). - -The abstract from the paper is: - -*We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space.* - -The original codebase can be found at [openai/shap-e](https://github.com/openai/shap-e). - -> [!TIP] -> See the [reuse components across pipelines](../../using-diffusers/loading#reusing-models-in-multiple-pipelines) section to learn how to efficiently load the same components into multiple pipelines. - -Make sure you have the following libraries installed. - -```py -# uncomment to install the necessary libraries in Colab -#!pip install -q diffusers transformers accelerate trimesh -``` - -## Text-to-3D - -To generate a gif of a 3D object, pass a text prompt to the [`ShapEPipeline`]. The pipeline generates a list of image frames which are used to create the 3D object. - -```py -import torch -from diffusers import ShapEPipeline - -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -pipe = ShapEPipeline.from_pretrained("openai/shap-e", dtype=torch.float16, variant="fp16") -pipe = pipe.to(device) - -guidance_scale = 15.0 -prompt = ["A firecracker", "A birthday cupcake"] - -images = pipe( - prompt, - guidance_scale=guidance_scale, - num_inference_steps=64, - frame_size=256, -).images -``` - -Now use the [`~utils.export_to_gif`] function to convert the list of image frames to a gif of the 3D object. - -```py -from diffusers.utils import export_to_gif - -export_to_gif(images[0], "firecracker_3d.gif") -export_to_gif(images[1], "cake_3d.gif") -``` - -
-
- -
prompt = "A firecracker"
-
-
- -
prompt = "A birthday cupcake"
-
-
- -## Image-to-3D - -To generate a 3D object from another image, use the [`ShapEImg2ImgPipeline`]. You can use an existing image or generate an entirely new one. Let's use the [Kandinsky 2.1](./kandinsky) model to generate a new image. - -```py -from diffusers import DiffusionPipeline -import torch - -prior_pipeline = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # or "mps", "xpu", "cpu" -pipeline = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16, use_safetensors=True).to("cuda") - -prompt = "A cheeseburger, white background" - -image_embeds, negative_image_embeds = prior_pipeline(prompt, guidance_scale=1.0).to_tuple() -image = pipeline( - prompt, - image_embeds=image_embeds, - negative_image_embeds=negative_image_embeds, -).images[0] - -image.save("burger.png") -``` - -Pass the cheeseburger to the [`ShapEImg2ImgPipeline`] to generate a 3D representation of it. - -```py -from PIL import Image -from diffusers import ShapEImg2ImgPipeline -from diffusers.utils import export_to_gif - -pipe = ShapEImg2ImgPipeline.from_pretrained("openai/shap-e-img2img", dtype=torch.float16, variant="fp16").to("cuda") # or "mps", "xpu", "cpu" - -guidance_scale = 3.0 -image = Image.open("burger.png").resize((256, 256)) - -images = pipe( - image, - guidance_scale=guidance_scale, - num_inference_steps=64, - frame_size=256, -).images - -gif_path = export_to_gif(images[0], "burger_3d.gif") -``` - -
-
- -
cheeseburger
-
-
- -
3D cheeseburger
-
-
- -## Generate mesh - -Shap-E is a flexible model that can also generate textured mesh outputs to be rendered for downstream applications. In this example, you'll convert the output into a `glb` file because the 🤗 Datasets library supports mesh visualization of `glb` files which can be rendered by the [Dataset viewer](https://huggingface.co/docs/hub/datasets-viewer#dataset-preview). - -You can generate mesh outputs for both the [`ShapEPipeline`] and [`ShapEImg2ImgPipeline`] by specifying the `output_type` parameter as `"mesh"`: - -```py -import torch -from diffusers import ShapEPipeline - -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -pipe = ShapEPipeline.from_pretrained("openai/shap-e", dtype=torch.float16, variant="fp16") -pipe = pipe.to(device) - -guidance_scale = 15.0 -prompt = "A birthday cupcake" - -images = pipe(prompt, guidance_scale=guidance_scale, num_inference_steps=64, frame_size=256, output_type="mesh").images -``` - -Use the [`~utils.export_to_ply`] function to save the mesh output as a `ply` file: - -> [!TIP] -> You can optionally save the mesh output as an `obj` file with the [`~utils.export_to_obj`] function. The ability to save the mesh output in a variety of formats makes it more flexible for downstream usage! - -```py -from diffusers.utils import export_to_ply - -ply_path = export_to_ply(images[0], "3d_cake.ply") -print(f"Saved to folder: {ply_path}") -``` - -Then you can convert the `ply` file to a `glb` file with the trimesh library: - -```py -import trimesh - -mesh = trimesh.load("3d_cake.ply") -mesh_export = mesh.export("3d_cake.glb", file_type="glb") -``` - -By default, the mesh output is focused from the bottom viewpoint but you can change the default viewpoint by applying a rotation transform: - -```py -import trimesh -import numpy as np - -mesh = trimesh.load("3d_cake.ply") -rot = trimesh.transformations.rotation_matrix(-np.pi / 2, [1, 0, 0]) -mesh = mesh.apply_transform(rot) -mesh_export = mesh.export("3d_cake.glb", file_type="glb") -``` - -Upload the mesh file to your dataset repository to visualize it with the Dataset viewer! - -
- -
- -## ShapEPipeline -[[autodoc]] ShapEPipeline - - all - - __call__ - -## ShapEImg2ImgPipeline -[[autodoc]] ShapEImg2ImgPipeline - - all - - __call__ - -## ShapEPipelineOutput -[[autodoc]] pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput diff --git a/examples/research_projects/promptdiffusion/convert_original_promptdiffusion_to_diffusers.py b/examples/research_projects/promptdiffusion/convert_original_promptdiffusion_to_diffusers.py index 31105b6a5890..0f217a37a07c 100644 --- a/examples/research_projects/promptdiffusion/convert_original_promptdiffusion_to_diffusers.py +++ b/examples/research_projects/promptdiffusion/convert_original_promptdiffusion_to_diffusers.py @@ -36,6 +36,7 @@ CLIPVisionModelWithProjection, ) +from diffusers import StableUnCLIPImageNormalizer from diffusers.models import ( AutoencoderKL, ControlNetModel, @@ -46,7 +47,6 @@ from diffusers.pipelines.paint_by_example import PaintByExampleImageEncoder from diffusers.pipelines.pipeline_utils import DiffusionPipeline from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker -from diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer import StableUnCLIPImageNormalizer from diffusers.schedulers import ( DDIMScheduler, DDPMScheduler, diff --git a/scripts/convert_echo_to_diffusers.py b/scripts/convert_echo_to_diffusers.py index 8d13fff8b21f..5f42051da26f 100644 --- a/scripts/convert_echo_to_diffusers.py +++ b/scripts/convert_echo_to_diffusers.py @@ -71,11 +71,11 @@ def write_modular_index(output_path: Path, base_model: str, repo_id: str | None) "_blocks_class_name": "EchoBlocks", "text_encoder": component_entry("transformers", "Gemma3ForConditionalGeneration", base_model, None), "tokenizer": component_entry("transformers", "GemmaTokenizerFast", base_model, None), - "connectors": component_entry("ltx2", "LTX2TextConnectors", echo_repo, "connectors"), + "connectors": component_entry("diffusers", "LTX2TextConnectors", echo_repo, "connectors"), "vae": component_entry("diffusers", "AutoencoderKLLTX2Video", echo_repo, "vae"), "audio_vae": component_entry("diffusers", "AutoencoderKLLTX2Audio", echo_repo, "audio_vae"), "transformer": component_entry("diffusers", "LTX2VideoTransformer3DModel", echo_repo, "transformer"), - "vocoder": component_entry("ltx2", "LTX2VocoderWithBWE", echo_repo, "vocoder"), + "vocoder": component_entry("diffusers", "LTX2VocoderWithBWE", echo_repo, "vocoder"), } (output_path / "modular_model_index.json").write_text(json.dumps(index, indent=2) + "\n", encoding="utf-8") diff --git a/scripts/convert_shap_e_to_diffusers.py b/scripts/convert_shap_e_to_diffusers.py index ac6543667af9..0622e1282668 100644 --- a/scripts/convert_shap_e_to_diffusers.py +++ b/scripts/convert_shap_e_to_diffusers.py @@ -5,7 +5,7 @@ from accelerate import load_checkpoint_and_dispatch from diffusers.models.transformers.prior_transformer import PriorTransformer -from diffusers.pipelines.shap_e import ShapERenderer +from diffusers.pipelines.deprecated.shap_e import ShapERenderer """ diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index 048f37f4061f..1435abc234cf 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -227,6 +227,8 @@ ] _import_structure["models"].extend( [ + "AceStepAudioTokenDetokenizer", + "AceStepAudioTokenizer", "AceStepConditionEncoder", "AceStepTransformer1DModel", "AllegroTransformer3DModel", @@ -236,6 +238,7 @@ "AsymmetricAutoencoderKL", "AttentionBackendName", "AudioLDM2ProjectionModel", + "AudioLDM2UNet2DConditionModel", "AuraFlowTransformer2DModel", "AutoencoderDC", "AutoencoderKL", @@ -329,6 +332,8 @@ "LTX2TextConnectors", "LTX2VideoDiffusionDecoderModel", "LTX2VideoTransformer3DModel", + "LTX2Vocoder", + "LTX2VocoderWithBWE", "LTXLatentUpsamplerModel", "LTXVideoTransformer3DModel", "Lumina2Transformer2DModel", @@ -378,6 +383,7 @@ "StableAudioDiTModel", "StableAudioProjectionModel", "StableCascadeUNet", + "StableUnCLIPImageNormalizer", "T2IAdapter", "T5FilmDecoder", "TensorParallelConfig", @@ -643,8 +649,6 @@ ) _import_structure["pipelines"].extend( [ - "AceStepAudioTokenDetokenizer", - "AceStepAudioTokenizer", "AceStepPipeline", "AllegroPipeline", "AltDiffusionImg2ImgPipeline", @@ -662,7 +666,6 @@ "AnyFlowFARPipeline", "AnyFlowPipeline", "AudioLDM2Pipeline", - "AudioLDM2UNet2DConditionModel", "AudioLDMPipeline", "AuraFlowPipeline", "BlipDiffusionControlNetPipeline", @@ -1142,6 +1145,8 @@ VaeImageProcessorLDM3D, ) from .models import ( + AceStepAudioTokenDetokenizer, + AceStepAudioTokenizer, AceStepConditionEncoder, AceStepTransformer1DModel, AllegroTransformer3DModel, @@ -1151,6 +1156,7 @@ AsymmetricAutoencoderKL, AttentionBackendName, AudioLDM2ProjectionModel, + AudioLDM2UNet2DConditionModel, AuraFlowTransformer2DModel, AutoencoderDC, AutoencoderKL, @@ -1244,6 +1250,8 @@ LTX2TextConnectors, LTX2VideoDiffusionDecoderModel, LTX2VideoTransformer3DModel, + LTX2Vocoder, + LTX2VocoderWithBWE, LTXLatentUpsamplerModel, LTXVideoTransformer3DModel, Lumina2Transformer2DModel, @@ -1292,6 +1300,7 @@ StableAudio3DurationEmbedder, StableAudioDiTModel, StableAudioProjectionModel, + StableUnCLIPImageNormalizer, T2IAdapter, T5FilmDecoder, TensorParallelConfig, @@ -1534,8 +1543,6 @@ ZImageModularPipeline, ) from .pipelines import ( - AceStepAudioTokenDetokenizer, - AceStepAudioTokenizer, AceStepPipeline, AllegroPipeline, AltDiffusionImg2ImgPipeline, @@ -1553,7 +1560,6 @@ AnyFlowFARPipeline, AnyFlowPipeline, AudioLDM2Pipeline, - AudioLDM2UNet2DConditionModel, AudioLDMPipeline, AuraFlowPipeline, BriaFibo2Pipeline, diff --git a/src/diffusers/models/__init__.py b/src/diffusers/models/__init__.py index 28075d72de80..409933b0304d 100755 --- a/src/diffusers/models/__init__.py +++ b/src/diffusers/models/__init__.py @@ -28,6 +28,10 @@ _import_structure["adapter"] = ["MultiAdapter", "T2IAdapter"] _import_structure["attention_dispatch"] = ["AttentionBackendName", "attention_backend"] _import_structure["auto_model"] = ["AutoModel"] + _import_structure["autoencoders.audio_tokenizer_ace_step"] = [ + "AceStepAudioTokenDetokenizer", + "AceStepAudioTokenizer", + ] _import_structure["autoencoders.autoencoder_asym_kl"] = ["AsymmetricAutoencoderKL"] _import_structure["autoencoders.autoencoder_cosmos3_audio"] = ["Cosmos3AVAEAudioTokenizer"] _import_structure["autoencoders.autoencoder_dc"] = ["AutoencoderDC"] @@ -69,6 +73,7 @@ _import_structure["autoencoders.ltx2_diffusion_decoder"] = ["LTX2VideoDiffusionDecoderModel"] _import_structure["autoencoders.minimax_music3_vocoder"] = ["MiniMaxMusic3Vocoder"] _import_structure["autoencoders.mmaudio_vocoder"] = ["MMAudioVocoder"] + _import_structure["autoencoders.vocoder_ltx2"] = ["LTX2Vocoder", "LTX2VocoderWithBWE"] _import_structure["autoencoders.vq_model"] = ["VQModel"] _import_structure["cache_utils"] = ["CacheMixin"] _import_structure["condition_embedders.condition_embedder_ace_step"] = ["AceStepConditionEncoder"] @@ -81,6 +86,7 @@ _import_structure["condition_embedders.condition_embedder_p_h3_image"] = ["PH3ImageTextConnector"] _import_structure["condition_embedders.condition_embedder_stable_audio"] = ["StableAudioProjectionModel"] _import_structure["condition_embedders.condition_embedder_stable_audio_3"] = ["StableAudio3DurationEmbedder"] + _import_structure["condition_embedders.condition_embedder_stable_unclip"] = ["StableUnCLIPImageNormalizer"] _import_structure["controlnets.controlnet"] = ["ControlNetModel"] _import_structure["controlnets.controlnet_cosmos"] = ["CosmosControlNetModel"] _import_structure["controlnets.controlnet_flux"] = ["FluxControlNetModel", "FluxMultiControlNetModel"] @@ -187,6 +193,7 @@ _import_structure["unets.unet_1d"] = ["UNet1DModel"] _import_structure["unets.unet_2d"] = ["UNet2DModel"] _import_structure["unets.unet_2d_condition"] = ["UNet2DConditionModel"] + _import_structure["unets.unet_2d_condition_audioldm2"] = ["AudioLDM2UNet2DConditionModel"] _import_structure["unets.unet_3d_condition"] = ["UNet3DConditionModel"] _import_structure["unets.unet_dreamlite"] = ["DreamLiteUNetModel"] _import_structure["unets.unet_i2vgen_xl"] = ["I2VGenXLUNet"] @@ -204,6 +211,8 @@ from .attention_dispatch import AttentionBackendName, attention_backend from .auto_model import AutoModel from .autoencoders import ( + AceStepAudioTokenDetokenizer, + AceStepAudioTokenizer, AsymmetricAutoencoderKL, AutoencoderDC, AutoencoderKL, @@ -241,6 +250,8 @@ Kandinsky6SRVAE, LongCatAudioDiTVae, LTX2VideoDiffusionDecoderModel, + LTX2Vocoder, + LTX2VocoderWithBWE, MiniMaxMusic3Vocoder, MMAudioVAE, MMAudioVocoder, @@ -259,6 +270,7 @@ ReduxImageEncoder, StableAudio3DurationEmbedder, StableAudioProjectionModel, + StableUnCLIPImageNormalizer, ) from .controlnets import ( ControlNetModel, @@ -365,6 +377,7 @@ ZImageTransformer2DModel, ) from .unets import ( + AudioLDM2UNet2DConditionModel, DreamLiteUNetModel, I2VGenXLUNet, Kandinsky3UNet, diff --git a/src/diffusers/models/autoencoders/__init__.py b/src/diffusers/models/autoencoders/__init__.py index 84e2fdf1c0b9..08e808d06363 100644 --- a/src/diffusers/models/autoencoders/__init__.py +++ b/src/diffusers/models/autoencoders/__init__.py @@ -1,3 +1,4 @@ +from .audio_tokenizer_ace_step import AceStepAudioTokenDetokenizer, AceStepAudioTokenizer from .autoencoder_asym_kl import AsymmetricAutoencoderKL from .autoencoder_cosmos3_audio import Cosmos3AVAEAudioTokenizer from .autoencoder_dc import AutoencoderDC @@ -39,4 +40,5 @@ from .ltx2_diffusion_decoder import LTX2VideoDiffusionDecoderModel from .minimax_music3_vocoder import MiniMaxMusic3Vocoder from .mmaudio_vocoder import MMAudioVocoder +from .vocoder_ltx2 import LTX2Vocoder, LTX2VocoderWithBWE from .vq_model import VQModel diff --git a/src/diffusers/models/autoencoders/audio_tokenizer_ace_step.py b/src/diffusers/models/autoencoders/audio_tokenizer_ace_step.py new file mode 100644 index 000000000000..d68a70680cb1 --- /dev/null +++ b/src/diffusers/models/autoencoders/audio_tokenizer_ace_step.py @@ -0,0 +1,431 @@ +# Copyright 2025 The ACE-Step Team and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Audio tokenizer / detokenizer for ACE-Step 1.5, used by cover conditioning.""" + +from typing import Optional, Tuple + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ...configuration_utils import ConfigMixin, register_to_config +from ...utils import logging +from ..condition_embedders.condition_embedder_ace_step import AceStepEncoderLayer +from ..modeling_utils import ModelMixin +from ..normalization import RMSNorm +from ..transformers.ace_step_transformer import ( + _ace_step_rotary_freqs, + _create_4d_mask, + _is_flash_attention_backend, +) + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +# --------------------------------------------------------------------------- # +# audio tokenizer / detokenizer # +# --------------------------------------------------------------------------- # + + +class _AceStepResidualFSQ(nn.Module): + """Minimal ResidualFSQ compatible with ACE-Step's saved tokenizer weights.""" + + def __init__( + self, + dim: int = 2048, + levels: Optional[list] = None, + num_quantizers: int = 1, + ): + super().__init__() + + if levels is None: + levels = [8, 8, 8, 5, 5, 5] + + self.levels = levels + self.num_quantizers = num_quantizers + self.codebook_dim = len(levels) + + self.project_in = nn.Linear(dim, self.codebook_dim) + self.project_out = nn.Linear(self.codebook_dim, dim) + + levels_tensor = torch.tensor(levels, dtype=torch.long) + basis = torch.cumprod(torch.tensor([1] + levels[:-1], dtype=torch.long), dim=0) + scales = torch.stack([levels_tensor.float() ** -i for i in range(num_quantizers)]) + self.register_buffer("_levels", levels_tensor, persistent=False) + self.register_buffer("_basis", basis, persistent=False) + self.register_buffer("scales", scales, persistent=False) + + @property + def codebook_size(self) -> int: + return int(torch.prod(self._levels).item()) + + def _indices_to_codes(self, indices: torch.Tensor) -> torch.Tensor: + levels = self._levels.to(device=indices.device) + basis = self._basis.to(device=indices.device) + level_indices = (indices.long().unsqueeze(-1) // basis) % levels + scale = 2.0 / (levels.to(dtype=torch.float32) - 1.0) + return level_indices.to(dtype=torch.float32) * scale - 1.0 + + def _codes_to_indices(self, codes: torch.Tensor) -> torch.Tensor: + levels = self._levels.to(device=codes.device, dtype=codes.dtype) + basis = self._basis.to(device=codes.device, dtype=codes.dtype) + level_indices = (codes + 1.0) / (2.0 / (levels - 1.0)) + return (level_indices * basis).sum(dim=-1).round().to(torch.long) + + def _quantize(self, x: torch.Tensor) -> torch.Tensor: + levels = self._levels.to(device=x.device, dtype=x.dtype) + levels_minus_one = levels - 1.0 + step = 2.0 / levels_minus_one + bracket = levels_minus_one * (x.clamp(-1.0, 1.0) + 1.0) / 2.0 + 0.5 + return step * torch.floor(bracket) - 1.0 + + def get_codes_from_indices(self, indices: torch.Tensor) -> torch.Tensor: + if indices.ndim == 2: + indices = indices.unsqueeze(-1) + if indices.shape[-1] != self.num_quantizers: + raise ValueError( + f"Expected audio code indices with last dimension {self.num_quantizers}, got {indices.shape[-1]}." + ) + + codes = [] + for quantizer_idx in range(self.num_quantizers): + code = self._indices_to_codes(indices[..., quantizer_idx]) + scale = self.scales[quantizer_idx].to(device=code.device, dtype=code.dtype) + codes.append(code * scale) + return torch.stack(codes, dim=0) + + def get_output_from_indices(self, indices: torch.Tensor) -> torch.Tensor: + codes = self.get_codes_from_indices(indices).sum(dim=0) + weight = self.project_out.weight.float() + bias = self.project_out.bias.float() if self.project_out.bias is not None else None + output = F.linear(codes.float(), weight, bias) + return output.to(dtype=self.project_out.weight.dtype) + + def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + input_dtype = hidden_states.dtype + weight = self.project_in.weight.float() + bias = self.project_in.bias.float() if self.project_in.bias is not None else None + hidden_states = F.linear(hidden_states.float(), weight, bias) + + levels = self._levels.to(device=hidden_states.device, dtype=hidden_states.dtype) + soft_clamp = 1.0 + (1.0 / (levels - 1.0)) + hidden_states = (hidden_states / soft_clamp).tanh() * soft_clamp + + quantized_out = torch.zeros_like(hidden_states) + residual = hidden_states + all_indices = [] + for scale in self.scales.to(device=hidden_states.device, dtype=hidden_states.dtype): + quantized = self._quantize(residual / scale) * scale + residual = residual - quantized.detach() + quantized_out = quantized_out + quantized + all_indices.append(self._codes_to_indices(quantized / scale)) + + weight = self.project_out.weight.float() + bias = self.project_out.bias.float() if self.project_out.bias is not None else None + quantized_out = F.linear(quantized_out.float(), weight, bias).to(dtype=input_dtype) + all_indices = torch.stack(all_indices, dim=-1) + return quantized_out, all_indices + + +class AceStepAttentionPooler(nn.Module): + """Attention pooler used by the ACE-Step audio tokenizer.""" + + def __init__( + self, + hidden_size: int = 2048, + intermediate_size: int = 6144, + num_attention_pooler_hidden_layers: int = 2, + num_attention_heads: int = 16, + num_key_value_heads: int = 8, + head_dim: int = 128, + rope_theta: float = 1000000.0, + attention_bias: bool = False, + attention_dropout: float = 0.0, + rms_norm_eps: float = 1e-6, + sliding_window: int = 128, + layer_types: list = None, + ): + super().__init__() + + if layer_types is None: + layer_types = [ + "sliding_attention" if bool((i + 1) % 2) else "full_attention" + for i in range(num_attention_pooler_hidden_layers) + ] + + self.embed_tokens = nn.Linear(hidden_size, hidden_size) + self.norm = RMSNorm(hidden_size, eps=rms_norm_eps) + self.special_token = nn.Parameter(torch.randn(1, 1, hidden_size) * 0.02) + self.head_dim = head_dim + self.rope_theta = rope_theta + self.sliding_window = sliding_window + self.layers = nn.ModuleList( + [ + AceStepEncoderLayer( + hidden_size=hidden_size, + num_attention_heads=num_attention_heads, + num_key_value_heads=num_key_value_heads, + head_dim=head_dim, + intermediate_size=intermediate_size, + attention_bias=attention_bias, + attention_dropout=attention_dropout, + rms_norm_eps=rms_norm_eps, + sliding_window=sliding_window if layer_types[i] == "sliding_attention" else None, + ) + for i in range(num_attention_pooler_hidden_layers) + ] + ) + self._layer_types = layer_types + + def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: + batch_size, num_patches, patch_size, _ = hidden_states.shape + hidden_states = self.embed_tokens(hidden_states) + special_token = self.special_token.to(device=hidden_states.device, dtype=hidden_states.dtype) + special_token = special_token.expand(batch_size, num_patches, -1, -1) + hidden_states = torch.cat([special_token, hidden_states], dim=2) + hidden_states = hidden_states.reshape(batch_size * num_patches, patch_size + 1, -1) + + seq_len = hidden_states.shape[1] + dtype = hidden_states.dtype + device = hidden_states.device + position_embeddings = _ace_step_rotary_freqs(seq_len, self.head_dim, self.rope_theta, device, dtype) + sliding_attn_mask = None + if not _is_flash_attention_backend(self.layers[0].self_attn.processor): + sliding_attn_mask = _create_4d_mask( + seq_len=seq_len, + dtype=dtype, + device=device, + attention_mask=None, + sliding_window=self.sliding_window, + is_sliding_window=True, + is_causal=False, + ) + + for i, layer_module in enumerate(self.layers): + mask = sliding_attn_mask if self._layer_types[i] == "sliding_attention" else None + hidden_states = layer_module( + hidden_states=hidden_states, + position_embeddings=position_embeddings, + attention_mask=mask, + ) + + hidden_states = self.norm(hidden_states) + hidden_states = hidden_states[:, 0, :] + return hidden_states.reshape(batch_size, num_patches, -1) + + +class AceStepAudioTokenDetokenizer(ModelMixin, ConfigMixin): + """Expands ACE-Step 5 Hz audio tokens back to 25 Hz acoustic conditioning.""" + + _supports_gradient_checkpointing = True + + @register_to_config + def __init__( + self, + hidden_size: int = 2048, + intermediate_size: int = 6144, + audio_acoustic_hidden_dim: int = 64, + pool_window_size: int = 5, + num_attention_pooler_hidden_layers: int = 2, + num_attention_heads: int = 16, + num_key_value_heads: int = 8, + head_dim: int = 128, + rope_theta: float = 1000000.0, + attention_bias: bool = False, + attention_dropout: float = 0.0, + rms_norm_eps: float = 1e-6, + sliding_window: int = 128, + layer_types: list = None, + ): + super().__init__() + + if layer_types is None: + layer_types = [ + "sliding_attention" if bool((i + 1) % 2) else "full_attention" + for i in range(num_attention_pooler_hidden_layers) + ] + + self.embed_tokens = nn.Linear(hidden_size, hidden_size) + self.norm = RMSNorm(hidden_size, eps=rms_norm_eps) + self.special_tokens = nn.Parameter(torch.randn(1, pool_window_size, hidden_size) * 0.02) + self.proj_out = nn.Linear(hidden_size, audio_acoustic_hidden_dim) + self.head_dim = head_dim + self.rope_theta = rope_theta + self.sliding_window = sliding_window + self.pool_window_size = pool_window_size + self.layers = nn.ModuleList( + [ + AceStepEncoderLayer( + hidden_size=hidden_size, + num_attention_heads=num_attention_heads, + num_key_value_heads=num_key_value_heads, + head_dim=head_dim, + intermediate_size=intermediate_size, + attention_bias=attention_bias, + attention_dropout=attention_dropout, + rms_norm_eps=rms_norm_eps, + sliding_window=sliding_window if layer_types[i] == "sliding_attention" else None, + ) + for i in range(num_attention_pooler_hidden_layers) + ] + ) + self._layer_types = layer_types + self.gradient_checkpointing = False + + def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: + r""" + Args: + hidden_states (`torch.Tensor`): + Quantized 5 Hz audio token embeddings from [`AceStepAudioTokenizer`], of shape `(batch_size, + num_tokens, hidden_size)`. + + Returns: + `torch.Tensor`: + 25 Hz acoustic conditioning of shape `(batch_size, num_tokens * pool_window_size, + audio_acoustic_hidden_dim)`. + """ + batch_size, num_tokens, _ = hidden_states.shape + hidden_states = self.embed_tokens(hidden_states) + hidden_states = hidden_states.unsqueeze(2).expand(-1, -1, self.pool_window_size, -1) + special_tokens = self.special_tokens.to(device=hidden_states.device, dtype=hidden_states.dtype) + hidden_states = hidden_states + special_tokens.unsqueeze(0) + hidden_states = hidden_states.reshape(batch_size * num_tokens, self.pool_window_size, -1) + + seq_len = hidden_states.shape[1] + dtype = hidden_states.dtype + device = hidden_states.device + position_embeddings = _ace_step_rotary_freqs(seq_len, self.head_dim, self.rope_theta, device, dtype) + sliding_attn_mask = None + if not _is_flash_attention_backend(self.layers[0].self_attn.processor): + sliding_attn_mask = _create_4d_mask( + seq_len=seq_len, + dtype=dtype, + device=device, + attention_mask=None, + sliding_window=self.sliding_window, + is_sliding_window=True, + is_causal=False, + ) + + for i, layer_module in enumerate(self.layers): + mask = sliding_attn_mask if self._layer_types[i] == "sliding_attention" else None + if torch.is_grad_enabled() and self.gradient_checkpointing: + hidden_states = self._gradient_checkpointing_func( + layer_module, hidden_states, position_embeddings, mask + ) + else: + hidden_states = layer_module( + hidden_states=hidden_states, + position_embeddings=position_embeddings, + attention_mask=mask, + ) + + hidden_states = self.norm(hidden_states) + hidden_states = self.proj_out(hidden_states) + return hidden_states.reshape(batch_size, num_tokens * self.pool_window_size, -1) + + +class AceStepAudioTokenizer(ModelMixin, ConfigMixin): + """Converts 25 Hz acoustic latents to ACE-Step 5 Hz audio tokens.""" + + _supports_gradient_checkpointing = True + + @register_to_config + def __init__( + self, + hidden_size: int = 2048, + intermediate_size: int = 6144, + audio_acoustic_hidden_dim: int = 64, + pool_window_size: int = 5, + fsq_dim: int = 2048, + fsq_input_levels: list = None, + fsq_input_num_quantizers: int = 1, + num_attention_pooler_hidden_layers: int = 2, + num_attention_heads: int = 16, + num_key_value_heads: int = 8, + head_dim: int = 128, + rope_theta: float = 1000000.0, + attention_bias: bool = False, + attention_dropout: float = 0.0, + rms_norm_eps: float = 1e-6, + sliding_window: int = 128, + layer_types: list = None, + ): + super().__init__() + + if fsq_input_levels is None: + fsq_input_levels = [8, 8, 8, 5, 5, 5] + + self.audio_acoustic_proj = nn.Linear(audio_acoustic_hidden_dim, hidden_size) + self.attention_pooler = AceStepAttentionPooler( + hidden_size=hidden_size, + intermediate_size=intermediate_size, + num_attention_pooler_hidden_layers=num_attention_pooler_hidden_layers, + num_attention_heads=num_attention_heads, + num_key_value_heads=num_key_value_heads, + head_dim=head_dim, + rope_theta=rope_theta, + attention_bias=attention_bias, + attention_dropout=attention_dropout, + rms_norm_eps=rms_norm_eps, + sliding_window=sliding_window, + layer_types=layer_types, + ) + self.quantizer = _AceStepResidualFSQ( + dim=fsq_dim, + levels=fsq_input_levels, + num_quantizers=fsq_input_num_quantizers, + ) + self.pool_window_size = pool_window_size + + def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + r""" + Args: + hidden_states (`torch.Tensor`): + Windowed 25 Hz acoustic latents of shape `(batch_size, num_tokens, pool_window_size, + audio_acoustic_hidden_dim)`. Use [`~AceStepAudioTokenizer.tokenize`] to pad and window raw latents. + + Returns: + `Tuple[torch.Tensor, torch.Tensor]`: + The quantized 5 Hz token embeddings of shape `(batch_size, num_tokens, hidden_size)` and their codebook + indices of shape `(batch_size, num_tokens, num_quantizers)`. + """ + input_dtype = hidden_states.dtype + hidden_states = self.audio_acoustic_proj(hidden_states) + hidden_states = self.attention_pooler(hidden_states) + quantized, indices = self.quantizer(hidden_states) + return quantized.to(dtype=input_dtype), indices + + def tokenize( + self, + hidden_states: torch.Tensor, + silence_latent: Optional[torch.Tensor] = None, + ) -> Tuple[torch.Tensor, torch.Tensor]: + batch_size, latent_length, acoustic_dim = hidden_states.shape + pad_len = (-latent_length) % self.pool_window_size + if pad_len: + if silence_latent is not None and silence_latent.shape[-1] == acoustic_dim: + pad = silence_latent[:, :pad_len, :].to(device=hidden_states.device, dtype=hidden_states.dtype) + pad = pad.expand(batch_size, -1, -1) + else: + pad = torch.zeros( + batch_size, pad_len, acoustic_dim, device=hidden_states.device, dtype=hidden_states.dtype + ) + hidden_states = torch.cat([hidden_states, pad], dim=1) + + num_patches = hidden_states.shape[1] // self.pool_window_size + hidden_states = hidden_states.reshape(batch_size, num_patches, self.pool_window_size, acoustic_dim) + return self(hidden_states) diff --git a/src/diffusers/models/autoencoders/vocoder_ltx2.py b/src/diffusers/models/autoencoders/vocoder_ltx2.py new file mode 100644 index 000000000000..83a49a8d2184 --- /dev/null +++ b/src/diffusers/models/autoencoders/vocoder_ltx2.py @@ -0,0 +1,621 @@ +# Copyright 2026 Lightricks and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ...configuration_utils import ConfigMixin, register_to_config +from ..modeling_utils import ModelMixin + + +def kaiser_sinc_filter1d(cutoff: float, half_width: float, kernel_size: int) -> torch.Tensor: + """ + Creates a Kaiser sinc kernel for low-pass filtering. + + Args: + cutoff (`float`): + Normalized frequency cutoff (relative to the sampling rate). Must be between 0 and 0.5 (the Nyquist + frequency). + half_width (`float`): + Used to determine the Kaiser window's beta parameter. + kernel_size: + Size of the Kaiser window (and ultimately the Kaiser sinc kernel). + + Returns: + `torch.Tensor` of shape `(kernel_size,)`: + The Kaiser sinc kernel. + """ + delta_f = 4 * half_width + half_size = kernel_size // 2 + amplitude = 2.285 * (half_size - 1) * math.pi * delta_f + 7.95 + if amplitude > 50.0: + beta = 0.1102 * (amplitude - 8.7) + elif amplitude >= 21.0: + beta = 0.5842 * (amplitude - 21) ** 0.4 + 0.07886 * (amplitude - 21.0) + else: + beta = 0.0 + + window = torch.kaiser_window(kernel_size, beta=beta, periodic=False) + + even = kernel_size % 2 == 0 + time = torch.arange(-half_size, half_size) + 0.5 if even else torch.arange(kernel_size) - half_size + + if cutoff == 0.0: + filter = torch.zeros_like(time) + else: + time = 2 * cutoff * time + sinc = torch.where( + time == 0, + torch.ones_like(time), + torch.sin(math.pi * time) / math.pi / time, + ) + filter = 2 * cutoff * window * sinc + filter = filter / filter.sum() + return filter + + +class DownSample1d(nn.Module): + """1D low-pass filter for antialias downsampling.""" + + def __init__( + self, + ratio: int = 2, + kernel_size: int | None = None, + use_padding: bool = True, + padding_mode: str = "replicate", + persistent: bool = True, + ): + super().__init__() + self.ratio = ratio + self.kernel_size = kernel_size or int(6 * ratio // 2) * 2 + self.pad_left = self.kernel_size // 2 + (self.kernel_size % 2) - 1 + self.pad_right = self.kernel_size // 2 + self.use_padding = use_padding + self.padding_mode = padding_mode + + cutoff = 0.5 / ratio + half_width = 0.6 / ratio + low_pass_filter = kaiser_sinc_filter1d(cutoff, half_width, self.kernel_size) + self.register_buffer("filter", low_pass_filter.view(1, 1, self.kernel_size), persistent=persistent) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + # x expected shape: [batch_size, num_channels, hidden_dim] + num_channels = x.shape[1] + if self.use_padding: + x = F.pad(x, (self.pad_left, self.pad_right), mode=self.padding_mode) + x_filtered = F.conv1d(x, self.filter.expand(num_channels, -1, -1), stride=self.ratio, groups=num_channels) + return x_filtered + + +class UpSample1d(nn.Module): + def __init__( + self, + ratio: int = 2, + kernel_size: int | None = None, + window_type: str = "kaiser", + padding_mode: str = "replicate", + persistent: bool = True, + ): + super().__init__() + self.ratio = ratio + self.padding_mode = padding_mode + + if window_type == "hann": + rolloff = 0.99 + lowpass_filter_width = 6 + width = math.ceil(lowpass_filter_width / rolloff) + self.kernel_size = 2 * width * ratio + 1 + self.pad = width + self.pad_left = 2 * width * ratio + self.pad_right = self.kernel_size - ratio + + time_axis = (torch.arange(self.kernel_size) / ratio - width) * rolloff + time_clamped = time_axis.clamp(-lowpass_filter_width, lowpass_filter_width) + window = torch.cos(time_clamped * math.pi / lowpass_filter_width / 2) ** 2 + sinc_filter = (torch.sinc(time_axis) * window * rolloff / ratio).view(1, 1, -1) + else: + # Kaiser sinc filter is BigVGAN default + self.kernel_size = int(6 * ratio // 2) * 2 if kernel_size is None else kernel_size + self.pad = self.kernel_size // ratio - 1 + self.pad_left = self.pad * self.ratio + (self.kernel_size - self.ratio) // 2 + self.pad_right = self.pad * self.ratio + (self.kernel_size - self.ratio + 1) // 2 + + sinc_filter = kaiser_sinc_filter1d( + cutoff=0.5 / ratio, + half_width=0.6 / ratio, + kernel_size=self.kernel_size, + ) + + self.register_buffer("filter", sinc_filter.view(1, 1, self.kernel_size), persistent=persistent) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + # x expected shape: [batch_size, num_channels, hidden_dim] + num_channels = x.shape[1] + x = F.pad(x, (self.pad, self.pad), mode=self.padding_mode) + low_pass_filter = self.filter.to(dtype=x.dtype, device=x.device).expand(num_channels, -1, -1) + x = self.ratio * F.conv_transpose1d(x, low_pass_filter, stride=self.ratio, groups=num_channels) + return x[..., self.pad_left : -self.pad_right] + + +class AntiAliasAct1d(nn.Module): + """ + Antialiasing activation for a 1D signal: upsamples, applies an activation (usually snakebeta), and then downsamples + to avoid aliasing. + """ + + def __init__( + self, + act_fn: str | nn.Module, + ratio: int = 2, + kernel_size: int = 12, + **kwargs, + ): + super().__init__() + self.upsample = UpSample1d(ratio=ratio, kernel_size=kernel_size) + if isinstance(act_fn, str): + if act_fn == "snakebeta": + act_fn = SnakeBeta(**kwargs) + elif act_fn == "snake": + act_fn = SnakeBeta(**kwargs) + else: + act_fn = nn.LeakyReLU(**kwargs) + self.act = act_fn + self.downsample = DownSample1d(ratio=ratio, kernel_size=kernel_size) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = self.upsample(x) + x = self.act(x) + x = self.downsample(x) + return x + + +class SnakeBeta(nn.Module): + """ + Implements the Snake and SnakeBeta activations, which help with learning periodic patterns. + """ + + def __init__( + self, + channels: int, + alpha: float = 1.0, + eps: float = 1e-9, + trainable_params: bool = True, + logscale: bool = True, + use_beta: bool = True, + ): + super().__init__() + self.eps = eps + self.logscale = logscale + self.use_beta = use_beta + + self.alpha = nn.Parameter(torch.zeros(channels) if self.logscale else torch.ones(channels) * alpha) + self.alpha.requires_grad = trainable_params + if use_beta: + self.beta = nn.Parameter(torch.zeros(channels) if self.logscale else torch.ones(channels) * alpha) + self.beta.requires_grad = trainable_params + + def forward(self, hidden_states: torch.Tensor, channel_dim: int = 1) -> torch.Tensor: + broadcast_shape = [1] * hidden_states.ndim + broadcast_shape[channel_dim] = -1 + alpha = self.alpha.view(broadcast_shape) + if self.use_beta: + beta = self.beta.view(broadcast_shape) + + if self.logscale: + alpha = torch.exp(alpha) + if self.use_beta: + beta = torch.exp(beta) + + amplitude = beta if self.use_beta else alpha + hidden_states = hidden_states + (1.0 / (amplitude + self.eps)) * torch.sin(hidden_states * alpha).pow(2) + return hidden_states + + +class ResBlock(nn.Module): + def __init__( + self, + channels: int, + kernel_size: int = 3, + stride: int = 1, + dilations: tuple[int, ...] = (1, 3, 5), + act_fn: str = "leaky_relu", + leaky_relu_negative_slope: float = 0.1, + antialias: bool = False, + antialias_ratio: int = 2, + antialias_kernel_size: int = 12, + padding_mode: str = "same", + ): + super().__init__() + self.dilations = dilations + + self.convs1 = nn.ModuleList( + [ + nn.Conv1d(channels, channels, kernel_size, stride=stride, dilation=dilation, padding=padding_mode) + for dilation in dilations + ] + ) + self.acts1 = nn.ModuleList() + for _ in range(len(self.convs1)): + if act_fn == "snakebeta": + act = SnakeBeta(channels, use_beta=True) + elif act_fn == "snake": + act = SnakeBeta(channels, use_beta=False) + else: + act = nn.LeakyReLU(negative_slope=leaky_relu_negative_slope) + + if antialias: + act = AntiAliasAct1d(act, ratio=antialias_ratio, kernel_size=antialias_kernel_size) + self.acts1.append(act) + + self.convs2 = nn.ModuleList( + [ + nn.Conv1d(channels, channels, kernel_size, stride=stride, dilation=1, padding=padding_mode) + for _ in range(len(dilations)) + ] + ) + self.acts2 = nn.ModuleList() + for _ in range(len(self.convs2)): + if act_fn == "snakebeta": + act = SnakeBeta(channels, use_beta=True) + elif act_fn == "snake": + act = SnakeBeta(channels, use_beta=False) + else: + act_fn = nn.LeakyReLU(negative_slope=leaky_relu_negative_slope) + + if antialias: + act = AntiAliasAct1d(act, ratio=antialias_ratio, kernel_size=antialias_kernel_size) + self.acts2.append(act) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + for act1, conv1, act2, conv2 in zip(self.acts1, self.convs1, self.acts2, self.convs2): + xt = act1(x) + xt = conv1(xt) + xt = act2(xt) + xt = conv2(xt) + x = x + xt + return x + + +class LTX2Vocoder(ModelMixin, ConfigMixin): + r""" + LTX 2.0 vocoder for converting generated mel spectrograms back to audio waveforms. + """ + + @register_to_config + def __init__( + self, + in_channels: int = 128, + hidden_channels: int = 1024, + out_channels: int = 2, + upsample_kernel_sizes: list[int] = [16, 15, 8, 4, 4], + upsample_factors: list[int] = [6, 5, 2, 2, 2], + resnet_kernel_sizes: list[int] = [3, 7, 11], + resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], + act_fn: str = "leaky_relu", + leaky_relu_negative_slope: float = 0.1, + antialias: bool = False, + antialias_ratio: int = 2, + antialias_kernel_size: int = 12, + final_act_fn: str | None = "tanh", # tanh, clamp, None + final_bias: bool = True, + output_sampling_rate: int = 24000, + ): + super().__init__() + self.num_upsample_layers = len(upsample_kernel_sizes) + self.resnets_per_upsample = len(resnet_kernel_sizes) + self.out_channels = out_channels + self.total_upsample_factor = math.prod(upsample_factors) + self.act_fn = act_fn + self.negative_slope = leaky_relu_negative_slope + self.final_act_fn = final_act_fn + + if self.num_upsample_layers != len(upsample_factors): + raise ValueError( + f"`upsample_kernel_sizes` and `upsample_factors` should be lists of the same length but are length" + f" {self.num_upsample_layers} and {len(upsample_factors)}, respectively." + ) + + if self.resnets_per_upsample != len(resnet_dilations): + raise ValueError( + f"`resnet_kernel_sizes` and `resnet_dilations` should be lists of the same length but are length" + f" {len(self.resnets_per_upsample)} and {len(resnet_dilations)}, respectively." + ) + + supported_act_fns = ["snakebeta", "snake", "leaky_relu"] + if self.act_fn not in supported_act_fns: + raise ValueError( + f"Unsupported activation function: {self.act_fn}. Currently supported values of `act_fn` are " + f"{supported_act_fns}." + ) + + self.conv_in = nn.Conv1d(in_channels, hidden_channels, kernel_size=7, stride=1, padding=3) + + self.upsamplers = nn.ModuleList() + self.resnets = nn.ModuleList() + input_channels = hidden_channels + for i, (stride, kernel_size) in enumerate(zip(upsample_factors, upsample_kernel_sizes)): + output_channels = input_channels // 2 + self.upsamplers.append( + nn.ConvTranspose1d( + input_channels, # hidden_channels // (2 ** i) + output_channels, # hidden_channels // (2 ** (i + 1)) + kernel_size, + stride=stride, + padding=(kernel_size - stride) // 2, + ) + ) + + for kernel_size, dilations in zip(resnet_kernel_sizes, resnet_dilations): + self.resnets.append( + ResBlock( + channels=output_channels, + kernel_size=kernel_size, + dilations=dilations, + act_fn=act_fn, + leaky_relu_negative_slope=leaky_relu_negative_slope, + antialias=antialias, + antialias_ratio=antialias_ratio, + antialias_kernel_size=antialias_kernel_size, + ) + ) + input_channels = output_channels + + if act_fn == "snakebeta" or act_fn == "snake": + # Always use antialiasing + act_out = SnakeBeta(channels=output_channels, use_beta=True) + self.act_out = AntiAliasAct1d(act_out, ratio=antialias_ratio, kernel_size=antialias_kernel_size) + elif act_fn == "leaky_relu": + # NOTE: does NOT use self.negative_slope, following the original code + self.act_out = nn.LeakyReLU() + + self.conv_out = nn.Conv1d(output_channels, out_channels, 7, stride=1, padding=3, bias=final_bias) + + def forward(self, hidden_states: torch.Tensor, time_last: bool = False) -> torch.Tensor: + r""" + Forward pass of the vocoder. + + Args: + hidden_states (`torch.Tensor`): + Input Mel spectrogram tensor of shape `(batch_size, num_channels, time, num_mel_bins)` if `time_last` + is `False` (the default) or shape `(batch_size, num_channels, num_mel_bins, time)` if `time_last` is + `True`. + time_last (`bool`, *optional*, defaults to `False`): + Whether the last dimension of the input is the time/frame dimension or the Mel bins dimension. + + Returns: + `torch.Tensor`: + Audio waveform tensor of shape (batch_size, out_channels, audio_length) + """ + + # Ensure that the time/frame dimension is last + if not time_last: + hidden_states = hidden_states.transpose(2, 3) + # Combine channels and frequency (mel bins) dimensions + hidden_states = hidden_states.flatten(1, 2) + + hidden_states = self.conv_in(hidden_states) + + for i in range(self.num_upsample_layers): + if self.act_fn == "leaky_relu": + # Other activations are inside each upsampling block + hidden_states = F.leaky_relu(hidden_states, negative_slope=self.negative_slope) + hidden_states = self.upsamplers[i](hidden_states) + + # Run all resnets in parallel on hidden_states + start = i * self.resnets_per_upsample + end = (i + 1) * self.resnets_per_upsample + resnet_outputs = torch.stack([self.resnets[j](hidden_states) for j in range(start, end)], dim=0) + + hidden_states = torch.mean(resnet_outputs, dim=0) + + hidden_states = self.act_out(hidden_states) + hidden_states = self.conv_out(hidden_states) + if self.final_act_fn == "tanh": + hidden_states = torch.tanh(hidden_states) + elif self.final_act_fn == "clamp": + hidden_states = torch.clamp(hidden_states, -1, 1) + + return hidden_states + + +class CausalSTFT(nn.Module): + """ + Performs a causal short-time Fourier transform (STFT) using causal Hann windows on a waveform. The DFT bases + multiplied by the Hann windows are pre-calculated and stored as buffers. For exact parity with training, the exact + buffers should be loaded from the checkpoint in bfloat16. + """ + + def __init__(self, filter_length: int = 512, hop_length: int = 80, window_length: int = 512): + super().__init__() + self.hop_length = hop_length + self.window_length = window_length + n_freqs = filter_length // 2 + 1 + + self.register_buffer("forward_basis", torch.zeros(n_freqs * 2, 1, filter_length), persistent=True) + self.register_buffer("inverse_basis", torch.zeros(n_freqs * 2, 1, filter_length), persistent=True) + + def forward(self, waveform: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + if waveform.ndim == 2: + waveform = waveform.unsqueeze(1) # [B, num_channels, num_samples] + + left_pad = max(0, self.window_length - self.hop_length) # causal: left-only + waveform = F.pad(waveform, (left_pad, 0)) + + spec = F.conv1d(waveform, self.forward_basis, stride=self.hop_length, padding=0) + n_freqs = spec.shape[1] // 2 + real, imag = spec[:, :n_freqs], spec[:, n_freqs:] + magnitude = torch.sqrt(real**2 + imag**2) + phase = torch.atan2(imag.float(), real.float()).to(dtype=real.dtype) + return magnitude, phase + + +class MelSTFT(nn.Module): + """ + Calculates a causal log-mel spectrogram from a waveform. Uses a pre-calculated mel filterbank, which should be + loaded from the checkpoint in bfloat16. + """ + + def __init__( + self, + filter_length: int = 512, + hop_length: int = 80, + window_length: int = 512, + num_mel_channels: int = 64, + ): + super().__init__() + self.stft_fn = CausalSTFT(filter_length, hop_length, window_length) + + num_freqs = filter_length // 2 + 1 + self.register_buffer("mel_basis", torch.zeros(num_mel_channels, num_freqs), persistent=True) + + def forward(self, waveform: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + magnitude, phase = self.stft_fn(waveform) + energy = torch.norm(magnitude, dim=1) + mel = torch.matmul(self.mel_basis.to(magnitude.dtype), magnitude) + log_mel = torch.log(torch.clamp(mel, min=1e-5)) + return log_mel, magnitude, phase, energy + + +class LTX2VocoderWithBWE(ModelMixin, ConfigMixin): + """ + LTX-2.X vocoder with bandwidth extension (BWE) upsampling. The vocoder and the BWE module run in sequence, with the + BWE module upsampling the vocoder output waveform to a higher sampling rate. The BWE module itself has the same + architecture as the original vocoder. + """ + + @register_to_config + def __init__( + self, + in_channels: int = 128, + hidden_channels: int = 1536, + out_channels: int = 2, + upsample_kernel_sizes: list[int] = [11, 4, 4, 4, 4, 4], + upsample_factors: list[int] = [5, 2, 2, 2, 2, 2], + resnet_kernel_sizes: list[int] = [3, 7, 11], + resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], + act_fn: str = "snakebeta", + leaky_relu_negative_slope: float = 0.1, + antialias: bool = True, + antialias_ratio: int = 2, + antialias_kernel_size: int = 12, + final_act_fn: str | None = None, + final_bias: bool = False, + bwe_in_channels: int = 128, + bwe_hidden_channels: int = 512, + bwe_out_channels: int = 2, + bwe_upsample_kernel_sizes: list[int] = [12, 11, 4, 4, 4], + bwe_upsample_factors: list[int] = [6, 5, 2, 2, 2], + bwe_resnet_kernel_sizes: list[int] = [3, 7, 11], + bwe_resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], + bwe_act_fn: str = "snakebeta", + bwe_leaky_relu_negative_slope: float = 0.1, + bwe_antialias: bool = True, + bwe_antialias_ratio: int = 2, + bwe_antialias_kernel_size: int = 12, + bwe_final_act_fn: str | None = None, + bwe_final_bias: bool = False, + filter_length: int = 512, + hop_length: int = 80, + window_length: int = 512, + num_mel_channels: int = 64, + input_sampling_rate: int = 16000, + output_sampling_rate: int = 48000, + ): + super().__init__() + + self.vocoder = LTX2Vocoder( + in_channels=in_channels, + hidden_channels=hidden_channels, + out_channels=out_channels, + upsample_kernel_sizes=upsample_kernel_sizes, + upsample_factors=upsample_factors, + resnet_kernel_sizes=resnet_kernel_sizes, + resnet_dilations=resnet_dilations, + act_fn=act_fn, + leaky_relu_negative_slope=leaky_relu_negative_slope, + antialias=antialias, + antialias_ratio=antialias_ratio, + antialias_kernel_size=antialias_kernel_size, + final_act_fn=final_act_fn, + final_bias=final_bias, + output_sampling_rate=input_sampling_rate, + ) + self.bwe_generator = LTX2Vocoder( + in_channels=bwe_in_channels, + hidden_channels=bwe_hidden_channels, + out_channels=bwe_out_channels, + upsample_kernel_sizes=bwe_upsample_kernel_sizes, + upsample_factors=bwe_upsample_factors, + resnet_kernel_sizes=bwe_resnet_kernel_sizes, + resnet_dilations=bwe_resnet_dilations, + act_fn=bwe_act_fn, + leaky_relu_negative_slope=bwe_leaky_relu_negative_slope, + antialias=bwe_antialias, + antialias_ratio=bwe_antialias_ratio, + antialias_kernel_size=bwe_antialias_kernel_size, + final_act_fn=bwe_final_act_fn, + final_bias=bwe_final_bias, + output_sampling_rate=output_sampling_rate, + ) + + self.mel_stft = MelSTFT( + filter_length=filter_length, + hop_length=hop_length, + window_length=window_length, + num_mel_channels=num_mel_channels, + ) + + self.resampler = UpSample1d( + ratio=output_sampling_rate // input_sampling_rate, + window_type="hann", + persistent=False, + ) + + def forward(self, mel_spec: torch.Tensor) -> torch.Tensor: + r""" + Args: + mel_spec (`torch.Tensor`): + Input Mel spectrogram tensor of shape `(batch_size, num_channels, time, num_mel_bins)`. + + Returns: + `torch.Tensor`: + Bandwidth-extended audio waveform of shape `(batch_size, out_channels, audio_length)` at + `output_sampling_rate`. + """ + # 1. Run stage 1 vocoder to get low sampling rate waveform + x = self.vocoder(mel_spec) + batch_size, num_channels, num_samples = x.shape + + # Pad to exact multiple of hop_length for exact mel frame count + remainder = num_samples % self.config.hop_length + if remainder != 0: + x = F.pad(x, (0, self.hop_length - remainder)) + + # 2. Compute mel spectrogram on vocoder output + mel, _, _, _ = self.mel_stft(x.flatten(0, 1)) + mel = mel.unflatten(0, (-1, num_channels)) + + # 3. Run bandwidth extender (BWE) on new mel spectrogram + mel_for_bwe = mel.transpose(2, 3) # [B, C, num_mel_bins, num_frames] --> [B, C, num_frames, num_mel_bins] + residual = self.bwe_generator(mel_for_bwe) + + # 4. Residual connection with resampler + skip = self.resampler(x) + waveform = torch.clamp(residual + skip, -1, 1) + output_samples = num_samples * self.config.output_sampling_rate // self.config.input_sampling_rate + waveform = waveform[..., :output_samples] + return waveform diff --git a/src/diffusers/models/condition_embedders/__init__.py b/src/diffusers/models/condition_embedders/__init__.py index 9f757c3de2fa..229c2a7ee564 100644 --- a/src/diffusers/models/condition_embedders/__init__.py +++ b/src/diffusers/models/condition_embedders/__init__.py @@ -12,3 +12,4 @@ from .condition_embedder_p_h3_image import PH3ImageTextConnector from .condition_embedder_stable_audio import StableAudioProjectionModel from .condition_embedder_stable_audio_3 import StableAudio3DurationEmbedder + from .condition_embedder_stable_unclip import StableUnCLIPImageNormalizer diff --git a/src/diffusers/models/condition_embedders/condition_embedder_stable_unclip.py b/src/diffusers/models/condition_embedders/condition_embedder_stable_unclip.py new file mode 100644 index 000000000000..d376cf990a5e --- /dev/null +++ b/src/diffusers/models/condition_embedders/condition_embedder_stable_unclip.py @@ -0,0 +1,55 @@ +# Copyright 2026 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from torch import nn + +from ...configuration_utils import ConfigMixin, register_to_config +from ..modeling_utils import ModelMixin + + +class StableUnCLIPImageNormalizer(ModelMixin, ConfigMixin): + """ + This class is used to hold the mean and standard deviation of the CLIP embedder used in stable unCLIP. + + It is used to normalize the image embeddings before the noise is applied and un-normalize the noised image + embeddings. + """ + + @register_to_config + def __init__( + self, + embedding_dim: int = 768, + ): + super().__init__() + + self.mean = nn.Parameter(torch.zeros(1, embedding_dim)) + self.std = nn.Parameter(torch.ones(1, embedding_dim)) + + def to( + self, + torch_device: str | torch.device | None = None, + torch_dtype: torch.dtype | None = None, + ): + self.mean = nn.Parameter(self.mean.to(torch_device).to(torch_dtype)) + self.std = nn.Parameter(self.std.to(torch_device).to(torch_dtype)) + return self + + def scale(self, embeds): + embeds = (embeds - self.mean) * 1.0 / self.std + return embeds + + def unscale(self, embeds): + embeds = (embeds * self.std) + self.mean + return embeds diff --git a/src/diffusers/models/unets/__init__.py b/src/diffusers/models/unets/__init__.py index d3b69d6d5e8c..da77d6a80a20 100644 --- a/src/diffusers/models/unets/__init__.py +++ b/src/diffusers/models/unets/__init__.py @@ -5,6 +5,7 @@ from .unet_1d import UNet1DModel from .unet_2d import UNet2DModel from .unet_2d_condition import UNet2DConditionModel + from .unet_2d_condition_audioldm2 import AudioLDM2UNet2DConditionModel from .unet_3d_condition import UNet3DConditionModel from .unet_dreamlite import DreamLiteUNetModel from .unet_i2vgen_xl import I2VGenXLUNet diff --git a/src/diffusers/models/unets/unet_2d_condition_audioldm2.py b/src/diffusers/models/unets/unet_2d_condition_audioldm2.py new file mode 100644 index 000000000000..6b43d5d0d5ef --- /dev/null +++ b/src/diffusers/models/unets/unet_2d_condition_audioldm2.py @@ -0,0 +1,1299 @@ +# Copyright 2026 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any + +import torch +import torch.nn as nn + +from ...configuration_utils import ConfigMixin, register_to_config +from ...loaders import UNet2DConditionLoadersMixin +from ...utils import logging +from ..activations import get_activation +from ..attention import AttentionMixin +from ..attention_processor import ( + ADDED_KV_ATTENTION_PROCESSORS, + CROSS_ATTENTION_PROCESSORS, + AttnAddedKVProcessor, + AttnProcessor, +) +from ..embeddings import TimestepEmbedding, Timesteps +from ..modeling_utils import ModelMixin +from ..resnet import Downsample2D, ResnetBlock2D, Upsample2D +from ..transformers.transformer_2d import Transformer2DModel +from .unet_2d_blocks import DownBlock2D, UpBlock2D +from .unet_2d_condition import UNet2DConditionOutput + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +class AudioLDM2UNet2DConditionModel(ModelMixin, AttentionMixin, ConfigMixin, UNet2DConditionLoadersMixin): + r""" + A conditional 2D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample + shaped output. Compared to the vanilla [`UNet2DConditionModel`], this variant optionally includes an additional + self-attention layer in each Transformer block, as well as multiple cross-attention layers. It also allows for up + to two cross-attention embeddings, `encoder_hidden_states` and `encoder_hidden_states_1`. + + This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented + for all models (such as downloading or saving). + + Parameters: + sample_size (`int` or `tuple[int, int]`, *optional*, defaults to `None`): + Height and width of input/output sample. + in_channels (`int`, *optional*, defaults to 4): Number of channels in the input sample. + out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. + flip_sin_to_cos (`bool`, *optional*, defaults to `False`): + Whether to flip the sin to cos in the time embedding. + freq_shift (`int`, *optional*, defaults to 0): The frequency shift to apply to the time embedding. + down_block_types (`tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): + The tuple of downsample blocks to use. + mid_block_type (`str`, *optional*, defaults to `"UNetMidBlock2DCrossAttn"`): + Block type for middle of UNet, it can only be `UNetMidBlock2DCrossAttn` for AudioLDM2. + up_block_types (`tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): + The tuple of upsample blocks to use. + only_cross_attention (`bool` or `tuple[bool]`, *optional*, default to `False`): + Whether to include self-attention in the basic transformer blocks, see + [`~models.attention.BasicTransformerBlock`]. + block_out_channels (`tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): + The tuple of output channels for each block. + layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. + downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution. + mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block. + act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. + norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. + If `None`, normalization and activation layers is skipped in post-processing. + norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization. + cross_attention_dim (`int` or `tuple[int]`, *optional*, defaults to 1280): + The dimension of the cross attention features. + transformer_layers_per_block (`int` or `tuple[int]`, *optional*, defaults to 1): + The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for + [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.CrossAttnUpBlock2D`], + [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. + attention_head_dim (`int`, *optional*, defaults to 8): The dimension of the attention heads. + num_attention_heads (`int`, *optional*): + The number of attention heads. If not defined, defaults to `attention_head_dim` + resnet_time_scale_shift (`str`, *optional*, defaults to `"default"`): Time scale shift config + for ResNet blocks (see [`~models.resnet.ResnetBlock2D`]). Choose from `default` or `scale_shift`. + class_embed_type (`str`, *optional*, defaults to `None`): + The type of class embedding to use which is ultimately summed with the time embeddings. Choose from `None`, + `"timestep"`, `"identity"`, `"projection"`, or `"simple_projection"`. + num_class_embeds (`int`, *optional*, defaults to `None`): + Input dimension of the learnable embedding matrix to be projected to `time_embed_dim`, when performing + class conditioning with `class_embed_type` equal to `None`. + time_embedding_type (`str`, *optional*, defaults to `positional`): + The type of position embedding to use for timesteps. Choose from `positional` or `fourier`. + time_embedding_dim (`int`, *optional*, defaults to `None`): + An optional override for the dimension of the projected time embedding. + time_embedding_act_fn (`str`, *optional*, defaults to `None`): + Optional activation function to use only once on the time embeddings before they are passed to the rest of + the UNet. Choose from `silu`, `mish`, `gelu`, and `swish`. + timestep_post_act (`str`, *optional*, defaults to `None`): + The second activation function to use in timestep embedding. Choose from `silu`, `mish` and `gelu`. + time_cond_proj_dim (`int`, *optional*, defaults to `None`): + The dimension of `cond_proj` layer in the timestep embedding. + conv_in_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_in` layer. + conv_out_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_out` layer. + projection_class_embeddings_input_dim (`int`, *optional*): The dimension of the `class_labels` input when + `class_embed_type="projection"`. Required when `class_embed_type="projection"`. + class_embeddings_concat (`bool`, *optional*, defaults to `False`): Whether to concatenate the time + embeddings with the class embeddings. + """ + + _supports_gradient_checkpointing = True + + @register_to_config + def __init__( + self, + sample_size: int | None = None, + in_channels: int = 4, + out_channels: int = 4, + flip_sin_to_cos: bool = True, + freq_shift: int = 0, + down_block_types: tuple[str] = ( + "CrossAttnDownBlock2D", + "CrossAttnDownBlock2D", + "CrossAttnDownBlock2D", + "DownBlock2D", + ), + mid_block_type: str = "UNetMidBlock2DCrossAttn", + up_block_types: tuple[str] = ("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D"), + only_cross_attention: bool | tuple[bool] = False, + block_out_channels: tuple[int] = (320, 640, 1280, 1280), + layers_per_block: int | tuple[int] = 2, + downsample_padding: int = 1, + mid_block_scale_factor: float = 1, + act_fn: str = "silu", + norm_num_groups: int | None = 32, + norm_eps: float = 1e-5, + cross_attention_dim: int | tuple[int] = 1280, + transformer_layers_per_block: int | tuple[int] = 1, + attention_head_dim: int | tuple[int] = 8, + num_attention_heads: int | tuple[int] | None = None, + use_linear_projection: bool = False, + class_embed_type: str | None = None, + num_class_embeds: int | None = None, + upcast_attention: bool = False, + resnet_time_scale_shift: str = "default", + time_embedding_type: str = "positional", + time_embedding_dim: int | None = None, + time_embedding_act_fn: str | None = None, + timestep_post_act: str | None = None, + time_cond_proj_dim: int | None = None, + conv_in_kernel: int = 3, + conv_out_kernel: int = 3, + projection_class_embeddings_input_dim: int | None = None, + class_embeddings_concat: bool = False, + ): + super().__init__() + + self.sample_size = sample_size + + if num_attention_heads is not None: + raise ValueError( + "At the moment it is not possible to define the number of attention heads via `num_attention_heads` because of a naming issue as described in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131. Passing `num_attention_heads` will only be supported in diffusers v0.19." + ) + + # If `num_attention_heads` is not defined (which is the case for most models) + # it will default to `attention_head_dim`. This looks weird upon first reading it and it is. + # The reason for this behavior is to correct for incorrectly named variables that were introduced + # when this library was created. The incorrect naming was only discovered much later in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131 + # Changing `attention_head_dim` to `num_attention_heads` for 40,000+ configurations is too backwards breaking + # which is why we correct for the naming here. + num_attention_heads = num_attention_heads or attention_head_dim + + # Check inputs + if len(down_block_types) != len(up_block_types): + raise ValueError( + f"Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: {down_block_types}. `up_block_types`: {up_block_types}." + ) + + if len(block_out_channels) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: {block_out_channels}. `down_block_types`: {down_block_types}." + ) + + if not isinstance(only_cross_attention, bool) and len(only_cross_attention) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`: {only_cross_attention}. `down_block_types`: {down_block_types}." + ) + + if not isinstance(num_attention_heads, int) and len(num_attention_heads) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: {num_attention_heads}. `down_block_types`: {down_block_types}." + ) + + if not isinstance(attention_head_dim, int) and len(attention_head_dim) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `attention_head_dim` as `down_block_types`. `attention_head_dim`: {attention_head_dim}. `down_block_types`: {down_block_types}." + ) + + if isinstance(cross_attention_dim, list) and len(cross_attention_dim) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: {cross_attention_dim}. `down_block_types`: {down_block_types}." + ) + + if not isinstance(layers_per_block, int) and len(layers_per_block) != len(down_block_types): + raise ValueError( + f"Must provide the same number of `layers_per_block` as `down_block_types`. `layers_per_block`: {layers_per_block}. `down_block_types`: {down_block_types}." + ) + + # input + conv_in_padding = (conv_in_kernel - 1) // 2 + self.conv_in = nn.Conv2d( + in_channels, block_out_channels[0], kernel_size=conv_in_kernel, padding=conv_in_padding + ) + + # time + if time_embedding_type == "positional": + time_embed_dim = time_embedding_dim or block_out_channels[0] * 4 + + self.time_proj = Timesteps(block_out_channels[0], flip_sin_to_cos, freq_shift) + timestep_input_dim = block_out_channels[0] + else: + raise ValueError(f"{time_embedding_type} does not exist. Please make sure to use `positional`.") + + self.time_embedding = TimestepEmbedding( + timestep_input_dim, + time_embed_dim, + act_fn=act_fn, + post_act_fn=timestep_post_act, + cond_proj_dim=time_cond_proj_dim, + ) + + # class embedding + if class_embed_type is None and num_class_embeds is not None: + self.class_embedding = nn.Embedding(num_class_embeds, time_embed_dim) + elif class_embed_type == "timestep": + self.class_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim, act_fn=act_fn) + elif class_embed_type == "identity": + self.class_embedding = nn.Identity(time_embed_dim, time_embed_dim) + elif class_embed_type == "projection": + if projection_class_embeddings_input_dim is None: + raise ValueError( + "`class_embed_type`: 'projection' requires `projection_class_embeddings_input_dim` be set" + ) + # The projection `class_embed_type` is the same as the timestep `class_embed_type` except + # 1. the `class_labels` inputs are not first converted to sinusoidal embeddings + # 2. it projects from an arbitrary input dimension. + # + # Note that `TimestepEmbedding` is quite general, being mainly linear layers and activations. + # When used for embedding actual timesteps, the timesteps are first converted to sinusoidal embeddings. + # As a result, `TimestepEmbedding` can be passed arbitrary vectors. + self.class_embedding = TimestepEmbedding(projection_class_embeddings_input_dim, time_embed_dim) + elif class_embed_type == "simple_projection": + if projection_class_embeddings_input_dim is None: + raise ValueError( + "`class_embed_type`: 'simple_projection' requires `projection_class_embeddings_input_dim` be set" + ) + self.class_embedding = nn.Linear(projection_class_embeddings_input_dim, time_embed_dim) + else: + self.class_embedding = None + + if time_embedding_act_fn is None: + self.time_embed_act = None + else: + self.time_embed_act = get_activation(time_embedding_act_fn) + + self.down_blocks = nn.ModuleList([]) + self.up_blocks = nn.ModuleList([]) + + if isinstance(only_cross_attention, bool): + only_cross_attention = [only_cross_attention] * len(down_block_types) + + if isinstance(num_attention_heads, int): + num_attention_heads = (num_attention_heads,) * len(down_block_types) + + if isinstance(cross_attention_dim, int): + cross_attention_dim = (cross_attention_dim,) * len(down_block_types) + + if isinstance(layers_per_block, int): + layers_per_block = [layers_per_block] * len(down_block_types) + + if isinstance(transformer_layers_per_block, int): + transformer_layers_per_block = [transformer_layers_per_block] * len(down_block_types) + + if class_embeddings_concat: + # The time embeddings are concatenated with the class embeddings. The dimension of the + # time embeddings passed to the down, middle, and up blocks is twice the dimension of the + # regular time embeddings + blocks_time_embed_dim = time_embed_dim * 2 + else: + blocks_time_embed_dim = time_embed_dim + + # down + output_channel = block_out_channels[0] + for i, down_block_type in enumerate(down_block_types): + input_channel = output_channel + output_channel = block_out_channels[i] + is_final_block = i == len(block_out_channels) - 1 + + down_block = get_down_block( + down_block_type, + num_layers=layers_per_block[i], + transformer_layers_per_block=transformer_layers_per_block[i], + in_channels=input_channel, + out_channels=output_channel, + temb_channels=blocks_time_embed_dim, + add_downsample=not is_final_block, + resnet_eps=norm_eps, + resnet_act_fn=act_fn, + resnet_groups=norm_num_groups, + cross_attention_dim=cross_attention_dim[i], + num_attention_heads=num_attention_heads[i], + downsample_padding=downsample_padding, + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention[i], + upcast_attention=upcast_attention, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + self.down_blocks.append(down_block) + + # mid + if mid_block_type == "UNetMidBlock2DCrossAttn": + self.mid_block = UNetMidBlock2DCrossAttn( + transformer_layers_per_block=transformer_layers_per_block[-1], + in_channels=block_out_channels[-1], + temb_channels=blocks_time_embed_dim, + resnet_eps=norm_eps, + resnet_act_fn=act_fn, + output_scale_factor=mid_block_scale_factor, + resnet_time_scale_shift=resnet_time_scale_shift, + cross_attention_dim=cross_attention_dim[-1], + num_attention_heads=num_attention_heads[-1], + resnet_groups=norm_num_groups, + use_linear_projection=use_linear_projection, + upcast_attention=upcast_attention, + ) + else: + raise ValueError( + f"unknown mid_block_type : {mid_block_type}. Should be `UNetMidBlock2DCrossAttn` for AudioLDM2." + ) + + # count how many layers upsample the images + self.num_upsamplers = 0 + + # up + reversed_block_out_channels = list(reversed(block_out_channels)) + reversed_num_attention_heads = list(reversed(num_attention_heads)) + reversed_layers_per_block = list(reversed(layers_per_block)) + reversed_cross_attention_dim = list(reversed(cross_attention_dim)) + reversed_transformer_layers_per_block = list(reversed(transformer_layers_per_block)) + only_cross_attention = list(reversed(only_cross_attention)) + + output_channel = reversed_block_out_channels[0] + for i, up_block_type in enumerate(up_block_types): + is_final_block = i == len(block_out_channels) - 1 + + prev_output_channel = output_channel + output_channel = reversed_block_out_channels[i] + input_channel = reversed_block_out_channels[min(i + 1, len(block_out_channels) - 1)] + + # add upsample block for all BUT final layer + if not is_final_block: + add_upsample = True + self.num_upsamplers += 1 + else: + add_upsample = False + + up_block = get_up_block( + up_block_type, + num_layers=reversed_layers_per_block[i] + 1, + transformer_layers_per_block=reversed_transformer_layers_per_block[i], + in_channels=input_channel, + out_channels=output_channel, + prev_output_channel=prev_output_channel, + temb_channels=blocks_time_embed_dim, + add_upsample=add_upsample, + resnet_eps=norm_eps, + resnet_act_fn=act_fn, + resnet_groups=norm_num_groups, + cross_attention_dim=reversed_cross_attention_dim[i], + num_attention_heads=reversed_num_attention_heads[i], + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention[i], + upcast_attention=upcast_attention, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + self.up_blocks.append(up_block) + prev_output_channel = output_channel + + # out + if norm_num_groups is not None: + self.conv_norm_out = nn.GroupNorm( + num_channels=block_out_channels[0], num_groups=norm_num_groups, eps=norm_eps + ) + + self.conv_act = get_activation(act_fn) + + else: + self.conv_norm_out = None + self.conv_act = None + + conv_out_padding = (conv_out_kernel - 1) // 2 + self.conv_out = nn.Conv2d( + block_out_channels[0], out_channels, kernel_size=conv_out_kernel, padding=conv_out_padding + ) + + # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_default_attn_processor + def set_default_attn_processor(self): + """ + Disables custom attention processors and sets the default attention implementation. + """ + if all(proc.__class__ in ADDED_KV_ATTENTION_PROCESSORS for proc in self.attn_processors.values()): + processor = AttnAddedKVProcessor() + elif all(proc.__class__ in CROSS_ATTENTION_PROCESSORS for proc in self.attn_processors.values()): + processor = AttnProcessor() + else: + raise ValueError( + f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}" + ) + + self.set_attn_processor(processor) + + # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_attention_slice + def set_attention_slice(self, slice_size): + r""" + Enable sliced attention computation. + + When this option is enabled, the attention module splits the input tensor in slices to compute attention in + several steps. This is useful for saving some memory in exchange for a small decrease in speed. + + Args: + slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): + When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If + `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is + provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` + must be a multiple of `slice_size`. + """ + sliceable_head_dims = [] + + def fn_recursive_retrieve_sliceable_dims(module: torch.nn.Module): + if hasattr(module, "set_attention_slice"): + sliceable_head_dims.append(module.sliceable_head_dim) + + for child in module.children(): + fn_recursive_retrieve_sliceable_dims(child) + + # retrieve number of attention layers + for module in self.children(): + fn_recursive_retrieve_sliceable_dims(module) + + num_sliceable_layers = len(sliceable_head_dims) + + if slice_size == "auto": + # half the attention head size is usually a good trade-off between + # speed and memory + slice_size = [dim // 2 for dim in sliceable_head_dims] + elif slice_size == "max": + # make smallest slice possible + slice_size = num_sliceable_layers * [1] + + slice_size = num_sliceable_layers * [slice_size] if not isinstance(slice_size, list) else slice_size + + if len(slice_size) != len(sliceable_head_dims): + raise ValueError( + f"You have provided {len(slice_size)}, but {self.config} has {len(sliceable_head_dims)} different" + f" attention layers. Make sure to match `len(slice_size)` to be {len(sliceable_head_dims)}." + ) + + for i in range(len(slice_size)): + size = slice_size[i] + dim = sliceable_head_dims[i] + if size is not None and size > dim: + raise ValueError(f"size {size} has to be smaller or equal to {dim}.") + + # Recursively walk through all the children. + # Any children which exposes the set_attention_slice method + # gets the message + def fn_recursive_set_attention_slice(module: torch.nn.Module, slice_size: list[int]): + if hasattr(module, "set_attention_slice"): + module.set_attention_slice(slice_size.pop()) + + for child in module.children(): + fn_recursive_set_attention_slice(child, slice_size) + + reversed_slice_size = list(reversed(slice_size)) + for module in self.children(): + fn_recursive_set_attention_slice(module, reversed_slice_size) + + def forward( + self, + sample: torch.Tensor, + timestep: torch.Tensor | float | int, + encoder_hidden_states: torch.Tensor, + class_labels: torch.Tensor | None = None, + timestep_cond: torch.Tensor | None = None, + attention_mask: torch.Tensor | None = None, + cross_attention_kwargs: dict[str, Any] | None = None, + encoder_attention_mask: torch.Tensor | None = None, + return_dict: bool = True, + encoder_hidden_states_1: torch.Tensor | None = None, + encoder_attention_mask_1: torch.Tensor | None = None, + ) -> UNet2DConditionOutput | tuple: + r""" + The [`AudioLDM2UNet2DConditionModel`] forward method. + + Args: + sample (`torch.Tensor`): + The noisy input tensor with the following shape `(batch, channel, height, width)`. + timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. + encoder_hidden_states (`torch.Tensor`): + The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. + class_labels (`torch.Tensor`, *optional*, defaults to `None`): + Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. + timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): + Conditional embeddings for timestep. If provided, the embeddings will be summed with the samples passed + through the `self.time_embedding` layer to obtain the timestep embeddings. + attention_mask (`torch.Tensor`, *optional*, defaults to `None`): + An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask + is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large + negative values to the attention scores corresponding to "discard" tokens. + encoder_attention_mask (`torch.Tensor`): + A cross-attention mask of shape `(batch, sequence_length)` is applied to `encoder_hidden_states`. If + `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, + which adds large negative values to the attention scores corresponding to "discard" tokens. + return_dict (`bool`, *optional*, defaults to `True`): + Whether or not to return a [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] instead of a plain + tuple. + cross_attention_kwargs (`dict`, *optional*): + A kwargs dictionary that if specified is passed along to the [`AttnProcessor`]. + encoder_hidden_states_1 (`torch.Tensor`, *optional*): + A second set of encoder hidden states with shape `(batch, sequence_length_2, feature_dim_2)`. Can be + used to condition the model on a different set of embeddings to `encoder_hidden_states`. + encoder_attention_mask_1 (`torch.Tensor`, *optional*): + A cross-attention mask of shape `(batch, sequence_length_2)` is applied to `encoder_hidden_states_1`. + If `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, + which adds large negative values to the attention scores corresponding to "discard" tokens. + + Returns: + [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] or `tuple`: + If `return_dict` is True, an [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] is returned, + otherwise a `tuple` is returned where the first element is the sample tensor. + """ + # By default samples have to be AT least a multiple of the overall upsampling factor. + # The overall upsampling factor is equal to 2 ** (# num of upsampling layers). + # However, the upsampling interpolation output size can be forced to fit any upsampling size + # on the fly if necessary. + default_overall_up_factor = 2**self.num_upsamplers + + # upsample size should be forwarded when sample is not a multiple of `default_overall_up_factor` + forward_upsample_size = False + upsample_size = None + + if any(s % default_overall_up_factor != 0 for s in sample.shape[-2:]): + logger.info("Forward upsample size to force interpolation output size.") + forward_upsample_size = True + + # ensure attention_mask is a bias, and give it a singleton query_tokens dimension + # expects mask of shape: + # [batch, key_tokens] + # adds singleton query_tokens dimension: + # [batch, 1, key_tokens] + # this helps to broadcast it as a bias over attention scores, which will be in one of the following shapes: + # [batch, heads, query_tokens, key_tokens] (e.g. torch sdp attn) + # [batch * heads, query_tokens, key_tokens] (e.g. xformers or classic attn) + if attention_mask is not None: + # assume that mask is expressed as: + # (1 = keep, 0 = discard) + # convert mask into a bias that can be added to attention scores: + # (keep = +0, discard = -10000.0) + attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0 + attention_mask = attention_mask.unsqueeze(1) + + # convert encoder_attention_mask to a bias the same way we do for attention_mask + if encoder_attention_mask is not None: + encoder_attention_mask = (1 - encoder_attention_mask.to(sample.dtype)) * -10000.0 + encoder_attention_mask = encoder_attention_mask.unsqueeze(1) + + if encoder_attention_mask_1 is not None: + encoder_attention_mask_1 = (1 - encoder_attention_mask_1.to(sample.dtype)) * -10000.0 + encoder_attention_mask_1 = encoder_attention_mask_1.unsqueeze(1) + + # 1. time + timesteps = timestep + if not torch.is_tensor(timesteps): + # TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can + # This would be a good case for the `match` statement (Python 3.10+) + is_mps = sample.device.type == "mps" + is_npu = sample.device.type == "npu" + if isinstance(timestep, float): + dtype = torch.float32 if (is_mps or is_npu) else torch.float64 + else: + dtype = torch.int32 if (is_mps or is_npu) else torch.int64 + timesteps = torch.tensor([timesteps], dtype=dtype, device=sample.device) + elif len(timesteps.shape) == 0: + timesteps = timesteps[None].to(sample.device) + + # broadcast to batch dimension in a way that's compatible with ONNX/Core ML + timesteps = timesteps.expand(sample.shape[0]) + + t_emb = self.time_proj(timesteps) + + # `Timesteps` does not contain any weights and will always return f32 tensors + # but time_embedding might actually be running in fp16. so we need to cast here. + # there might be better ways to encapsulate this. + t_emb = t_emb.to(dtype=sample.dtype) + + emb = self.time_embedding(t_emb, timestep_cond) + aug_emb = None + + if self.class_embedding is not None: + if class_labels is None: + raise ValueError("class_labels should be provided when num_class_embeds > 0") + + if self.config.class_embed_type == "timestep": + class_labels = self.time_proj(class_labels) + + # `Timesteps` does not contain any weights and will always return f32 tensors + # there might be better ways to encapsulate this. + class_labels = class_labels.to(dtype=sample.dtype) + + class_emb = self.class_embedding(class_labels).to(dtype=sample.dtype) + + if self.config.class_embeddings_concat: + emb = torch.cat([emb, class_emb], dim=-1) + else: + emb = emb + class_emb + + emb = emb + aug_emb if aug_emb is not None else emb + + if self.time_embed_act is not None: + emb = self.time_embed_act(emb) + + # 2. pre-process + sample = self.conv_in(sample) + + # 3. down + down_block_res_samples = (sample,) + for downsample_block in self.down_blocks: + if hasattr(downsample_block, "has_cross_attention") and downsample_block.has_cross_attention: + sample, res_samples = downsample_block( + hidden_states=sample, + temb=emb, + encoder_hidden_states=encoder_hidden_states, + attention_mask=attention_mask, + cross_attention_kwargs=cross_attention_kwargs, + encoder_attention_mask=encoder_attention_mask, + encoder_hidden_states_1=encoder_hidden_states_1, + encoder_attention_mask_1=encoder_attention_mask_1, + ) + else: + sample, res_samples = downsample_block(hidden_states=sample, temb=emb) + + down_block_res_samples += res_samples + + # 4. mid + if self.mid_block is not None: + sample = self.mid_block( + sample, + emb, + encoder_hidden_states=encoder_hidden_states, + attention_mask=attention_mask, + cross_attention_kwargs=cross_attention_kwargs, + encoder_attention_mask=encoder_attention_mask, + encoder_hidden_states_1=encoder_hidden_states_1, + encoder_attention_mask_1=encoder_attention_mask_1, + ) + + # 5. up + for i, upsample_block in enumerate(self.up_blocks): + is_final_block = i == len(self.up_blocks) - 1 + + res_samples = down_block_res_samples[-len(upsample_block.resnets) :] + down_block_res_samples = down_block_res_samples[: -len(upsample_block.resnets)] + + # if we have not reached the final block and need to forward the + # upsample size, we do it here + if not is_final_block and forward_upsample_size: + upsample_size = down_block_res_samples[-1].shape[2:] + + if hasattr(upsample_block, "has_cross_attention") and upsample_block.has_cross_attention: + sample = upsample_block( + hidden_states=sample, + temb=emb, + res_hidden_states_tuple=res_samples, + encoder_hidden_states=encoder_hidden_states, + cross_attention_kwargs=cross_attention_kwargs, + upsample_size=upsample_size, + attention_mask=attention_mask, + encoder_attention_mask=encoder_attention_mask, + encoder_hidden_states_1=encoder_hidden_states_1, + encoder_attention_mask_1=encoder_attention_mask_1, + ) + else: + sample = upsample_block( + hidden_states=sample, temb=emb, res_hidden_states_tuple=res_samples, upsample_size=upsample_size + ) + + # 6. post-process + if self.conv_norm_out: + sample = self.conv_norm_out(sample) + sample = self.conv_act(sample) + sample = self.conv_out(sample) + + if not return_dict: + return (sample,) + + return UNet2DConditionOutput(sample=sample) + + +def get_down_block( + down_block_type, + num_layers, + in_channels, + out_channels, + temb_channels, + add_downsample, + resnet_eps, + resnet_act_fn, + transformer_layers_per_block=1, + num_attention_heads=None, + resnet_groups=None, + cross_attention_dim=None, + downsample_padding=None, + use_linear_projection=False, + only_cross_attention=False, + upcast_attention=False, + resnet_time_scale_shift="default", +): + down_block_type = down_block_type[7:] if down_block_type.startswith("UNetRes") else down_block_type + if down_block_type == "DownBlock2D": + return DownBlock2D( + num_layers=num_layers, + in_channels=in_channels, + out_channels=out_channels, + temb_channels=temb_channels, + add_downsample=add_downsample, + resnet_eps=resnet_eps, + resnet_act_fn=resnet_act_fn, + resnet_groups=resnet_groups, + downsample_padding=downsample_padding, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + elif down_block_type == "CrossAttnDownBlock2D": + if cross_attention_dim is None: + raise ValueError("cross_attention_dim must be specified for CrossAttnDownBlock2D") + return CrossAttnDownBlock2D( + num_layers=num_layers, + transformer_layers_per_block=transformer_layers_per_block, + in_channels=in_channels, + out_channels=out_channels, + temb_channels=temb_channels, + add_downsample=add_downsample, + resnet_eps=resnet_eps, + resnet_act_fn=resnet_act_fn, + resnet_groups=resnet_groups, + downsample_padding=downsample_padding, + cross_attention_dim=cross_attention_dim, + num_attention_heads=num_attention_heads, + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention, + upcast_attention=upcast_attention, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + raise ValueError(f"{down_block_type} does not exist.") + + +def get_up_block( + up_block_type, + num_layers, + in_channels, + out_channels, + prev_output_channel, + temb_channels, + add_upsample, + resnet_eps, + resnet_act_fn, + transformer_layers_per_block=1, + num_attention_heads=None, + resnet_groups=None, + cross_attention_dim=None, + use_linear_projection=False, + only_cross_attention=False, + upcast_attention=False, + resnet_time_scale_shift="default", +): + up_block_type = up_block_type[7:] if up_block_type.startswith("UNetRes") else up_block_type + if up_block_type == "UpBlock2D": + return UpBlock2D( + num_layers=num_layers, + in_channels=in_channels, + out_channels=out_channels, + prev_output_channel=prev_output_channel, + temb_channels=temb_channels, + add_upsample=add_upsample, + resnet_eps=resnet_eps, + resnet_act_fn=resnet_act_fn, + resnet_groups=resnet_groups, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + elif up_block_type == "CrossAttnUpBlock2D": + if cross_attention_dim is None: + raise ValueError("cross_attention_dim must be specified for CrossAttnUpBlock2D") + return CrossAttnUpBlock2D( + num_layers=num_layers, + transformer_layers_per_block=transformer_layers_per_block, + in_channels=in_channels, + out_channels=out_channels, + prev_output_channel=prev_output_channel, + temb_channels=temb_channels, + add_upsample=add_upsample, + resnet_eps=resnet_eps, + resnet_act_fn=resnet_act_fn, + resnet_groups=resnet_groups, + cross_attention_dim=cross_attention_dim, + num_attention_heads=num_attention_heads, + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention, + upcast_attention=upcast_attention, + resnet_time_scale_shift=resnet_time_scale_shift, + ) + raise ValueError(f"{up_block_type} does not exist.") + + +class CrossAttnDownBlock2D(nn.Module): + def __init__( + self, + in_channels: int, + out_channels: int, + temb_channels: int, + dropout: float = 0.0, + num_layers: int = 1, + transformer_layers_per_block: int = 1, + resnet_eps: float = 1e-6, + resnet_time_scale_shift: str = "default", + resnet_act_fn: str = "swish", + resnet_groups: int = 32, + resnet_pre_norm: bool = True, + num_attention_heads=1, + cross_attention_dim=1280, + output_scale_factor=1.0, + downsample_padding=1, + add_downsample=True, + use_linear_projection=False, + only_cross_attention=False, + upcast_attention=False, + ): + super().__init__() + resnets = [] + attentions = [] + + self.has_cross_attention = True + self.num_attention_heads = num_attention_heads + + if isinstance(cross_attention_dim, int): + cross_attention_dim = (cross_attention_dim,) + if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: + raise ValueError( + "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " + f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" + ) + self.cross_attention_dim = cross_attention_dim + + for i in range(num_layers): + in_channels = in_channels if i == 0 else out_channels + resnets.append( + ResnetBlock2D( + in_channels=in_channels, + out_channels=out_channels, + temb_channels=temb_channels, + eps=resnet_eps, + groups=resnet_groups, + dropout=dropout, + time_embedding_norm=resnet_time_scale_shift, + non_linearity=resnet_act_fn, + output_scale_factor=output_scale_factor, + pre_norm=resnet_pre_norm, + ) + ) + for j in range(len(cross_attention_dim)): + attentions.append( + Transformer2DModel( + num_attention_heads, + out_channels // num_attention_heads, + in_channels=out_channels, + num_layers=transformer_layers_per_block, + cross_attention_dim=cross_attention_dim[j], + norm_num_groups=resnet_groups, + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention, + upcast_attention=upcast_attention, + double_self_attention=True if cross_attention_dim[j] is None else False, + ) + ) + self.attentions = nn.ModuleList(attentions) + self.resnets = nn.ModuleList(resnets) + + if add_downsample: + self.downsamplers = nn.ModuleList( + [ + Downsample2D( + out_channels, use_conv=True, out_channels=out_channels, padding=downsample_padding, name="op" + ) + ] + ) + else: + self.downsamplers = None + + self.gradient_checkpointing = False + + def forward( + self, + hidden_states: torch.Tensor, + temb: torch.Tensor | None = None, + encoder_hidden_states: torch.Tensor | None = None, + attention_mask: torch.Tensor | None = None, + cross_attention_kwargs: dict[str, Any] | None = None, + encoder_attention_mask: torch.Tensor | None = None, + encoder_hidden_states_1: torch.Tensor | None = None, + encoder_attention_mask_1: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, tuple[torch.Tensor, ...]]: + output_states = () + num_layers = len(self.resnets) + num_attention_per_layer = len(self.attentions) // num_layers + + encoder_hidden_states_1 = ( + encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states + ) + encoder_attention_mask_1 = ( + encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask + ) + + for i in range(num_layers): + if torch.is_grad_enabled() and self.gradient_checkpointing: + hidden_states = self._gradient_checkpointing_func(self.resnets[i], hidden_states, temb) + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self._gradient_checkpointing_func( + self.attentions[i * num_attention_per_layer + idx], + hidden_states, + forward_encoder_hidden_states, + None, # timestep + None, # class_labels + cross_attention_kwargs, + attention_mask, + forward_encoder_attention_mask, + )[0] + else: + hidden_states = self.resnets[i](hidden_states, temb) + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self.attentions[i * num_attention_per_layer + idx]( + hidden_states, + attention_mask=attention_mask, + encoder_hidden_states=forward_encoder_hidden_states, + encoder_attention_mask=forward_encoder_attention_mask, + return_dict=False, + )[0] + + output_states = output_states + (hidden_states,) + + if self.downsamplers is not None: + for downsampler in self.downsamplers: + hidden_states = downsampler(hidden_states) + + output_states = output_states + (hidden_states,) + + return hidden_states, output_states + + +class UNetMidBlock2DCrossAttn(nn.Module): + def __init__( + self, + in_channels: int, + temb_channels: int, + dropout: float = 0.0, + num_layers: int = 1, + transformer_layers_per_block: int = 1, + resnet_eps: float = 1e-6, + resnet_time_scale_shift: str = "default", + resnet_act_fn: str = "swish", + resnet_groups: int = 32, + resnet_pre_norm: bool = True, + num_attention_heads=1, + output_scale_factor=1.0, + cross_attention_dim=1280, + use_linear_projection=False, + upcast_attention=False, + ): + super().__init__() + + self.has_cross_attention = True + self.num_attention_heads = num_attention_heads + resnet_groups = resnet_groups if resnet_groups is not None else min(in_channels // 4, 32) + + if isinstance(cross_attention_dim, int): + cross_attention_dim = (cross_attention_dim,) + if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: + raise ValueError( + "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " + f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" + ) + self.cross_attention_dim = cross_attention_dim + + # there is always at least one resnet + resnets = [ + ResnetBlock2D( + in_channels=in_channels, + out_channels=in_channels, + temb_channels=temb_channels, + eps=resnet_eps, + groups=resnet_groups, + dropout=dropout, + time_embedding_norm=resnet_time_scale_shift, + non_linearity=resnet_act_fn, + output_scale_factor=output_scale_factor, + pre_norm=resnet_pre_norm, + ) + ] + attentions = [] + + for i in range(num_layers): + for j in range(len(cross_attention_dim)): + attentions.append( + Transformer2DModel( + num_attention_heads, + in_channels // num_attention_heads, + in_channels=in_channels, + num_layers=transformer_layers_per_block, + cross_attention_dim=cross_attention_dim[j], + norm_num_groups=resnet_groups, + use_linear_projection=use_linear_projection, + upcast_attention=upcast_attention, + double_self_attention=True if cross_attention_dim[j] is None else False, + ) + ) + resnets.append( + ResnetBlock2D( + in_channels=in_channels, + out_channels=in_channels, + temb_channels=temb_channels, + eps=resnet_eps, + groups=resnet_groups, + dropout=dropout, + time_embedding_norm=resnet_time_scale_shift, + non_linearity=resnet_act_fn, + output_scale_factor=output_scale_factor, + pre_norm=resnet_pre_norm, + ) + ) + + self.attentions = nn.ModuleList(attentions) + self.resnets = nn.ModuleList(resnets) + + self.gradient_checkpointing = False + + def forward( + self, + hidden_states: torch.Tensor, + temb: torch.Tensor | None = None, + encoder_hidden_states: torch.Tensor | None = None, + attention_mask: torch.Tensor | None = None, + cross_attention_kwargs: dict[str, Any] | None = None, + encoder_attention_mask: torch.Tensor | None = None, + encoder_hidden_states_1: torch.Tensor | None = None, + encoder_attention_mask_1: torch.Tensor | None = None, + ) -> torch.Tensor: + hidden_states = self.resnets[0](hidden_states, temb) + num_attention_per_layer = len(self.attentions) // (len(self.resnets) - 1) + + encoder_hidden_states_1 = ( + encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states + ) + encoder_attention_mask_1 = ( + encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask + ) + + for i in range(len(self.resnets[1:])): + if torch.is_grad_enabled() and self.gradient_checkpointing: + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self._gradient_checkpointing_func( + self.attentions[i * num_attention_per_layer + idx], + hidden_states, + forward_encoder_hidden_states, + None, # timestep + None, # class_labels + cross_attention_kwargs, + attention_mask, + forward_encoder_attention_mask, + )[0] + hidden_states = self._gradient_checkpointing_func(self.resnets[i + 1], hidden_states, temb) + else: + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self.attentions[i * num_attention_per_layer + idx]( + hidden_states, + attention_mask=attention_mask, + encoder_hidden_states=forward_encoder_hidden_states, + encoder_attention_mask=forward_encoder_attention_mask, + return_dict=False, + )[0] + + hidden_states = self.resnets[i + 1](hidden_states, temb) + + return hidden_states + + +class CrossAttnUpBlock2D(nn.Module): + def __init__( + self, + in_channels: int, + out_channels: int, + prev_output_channel: int, + temb_channels: int, + dropout: float = 0.0, + num_layers: int = 1, + transformer_layers_per_block: int = 1, + resnet_eps: float = 1e-6, + resnet_time_scale_shift: str = "default", + resnet_act_fn: str = "swish", + resnet_groups: int = 32, + resnet_pre_norm: bool = True, + num_attention_heads=1, + cross_attention_dim=1280, + output_scale_factor=1.0, + add_upsample=True, + use_linear_projection=False, + only_cross_attention=False, + upcast_attention=False, + ): + super().__init__() + resnets = [] + attentions = [] + + self.has_cross_attention = True + self.num_attention_heads = num_attention_heads + + if isinstance(cross_attention_dim, int): + cross_attention_dim = (cross_attention_dim,) + if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: + raise ValueError( + "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " + f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" + ) + self.cross_attention_dim = cross_attention_dim + + for i in range(num_layers): + res_skip_channels = in_channels if (i == num_layers - 1) else out_channels + resnet_in_channels = prev_output_channel if i == 0 else out_channels + + resnets.append( + ResnetBlock2D( + in_channels=resnet_in_channels + res_skip_channels, + out_channels=out_channels, + temb_channels=temb_channels, + eps=resnet_eps, + groups=resnet_groups, + dropout=dropout, + time_embedding_norm=resnet_time_scale_shift, + non_linearity=resnet_act_fn, + output_scale_factor=output_scale_factor, + pre_norm=resnet_pre_norm, + ) + ) + for j in range(len(cross_attention_dim)): + attentions.append( + Transformer2DModel( + num_attention_heads, + out_channels // num_attention_heads, + in_channels=out_channels, + num_layers=transformer_layers_per_block, + cross_attention_dim=cross_attention_dim[j], + norm_num_groups=resnet_groups, + use_linear_projection=use_linear_projection, + only_cross_attention=only_cross_attention, + upcast_attention=upcast_attention, + double_self_attention=True if cross_attention_dim[j] is None else False, + ) + ) + self.attentions = nn.ModuleList(attentions) + self.resnets = nn.ModuleList(resnets) + + if add_upsample: + self.upsamplers = nn.ModuleList([Upsample2D(out_channels, use_conv=True, out_channels=out_channels)]) + else: + self.upsamplers = None + + self.gradient_checkpointing = False + + def forward( + self, + hidden_states: torch.Tensor, + res_hidden_states_tuple: tuple[torch.Tensor, ...], + temb: torch.Tensor | None = None, + encoder_hidden_states: torch.Tensor | None = None, + cross_attention_kwargs: dict[str, Any] | None = None, + upsample_size: int | None = None, + attention_mask: torch.Tensor | None = None, + encoder_attention_mask: torch.Tensor | None = None, + encoder_hidden_states_1: torch.Tensor | None = None, + encoder_attention_mask_1: torch.Tensor | None = None, + ) -> torch.Tensor: + num_layers = len(self.resnets) + num_attention_per_layer = len(self.attentions) // num_layers + + encoder_hidden_states_1 = ( + encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states + ) + encoder_attention_mask_1 = ( + encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask + ) + + for i in range(num_layers): + # pop res hidden states + res_hidden_states = res_hidden_states_tuple[-1] + res_hidden_states_tuple = res_hidden_states_tuple[:-1] + hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1) + + if torch.is_grad_enabled() and self.gradient_checkpointing: + hidden_states = self._gradient_checkpointing_func(self.resnets[i], hidden_states, temb) + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self._gradient_checkpointing_func( + self.attentions[i * num_attention_per_layer + idx], + hidden_states, + forward_encoder_hidden_states, + None, # timestep + None, # class_labels + cross_attention_kwargs, + attention_mask, + forward_encoder_attention_mask, + )[0] + else: + hidden_states = self.resnets[i](hidden_states, temb) + for idx, cross_attention_dim in enumerate(self.cross_attention_dim): + if cross_attention_dim is not None and idx <= 1: + forward_encoder_hidden_states = encoder_hidden_states + forward_encoder_attention_mask = encoder_attention_mask + elif cross_attention_dim is not None and idx > 1: + forward_encoder_hidden_states = encoder_hidden_states_1 + forward_encoder_attention_mask = encoder_attention_mask_1 + else: + forward_encoder_hidden_states = None + forward_encoder_attention_mask = None + hidden_states = self.attentions[i * num_attention_per_layer + idx]( + hidden_states, + attention_mask=attention_mask, + encoder_hidden_states=forward_encoder_hidden_states, + encoder_attention_mask=forward_encoder_attention_mask, + return_dict=False, + )[0] + + if self.upsamplers is not None: + for upsampler in self.upsamplers: + hidden_states = upsampler(hidden_states, upsample_size) + + return hidden_states diff --git a/src/diffusers/modular_pipelines/echo/decoders.py b/src/diffusers/modular_pipelines/echo/decoders.py index 846f1ec559d5..0bc695a658d8 100644 --- a/src/diffusers/modular_pipelines/echo/decoders.py +++ b/src/diffusers/modular_pipelines/echo/decoders.py @@ -18,7 +18,7 @@ from ...configuration_utils import FrozenDict from ...models import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video -from ...pipelines.ltx2.vocoder import LTX2Vocoder +from ...models.autoencoders import LTX2Vocoder from ...utils.torch_utils import randn_tensor from ...video_processor import VideoProcessor from ..modular_pipeline import ModularPipelineBlocks, PipelineState diff --git a/src/diffusers/modular_pipelines/ltx2/decoders.py b/src/diffusers/modular_pipelines/ltx2/decoders.py index 34a2e9b6a74f..6cf31e9a1939 100644 --- a/src/diffusers/modular_pipelines/ltx2/decoders.py +++ b/src/diffusers/modular_pipelines/ltx2/decoders.py @@ -18,13 +18,7 @@ from ...configuration_utils import FrozenDict from ...models import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2VideoDiffusionDecoderModel - -# NOTE (modular.md gotcha #1): `LTX2Vocoder` / `LTX2VocoderWithBWE` currently live under -# `diffusers.pipelines.ltx2.vocoder`, and modular blocks must not import from `diffusers.pipelines.*`. -# They are already `ModelMixin` / `ConfigMixin` model classes, so the clean fix is to relocate them to -# `src/diffusers/models/` and re-export from `diffusers.models` before this lands. Imported from the -# pipelines path here only so the draft is runnable; switch to the models path once moved. -from ...pipelines.ltx2.vocoder import LTX2Vocoder +from ...models.autoencoders import LTX2Vocoder from ...utils import logging from ...utils.torch_utils import randn_tensor from ...video_processor import VideoProcessor diff --git a/src/diffusers/pipelines/__init__.py b/src/diffusers/pipelines/__init__.py index 7f0db0230b1d..33e8f812e721 100644 --- a/src/diffusers/pipelines/__init__.py +++ b/src/diffusers/pipelines/__init__.py @@ -113,6 +113,9 @@ "PIAPipeline", "PaintByExamplePipeline", "SemanticStableDiffusionPipeline", + "ShapEImg2ImgPipeline", + "ShapEPipeline", + "shap_e", "StableDiffusionAttendAndExcitePipeline", "StableDiffusionControlNetXSPipeline", "StableDiffusionDiffEditPipeline", @@ -394,7 +397,6 @@ "SanaWMLTX2Refiner", "SanaWMPipelineOutput", ] - _import_structure["shap_e"] = ["ShapEImg2ImgPipeline", "ShapEPipeline"] _import_structure["stable_audio"] = [ "StableAudioProjectionModel", "StableAudioPipeline", @@ -658,6 +660,8 @@ PaintByExamplePipeline, PIAPipeline, SemanticStableDiffusionPipeline, + ShapEImg2ImgPipeline, + ShapEPipeline, StableDiffusionAttendAndExcitePipeline, StableDiffusionControlNetXSPipeline, StableDiffusionDiffEditPipeline, @@ -689,6 +693,7 @@ WuerstchenCombinedPipeline, WuerstchenDecoderPipeline, WuerstchenPriorPipeline, + shap_e, ) from .diffusion_gemma import DiffusionGemmaPipeline, DiffusionGemmaPipelineOutput from .dreamlite import ( @@ -872,7 +877,6 @@ SanaWMPipeline, SanaWMPipelineOutput, ) - from .shap_e import ShapEImg2ImgPipeline, ShapEPipeline from .stable_audio import StableAudioPipeline, StableAudioProjectionModel from .stable_audio_3 import ( StableAudio3AudioToAudioPipeline, diff --git a/src/diffusers/pipelines/ace_step/modeling_ace_step.py b/src/diffusers/pipelines/ace_step/modeling_ace_step.py index 199e97fc9169..6bafac420ea0 100644 --- a/src/diffusers/pipelines/ace_step/modeling_ace_step.py +++ b/src/diffusers/pipelines/ace_step/modeling_ace_step.py @@ -12,411 +12,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Pipeline-specific models for ACE-Step 1.5. - -Holds the audio tokenizer / detokenizer used by cover conditioning. The condition encoder now lives in -``diffusers/models/condition_embedders/condition_embedder_ace_step.py``; the names below are kept as deprecated aliases -for backward compatibility. -""" - -from typing import Optional, Tuple - -import torch -import torch.nn as nn -import torch.nn.functional as F - -from ...configuration_utils import ConfigMixin, register_to_config +from ...models.autoencoders.audio_tokenizer_ace_step import ( + AceStepAudioTokenDetokenizer as _AceStepAudioTokenDetokenizer, +) +from ...models.autoencoders.audio_tokenizer_ace_step import AceStepAudioTokenizer as _AceStepAudioTokenizer from ...models.condition_embedders.condition_embedder_ace_step import ( AceStepConditionEncoder as _AceStepConditionEncoder, ) -from ...models.condition_embedders.condition_embedder_ace_step import AceStepEncoderLayer from ...models.condition_embedders.condition_embedder_ace_step import AceStepLyricEncoder as _AceStepLyricEncoder from ...models.condition_embedders.condition_embedder_ace_step import AceStepTimbreEncoder as _AceStepTimbreEncoder -from ...models.modeling_utils import ModelMixin -from ...models.normalization import RMSNorm -from ...models.transformers.ace_step_transformer import ( - _ace_step_rotary_freqs, - _create_4d_mask, - _is_flash_attention_backend, -) -from ...utils import deprecate, logging - - -logger = logging.get_logger(__name__) # pylint: disable=invalid-name - - -# --------------------------------------------------------------------------- # -# audio tokenizer / detokenizer # -# --------------------------------------------------------------------------- # - - -class _AceStepResidualFSQ(nn.Module): - """Minimal ResidualFSQ compatible with ACE-Step's saved tokenizer weights.""" - - def __init__( - self, - dim: int = 2048, - levels: Optional[list] = None, - num_quantizers: int = 1, - ): - super().__init__() - - if levels is None: - levels = [8, 8, 8, 5, 5, 5] - - self.levels = levels - self.num_quantizers = num_quantizers - self.codebook_dim = len(levels) - - self.project_in = nn.Linear(dim, self.codebook_dim) - self.project_out = nn.Linear(self.codebook_dim, dim) - - levels_tensor = torch.tensor(levels, dtype=torch.long) - basis = torch.cumprod(torch.tensor([1] + levels[:-1], dtype=torch.long), dim=0) - scales = torch.stack([levels_tensor.float() ** -i for i in range(num_quantizers)]) - self.register_buffer("_levels", levels_tensor, persistent=False) - self.register_buffer("_basis", basis, persistent=False) - self.register_buffer("scales", scales, persistent=False) - - @property - def codebook_size(self) -> int: - return int(torch.prod(self._levels).item()) - - def _indices_to_codes(self, indices: torch.Tensor) -> torch.Tensor: - levels = self._levels.to(device=indices.device) - basis = self._basis.to(device=indices.device) - level_indices = (indices.long().unsqueeze(-1) // basis) % levels - scale = 2.0 / (levels.to(dtype=torch.float32) - 1.0) - return level_indices.to(dtype=torch.float32) * scale - 1.0 - - def _codes_to_indices(self, codes: torch.Tensor) -> torch.Tensor: - levels = self._levels.to(device=codes.device, dtype=codes.dtype) - basis = self._basis.to(device=codes.device, dtype=codes.dtype) - level_indices = (codes + 1.0) / (2.0 / (levels - 1.0)) - return (level_indices * basis).sum(dim=-1).round().to(torch.long) - - def _quantize(self, x: torch.Tensor) -> torch.Tensor: - levels = self._levels.to(device=x.device, dtype=x.dtype) - levels_minus_one = levels - 1.0 - step = 2.0 / levels_minus_one - bracket = levels_minus_one * (x.clamp(-1.0, 1.0) + 1.0) / 2.0 + 0.5 - return step * torch.floor(bracket) - 1.0 - - def get_codes_from_indices(self, indices: torch.Tensor) -> torch.Tensor: - if indices.ndim == 2: - indices = indices.unsqueeze(-1) - if indices.shape[-1] != self.num_quantizers: - raise ValueError( - f"Expected audio code indices with last dimension {self.num_quantizers}, got {indices.shape[-1]}." - ) - - codes = [] - for quantizer_idx in range(self.num_quantizers): - code = self._indices_to_codes(indices[..., quantizer_idx]) - scale = self.scales[quantizer_idx].to(device=code.device, dtype=code.dtype) - codes.append(code * scale) - return torch.stack(codes, dim=0) - - def get_output_from_indices(self, indices: torch.Tensor) -> torch.Tensor: - codes = self.get_codes_from_indices(indices).sum(dim=0) - weight = self.project_out.weight.float() - bias = self.project_out.bias.float() if self.project_out.bias is not None else None - output = F.linear(codes.float(), weight, bias) - return output.to(dtype=self.project_out.weight.dtype) - - def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - input_dtype = hidden_states.dtype - weight = self.project_in.weight.float() - bias = self.project_in.bias.float() if self.project_in.bias is not None else None - hidden_states = F.linear(hidden_states.float(), weight, bias) - - levels = self._levels.to(device=hidden_states.device, dtype=hidden_states.dtype) - soft_clamp = 1.0 + (1.0 / (levels - 1.0)) - hidden_states = (hidden_states / soft_clamp).tanh() * soft_clamp - - quantized_out = torch.zeros_like(hidden_states) - residual = hidden_states - all_indices = [] - for scale in self.scales.to(device=hidden_states.device, dtype=hidden_states.dtype): - quantized = self._quantize(residual / scale) * scale - residual = residual - quantized.detach() - quantized_out = quantized_out + quantized - all_indices.append(self._codes_to_indices(quantized / scale)) - - weight = self.project_out.weight.float() - bias = self.project_out.bias.float() if self.project_out.bias is not None else None - quantized_out = F.linear(quantized_out.float(), weight, bias).to(dtype=input_dtype) - all_indices = torch.stack(all_indices, dim=-1) - return quantized_out, all_indices - - -class AceStepAttentionPooler(nn.Module): - """Attention pooler used by the ACE-Step audio tokenizer.""" - - def __init__( - self, - hidden_size: int = 2048, - intermediate_size: int = 6144, - num_attention_pooler_hidden_layers: int = 2, - num_attention_heads: int = 16, - num_key_value_heads: int = 8, - head_dim: int = 128, - rope_theta: float = 1000000.0, - attention_bias: bool = False, - attention_dropout: float = 0.0, - rms_norm_eps: float = 1e-6, - sliding_window: int = 128, - layer_types: list = None, - ): - super().__init__() - - if layer_types is None: - layer_types = [ - "sliding_attention" if bool((i + 1) % 2) else "full_attention" - for i in range(num_attention_pooler_hidden_layers) - ] - - self.embed_tokens = nn.Linear(hidden_size, hidden_size) - self.norm = RMSNorm(hidden_size, eps=rms_norm_eps) - self.special_token = nn.Parameter(torch.randn(1, 1, hidden_size) * 0.02) - self.head_dim = head_dim - self.rope_theta = rope_theta - self.sliding_window = sliding_window - self.layers = nn.ModuleList( - [ - AceStepEncoderLayer( - hidden_size=hidden_size, - num_attention_heads=num_attention_heads, - num_key_value_heads=num_key_value_heads, - head_dim=head_dim, - intermediate_size=intermediate_size, - attention_bias=attention_bias, - attention_dropout=attention_dropout, - rms_norm_eps=rms_norm_eps, - sliding_window=sliding_window if layer_types[i] == "sliding_attention" else None, - ) - for i in range(num_attention_pooler_hidden_layers) - ] - ) - self._layer_types = layer_types - - def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: - batch_size, num_patches, patch_size, _ = hidden_states.shape - hidden_states = self.embed_tokens(hidden_states) - special_token = self.special_token.to(device=hidden_states.device, dtype=hidden_states.dtype) - special_token = special_token.expand(batch_size, num_patches, -1, -1) - hidden_states = torch.cat([special_token, hidden_states], dim=2) - hidden_states = hidden_states.reshape(batch_size * num_patches, patch_size + 1, -1) - - seq_len = hidden_states.shape[1] - dtype = hidden_states.dtype - device = hidden_states.device - position_embeddings = _ace_step_rotary_freqs(seq_len, self.head_dim, self.rope_theta, device, dtype) - sliding_attn_mask = None - if not _is_flash_attention_backend(self.layers[0].self_attn.processor): - sliding_attn_mask = _create_4d_mask( - seq_len=seq_len, - dtype=dtype, - device=device, - attention_mask=None, - sliding_window=self.sliding_window, - is_sliding_window=True, - is_causal=False, - ) - - for i, layer_module in enumerate(self.layers): - mask = sliding_attn_mask if self._layer_types[i] == "sliding_attention" else None - hidden_states = layer_module( - hidden_states=hidden_states, - position_embeddings=position_embeddings, - attention_mask=mask, - ) - - hidden_states = self.norm(hidden_states) - hidden_states = hidden_states[:, 0, :] - return hidden_states.reshape(batch_size, num_patches, -1) - - -class AceStepAudioTokenDetokenizer(ModelMixin, ConfigMixin): - """Expands ACE-Step 5 Hz audio tokens back to 25 Hz acoustic conditioning.""" - - _supports_gradient_checkpointing = True - - @register_to_config - def __init__( - self, - hidden_size: int = 2048, - intermediate_size: int = 6144, - audio_acoustic_hidden_dim: int = 64, - pool_window_size: int = 5, - num_attention_pooler_hidden_layers: int = 2, - num_attention_heads: int = 16, - num_key_value_heads: int = 8, - head_dim: int = 128, - rope_theta: float = 1000000.0, - attention_bias: bool = False, - attention_dropout: float = 0.0, - rms_norm_eps: float = 1e-6, - sliding_window: int = 128, - layer_types: list = None, - ): - super().__init__() - - if layer_types is None: - layer_types = [ - "sliding_attention" if bool((i + 1) % 2) else "full_attention" - for i in range(num_attention_pooler_hidden_layers) - ] - - self.embed_tokens = nn.Linear(hidden_size, hidden_size) - self.norm = RMSNorm(hidden_size, eps=rms_norm_eps) - self.special_tokens = nn.Parameter(torch.randn(1, pool_window_size, hidden_size) * 0.02) - self.proj_out = nn.Linear(hidden_size, audio_acoustic_hidden_dim) - self.head_dim = head_dim - self.rope_theta = rope_theta - self.sliding_window = sliding_window - self.pool_window_size = pool_window_size - self.layers = nn.ModuleList( - [ - AceStepEncoderLayer( - hidden_size=hidden_size, - num_attention_heads=num_attention_heads, - num_key_value_heads=num_key_value_heads, - head_dim=head_dim, - intermediate_size=intermediate_size, - attention_bias=attention_bias, - attention_dropout=attention_dropout, - rms_norm_eps=rms_norm_eps, - sliding_window=sliding_window if layer_types[i] == "sliding_attention" else None, - ) - for i in range(num_attention_pooler_hidden_layers) - ] - ) - self._layer_types = layer_types - self.gradient_checkpointing = False - - def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: - batch_size, num_tokens, _ = hidden_states.shape - hidden_states = self.embed_tokens(hidden_states) - hidden_states = hidden_states.unsqueeze(2).expand(-1, -1, self.pool_window_size, -1) - special_tokens = self.special_tokens.to(device=hidden_states.device, dtype=hidden_states.dtype) - hidden_states = hidden_states + special_tokens.unsqueeze(0) - hidden_states = hidden_states.reshape(batch_size * num_tokens, self.pool_window_size, -1) - - seq_len = hidden_states.shape[1] - dtype = hidden_states.dtype - device = hidden_states.device - position_embeddings = _ace_step_rotary_freqs(seq_len, self.head_dim, self.rope_theta, device, dtype) - sliding_attn_mask = None - if not _is_flash_attention_backend(self.layers[0].self_attn.processor): - sliding_attn_mask = _create_4d_mask( - seq_len=seq_len, - dtype=dtype, - device=device, - attention_mask=None, - sliding_window=self.sliding_window, - is_sliding_window=True, - is_causal=False, - ) - - for i, layer_module in enumerate(self.layers): - mask = sliding_attn_mask if self._layer_types[i] == "sliding_attention" else None - if torch.is_grad_enabled() and self.gradient_checkpointing: - hidden_states = self._gradient_checkpointing_func( - layer_module, hidden_states, position_embeddings, mask - ) - else: - hidden_states = layer_module( - hidden_states=hidden_states, - position_embeddings=position_embeddings, - attention_mask=mask, - ) - - hidden_states = self.norm(hidden_states) - hidden_states = self.proj_out(hidden_states) - return hidden_states.reshape(batch_size, num_tokens * self.pool_window_size, -1) - - -class AceStepAudioTokenizer(ModelMixin, ConfigMixin): - """Converts 25 Hz acoustic latents to ACE-Step 5 Hz audio tokens.""" - - _supports_gradient_checkpointing = True - - @register_to_config - def __init__( - self, - hidden_size: int = 2048, - intermediate_size: int = 6144, - audio_acoustic_hidden_dim: int = 64, - pool_window_size: int = 5, - fsq_dim: int = 2048, - fsq_input_levels: list = None, - fsq_input_num_quantizers: int = 1, - num_attention_pooler_hidden_layers: int = 2, - num_attention_heads: int = 16, - num_key_value_heads: int = 8, - head_dim: int = 128, - rope_theta: float = 1000000.0, - attention_bias: bool = False, - attention_dropout: float = 0.0, - rms_norm_eps: float = 1e-6, - sliding_window: int = 128, - layer_types: list = None, - ): - super().__init__() - - if fsq_input_levels is None: - fsq_input_levels = [8, 8, 8, 5, 5, 5] - - self.audio_acoustic_proj = nn.Linear(audio_acoustic_hidden_dim, hidden_size) - self.attention_pooler = AceStepAttentionPooler( - hidden_size=hidden_size, - intermediate_size=intermediate_size, - num_attention_pooler_hidden_layers=num_attention_pooler_hidden_layers, - num_attention_heads=num_attention_heads, - num_key_value_heads=num_key_value_heads, - head_dim=head_dim, - rope_theta=rope_theta, - attention_bias=attention_bias, - attention_dropout=attention_dropout, - rms_norm_eps=rms_norm_eps, - sliding_window=sliding_window, - layer_types=layer_types, - ) - self.quantizer = _AceStepResidualFSQ( - dim=fsq_dim, - levels=fsq_input_levels, - num_quantizers=fsq_input_num_quantizers, - ) - self.pool_window_size = pool_window_size - - def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - input_dtype = hidden_states.dtype - hidden_states = self.audio_acoustic_proj(hidden_states) - hidden_states = self.attention_pooler(hidden_states) - quantized, indices = self.quantizer(hidden_states) - return quantized.to(dtype=input_dtype), indices - - def tokenize( - self, - hidden_states: torch.Tensor, - silence_latent: Optional[torch.Tensor] = None, - ) -> Tuple[torch.Tensor, torch.Tensor]: - batch_size, latent_length, acoustic_dim = hidden_states.shape - pad_len = (-latent_length) % self.pool_window_size - if pad_len: - if silence_latent is not None and silence_latent.shape[-1] == acoustic_dim: - pad = silence_latent[:, :pad_len, :].to(device=hidden_states.device, dtype=hidden_states.dtype) - pad = pad.expand(batch_size, -1, -1) - else: - pad = torch.zeros( - batch_size, pad_len, acoustic_dim, device=hidden_states.device, dtype=hidden_states.dtype - ) - hidden_states = torch.cat([hidden_states, pad], dim=1) - - num_patches = hidden_states.shape[1] // self.pool_window_size - hidden_states = hidden_states.reshape(batch_size, num_patches, self.pool_window_size, acoustic_dim) - return self(hidden_states) +from ...utils import deprecate # Deprecated aliases kept for backward compatibility. The warning is raised from `__new__` rather than `__init__` so @@ -452,3 +57,25 @@ def __new__(cls, *args, **kwargs): "Import it with `from diffusers import AceStepConditionEncoder` instead.", ) return super().__new__(cls) + + +class AceStepAudioTokenizer(_AceStepAudioTokenizer): + def __new__(cls, *args, **kwargs): + deprecate( + "AceStepAudioTokenizer", + "1.0.0", + "Importing `AceStepAudioTokenizer` from `diffusers.pipelines.ace_step.modeling_ace_step` is deprecated. " + "Import it with `from diffusers import AceStepAudioTokenizer` instead.", + ) + return super().__new__(cls) + + +class AceStepAudioTokenDetokenizer(_AceStepAudioTokenDetokenizer): + def __new__(cls, *args, **kwargs): + deprecate( + "AceStepAudioTokenDetokenizer", + "1.0.0", + "Importing `AceStepAudioTokenDetokenizer` from `diffusers.pipelines.ace_step.modeling_ace_step` is deprecated. " + "Import it with `from diffusers import AceStepAudioTokenDetokenizer` instead.", + ) + return super().__new__(cls) diff --git a/src/diffusers/pipelines/ace_step/pipeline_ace_step.py b/src/diffusers/pipelines/ace_step/pipeline_ace_step.py index 2b27db5df98a..29f437e58443 100644 --- a/src/diffusers/pipelines/ace_step/pipeline_ace_step.py +++ b/src/diffusers/pipelines/ace_step/pipeline_ace_step.py @@ -22,13 +22,13 @@ from ...guiders.adaptive_projected_guidance import MomentumBuffer, normalized_guidance from ...loaders import AceStepLoraLoaderMixin from ...models import AutoencoderOobleck +from ...models.autoencoders import AceStepAudioTokenDetokenizer, AceStepAudioTokenizer from ...models.condition_embedders import AceStepConditionEncoder from ...models.transformers.ace_step_transformer import AceStepTransformer1DModel from ...schedulers import FlowMatchEulerDiscreteScheduler from ...utils import logging, replace_example_docstring from ...utils.torch_utils import randn_tensor from ..pipeline_utils import AudioPipelineOutput, DiffusionPipeline -from .modeling_ace_step import AceStepAudioTokenDetokenizer, AceStepAudioTokenizer logger = logging.get_logger(__name__) # pylint: disable=invalid-name diff --git a/src/diffusers/pipelines/audioldm2/modeling_audioldm2.py b/src/diffusers/pipelines/audioldm2/modeling_audioldm2.py index 6b918cadf800..a9538c6bd00b 100644 --- a/src/diffusers/pipelines/audioldm2/modeling_audioldm2.py +++ b/src/diffusers/pipelines/audioldm2/modeling_audioldm2.py @@ -12,1285 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. -from typing import Any - -import torch -import torch.nn as nn - -from ...configuration_utils import ConfigMixin, register_to_config -from ...loaders import UNet2DConditionLoadersMixin -from ...models.activations import get_activation -from ...models.attention import AttentionMixin -from ...models.attention_processor import ( - ADDED_KV_ATTENTION_PROCESSORS, - CROSS_ATTENTION_PROCESSORS, - AttnAddedKVProcessor, - AttnProcessor, -) from ...models.condition_embedders.condition_embedder_audioldm2 import ( AudioLDM2ProjectionModel as _AudioLDM2ProjectionModel, ) -from ...models.embeddings import TimestepEmbedding, Timesteps -from ...models.modeling_utils import ModelMixin -from ...models.resnet import Downsample2D, ResnetBlock2D, Upsample2D -from ...models.transformers.transformer_2d import Transformer2DModel -from ...models.unets.unet_2d_blocks import DownBlock2D, UpBlock2D -from ...models.unets.unet_2d_condition import UNet2DConditionOutput -from ...utils import deprecate, logging - - -logger = logging.get_logger(__name__) # pylint: disable=invalid-name - - -class AudioLDM2UNet2DConditionModel(ModelMixin, AttentionMixin, ConfigMixin, UNet2DConditionLoadersMixin): - r""" - A conditional 2D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample - shaped output. Compared to the vanilla [`UNet2DConditionModel`], this variant optionally includes an additional - self-attention layer in each Transformer block, as well as multiple cross-attention layers. It also allows for up - to two cross-attention embeddings, `encoder_hidden_states` and `encoder_hidden_states_1`. - - This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented - for all models (such as downloading or saving). - - Parameters: - sample_size (`int` or `tuple[int, int]`, *optional*, defaults to `None`): - Height and width of input/output sample. - in_channels (`int`, *optional*, defaults to 4): Number of channels in the input sample. - out_channels (`int`, *optional*, defaults to 4): Number of channels in the output. - flip_sin_to_cos (`bool`, *optional*, defaults to `False`): - Whether to flip the sin to cos in the time embedding. - freq_shift (`int`, *optional*, defaults to 0): The frequency shift to apply to the time embedding. - down_block_types (`tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): - The tuple of downsample blocks to use. - mid_block_type (`str`, *optional*, defaults to `"UNetMidBlock2DCrossAttn"`): - Block type for middle of UNet, it can only be `UNetMidBlock2DCrossAttn` for AudioLDM2. - up_block_types (`tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): - The tuple of upsample blocks to use. - only_cross_attention (`bool` or `tuple[bool]`, *optional*, default to `False`): - Whether to include self-attention in the basic transformer blocks, see - [`~models.attention.BasicTransformerBlock`]. - block_out_channels (`tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): - The tuple of output channels for each block. - layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. - downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution. - mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block. - act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. - norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. - If `None`, normalization and activation layers is skipped in post-processing. - norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization. - cross_attention_dim (`int` or `tuple[int]`, *optional*, defaults to 1280): - The dimension of the cross attention features. - transformer_layers_per_block (`int` or `tuple[int]`, *optional*, defaults to 1): - The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for - [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.CrossAttnUpBlock2D`], - [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. - attention_head_dim (`int`, *optional*, defaults to 8): The dimension of the attention heads. - num_attention_heads (`int`, *optional*): - The number of attention heads. If not defined, defaults to `attention_head_dim` - resnet_time_scale_shift (`str`, *optional*, defaults to `"default"`): Time scale shift config - for ResNet blocks (see [`~models.resnet.ResnetBlock2D`]). Choose from `default` or `scale_shift`. - class_embed_type (`str`, *optional*, defaults to `None`): - The type of class embedding to use which is ultimately summed with the time embeddings. Choose from `None`, - `"timestep"`, `"identity"`, `"projection"`, or `"simple_projection"`. - num_class_embeds (`int`, *optional*, defaults to `None`): - Input dimension of the learnable embedding matrix to be projected to `time_embed_dim`, when performing - class conditioning with `class_embed_type` equal to `None`. - time_embedding_type (`str`, *optional*, defaults to `positional`): - The type of position embedding to use for timesteps. Choose from `positional` or `fourier`. - time_embedding_dim (`int`, *optional*, defaults to `None`): - An optional override for the dimension of the projected time embedding. - time_embedding_act_fn (`str`, *optional*, defaults to `None`): - Optional activation function to use only once on the time embeddings before they are passed to the rest of - the UNet. Choose from `silu`, `mish`, `gelu`, and `swish`. - timestep_post_act (`str`, *optional*, defaults to `None`): - The second activation function to use in timestep embedding. Choose from `silu`, `mish` and `gelu`. - time_cond_proj_dim (`int`, *optional*, defaults to `None`): - The dimension of `cond_proj` layer in the timestep embedding. - conv_in_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_in` layer. - conv_out_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_out` layer. - projection_class_embeddings_input_dim (`int`, *optional*): The dimension of the `class_labels` input when - `class_embed_type="projection"`. Required when `class_embed_type="projection"`. - class_embeddings_concat (`bool`, *optional*, defaults to `False`): Whether to concatenate the time - embeddings with the class embeddings. - """ - - _supports_gradient_checkpointing = True - - @register_to_config - def __init__( - self, - sample_size: int | None = None, - in_channels: int = 4, - out_channels: int = 4, - flip_sin_to_cos: bool = True, - freq_shift: int = 0, - down_block_types: tuple[str] = ( - "CrossAttnDownBlock2D", - "CrossAttnDownBlock2D", - "CrossAttnDownBlock2D", - "DownBlock2D", - ), - mid_block_type: str = "UNetMidBlock2DCrossAttn", - up_block_types: tuple[str] = ("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D"), - only_cross_attention: bool | tuple[bool] = False, - block_out_channels: tuple[int] = (320, 640, 1280, 1280), - layers_per_block: int | tuple[int] = 2, - downsample_padding: int = 1, - mid_block_scale_factor: float = 1, - act_fn: str = "silu", - norm_num_groups: int | None = 32, - norm_eps: float = 1e-5, - cross_attention_dim: int | tuple[int] = 1280, - transformer_layers_per_block: int | tuple[int] = 1, - attention_head_dim: int | tuple[int] = 8, - num_attention_heads: int | tuple[int] | None = None, - use_linear_projection: bool = False, - class_embed_type: str | None = None, - num_class_embeds: int | None = None, - upcast_attention: bool = False, - resnet_time_scale_shift: str = "default", - time_embedding_type: str = "positional", - time_embedding_dim: int | None = None, - time_embedding_act_fn: str | None = None, - timestep_post_act: str | None = None, - time_cond_proj_dim: int | None = None, - conv_in_kernel: int = 3, - conv_out_kernel: int = 3, - projection_class_embeddings_input_dim: int | None = None, - class_embeddings_concat: bool = False, - ): - super().__init__() - - self.sample_size = sample_size - - if num_attention_heads is not None: - raise ValueError( - "At the moment it is not possible to define the number of attention heads via `num_attention_heads` because of a naming issue as described in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131. Passing `num_attention_heads` will only be supported in diffusers v0.19." - ) - - # If `num_attention_heads` is not defined (which is the case for most models) - # it will default to `attention_head_dim`. This looks weird upon first reading it and it is. - # The reason for this behavior is to correct for incorrectly named variables that were introduced - # when this library was created. The incorrect naming was only discovered much later in https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131 - # Changing `attention_head_dim` to `num_attention_heads` for 40,000+ configurations is too backwards breaking - # which is why we correct for the naming here. - num_attention_heads = num_attention_heads or attention_head_dim - - # Check inputs - if len(down_block_types) != len(up_block_types): - raise ValueError( - f"Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: {down_block_types}. `up_block_types`: {up_block_types}." - ) - - if len(block_out_channels) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: {block_out_channels}. `down_block_types`: {down_block_types}." - ) - - if not isinstance(only_cross_attention, bool) and len(only_cross_attention) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`: {only_cross_attention}. `down_block_types`: {down_block_types}." - ) - - if not isinstance(num_attention_heads, int) and len(num_attention_heads) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: {num_attention_heads}. `down_block_types`: {down_block_types}." - ) - - if not isinstance(attention_head_dim, int) and len(attention_head_dim) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `attention_head_dim` as `down_block_types`. `attention_head_dim`: {attention_head_dim}. `down_block_types`: {down_block_types}." - ) - - if isinstance(cross_attention_dim, list) and len(cross_attention_dim) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: {cross_attention_dim}. `down_block_types`: {down_block_types}." - ) - - if not isinstance(layers_per_block, int) and len(layers_per_block) != len(down_block_types): - raise ValueError( - f"Must provide the same number of `layers_per_block` as `down_block_types`. `layers_per_block`: {layers_per_block}. `down_block_types`: {down_block_types}." - ) - - # input - conv_in_padding = (conv_in_kernel - 1) // 2 - self.conv_in = nn.Conv2d( - in_channels, block_out_channels[0], kernel_size=conv_in_kernel, padding=conv_in_padding - ) - - # time - if time_embedding_type == "positional": - time_embed_dim = time_embedding_dim or block_out_channels[0] * 4 - - self.time_proj = Timesteps(block_out_channels[0], flip_sin_to_cos, freq_shift) - timestep_input_dim = block_out_channels[0] - else: - raise ValueError(f"{time_embedding_type} does not exist. Please make sure to use `positional`.") - - self.time_embedding = TimestepEmbedding( - timestep_input_dim, - time_embed_dim, - act_fn=act_fn, - post_act_fn=timestep_post_act, - cond_proj_dim=time_cond_proj_dim, - ) - - # class embedding - if class_embed_type is None and num_class_embeds is not None: - self.class_embedding = nn.Embedding(num_class_embeds, time_embed_dim) - elif class_embed_type == "timestep": - self.class_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim, act_fn=act_fn) - elif class_embed_type == "identity": - self.class_embedding = nn.Identity(time_embed_dim, time_embed_dim) - elif class_embed_type == "projection": - if projection_class_embeddings_input_dim is None: - raise ValueError( - "`class_embed_type`: 'projection' requires `projection_class_embeddings_input_dim` be set" - ) - # The projection `class_embed_type` is the same as the timestep `class_embed_type` except - # 1. the `class_labels` inputs are not first converted to sinusoidal embeddings - # 2. it projects from an arbitrary input dimension. - # - # Note that `TimestepEmbedding` is quite general, being mainly linear layers and activations. - # When used for embedding actual timesteps, the timesteps are first converted to sinusoidal embeddings. - # As a result, `TimestepEmbedding` can be passed arbitrary vectors. - self.class_embedding = TimestepEmbedding(projection_class_embeddings_input_dim, time_embed_dim) - elif class_embed_type == "simple_projection": - if projection_class_embeddings_input_dim is None: - raise ValueError( - "`class_embed_type`: 'simple_projection' requires `projection_class_embeddings_input_dim` be set" - ) - self.class_embedding = nn.Linear(projection_class_embeddings_input_dim, time_embed_dim) - else: - self.class_embedding = None - - if time_embedding_act_fn is None: - self.time_embed_act = None - else: - self.time_embed_act = get_activation(time_embedding_act_fn) - - self.down_blocks = nn.ModuleList([]) - self.up_blocks = nn.ModuleList([]) - - if isinstance(only_cross_attention, bool): - only_cross_attention = [only_cross_attention] * len(down_block_types) - - if isinstance(num_attention_heads, int): - num_attention_heads = (num_attention_heads,) * len(down_block_types) - - if isinstance(cross_attention_dim, int): - cross_attention_dim = (cross_attention_dim,) * len(down_block_types) - - if isinstance(layers_per_block, int): - layers_per_block = [layers_per_block] * len(down_block_types) - - if isinstance(transformer_layers_per_block, int): - transformer_layers_per_block = [transformer_layers_per_block] * len(down_block_types) - - if class_embeddings_concat: - # The time embeddings are concatenated with the class embeddings. The dimension of the - # time embeddings passed to the down, middle, and up blocks is twice the dimension of the - # regular time embeddings - blocks_time_embed_dim = time_embed_dim * 2 - else: - blocks_time_embed_dim = time_embed_dim - - # down - output_channel = block_out_channels[0] - for i, down_block_type in enumerate(down_block_types): - input_channel = output_channel - output_channel = block_out_channels[i] - is_final_block = i == len(block_out_channels) - 1 - - down_block = get_down_block( - down_block_type, - num_layers=layers_per_block[i], - transformer_layers_per_block=transformer_layers_per_block[i], - in_channels=input_channel, - out_channels=output_channel, - temb_channels=blocks_time_embed_dim, - add_downsample=not is_final_block, - resnet_eps=norm_eps, - resnet_act_fn=act_fn, - resnet_groups=norm_num_groups, - cross_attention_dim=cross_attention_dim[i], - num_attention_heads=num_attention_heads[i], - downsample_padding=downsample_padding, - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention[i], - upcast_attention=upcast_attention, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - self.down_blocks.append(down_block) - - # mid - if mid_block_type == "UNetMidBlock2DCrossAttn": - self.mid_block = UNetMidBlock2DCrossAttn( - transformer_layers_per_block=transformer_layers_per_block[-1], - in_channels=block_out_channels[-1], - temb_channels=blocks_time_embed_dim, - resnet_eps=norm_eps, - resnet_act_fn=act_fn, - output_scale_factor=mid_block_scale_factor, - resnet_time_scale_shift=resnet_time_scale_shift, - cross_attention_dim=cross_attention_dim[-1], - num_attention_heads=num_attention_heads[-1], - resnet_groups=norm_num_groups, - use_linear_projection=use_linear_projection, - upcast_attention=upcast_attention, - ) - else: - raise ValueError( - f"unknown mid_block_type : {mid_block_type}. Should be `UNetMidBlock2DCrossAttn` for AudioLDM2." - ) - - # count how many layers upsample the images - self.num_upsamplers = 0 - - # up - reversed_block_out_channels = list(reversed(block_out_channels)) - reversed_num_attention_heads = list(reversed(num_attention_heads)) - reversed_layers_per_block = list(reversed(layers_per_block)) - reversed_cross_attention_dim = list(reversed(cross_attention_dim)) - reversed_transformer_layers_per_block = list(reversed(transformer_layers_per_block)) - only_cross_attention = list(reversed(only_cross_attention)) - - output_channel = reversed_block_out_channels[0] - for i, up_block_type in enumerate(up_block_types): - is_final_block = i == len(block_out_channels) - 1 - - prev_output_channel = output_channel - output_channel = reversed_block_out_channels[i] - input_channel = reversed_block_out_channels[min(i + 1, len(block_out_channels) - 1)] - - # add upsample block for all BUT final layer - if not is_final_block: - add_upsample = True - self.num_upsamplers += 1 - else: - add_upsample = False - - up_block = get_up_block( - up_block_type, - num_layers=reversed_layers_per_block[i] + 1, - transformer_layers_per_block=reversed_transformer_layers_per_block[i], - in_channels=input_channel, - out_channels=output_channel, - prev_output_channel=prev_output_channel, - temb_channels=blocks_time_embed_dim, - add_upsample=add_upsample, - resnet_eps=norm_eps, - resnet_act_fn=act_fn, - resnet_groups=norm_num_groups, - cross_attention_dim=reversed_cross_attention_dim[i], - num_attention_heads=reversed_num_attention_heads[i], - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention[i], - upcast_attention=upcast_attention, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - self.up_blocks.append(up_block) - prev_output_channel = output_channel - - # out - if norm_num_groups is not None: - self.conv_norm_out = nn.GroupNorm( - num_channels=block_out_channels[0], num_groups=norm_num_groups, eps=norm_eps - ) - - self.conv_act = get_activation(act_fn) - - else: - self.conv_norm_out = None - self.conv_act = None - - conv_out_padding = (conv_out_kernel - 1) // 2 - self.conv_out = nn.Conv2d( - block_out_channels[0], out_channels, kernel_size=conv_out_kernel, padding=conv_out_padding - ) - - # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_default_attn_processor - def set_default_attn_processor(self): - """ - Disables custom attention processors and sets the default attention implementation. - """ - if all(proc.__class__ in ADDED_KV_ATTENTION_PROCESSORS for proc in self.attn_processors.values()): - processor = AttnAddedKVProcessor() - elif all(proc.__class__ in CROSS_ATTENTION_PROCESSORS for proc in self.attn_processors.values()): - processor = AttnProcessor() - else: - raise ValueError( - f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}" - ) - - self.set_attn_processor(processor) - - # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_attention_slice - def set_attention_slice(self, slice_size): - r""" - Enable sliced attention computation. - - When this option is enabled, the attention module splits the input tensor in slices to compute attention in - several steps. This is useful for saving some memory in exchange for a small decrease in speed. - - Args: - slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): - When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If - `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is - provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` - must be a multiple of `slice_size`. - """ - sliceable_head_dims = [] - - def fn_recursive_retrieve_sliceable_dims(module: torch.nn.Module): - if hasattr(module, "set_attention_slice"): - sliceable_head_dims.append(module.sliceable_head_dim) - - for child in module.children(): - fn_recursive_retrieve_sliceable_dims(child) - - # retrieve number of attention layers - for module in self.children(): - fn_recursive_retrieve_sliceable_dims(module) - - num_sliceable_layers = len(sliceable_head_dims) - - if slice_size == "auto": - # half the attention head size is usually a good trade-off between - # speed and memory - slice_size = [dim // 2 for dim in sliceable_head_dims] - elif slice_size == "max": - # make smallest slice possible - slice_size = num_sliceable_layers * [1] - - slice_size = num_sliceable_layers * [slice_size] if not isinstance(slice_size, list) else slice_size - - if len(slice_size) != len(sliceable_head_dims): - raise ValueError( - f"You have provided {len(slice_size)}, but {self.config} has {len(sliceable_head_dims)} different" - f" attention layers. Make sure to match `len(slice_size)` to be {len(sliceable_head_dims)}." - ) - - for i in range(len(slice_size)): - size = slice_size[i] - dim = sliceable_head_dims[i] - if size is not None and size > dim: - raise ValueError(f"size {size} has to be smaller or equal to {dim}.") - - # Recursively walk through all the children. - # Any children which exposes the set_attention_slice method - # gets the message - def fn_recursive_set_attention_slice(module: torch.nn.Module, slice_size: list[int]): - if hasattr(module, "set_attention_slice"): - module.set_attention_slice(slice_size.pop()) - - for child in module.children(): - fn_recursive_set_attention_slice(child, slice_size) - - reversed_slice_size = list(reversed(slice_size)) - for module in self.children(): - fn_recursive_set_attention_slice(module, reversed_slice_size) - - def forward( - self, - sample: torch.Tensor, - timestep: torch.Tensor | float | int, - encoder_hidden_states: torch.Tensor, - class_labels: torch.Tensor | None = None, - timestep_cond: torch.Tensor | None = None, - attention_mask: torch.Tensor | None = None, - cross_attention_kwargs: dict[str, Any] | None = None, - encoder_attention_mask: torch.Tensor | None = None, - return_dict: bool = True, - encoder_hidden_states_1: torch.Tensor | None = None, - encoder_attention_mask_1: torch.Tensor | None = None, - ) -> UNet2DConditionOutput | tuple: - r""" - The [`AudioLDM2UNet2DConditionModel`] forward method. - - Args: - sample (`torch.Tensor`): - The noisy input tensor with the following shape `(batch, channel, height, width)`. - timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. - encoder_hidden_states (`torch.Tensor`): - The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. - encoder_attention_mask (`torch.Tensor`): - A cross-attention mask of shape `(batch, sequence_length)` is applied to `encoder_hidden_states`. If - `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, - which adds large negative values to the attention scores corresponding to "discard" tokens. - return_dict (`bool`, *optional*, defaults to `True`): - Whether or not to return a [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] instead of a plain - tuple. - cross_attention_kwargs (`dict`, *optional*): - A kwargs dictionary that if specified is passed along to the [`AttnProcessor`]. - encoder_hidden_states_1 (`torch.Tensor`, *optional*): - A second set of encoder hidden states with shape `(batch, sequence_length_2, feature_dim_2)`. Can be - used to condition the model on a different set of embeddings to `encoder_hidden_states`. - encoder_attention_mask_1 (`torch.Tensor`, *optional*): - A cross-attention mask of shape `(batch, sequence_length_2)` is applied to `encoder_hidden_states_1`. - If `True` the mask is kept, otherwise if `False` it is discarded. Mask will be converted into a bias, - which adds large negative values to the attention scores corresponding to "discard" tokens. - - Returns: - [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] or `tuple`: - If `return_dict` is True, an [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] is returned, - otherwise a `tuple` is returned where the first element is the sample tensor. - """ - # By default samples have to be AT least a multiple of the overall upsampling factor. - # The overall upsampling factor is equal to 2 ** (# num of upsampling layers). - # However, the upsampling interpolation output size can be forced to fit any upsampling size - # on the fly if necessary. - default_overall_up_factor = 2**self.num_upsamplers - - # upsample size should be forwarded when sample is not a multiple of `default_overall_up_factor` - forward_upsample_size = False - upsample_size = None - - if any(s % default_overall_up_factor != 0 for s in sample.shape[-2:]): - logger.info("Forward upsample size to force interpolation output size.") - forward_upsample_size = True - - # ensure attention_mask is a bias, and give it a singleton query_tokens dimension - # expects mask of shape: - # [batch, key_tokens] - # adds singleton query_tokens dimension: - # [batch, 1, key_tokens] - # this helps to broadcast it as a bias over attention scores, which will be in one of the following shapes: - # [batch, heads, query_tokens, key_tokens] (e.g. torch sdp attn) - # [batch * heads, query_tokens, key_tokens] (e.g. xformers or classic attn) - if attention_mask is not None: - # assume that mask is expressed as: - # (1 = keep, 0 = discard) - # convert mask into a bias that can be added to attention scores: - # (keep = +0, discard = -10000.0) - attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0 - attention_mask = attention_mask.unsqueeze(1) - - # convert encoder_attention_mask to a bias the same way we do for attention_mask - if encoder_attention_mask is not None: - encoder_attention_mask = (1 - encoder_attention_mask.to(sample.dtype)) * -10000.0 - encoder_attention_mask = encoder_attention_mask.unsqueeze(1) - - if encoder_attention_mask_1 is not None: - encoder_attention_mask_1 = (1 - encoder_attention_mask_1.to(sample.dtype)) * -10000.0 - encoder_attention_mask_1 = encoder_attention_mask_1.unsqueeze(1) - - # 1. time - timesteps = timestep - if not torch.is_tensor(timesteps): - # TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can - # This would be a good case for the `match` statement (Python 3.10+) - is_mps = sample.device.type == "mps" - is_npu = sample.device.type == "npu" - if isinstance(timestep, float): - dtype = torch.float32 if (is_mps or is_npu) else torch.float64 - else: - dtype = torch.int32 if (is_mps or is_npu) else torch.int64 - timesteps = torch.tensor([timesteps], dtype=dtype, device=sample.device) - elif len(timesteps.shape) == 0: - timesteps = timesteps[None].to(sample.device) - - # broadcast to batch dimension in a way that's compatible with ONNX/Core ML - timesteps = timesteps.expand(sample.shape[0]) - - t_emb = self.time_proj(timesteps) - - # `Timesteps` does not contain any weights and will always return f32 tensors - # but time_embedding might actually be running in fp16. so we need to cast here. - # there might be better ways to encapsulate this. - t_emb = t_emb.to(dtype=sample.dtype) - - emb = self.time_embedding(t_emb, timestep_cond) - aug_emb = None - - if self.class_embedding is not None: - if class_labels is None: - raise ValueError("class_labels should be provided when num_class_embeds > 0") - - if self.config.class_embed_type == "timestep": - class_labels = self.time_proj(class_labels) - - # `Timesteps` does not contain any weights and will always return f32 tensors - # there might be better ways to encapsulate this. - class_labels = class_labels.to(dtype=sample.dtype) - - class_emb = self.class_embedding(class_labels).to(dtype=sample.dtype) - - if self.config.class_embeddings_concat: - emb = torch.cat([emb, class_emb], dim=-1) - else: - emb = emb + class_emb - - emb = emb + aug_emb if aug_emb is not None else emb - - if self.time_embed_act is not None: - emb = self.time_embed_act(emb) - - # 2. pre-process - sample = self.conv_in(sample) - - # 3. down - down_block_res_samples = (sample,) - for downsample_block in self.down_blocks: - if hasattr(downsample_block, "has_cross_attention") and downsample_block.has_cross_attention: - sample, res_samples = downsample_block( - hidden_states=sample, - temb=emb, - encoder_hidden_states=encoder_hidden_states, - attention_mask=attention_mask, - cross_attention_kwargs=cross_attention_kwargs, - encoder_attention_mask=encoder_attention_mask, - encoder_hidden_states_1=encoder_hidden_states_1, - encoder_attention_mask_1=encoder_attention_mask_1, - ) - else: - sample, res_samples = downsample_block(hidden_states=sample, temb=emb) - - down_block_res_samples += res_samples - - # 4. mid - if self.mid_block is not None: - sample = self.mid_block( - sample, - emb, - encoder_hidden_states=encoder_hidden_states, - attention_mask=attention_mask, - cross_attention_kwargs=cross_attention_kwargs, - encoder_attention_mask=encoder_attention_mask, - encoder_hidden_states_1=encoder_hidden_states_1, - encoder_attention_mask_1=encoder_attention_mask_1, - ) - - # 5. up - for i, upsample_block in enumerate(self.up_blocks): - is_final_block = i == len(self.up_blocks) - 1 - - res_samples = down_block_res_samples[-len(upsample_block.resnets) :] - down_block_res_samples = down_block_res_samples[: -len(upsample_block.resnets)] - - # if we have not reached the final block and need to forward the - # upsample size, we do it here - if not is_final_block and forward_upsample_size: - upsample_size = down_block_res_samples[-1].shape[2:] - - if hasattr(upsample_block, "has_cross_attention") and upsample_block.has_cross_attention: - sample = upsample_block( - hidden_states=sample, - temb=emb, - res_hidden_states_tuple=res_samples, - encoder_hidden_states=encoder_hidden_states, - cross_attention_kwargs=cross_attention_kwargs, - upsample_size=upsample_size, - attention_mask=attention_mask, - encoder_attention_mask=encoder_attention_mask, - encoder_hidden_states_1=encoder_hidden_states_1, - encoder_attention_mask_1=encoder_attention_mask_1, - ) - else: - sample = upsample_block( - hidden_states=sample, temb=emb, res_hidden_states_tuple=res_samples, upsample_size=upsample_size - ) - - # 6. post-process - if self.conv_norm_out: - sample = self.conv_norm_out(sample) - sample = self.conv_act(sample) - sample = self.conv_out(sample) - - if not return_dict: - return (sample,) - - return UNet2DConditionOutput(sample=sample) - - -def get_down_block( - down_block_type, - num_layers, - in_channels, - out_channels, - temb_channels, - add_downsample, - resnet_eps, - resnet_act_fn, - transformer_layers_per_block=1, - num_attention_heads=None, - resnet_groups=None, - cross_attention_dim=None, - downsample_padding=None, - use_linear_projection=False, - only_cross_attention=False, - upcast_attention=False, - resnet_time_scale_shift="default", -): - down_block_type = down_block_type[7:] if down_block_type.startswith("UNetRes") else down_block_type - if down_block_type == "DownBlock2D": - return DownBlock2D( - num_layers=num_layers, - in_channels=in_channels, - out_channels=out_channels, - temb_channels=temb_channels, - add_downsample=add_downsample, - resnet_eps=resnet_eps, - resnet_act_fn=resnet_act_fn, - resnet_groups=resnet_groups, - downsample_padding=downsample_padding, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - elif down_block_type == "CrossAttnDownBlock2D": - if cross_attention_dim is None: - raise ValueError("cross_attention_dim must be specified for CrossAttnDownBlock2D") - return CrossAttnDownBlock2D( - num_layers=num_layers, - transformer_layers_per_block=transformer_layers_per_block, - in_channels=in_channels, - out_channels=out_channels, - temb_channels=temb_channels, - add_downsample=add_downsample, - resnet_eps=resnet_eps, - resnet_act_fn=resnet_act_fn, - resnet_groups=resnet_groups, - downsample_padding=downsample_padding, - cross_attention_dim=cross_attention_dim, - num_attention_heads=num_attention_heads, - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention, - upcast_attention=upcast_attention, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - raise ValueError(f"{down_block_type} does not exist.") - - -def get_up_block( - up_block_type, - num_layers, - in_channels, - out_channels, - prev_output_channel, - temb_channels, - add_upsample, - resnet_eps, - resnet_act_fn, - transformer_layers_per_block=1, - num_attention_heads=None, - resnet_groups=None, - cross_attention_dim=None, - use_linear_projection=False, - only_cross_attention=False, - upcast_attention=False, - resnet_time_scale_shift="default", -): - up_block_type = up_block_type[7:] if up_block_type.startswith("UNetRes") else up_block_type - if up_block_type == "UpBlock2D": - return UpBlock2D( - num_layers=num_layers, - in_channels=in_channels, - out_channels=out_channels, - prev_output_channel=prev_output_channel, - temb_channels=temb_channels, - add_upsample=add_upsample, - resnet_eps=resnet_eps, - resnet_act_fn=resnet_act_fn, - resnet_groups=resnet_groups, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - elif up_block_type == "CrossAttnUpBlock2D": - if cross_attention_dim is None: - raise ValueError("cross_attention_dim must be specified for CrossAttnUpBlock2D") - return CrossAttnUpBlock2D( - num_layers=num_layers, - transformer_layers_per_block=transformer_layers_per_block, - in_channels=in_channels, - out_channels=out_channels, - prev_output_channel=prev_output_channel, - temb_channels=temb_channels, - add_upsample=add_upsample, - resnet_eps=resnet_eps, - resnet_act_fn=resnet_act_fn, - resnet_groups=resnet_groups, - cross_attention_dim=cross_attention_dim, - num_attention_heads=num_attention_heads, - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention, - upcast_attention=upcast_attention, - resnet_time_scale_shift=resnet_time_scale_shift, - ) - raise ValueError(f"{up_block_type} does not exist.") - - -class CrossAttnDownBlock2D(nn.Module): - def __init__( - self, - in_channels: int, - out_channels: int, - temb_channels: int, - dropout: float = 0.0, - num_layers: int = 1, - transformer_layers_per_block: int = 1, - resnet_eps: float = 1e-6, - resnet_time_scale_shift: str = "default", - resnet_act_fn: str = "swish", - resnet_groups: int = 32, - resnet_pre_norm: bool = True, - num_attention_heads=1, - cross_attention_dim=1280, - output_scale_factor=1.0, - downsample_padding=1, - add_downsample=True, - use_linear_projection=False, - only_cross_attention=False, - upcast_attention=False, - ): - super().__init__() - resnets = [] - attentions = [] - - self.has_cross_attention = True - self.num_attention_heads = num_attention_heads - - if isinstance(cross_attention_dim, int): - cross_attention_dim = (cross_attention_dim,) - if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: - raise ValueError( - "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " - f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" - ) - self.cross_attention_dim = cross_attention_dim - - for i in range(num_layers): - in_channels = in_channels if i == 0 else out_channels - resnets.append( - ResnetBlock2D( - in_channels=in_channels, - out_channels=out_channels, - temb_channels=temb_channels, - eps=resnet_eps, - groups=resnet_groups, - dropout=dropout, - time_embedding_norm=resnet_time_scale_shift, - non_linearity=resnet_act_fn, - output_scale_factor=output_scale_factor, - pre_norm=resnet_pre_norm, - ) - ) - for j in range(len(cross_attention_dim)): - attentions.append( - Transformer2DModel( - num_attention_heads, - out_channels // num_attention_heads, - in_channels=out_channels, - num_layers=transformer_layers_per_block, - cross_attention_dim=cross_attention_dim[j], - norm_num_groups=resnet_groups, - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention, - upcast_attention=upcast_attention, - double_self_attention=True if cross_attention_dim[j] is None else False, - ) - ) - self.attentions = nn.ModuleList(attentions) - self.resnets = nn.ModuleList(resnets) - - if add_downsample: - self.downsamplers = nn.ModuleList( - [ - Downsample2D( - out_channels, use_conv=True, out_channels=out_channels, padding=downsample_padding, name="op" - ) - ] - ) - else: - self.downsamplers = None - - self.gradient_checkpointing = False - - def forward( - self, - hidden_states: torch.Tensor, - temb: torch.Tensor | None = None, - encoder_hidden_states: torch.Tensor | None = None, - attention_mask: torch.Tensor | None = None, - cross_attention_kwargs: dict[str, Any] | None = None, - encoder_attention_mask: torch.Tensor | None = None, - encoder_hidden_states_1: torch.Tensor | None = None, - encoder_attention_mask_1: torch.Tensor | None = None, - ): - output_states = () - num_layers = len(self.resnets) - num_attention_per_layer = len(self.attentions) // num_layers - - encoder_hidden_states_1 = ( - encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states - ) - encoder_attention_mask_1 = ( - encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask - ) - - for i in range(num_layers): - if torch.is_grad_enabled() and self.gradient_checkpointing: - hidden_states = self._gradient_checkpointing_func(self.resnets[i], hidden_states, temb) - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self._gradient_checkpointing_func( - self.attentions[i * num_attention_per_layer + idx], - hidden_states, - forward_encoder_hidden_states, - None, # timestep - None, # class_labels - cross_attention_kwargs, - attention_mask, - forward_encoder_attention_mask, - )[0] - else: - hidden_states = self.resnets[i](hidden_states, temb) - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self.attentions[i * num_attention_per_layer + idx]( - hidden_states, - attention_mask=attention_mask, - encoder_hidden_states=forward_encoder_hidden_states, - encoder_attention_mask=forward_encoder_attention_mask, - return_dict=False, - )[0] - - output_states = output_states + (hidden_states,) - - if self.downsamplers is not None: - for downsampler in self.downsamplers: - hidden_states = downsampler(hidden_states) - - output_states = output_states + (hidden_states,) - - return hidden_states, output_states - - -class UNetMidBlock2DCrossAttn(nn.Module): - def __init__( - self, - in_channels: int, - temb_channels: int, - dropout: float = 0.0, - num_layers: int = 1, - transformer_layers_per_block: int = 1, - resnet_eps: float = 1e-6, - resnet_time_scale_shift: str = "default", - resnet_act_fn: str = "swish", - resnet_groups: int = 32, - resnet_pre_norm: bool = True, - num_attention_heads=1, - output_scale_factor=1.0, - cross_attention_dim=1280, - use_linear_projection=False, - upcast_attention=False, - ): - super().__init__() - - self.has_cross_attention = True - self.num_attention_heads = num_attention_heads - resnet_groups = resnet_groups if resnet_groups is not None else min(in_channels // 4, 32) - - if isinstance(cross_attention_dim, int): - cross_attention_dim = (cross_attention_dim,) - if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: - raise ValueError( - "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " - f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" - ) - self.cross_attention_dim = cross_attention_dim - - # there is always at least one resnet - resnets = [ - ResnetBlock2D( - in_channels=in_channels, - out_channels=in_channels, - temb_channels=temb_channels, - eps=resnet_eps, - groups=resnet_groups, - dropout=dropout, - time_embedding_norm=resnet_time_scale_shift, - non_linearity=resnet_act_fn, - output_scale_factor=output_scale_factor, - pre_norm=resnet_pre_norm, - ) - ] - attentions = [] - - for i in range(num_layers): - for j in range(len(cross_attention_dim)): - attentions.append( - Transformer2DModel( - num_attention_heads, - in_channels // num_attention_heads, - in_channels=in_channels, - num_layers=transformer_layers_per_block, - cross_attention_dim=cross_attention_dim[j], - norm_num_groups=resnet_groups, - use_linear_projection=use_linear_projection, - upcast_attention=upcast_attention, - double_self_attention=True if cross_attention_dim[j] is None else False, - ) - ) - resnets.append( - ResnetBlock2D( - in_channels=in_channels, - out_channels=in_channels, - temb_channels=temb_channels, - eps=resnet_eps, - groups=resnet_groups, - dropout=dropout, - time_embedding_norm=resnet_time_scale_shift, - non_linearity=resnet_act_fn, - output_scale_factor=output_scale_factor, - pre_norm=resnet_pre_norm, - ) - ) - - self.attentions = nn.ModuleList(attentions) - self.resnets = nn.ModuleList(resnets) - - self.gradient_checkpointing = False - - def forward( - self, - hidden_states: torch.Tensor, - temb: torch.Tensor | None = None, - encoder_hidden_states: torch.Tensor | None = None, - attention_mask: torch.Tensor | None = None, - cross_attention_kwargs: dict[str, Any] | None = None, - encoder_attention_mask: torch.Tensor | None = None, - encoder_hidden_states_1: torch.Tensor | None = None, - encoder_attention_mask_1: torch.Tensor | None = None, - ) -> torch.Tensor: - hidden_states = self.resnets[0](hidden_states, temb) - num_attention_per_layer = len(self.attentions) // (len(self.resnets) - 1) - - encoder_hidden_states_1 = ( - encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states - ) - encoder_attention_mask_1 = ( - encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask - ) - - for i in range(len(self.resnets[1:])): - if torch.is_grad_enabled() and self.gradient_checkpointing: - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self._gradient_checkpointing_func( - self.attentions[i * num_attention_per_layer + idx], - hidden_states, - forward_encoder_hidden_states, - None, # timestep - None, # class_labels - cross_attention_kwargs, - attention_mask, - forward_encoder_attention_mask, - )[0] - hidden_states = self._gradient_checkpointing_func(self.resnets[i + 1], hidden_states, temb) - else: - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self.attentions[i * num_attention_per_layer + idx]( - hidden_states, - attention_mask=attention_mask, - encoder_hidden_states=forward_encoder_hidden_states, - encoder_attention_mask=forward_encoder_attention_mask, - return_dict=False, - )[0] - - hidden_states = self.resnets[i + 1](hidden_states, temb) - - return hidden_states - - -class CrossAttnUpBlock2D(nn.Module): - def __init__( - self, - in_channels: int, - out_channels: int, - prev_output_channel: int, - temb_channels: int, - dropout: float = 0.0, - num_layers: int = 1, - transformer_layers_per_block: int = 1, - resnet_eps: float = 1e-6, - resnet_time_scale_shift: str = "default", - resnet_act_fn: str = "swish", - resnet_groups: int = 32, - resnet_pre_norm: bool = True, - num_attention_heads=1, - cross_attention_dim=1280, - output_scale_factor=1.0, - add_upsample=True, - use_linear_projection=False, - only_cross_attention=False, - upcast_attention=False, - ): - super().__init__() - resnets = [] - attentions = [] - - self.has_cross_attention = True - self.num_attention_heads = num_attention_heads - - if isinstance(cross_attention_dim, int): - cross_attention_dim = (cross_attention_dim,) - if isinstance(cross_attention_dim, (list, tuple)) and len(cross_attention_dim) > 4: - raise ValueError( - "Only up to 4 cross-attention layers are supported. Ensure that the length of cross-attention " - f"dims is less than or equal to 4. Got cross-attention dims {cross_attention_dim} of length {len(cross_attention_dim)}" - ) - self.cross_attention_dim = cross_attention_dim - - for i in range(num_layers): - res_skip_channels = in_channels if (i == num_layers - 1) else out_channels - resnet_in_channels = prev_output_channel if i == 0 else out_channels - - resnets.append( - ResnetBlock2D( - in_channels=resnet_in_channels + res_skip_channels, - out_channels=out_channels, - temb_channels=temb_channels, - eps=resnet_eps, - groups=resnet_groups, - dropout=dropout, - time_embedding_norm=resnet_time_scale_shift, - non_linearity=resnet_act_fn, - output_scale_factor=output_scale_factor, - pre_norm=resnet_pre_norm, - ) - ) - for j in range(len(cross_attention_dim)): - attentions.append( - Transformer2DModel( - num_attention_heads, - out_channels // num_attention_heads, - in_channels=out_channels, - num_layers=transformer_layers_per_block, - cross_attention_dim=cross_attention_dim[j], - norm_num_groups=resnet_groups, - use_linear_projection=use_linear_projection, - only_cross_attention=only_cross_attention, - upcast_attention=upcast_attention, - double_self_attention=True if cross_attention_dim[j] is None else False, - ) - ) - self.attentions = nn.ModuleList(attentions) - self.resnets = nn.ModuleList(resnets) - - if add_upsample: - self.upsamplers = nn.ModuleList([Upsample2D(out_channels, use_conv=True, out_channels=out_channels)]) - else: - self.upsamplers = None - - self.gradient_checkpointing = False - - def forward( - self, - hidden_states: torch.Tensor, - res_hidden_states_tuple: tuple[torch.Tensor, ...], - temb: torch.Tensor | None = None, - encoder_hidden_states: torch.Tensor | None = None, - cross_attention_kwargs: dict[str, Any] | None = None, - upsample_size: int | None = None, - attention_mask: torch.Tensor | None = None, - encoder_attention_mask: torch.Tensor | None = None, - encoder_hidden_states_1: torch.Tensor | None = None, - encoder_attention_mask_1: torch.Tensor | None = None, - ): - num_layers = len(self.resnets) - num_attention_per_layer = len(self.attentions) // num_layers - - encoder_hidden_states_1 = ( - encoder_hidden_states_1 if encoder_hidden_states_1 is not None else encoder_hidden_states - ) - encoder_attention_mask_1 = ( - encoder_attention_mask_1 if encoder_hidden_states_1 is not None else encoder_attention_mask - ) - - for i in range(num_layers): - # pop res hidden states - res_hidden_states = res_hidden_states_tuple[-1] - res_hidden_states_tuple = res_hidden_states_tuple[:-1] - hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1) - - if torch.is_grad_enabled() and self.gradient_checkpointing: - hidden_states = self._gradient_checkpointing_func(self.resnets[i], hidden_states, temb) - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self._gradient_checkpointing_func( - self.attentions[i * num_attention_per_layer + idx], - hidden_states, - forward_encoder_hidden_states, - None, # timestep - None, # class_labels - cross_attention_kwargs, - attention_mask, - forward_encoder_attention_mask, - )[0] - else: - hidden_states = self.resnets[i](hidden_states, temb) - for idx, cross_attention_dim in enumerate(self.cross_attention_dim): - if cross_attention_dim is not None and idx <= 1: - forward_encoder_hidden_states = encoder_hidden_states - forward_encoder_attention_mask = encoder_attention_mask - elif cross_attention_dim is not None and idx > 1: - forward_encoder_hidden_states = encoder_hidden_states_1 - forward_encoder_attention_mask = encoder_attention_mask_1 - else: - forward_encoder_hidden_states = None - forward_encoder_attention_mask = None - hidden_states = self.attentions[i * num_attention_per_layer + idx]( - hidden_states, - attention_mask=attention_mask, - encoder_hidden_states=forward_encoder_hidden_states, - encoder_attention_mask=forward_encoder_attention_mask, - return_dict=False, - )[0] - - if self.upsamplers is not None: - for upsampler in self.upsamplers: - hidden_states = upsampler(hidden_states, upsample_size) - - return hidden_states +from ...models.unets.unet_2d_condition_audioldm2 import AudioLDM2UNet2DConditionModel as _AudioLDM2UNet2DConditionModel +from ...utils import deprecate # Deprecated alias kept for backward compatibility. The warning is raised from `__new__` rather than `__init__` so @@ -1304,3 +30,14 @@ def __new__(cls, *args, **kwargs): "Import it with `from diffusers import AudioLDM2ProjectionModel` instead.", ) return super().__new__(cls) + + +class AudioLDM2UNet2DConditionModel(_AudioLDM2UNet2DConditionModel): + def __new__(cls, *args, **kwargs): + deprecate( + "AudioLDM2UNet2DConditionModel", + "1.0.0", + "Importing `AudioLDM2UNet2DConditionModel` from `diffusers.pipelines.audioldm2.modeling_audioldm2` is deprecated. " + "Import it with `from diffusers import AudioLDM2UNet2DConditionModel` instead.", + ) + return super().__new__(cls) diff --git a/src/diffusers/pipelines/audioldm2/pipeline_audioldm2.py b/src/diffusers/pipelines/audioldm2/pipeline_audioldm2.py index 9a5d69c0df09..049acd133cd0 100644 --- a/src/diffusers/pipelines/audioldm2/pipeline_audioldm2.py +++ b/src/diffusers/pipelines/audioldm2/pipeline_audioldm2.py @@ -34,6 +34,7 @@ from ...dependency_versions_table import deps from ...models import AutoencoderKL from ...models.condition_embedders import AudioLDM2ProjectionModel +from ...models.unets import AudioLDM2UNet2DConditionModel from ...schedulers import KarrasDiffusionSchedulers from ...utils import ( is_accelerate_available, @@ -44,7 +45,6 @@ from ...utils.import_utils import is_transformers_version from ...utils.torch_utils import empty_device_cache, randn_tensor from ..pipeline_utils import AudioPipelineOutput, DiffusionPipeline -from .modeling_audioldm2 import AudioLDM2UNet2DConditionModel if is_librosa_available(): diff --git a/src/diffusers/pipelines/deprecated/__init__.py b/src/diffusers/pipelines/deprecated/__init__.py index 3eec8e849592..e433928b62a4 100644 --- a/src/diffusers/pipelines/deprecated/__init__.py +++ b/src/diffusers/pipelines/deprecated/__init__.py @@ -61,6 +61,7 @@ _import_structure["musicldm"] = ["MusicLDMPipeline"] _import_structure["paint_by_example"] = ["PaintByExamplePipeline"] _import_structure["pia"] = ["PIAPipeline"] + _import_structure["shap_e"] = ["ShapEImg2ImgPipeline", "ShapEPipeline"] _import_structure["semantic_stable_diffusion"] = ["SemanticStableDiffusionPipeline"] _import_structure["stable_diffusion_attend_and_excite"] = ["StableDiffusionAttendAndExcitePipeline"] _import_structure["stable_diffusion_diffedit"] = ["StableDiffusionDiffEditPipeline"] @@ -154,6 +155,7 @@ from .paint_by_example import PaintByExamplePipeline from .pia import PIAPipeline from .semantic_stable_diffusion import SemanticStableDiffusionPipeline + from .shap_e import ShapEImg2ImgPipeline, ShapEPipeline from .spectrogram_diffusion import SpectrogramDiffusionPipeline from .stable_diffusion_attend_and_excite import StableDiffusionAttendAndExcitePipeline from .stable_diffusion_diffedit import StableDiffusionDiffEditPipeline diff --git a/src/diffusers/pipelines/shap_e/__init__.py b/src/diffusers/pipelines/deprecated/shap_e/__init__.py similarity index 91% rename from src/diffusers/pipelines/shap_e/__init__.py rename to src/diffusers/pipelines/deprecated/shap_e/__init__.py index 4ed563c4a51f..c80732bf4de2 100644 --- a/src/diffusers/pipelines/shap_e/__init__.py +++ b/src/diffusers/pipelines/deprecated/shap_e/__init__.py @@ -1,6 +1,6 @@ from typing import TYPE_CHECKING -from ...utils import ( +from ....utils import ( DIFFUSERS_SLOW_IMPORT, OptionalDependencyNotAvailable, _LazyModule, @@ -17,7 +17,7 @@ if not (is_transformers_available() and is_torch_available()): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: - from ...utils import dummy_torch_and_transformers_objects # noqa F403 + from ....utils import dummy_torch_and_transformers_objects # noqa F403 _dummy_objects.update(get_objects_from_module(dummy_torch_and_transformers_objects)) else: @@ -41,7 +41,7 @@ raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: - from ...utils.dummy_torch_and_transformers_objects import * + from ....utils.dummy_torch_and_transformers_objects import * else: from .camera import create_pan_cameras from .pipeline_shap_e import ShapEPipeline diff --git a/src/diffusers/pipelines/shap_e/camera.py b/src/diffusers/pipelines/deprecated/shap_e/camera.py similarity index 100% rename from src/diffusers/pipelines/shap_e/camera.py rename to src/diffusers/pipelines/deprecated/shap_e/camera.py diff --git a/src/diffusers/pipelines/shap_e/pipeline_shap_e.py b/src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e.py similarity index 93% rename from src/diffusers/pipelines/shap_e/pipeline_shap_e.py rename to src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e.py index 56e8f57dab35..5fdc85c2a724 100644 --- a/src/diffusers/pipelines/shap_e/pipeline_shap_e.py +++ b/src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e.py @@ -20,16 +20,16 @@ import torch from transformers import CLIPTextModelWithProjection, CLIPTokenizer -from ...models import PriorTransformer -from ...schedulers import HeunDiscreteScheduler -from ...utils import ( +from ....models import PriorTransformer +from ....schedulers import HeunDiscreteScheduler +from ....utils import ( BaseOutput, is_torch_xla_available, logging, replace_example_docstring, ) -from ...utils.torch_utils import randn_tensor -from ..pipeline_utils import DiffusionPipeline +from ....utils.torch_utils import randn_tensor +from ...pipeline_utils import DeprecatedPipelineMixin, DiffusionPipeline from .renderer import ShapERenderer @@ -40,6 +40,7 @@ else: XLA_AVAILABLE = False + logger = logging.get_logger(__name__) # pylint: disable=invalid-name @@ -84,7 +85,7 @@ class ShapEPipelineOutput(BaseOutput): images: list[list[PIL.Image.Image]] | list[list[np.ndarray]] -class ShapEPipeline(DiffusionPipeline): +class ShapEPipeline(DeprecatedPipelineMixin, DiffusionPipeline): """ Pipeline for generating latent representation of a 3D asset and rendering with the NeRF method. @@ -105,6 +106,8 @@ class ShapEPipeline(DiffusionPipeline): rendering method. """ + _last_supported_version = "0.43.0" + model_cpu_offload_seq = "text_encoder->prior" _exclude_from_cpu_offload = ["shap_e_renderer"] @@ -228,15 +231,15 @@ def __call__( The output format of the generated image. Choose between `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`), `"latent"` (`torch.Tensor`), or mesh ([`MeshDecoderOutput`]). return_dict (`bool`, *optional*, defaults to `True`): - Whether or not to return a [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] instead of a plain - tuple. + Whether or not to return a [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] instead + of a plain tuple. Examples: Returns: - [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] or `tuple`: - If `return_dict` is `True`, [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] is returned, - otherwise a `tuple` is returned where the first element is a list with the generated images. + [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] or `tuple`: + If `return_dict` is `True`, [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] is + returned, otherwise a `tuple` is returned where the first element is a list with the generated images. """ if isinstance(prompt, str): diff --git a/src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py b/src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e_img2img.py similarity index 93% rename from src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py rename to src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e_img2img.py index 403d5fa8a743..7d95d2ddfbee 100644 --- a/src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py +++ b/src/diffusers/pipelines/deprecated/shap_e/pipeline_shap_e_img2img.py @@ -19,16 +19,16 @@ import torch from transformers import CLIPImageProcessor, CLIPVisionModel -from ...models import PriorTransformer -from ...schedulers import HeunDiscreteScheduler -from ...utils import ( +from ....models import PriorTransformer +from ....schedulers import HeunDiscreteScheduler +from ....utils import ( BaseOutput, is_torch_xla_available, logging, replace_example_docstring, ) -from ...utils.torch_utils import randn_tensor -from ..pipeline_utils import DiffusionPipeline +from ....utils.torch_utils import randn_tensor +from ...pipeline_utils import DeprecatedPipelineMixin, DiffusionPipeline from .renderer import ShapERenderer @@ -39,6 +39,7 @@ else: XLA_AVAILABLE = False + logger = logging.get_logger(__name__) # pylint: disable=invalid-name @@ -85,7 +86,7 @@ class ShapEPipelineOutput(BaseOutput): images: PIL.Image.Image | np.ndarray -class ShapEImg2ImgPipeline(DiffusionPipeline): +class ShapEImg2ImgPipeline(DeprecatedPipelineMixin, DiffusionPipeline): """ Pipeline for generating latent representation of a 3D asset and rendering with the NeRF method from an image. @@ -106,6 +107,8 @@ class ShapEImg2ImgPipeline(DiffusionPipeline): rendering method. """ + _last_supported_version = "0.43.0" + model_cpu_offload_seq = "image_encoder->prior" _exclude_from_cpu_offload = ["shap_e_renderer"] @@ -211,15 +214,15 @@ def __call__( The output format of the generated image. Choose between `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`), `"latent"` (`torch.Tensor`), or mesh ([`MeshDecoderOutput`]). return_dict (`bool`, *optional*, defaults to `True`): - Whether or not to return a [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] instead of a plain - tuple. + Whether or not to return a [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] instead + of a plain tuple. Examples: Returns: - [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] or `tuple`: - If `return_dict` is `True`, [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] is returned, - otherwise a `tuple` is returned where the first element is a list with the generated images. + [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] or `tuple`: + If `return_dict` is `True`, [`~pipelines.deprecated.shap_e.pipeline_shap_e.ShapEPipelineOutput`] is + returned, otherwise a `tuple` is returned where the first element is a list with the generated images. """ if isinstance(image, PIL.Image.Image): diff --git a/src/diffusers/pipelines/shap_e/renderer.py b/src/diffusers/pipelines/deprecated/shap_e/renderer.py similarity index 99% rename from src/diffusers/pipelines/shap_e/renderer.py rename to src/diffusers/pipelines/deprecated/shap_e/renderer.py index 0c2058c887fc..73e913ed4380 100644 --- a/src/diffusers/pipelines/shap_e/renderer.py +++ b/src/diffusers/pipelines/deprecated/shap_e/renderer.py @@ -20,9 +20,9 @@ import torch.nn.functional as F from torch import nn -from ...configuration_utils import ConfigMixin, register_to_config -from ...models import ModelMixin -from ...utils import BaseOutput +from ....configuration_utils import ConfigMixin, register_to_config +from ....models import ModelMixin +from ....utils import BaseOutput from .camera import create_pan_cameras diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2.py index 0d5341d4f41e..e7462aa5dfdf 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2.py @@ -31,7 +31,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...image_processor import PipelineImageInput from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler @@ -55,7 +55,6 @@ from .utils import ( conditioning_frame_rate as get_conditioning_frame_rate, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_condition.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_condition.py index d876dec5b399..9f28ff5ee9b6 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_condition.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_condition.py @@ -33,7 +33,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...image_processor import PipelineImageInput from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler @@ -59,7 +59,6 @@ from .utils import ( conditioning_frame_rate as get_conditioning_frame_rate, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr.py index f9c6729f2e02..ea232a3a331e 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr.py @@ -31,7 +31,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...image_processor import PipelineImageInput from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler, LTXEulerAncestralRFScheduler @@ -61,7 +61,6 @@ resolve_default_image_crf, trim_canvas, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr_temporal_refine.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr_temporal_refine.py index df71dda6d644..7a46047760c9 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr_temporal_refine.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_dfr_temporal_refine.py @@ -28,7 +28,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.latent_upscaler import LTX2LatentUpsamplerModel from ...models.transformers import LTX2VideoTransformer3DModel @@ -49,7 +49,6 @@ apply_image_conditioning_crf, resolve_default_image_crf, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_hdr_lora.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_hdr_lora.py index 261521fdef18..018dd5dbeeec 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_hdr_lora.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_hdr_lora.py @@ -29,7 +29,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler @@ -38,7 +38,6 @@ from ..pipeline_utils import DiffusionPipeline from .image_processor import LTX2VideoHDRProcessor from .pipeline_output import LTX2PipelineOutput -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_ic_lora.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_ic_lora.py index 0e01c9610388..5f6253589525 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_ic_lora.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_ic_lora.py @@ -33,7 +33,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...image_processor import PipelineImageInput from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler @@ -59,7 +59,6 @@ from .utils import ( conditioning_frame_rate as get_conditioning_frame_rate, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/pipeline_ltx2_image2video.py b/src/diffusers/pipelines/ltx2/pipeline_ltx2_image2video.py index 0bc45b0c1895..82a5ddb31d5c 100644 --- a/src/diffusers/pipelines/ltx2/pipeline_ltx2_image2video.py +++ b/src/diffusers/pipelines/ltx2/pipeline_ltx2_image2video.py @@ -31,7 +31,7 @@ from ...callbacks import MultiPipelineCallbacks, PipelineCallback from ...image_processor import PipelineImageInput from ...loaders import FromSingleFileMixin, LTX2LoraLoaderMixin -from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video +from ...models.autoencoders import AutoencoderKLLTX2Audio, AutoencoderKLLTX2Video, LTX2Vocoder, LTX2VocoderWithBWE from ...models.condition_embedders import LTX2TextConnectors from ...models.transformers import LTX2VideoTransformer3DModel from ...schedulers import FlowMatchEulerDiscreteScheduler @@ -56,7 +56,6 @@ from .utils import ( conditioning_frame_rate as get_conditioning_frame_rate, ) -from .vocoder import LTX2Vocoder, LTX2VocoderWithBWE if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/ltx2/vocoder.py b/src/diffusers/pipelines/ltx2/vocoder.py index f0004f2ec02d..bb5f14357e4c 100644 --- a/src/diffusers/pipelines/ltx2/vocoder.py +++ b/src/diffusers/pipelines/ltx2/vocoder.py @@ -1,597 +1,41 @@ -import math - -import torch -import torch.nn as nn -import torch.nn.functional as F - -from ...configuration_utils import ConfigMixin, register_to_config -from ...models.modeling_utils import ModelMixin - - -def kaiser_sinc_filter1d(cutoff: float, half_width: float, kernel_size: int) -> torch.Tensor: - """ - Creates a Kaiser sinc kernel for low-pass filtering. - - Args: - cutoff (`float`): - Normalized frequency cutoff (relative to the sampling rate). Must be between 0 and 0.5 (the Nyquist - frequency). - half_width (`float`): - Used to determine the Kaiser window's beta parameter. - kernel_size: - Size of the Kaiser window (and ultimately the Kaiser sinc kernel). - - Returns: - `torch.Tensor` of shape `(kernel_size,)`: - The Kaiser sinc kernel. - """ - delta_f = 4 * half_width - half_size = kernel_size // 2 - amplitude = 2.285 * (half_size - 1) * math.pi * delta_f + 7.95 - if amplitude > 50.0: - beta = 0.1102 * (amplitude - 8.7) - elif amplitude >= 21.0: - beta = 0.5842 * (amplitude - 21) ** 0.4 + 0.07886 * (amplitude - 21.0) - else: - beta = 0.0 - - window = torch.kaiser_window(kernel_size, beta=beta, periodic=False) - - even = kernel_size % 2 == 0 - time = torch.arange(-half_size, half_size) + 0.5 if even else torch.arange(kernel_size) - half_size - - if cutoff == 0.0: - filter = torch.zeros_like(time) - else: - time = 2 * cutoff * time - sinc = torch.where( - time == 0, - torch.ones_like(time), - torch.sin(math.pi * time) / math.pi / time, - ) - filter = 2 * cutoff * window * sinc - filter = filter / filter.sum() - return filter - - -class DownSample1d(nn.Module): - """1D low-pass filter for antialias downsampling.""" - - def __init__( - self, - ratio: int = 2, - kernel_size: int | None = None, - use_padding: bool = True, - padding_mode: str = "replicate", - persistent: bool = True, - ): - super().__init__() - self.ratio = ratio - self.kernel_size = kernel_size or int(6 * ratio // 2) * 2 - self.pad_left = self.kernel_size // 2 + (self.kernel_size % 2) - 1 - self.pad_right = self.kernel_size // 2 - self.use_padding = use_padding - self.padding_mode = padding_mode - - cutoff = 0.5 / ratio - half_width = 0.6 / ratio - low_pass_filter = kaiser_sinc_filter1d(cutoff, half_width, self.kernel_size) - self.register_buffer("filter", low_pass_filter.view(1, 1, self.kernel_size), persistent=persistent) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - # x expected shape: [batch_size, num_channels, hidden_dim] - num_channels = x.shape[1] - if self.use_padding: - x = F.pad(x, (self.pad_left, self.pad_right), mode=self.padding_mode) - x_filtered = F.conv1d(x, self.filter.expand(num_channels, -1, -1), stride=self.ratio, groups=num_channels) - return x_filtered - - -class UpSample1d(nn.Module): - def __init__( - self, - ratio: int = 2, - kernel_size: int | None = None, - window_type: str = "kaiser", - padding_mode: str = "replicate", - persistent: bool = True, - ): - super().__init__() - self.ratio = ratio - self.padding_mode = padding_mode - - if window_type == "hann": - rolloff = 0.99 - lowpass_filter_width = 6 - width = math.ceil(lowpass_filter_width / rolloff) - self.kernel_size = 2 * width * ratio + 1 - self.pad = width - self.pad_left = 2 * width * ratio - self.pad_right = self.kernel_size - ratio - - time_axis = (torch.arange(self.kernel_size) / ratio - width) * rolloff - time_clamped = time_axis.clamp(-lowpass_filter_width, lowpass_filter_width) - window = torch.cos(time_clamped * math.pi / lowpass_filter_width / 2) ** 2 - sinc_filter = (torch.sinc(time_axis) * window * rolloff / ratio).view(1, 1, -1) - else: - # Kaiser sinc filter is BigVGAN default - self.kernel_size = int(6 * ratio // 2) * 2 if kernel_size is None else kernel_size - self.pad = self.kernel_size // ratio - 1 - self.pad_left = self.pad * self.ratio + (self.kernel_size - self.ratio) // 2 - self.pad_right = self.pad * self.ratio + (self.kernel_size - self.ratio + 1) // 2 - - sinc_filter = kaiser_sinc_filter1d( - cutoff=0.5 / ratio, - half_width=0.6 / ratio, - kernel_size=self.kernel_size, - ) - - self.register_buffer("filter", sinc_filter.view(1, 1, self.kernel_size), persistent=persistent) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - # x expected shape: [batch_size, num_channels, hidden_dim] - num_channels = x.shape[1] - x = F.pad(x, (self.pad, self.pad), mode=self.padding_mode) - low_pass_filter = self.filter.to(dtype=x.dtype, device=x.device).expand(num_channels, -1, -1) - x = self.ratio * F.conv_transpose1d(x, low_pass_filter, stride=self.ratio, groups=num_channels) - return x[..., self.pad_left : -self.pad_right] - - -class AntiAliasAct1d(nn.Module): - """ - Antialiasing activation for a 1D signal: upsamples, applies an activation (usually snakebeta), and then downsamples - to avoid aliasing. - """ - - def __init__( - self, - act_fn: str | nn.Module, - ratio: int = 2, - kernel_size: int = 12, - **kwargs, - ): - super().__init__() - self.upsample = UpSample1d(ratio=ratio, kernel_size=kernel_size) - if isinstance(act_fn, str): - if act_fn == "snakebeta": - act_fn = SnakeBeta(**kwargs) - elif act_fn == "snake": - act_fn = SnakeBeta(**kwargs) - else: - act_fn = nn.LeakyReLU(**kwargs) - self.act = act_fn - self.downsample = DownSample1d(ratio=ratio, kernel_size=kernel_size) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - x = self.upsample(x) - x = self.act(x) - x = self.downsample(x) - return x - - -class SnakeBeta(nn.Module): - """ - Implements the Snake and SnakeBeta activations, which help with learning periodic patterns. - """ - - def __init__( - self, - channels: int, - alpha: float = 1.0, - eps: float = 1e-9, - trainable_params: bool = True, - logscale: bool = True, - use_beta: bool = True, - ): - super().__init__() - self.eps = eps - self.logscale = logscale - self.use_beta = use_beta - - self.alpha = nn.Parameter(torch.zeros(channels) if self.logscale else torch.ones(channels) * alpha) - self.alpha.requires_grad = trainable_params - if use_beta: - self.beta = nn.Parameter(torch.zeros(channels) if self.logscale else torch.ones(channels) * alpha) - self.beta.requires_grad = trainable_params - - def forward(self, hidden_states: torch.Tensor, channel_dim: int = 1) -> torch.Tensor: - broadcast_shape = [1] * hidden_states.ndim - broadcast_shape[channel_dim] = -1 - alpha = self.alpha.view(broadcast_shape) - if self.use_beta: - beta = self.beta.view(broadcast_shape) - - if self.logscale: - alpha = torch.exp(alpha) - if self.use_beta: - beta = torch.exp(beta) - - amplitude = beta if self.use_beta else alpha - hidden_states = hidden_states + (1.0 / (amplitude + self.eps)) * torch.sin(hidden_states * alpha).pow(2) - return hidden_states - - -class ResBlock(nn.Module): - def __init__( - self, - channels: int, - kernel_size: int = 3, - stride: int = 1, - dilations: tuple[int, ...] = (1, 3, 5), - act_fn: str = "leaky_relu", - leaky_relu_negative_slope: float = 0.1, - antialias: bool = False, - antialias_ratio: int = 2, - antialias_kernel_size: int = 12, - padding_mode: str = "same", - ): - super().__init__() - self.dilations = dilations - - self.convs1 = nn.ModuleList( - [ - nn.Conv1d(channels, channels, kernel_size, stride=stride, dilation=dilation, padding=padding_mode) - for dilation in dilations - ] - ) - self.acts1 = nn.ModuleList() - for _ in range(len(self.convs1)): - if act_fn == "snakebeta": - act = SnakeBeta(channels, use_beta=True) - elif act_fn == "snake": - act = SnakeBeta(channels, use_beta=False) - else: - act = nn.LeakyReLU(negative_slope=leaky_relu_negative_slope) - - if antialias: - act = AntiAliasAct1d(act, ratio=antialias_ratio, kernel_size=antialias_kernel_size) - self.acts1.append(act) - - self.convs2 = nn.ModuleList( - [ - nn.Conv1d(channels, channels, kernel_size, stride=stride, dilation=1, padding=padding_mode) - for _ in range(len(dilations)) - ] - ) - self.acts2 = nn.ModuleList() - for _ in range(len(self.convs2)): - if act_fn == "snakebeta": - act = SnakeBeta(channels, use_beta=True) - elif act_fn == "snake": - act = SnakeBeta(channels, use_beta=False) - else: - act_fn = nn.LeakyReLU(negative_slope=leaky_relu_negative_slope) - - if antialias: - act = AntiAliasAct1d(act, ratio=antialias_ratio, kernel_size=antialias_kernel_size) - self.acts2.append(act) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - for act1, conv1, act2, conv2 in zip(self.acts1, self.convs1, self.acts2, self.convs2): - xt = act1(x) - xt = conv1(xt) - xt = act2(xt) - xt = conv2(xt) - x = x + xt - return x - - -class LTX2Vocoder(ModelMixin, ConfigMixin): - r""" - LTX 2.0 vocoder for converting generated mel spectrograms back to audio waveforms. - """ - - @register_to_config - def __init__( - self, - in_channels: int = 128, - hidden_channels: int = 1024, - out_channels: int = 2, - upsample_kernel_sizes: list[int] = [16, 15, 8, 4, 4], - upsample_factors: list[int] = [6, 5, 2, 2, 2], - resnet_kernel_sizes: list[int] = [3, 7, 11], - resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], - act_fn: str = "leaky_relu", - leaky_relu_negative_slope: float = 0.1, - antialias: bool = False, - antialias_ratio: int = 2, - antialias_kernel_size: int = 12, - final_act_fn: str | None = "tanh", # tanh, clamp, None - final_bias: bool = True, - output_sampling_rate: int = 24000, - ): - super().__init__() - self.num_upsample_layers = len(upsample_kernel_sizes) - self.resnets_per_upsample = len(resnet_kernel_sizes) - self.out_channels = out_channels - self.total_upsample_factor = math.prod(upsample_factors) - self.act_fn = act_fn - self.negative_slope = leaky_relu_negative_slope - self.final_act_fn = final_act_fn - - if self.num_upsample_layers != len(upsample_factors): - raise ValueError( - f"`upsample_kernel_sizes` and `upsample_factors` should be lists of the same length but are length" - f" {self.num_upsample_layers} and {len(upsample_factors)}, respectively." - ) - - if self.resnets_per_upsample != len(resnet_dilations): - raise ValueError( - f"`resnet_kernel_sizes` and `resnet_dilations` should be lists of the same length but are length" - f" {len(self.resnets_per_upsample)} and {len(resnet_dilations)}, respectively." - ) - - supported_act_fns = ["snakebeta", "snake", "leaky_relu"] - if self.act_fn not in supported_act_fns: - raise ValueError( - f"Unsupported activation function: {self.act_fn}. Currently supported values of `act_fn` are " - f"{supported_act_fns}." - ) - - self.conv_in = nn.Conv1d(in_channels, hidden_channels, kernel_size=7, stride=1, padding=3) - - self.upsamplers = nn.ModuleList() - self.resnets = nn.ModuleList() - input_channels = hidden_channels - for i, (stride, kernel_size) in enumerate(zip(upsample_factors, upsample_kernel_sizes)): - output_channels = input_channels // 2 - self.upsamplers.append( - nn.ConvTranspose1d( - input_channels, # hidden_channels // (2 ** i) - output_channels, # hidden_channels // (2 ** (i + 1)) - kernel_size, - stride=stride, - padding=(kernel_size - stride) // 2, - ) - ) - - for kernel_size, dilations in zip(resnet_kernel_sizes, resnet_dilations): - self.resnets.append( - ResBlock( - channels=output_channels, - kernel_size=kernel_size, - dilations=dilations, - act_fn=act_fn, - leaky_relu_negative_slope=leaky_relu_negative_slope, - antialias=antialias, - antialias_ratio=antialias_ratio, - antialias_kernel_size=antialias_kernel_size, - ) - ) - input_channels = output_channels - - if act_fn == "snakebeta" or act_fn == "snake": - # Always use antialiasing - act_out = SnakeBeta(channels=output_channels, use_beta=True) - self.act_out = AntiAliasAct1d(act_out, ratio=antialias_ratio, kernel_size=antialias_kernel_size) - elif act_fn == "leaky_relu": - # NOTE: does NOT use self.negative_slope, following the original code - self.act_out = nn.LeakyReLU() - - self.conv_out = nn.Conv1d(output_channels, out_channels, 7, stride=1, padding=3, bias=final_bias) - - def forward(self, hidden_states: torch.Tensor, time_last: bool = False) -> torch.Tensor: - r""" - Forward pass of the vocoder. - - Args: - hidden_states (`torch.Tensor`): - Input Mel spectrogram tensor of shape `(batch_size, num_channels, time, num_mel_bins)` if `time_last` - is `False` (the default) or shape `(batch_size, num_channels, num_mel_bins, time)` if `time_last` is - `True`. - time_last (`bool`, *optional*, defaults to `False`): - Whether the last dimension of the input is the time/frame dimension or the Mel bins dimension. - - Returns: - `torch.Tensor`: - Audio waveform tensor of shape (batch_size, out_channels, audio_length) - """ - - # Ensure that the time/frame dimension is last - if not time_last: - hidden_states = hidden_states.transpose(2, 3) - # Combine channels and frequency (mel bins) dimensions - hidden_states = hidden_states.flatten(1, 2) - - hidden_states = self.conv_in(hidden_states) - - for i in range(self.num_upsample_layers): - if self.act_fn == "leaky_relu": - # Other activations are inside each upsampling block - hidden_states = F.leaky_relu(hidden_states, negative_slope=self.negative_slope) - hidden_states = self.upsamplers[i](hidden_states) - - # Run all resnets in parallel on hidden_states - start = i * self.resnets_per_upsample - end = (i + 1) * self.resnets_per_upsample - resnet_outputs = torch.stack([self.resnets[j](hidden_states) for j in range(start, end)], dim=0) - - hidden_states = torch.mean(resnet_outputs, dim=0) - - hidden_states = self.act_out(hidden_states) - hidden_states = self.conv_out(hidden_states) - if self.final_act_fn == "tanh": - hidden_states = torch.tanh(hidden_states) - elif self.final_act_fn == "clamp": - hidden_states = torch.clamp(hidden_states, -1, 1) - - return hidden_states - - -class CausalSTFT(nn.Module): - """ - Performs a causal short-time Fourier transform (STFT) using causal Hann windows on a waveform. The DFT bases - multiplied by the Hann windows are pre-calculated and stored as buffers. For exact parity with training, the exact - buffers should be loaded from the checkpoint in bfloat16. - """ - - def __init__(self, filter_length: int = 512, hop_length: int = 80, window_length: int = 512): - super().__init__() - self.hop_length = hop_length - self.window_length = window_length - n_freqs = filter_length // 2 + 1 - - self.register_buffer("forward_basis", torch.zeros(n_freqs * 2, 1, filter_length), persistent=True) - self.register_buffer("inverse_basis", torch.zeros(n_freqs * 2, 1, filter_length), persistent=True) - - def forward(self, waveform: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: - if waveform.ndim == 2: - waveform = waveform.unsqueeze(1) # [B, num_channels, num_samples] - - left_pad = max(0, self.window_length - self.hop_length) # causal: left-only - waveform = F.pad(waveform, (left_pad, 0)) - - spec = F.conv1d(waveform, self.forward_basis, stride=self.hop_length, padding=0) - n_freqs = spec.shape[1] // 2 - real, imag = spec[:, :n_freqs], spec[:, n_freqs:] - magnitude = torch.sqrt(real**2 + imag**2) - phase = torch.atan2(imag.float(), real.float()).to(dtype=real.dtype) - return magnitude, phase - - -class MelSTFT(nn.Module): - """ - Calculates a causal log-mel spectrogram from a waveform. Uses a pre-calculated mel filterbank, which should be - loaded from the checkpoint in bfloat16. - """ - - def __init__( - self, - filter_length: int = 512, - hop_length: int = 80, - window_length: int = 512, - num_mel_channels: int = 64, - ): - super().__init__() - self.stft_fn = CausalSTFT(filter_length, hop_length, window_length) - - num_freqs = filter_length // 2 + 1 - self.register_buffer("mel_basis", torch.zeros(num_mel_channels, num_freqs), persistent=True) - - def forward(self, waveform: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: - magnitude, phase = self.stft_fn(waveform) - energy = torch.norm(magnitude, dim=1) - mel = torch.matmul(self.mel_basis.to(magnitude.dtype), magnitude) - log_mel = torch.log(torch.clamp(mel, min=1e-5)) - return log_mel, magnitude, phase, energy - - -class LTX2VocoderWithBWE(ModelMixin, ConfigMixin): - """ - LTX-2.X vocoder with bandwidth extension (BWE) upsampling. The vocoder and the BWE module run in sequence, with the - BWE module upsampling the vocoder output waveform to a higher sampling rate. The BWE module itself has the same - architecture as the original vocoder. - """ - - @register_to_config - def __init__( - self, - in_channels: int = 128, - hidden_channels: int = 1536, - out_channels: int = 2, - upsample_kernel_sizes: list[int] = [11, 4, 4, 4, 4, 4], - upsample_factors: list[int] = [5, 2, 2, 2, 2, 2], - resnet_kernel_sizes: list[int] = [3, 7, 11], - resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], - act_fn: str = "snakebeta", - leaky_relu_negative_slope: float = 0.1, - antialias: bool = True, - antialias_ratio: int = 2, - antialias_kernel_size: int = 12, - final_act_fn: str | None = None, - final_bias: bool = False, - bwe_in_channels: int = 128, - bwe_hidden_channels: int = 512, - bwe_out_channels: int = 2, - bwe_upsample_kernel_sizes: list[int] = [12, 11, 4, 4, 4], - bwe_upsample_factors: list[int] = [6, 5, 2, 2, 2], - bwe_resnet_kernel_sizes: list[int] = [3, 7, 11], - bwe_resnet_dilations: list[list[int]] = [[1, 3, 5], [1, 3, 5], [1, 3, 5]], - bwe_act_fn: str = "snakebeta", - bwe_leaky_relu_negative_slope: float = 0.1, - bwe_antialias: bool = True, - bwe_antialias_ratio: int = 2, - bwe_antialias_kernel_size: int = 12, - bwe_final_act_fn: str | None = None, - bwe_final_bias: bool = False, - filter_length: int = 512, - hop_length: int = 80, - window_length: int = 512, - num_mel_channels: int = 64, - input_sampling_rate: int = 16000, - output_sampling_rate: int = 48000, - ): - super().__init__() - - self.vocoder = LTX2Vocoder( - in_channels=in_channels, - hidden_channels=hidden_channels, - out_channels=out_channels, - upsample_kernel_sizes=upsample_kernel_sizes, - upsample_factors=upsample_factors, - resnet_kernel_sizes=resnet_kernel_sizes, - resnet_dilations=resnet_dilations, - act_fn=act_fn, - leaky_relu_negative_slope=leaky_relu_negative_slope, - antialias=antialias, - antialias_ratio=antialias_ratio, - antialias_kernel_size=antialias_kernel_size, - final_act_fn=final_act_fn, - final_bias=final_bias, - output_sampling_rate=input_sampling_rate, - ) - self.bwe_generator = LTX2Vocoder( - in_channels=bwe_in_channels, - hidden_channels=bwe_hidden_channels, - out_channels=bwe_out_channels, - upsample_kernel_sizes=bwe_upsample_kernel_sizes, - upsample_factors=bwe_upsample_factors, - resnet_kernel_sizes=bwe_resnet_kernel_sizes, - resnet_dilations=bwe_resnet_dilations, - act_fn=bwe_act_fn, - leaky_relu_negative_slope=bwe_leaky_relu_negative_slope, - antialias=bwe_antialias, - antialias_ratio=bwe_antialias_ratio, - antialias_kernel_size=bwe_antialias_kernel_size, - final_act_fn=bwe_final_act_fn, - final_bias=bwe_final_bias, - output_sampling_rate=output_sampling_rate, +# Copyright 2026 Lightricks and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ...models.autoencoders.vocoder_ltx2 import LTX2Vocoder as _LTX2Vocoder +from ...models.autoencoders.vocoder_ltx2 import LTX2VocoderWithBWE as _LTX2VocoderWithBWE +from ...utils import deprecate + + +# Deprecated aliases kept for backward compatibility. The warning is raised from `__new__` rather than `__init__` so +# that the parent's `__init__` signature, which `ConfigMixin` inspects when loading a config, stays intact. +class LTX2Vocoder(_LTX2Vocoder): + def __new__(cls, *args, **kwargs): + deprecate( + "LTX2Vocoder", + "1.0.0", + "Importing `LTX2Vocoder` from `diffusers.pipelines.ltx2.vocoder` is deprecated. " + "Import it with `from diffusers import LTX2Vocoder` instead.", ) + return super().__new__(cls) - self.mel_stft = MelSTFT( - filter_length=filter_length, - hop_length=hop_length, - window_length=window_length, - num_mel_channels=num_mel_channels, - ) - self.resampler = UpSample1d( - ratio=output_sampling_rate // input_sampling_rate, - window_type="hann", - persistent=False, +class LTX2VocoderWithBWE(_LTX2VocoderWithBWE): + def __new__(cls, *args, **kwargs): + deprecate( + "LTX2VocoderWithBWE", + "1.0.0", + "Importing `LTX2VocoderWithBWE` from `diffusers.pipelines.ltx2.vocoder` is deprecated. " + "Import it with `from diffusers import LTX2VocoderWithBWE` instead.", ) - - def forward(self, mel_spec: torch.Tensor) -> torch.Tensor: - # 1. Run stage 1 vocoder to get low sampling rate waveform - x = self.vocoder(mel_spec) - batch_size, num_channels, num_samples = x.shape - - # Pad to exact multiple of hop_length for exact mel frame count - remainder = num_samples % self.config.hop_length - if remainder != 0: - x = F.pad(x, (0, self.hop_length - remainder)) - - # 2. Compute mel spectrogram on vocoder output - mel, _, _, _ = self.mel_stft(x.flatten(0, 1)) - mel = mel.unflatten(0, (-1, num_channels)) - - # 3. Run bandwidth extender (BWE) on new mel spectrogram - mel_for_bwe = mel.transpose(2, 3) # [B, C, num_mel_bins, num_frames] --> [B, C, num_frames, num_mel_bins] - residual = self.bwe_generator(mel_for_bwe) - - # 4. Residual connection with resampler - skip = self.resampler(x) - waveform = torch.clamp(residual + skip, -1, 1) - output_samples = num_samples * self.config.output_sampling_rate // self.config.input_sampling_rate - waveform = waveform[..., :output_samples] - return waveform + return super().__new__(cls) diff --git a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py index 0c8fd842fcba..820ffe4e121f 100644 --- a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py +++ b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py @@ -39,6 +39,7 @@ PriorTransformer, UNet2DConditionModel, ) +from ...models.condition_embedders import StableUnCLIPImageNormalizer from ...schedulers import ( DDIMScheduler, DDPMScheduler, @@ -57,7 +58,6 @@ from ..latent_diffusion.pipeline_latent_diffusion import LDMBertConfig, LDMBertModel from ..pipeline_utils import DiffusionPipeline from .safety_checker import StableDiffusionSafetyChecker -from .stable_unclip_image_normalizer import StableUnCLIPImageNormalizer if is_accelerate_available(): diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip.py index 1d6feec05846..a88c83a088d6 100644 --- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip.py +++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip.py @@ -22,6 +22,7 @@ from ...image_processor import VaeImageProcessor from ...loaders import StableDiffusionLoraLoaderMixin, TextualInversionLoaderMixin from ...models import AutoencoderKL, PriorTransformer, UNet2DConditionModel +from ...models.condition_embedders import StableUnCLIPImageNormalizer from ...models.embeddings import get_timestep_embedding from ...models.lora import adjust_lora_scale_text_encoder from ...schedulers import KarrasDiffusionSchedulers @@ -36,7 +37,6 @@ ) from ...utils.torch_utils import randn_tensor from ..pipeline_utils import DiffusionPipeline, ImagePipelineOutput, StableDiffusionMixin -from .stable_unclip_image_normalizer import StableUnCLIPImageNormalizer if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py index c560b411620e..4cccbcc29172 100644 --- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py +++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py @@ -22,6 +22,7 @@ from ...image_processor import VaeImageProcessor from ...loaders import StableDiffusionLoraLoaderMixin, TextualInversionLoaderMixin from ...models import AutoencoderKL, UNet2DConditionModel +from ...models.condition_embedders import StableUnCLIPImageNormalizer from ...models.embeddings import get_timestep_embedding from ...models.lora import adjust_lora_scale_text_encoder from ...schedulers import KarrasDiffusionSchedulers @@ -36,7 +37,6 @@ ) from ...utils.torch_utils import randn_tensor from ..pipeline_utils import DiffusionPipeline, ImagePipelineOutput, StableDiffusionMixin -from .stable_unclip_image_normalizer import StableUnCLIPImageNormalizer if is_torch_xla_available(): diff --git a/src/diffusers/pipelines/stable_diffusion/stable_unclip_image_normalizer.py b/src/diffusers/pipelines/stable_diffusion/stable_unclip_image_normalizer.py index 1a29bc9bb63b..646d4cf8bbb3 100644 --- a/src/diffusers/pipelines/stable_diffusion/stable_unclip_image_normalizer.py +++ b/src/diffusers/pipelines/stable_diffusion/stable_unclip_image_normalizer.py @@ -12,44 +12,20 @@ # See the License for the specific language governing permissions and # limitations under the License. -import torch -from torch import nn - -from ...configuration_utils import ConfigMixin, register_to_config -from ...models.modeling_utils import ModelMixin - - -class StableUnCLIPImageNormalizer(ModelMixin, ConfigMixin): - """ - This class is used to hold the mean and standard deviation of the CLIP embedder used in stable unCLIP. - - It is used to normalize the image embeddings before the noise is applied and un-normalize the noised image - embeddings. - """ - - @register_to_config - def __init__( - self, - embedding_dim: int = 768, - ): - super().__init__() - - self.mean = nn.Parameter(torch.zeros(1, embedding_dim)) - self.std = nn.Parameter(torch.ones(1, embedding_dim)) - - def to( - self, - torch_device: str | torch.device | None = None, - torch_dtype: torch.dtype | None = None, - ): - self.mean = nn.Parameter(self.mean.to(torch_device).to(torch_dtype)) - self.std = nn.Parameter(self.std.to(torch_device).to(torch_dtype)) - return self - - def scale(self, embeds): - embeds = (embeds - self.mean) * 1.0 / self.std - return embeds - - def unscale(self, embeds): - embeds = (embeds * self.std) + self.mean - return embeds +from ...models.condition_embedders.condition_embedder_stable_unclip import ( + StableUnCLIPImageNormalizer as _StableUnCLIPImageNormalizer, +) +from ...utils import deprecate + + +# Deprecated alias kept for backward compatibility. The warning is raised from `__new__` rather than `__init__` so +# that the parent's `__init__` signature, which `ConfigMixin` inspects when loading a config, stays intact. +class StableUnCLIPImageNormalizer(_StableUnCLIPImageNormalizer): + def __new__(cls, *args, **kwargs): + deprecate( + "StableUnCLIPImageNormalizer", + "1.0.0", + "Importing `StableUnCLIPImageNormalizer` from `diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer` is deprecated. " + "Import it with `from diffusers import StableUnCLIPImageNormalizer` instead.", + ) + return super().__new__(cls) diff --git a/src/diffusers/utils/deprecation_utils.py b/src/diffusers/utils/deprecation_utils.py index 48b427d9e1ff..a06fafbafccf 100644 --- a/src/diffusers/utils/deprecation_utils.py +++ b/src/diffusers/utils/deprecation_utils.py @@ -72,15 +72,21 @@ def _maybe_remap_transformers_class(class_name: str) -> str | None: # reference them as `[, ]`, so they are resolved to the top-level `diffusers` export # instead of the deprecated alias left in the pipeline folder. _RELOCATED_PIPELINE_CLASSES = { + ("ace_step", "AceStepAudioTokenDetokenizer"), + ("ace_step", "AceStepAudioTokenizer"), ("ace_step", "AceStepConditionEncoder"), ("audioldm2", "AudioLDM2ProjectionModel"), + ("audioldm2", "AudioLDM2UNet2DConditionModel"), ("flux", "ReduxImageEncoder"), ("ltx", "LTXLatentUpsamplerModel"), ("ltx2", "LTX2LatentUpsamplerModel"), ("ltx2", "LTX2TextConnectors"), + ("ltx2", "LTX2Vocoder"), + ("ltx2", "LTX2VocoderWithBWE"), ("stable_audio", "StableAudioProjectionModel"), ("stable_audio_3", "StableAudio3DurationEmbedder"), ("stable_diffusion", "CLIPImageProjection"), + ("stable_diffusion", "StableUnCLIPImageNormalizer"), } diff --git a/src/diffusers/utils/dummy_pt_objects.py b/src/diffusers/utils/dummy_pt_objects.py index 5884b302c8ae..e11524497db0 100644 --- a/src/diffusers/utils/dummy_pt_objects.py +++ b/src/diffusers/utils/dummy_pt_objects.py @@ -439,6 +439,36 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch"]) +class AceStepAudioTokenDetokenizer(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + +class AceStepAudioTokenizer(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + class AceStepConditionEncoder(metaclass=DummyObject): _backends = ["torch"] @@ -574,6 +604,21 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch"]) +class AudioLDM2UNet2DConditionModel(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + class AuraFlowTransformer2DModel(metaclass=DummyObject): _backends = ["torch"] @@ -1969,6 +2014,36 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch"]) +class LTX2Vocoder(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + +class LTX2VocoderWithBWE(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + class LTXLatentUpsamplerModel(metaclass=DummyObject): _backends = ["torch"] @@ -2689,6 +2764,21 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch"]) +class StableUnCLIPImageNormalizer(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + class T2IAdapter(metaclass=DummyObject): _backends = ["torch"] diff --git a/src/diffusers/utils/dummy_torch_and_transformers_objects.py b/src/diffusers/utils/dummy_torch_and_transformers_objects.py index 2ef88ff59bc2..c5f61f870a6d 100644 --- a/src/diffusers/utils/dummy_torch_and_transformers_objects.py +++ b/src/diffusers/utils/dummy_torch_and_transformers_objects.py @@ -1397,36 +1397,6 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch", "transformers"]) -class AceStepAudioTokenDetokenizer(metaclass=DummyObject): - _backends = ["torch", "transformers"] - - def __init__(self, *args, **kwargs): - requires_backends(self, ["torch", "transformers"]) - - @classmethod - def from_config(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - @classmethod - def from_pretrained(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - -class AceStepAudioTokenizer(metaclass=DummyObject): - _backends = ["torch", "transformers"] - - def __init__(self, *args, **kwargs): - requires_backends(self, ["torch", "transformers"]) - - @classmethod - def from_config(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - @classmethod - def from_pretrained(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - class AceStepPipeline(metaclass=DummyObject): _backends = ["torch", "transformers"] @@ -1682,21 +1652,6 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch", "transformers"]) -class AudioLDM2UNet2DConditionModel(metaclass=DummyObject): - _backends = ["torch", "transformers"] - - def __init__(self, *args, **kwargs): - requires_backends(self, ["torch", "transformers"]) - - @classmethod - def from_config(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - @classmethod - def from_pretrained(cls, *args, **kwargs): - requires_backends(cls, ["torch", "transformers"]) - - class AudioLDMPipeline(metaclass=DummyObject): _backends = ["torch", "transformers"] diff --git a/tests/models/test_relocated_class_loading.py b/tests/models/test_relocated_class_loading.py index f232ce813f9d..fe77f0434643 100644 --- a/tests/models/test_relocated_class_loading.py +++ b/tests/models/test_relocated_class_loading.py @@ -29,6 +29,8 @@ # (pipeline folder, deprecated module path, class name, init kwargs) RELOCATED_CLASSES = [ + ("ace_step", "diffusers.pipelines.ace_step.modeling_ace_step", "AceStepAudioTokenDetokenizer", {}), + ("ace_step", "diffusers.pipelines.ace_step.modeling_ace_step", "AceStepAudioTokenizer", {}), ("ace_step", "diffusers.pipelines.ace_step.modeling_ace_step", "AceStepConditionEncoder", {}), ( "audioldm2", @@ -36,10 +38,26 @@ "AudioLDM2ProjectionModel", {"text_encoder_dim": 8, "text_encoder_1_dim": 8, "langauge_model_dim": 16}, ), + ( + "audioldm2", + "diffusers.pipelines.audioldm2.modeling_audioldm2", + "AudioLDM2UNet2DConditionModel", + { + "block_out_channels": (8, 16), + "layers_per_block": 1, + "norm_num_groups": 8, + "down_block_types": ("DownBlock2D", "CrossAttnDownBlock2D"), + "up_block_types": ("CrossAttnUpBlock2D", "UpBlock2D"), + "cross_attention_dim": ([None, 8], [None, 8]), + "attention_head_dim": 2, + }, + ), ("flux", "diffusers.pipelines.flux.modeling_flux", "ReduxImageEncoder", {}), ("ltx", "diffusers.pipelines.ltx.modeling_latent_upsampler", "LTXLatentUpsamplerModel", {}), ("ltx2", "diffusers.pipelines.ltx2.latent_upsampler", "LTX2LatentUpsamplerModel", {}), ("ltx2", "diffusers.pipelines.ltx2.connectors", "LTX2TextConnectors", {}), + ("ltx2", "diffusers.pipelines.ltx2.vocoder", "LTX2Vocoder", {}), + ("ltx2", "diffusers.pipelines.ltx2.vocoder", "LTX2VocoderWithBWE", {}), ( "stable_audio", "diffusers.pipelines.stable_audio.modeling_stable_audio", @@ -53,9 +71,33 @@ {}, ), ("stable_diffusion", "diffusers.pipelines.stable_diffusion.clip_image_project_model", "CLIPImageProjection", {}), + ( + "stable_diffusion", + "diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer", + "StableUnCLIPImageNormalizer", + {}, + ), ] IDS = [name for _, _, name, _ in RELOCATED_CLASSES] +# Relocated symbols that are not top-level `diffusers` exports and so are not in `_RELOCATED_PIPELINE_CLASSES`: +# (deprecated module path, new module path, symbol name, call kwargs) +RELOCATED_HELPERS = [ + ( + "diffusers.pipelines.ace_step.modeling_ace_step", + "diffusers.models.condition_embedders.condition_embedder_ace_step", + "AceStepLyricEncoder", + {}, + ), + ( + "diffusers.pipelines.ace_step.modeling_ace_step", + "diffusers.models.condition_embedders.condition_embedder_ace_step", + "AceStepTimbreEncoder", + {}, + ), +] +HELPER_IDS = [name for _, _, name, _ in RELOCATED_HELPERS] + def _future_warnings(fn): with warnings.catch_warnings(record=True) as caught: @@ -83,3 +125,16 @@ def test_model_index_entry_resolves_to_new_class(folder, module, name, kwargs): class_obj, _ = get_class_obj_and_candidates(folder, name, ALL_IMPORTABLE_CLASSES, pipelines, True) assert class_obj is new_cls assert simple_get_class_obj(folder, name) is new_cls + + +@pytest.mark.parametrize("module, new_module, name, kwargs", RELOCATED_HELPERS, ids=HELPER_IDS) +def test_deprecated_helper_path_warns(module, new_module, name, kwargs): + new_obj = getattr(importlib.import_module(new_module), name) + old_obj = getattr(importlib.import_module(module), name) + if isinstance(old_obj, type): + assert issubclass(old_obj, new_obj) + + with torch.device("meta"): + assert not _future_warnings(lambda: new_obj(**kwargs)) + messages = _future_warnings(lambda: old_obj(**kwargs)) + assert any(name in message and "deprecated" in message for message in messages), messages diff --git a/tests/pipelines/ltx2/testing_utils.py b/tests/pipelines/ltx2/testing_utils.py index 1e3a2e225ebf..a45ae626e052 100644 --- a/tests/pipelines/ltx2/testing_utils.py +++ b/tests/pipelines/ltx2/testing_utils.py @@ -29,12 +29,12 @@ AutoencoderKLLTX2Video, FlowMatchEulerDiscreteScheduler, LTX2VideoTransformer3DModel, + LTX2Vocoder, LTXEulerAncestralRFScheduler, ) from diffusers.models.condition_embedders import LTX2TextConnectors from diffusers.models.latent_upscaler import LTX2LatentUpsamplerModel from diffusers.pipelines.ltx2 import LTX2DurationHead -from diffusers.pipelines.ltx2.vocoder import LTX2Vocoder from ...testing_utils import torch_device from ..testing_utils import ( diff --git a/tests/pipelines/shap_e/__init__.py b/tests/pipelines/shap_e/__init__.py deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/tests/pipelines/shap_e/test_shap_e.py b/tests/pipelines/shap_e/test_shap_e.py deleted file mode 100644 index aa8cc23aa8e3..000000000000 --- a/tests/pipelines/shap_e/test_shap_e.py +++ /dev/null @@ -1,260 +0,0 @@ -# Copyright 2026 HuggingFace Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import gc - -import numpy as np -import pytest -import torch -from transformers import CLIPTextConfig, CLIPTextModelWithProjection, CLIPTokenizer - -from diffusers import HeunDiscreteScheduler, PriorTransformer, ShapEPipeline -from diffusers.pipelines.shap_e import ShapERenderer - -from ...testing_utils import ( - backend_empty_cache, - load_numpy, - nightly, - require_torch_accelerator, - torch_device, -) -from ..testing_utils import ( - BasePipelineTesterConfig, - MemoryTesterMixin, - PipelineTesterMixin, - assert_mean_pixel_difference, -) - - -class ShapEPipelineTesterConfig(BasePipelineTesterConfig): - pipeline_class = ShapEPipeline - required_input_params_in_call_signature = frozenset(["prompt"]) - batch_input_params = frozenset(["prompt"]) - # Shap-E renders a 3D asset, so `guidance_scale` and `frame_size` join the canonical optional parameters. - optional_input_params = frozenset( - [ - "num_inference_steps", - "num_images_per_prompt", - "generator", - "latents", - "guidance_scale", - "frame_size", - "output_type", - "return_dict", - ] - ) - # The dummy inputs request latents, so a sample is the flattened latent grid rather than an image. - output_shape = (32, 16) - - @property - def text_embedder_hidden_size(self): - return 16 - - @property - def time_input_dim(self): - return 16 - - @property - def time_embed_dim(self): - return self.time_input_dim * 4 - - @property - def renderer_dim(self): - return 8 - - @property - def dummy_tokenizer(self): - tokenizer = CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip") - return tokenizer - - @property - def dummy_text_encoder(self): - torch.manual_seed(0) - config = CLIPTextConfig( - bos_token_id=0, - eos_token_id=2, - hidden_size=self.text_embedder_hidden_size, - projection_dim=self.text_embedder_hidden_size, - intermediate_size=37, - layer_norm_eps=1e-05, - num_attention_heads=4, - num_hidden_layers=5, - pad_token_id=1, - vocab_size=1000, - ) - return CLIPTextModelWithProjection(config) - - @property - def dummy_prior(self): - torch.manual_seed(0) - - model_kwargs = { - "num_attention_heads": 2, - "attention_head_dim": 16, - "embedding_dim": self.time_input_dim, - "num_embeddings": 32, - "embedding_proj_dim": self.text_embedder_hidden_size, - "time_embed_dim": self.time_embed_dim, - "num_layers": 1, - "clip_embed_dim": self.time_input_dim * 2, - "additional_embeddings": 0, - "time_embed_act_fn": "gelu", - "norm_in_type": "layer", - "encoder_hid_proj_type": None, - "added_emb_type": None, - } - - model = PriorTransformer(**model_kwargs) - return model - - @property - def dummy_renderer(self): - torch.manual_seed(0) - - model_kwargs = { - "param_shapes": ( - (self.renderer_dim, 93), - (self.renderer_dim, 8), - (self.renderer_dim, 8), - (self.renderer_dim, 8), - ), - "d_latent": self.time_input_dim, - "d_hidden": self.renderer_dim, - "n_output": 12, - "background": ( - 0.1, - 0.1, - 0.1, - ), - } - model = ShapERenderer(**model_kwargs) - return model - - def get_dummy_components(self): - prior = self.dummy_prior - text_encoder = self.dummy_text_encoder - tokenizer = self.dummy_tokenizer - shap_e_renderer = self.dummy_renderer - - scheduler = HeunDiscreteScheduler( - beta_schedule="exp", - num_train_timesteps=1024, - prediction_type="sample", - use_karras_sigmas=True, - clip_sample=True, - clip_sample_range=1.0, - ) - return { - "prior": prior, - "text_encoder": text_encoder, - "tokenizer": tokenizer, - "shap_e_renderer": shap_e_renderer, - "scheduler": scheduler, - } - - def get_dummy_inputs(self): - return { - "prompt": "horse", - "generator": self.get_generator(0), - "num_inference_steps": 1, - "frame_size": 32, - "output_type": "latent", - } - - -class TestShapEPipeline(ShapEPipelineTesterConfig, PipelineTesterMixin): - def test_shap_e(self): - pipe = self.get_pipeline().to(torch_device) - - output = pipe(**self.get_dummy_inputs()) - image = output.images[0] - image = image.cpu().numpy() - image_slice = image[-3:, -3:] - - assert image.shape == self.output_shape - - expected_slice = np.array([-1.0000, -0.6097, 1.0000, -0.9113, -0.6667, 0.8234, -0.7639, -0.2809, 0.6456]) - assert np.abs(image_slice.flatten() - expected_slice).max() < 1e-2 - - def test_inference_batch_consistent(self): - # NOTE: Larger batch sizes cause this test to timeout, only test on smaller batches - super().test_inference_batch_consistent(batch_sizes=[1, 2]) - - def test_inference_batch_single_identical(self): - super().test_inference_batch_single_identical(batch_size=2, expected_max_diff=6e-3) - - def test_num_images_per_prompt(self): - # The base test sweeps batch size x images-per-prompt; rendering makes that too slow here, so check the - # single interesting combination. - pipe = self.get_pipeline().to(torch_device) - - batch_size = 1 - num_images_per_prompt = 2 - - inputs = self.get_dummy_inputs() - for key in inputs.keys(): - if key in self.batch_input_params: - inputs[key] = batch_size * [inputs[key]] - - images = pipe(**inputs, num_images_per_prompt=num_images_per_prompt)[0] - - assert images.shape[0] == batch_size * num_images_per_prompt - - def test_save_load_local(self, tmp_path, base_pipe_output): - super().test_save_load_local(tmp_path, base_pipe_output, expected_max_difference=5e-3) - - -class TestShapEPipelineMemory(ShapEPipelineTesterConfig, MemoryTesterMixin): - """Memory optimization tests (CPU offload, group offload, layerwise casting) for the Shap-E pipeline.""" - - @pytest.mark.skip("Key error is raised with accelerate") - def test_sequential_cpu_offload_forward_pass(self): - pass - - -@nightly -@require_torch_accelerator -class TestShapEPipelineIntegration: - @pytest.fixture(autouse=True) - def cleanup(self): - # clean up the VRAM before and after each test - gc.collect() - backend_empty_cache(torch_device) - yield - gc.collect() - backend_empty_cache(torch_device) - - def test_shap_e(self): - expected_image = load_numpy( - "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main" - "/shap_e/test_shap_e_np_out.npy" - ) - pipe = ShapEPipeline.from_pretrained("openai/shap-e") - pipe = pipe.to(torch_device) - pipe.set_progress_bar_config(disable=None) - - generator = torch.Generator(device=torch_device).manual_seed(0) - - images = pipe( - "a shark", - generator=generator, - guidance_scale=15.0, - num_inference_steps=64, - frame_size=64, - output_type="np", - ).images[0] - - assert images.shape == (20, 64, 64, 3) - - assert_mean_pixel_difference(images, expected_image) diff --git a/tests/pipelines/shap_e/test_shap_e_img2img.py b/tests/pipelines/shap_e/test_shap_e_img2img.py deleted file mode 100644 index bab08826d170..000000000000 --- a/tests/pipelines/shap_e/test_shap_e_img2img.py +++ /dev/null @@ -1,276 +0,0 @@ -# Copyright 2026 HuggingFace Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import gc -import random - -import numpy as np -import pytest -import torch -from transformers import CLIPImageProcessor, CLIPVisionConfig, CLIPVisionModel - -from diffusers import HeunDiscreteScheduler, PriorTransformer, ShapEImg2ImgPipeline -from diffusers.pipelines.shap_e import ShapERenderer - -from ...testing_utils import ( - backend_empty_cache, - floats_tensor, - load_image, - load_numpy, - nightly, - require_torch_accelerator, - torch_device, -) -from ..testing_utils import ( - BasePipelineTesterConfig, - MemoryTesterMixin, - PipelineTesterMixin, - assert_mean_pixel_difference, -) - - -class ShapEImg2ImgPipelineTesterConfig(BasePipelineTesterConfig): - pipeline_class = ShapEImg2ImgPipeline - required_input_params_in_call_signature = frozenset(["image"]) - batch_input_params = frozenset(["image"]) - # Shap-E renders a 3D asset, so `guidance_scale` and `frame_size` join the canonical optional parameters. - optional_input_params = frozenset( - [ - "num_inference_steps", - "num_images_per_prompt", - "generator", - "latents", - "guidance_scale", - "frame_size", - "output_type", - "return_dict", - ] - ) - # The dummy inputs request latents, so a sample is the flattened latent grid rather than an image. - output_shape = (32, 16) - - @property - def text_embedder_hidden_size(self): - return 16 - - @property - def time_input_dim(self): - return 16 - - @property - def time_embed_dim(self): - return self.time_input_dim * 4 - - @property - def renderer_dim(self): - return 8 - - @property - def dummy_image_encoder(self): - torch.manual_seed(0) - config = CLIPVisionConfig( - hidden_size=self.text_embedder_hidden_size, - image_size=32, - projection_dim=self.text_embedder_hidden_size, - intermediate_size=24, - num_attention_heads=2, - num_channels=3, - num_hidden_layers=5, - patch_size=1, - ) - - model = CLIPVisionModel(config) - return model - - @property - def dummy_image_processor(self): - image_processor = CLIPImageProcessor( - crop_size=224, - do_center_crop=True, - do_normalize=True, - do_resize=True, - image_mean=[0.48145466, 0.4578275, 0.40821073], - image_std=[0.26862954, 0.26130258, 0.27577711], - resample=3, - size=224, - ) - - return image_processor - - @property - def dummy_prior(self): - torch.manual_seed(0) - - model_kwargs = { - "num_attention_heads": 2, - "attention_head_dim": 16, - "embedding_dim": self.time_input_dim, - "num_embeddings": 32, - "embedding_proj_dim": self.text_embedder_hidden_size, - "time_embed_dim": self.time_embed_dim, - "num_layers": 1, - "clip_embed_dim": self.time_input_dim * 2, - "additional_embeddings": 0, - "time_embed_act_fn": "gelu", - "norm_in_type": "layer", - "embedding_proj_norm_type": "layer", - "encoder_hid_proj_type": None, - "added_emb_type": None, - } - - model = PriorTransformer(**model_kwargs) - return model - - @property - def dummy_renderer(self): - torch.manual_seed(0) - - model_kwargs = { - "param_shapes": ( - (self.renderer_dim, 93), - (self.renderer_dim, 8), - (self.renderer_dim, 8), - (self.renderer_dim, 8), - ), - "d_latent": self.time_input_dim, - "d_hidden": self.renderer_dim, - "n_output": 12, - "background": ( - 0.1, - 0.1, - 0.1, - ), - } - model = ShapERenderer(**model_kwargs) - return model - - def get_dummy_components(self): - prior = self.dummy_prior - image_encoder = self.dummy_image_encoder - image_processor = self.dummy_image_processor - shap_e_renderer = self.dummy_renderer - - scheduler = HeunDiscreteScheduler( - beta_schedule="exp", - num_train_timesteps=1024, - prediction_type="sample", - use_karras_sigmas=True, - clip_sample=True, - clip_sample_range=1.0, - ) - return { - "prior": prior, - "image_encoder": image_encoder, - "image_processor": image_processor, - "shap_e_renderer": shap_e_renderer, - "scheduler": scheduler, - } - - def get_dummy_inputs(self): - input_image = floats_tensor((1, 3, 32, 32), rng=random.Random(0)).to(torch_device) - return { - "image": input_image, - "generator": self.get_generator(0), - "num_inference_steps": 1, - "frame_size": 32, - "output_type": "latent", - } - - -class TestShapEImg2ImgPipeline(ShapEImg2ImgPipelineTesterConfig, PipelineTesterMixin): - def test_shap_e(self): - pipe = self.get_pipeline().to(torch_device) - - output = pipe(**self.get_dummy_inputs()) - image = output.images[0] - image_slice = image[-3:, -3:].cpu().numpy() - - assert image.shape == self.output_shape - - expected_slice = np.array( - [-1.0, -0.08255208, 1.0, -0.99469435, -0.19953543, 0.99999994, -0.9133135, 0.24063873, 0.8238962] - ) - - assert np.abs(image_slice.flatten() - expected_slice).max() < 1e-2 - - def test_inference_batch_single_identical(self): - super().test_inference_batch_single_identical(batch_size=2, expected_max_diff=6e-3) - - def test_num_images_per_prompt(self): - # The base test sweeps batch size x images-per-prompt; rendering makes that too slow here, so check the - # single interesting combination. - pipe = self.get_pipeline().to(torch_device) - - batch_size = 1 - num_images_per_prompt = 2 - - inputs = self.get_dummy_inputs() - for key in inputs.keys(): - if key in self.batch_input_params: - inputs[key] = batch_size * [inputs[key]] - - images = pipe(**inputs, num_images_per_prompt=num_images_per_prompt)[0] - - assert images.shape[0] == batch_size * num_images_per_prompt - - def test_save_load_local(self, tmp_path, base_pipe_output): - super().test_save_load_local(tmp_path, base_pipe_output, expected_max_difference=5e-3) - - -class TestShapEImg2ImgPipelineMemory(ShapEImg2ImgPipelineTesterConfig, MemoryTesterMixin): - """Memory optimization tests (CPU offload, group offload, layerwise casting) for the Shap-E img2img pipeline.""" - - @pytest.mark.skip("Key error is raised with accelerate") - def test_sequential_cpu_offload_forward_pass(self): - pass - - -@nightly -@require_torch_accelerator -class TestShapEImg2ImgPipelineIntegration: - @pytest.fixture(autouse=True) - def cleanup(self): - # clean up the VRAM before and after each test - gc.collect() - backend_empty_cache(torch_device) - yield - gc.collect() - backend_empty_cache(torch_device) - - def test_shap_e_img2img(self): - input_image = load_image( - "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/shap_e/corgi.png" - ) - expected_image = load_numpy( - "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main" - "/shap_e/test_shap_e_img2img_out.npy" - ) - pipe = ShapEImg2ImgPipeline.from_pretrained("openai/shap-e-img2img") - pipe = pipe.to(torch_device) - pipe.set_progress_bar_config(disable=None) - - generator = torch.Generator(device=torch_device).manual_seed(0) - - images = pipe( - input_image, - generator=generator, - guidance_scale=3.0, - num_inference_steps=64, - frame_size=64, - output_type="np", - ).images[0] - - assert images.shape == (20, 64, 64, 3) - - assert_mean_pixel_difference(images, expected_image) diff --git a/tests/pipelines/stable_unclip/test_stable_unclip.py b/tests/pipelines/stable_unclip/test_stable_unclip.py index befd4e2bf192..e78223c7899d 100644 --- a/tests/pipelines/stable_unclip/test_stable_unclip.py +++ b/tests/pipelines/stable_unclip/test_stable_unclip.py @@ -9,10 +9,10 @@ DDIMScheduler, DDPMScheduler, PriorTransformer, + StableUnCLIPImageNormalizer, StableUnCLIPPipeline, UNet2DConditionModel, ) -from diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer import StableUnCLIPImageNormalizer from ...testing_utils import ( backend_empty_cache, diff --git a/tests/pipelines/stable_unclip/test_stable_unclip_img2img.py b/tests/pipelines/stable_unclip/test_stable_unclip_img2img.py index 5f9ee5a5eaf6..f30f61efcf99 100644 --- a/tests/pipelines/stable_unclip/test_stable_unclip_img2img.py +++ b/tests/pipelines/stable_unclip/test_stable_unclip_img2img.py @@ -12,9 +12,15 @@ CLIPVisionModelWithProjection, ) -from diffusers import AutoencoderKL, DDIMScheduler, DDPMScheduler, StableUnCLIPImg2ImgPipeline, UNet2DConditionModel +from diffusers import ( + AutoencoderKL, + DDIMScheduler, + DDPMScheduler, + StableUnCLIPImageNormalizer, + StableUnCLIPImg2ImgPipeline, + UNet2DConditionModel, +) from diffusers.pipelines.pipeline_utils import DiffusionPipeline -from diffusers.pipelines.stable_diffusion.stable_unclip_image_normalizer import StableUnCLIPImageNormalizer from ...testing_utils import ( assert_tensors_close,