Skip to content

Add export_to_wav to diffusers.utils and use it in diffusers serve / diffusers run #15360

Open

Description

@moon-bot-app

Context

#15017 (diffusers serve) has this TODO in src/diffusers/commands/serve/backends.py, in the audio branch of the output encoding:

# TODO(DN6): write the files with `export_to_wav` once it is added to `diffusers.utils`.
for index, audio in enumerate(arrays):
    data = np.asarray(audio)
    if data.dtype.kind == "f":
        data = (np.clip(data, -1.0, 1.0) * 32767).astype(np.int16)
    else:
        data = data.astype(np.int16)
    # `wave` takes interleaved samples x channels; the shorter axis is taken to be the channels.
    if data.ndim == 1:
        data = data[:, None]
    elif data.shape[0] < data.shape[1]:
        data = data.T
    filename = f"{name}-{index}.wav"
    with wave.open(str(generation.directory / filename), "wb") as file:
        ...

diffusers.utils.export_utils already has export_to_gif, export_to_ply, export_to_obj and export_to_video, but nothing for audio. Because of that, WAV writing is now implemented by hand in at least two places:

  • src/diffusers/commands/run.py: _save_audios (on main), which writes 16-bit PCM WAV with the stdlib wave module
  • src/diffusers/commands/serve/backends.py (Add diffusers serve to CLI #15017): a near-copy of the same logic

The audio pipeline docstrings (AudioLDM, AudioLDM2, MusicLDM, Dance Diffusion) also tell users to call scipy.io.wavfile.write themselves.

Proposal

Add export_to_wav to diffusers/utils/export_utils.py and export it from diffusers.utils, following the existing export_to_* helpers. Roughly:

def export_to_wav(
    audio: np.ndarray | torch.Tensor,
    output_wav_path: str | None = None,
    sample_rate: int = 16000,
) -> str:
    ...
  • Accepts numpy arrays or torch tensors (converted with .detach().float().cpu().numpy())
  • Float input is clipped to [-1, 1] and converted to int16. Integer input is cast to int16.
  • 1D input is treated as mono. For 2D input, the shorter axis is treated as channels and transposed to samples × channels if needed (same heuristic used today).
  • Uses the stdlib wave module, so it adds no scipy dependency
  • If output_wav_path is not given, writes to a temp file (same as the other export_to_* helpers) and returns the path

Follow-ups

  • Replace the TODO in commands/serve/backends.py with a call to export_to_wav
  • Make _save_audios in commands/run.py use it too
  • Optionally, update the audio pipeline docstring examples to use export_to_wav instead of scipy.io.wavfile.write
  • Add a small unit test covering mono/stereo layouts and float/int inputs

Requested by Sayak Paul - Slack thread - Agent trace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions