Repository navigation
[MiniMax-H3, Pika H3 Flex] Resample reference soundtracks without torchaudio - #15199
christopher5106 wants to merge 1 commit into
Conversation
…chaudio The ref2va setup steps of MiniMax-H3 and Pika H3 Flex resample a reference soundtrack onto the audio VAE's sample rate with `torchaudio.transforms.Resample`, an optional dependency that is not installed with diffusers. Any soundtrack at another rate than the VAE's (e.g. the 44.1 / 48 kHz track of most videos) fails with an ImportError unless torchaudio is installed, and torchaudio has no release for recent torch versions. Add `diffusers.utils.torch_utils.resample_audio`, a plain-PyTorch port of torchaudio's default windowed-sinc resampler (`sinc_interp_hann`, `lowpass_filter_width=6`, `rolloff=0.99`), and use it in both steps. It matches `torchaudio.transforms.Resample` to float32 rounding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hi @christopher5106, thanks for the PR! It does not appear to link an issue it fixes.
See the contribution guide for more details. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an existing issue, the author box is ticked, or the |
|
This PR has been automatically closed because it does not link an issue and the reminder above was not addressed within 10 days. If this PR is still relevant, please link the issue it fixes (e.g. We are experimenting with this process to keep the review queue manageable, and it will sometimes get it wrong. If you think this PR should stay open, please just say so here and we will reopen it — no need to justify it at length. Thanks again for contributing, and sorry for the noise if we closed this by mistake! |
|
@sayakpaul this PR should not have been closed, because it links an issue |
|
This PR has been automatically closed because it does not link an issue and the reminder above was not addressed within 10 days. If this PR is still relevant, please link the issue it fixes (e.g. We are experimenting with this process to keep the review queue manageable, and it will sometimes get it wrong. If you think this PR should stay open, please just say so here and we will reopen it — no need to justify it at length. Thanks again for contributing, and sorry for the noise if we closed this by mistake! |
|
@sayakpaul second closing 😂 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
hey @christopher5106 for this, I think it should work on recent torch version now
let me know if it's not the case, otherwise we will close the PR and issue if it works when user install torchaudio |
What does this PR do?
Fixes #15245
The ref2va setup steps of MiniMax-H3 (
MiniMaxH3Ref2VASetupStep) and Pika H3 Flex (PikaH3FlexRef2VASetupStep) resample a reference soundtrack onto the audio VAE's sample rate withtorchaudio.transforms.Resample. torchaudio is not a diffusers dependency, so a reference soundtrack at any other rate than the VAE's 32 kHz (the 44.1 / 48 kHz audio track of most videos) fails:Installing torchaudio doesn't help everywhere: it has no release for recent torch versions (the last on PyPI is 2.11.1).
This PR adds
diffusers.utils.torch_utils.resample_audio, a plain-PyTorch port of torchaudio's default band-limited resampler (Hann-windowed sinc,lowpass_filter_width=6,rolloff=0.99), and uses it in both steps. Its output matchestorchaudio.transforms.Resampleto float32 rounding: identical at 48 → 32 kHz, within 4e-5 at 44.1 kHz. torchaudio's owntransforms.Resampleandfunctional.resamplediffer from each other by about as much.echostill imports torchaudio, because it also needsMelSpectrogram, so this PR doesn't change it.Tests
tests/others/test_utils.py::TestResampleAudio: parity withtorchaudio.transforms.Resamplefor 48→32, 44.1→32, 16→48 and 22.05→44.1 kHz (skipped without torchaudio), a 1 kHz tone keeping its frequency through 48 → 32 kHz, and checks on shape, dtype, the same-rate identity and the integer-input error.tests/modular_pipelines/minimax_h3:_normalize_audio_conditionresamples a 48 kHz mono reference withtorchaudioblocked insys.modules.The full
minimax_h3andpika_h3_flexsuites pass on CPU, except 4 Pika H3 Flex ref2va tests (test_inference_is_not_nan,test_save_from_pretrained, and the twoauto_cpu_offloadones), which fail the same way onmainwithout this change.Before submitting