Repository navigation
fix: Fix default RNG synchronization for ditribution - #15141
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…sers into sync-parallel-rng-seed
|
|
||
| @is_tensor_parallel | ||
| @is_context_parallel | ||
| @pytest.mark.skipif(_parallel_device_type() is None, reason="test requires two accelerators") |
There was a problem hiding this comment.
Can we use require_torch_multi_accelerator?
diffusers/tests/testing_utils.py
Line 600 in 84532a0
There was a problem hiding this comment.
There are tests using the decorator elsewhere eg. ContextParallelAttentionBackendsTesterMixin, simply adding neuron/tpu to the decorator enables ~240 existing multi-accelerator tests that don't necessarily pass yet.
But I can add neuron/tpu options to the decorator and enable them only for the rng test rn, maybe later we can include neuron/tpu for all related tests.
afe646e to
a0330c8
Compare
sayakpaul
left a comment
There was a problem hiding this comment.
I opened JingyaHuang#3 to make testing a bit more generalised. LMK your thoughts.
…ests generalize the test case for sync rng for all supported models.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
sayakpaul
left a comment
There was a problem hiding this comment.
After https://github.com/huggingface/diffusers/pull/15137/changes were merged, the tests here in this PR started failing.
So, I opened JingyaHuang#4. After that we're good to go.
Handle list-valued main inputs in the parallel RNG sync tests
|
Failing test is unrelated. |
What does this PR do?
Synchronize default RNGs across each parallel group so inference without an explicit generator uses matching random draws on all ranks.
Add tests covering different seeds and matching seeds with different RNG offsets.
Before submitting
self-reviewskill on the diff?documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.