Skip to content

[core] keep every CPU-overflow component in the balanced device map - #15205

Open
charchit7 wants to merge 2 commits into
huggingface:mainfrom
charchit7:fix-balanced-device-map-cpu-overflow
Open

charchit7 wants to merge 2 commits into
huggingface:mainfrom
charchit7:fix-balanced-device-map-cpu-overflow

Conversation

@charchit7

@charchit7 charchit7 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #15204

With device_map="balanced", each component that didn't fit on an accelerator overwrote the "cpu" entry in _assign_components_to_devices. Only the last one stayed in the map. The dropped components then loaded as plain CPU modules without accelerate's offload hooks, so inference failed with a device mismatch (repro in the linked issue).

This PR appends to the "cpu" entry instead, the same way the accelerator entries are handled a few lines below.

Behavior change: if every component overflows, hf_device_map now has more than one entry. _is_pipeline_device_mapped() then returns True, so .to() / enable_model_cpu_offload() ask for reset_device_map() first. That's the same as any other balanced map. Before, the single-entry map slipped past that check.

Tests

Added TestAssignComponentsToDevices to tests/pipelines/test_pipeline_utils.py (CPU-only, no downloads).

$ pytest tests/pipelines/test_pipeline_utils.py -k AssignComponents
3 passed, 58 deselected

# same tests against main:
>       assert mapping["cpu"] == ["transformer", "text_encoder"]
E       AssertionError: assert ['text_encoder'] == ['transformer...text_encoder']
1 failed, 2 passed, 58 deselected

End-to-end repro from the issue (SD 1.5, fp16, max_memory={0: "700MB"}, NVIDIA B200):

# main
hf_device_map: {'vae': 'cpu', 'safety_checker': 0}
RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)

# this PR
hf_device_map: {'unet': 'cpu', 'text_encoder': 'cpu', 'vae': 'cpu', 'safety_checker': 0}
inference OK

Same result on Apple Silicon (MPS).

ruff check and ruff format --check pass on the changed files.

Self-review notes

Ran the self-review skill against .ai/references/review-rules.md and testing.md. No blocking issues and no dead code.

Left for the reviewer:

  • The behavior change above, when every component overflows.
  • device_id_component_mapping.setdefault("cpu", []).append(component) would be one line shorter. I kept the if/else so it matches the accelerator branch right below it.

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Was this discussed/approved via a GitHub issue?
  • Did you write any new necessary tests?

@sayakpaul

Comment thread tests/pipelines/test_pipeline_utils.py Outdated
from ..testing_utils import require_torch_accelerator, torch_device


class TestAssignComponentsToDevices:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should find a more general way to test it instead of using the private methods.

@charchit7 charchit7 Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, yeah, makes sense, thanks.

I replaced them with one test that goes through from_pretrained: it loads the tiny SD pipeline with device_map="balanced" and a max_memory budget that only fits the text encoder, then checks that unet and vae are both in hf_device_map under "cpu" and that inference runs. On main it fails with KeyError: 'unet'.

It needs an accelerator, so it's skipped on the CPU jobs. I ran it on CUDA and MPS.

@sayakpaul sayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Just one comment.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

device_map="balanced" drops all but the last CPU-overflow component, which then loads without offload hooks and breaks inference

3 participants