Skip to content

feat(tpu): add SimpleFSDP from the torch_tpu reference code - #469

Draft
hodaaaaaaaaaa wants to merge 1 commit into
openrl-dev:feat/tpufrom
hodaaaaaaaaaa:feat/tpu-vendor-simple-fsdp
Draft

hodaaaaaaaaaa wants to merge 1 commit into
openrl-dev:feat/tpufrom
hodaaaaaaaaaa:feat/tpu-vendor-simple-fsdp

Conversation

@hodaaaaaaaaaa

Copy link
Copy Markdown
Collaborator

Problem

A model that does not fit one TPU chip cannot be trained today. The TPU trainer
runs on one chip.

Change

Adds SimpleFSDP, which shards a model's weights over several chips. The
implementation is taken from the torch_tpu reference code
(examples/fsdp/simple_fsdp.py). Nothing calls it yet. A test shards a tiny
model over two CPU processes and checks the gradients and the updated weights
against a one-process run.

Signed-off-by: Hoda Mohammadi <hodamo@google.com>
@hodaaaaaaaaaa
hodaaaaaaaaaa force-pushed the feat/tpu-vendor-simple-fsdp branch from e8679fd to 3316654 Compare November 20, 2026 00:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant