fix(grpo): reject epoch bounds that train zero steps

[bzantium]

What does this PR do ?

Rejects, at setup time, the two max_num_epochs bounds that make a run train zero steps — and stops the async bound from being written into the config it derives from.

Three follow-ups to #3248.

max_num_epochs <= 0 was skipped by that commit's > 0 guard, so an async run went to max_num_steps (default 1,000,000) unbounded. Both synchronous trainers gate on while current_epoch < max_num_epochs (grpo.py, grpo_sync.py) and fall straight through, returning 0 having trained nothing.

An empty training dataloader made the bound 0. StatefulDataLoader is built with drop_last=True, so a dataset smaller than num_prompts_per_step yields no batches; the clamp becomes 0, a fresh run satisfies step >= max_num_steps before the collector starts, and it prints "Async GRPO training is already complete … limit of 0 steps" and returns 0. Before #3248 the same misconfiguration failed loudly at the exhausted collector.

Both are configuration errors, so they are rejected in setup() beside the other _validate_* helpers rather than at the top of async_grpo_train. That placement matters three ways:

• it fails before the Ray placement groups and engines are built, per the "validate the complete boundary before allocating Ray placement groups" comment already in setup();
• it covers the synchronous trainers too, which otherwise keep exiting 0 with nothing trained — checking only the async path would trade one async/sync divergence for another;
• setup() derives the Megatron train_iters from the same two operands with no max(..., 1) floor (ppo.py and single_controller_utils/setup.py both have one), so a non-positive bound handed Megatron-Bridge a scheduler horizon of 0.

The <= 0 rejection follows _validate_algo_settings in single_controller_utils/config.py, which makes the same call for the same field, with the same reasoning in its comment.

Third, the bound was applied by mutating master_config.grpo.max_num_steps. init_tmp_checkpoint(step + 1, vars(grpo_save_state), master_config) serializes that object into every checkpoint, so a run configured for max_num_steps: 1000000 recorded the derived value instead of what the user wrote. It is a local now, as setup() already does with total_train_iters.

Issues

Follows up #3248. Our earlier PR for the same underlying bug, #3948, is closed as superseded by it.

Usage

Unchanged for valid configs. The two rejected cases now fail at setup:

ValueError: grpo.max_num_epochs=0 trains zero steps: the training loop gates on
current_epoch < max_num_epochs, and GRPO has no -1 convention. Set a positive
grpo.max_num_epochs and bound the run with grpo.max_num_steps.

ValueError: The training dataloader yields 0 batches, so the run is bounded at 0
steps. The dataloader drops the last partial batch, so a dataset smaller than
grpo.num_prompts_per_step=32 yields none at all. Lower grpo.num_prompts_per_step
or use a larger dataset.

On -1: v1 async PPO uses max_num_epochs: -1 to mean "no epoch bound", and examples/run_ppo.py raises unless it is exactly that. GRPO has never had that sentinel — GRPOConfig.max_num_epochs is int = 1, no shipped GRPO recipe or doc uses -1 or 0, and _validate_algo_settings already rejects <= 0 for GRPOConfig on the SingleController path. Raising here cannot break a PPO recipe. If GRPO ever wants "unbounded", the shape used elsewhere in the codebase is None, which this leaves available.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• Two tests added for the validator, in the shape of the neighbouring _validate_* tests. The existing test_async_grpo_exit_on_max_epochs keeps covering the clamp; its assertion changed from "the config was mutated" to "the config is untouched".
• The unit tests were not run locally — tests/unit/algorithms/test_grpo.py needs the nemo module, which is not importable in the environment this was written in.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.