Repository navigation
Feature/multi gpu - #25
Merged
Merged
Conversation
|
Sorry @gjbex, your pull request is larger than the review limit of 150,000 diff characters |
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
🟡 Changes recommended
Rank-local snapshot collision handling can leave distributed workers blocked in collectives.
2 open findings
What changed in this PR
Adds multi-GPU teaching material while modernizing optimization examples and consolidating environments.
Changes:
- Adds native PyTorch and Lightning DDP, FSDP2, checkpointing, sharding, and memory exercises.
- Replaces legacy Hyperopt/Julia examples with Optuna-based material.
- Updates environment definitions, metadata, and documentation.
| File | Description |
|---|---|
training.toml |
Adds language and maintainer metadata. |
source-code/README.md |
Updates optimization overview. |
source-code/pytorch/README.md |
Lists new PyTorch exercises. |
source-code/pytorch/memory-diagnostics/SOLUTION.md |
Adds diagnostic solution. |
source-code/pytorch/memory-diagnostics/README.md |
Documents memory exercise. |
source-code/pytorch/memory-diagnostics/memory_diagnostics.py |
Implements memory diagnostics. |
source-code/pytorch/memory-diagnostics/compare_memory_reports.py |
Compares diagnostic reports. |
source-code/pytorch/fsdp/README.md |
Documents FSDP2 benchmark. |
source-code/pytorch/fsdp/fsdp2_transformer.py |
Implements DDP/FSDP2 comparison. |
source-code/pytorch/environment.yml |
Removes obsolete local environment. |
source-code/pytorch/ddp/README.md |
Documents native DDP benchmark. |
source-code/pytorch/ddp/ddp_resnet50.py |
Implements ResNet-50 DDP benchmark. |
source-code/pytorch/ddp/data-sharding/solution_data_sharding.py |
Adds correct sharding solution. |
source-code/pytorch/ddp/data-sharding/README.md |
Documents sharding exercise. |
source-code/pytorch/ddp/data-sharding/buggy_data_sharding.py |
Adds diagnostic starter. |
source-code/pytorch/activation-checkpointing/README.md |
Documents checkpointing exercise. |
source-code/pytorch/activation-checkpointing/activation_checkpointing.py |
Implements checkpoint benchmark. |
source-code/pytorch-lightning/README.md |
Lists Lightning exercises. |
source-code/pytorch-lightning/fsdp/README.md |
Documents Lightning FSDP2. |
source-code/pytorch-lightning/ddp/README.md |
Documents Lightning DDP. |
source-code/pytorch-lightning/ddp/ddp_resnet50.py |
Implements Lightning DDP benchmark. |
source-code/pytorch-lightning/activation-checkpointing/README.md |
Documents Lightning checkpointing. |
source-code/pytorch-lightning/activation-checkpointing/activation_checkpointing.py |
Implements Lightning checkpointing. |
source-code/pinns/README.md |
Adds parameterized PINN material. |
source-code/pinns/environment.yml |
Removes redundant environment. |
source-code/parameter-optimization/simple.py |
Adds minimal Optuna example. |
source-code/parameter-optimization/README.md |
Reframes optimization material. |
source-code/parameter-optimization/optuna/README.md |
Removes superseded documentation. |
source-code/parameter-optimization/JuliaProcesses/trials_intel.txt |
Removes legacy trial data. |
source-code/parameter-optimization/JuliaProcesses/trials_gcc.txt |
Removes legacy trial data. |
source-code/parameter-optimization/JuliaProcesses/README.md |
Removes legacy documentation. |
source-code/parameter-optimization/JuliaProcesses/Makefile |
Removes legacy build setup. |
source-code/parameter-optimization/JuliaProcesses/julia_optimize.py |
Removes Hyperopt workflow. |
source-code/parameter-optimization/JuliaProcesses/julia_omp.cpp |
Removes Julia-set workload. |
source-code/parameter-optimization/JuliaProcesses/julia_intel.pbs |
Removes Intel PBS job. |
source-code/parameter-optimization/JuliaProcesses/julia_gcc.pbs |
Removes GCC PBS job. |
source-code/parameter-optimization/JuliaJobs/README.md |
Removes legacy documentation. |
source-code/parameter-optimization/JuliaJobs/Makefile |
Removes legacy build setup. |
source-code/parameter-optimization/JuliaJobs/julia.pbs |
Removes legacy PBS job. |
source-code/parameter-optimization/JuliaJobs/julia_optimize.py |
Removes job-based optimizer. |
source-code/parameter-optimization/JuliaJobs/julia_omp.cpp |
Removes Julia-set workload. |
source-code/parameter-optimization/hyperopt_imize.py |
Removes Hyperopt example. |
source-code/parameter-optimization/continuous_process.py |
Removes legacy objective script. |
source-code/keras/README.md |
Documents Keras environments. |
source-code/keras/environment_keras.yml |
Adds CPU Keras environment. |
source-code/keras/environment_keras_gpu.yml |
Adds GPU Keras environment. |
README.md |
Links the multi-GPU presentation. |
environment.yml |
Switches optimization stack to Optuna. |
environment_gpu.yml |
Updates GPU teaching dependencies. |
.gitignore |
Ignores presentation lock files. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+479
to
+483
| if rank_snapshot is not None and rank_snapshot.exists(): | ||
| if not arguments.overwrite_snapshot: | ||
| raise FileExistsError( | ||
| f"snapshot already exists: {rank_snapshot}; enable overwrite explicitly" | ||
| ) |
Comment on lines
+166
to
+167
| if arguments.snapshot is not None: | ||
| validate_new_file(parser, arguments.snapshot, arguments.overwrite_snapshot) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.