Repository navigation

All pull requests

Synchronize Lookahead slow weights after each k fast steps
#571Afloat16 opened Nov 12, 2026
Accumulate Shampoo mode statistics before preconditioning
#568Afloat16 opened Nov 12, 2026
Preserve precision in NovoGrad gradient norm accumulation
#570Afloat16 opened Nov 12, 2026
Warm up Apollo groups toward their own learning rates
#569Afloat16 opened Nov 12, 2026
Support strided parameters in AdamP and SGDP projections
#567Afloat16 opened Nov 12, 2026
Preserve the SWATS momentum buffer during SGD updates
#566Afloat16 opened Nov 12, 2026
Initialize MADGRAD state lazily from each parameter group
#565Afloat16 opened Nov 12, 2026
Key RAdam cached step sizes by their hyperparameters
#564Afloat16 opened Nov 12, 2026
Preserve LAMB update options when resuming from checkpoints
#563Afloat16 opened Nov 12, 2026
Preserve AdaBound initial rates across added groups and checkpoints
#562Afloat16 opened Nov 12, 2026
Preserve the row-normalization axis in batched Adafactor
#561Afloat16 opened Oct 8, 2026
Remember the first PID gradient for the next derivative
#560shaneraphel opened Sep 25, 20261 task done
Keep a gradient second moment when AdaBelief beta1 is 0
#559shaneraphel opened Sep 25, 20263 tasks done
Drop kappa from the A2Grad examples
#558shaneraphel opened Sep 25, 20262 tasks done
Apply LAMB debias to the moments inside the trust ratio
#557shaneraphel opened Sep 25, 20263 tasks done
Estimate the Adahessian trace on both complex components
#556shaneraphel opened Sep 25, 20264 tasks done
Run AdaBound on the real and imaginary parts
#555shaneraphel opened Sep 25, 20263 tasks done
Scale SGDW parameters by (1 - lr * weight_decay)
#554shaneraphel opened Sep 25, 20261 task done
Add the Muon optimizer
#553shaneraphel opened Sep 25, 20263 of 4 tasks
Add the AdaNorm optimizer
#552shaneraphel opened Sep 25, 20263 of 4 tasks
Add the Aida optimizer
#551shaneraphel opened Sep 25, 20263 of 4 tasks
Add the Sophia optimizer
#550shaneraphel opened Sep 25, 20263 of 4 tasks
Make Adafactor relative-step constants configurable
#549shaneraphel opened Sep 25, 20261 of 2 tasks
Raise Shampoo preconditioners to -1/(2k) per the paper
#548shaneraphel opened Sep 25, 20262 of 3 tasks
Support Conv1d and Conv3d weights in Adahessian trace
#547shaneraphel opened Sep 25, 20262 tasks done

Pull requests