This Simple Optimizer Is Revolutionizing How We Train AI [Muon] скачать в хорошем качестве

This Simple Optimizer Is Revolutionizing How We Train AI [Muon]

The Muon optimizer has demonstrated remarkable performance in accelerating machine learning model training, often outperforming the widely used AdamW optimizer. In this video, we will cover the basic concept of how Muon works and discuss some recent improvements that make it scalable for large-scale LLM training. 00:00 Why Muon? 00:36 Reviewing Adam 02:13 Linear layer 04:24 Solving orthogonalization with SVD 06:28 Newton-Schulz iteration - Odd polynomial matrix 08:11 Newton-Schulz iteration - Example 10:35 The Muon optimizer 11:49 The exploding attention logit crisis 15:13 MuonClip: Extending QK-clip to Multi-head Latent Attention (MLA) 17:24 Results of MuonClip References: Muon: An optimizer for hidden layers in neural networks https://kellerjordan.github.io/posts/... Deriving Muon https://jeremybernste.in/writing/deri... Old Optimizer, New Norm: An Anthology https://arxiv.org/abs/2409.20325 Muon is Scalable for LLM Training: https://arxiv.org/abs/2502.16982 MuonClip: https://arxiv.org/abs/2507.20534 Fantastic pretraining optimizers and where to find them: https://arxiv.org/abs/2409.20325 Check out my other video to learn more about AdamW:    • The Algorithm that Helps Machines Learn [A...   Video made with Manim: https://www.manim.community/