Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum (Feb 2026) скачать в хорошем качестве

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum (Feb 2026)

Title: Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum (Feb 2026) Link: http://arxiv.org/abs/2602.17080v2 Date: February 2026 Summary: This paper introduces NAMO and NAMO-D, two new optimization algorithms that integrate orthogonalized momentum with Adam-style adaptive noise adaptation. NAMO uses a single adaptive scalar to preserve update orthogonality, while NAMO-D employs neuron-wise scaling with clamping to align with near block-diagonal Hessian structures. The authors establish optimal convergence rates for both methods and demonstrate superior performance over AdamW and Muon baselines in pretraining GPT-2 models. Key Topics: Stochastic Optimization Orthogonalized Momentum Adaptive Learning Rates Large Language Models Convergence Analysis Deep Learning Optimizers Chapters: 00:00 - Podcast Introduction 01:32 - Comparing Optimizer Philosophies 02:53 - Solving Muon Stochastic Instability 03:43 - NAMO Global Scalar Scaling 05:13 - NEMOD Diagonal Matrix Update 06:45 - Clamping Parameter Dynamics 08:02 - Establishing Convergence Proofs 09:23 - Benchmarking GPT-2 Performance 11:07 - Testing Learning Rate Robustness 12:36 - Iterative Newton Schultz Efficiency 13:36 - Adaptive Weight Decay Implementation 15:00 - Engineering Deployment Strategy 16:25 - Optimizer Summary and Conclusion Stock video credits: Google DeepMind - https://www.pexels.com/@googledeepmind olia danilevich - https://www.pexels.com/@olia-danilevich fauxels - https://www.pexels.com/@fauxels Pressmaster - https://www.pexels.com/@pressmaster StefWithAnF - https://www.pexels.com/@stefwithanf-1... Soumya - https://www.pexels.com/@soumya-1446957 Nino Souza - https://www.pexels.com/@ninosouza Anete Lusina - https://www.pexels.com/@anete-lusina Silviu Din - https://www.pexels.com/@silviu-din-16... Colors Motion Graphics - https://www.pexels.com/@colors-motion... Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulins... cottonbro studio - https://www.pexels.com/@cottonbro Claudiu Ciobanu - https://www.pexels.com/@claudiuciobanu tunnel motions - https://www.pexels.com/@tunnelmotions @svetjekolem - https://www.pexels.com/@svetjekolem Pachon in Motion - https://www.pexels.com/@pachon-in-mot... Colin Jones - https://www.pexels.com/@larchmedia José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro Pixabay - https://www.pexels.com/@pixabay Ron Lach - https://www.pexels.com/@ron-lach Max Fischer - https://www.pexels.com/@max-fischer Adis Resic - https://www.pexels.com/@adis-resic-29... Engin Akyurt - https://www.pexels.com/@enginakyurt Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk Kindel Media - https://www.pexels.com/@kindelmedia Caleb Oquendo - https://www.pexels.com/@caleboquendo Mikhail Nilov - https://www.pexels.com/@mikhail-nilov Stas Knop - https://www.pexels.com/@stasknop Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev Tom Fisk - https://www.pexels.com/@tomfisk Ketut Subiyanto - https://www.pexels.com/@ketut-subiyanto Stefanie Jockschat - https://www.pexels.com/@stefaniejocks... Tiger Lily - https://www.pexels.com/@tiger-lily