Mixture of Experts Explained: From Vowel Recognition to Trillion-Parameter LLMs (Llama 4, DeepSeek) скачать в хорошем качестве

Mixture of Experts Explained: From Vowel Recognition to Trillion-Parameter LLMs (Llama 4, DeepSeek)

Mixture of Experts (MoE) is everywhere: Meta / Llama 4, DeepSeek, Mistral. But how does it actually work? Do experts specialize? Why does this design scale better than dense models? In this video, we go deep: 🔹 Walk through the full history of MoE—from vowel recognition in 1991 to trillion-parameter models 🔹 Reproduce the original paper live in Colab 🔹 Dissect modern architectures like Switch Transformer, DeepSeek-MoE, and Mixtral 🔹 Explain why sparsity works, how gating networks operate, and whether experts actually specialize 🔹 Explore training tricks like noise injection and load balancing 🔹 Discuss expert specialization. Whether you’re an ML researcher, engineer, or just LLM-curious—you'll find value in this video. 🧠 Free resources (slides, reading list, Colab) are available on my Patreon for free 👉   / juliaturc   00:00 Intro & Motivation 01:00 The Scaling Problem 01:49 The Original MoE Paper (1991) 03:43 Colab Repro of Original Paper 09:54 Sparse MoE Revival (2017) 16:03 Switch Transformer & K=1 (2019) 20:28 Modern Open-Source MoEs (Mixtral, DeepSeek, LLaMA 4) 23:02 Do experts specialize? 25:41 Parallelization