Understanding Mixture of Experts скачать в хорошем качестве

Understanding Mixture of Experts

For ADVANCED fine-tuning scripts, function-calling Llama 2 and more... check out Trelis.com Slides: https://tinyurl.com/mtm54unn Binary-tree/FFF paper: https://arxiv.org/pdf/2308.14711.pdf MoE papers: https://arxiv.org/pdf/2208.02813.pdf ; https://arxiv.org/pdf/1701.06538.pdf Reddit thread: https://tinyurl.com/ythsu2nd Chapters 0:00 GPT-3, GPT-4 and Mixture of Experts 0:55 Why Mixture of Experts? 2:35 The idea behind Mixture of Experts 3:59 How to train MoE 5:41 Problems training MoE 7:54 Adding noise during training 9:06 Adjusting the loss function for router evenness 10:56 Is MoE useful for LLMs on laptops? 12:37 How might MoE help big companies like OpenAI? 14:22 Disadvantages of MoE 15:42 Binary tree MoE (fast feed forward) 18:15 Data on GPT vs MoE vs FFF 21:55 Inference speed up with binary tree MoE 23:48 Recap - Does MoE make sense? 25:05 Why might big companies use MoE?