Скачать с ютуб How FlashAttention Accelerates Generative AI Revolution в хорошем качестве

How FlashAttention Accelerates Generative AI Revolution

FlashAttention is an IO-aware algorithm for computing attention used in Transformers. It's fast, memory-efficient, and exact. It has become a standard tool for speeding up LLM training and inference. Join me and learn how FlashAttention works! References: [OnlineSoftmax] https://arxiv.org/abs/1805.02867 [From Online Softmax to FlashAttention] https://courses.cs.washington.edu/cou... [FlashAttention] https://arxiv.org/abs/2205.14135 [FlashAttention-2] https://arxiv.org/abs/2307.08691 [FlashAttention-3] https://arxiv.org/abs/2407.08608 Chapters: 00:00 - Prior Attempts for Speeding Up Attention 01:01 - Why is Self-Attention Slow? 03:35 - IO-aware Algorithm - Tiling 05:38 - Safe Softmax 07:05 - Online Softmax 08:24 - FlashAttention