Flash Attention 2.0 with Tri Dao (author)! | Discord server talks скачать в хорошем качестве

Flash Attention 2.0 with Tri Dao (author)! | Discord server talks

❤️ Become The AI Epiphany Patreon ❤️   / theaiepiphany   👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦   / discord   Hey guys! I've been uploading less frequently lately since I started my startup - as I'm trying to focus and learn as much as possible. Until I get back into the rhythm I'll be experimenting a bit with the channel: expect talks from my Discord server, live streaming, etc. Here is a talk we had in my Discord server with Tri Dao from Stanford, talking about his recent work on Flash Attention 2.0! ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ My Flash Attention blog:   / eli5-flash-attention   ✅ Horace's blog: https://horace.io/brrr_intro.html ✅ Flash Attention paper: https://arxiv.org/abs/2205.14135 ✅ Flash Attention 2.0 paper: https://tridao.me/publications/flash2... ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00 Main talk starts - intro & motivation 10:20 Behind the scenes: how Tri got started with Flash Attention 16:40 Motivation: modelling long sequences 19:50 Brief recap of attention 23:15 Memory bottleneck, IO awareness 35:10 Flash Attention 2.0 improvements 37:20 Behind the scenes of Flash Attention 2.0 (refactor of CUTLASS 3) 43:45 Future directions 46:45 Q&A ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany -   / theaiepiphany   One-time donation - https://www.paypal.com/paypalme/theai... Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn -   / aleksagordic   🐦 Twitter -   / gordic_aleksa   👨‍👩‍👧‍👦 Discord -   / discord   📺 YouTube -    / theaiepiphany   📚 Medium -   / gordicaleksa   💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #flashattention #tridao #mlsys