Lecture 28: Liger Kernel - Efficient Triton Kernels for LLM Training скачать в хорошем качестве

Lecture 28: Liger Kernel - Efficient Triton Kernels for LLM Training

Byron Hsu presents LinkedIn's open-source collection of Triton kernels for efficient LLM training. TIMESTAMPS 00:00 Host Opening 00:22 Main Focus 01:18 Outline 03:03 LLM Training Bottleneck 05:27 Live Demo - PyTorch Profiler 10:41 Why Triton 12:53 QA 13:49 Example - RMS Norm 18:00 QA 20:03 RMS Norm Tricks 21:20 Live Code - RMS Nrom 25:40 QA 28:14 Example - Fused Linear Cross Entropy 30:58 Gradient Checkpointing 31:51 Gradient-in-forward 32:53 QA 35:01 Chunking 36:23 QA 37:56 Live Code - Fused Linear Cross Entropy 39:59 QA 41:15 Convergence Test 42:39 Live Code - Convergence Test 44:12 Contiguity 45:15 Live Code - Contiguity 48:01 QA 49:45 Memory Address 50:38 Live Code - Memory Address 52:11 QA 1:00:28 QA - Liger Kernel 1:09:12 Acknowledgement Slides: https://docs.google.com/presentation/... Notebooks: https://github.com/cuda-mode/lectures...