Let's pretrain a 3B LLM from scratch: on 16+ H100 GPUs, no detail skipped. скачать в хорошем качестве

Let's pretrain a 3B LLM from scratch: on 16+ H100 GPUs, no detail skipped.

We learn to pretrain a 3B parameter LLM across multiple H100 machines from scratch skipping no details. Learn to handle OOM errors, how to develop on cheap GPUs before scaling to multi-GPU. Finally, we end with running multinode with FSDP and explain how to take the model beyond 3B params. This is a full lecture with no edits or details skipped. At the end of this lecture you will improve your set of skills and intuition needed for pretraining and scaling LLMs beyond a simple demo. We start tuning and developing on cheap A10G GPUs. Then we run on 8 H100 GPUs and finally scale it to 2 machines, for a total of 16 H100 GPUs. This workflow saves a ton in cloud costs. I start at 1B parameters and scale it to 3B. To go beyond 3B, simply use the same process but with more machines. Chapters: 00:00 Introduction 01:40 Run the Llama template 02:19 Llama template overview 05:00 Run the template on 1 GPU (A10G) 06:20 Monitor GPU memory usage 06:40 Code walkthrough 10:30 How to handle OOM (out of memory) errors 13:20 Connect local VSCode (optional) 14:40 Overview of hyperparameters 15:50 Run a hyperparameter sweep to find the context window 24:50 Speed up by 2x on 4 GPUs (A10G) 29:40 VRAM vs power for profiling 33:07 From 1B to 3B parameters 37:00 How to release ghost GPU memory 42:00 Change to machine with 8 x H100 GPUs 42:20 Number of parameters vs data size 45:00 Hyperparameter sweep results 48:00 3B params on the H100 at 4x speed 54:40 Troubleshoot Tensorboard error 58:40 TensorBoard and artifacts on separate Studio for analysis 1:02:00 Measure cloud costs spent so far 1:05:00 Discuss and view data concerns 1:10:20 Getting to steady state 1:10:50 How to increase speed for the 3B parameter model 1:16:00 How to run DeepSpeed, FSDP and other scaling techniques 1:20:00 Start training with multi-node (multiple machines) 1:28:00 Monitor multi-node training 1:29:00 Summary