The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert скачать в хорошем качестве

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

The origins of Reinforcement Learning from Human Feedback, RLHF, sociology's influence on it, the tension between human vs synthetic data, and emerging research in the field. Full notes and writeup: https://www.latent.space/p/rlhf-201 Timestamps [00:00:00] Introductions and background on the lecture origins [00:05:17] History of RL and its applications [00:10:09] Intellectual history of RLHF [00:13:47] RLHF for decision-making and pre-deep RL vs deep RL [00:20:19] Initial papers and intuitions around RLHF [00:27:57] The three phases of RLHF [00:31:09] Overfitting issues [00:34:47] How preferences get defined [00:40:35] Ballpark on LLaMA2 costs [00:42:50] Synthetic data for training [00:47:25] Technical deep dive in the RLHF process [00:54:34] Projection / best event sampling [00:57:49] Constitutional AI [01:04:13] DPO [01:08:54] What's the Allen Institute for AI? [01:13:43] Benchmarks and models comparisons