Pierre Clavier - ShiQ: Bringing back Bellman to LLMs скачать в хорошем качестве

Pierre Clavier - ShiQ: Bringing back Bellman to LLMs

The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM, seen as an initial policy. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness comes from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLMs. However, naively applying a Q-learning-style update to the model's logits is ineffective due to the specificity of LLMs. Our core contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we carefully adapt insights from the RL literature to account for LLM-specific characteristics, ensuring that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally, we evaluate ShiQ on both synthetic data and real-world benchmarks, e.g., UltraFeedback and BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings Pierre Clavier is a Research Scientist at Cohere, where he works on Reinforcement Learning applied to Large Language Models. Before joining Cohere, he was a PhD candidate in Machine Learning at CMAP in École Polytechnique, under the supervision of Stéphanie Allassonnière and Erwan Le Pennec, and in close collaboration with Matthieu Geist. He was also part of the HeKA team at Inria Paris. He holds a Master’s degree in Mathematics and Machine Learning from the MVA program at ENS Paris-Saclay. From January to March 2024, he was a visiting researcher at Caltech in the Computing + Mathematical Sciences Department, where he was supervised by Adam Wierman and Eric Mazumdar. This session is brought to you by the Cohere Labs Open Science Community - a space where ML researchers, engineers, linguists, social scientists, and lifelong learners connect and collaborate with each other. We'd like to extend a special thank you to Anier Velasco Sotomayor, Thang Chu, and Andrej Jovanović, Leads of our ML Theory group for their dedication in organizing this event. If you’re interested in sharing your work, we welcome you to join us! Simply fill out the form at https://forms.gle/ALND9i6KouEEpCnz6 to express your interest in becoming a speaker. Join the Cohere Labs Open Science Community to see a full list of upcoming events (https://tinyurl.com/CohereLabsCommuni....