Cengiz Pehlevan: Toward a Theory of Neural Scaling Laws скачать в хорошем качестве

Cengiz Pehlevan: Toward a Theory of Neural Scaling Laws

The Talk given by Cengiz Pehlevan KUIS AI Talks on December 30,2025 𝐓𝐢𝐭𝐥𝐞: Toward a Theory of Neural Scaling Laws 𝐀𝐛𝐬𝐭𝐫𝐚𝐜𝐭:Neural scaling laws, which describe power-law relationships between performance and resources such as model size, data, and compute, have become central to recent progress in AI. Yet we still lack a clear theoretical understanding of what determines these exponents and when such laws should hold. In this talk I will begin with our past work on neural scaling laws in simplified but analytically tractable settings, highlighting the role of data structure, optimization dynamics, and model architecture. While these theories are powerful, they do not account for phenomena observed in transformers, such as in-context learning. I will then present a new theory of scaling laws for in-context regression in transformers, characterizing how performance depends on computational and statistical resources such as width, depth, number of training steps, batch size, and data per context. Together, these results aim to clarify when and why scaling laws arise and to provide principled guidance for the design and training of large transformer-based models. Short Bio: Cengiz Pehlevan is an Assistant Professor of Applied Mathematics at Harvard University and an Associate Faculty Member at the Kempner Institute. His research develops mathematical theory for learning in biological and artificial neural networks. He is a recipient of a Sloan Research Fellowship in Neuroscience, an NSF CAREER Award, and a Google Faculty Research Award. Previously, he held research positions at the Flatiron Institute’s Center for Computational Biology and Janelia Research Campus, and was a Swartz Fellow at Harvard University. He holds a Ph.D. in physics from Brown University.