Heavy-tailed Noise & Stochastic Gradient Descent - Wesley Erickson - PIBBSS Symposium скачать в хорошем качестве

Heavy-tailed Noise & Stochastic Gradient Descent - Wesley Erickson - PIBBSS Symposium

This video was recorded during the 2024 PIBBSS Symposium. Read more about it on our website: https://pibbss.ai/symposium-24/ About the talk: Heavy-tailed distributions describe systems where extreme events dominate the dynamics. These distributions are prominent in the gradient noise of Stochastic Gradient Descent, and have strong connections to the ability for SGD to generalize. Despite this, it is common in machine learning to assume Normal/Gaussian dynamics, which may impact safety by underestimating the possibility of rare events or sudden changes in behavior or ability. In this talk I explore simulations of ML systems exhibiting heavy-tailed noise, discuss how this noise arises, and how its characteristics may act as a useful statistical signature for monitoring and interpreting ML systems.