Choosing the Learning Rate with LR Finder скачать в хорошем качестве

Choosing the Learning Rate with LR Finder

Setting the learning rate for stochastic gradient descent (SGD) is crucially important when training neural network because it controls both the speed of convergence and the ultimate performance of the network. Set the learning too low and you could be twiddling your thumbs for quite some time as the parameters update very slowly. Set it too high and the updates will skip over optimal solutions, or worse the optimizer might not converge at all! Leslie Smith from the U.S. Naval Research Laboratory presented a method for finding a good learning rate in a paper called "Cyclical Learning Rates for Training Neural Networks" (https://arxiv.org/abs/1506.01186). In this video we implement a 'Learning Rate Finder' in MXNet with the Gluon API that you can use while training your own networks. Code and more information can be found at https://mxnet.incubator.apache.org/tu... Intro Music: Candy-Coloured Sky by Catmosphere