Implementing the Adam Optimiser By Hand In JAX // Hi, JAX! Act I // Lecture 05 скачать в хорошем качестве

Implementing the Adam Optimiser By Hand In JAX // Hi, JAX! Act I // Lecture 05

Welcome to the fifth and final lecture for Act I of "Hi, JAX!", an introduction to vanilla JAX for deep learning research! In this lecture we'll build on our efforts from last time and enhance our SimpLeNet training code. Instead of using stateless stochastic gradient descent, we'll implement the famous adaptive moment estimating optimiser "Adam". This will be a good chance to practice navigating stateful computation without relying on mutability and to reinforce other topics we have learned in earlier lectures such as jax.tree.map. And in the end, we should see a nice accuracy boost from using a more efficient optimiser. Links: Course webpage: https://github.com/matomatical/hijax Course playlist:    • Hi, JAX! Introduction to vanilla JAX for d...   Chapters: 00:00 Introduction 02:34 Copying code from last time 04:11 The plan 07:20 Defining optimiser state 12:06 Initialising the state 16:41 Designing the update API 19:31 Compute the new optimiser state 24:52 Compute the model update 30:12 Training with the new optimiser 31:31 Summing up 32:17 Challenge