Gumbel Softmax Quantization: Differentiable Discrete Sampling скачать в хорошем качестве

Gumbel Softmax Quantization: Differentiable Discrete Sampling

Code: https://github.com/priyammaz/PyTorch-... Gumbel Softmax Quantization is a differentiable alternative to the standard Vector Quantization we saw before. The reason this is important is we get to avoid that whole pairwise distance and Argmin operation we do for VQVAEs, and get clean gradients to update our codebook without any codebook/commitment loss! Today we start with the Gumbel-Max trick, derive its relation to Softmax, and then extend to the Gumbel Softmax from there! The derivation we do is pretty close to the example given by the Princeton CS group (https://lips.cs.princeton.edu/the-gum...) but I fill in some of the mathematical details they skipped! I hope you already have seen the following pre-reqs: VAE:    • Variational AutoEncoders (VAE) Implementation   VQVAE:    • Vector Quantized Variational AutoEncoder (...   Timestamps: 00:00:00 - Introduction 00:00:30 - Recap VQVAE 00:02:32 - Selecting Codes is a Sampling Problem 00:04:20 - What is the Gumbel Distribution? 00:09:08 - The Gumbel-Max Trick vs Multinomial Sampling 00:13:08 - Prove Gumbel-Max is Equivalent to Multinomial Sampling 00:39:24 - How to Sample from Gumbel Distribution? 00:43:24 - Simulation to Show Equivalence 00:44:50 - Moving to Gumbel Softmax 00:48:00 - Implementing Gumbel Softmax 00:50:00 - Effect of the Temperature Parameter Tau 00:52:36 - Implement One-Hot-Encoded Outputs (Hard Outputs) 00:58:03 - Implement Gumbel Softmax Quantizer 01:06:00 - Training Model with Annealed Tau 01:10:51 - Importance in Wav2Vec2 Socials! X   / data_adventurer   Instagram   / nixielights   Linkedin   / priyammaz   Discord   / discord   🚀 Github: https://github.com/priyammaz 🌐 Website: https://www.priyammazumdar.com/