NLP Series III: Text Retrieval Embeddings, E5. Algorithm and code read. скачать в хорошем качестве

NLP Series III: Text Retrieval Embeddings, E5. Algorithm and code read.

This video is part of a larger series about text processing models:    • NLP Series   This video talks about foundational principles behind pretraining of modern text retrieval embeddings, such as Contrastive Loss, Negatives and Hard Negative Mining, and Knowledge Distillation. It talks about specific model for retrieval embeddings - E5 - the first one that managed to beat bm25 baseline. Second part of the video goes into the pretraining source code of the microsoft/unilm repo. Important links: E5 paper: https://arxiv.org/pdf/2212.03533 Source code: https://github.com/microsoft/unilm/tr... Paper about temperature and contrastive loss visualisation: https://arxiv.org/pdf/2012.09740 Installation instructions: https://github.com/adensur/blog/blob/... Code shown in the video & presentation: https://github.com/adensur/blog/tree/... 00:00 - Intro 01:00 - What are retrieval embeddings 02:28 - E5 paper 03:23 - MTEB leaderboard 05:46 - Transformers taxonomy 10:26 - E5 is BERT 11:21 - Bert Architecture overview 14:33 - Cross Encoders and Bi-Encoders 17:08 - Contrastive Loss 20:50 - Temperature and its visualization 22:34 - Negatives and Hard Negatives 27:58 - Knowledge Distillation 30:37 - Code Reading: Setup and intro 35:21 - Dataset and dataloader 43:34 - Forward Pass of the model 48:50 - In Batch Negatives, Contrastive Loss 54:40 - Knowledge Distillation