Self-Supervised Learning Collapse: a Quick CIFAR-10 Experiment скачать в хорошем качестве

Self-Supervised Learning Collapse: a Quick CIFAR-10 Experiment

I ran a simple experiment to understand collapse in self-supervised learning using. I tried solutions like stop-gradient and parameter copying. I also plotted the singular values of the covariance matrix at the projection layer. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 🔗 notes: https://mashaan14.github.io/YouTube-c... ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ website https://mashaan14.github.io/mashaan/ github https://github.com/mashaan14 X   / mashaan_14   linkedin   / mashaan   ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ sponsor this channel on GitHub Sponsors: https://github.com/sponsors/mashaan14 ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Chapters: 00:00 start 00:13 what is SSL collapse? 02:10 importing CIFAR-10 02:51 ViT architecture 04:02 training loop 04:45 COLLAPSE!!! 05:11 stopping the gradient on teacher network 06:26 copying the parameters to the teacher network 07:20 how to prevent collapse 07:57 plotting the singular values ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ References: Chen, Xinlei, and Kaiming He. "Exploring simple siamese representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. Jing, Li, et al. "Understanding dimensional collapse in contrastive self-supervised learning." arXiv preprint arXiv:2110.09348 (2021). Wang, Xiao, et al. "On the importance of asymmetry for siamese representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #Collapse #SelfSupervisedLerning #ViT #VisionTransformer #ContrastiveLearning #IJEPA #JointEmbedding #DINO #iBOT #SimCLR #BYOL #SwAV #Dimensional #PyTorch #DeepLearningTutorial #MachineLearningProject #AIResearch #CodingTutorial