Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models скачать в хорошем качестве

Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models

Traditional multimodal learners find unified representations for tasks like visual question answering, but rely heavily on paired datasets. However, an overlooked yet potentially powerful question is: can one leverage auxiliary unpaired multimodal data to directly enhance representation learning in a target modality? We introduce UML: Unpaired Multimodal Learner, a modality-agnostic training paradigm in which a single model alternately processes inputs from different modalities while sharing parameters across them. This design exploits the assumption that different modalities are projections of a shared underlying reality, allowing the model to benefit from cross-modal structure without requiring explicit pairs. Theoretically, under linear data-generating assumptions, we show that unpaired auxiliary data can yield representations strictly more informative about the data-generating process than unimodal training. Empirically, we show that using unpaired data from auxiliary modalities---such as text, audio, or images---consistently improves downstream performance across diverse unimodal targets such as image and audio. Resources Project page: https://unpaired-multimodal.github.io/ About the Speaker Sharut Gupta is a fourth-year Ph.D student at MIT CSAIL, advised by Prof. Phillip Isola and Prof. Stefanie Jegelka. Prior to this, she completed her undergraduate studies in Mathematics and Computing at the Indian Institute of Technology, Delhi (IIT Delhi), during which she worked with Prof. Yoshua Bengio on her thesis. She has also spent time at Meta SuperIntelligence Labs (Meta AI), and Google DeepMind.