MedAI #56: Fundamentals of Multimodal Representation Learning | Paul Pu Liang скачать в хорошем качестве

MedAI #56: Fundamentals of Multimodal Representation Learning | Paul Pu Liang

Title: Fundamentals of Multimodal Representation Learning - Towards Generalization and Quantification Speaker: Paul Pu Liang Abstract: In recent years, the quest for artificial intelligence capable of digital, physical, and social intelligence has led to an explosion of interest in multimodal datasets and algorithms. This research area of multimodal machine learning studies the computational and theoretical foundations of learning from heterogeneous data sources. This talk studies two core challenges in multimodal learning- (1) constructing multimodal models and datasets that enable generalization across many modalities and different tasks, and (2) designing quantification methods to comprehensively understand the internal mechanics of multimodal representations and gain insights for safe real-world deployment. In the first part, we study generalization in multimodal learning. Generalization is particularly beneficial when one modality has limited resources such as the lack of annotated data, noisy inputs, or unreliable labels, and presents a step towards processing many diverse and understudied modalities. To enable the study of generalization, we introduce MultiBench, a unified large-scale benchmark across a wide range of modalities, tasks, and research areas. Using MultiBench, we study generalization with parallel modalities, as well as in non-parallel scenarios, where we are presented with many modalities, but each task is defined only over a small subset of them. The second part studies quantification of multimodal models via MultiViz, our recent attempt at a framework to understand the internal modeling of multimodal information and cross-modal interactions. We conclude this talk by discussing how future work can leverage these ideas to drive progress towards more general, scalable, and explainable multimodal models. Speaker Bio: Paul Liang is a Ph.D. student in Machine Learning at CMU, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. His research lies in the foundations of multimodal machine learning with applications in socially intelligent AI, understanding human and machine intelligence, natural language processing, healthcare, and education. His research is generously supported by a Facebook PhD Fellowship and a Center for Machine Learning and Health Fellowship, and has been recognized by awards at the NeurIPS 2019 workshop on federated learning and ICMI 2017. He regularly organizes courses, workshops, and tutorials on multimodal learning and was a workflow chair for ICML 2019. Website at https://www.cs.cmu.edu/~pliang/ ------ The MedAI Group Exchange Sessions are a platform where we can critically examine key topics in AI and medicine, generate fresh ideas and discussion around their intersection and most importantly, learn from each other. We will be having weekly sessions where invited speakers will give a talk presenting their work followed by an interactive discussion and Q&A. Our sessions are held every Thursday from 1pm-2pm PST. To get notifications about upcoming sessions, please join our mailing list: https://mailman.stanford.edu/mailman/... For more details about MedAI, check out our website: https://medai.stanford.edu. You can follow us on Twitter @MedaiStanford Organized by members of the Rubin Lab (http://rubinlab.stanford.edu) Nandita Bhaskhar (https://www.stanford.edu/~nanbhas) Siyi Tang (https://siyitang.me)