SANE 2022 | Chuang Gan - Learning to Perceive Physical Scenes from Multi-Sensory Data скачать в хорошем качестве

SANE 2022 | Chuang Gan - Learning to Perceive Physical Scenes from Multi-Sensory Data

Chuang Gan, assistant professor at UMass Amherst and research manager at MIT-IBM Watson AI Lab in Cambridge, MA, presents his work on learning to perceive physical scenes from multi-sensory data at the SANE 2022 workshop in Kendall Square, October 6, 2022. More info on the SANE workshop series: http://www.saneworkshop.org/ Abstract: Human sensory perception of the physical world is rich and multimodal and can flexibly integrate input from all five sensory modalities -- vision, touch, smell, hearing, and taste. However, in AI, attention has primarily focused on visual perception. In this talk, I will introduce my efforts in connecting vision with sound, allowing machine perception systems to see objects and infer physics from multi-sensory data. In the first part of my talk, I will discuss a self-supervised approach that could learn to parse images and separate the sound sources by watching and listening to unlabeled videos without requiring additional manual supervision. In the second part of my talk, I will introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene and infer the underlying causal structure in 3D environments through visual and auditory observations. Finally, I will talk about how we may further build embodied agents to seek the sound source of repeating environmental sound (e.g., alarm) or identify what object has fallen and where from an intermittent impact sound. Bio: Chuang Gan is an assistant professor at UMass Amherst and research manager at MIT-IBM Watson AI Lab. Before that, he was a researcher at MIT, working with Prof. Antonio Torralba and Prof. Josh Tenenbaum. He completed his Ph.D. with the highest honor at Tsinghua University, supervised by Prof. Andrew Chi-Chih Yao. His research interests sit at the intersection of computer vision, machine learning, and cognitive science. His research works have been recognized by Microsoft Fellowship, Baidu Fellowship, and media coverage from BBC, WIRED, Forbes, and MIT Tech Review. He has been an area chair of CVPR, ICCV, ECCV, ICML, ICLR, NeurIPS, and ACL and an associate editor of IEEE Transactions on Image Processing.