Vincent Leroy: From CroCo to MASt3R: A Paradigm Change in 3D Vision скачать в хорошем качестве

Vincent Leroy: From CroCo to MASt3R: A Paradigm Change in 3D Vision

Abstract: In the rapidly evolving field of 3D vision, a multitude of heterogeneous downstream tasks coexist, such as visual localization, depth estimation, 3d reconstruction, etc. The current paradigm is to develop a dedicated method to solve each task, thereby largely ignoring their potential inter-connections and synergies. Developing unified models able to handle multiple 3D geometric downstream tasks remains a challenge. This presentation introduces three interconnected advancements in this respect: CroCo, DUSt3R and MASt3R. CroCo, a self-supervised pre-training framework, utilizes a pretext task to lay foundations for DUSt3R/MASt3R, a unified foundational model for geometric 3D vision. Cross-view completion, or CroCo in short, first serves to learn robust representations of 3D geometry from pairs of images depicting the same scene from different viewpoints. By masking parts of an image and predicting these given another viewpoint of the scene, CroCo effectively captures priors about spatial relationships and geometric information, setting a strong foundation for downstream 3D vision tasks. Then, building on the robust pre-trained models provided by CroCo, DUSt3R introduces a novel approach for Dense Unconstrained Stereo 3D Reconstruction. This method revolutionizes traditional multi-view stereo reconstruction by regressing pointmaps that encode scene geometry without requiring calibrated or posed cameras. DUSt3R simplifies the complex pipeline of traditional 3D reconstruction methods, significantly reducing computational overhead and enhancing performance across various benchmarks. MASt3R further extends DUSt3R by adding the ability to establish accurate pixel correspondences. The journey from CroCo to MASt3R exemplify a significant paradigm shift in 3D vision technologies. This presentation will delve into the methodologies, innovations, and synergistic integration of these frameworks, demonstrating their impact on the field and potential future directions. The discussion aims to highlight how these advancements unify and streamline the processing of 3D visual data, offering new perspectives and capabilities in robotic navigation, cultural heritage preservation, and beyond. Bio: I'm a research scientist in Geometric Deep Learning at Naver Labs Europe. I joined 5 years ago, in 2019, after completing my PhD on Multi-View Stereo Reconstruction for dynamic shapes at the INRIA Grenoble-Alpes under the supervision of E.Boyer and J-S. Franco. Other than that, all you need to know is I am a simple man: I like hiking in the mountains and finding simple solutions to complex problems. Interestingly, the latter usually comes with the former.