Jim Fan from Nvidia AI Building foundation model for robots! скачать в хорошем качестве

Jim Fan from Nvidia AI Building foundation model for robots!

Join Dr. Jim Fan as he delves into the fascinating world of embodied AI! At the San Francisco GenAI Summit on May 29, Jim shared beliefs regarding future trends and covered the latest advancements from Nvidia including MineDojo, Voyager, MetaMorph, and Dr. Eureka. MineCLIP uses reinforcement learning from human feedback (RLHF) to train agents in the Minecraft game environment using video and text data. Voyager is an agent that can explore Minecraft for hours, generating code snippets as skills using - GPT-4 and refining them through trial and error. MetaMorph is a universal policy model that can control thousands of different robot morphologies by tokenizing their kinematic structures. Eureka utilizes GPT-4 to automatically generate reward functions for reinforcement learning in physics simulations like IsaacSim, sometimes outperforming human experts. Domain randomization allows simulation-trained policies to be transferred to real robots by randomizing parameters during training. Nvidia aims to create a "foundation agent" that can generalize across different embodiments using language instructions, similar to how ChatGPT generalizes across text. Project Groot focuses on developing a foundation model specifically for humanoid robots, leveraging decreasing hardware costs and simulation at scale. Chapters: 0:00 MineDojo 2:14 Voyager 4:14 MetaMorph 6:38 Isaac Sim 7:14 Eureka 9:30 Dr. Eureka 11:02 Insights 14:14 F&A Note: this video is a condensed version of the original 40-minute talk. #nvidiaai #robotics