Deep Dive - Cooking with Gas: Building a Data Platform at US Foods скачать в хорошем качестве

Deep Dive - Cooking with Gas: Building a Data Platform at US Foods

Join Dagster Developer Advocate Alex Noonan and US Foods Lead Machine Learning Engineer Lee Littlejohn for an in-depth discussion on how US Foods successfully built a centralized data platform using Dagster. Discover the core principles, tool selection, implementation strategies, and challenges encountered during the transformation from a chaotic to a structured data ecosystem. Gain valuable insights into the modular design, dependency management, containerization, and environment-specific configurations. This deep dive is an essential watch for data engineers and platform owners aiming to streamline their data operations and enhance efficiency. 00:00 Introduction and Overview 00:58 Meet Lee Littlejohn and US Foods 01:19 Drawing Parallels Between Kitchens and Data Platforms 02:43 Core Principles of Platform Redesign 05:47 Challenges Faced in the Data Ecosystem 08:09 Tool Selection and Dagster's Role 10:32 Dependency Management with Poetry 12:33 Containerization Strategy 13:34 AWS CDK for Cloud Resource Management 14:53 Automating Deployments with GitHub Actions 16:13 Code Organization and Team Structure 17:58 Environment-Specific Configurations 20:13 Snowflake Data Organization 20:24 Modular Mapping and Access Control 20:59 Zero Copy Cloning and Environmental Refresh 21:40 Integrations with DBT and SageMaker 23:06 End-to-End Workflow Example 24:05 Benefits of the New System 24:44 Adoption and Cultural Shift 27:06 Q&A Session