Data Platform Week - Declarative Automation with Owen Kephart скачать в хорошем качестве

Data Platform Week - Declarative Automation with Owen Kephart

Revolutionizing Data Workflows with Dagster's Declarative Automation Join Owen Kephart, a software engineer at Dagster, as he introduces Dagster’s new declarative automation system designed to optimize data updates across your data platform. Learn how this system builds on previous policies to offer a more flexible, predictable, and user-friendly solution to data automation challenges. Owen discusses the limitations of traditional methods like cron schedules and cross-team job groupings and showcases how Dagster's intelligent, event-based orchestration and partitioned assets offer a superior alternative. Discover how this system simplifies managing data dependencies, improves data quality monitoring, and reduces organizational costs, making it a game-changer for data teams. 00:00 Introduction to Dagster's New Declarative Automation System 00:25 Challenges with Traditional Automation Methods 03:17 Dagster's Declarative Automation Solution 04:11 Example: Processing Raw Data with Dagster 06:32 Advanced Automation Features 08:35 Managing Partitioned Assets 10:22 Data Quality Management with Declarative Automation 12:10 Conclusion and Future Outlook