Building a robust data pipeline with the dAG stack dbt, Airflow, Great Expectations скачать в хорошем качестве

Building a robust data pipeline with the dAG stack dbt, Airflow, Great Expectations

Presented by Sam Bail at Airflow Summit 2021. Data quality has become a much discussed topic in the fields of data engineering and data science, and it has become clear that data validation is absolutely crucial to ensuring the reliability of any data products and insights produced by an organization’s data pipelines. This session will outline patterns for combining three popular open source tools in the data ecosystem - dbt, Airflow, and Great Expectations - and use them to build a robust data pipeline with data validation at each critical step. 0:00 Welcome 3:20 Quick review of dbt 6:32 Overview of Great Expectations 14:00 Integrating dbt and Airflow 27:50 Testing with dbt and Great Expectations 41:40 Wrap-up