Migrating Airflow to Dagster with Airlift скачать в хорошем качестве

Migrating Airflow to Dagster with Airlift

Learn about Airlift, a new toolkit that simplifies migration by supporting the observation and federation of multiple Airflow instances within Dagster. Understand how Airlift facilitates the representation of Airflow DAGs as Dagster assets, provides rich metadata, and ensures data quality through asset checks. Explore how the Observe phase maps individual Airflow tasks to Dagster assets, and the final Migration phase proxies Airflow DAGs to invoke computation within Dagster. Airlift allows for incremental, zero-downtime migration, enabling teams to transition at their own pace while preserving Airflow DAG structures and history, ultimately building a more powerful data platform. 00:00 Introduction to Migrating from Airflow to Dagster 00:22 Introducing Airlift: A New Toolkit for Migration 00:32 Peering: Viewing Multiple Airflow Instances in Dagster 00:55 Observe: Mapping Airflow Tasks to Dagster Assets 01:22 Migration: Proxying Airflow DAGs to Dagster Compute 01:43 The Power of Incremental Migration with Airlift 02:00 Conclusion: Build a More Powerful Data Platform with Airlift