Apache Airflow: Adios SubDAGs! Welcome TaskGroups! скачать в хорошем качестве

Apache Airflow: Adios SubDAGs! Welcome TaskGroups!

Adios SubDAGs! Welcome TaskGroups! In Airflow 2.0, you should forget about SubDAGs as a new easier, more efficient concept just comes in, TASKGROUP! 👍 Smash the like button to become an Airflow Super Hero! ❤️ Subscribe to my channel to become a master of Airflow 🏆 Take my course : https://www.udemy.com/course/the-ulti... to join the legends of Airflow 🚨 My Patreon:   / marclamberti   to support my work and be friend for life 1. Problem SubDAGs are complex. Too complex for their goal. Before Airflow 2.0, to group tasks, you had to define SubDAGs. To define a SubDAG you need 3 things. First, import the SubdagOperator. Second, create a python function returning a DAG. Third, instantiate a new DAG just to group the tasks in that python function. You had to take care of the executor, potential deadlocks, if the default arguments are well shared between the parent DAG and the Subdag, checking if the id of the SubDAG is well formatted and so on. TOO MUCH COMPLEXITY! Well, that's come to an end. 2. Solution TaskGroups! Since Airflow 2.0, you can forget about SubDAGs and start leveraging the power of TaskGroups. No need to create a python function, to instantiate a DAG, to use an Operator, no need of that. The ONLY thing you have to do is to import the TaskGroup class. Then, instantiate a TaskGroup, put your tasks under the task group and THAT'S IT. 3. Benefits TaskGroups are much easier to use, faster to implement, less prone to errors and efficient. They do what SubDAG do but in a much better way. You won't have to take care of potential deadlocks. Pools are respected, the executor is the same as the one you use for all of your tasks (By default, the sequential executor is set to the SubDAGOperator). Go with TaskGroups :) 5. BranchPythonOperator task skipped When a task is not triggered by the BranchPythonOperator, it is skipped. The problem is, if you have a task which depends both on the skipped task and the task triggered by the BranchPythonOperator, that task will be skipped as well. Why? Trigger rules! By default, if one of the parents of a task is skipped, then the task is skipped as well. You can fix this with the trigger rule none_failed_or_skipped Enjoy!