Airflow XCom for Beginners - All you have to know in 10 mins скачать в хорошем качестве

Airflow XCom for Beginners - All you have to know in 10 mins

Airflow XCom for Beginners - All you have to know in 10 mins to share data between tasks. 👍 Smash the like button to become better at Airflow ❤️ Subscribe to my channel to become a master of Airflow 🏆 Take my course : https://www.udemy.com/course/the-ulti... to join the legends of Airflow 🚨 My Patreon:   / marclamberti   to support my work and be friend for life The materials: https://www.notion.so/Airflow-XCOM-Al... The blogs post: https://marclamberti.com/blog/airflow... 1. Use case You have a data pipelines with 5 different tasks. The first task downloads data, 3 tasks train machine learning models, the last task chooses the best model. Each task training a machine learning model, produces an accuracy. According to this accuracy, you want to choose the best model in the last task. Question, how can you share the accuracies produced in the 3 training model tasks with the last task? XComs! 2. What is a XCom? XCom stands for cross communication and allows to share messages and small amount of data between tasks in data pipelines. A XCom is composed of a key (identifier), a value (must be serializable), a timestamp (when was created), an execution date (to which dagrun the XCom belongs with), a task id (which task created the XCom), a dag id (same as the task but for the dag). XComs are stored in the database of Airflow. 3. How to push a XCom 2 ways, with the return keywords or with xcom_push. Any value returned from an operator is automatically pushed as a XCom with the key return_value. With xcom_push you have to specify the key as well as the value. To use xcom_push, you have to access the task instance object corresponding to your task. 4. How to pull a XCom With xcom_pull. Again, to call xcom_pull, you have to access the task instance object of your task. xcom_pull expects two arguments, the key and a list of task ids. 5. XCom limitations Different xcom size limit according to the database used. Create implicit dependencies between your tasks. Enjoy!