How to build an automated data pipeline using Airflow, dbt, Postgres, and Superset (Windows 11 WSL) скачать в хорошем качестве

How to build an automated data pipeline using Airflow, dbt, Postgres, and Superset (Windows 11 WSL)

How to build an automated ELT pipeline from ingesting live data from an API to creating a near-realtime dashboard, using Windows WSL Ubuntu 24 and Docker images of Apache Airflow 3.0 (the latest), dbt, Postgres, and Apache Superset. In this live coding tutorial, I walk you through the thinking process, and show you every step. This tutorial gets all the tools communicating with each other but does not explore too much into the functionalities of each. This video is version 1. Improvements are underway. Looking to get personalized mentorship for landing your next data role? Data Engineer Academy helped me get started in my career as a Data Engineer. Use my referral link to get 10% off of your tuition: https://my.dataengineeracademy.com/lo... 0:00 - Intro 1:25 - Installing Visual Studio Code 2:05 - Setting up a WSL Linux Instance 5:56 - Installing and Setting up Docker for WSL 7:02 - Weatherstack API 7:59 - API Request using Python 13:48 - Setting up Postgres using Docker Compose 17:48 - Ingest Data into Postgres with psycopg2 27:24 - Setting up Airflow using Docker Compose 34:19 - Creating a DAG for Data Ingestion 43:40 - Setting up dbt using Docker Compose 50:16 - Creating dbt Models for Analytics 57:26 - Creating a DAG for dbt 1:04:34 - Creating DAG dependencies 1:05:38 - Setting up Superset using Docker Compose 1:16:38 - Creating a Real-time Superset Dashboard