End-to-End Data Engineering Project | GCP, Kafka, PySpark, Airflow (Full Walkthrough) скачать в хорошем качестве

End-to-End Data Engineering Project | GCP, Kafka, PySpark, Airflow (Full Walkthrough)

I built a full end-to-end Data Engineering pipeline using GCP, Apache Kafka, PySpark, Dataproc, Airflow, BigQuery, and Looker Studio. This project demonstrates real-world batch + streaming ETL workflows just like Uber, Lyft, and DoorDash use. 🚀 What’s inside this video: 00:00 – Intro 00:45 – Architecture Overview 02:10 – GCS Raw Zone Setup 03:40 – Dataproc & PySpark Processing 06:00 – BigQuery Partitioned Tables 08:20 – Airflow Orchestration (DAG Walkthrough) 10:10 – Kafka Real-Time Streaming (Producer + Consumer) 12:00 – Spark Structured Streaming 13:20 – Final Dashboard + Insights 14:00 – Closing Notes 🛠️ Tech Used Google Cloud Storage (Raw + Processed) Dataproc (PySpark) BigQuery Apache Kafka (Zookeeper + Brokers) Spark Structured Streaming Apache Airflow (Orchestration) Looker Studio (Dashboard) 📊 LIVE Dashboard https://lookerstudio.google.com/reporting/... 💻 GitHub Source Code https://github.com/Ruthvikyadavm/gcp-data-... 👋 About Me I’m Ruthvik — a Cloud Data Engineer specializing in ETL pipelines, streaming systems, and data platforms on GCP/AWS. If you’re a recruiter/hiring manager and want to view my full resume or discuss opportunities: 📧 [email protected] 🔗 LinkedIn: https://www.linkedin.com/in/ruthvikyadav/ #DataEngineering #GCP #Kafka #PySpark #Airflow #BigQuery #DataPipeline #CloudComputing #PortfolioProject