End-to-End Spotify Data Engineering Project | Azure ADF + Databricks DLT + SCD Type 2 (CDC Demo) скачать в хорошем качестве

End-to-End Spotify Data Engineering Project | Azure ADF + Databricks DLT + SCD Type 2 (CDC Demo)

In this demo, I walk through an end-to-end Spotify-style Data Engineering project built on the Azure Data Platform, showcasing incremental ingestion, CDC handling, and SCD Type 2 processing using modern lakehouse patterns. The pipeline ingests dimensional and fact data from Azure SQL Database using Azure Data Factory, lands incremental data into ADLS Gen2, and processes it through Bronze → Silver → Gold layers in Databricks. 🔹 What this project covers: Incremental ingestion with watermark-based CDC in Azure Data Factory Metadata-driven pipelines using JSON CDC files Streaming transformations in Databricks Delta Live Tables (DLT)-style processing SCD Type 2 implementation using create_auto_cdc_flow() Data quality enforcement with expectations Star schema modeling (Dimensions + Fact) 🔹 Business entities modeled: DimUser DimArtist DimTrack DimDate FactStream This demo is ideal for Data Engineers, Azure professionals, and anyone preparing for Databricks / Lakehouse interviews or building production-grade analytics pipelines. #DataEngineering #AzureDataFactory #Databricks #DeltaLiveTables #SCDType2 #CDC #SpotifyDataEngineering #AzureADF #ADLSGen2 #Lakehouse #BigData #MedallionArchitecture #PySpark #DataEngineer #LearningByDoing