Databricks Streaming Tables Explained | Spark Declarative Pipelines Bronze Layer Episode 2 скачать в хорошем качестве

Databricks Streaming Tables Explained | Spark Declarative Pipelines Bronze Layer Episode 2

🚀 Full Databricks Lakeflow Masterclass (32+ Episodes)    • Databricks Lakeflow Masterclass   📚 Start the course here: 1️⃣ Lakeflow Architecture    • Databricks Lakeflow Explained (2026) | Arc...   2️⃣ Lakeflow Connect    • 1️⃣ Lakeflow Connect Explained (2026) | Da...   This video is Episode 2 of the Databricks Lakeflow Declarative Pipelines Masterclass 2026, where we build production-grade data pipelines using Spark Declarative Pipelines. In this episode we focus on Streaming Tables, the foundation of the Bronze layer in a Lakeflow pipeline. Streaming tables allow you to continuously ingest raw data into the Lakehouse using Auto Loader and Spark Structured Streaming, while preserving the full data history. Using the @dp.table decorator, we define declarative ingestion pipelines that automatically process new data, maintain checkpoints, and scale efficiently. What You Will Learn • What Streaming Tables are in Databricks Lakeflow • The role of the Bronze layer in Medallion architecture • How to configure Auto Loader for incremental ingestion • Schema inference vs explicit schema definition • Handling schema evolution in streaming ingestion • Working with JSON, CSV, and Parquet ingestion pipelines • Adding metadata columns for observability and debugging • Performance optimization for high-volume ingestion pipelines GitHub Code All code used in this tutorial is available here: https://github.com/AhmedMahmoud2/data... Example file used in this episode: 02_streaming_tables_bronze.py Lakeflow Declarative Pipelines Masterclass 2026 Episode 1 — Introduction to Spark Declarative Pipelines Episode 2 — Streaming Tables (Bronze Layer) Episode 3 — Materialized Views (Silver Layer) Episode 4 — Data Quality Expectations Episode 5 — CDC Pipelines Episode 6 — Schema Evolution Episode 7 — Slowly Changing Dimensions (SCD Type 2) ▶ Previous Episode Databricks Lakeflow Pipelines Explained | Spark Declarative Pipelines Introduction (Episode 1)    • Databricks Lakeflow Pipelines Explained | ...   ▶ Next Episode Databricks Materialized Views Explained | Spark Declarative Pipelines Silver & Gold Layer Episode 3    • Databricks Materialized Views Explained | ...   About the Author Ahmed Mahmoud Principal Data Engineer & AI Architect Sharing practical tutorials on: • Databricks • Lakehouse Architecture • Data Engineering • AI-ready Data Platforms