3.11 Building Real Time Analytics on Lakehouse: Streaming and Transformations скачать в хорошем качестве

3.11 Building Real Time Analytics on Lakehouse: Streaming and Transformations

   • Microsoft Fabric For Beginners   In this tutorial, I explain how to build Real-Time Analytics on ever-changing Lakehouse tables. Finally, I demonstrate PySpark code that leverages Change Data Feed and Spark Structured Streaming to build low-latency streaming pipelines on the Lakehouses. Chapters: 00:00- Introduction 00:43- Preview 03:08- Engineering Challenges of Building Streaming Applications on Lakehouses 07:57- Demo- prepare data 09:54- Applying transformations and migrating historical data 10:46- Enabling Change Data Feed 14:28- Using window functions to de-duplicate Change Data Feed data 15:30- Using streaming ForEachBatch destination to handle the 'Merge into' logic 17:27- Handling Delete, Update and Insert transactions inside forEachBatch function 18:05- Building Streaming pipeline to stream changed data to Lakehouse destination 20:15- Validation of streaming functionality You can download the demo artifacts here: Notebook: https://github.com/fazizov/youtube/bl... Data: https://github.com/fazizov/youtube/bl... Please subscribe:    / @fazizov   Follow me on Linkedin: www.linkedin.com/in/fikrat-azizov-865a0528 and X: @fikrat_azizov Check out this tutorial to learn more about Spark Structured Streaming,Change Data Feed and Time Travel:    • 3.10 Unlock The Power Of Change Data Feed ...      • 3.08 Discover The Exciting World Of Spark ...  ,    • 3.09 Unleash The Power Of Spark Streaming ...   Official Documentation: https://learn.microsoft.com/en-us/fab... https://learn.microsoft.com/en-us/fab... https://docs.delta.io/2.0.0/delta-cha... https://delta.io/blog/2023-02-01-delt... https://spark.apache.org/docs/latest/... Hashtags: #realtimeanalytics, #streaming, #microsoftfabric ,#azure,#microsoft, #dataengineering,#cloudcomputing, #dataanalytics, #lakehouse, #azuretutorial, #azuretraining, #datapipeline, #dataextraction , #dataintegration, #datatransfer, #dataflow, #spark, #deltalake, #synapse, #synapsedataenginering, #demo, #datalake, #transformation, #ingested, #datawarehouse, #dataintegration, #azuredatabricks ,#databricks, #bigdata, #bigdatatechnologies, #pyspark, #sparksql, #notebook ,#transformationvideo, #bronze, #medallion , #modeling, #facts, #silver, #gold, #historical data, #timetravel, #incremental,#datafactory,