Azure Synapse Analytics | Spark pool | Catalyst Optimizer and Adaptive Query Execution скачать в хорошем качестве

Azure Synapse Analytics | Spark pool | Catalyst Optimizer and Adaptive Query Execution

Today, as businesses demand faster time to insights, data engineers, data scientists and developers are expecting platform to provide not only quicker way to write their code so that learning curve is minimized but also to optimize it in the best possible way during execution so that it can run faster. This will let them to focus on solving business problems while platform will do all the heavy-lifting behind the scene. This video talks about Spark Catalyst Optimizer and Adaptive Query Execution, an enhancement made in Spark 3.0. The Catalyst Optimizer is a query optimization engine which optimizes your code on submission for execution as well as during execution of your code whereas Adaptive Query Execution (AQE) reoptimizes and adjusts query plans based on runtime metrics collected during the execution of the query. 00:00 Introduction 00:43 Catalyst Optimizer 06:03 Physical Execution of Spark Jobs 07:57 RDD vs Dataframe (Spark SQL) 09:50 Catalyst Optimizer - Its stages and optimization 15:10 RDD vs Dataframe vs Dataset APIs 20:56 Adaptive Query Execution (AQE) 28:13 Demo - Understand and Analyze Execution Plan Thank you once again for watching, please do like, subscribe and let me know your feedback or any specific topic you would like me to cover next. GitHub Repo to download deck and script used in the video: https://github.com/AasTrailblazers/Az... Apache Spark https://spark.apache.org/ RDD vs Dataframe vs Dataset APIs https://docs.microsoft.com/en-us/azur... Dataframe - performance comparison with RDD https://databricks.com/blog/2015/02/1... Adaptive Query Execution https://databricks.com/blog/2020/05/2... Adaptive Query Execution Configuration https://spark.apache.org/docs/latest/...