Cardinality Estimation in Apache Spark 2.3 (Ron Hu & Zhenhua Wang) скачать в хорошем качестве

Cardinality Estimation in Apache Spark 2.3 (Ron Hu & Zhenhua Wang)

Ron Hu, a Principal Big Data Architect at Huawei Technologies, and Zhenhua Wang, a Research Engineer at Huawei Technologies, explain how Apache Spark 2.2 shipped with a state-of-art cost-based optimization framework that collects and leverages a variety of per-column data statistics (e.g., cardinality, number of distinct values, NULL values, max/min, avg/max length, etc.) to improve the quality of query execution plans. Skewed data distributions are often inherent in many real world applications. In order to deal with skewed distributions effectively, we added equal-height histograms to Apache Spark 2.3. Leveraging reliable statistics and histogram helps Spark make better decisions in picking the most optimal query plan for real world scenarios. Learn more here: https://databricks.com/session/cardin... Article you might like: https://databricks.com/session/deep-l... About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unifie... Connect with us: Website: https://databricks.com Facebook:   / databricksinc   Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...