Flash for Apache Spark Shuffle with Cosco скачать в хорошем качестве

Flash for Apache Spark Shuffle with Cosco

Cosco is an efficient and reliable shuffle-as-a-service that powers Spark jobs at Facebook warehouse scale. Cosco is built on in-memory aggregation across a shared pool of distributed memory and provides much more efficient disk usage than Spark’s built-in shuffle. In this talk, we present how adding a little flash to Cosco goes a long way in further improving shuffle efficiency: Flash decreases memory usage, and larger write-ahead (aggregation) buffers further help decrease disk IO. We also demonstrate, via careful experiments and analysis, that dynamically leveraging both memory and flash protects flash endurance even for write-once/read-once workloads like shuffle. Finally, the long time-scale at which flash’s endurance bottleneck applies allows it to gracefully absorb short-term spikes in workload. We discuss how flash fits into Cosco’s architecture and deployment model at Facebook, learnings from deploying at scale in production, and potential future work. We first presented Cosco at Spark+AI Summit 2019. About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unifie... Connect with us: Website: https://databricks.com Facebook:   / databricksinc   Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...