Taking Advantage of a Disaggregated Storage and Compute Architecture - Brian Cho and Ergin Seyfe скачать в хорошем качестве

Taking Advantage of a Disaggregated Storage and Compute Architecture - Brian Cho and Ergin Seyfe

Brian Cho, a software engineer at Facebook, and Ergin Seyfe, a software engineer on BigCompute team at Facebook, talk about how the hardware trend is moving to disaggregated storage and compute architecture in order to improve cost efficiency and scale at finer granular. For Spark, this means that compute machines store and access temporary data (shuffle, spill, persisted RDD/dataframes) on storage machines over the network. Disaggregated storage has many advantages over collocated storage where each node is constrained to the space of its local disks. Over half of our Spark workload is now running on disaggregated infrastructure with individual jobs shuffling more than 300 TB compressed. Learn more here: https://databricks.com/session/taking... Article you might like: https://databricks.com/session/transp... About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unifie... Connect with us: Website: https://databricks.com Facebook:   / databricksinc   Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...