Spark Right-Sizing: Saving Thousands of PBHrs of Compute at LinkedIn скачать в хорошем качестве

Spark Right-Sizing: Saving Thousands of PBHrs of Compute at LinkedIn

At LinkedIn, we manage over 400,000 daily Spark applications consuming 200+ PBHrs of compute daily. To address the challenges posed by manual configuration of Spark's memory tuning options, which led to low memory utilization and frequent OOM errors, we developed an automated Spark executor memory right-sizing system. Our approach, utilizing a policy-based system with nearline and real-time feedback loops, automates memory tuning, leading to more efficient resource allocation, improved user productivity and increased job reliability. By leveraging historical data and real-time error classification, we dynamically adjust memory, significantly narrowing the gap between allocated and utilized resources while reducing failures. This initiative has achieved a 13% increase in memory utilization and a 90% drop in OOM-related job failures, saving us 1000s of PBHrs of compute every year. Talk By: Shreyesh Arangath, Senior Software Engineer, LinkedIn Here's more to explore: The best data warehouse is a lakehouse: https://www.databricks.com/product/da... How to migrate legacy data warehouses: https://www.databricks.com/resources/... See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dat... Connect with us: Website: https://databricks.com Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Facebook:   / databricksinc