Sparkless Patterns: The New Data Stack and Modern Data Architecture - Alon Agmon скачать в хорошем качестве

Sparkless Patterns: The New Data Stack and Modern Data Architecture - Alon Agmon

In recent years, we have witnessed the emergence of powerful frameworks and libraries, such as Apache DataFusion, DuckDB, and Polars, that enable us to query and transform vast amounts of data at blazing speeds, even with modest compute resources. These frameworks not only enhance our capabilities but also pave the way for new kinds of data architectures and design patterns that challenge the traditional design of our ETL processes and query infrastructure. Workloads that once required a distributed Spark cluster to complete in a reasonable time, or queries that needed data warehouses to scan vast datasets, can now be processed more efficiently and with less overhead. In this session, I will focus on three major frameworks: DuckDB, Apache DataFusion, and Polars, which are central to what many refer to as the 'new data stack' and its transformative capabilities. We will discuss the significant advantages these frameworks offer to data engineers, especially in cloud-native environments, and demonstrate the new data design patterns they enable through real use cases that utilize them. Alon is an engineering group manager at AppsFlyer. Alon has been managing data engineering teams over the past few years, and handling exceptions for almost two decades. Intrigued by distributed systems, the big data ecosystem, and technological disruption. Fearless adopter and supporter of cutting edge data technologies. hayaData 2024, Tel-Aviv, Israel https://www.haya-data.com/