Apache Fluss and the Seven Deadly Sins of Streaming скачать в хорошем качестве

Apache Fluss and the Seven Deadly Sins of Streaming

For the last decade, Apache Kafka has been at the core of streaming data pipelines. After years in the trenches, we've learned some hard lessons and taken a fresh look at what true streaming analytics really needs. We refer to existing pain points as the seven deadly sins of streaming analytics: 1. Complex architectures that require multiple data copies, resulting in redundant storage costs 2. Inability to query data streams directly 3. Complex Multi-Joins with Large States 4. Separate links for streaming and Lakehouses (think Lambda-style architectures) 5. Lakehouse data freshness - latencies can vary from multiple double-digit minutes - to hourly - to daily intervals 6. Network Data Transmission and Costs 7. Changelog Generation, for keeping track of all the changes that happen at your events In response to these challenges, while seeking better and modern alternatives to address more efficiently the rising demand for real-time Data & AI, Apache Fluss was created, a columnar streaming storage layer. In this presentation, we will examine modern streaming data architectures and discuss concrete use cases it empowers, like real-time dashboards, streaming ETL and customer 360. We will see how architecting around Apache Fluss helps address these ""seven deadly sins", by introducing streams as tables that unlock new capabilities for delivering the next generation of real-time analytics, in a single unified streaming/lakehouse solution. The goal? A unified architecture that helps reduce complexity and delivers better analytics to data streams and better data freshness to data lakehouses Watch all conference recordings online now at: https://www.ververica.academy/learner... Discover more about Flink Forward: https://www.flink-forward.org/