Achieving Sub‑100ms Real‑Time Stream Processing with an S3‑Native Architecture (RisingWave Keynote) скачать в хорошем качестве

Achieving Sub‑100ms Real‑Time Stream Processing with an S3‑Native Architecture (RisingWave Keynote)

Yingjun Wu, Founder and CEO at ‪@risingwave-labs‬ , shared his experience in a techy talk at our latest applydata Meetup at the Berlin office of ‪@Diconium‬ . About his talk: Stream processing systems have traditionally relied on local storage engines, such as RocksDB, to achieve low-latency performance. While this approach works well in single-node setups, it does not scale efficiently in cloud environments, where elasticity and the separation of compute and storage are crucial. This topic explores how RisingWave rethinks stream processing architecture by building directly on top of S3, while still achieving sub-100 ms latency. Central to this architecture is Hummock, a log-structured state engine designed specifically for object storage. Hummock organizes state into a three-tier hierarchy: an in-memory cache for the hottest keys, a disk cache managed by Foyer for warm data, and S3 as the persistent cold tier. Queries are never executed directly on S3, avoiding the variable performance typical of object storage. The system also leverages remote compaction, which offloads heavy maintenance tasks from query nodes and prevents interference between user queries and background operations. Fine-grained caching policies and eviction strategies further ensure consistent query performance while maintaining cloud-native scalability and elasticity. This topic provides a detailed look at designing streaming systems that balance durability, scalability, and low latency in a cloud-native, S3-based environment, offering insights for both system architects and engineers building real-time data platforms. Timestamps: 00:00 Introduction applydata Meetup 00:15 Introduction to streaming 00:45 What's RisingWave? 05:32 Design principles of RisingWave 08:52 S3 storage & architecture details 15:28 Challenges 15:50 Performance & Cost 21:02 Foyer: hybrid cache 24:42 Achieving 100ms Latency 26:10 Summary #streamprocessing #risingwave #datastreaming #datakeynote #meetup