Structured Streaming Use-Cases at Apple скачать в хорошем качестве

Structured Streaming Use-Cases at Apple

Structured streaming plays an important role in current data infrastructure. In response to tremendous streaming requirements, we have actively worked on developing structured streaming in Spark in the past few months. In this talk, Kristine Guo and Liang-Chi Hsieh will detail some of the issues that arose when applying structured streaming and what was done to address them. Specifically, they will cover: How streaming applications that need to maintain large amounts of state require a scalable state store provider as an alternative to the in-memory solution built in with Spark. Structured streaming is currently missing session window support and although a map/flatMapWithState API may be used to implement a custom window, this approach does not generalize well across applications and is hard to maintain. Why we focused on structured streaming efforts like RocksDB state store and session windowing. Finally, they will detail how these features can help to compute aggregates over dynamic batches with minimum size requirements and perform stream-stream joins, while supporting high RPS and throughput. Connect with us: Website: https://databricks.com Facebook:   / databricksinc   Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...