Optimal Data Lake for analytics: Apache Kafka and ClickHouse by Olena Kutsenko скачать в хорошем качестве

Optimal Data Lake for analytics: Apache Kafka and ClickHouse by Olena Kutsenko

For more info on the next Devoxx UK event 👉 www.devoxx.co.uk Apache Kafka is amazing at handling real-time data feeds. However, in certain cases we need to come back to older records to analyse and process data at later times. This is challenging because storing records indefinitely and doing large scans of data in Apache Kafka is not optimal. ClickHouse, on the other hand, is a scalable and reliable storage designed to handle petabytes of data and, at the same time, it is a powerful open-source tool for fast online analytical processing. It was initially developed by Yandex and is now used by many companies for data analytics. What's more ClickHouse has a built-in table engine to publish and subscribe to Apache Kafka data feeds. Hence, we can use Apache Kafka and ClickHouse in collaboration to transition older records to a data lake in order to perform analytics at scale on top of fresh data that Kafka provides. In this talk you’ll learn how to use Apache Kafka together with ClickHouse and how to query the data stored in the data lake. This session is for those who want to perform analytics with fast response time over a huge volume of data without the need to downsample it.