Handling 2 Million Apache Kafka Messages Per Second at Honeycomb скачать в хорошем качестве

Handling 2 Million Apache Kafka Messages Per Second at Honeycomb

https://cnfl.io/podcast-episode-204 | How many messages can Apache Kafka® process per second? At Honeycomb, it’s easily over one million messages. In this episode, get a taste of how Honeycomb uses Kafka at massive scale. Liz Fong-Jones (Principal Developer Advocate, Honeycomb) explains how Honeycomb manages Kafka-based telemetry ingestion pipelines and scales Kafka clusters. And what is Honeycomb? Honeycomb is an observability platform that helps you visualize, analyze, and improve cloud application quality and performance. Their data volume has grown by a factor of 10 throughout the pandemic, while the total cost of ownership has only gone up by 20%. But how, you ask? As a developer advocate for site reliability engineering (SRE) and observability, Liz works alongside the platform engineering team on optimizing infrastructure for reliability and cost. Two years ago, the team was facing the prospect of growing from 20 Kafka brokers to 200 Kafka brokers as data volume increased. The challenge was to scale and shuffle data between the number of brokers while maintaining cost efficiency. The Honeycomb engineering team has experimented with using sc1 or st1 EBS hard disks to store the majority of longer-term archives and keep only the latest hours of data on NVMe instance storage. However, this approach to cost reduction was not ideal, which resulted in needing to keep data that is older than 24 hours on SSD. The team began to explore and adopt Zstandard compression to decrease bandwidth and disk size; however, the clusters were still struggling to keep up. When Confluent Platform 6.0 rolled out Tiered Storage, the team saw it as a feature to help them break away from being storage bound. Before bringing the feature into production, the team did a proof of concept, which helped them gain confidence as they watched Kafka tolerate broker death and reduce latencies in fetching historical data. Tiered Storage now shrinks their clusters significantly so that they can hold on to local NVMe SSD and the tiered data is only stored once on Amazon S3, rather than consuming SSD on all replicas. In combination with the AWS Im4gn instance, Tiered Storage allows the team to scale for long-term growth. Honeycomb also saved 87% on the cost per megabyte of Kafka throughput by optimizing their Kafka clusters. EPISODE LINKS ► Tiered Storage: https://cnfl.io/tiered-storage-episod... ► Introducing Confluent Platform 6.0: https://cnfl.io/confluent-6-0-tiered-... ► Scaling Kafka at Honeycomb: https://www.honeycomb.io/blog/scaling... ► Join the Confluent Community: https://cnfl.io/join-confluent-commun... ► Learn more with Kafka tutorials, resources, and guides: https://cnfl.io/confluent-developer-e... ► Live demo: Intro to Event-Driven Microservices with Confluent: https://cnfl.io/event-driven-microser... ► Use PODCAST100 to get $100 of free Confluent Cloud usage: https://cnfl.io/try-confluent-cloud-e... ► Promo code details: https://cnfl.io/podcast100-details-ep... TIMESTAMPS 0:00 - Intro 01:16 - Observability platform architecture 06:32 - Scaling Kafka clusters 07:55 - 70 partitions per volume 11:25 - Kafka on ARM 21:19 - Kafka was our recovery plan 35:54 - 87% reduction cost per megabyte 40:03 - It's a wrap ABOUT CONFLUENT Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io. #datapipeline #kafkastreams #apachekafka #kafka #confluent