MirrorMaker 2’s Offset Translation Isn’t Exactly Once (and That’s Okay) (Aiven Oy) | Current 24 скачать в хорошем качестве

MirrorMaker 2’s Offset Translation Isn’t Exactly Once (and That’s Okay) (Aiven Oy) | Current 24

More: https://current.confluent.io | Is your streaming application ready for a cluster failover event? MirrorMaker 2 offers replication of topic data and consumer offsets between Kafka clusters, and allows applications to recover from the loss of a Kafka cluster. MirrorMaker2 also includes support for exactly-once data replication with KIP-618, preventing errors from duplicating data. But turning on exactly-once doesn’t prevent data from being re-delivered to your application when using translated consumer offsets. Why? In this talk, we’ll be discussing the history of offset translation across the MirrorMaker2 versions, the existing translation algorithm, and its weaknesses. We’ll explain the design constraints surrounding the algorithm, and how MirrorMaker2 tries to strike a balance between memory usage, offset accuracy, offset precision, and translation availability. Like anything involving these kinds of trade-offs, the result isn’t perfect. So, we’ll also describe what you can expect from your offset translation in practice and explain how to get the best results from offset translation in your environment. Be prepared for your next Kafka failover by learning about the offset translation system and its limitations. LEARN MORE ► Confluent Developer: https://developer.confluent.io CONNECT Subscribe on YouTube:    / @confluentdeveloper   Community Slack: confluentcommunity.slack.com X: https://x.com/confluentinc Linkedin:   / confluent   GitHub: https://github.com/confluentinc Site: https://developer.confluent.io ABOUT CONFLUENT DEVELOPER Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io. #apachekafka #apacheflink #confluent