Extending Flink state serialization for better performance and smaller checkpoint size - Grebennikov скачать в хорошем качестве

Extending Flink state serialization for better performance and smaller checkpoint size - Grebennikov

Operations with Flink state are a common source of performance issues for a typical stateful stream processing application. One tiny mistake can easily make your job to spend most of a precious CPU time in serialization and inflate a checkpoint size to the sky. In this talk we’ll focus on a Flink serialization framework and common problems happening around it: Is Kryo fallback is really that expensive from the CPU and state size perspective? How to plug your own or existing serializers into the Flink (like protobuf). Using Scala sealed traits without Kryo fallback. Using custom integer variable-length encoding and delta encoding for primitive arrays to further reduce the state size.