LiquidCache: Pushdown Caching for DataFusion скачать в хорошем качестве

LiquidCache: Pushdown Caching for DataFusion

LiquidCache, presented by Xiangpeng Hao from the University of Wisconsin-Madison, details a solution to the inefficiencies of conventional caching, which simply mirrors storage and understands neither compute nor storage, leading to resource waste and slow decoding times. LiquidCache introduces "pushdown caching" by re-encoding data into an optimized, ephemeral "Liquid format" that evolves based on access frequency. This new approach allows LiquidCache to understand both storage and compute, enabling it to push filters directly into the encoding itself and perform encoded evaluation and selective decoding. By only decoding the data that survives the filter, LiquidCache achieves up to 10x faster decoding and significantly less network traffic. It is designed as a shared service that multiple compute engines like Spark, DuckDB, and DataFusion can leverage for independent scaling without needing special hardware.