Accelerating LLM Serving with Prompt Cache Offloading via CXL скачать в хорошем качестве

Accelerating LLM Serving with Prompt Cache Offloading via CXL

Don Moon Director - SK hynix, Taeho Hwang Researcher - SK hynix LLM inference uses a key-value (KV) cache to store intermediate activations for token-wise reuse, cutting redundant computation and latency. Prompt caching extends this across multi-turn or long-session workloads, but as user counts and sequence lengths grow, GPU HBM alone cannot sustain the full cache footprint, forcing offloads into higher-capacity media like host DRAM. However, host DRAM is limited in scalability. We address this by integrating CXL-attached memory via PCIe switches, unlocking multi-terabyte capacity beyond host limits. In ShareGPT benchmarks, augmenting a GPU with an additional 80GB of CXL memory (alongside host DRAM) raised prompt cache hit rates by 2.5x, reduced time-to-first-token by 32.9%, and boost overall throughput by 38.5%. This cost-effective tiered design delivers consistent low-latency responses under heavy load while notably reducing infrastructure costs , making LLM service deployment both faster and economical.