Scalable Inference Algorithms for Large Language Models | Woomin Song, KAIST | AER LABS скачать в хорошем качестве

Scalable Inference Algorithms for Large Language Models | Woomin Song, KAIST | AER LABS

Scalable Inference Algorithms for LLMs: REFORM & STAND In this presentation, Woomin Song introduces two training-free frameworks for efficient LLM inference: REFORM for long-context processing and STAND for accelerating test-time scaling. Part 1: REFORM (NeurIPS 2025) Learn how REFORM overcomes the quadratic computational cost of Transformer attention and KV cache memory bottlenecks. By combining Recurrent Chunking with On-Demand Cache Recomputation, REFORM achieves 75% accuracy on 1M-token Needle-In-A-Haystack benchmarks while significantly reducing latency and memory usage. Part 2: STAND (EMNLP 2025) Discover how STAND accelerates test-time scaling (chain-of-thought reasoning, majority voting, tree search) through model-free speculative decoding. By leveraging cross-trajectory n-gram overlaps and stochastic drafting, STAND achieves the same accuracy in under 40% of the decoding time. Both works were conducted during the speaker's internship at Amazon. Speaker: Woomin Song | Integrated M.S. + Ph.D. Student at KAIST Affiliation: KAIST (Korea Advanced Institute of Science and Technology) [Resume & Profile] https://woominsong.github.io/ --- Timestamps: [Part 1: REFORM - Long Context Processing] [00:00] Introduction: Scalable Inference Algorithms for LLMs [00:42] The Problem: Quadratic computational costs and KV cache bottlenecks [01:52] The Challenge: Pre-trained context length limits [02:18] Existing Solutions: Recurrent Compression (StreamingLLM, H2O) [03:36] Existing Solutions: Random Access approaches and their limitations [04:28] Introducing REFORM: Best of both worlds [05:08] Key Observation: Attention heads as token selectors using cosine similarity [05:52] Methodology Overview: Compress, Gather, and Recompute stages [06:28] Step 1: Compress - Recurrent chunking with early exit strategy [08:12] Handling KV Cache: Token eviction using attention scores [08:52] Step 2: Gather - Cosine similarity search for relevant tokens [09:16] Step 3: Recompute - Forwarding gathered inputs for generation [09:32] Evaluation: Needle-In-A-Haystack (NIAH) benchmark results [10:24] Synthetic Benchmarks: Comparison with InfLLM (23% vs 75% at 1M tokens) [10:52] Realistic Benchmarks: InfiniteBench, RepoEval, and MM-NIAH results [11:28] Efficiency Analysis: Inference time and peak GPU memory savings [12:16] Comparison with RAG: Architecture-level advantages [13:24] Ablation Studies: Compression strategies and head selection [Part 2: STAND - Test-Time Scaling Acceleration] [14:08] Introduction: Test-time scaling and the latency problem [15:12] Background: Chain-of-thought, majority voting, and tree search [16:32] The Research Problem: Speeding up without compromising accuracy [17:04] Speculative Decoding: Draft-then-verify framework [18:16] Key Observation: High n-gram overlap across reasoning trajectories [19:08] Model-Free Drafters: Leveraging cross-trajectory information [20:04] Stochastic vs Deterministic Drafting: Why sampling matters [21:16] STAND Components: N-gram drafter with probability awareness [22:08] Optimization Techniques: Gumbel top-k trick for faster sampling [22:32] Tree Drafting: Optimizing tree structure for higher acceptance [23:16] Evaluation: AIME 2024, GPQA Diamond, and LiveCodeBench results [24:28] Results: Same accuracy in under 40% decoding time [25:04] Batch Decoding Scenarios: STAND remains effective in parallel inference [25:32] Ablation Studies: Contribution of stochastic drafting and tree optimization [26:24] Key Finding: Deeper and narrower tree structures perform better [26:52] Summary: N-gram based speculative decoding for test-time scaling [Q&A Session] [27:28] Q&A: How speculative decoding ensures output correctness [31:04] Q&A: Greedy decoding vs sampling scenarios [33:28] Q&A: Tree drafting explanation and benefits [38:24] Q&A: Batch decoding and high-throughput inference scenarios --- Hosted by AER Labs #REFORM #STAND #KAIST #LLM #LongContext #SpeculativeDecoding #TestTimeScaling #DeepLearning #Transformer #Inference #AIResearch #NLP #MachineLearning #NeurIPS2025 #EMNLP2025TAND for accelerating test-time scaling.