SSA: Training Better Sparse Attention for LLMs скачать в хорошем качестве

SSA: Training Better Sparse Attention for LLMs

In this AI Research Roundup episode, Alex discusses the paper: 'SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space(2511.20102v1)' This work tackles the quadratic cost of full attention in large language models and the performance drop often seen with training-free sparse attention. The authors identify a key issue in native sparse-attention methods: excluded key–value pairs receive no gradients, preventing them from learning proper suppression and paradoxically reducing sparsity. SSA introduces a unified training framework that jointly considers sparse and full attention at every layer, enforcing bidirectional alignment in feature space. This preserves gradient flow to all tokens while explicitly encouraging sparse-attention outputs to match full-attention behavior, resulting in stronger sparsity and more efficient long-context processing. Paper URL: https://arxiv.org/pdf/2511.20102 #AI #MachineLearning #DeepLearning #LargeLanguageModels #SparseAttention #TransformerModels #LongContext