学习何时关注:为长上下文LLM的条件记忆访问
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
AI总结:
本文提出L2A方法,通过条件性长程记忆访问提升长上下文LLM性能,使Qwen 2.5和Qwen 3模型的有效上下文长度从32K扩展到128K,同时提升训练效率并减少内存消耗。
摘要:
语言模型在超出预训练上下文长度时难以泛化,限制了长视界推理和检索。继续在长上下文数据上预训练有帮助但成本高,因为注意力机制具有二次扩展性。我们观察到大多数标记不需要(全局)对整个序列的注意力,可以依赖局部上下文。基于此,我们提出了L2A(学习关注),一种层,通过决定何时调用全局注意力来实现条件(标记级)长程记忆访问。我们在Qwen 2.5和Qwen 3模型上评估L2A,将有效上下文长度从32K扩展到128K。L2A在标准长上下文训练性能内3%以内,同时跳过80%的全局注意力,优于先前基线。我们还设计了定制的Triton内核,高效实现此标记级条件注意力,实现训练吞吐量和时间到第一个标记的2倍改进。此外,L2A使后训练剪枝高稀疏性全局注意力层成为可能,减少KV缓存内存高达50%,性能损失可忽略。
英文摘要:
Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval. Continued pretraining on long-context data can help but is expensive due to the quadratic scaling of Attention. We observe that most tokens do not require (Global) Attention over the entire sequence and can rely on local context. Based on this, we propose L2A (Learning To Attend), a layer that enables conditional (token-wise) long-range memory access by deciding when to invoke global attention. We evaluate L2A on Qwen 2.5 and Qwen 3 models, extending their effective context length from 32K to 128K tokens. L2A matches the performance of standard long-context training to within 3\% while skipping Global Attention for $\sim$80\% of tokens, outperforming prior baselines. We also design custom Triton kernels to efficiently implement this token-wise conditional Attention on GPUs, achieving up to $\sim$2$\times$ improvements in training throughput and time-to-first-token over FlashAttention. Moreover, L2A enables post-training pruning of highly sparse Global Attention layers, reducing KV cache memory by up to 50\% with negligible performance loss. Our code is released under Apache 2.0 at https://github.com/awslabs/hybrid-model-factory/tree/main/examples/research/L2A.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。