arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.22175cs.CL
收藏

DySCO:动态注意力缩放解码用于长上下文语言模型

DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models

Princeton Language and Intelligence, Princeton University(普林斯顿语言与智能,普林斯顿大学);Department of Computer Science, New York University(纽约大学计算机科学系)

Xi Ye, Wuwei Zhang, Fangcong Yin, Howard Yen, Danqi Chen

AI总结:

DySCO通过动态调整注意力机制提升长上下文推理能力,利用检索头识别关键token并增强其权重,在多个基准测试中取得25%的性能提升。

摘要:

理解并推理长上下文是语言模型的关键能力。尽管近期模型支持更长的上下文窗口,但随着输入长度增加,准确性往往下降。在实践中,模型难以在解码过程中保持注意力与最相关上下文的一致性。本文提出DYSCO,一种新的解码算法,通过检索头识别任务相关token并显式增强其权重,动态调整生成过程中的注意力以更好地利用相关上下文。该方法无需训练,可直接应用于任何现成的语言模型。在多个指令调优和推理模型上,DySCO在挑战性长上下文推理基准中表现优异,128K上下文长度下相对提升最高达25%(MRCR和LongBenchV2)。进一步分析强调动态注意力重缩放和检索头引导选择对方法有效性的重要性,并提供了解码时注意力行为的可解释见解。代码可在https://github.com/princeton-pli/DySCO获取。

英文摘要:

Understanding and reasoning over long contexts is a crucial capability for language models (LMs). Although recent models support increasingly long context windows, their accuracy often deteriorates as input length grows. In practice, models often struggle to keep attention aligned with the most relevant context throughout decoding. In this work, we propose DYSCO, a novel decoding algorithm for improving long-context reasoning. DYSCO leverages retrieval heads--a subset of attention heads specialized for longcontext retrieval--to identify task-relevant tokens at each decoding step and explicitly up-weight them. By doing so, DYSCO dynamically adjusts attention during generation to better utilize relevant context. The method is training-free and can be applied directly to any off-the-shelf LMs. Across multiple instruction-tuned and reasoning models, DYSCO consistently improves performance on challenging long-context reasoning benchmarks, yielding relative gains of up to 25% on MRCR and LongBenchV2 at 128K context length with modest additional compute. Further analysis highlights the importance of both dynamic attention rescaling and retrievalhead guided selection for the effectiveness of the method, while providing interpretability insights into decoding-time attention behavior. Our code is available at https://github.com/princeton-pli/DySCO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑