LongHeads:多头注意力机制暗中是一个长上下文处理器
LongHeads: Multi-Head Attention is Secretly a Long Context Processor
AI总结:
本文提出无需训练的LongHeads框架,通过块选择策略让多头注意力各自处理分布内长度的上下文块,解决LLM长上下文处理中的泛化与计算问题,在128k密钥检索任务上达到100%准确率。
摘要:
大型语言模型(LLMs)在众多领域取得了令人瞩目的性能,但由于有限的长度泛化能力和注意力机制二次方的计算需求,往往难以有效且高效地处理长输入。许多人试图通过将注意力窗口限制在预训练长度内来缓解这一问题。然而,这些方法引入了新的问题,例如忽略中间上下文以及需要额外训练。为了解决这些问题,我们提出了LongHeads,这是一个无需训练的框架,通过释放多头注意力未开发的潜力来增强LLM的长上下文能力。我们不让每个头都关注完整句子(这由于分布外(OOD)问题而难以泛化到更长序列),而是允许每个头通过选择并关注重要的上下文块来处理分布内长度。为此,我们提出了一种块选择策略,该策略依赖于查询和键表示之间的内在相关性,高效地将上下文块分配给不同的头。这样,每个头确保能在训练长度内有效处理所关注的标记,而不同层中的不同头可以共同处理更长的上下文。LongHeads在线性时间内高效运行,与许多使用相对位置编码的LLM无缝契合。LongHeads在密钥检索任务上于128k长度实现了100%的准确率,验证了LongHeads在扩展现有模型可用上下文窗口方面的有效性。我们在 https://github.com/LuLuLuyi/LongHeads 发布了代码。
英文摘要:
Large language models (LLMs) have achieved impressive performance in numerous domains but often struggle to process lengthy inputs effectively and efficiently due to limited length generalization and attention's quadratic computational demands. Many sought to mitigate this by restricting the attention window within the pre-trained length. However, these methods introduce new issues such as ignoring the middle context and requiring additional training. To address these problems, we propose LongHeads, a training-free framework that enhances LLM's long context ability by unlocking multi-head attention's untapped potential. Instead of allowing each head to attend to the full sentence, which struggles with generalizing to longer sequences due to out-of-distribution (OOD) issues, we allow each head to process in-distribution length by selecting and attending to important context chunks. To this end, we propose a chunk selection strategy that relies on the inherent correlation between the query and the key representations, efficiently distributing context chunks to different heads. In this way, each head ensures it can effectively process attended tokens within the trained length, while different heads in different layers can collectively process longer contexts. LongHeads works efficiently in linear time, fits seamlessly with many LLMs that use relative positional encoding. LongHeads achieves 100% accuracy at the 128k length on passkey retrieval task, verifying LongHeads's efficacy in extending the usable context window for existing models. We release our code at https://github.com/LuLuLuyi/LongHeads .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。