arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.15225cs.CLcs.AIcs.LG
收藏

SEAL:缩放以强调长上下文检索中的注意力

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval

Department of Convergence IT Engineering(融合IT工程系);Department of Computer Science and Engineering(计算机科学与工程系);Graduate School of Artificial Intelligence(人工智能研究生院)

Changhun Lee, Minsang Seok, Jun-gyu Jin, Younghyun Cho, Eunhyeok Park

AI总结:

提出SEAL方法,通过识别并调整与长上下文检索相关的注意力头,利用生成数据增强其强度,显著提升LLM长上下文检索性能,并可扩展上下文限制。

摘要:

尽管许多先进的LLM被设计用于处理长序列数据,但我们仍能观察到即使在序列长度限制内,质量也会显著下降。在这项工作中,我们引入了一种名为“缩放以强调长上下文检索的注意力”(SEAL)的新方法,该方法增强了大型语言模型(LLM)在长上下文中的检索性能。我们观察到,特定的注意力头与长上下文检索密切相关,与检索分数呈正相关或负相关,调整这些头的强度能在很大程度上提升LLM在长上下文中的质量。基于这一洞察,我们提出了一种基于学习的机制,利用生成的数据来强调这些注意力头。通过应用SEAL,我们在各种任务和模型上实现了长上下文检索性能的显著提升。此外,当与现有的免训练上下文扩展技术结合时,SEAL扩展了LLM的上下文限制,同时保持了高度可靠的输出。

英文摘要:

While many advanced LLMs are designed to handle long sequence data, we can still observe notable quality degradation even within the sequence limit. In this work, we introduce a novel approach called Scaling to Emphasize Attention for Long-context retrieval (SEAL), which enhances the retrieval performance of large language models (LLMs) over long contexts. We observe that specific attention heads are closely tied to long-context retrieval, showing positive or negative correlation with retrieval scores, and adjusting the strength of these heads boosts the quality of LLMs in long context by a large margin. Built on this insight, we propose a learning-based mechanism that leverages generated data to emphasize these heads. By applying SEAL, we achieve significant improvements in long-context retrieval performance across various tasks and models. Additionally, when combined with existing training-free context extension techniques, SEAL extends the contextual limits of LLMs while maintaining highly reliable outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑