arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.12962cs.CL
收藏

无限检索:长上下文处理中的注意力增强型 LLM

Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing

Xiaoju Ye, Zhichun Wang, Jingyuan Wang

AI总结:

针对 LLM 上下文窗口受限问题,本文提出 InfiniRetri,利用模型自身注意力实现无限长度输入检索,无需训练,并在 NIH 和真实基准上取得 SOTA 表现。

摘要:

受限于大语言模型(LLM)的上下文窗口大小,处理输入 token 超过上限的各类任务一直具有挑战性,无论是简单的直接检索任务,还是复杂的多跳推理任务。尽管已有多种方法被提出以增强 LLM 的长上下文处理能力,但它们要么产生高昂的训练后成本,要么需要额外的工具模块(例如 RAG),要么在现实任务中未表现出显著提升。我们的工作观察了各层中注意力分布与生成答案之间的相关性,并通过实验证实注意力分配与检索增强能力相一致。基于上述洞见,我们提出了一种新颖方法 InfiniRetri,该方法利用 LLM 自身的注意力信息,在无限长度的输入中实现准确检索。评估表明,使用一个 0.5B 参数模型,InfiniRetri 在超过 1M token 的 Needle-In-a-Haystack(NIH,大海捞针)测试中达到 100% 准确率,超过其他方法或更大规模模型,并创造了新的最优水平(SOTA)。此外,我们的方法在真实世界基准上取得显著性能提升,最大提升达 288%。同时,InfiniRetri 可应用于任何基于 Transformer 的 LLM,无需额外训练,并大幅降低长文本上的推理延迟和计算开销。综上,我们的全面研究展示了 InfiniRetri 的实际应用潜力,并为在无限长度 token 下利用 LLM 自身能力检索信息建立了新范式。代码将在链接中发布。

英文摘要:

Limited by the context window size of Large Language Models(LLMs), handling various tasks with input tokens exceeding the upper limit has been challenging, whether it is a simple direct retrieval task or a complex multi-hop reasoning task. Although various methods have been proposed to enhance the long-context processing capabilities of LLMs, they either incur substantial post-training costs, or require additional tool modules(e.g.,RAG), or have not shown significant improvement in realistic tasks. Our work observes the correlation between the attention distribution and generated answers across each layer, and establishes the attention allocation aligns with retrieval-augmented capabilities through experiments. Drawing on the above insights, we propose a novel method InfiniRetri that leverages the LLMs's own attention information to enable accurate retrieval across inputs of infinitely length. Our evaluations indicate that InfiniRetri achieves 100% accuracy in the Needle-In-a-Haystack(NIH) test over 1M tokens using a 0.5B parameter model, surpassing other method or larger models and setting a new state-of-the-art(SOTA). Moreover, our method achieves significant performance improvements on real-world benchmarks, with a maximum 288% improvement. In addition, InfiniRetri can be applied to any Transformer-based LLMs without additional training and substantially reduces inference latency and compute overhead in long texts. In summary, our comprehensive studies show InfiniRetri's potential for practical applications and creates a paradigm for retrievaling information using LLMs own capabilities under infinite-length tokens. Code will be released in link.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑