arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.04797cs.CLcs.LG
收藏

居中有效:通过即插即用位置编码让语言模型更好地利用长上下文

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Oxford(牛津大学);Microsoft(微软公司);Carnegie Mellon University(卡内基梅隆大学)

Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

AI总结:

针对大语言模型“中间迷失”、难以识别长上下文中间相关信息的问题,提出即插即用的多尺度位置编码Ms-PoE,通过位置索引重缩放与注意力头差异化缩放,无需微调即可提升模型长上下文处理能力,在Zero-SCROLLS基准上最高提升3.8的平均准确率。

摘要:

本文旨在攻克大语言模型(LLM)面临的“中间迷失”难题。尽管近期进展已成功让大语言模型能对至多400万token执行稳定的语言建模,但多数大语言模型在识别位于上下文中间的相关信息时仍存在持续性困难,这一问题尚未得到充分解决。为解决该问题,本文提出多尺度位置编码(Multi-scale Positional Encoding,Ms-PoE),这是一种简单有效的即插即用方法,无需微调也不会引入任何额外开销,就能提升大语言模型处理上下文中间相关信息的能力。Ms-PoE利用位置索引重缩放来缓解RoPE带来的长程衰减效应,同时为不同注意力头精心分配不同的缩放比例,以保留预训练阶段学到的核心知识,形成从短距离到长距离的多尺度上下文融合。针对多种大语言模型开展的大量实验证明了该方法的有效性。值得注意的是,在Zero-SCROLLS基准测试上,Ms-PoE相较原始大语言模型实现了最高达3.8的平均准确率提升。代码可在https://github.com/VITA-Group/Ms-PoE获取。

英文摘要:

This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑