arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07138cs.CL
收藏

一种通过聚焦中间的位置编码扩展长上下文的高效方法

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding

BIGAI(北京智源人工智能研究院);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)

Tong Wu, Yanpeng Zhao, Zilong Zheng

AI总结:

提出CREAM方法,通过操纵位置索引和截断高斯采样,在预训练窗口微调即可将LLM上下文扩展至256K,有效缓解“迷失在中间”问题。

摘要:

最近,人们开发了许多方法来扩展预训练大型语言模型(LLMs)的上下文长度,但这些方法通常需要在目标长度($\gg4K$)下进行微调,并且难以有效利用上下文中间部分的信息。为了解决这些问题,我们提出了带有高斯中间采样的连续性-相对性索引($\ exttt{CREAM}$),它通过操纵位置索引来插值位置编码。除了简单之外,$\ exttt{CREAM}$在训练上也很高效:它只需要在预训练的上下文窗口(例如,Llama 2-4K)进行微调,就可以将LLMs扩展到更长的目标上下文长度(例如,256K)。为了确保模型更加关注中间的信息,我们引入了截断高斯分布,以鼓励在微调期间从上下文的中间部分进行采样,从而缓解长上下文LLMs面临的“迷失在中间”问题。实验结果表明,$\ exttt{CREAM}$成功地将LLMs扩展到目标长度,适用于$\ exttt{Llama2-7B}$的Base和Chat版本,实现了“Never Miss A Beat”。我们的代码已在https://github.com/bigai-nlco/cream公开发布。

英文摘要:

Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle ($\texttt{CREAM}$), which interpolates positional encodings by manipulating position indices. Apart from being simple, $\texttt{CREAM}$ is training-efficient: it only requires fine-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs to a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the "Lost-in-the-Middle" problem faced by long-context LLMs. Experimental results show that $\texttt{CREAM}$ successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with "Never Miss A Beat". Our code is publicly available at https://github.com/bigai-nlco/cream.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑