最近,人们开发了许多方法来扩展预训练大型语言模型(LLMs)的上下文长度,但这些方法通常需要在目标长度($\gg4K$)下进行微调,并且难以有效利用上下文中间部分的信息。为了解决这些问题,我们提出了带有高斯中间采样的连续性-相对性索引($\ exttt{CREAM}$),它通过操纵位置索引来插值位置编码。除了简单之外,$\ exttt{CREAM}$在训练上也很高效:它只需要在预训练的上下文窗口(例如,Llama 2-4K)进行微调,就可以将LLMs扩展到更长的目标上下文长度(例如,256K)。为了确保模型更加关注中间的信息,我们引入了截断高斯分布,以鼓励在微调期间从上下文的中间部分进行采样,从而缓解长上下文LLMs面临的“迷失在中间”问题。实验结果表明,$\ exttt{CREAM}$成功地将LLMs扩展到目标长度,适用于$\ exttt{Llama2-7B}$的Base和Chat版本,实现了“Never Miss A Beat”。我们的代码已在https://github.com/bigai-nlco/cream公开发布。
英文摘要:
Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle ($\texttt{CREAM}$), which interpolates positional encodings by manipulating position indices. Apart from being simple, $\texttt{CREAM}$ is training-efficient: it only requires fine-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs to a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the "Lost-in-the-Middle" problem faced by long-context LLMs. Experimental results show that $\texttt{CREAM}$ successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with "Never Miss A Beat". Our code is publicly available at https://github.com/bigai-nlco/cream.
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
LongHeads: Multi-Head Attention is Secretly a Long Context Processor
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);International Human Phenome Institutes, Shanghai, China(国际人类表型组研究院(上海))
GPT-5 vs Other LLMs in Long Short-Context Performance
Bernoulli Institute RUG(伯努利研究所);ISRC, Khazar University(ISRC、哈扎尔大学);Department of Computer Engineering University of Guilan(计算机工程系、盖兰大学);QCE Department TU Delft(QCE部门、代尔夫特理工大学)
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Instituto Superior Técnico and INESC-ID, University of Lisbon(里斯本大学高级技术学院与INESC-ID研究所);NOVA LINCS, NOVA School of Science and Technology(NOVA科技大学NOVA LINCS研究所)