arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.06113cs.AI
收藏

迈向无限长度外推:一种统一方法

Towards Infinite Length Extrapolation: A Unified Approach

Department of Computer Science(计算机科学系);Indian Institute of Science(印度科学研究院)

Nitin Vetcha

AI总结:

本文提出了一种统一框架,通过自适应位置编码方法解决长序列处理中的限制,实现了无限上下文外推。

摘要:

大型语言模型(LLMs)已经革新了自然语言处理,但它们处理长序列的能力从根本上受到训练期间上下文窗口大小的限制。现有的长度外推方法往往面临性能下降或计算效率低下的问题。因此,我们使用了一个统一的框架,将位置编码方法重新解释为将注意力分数分解为乘法变换和加性偏差的组合。这种视角不仅涵盖了流行的相对位置嵌入和注意力偏差调节方法,还揭示了这些方法在处理长距离依赖性方面的固有局限性。为了解决这些不足,受我们框架的启发,我们引入了自适应位置编码(APE),它利用自适应频率调制和一个精心设计的衰减偏差,该偏差结合了线性、对数和平方根项。我们的理论分析建立了无限上下文外推的条件,确保softmax归一化在无界序列上保持良好定义,同时保持长距离相关性、熵有界性和梯度位置敏感性。我们通过在TinyStories数据集上的实验案例研究以及一个新合成的数据集Long Tiny Stories(包含长达32,000字的故事)来验证我们的主张。相关代码、数据集和模型权重可在https://anonymous.4open.science/r/Check-2DAD/上获取。

英文摘要:

Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To address these shortcomings, motivated by our framework, we introduce Adaptive Positional Encoding (APE), which leverages adaptive frequency modulation and an intricately designed decay bias that incorporates linear, logarithmic, and square-root terms. Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax normalization remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity. We substantiate our claims with an experimental case study on TinyStories dataset as well as a new synthetic dataset, \emph{Long Tiny Stories} featuring stories up to 32,000 words. Relevant code, dataset and model weights are available at https://anonymous.4open.science/r/Check-2DAD/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑