在大规模语言模型领域,当序列外推至超过最大允许长度时会出现一个重大挑战。这是因为模型的位置嵌入机制仅限于训练期间遇到的位置,从而无法有效表示更长序列中的位置。我们分析了面向长上下文的传统位置编码方法,发现以下特征。(1)当将表示维度视为时间轴时,旋转位置嵌入(RoPE)可被解释为使用类 Haar 小波的受限小波变换。然而,由于它仅使用固定的尺度参数,并未充分利用小波变换通过多尺度(窗口大小)捕捉非平稳信号细微变化的优势。这一局限性可以解释 RoPE 在外推中表现不佳的原因。(2)先前研究以及我们自己的分析表明,带线性偏置的注意力(ALiBi)的功能类似于使用不同大小窗口的窗口化注意力。然而,由于它限制了模型的感受野,在捕捉深层依赖方面存在局限。基于这些见解,我们提出了一种新的位置表示方法,该方法在不限制模型注意力场的情况下,利用小波变换捕捉多尺度(即窗口大小)。实验结果表明,这种新方法在短上下文和长上下文中均提升了模型性能。特别是,我们的方法能够在不限制模型注意力场的情况下外推位置信息。
英文摘要:
In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. This is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences. We analyzed conventional position encoding methods for long contexts and found the following characteristics. (1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. However, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). This limitation could explain why RoPE performs poorly in extrapolation. (2) Previous research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes. However, it has limitations in capturing deep dependencies because it restricts the receptive field of the model. From these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field. Experimental results show that this new method improves the performance of the model in both short and long contexts. In particular, our method allows extrapolation of position information without limiting the model's attention field.
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Bytedance(字节跳动);Zhejiang Lab(浙江省实验室);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室)
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
MALTA – Machine Learning Theory and Applications Lab, PUCRS – Porto Alegre, Brazil(MALTA机器学习理论与应用实验室,PUCRS-波士顿实验室,巴西);Kunumi Institute, Brazil(昆米研究所,巴西)
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Baichuan Inc.(百川公司);School of Information, Renmin University of China(中国人民大学信息学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系)
Layer-wise Positional Bias in Short-Context Language Modeling
Tehran Institute for Advanced Studies, Khatam University, Iran(德黑兰高级研究院,卡坦大学,伊朗);School of Electrical and Computer Engineering, University of Tehran, Iran(德黑兰理工大学电气与计算机工程学院,伊朗)
Cascaded Semantic and Positional Self-Attention Network for Document Classification
College of Internet of Things Engineering, Hohai University(河海大学物联网工程学院);Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学类脑智能科学与技术研究院);Department of Computer & Information Sciences, Temple University(天普大学计算机与信息科学系)
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
Nanjing University of Science and Technology(南京理工大学);Alibaba Group(阿里巴巴集团);Huzhou Normal University(湖州师范学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Microsoft Corporation(微软公司);School of Computer Science, Peking University(北京大学计算机学院);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)