arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.02004cs.CL
收藏

基于小波的长上下文位置表示方法

Wavelet-based Positional Representation for Long Context

NTT Human Informatics Laboratories, NTT Corporation(NTT公司NTT人类信息学研究所)

Yui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito

AI总结:

本文针对长上下文位置外推问题,提出基于小波变换的多尺度位置表示方法,在不限制注意力场的前提下提升模型在短长上下文中的性能。

摘要:

在大规模语言模型领域,当序列外推至超过最大允许长度时会出现一个重大挑战。这是因为模型的位置嵌入机制仅限于训练期间遇到的位置,从而无法有效表示更长序列中的位置。我们分析了面向长上下文的传统位置编码方法,发现以下特征。(1)当将表示维度视为时间轴时,旋转位置嵌入(RoPE)可被解释为使用类 Haar 小波的受限小波变换。然而,由于它仅使用固定的尺度参数,并未充分利用小波变换通过多尺度(窗口大小)捕捉非平稳信号细微变化的优势。这一局限性可以解释 RoPE 在外推中表现不佳的原因。(2)先前研究以及我们自己的分析表明,带线性偏置的注意力(ALiBi)的功能类似于使用不同大小窗口的窗口化注意力。然而,由于它限制了模型的感受野,在捕捉深层依赖方面存在局限。基于这些见解,我们提出了一种新的位置表示方法,该方法在不限制模型注意力场的情况下,利用小波变换捕捉多尺度(即窗口大小)。实验结果表明,这种新方法在短上下文和长上下文中均提升了模型性能。特别是,我们的方法能够在不限制模型注意力场的情况下外推位置信息。

英文摘要:

In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. This is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences. We analyzed conventional position encoding methods for long contexts and found the following characteristics. (1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. However, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). This limitation could explain why RoPE performs poorly in extrapolation. (2) Previous research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes. However, it has limitations in capturing deep dependencies because it restricts the receptive field of the model. From these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field. Experimental results show that this new method improves the performance of the model in both short and long contexts. In particular, our method allows extrapolation of position information without limiting the model's attention field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑