共振 RoPE:提升大语言模型的上下文长度泛化能力
Resonance RoPE: Improving Context Length Generalization of Large Language Models
AI总结:
本文提出共振 RoPE 方法,通过改进 RoPE 对分布外位置的插值来提升大语言模型在短训练长测试场景下的长度泛化能力,并引入 PosGen 基准进行细粒度分析,实验证明其能有效提升模型性能。
摘要:
本文针对配备旋转位置嵌入(RoPE)的大语言模型(LLMs)在短序列训练、长序列测试(TSTL)场景中所面临的挑战,即模型在较短序列上预训练后,难以处理较长序列中的分布外(OOD)token 位置。我们提出共振 RoPE,这是一种新方法,旨在通过改进 RoPE 特征在 OOD 位置上的插值方式来缩小 TSTL 场景中的泛化差距,在不增加额外在线计算成本的情况下显著提升模型性能。此外,我们提出 PosGen,这是一个新的合成基准,专门为 TSTL 场景中的细粒度行为分析而设计,旨在将长上下文上 token 生成难度持续增加的问题与识别新 token 位置的挑战分离开来。我们在合成任务上的实验表明,应用共振 RoPE 后,Transformer 能够更好且更稳健地识别 OOD 位置。我们广泛的大语言模型实验还表明,将共振 RoPE 应用于当前最先进的 RoPE 缩放方法 YaRN 后,在上游语言建模任务和多种下游长文本应用中均表现出优越的性能。
英文摘要:
This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream language modeling tasks and a variety of downstream long-text applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。