MrRoPE: 混基 radix 旋转位置嵌入
MrRoPE: Mixed-radix Rotary Position Embedding
AI总结:
MrRoPE通过混合基数转换策略,统一了RoPE扩展方法,实现了无需微调的长序列泛化,提升了编码长度上限和检索准确性。
摘要:
旋转位置嵌入(RoPE)扩展指的是对旋转位置嵌入方案进行修改或泛化,以处理比预训练期间遇到的更长的序列。然而,当前的扩展策略高度多样化,缺乏统一的理论基础。在本文中,我们提出了MrRoPE(混合基RoPE),一种基于基数系统转换视角的通用编码公式,它优雅地将各种RoPE扩展方法作为不同的基数转换策略统一起来。基于这一理论,我们引入了两种无需训练的扩展方法,MrRoPE-Uni和MrRoPE-Pro,分别利用统一和渐进的基数转换策略,以实现'训练短,测试长'的泛化。无需微调,MrRoPE-Pro在128K上下文Needle-in-a-Haystack测试中保持超过85%的召回率,并在Infinite-Bench检索和对话子集上实现了比YaRN更高的准确性。理论分析证实,MrRoPE-Pro有效提高了RoPE可达到编码长度的上限,这进一步验证了我们理论和方法的可靠性和实用性。
英文摘要:
Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose MrRoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, MrRoPE-Uni and MrRoPE-Pro, which leverage uniform and progressive radix conversion strategies, respectively, to achieve 'train short, test long' generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN's accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。