高维旋转位置嵌入
Higher-Dimensional Rotary Position Embedding
浏览论文内容
中文总结 AI 辅助
针对RoPE跨通道混合与鲁棒性不足的问题，提出HD-RoPE，通过扩展至高维旋转并引入Paley-I正交基，在无额外参数的情况下提升性能，获基准测试显著改善。
中文摘要 AI 辅助
Transformer在长上下文建模中大多依赖位置嵌入机制。旋转位置嵌入（RoPE）通过独立的二维旋转嵌入位置信息，在自注意力中形成相对位置项。然而，其成对、基于块且解耦的结构限制了跨通道的深度混合与鲁棒性。我们提出HD-RoPE，将RoPE从独立二维旋转扩展至高维旋转，并引入Paley-I正交基，以在每个旋转子空间内实现平衡、各向同性且密集的相位混合。这在保持正交稳定性与相对位置闭合性的同时，显著增强了通道耦合与旋转自由度。此外，HD-RoPE易于优化以提升工程效率，且无需引入额外可训练参数。我们开展了广泛评估，结果表明HD-RoPE在各类流行基准及长短上下文场景中，均较标准RoPE实现了显著的性能提升。
英文摘要
Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.
发表机构
- The Chinese University of Hong Kong（香港中文大学）
- Tencent Hunyuan（腾讯混元）
- Tsinghua University（清华大学）
- University of Macau（澳门大学）
机构由 AI 辅助整理，请以论文原文为准。
