发表机构
Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
WavePrune通过将RoPE每个通道限制在第一个旋转周期内，消除位置混叠干扰，无需微调即可提升长上下文性能，并利用稀疏性实现推理加速，证明RoPE周期结构在首周期外冗余。
AI 中文摘要
旋转位置编码（RoPE）通过以通道特定频率旋转查询向量和键向量的每个二维通道来编码令牌位置，使得注意力对数在位置共同偏移时保持不变。然而，这种旋转具有周期性，导致位置混叠，即相隔一个完整旋转周期的相对位置变得难以区分。为了解决这一问题，我们提出了WavePrune，它将每个通道限制在其第一个旋转周期内。我们证明，这消除了由位置混叠引起的注意力图中的干扰，并提升了整体长上下文性能。具体而言，WavePrune在我们测试的五个模型中的四个上，无需任何额外微调即可提高HELMET分数（例如，在Qwen3-8B上从35.7提高到40.0）。当从头开始预训练模型时，WavePrune在外推长度上也比不使用WavePrune的预训练实现了更低的验证损失。由于WavePrune将每个通道限制在一个滑动窗口内，它引入了细粒度的稀疏性，我们与硬件对齐的CUDA内核利用了这一特性，在32K上下文下实现了相对于FlashAttention-2的1.15倍预填充和1.24倍解码加速。综合这些结果，我们表明RoPE的周期结构（被广泛认为是必不可少的）在第一个旋转周期之外很大程度上是冗余的。
英文摘要
Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.
