反周期位置编码：莫比乌斯边界条件使上下文检索可靠
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
浏览论文内容
中文总结 AI 辅助
研究提出基于反周期频率阶梯的莫比乌斯旋转位置编码，经数值验证和模型预训练，发现其能使上下文检索可靠，混合困惑度不变，不同频率阶梯效果有差异，还探讨了交换频率表及NoPE的影响，为单针检索提供保障。
中文摘要 AI 辅助
莫比乌斯旋转位置编码（Möbius RoPE）基于反周期频率阶梯\(\theta_i=\pi(2i + 1)/N\)构建。每个旋转平面在训练上下文中以\(\pi\)的奇数倍前进，位置全纯性为\(-1\)，序列两端通过封闭形式的狄利克雷“偶极子”确定性耦合，这是位置编码中的首个反周期边界条件。通过数值验证理论至约\(10^{-6}\)，预训练了48个模型。混合困惑度不变，但上下文512时的针在草垛检索变得可靠，非周期阶梯无此效果，周期阶梯只有部分效果。交换训练模型频率表回标准RoPE会使检索崩溃。NoPE在短上下文更可靠但困惑度增加且外推最差，只有反周期混合能兼顾困惑度和可靠性。该效果局限于训练窗口内的单针检索，一行频率交换可提供防检索种子抽奖的零成本保障。
英文摘要
Möbius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder $θ_i=π(2i+1)/N$: every rotation plane advances by an odd multiple of $π$ across the training context, so the positional holonomy is $-1$ and the two ends of the sequence are deterministically coupled through a closed-form Dirichlet "dipole"; to our knowledge this is the first anti-periodic boundary condition in positional encoding. We verify the theory numerically to $\sim 10^{-6}$ and pretrain 48 models spanning six 160M-class and three 410M-class arms (2B FineWeb-Edu tokens each; the hybrid arm puts Möbius frequencies on 25% of heads). Hybrid perplexity is unchanged (29.66 vs. 29.72), but needle-in-a-haystack retrieval becomes reliable: $90.3\pm5.7\%$ versus $63.3\pm31.4\%$ at context 512 ($n=6$ seeds), observed worst seed 86% versus 14%, robust variance tests $p=0.013$-$0.029$ (unadjusted), recurring at 410M (Levene $p=0.040$). Matched controls isolate the mechanism: an aperiodic ladder in the same frequency band reproduces none of the effect, and a periodic (holonomy $+1$) ladder only a fraction. Swapping trained models' frequency table back to standard RoPE (weights frozen) collapses retrieval, with damage concentrated on far needles: trained models depend on this long-range geometry. A NoPE arm is even more reliable at short context but pays a 13% perplexity tax and extrapolates worst; only the anti-periodic hybrid pairs baseline perplexity with a high reliability floor. The effect is scoped to single-needle retrieval within the training window; a one-line frequency swap thus provides zero-cost insurance against the retrieval seed lottery.
发表机构
- JRTI（联合研究与技术研究所）
机构由 AI 辅助整理，请以论文原文为准。
