AI 中文总结
HySPE通过辛对剪切实现双曲位置编码，在保持数值稳定和低延迟的同时，显著提升Transformer的长序列外推能力。
AI 中文摘要
我们引入了双曲辛位置编码（HySPE），将位置注意力建立在非紧致辛变换之上。经典的旋转位置编码（RoPE）通过旋转参数化$\text{Sp}(2,\mathbb{R})$的紧致椭圆分支，而HySPE则通过对剪切的双重对称组合实现了其双曲分支，从而在每个通道对上产生具有两个谱衰减率的共形辛收缩。为了消除朴素绝对分解中固有的指数表示漂移，我们在其不变特征基中对该算子进行对角化，并引入带有自适应居中执行的块级坐标重基。这保证了与长度无关的数值界限，同时匹配缓存的RoPE前向延迟（在RTX 4090上为7.21毫秒）。在TinyShakespeare数据集上，HySPE-UltraLong在高达16倍零样本外推（$L=4096$）时保持不变的困惑度4.810，而RoPE则退化至131.198。当扩展到WikiText-103上的51M参数子词Transformer（$L_{\text{train}}=512$）时，HySPE在域内与RoPE紧密匹配，同时稳健地外推至长度8192，将尾部困惑度相对于RoPE降低了83.9%。尽管这些受控实验确立了HySPE的外推稳健性和数值稳定性，但评估其在大型基础模型上的扩展行为仍是未来研究的重要方向。
英文摘要
We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of $\Sp(2,\R)$ via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21\,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to $16\times$ zero-shot extrapolation ($L=4096$), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 ($L_{\text{train}}=512$), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9\% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.
Comments11 pages
