SANE：用于稳定极端上下文Delta规则模型的状态异常中和
SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
- College of Science, Shantou University（汕头大学理学院）
- Yuanshi Intelligence（元氏智能）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对Delta规则循环模型在极端上下文外推下的局部范数爆炸问题，提出SANE方法，在安全阈值下可稳定长序列推理且性能不下降，过松阈值会导致推理能力丧失。
AI中文摘要：
Delta规则循环模型维护固定大小的状态，支持O(1)推理内存，但在极端上下文外推下可能变得不稳定。通过跟踪RWKV-7在长达1亿token的序列上的表现，我们经验性地识别出一种独特的失效模式：在相对稀疏的基底上发生局部范数爆炸，而非全局状态饱和。对循环更新的分析表明，持续衰减使更新微弱的条目保持较小值，而不均匀的注入则让少数通道积累极值。基于这一诊断，我们提出状态异常中和（SANE），该方法在块边界处应用自适应tanh压缩，同时保留块内并行结构。在安全阈值范围（3≤α≤5）内，SANE在11个短上下文推理基准上与基线匹配，无统计学意义上的性能下降。在超过训练长度24000倍以上的1亿token前缀后，SANE仍保持功能推理能力（得分33.46至35.56），而基线则遭遇数值溢出。相比之下，过松的阈值（α≥8）虽保持数值稳定，但完全丧失推理能力，表明仅数值稳定无法保证功能推理，揭示了状态压缩中存在容量-稳定性权衡。
英文摘要：
Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le α\le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($α\ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.
