arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.16636cs.CLcs.AI
收藏

SuPER-Mamba:用于Mamba中扩展上下文路由的惊讶度优先策略

SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba

School of Information Technology and Electrical Engineering(信息技术与电气工程学院);The University of Queensland(昆士兰大学)

Zhetong Zhang, Gayan K Kulatilleke

AI总结:

SuPER-Mamba是一种无需训练的框架,利用惊讶度作为内在信号优化预训练Mamba的长距离状态路由,在LongBench-E基准上使平均得分大幅提升,验证了其有效性。

摘要:

状态空间模型可为长序列提供高效的线性复杂度处理,但随着上下文长度增加,其性能常出现下降。我们提出SuPER-Mamba,这是一种无需训练的框架,利用惊讶度(Surprisal)作为模型内在信号,在预训练Mamba模型中进行长距离状态路由。SuPER-Mamba通过下一个token预测计算位置级惊讶度,经局部池化聚合相邻响应,再用所得分数优先进行全局通道更新,无需依赖查询的评分或额外训练。我们在LongBench-E基准上的三个状态空间模型上评估SuPER-Mamba,结果显示其平均得分从8.06提升至15.69,较基础模型实现94.7%的相对增益,较现有最优基线提升13.1%。消融实验和长度分析进一步验证了所提路由框架的有效性。

英文摘要:

State space models provide efficient linear-complexity processing for long sequences, but their performance frequently degrades as context length grows. We propose SuPER-Mamba, a training-free framework that uses Surprisal as a model-intrinsic signal for long-range state routing in pretrained Mamba models. SuPER-Mamba computes position-wise Surprisal from next-token predictions, aggregates neighboring responses through Local Pooling, and uses the resulting scores to prioritize Global-Channel updates without query-dependent scoring or additional training. We evaluate SuPER-Mamba on three state-space models across the LongBench-E benchmark. The results show that SuPER-Mamba improves the average score from 8.06 to 15.69, corresponding to a 94.7% relative gain over the Base Models and a 13.1% improvement over the state-of-the-art prior baseline. Ablation and length-wise analyses further support the effectiveness of the proposed routing framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑