arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12078cs.AIcs.CL
收藏

微小递归推理与Mamba-2注意力混合

Tiny Recursive Reasoning with Mamba-2 Attention Hybrid

Intercom Dublin, Ireland(Intercom都柏林,爱尔兰)

Wenlong Wang, Fergal Reid

AI总结:

本文探讨将Mamba-2混合操作符替换TRM中的Transformer块,验证其在递归框架下保持推理能力,提升抽象推理任务表现。

摘要:

最近关于递归推理模型如TRM的研究表明,微小网络(7M参数)可通过隐含递归在抽象推理任务中实现强性能,通过隐藏表征空间中的迭代细化实现。这引发了关于操作符选择的自然问题:Mamba-2的状态空间递归本身是一种迭代细化形式,使其成为递归推理的自然候选者——但将Mamba-2引入递归框架是否能保持推理能力?我们通过将TRM中的Transformer块替换为Mamba-2混合操作符进行研究,保持参数量一致(6.83M vs 6.86M参数)。在ARC-AGI-1上,混合模型在pass@2(官方指标)上提升+2.0%(45.88% vs 43.88%),并在更高K值时持续表现更优(pass@100时+4.75%),同时保持pass@1性能一致。这表明模型生成正确解决方案的可靠性提升,同时保持top-1选择率相似。结果验证了Mamba-2混合操作符在递归框架内保持推理能力,确立了基于SSM的操作符作为递归操作符设计空间中的可行候选,并为理解递归推理的最佳混合策略迈出第一步。

英文摘要:

Recent work on recursive reasoning models like TRM demonstrates that tiny networks (7M parameters) can achieve strong performance on abstract reasoning tasks through latent recursion -- iterative refinement in hidden representation space without emitting intermediate tokens. This raises a natural question about operator choice: Mamba-2's state space recurrence is itself a form of iterative refinement, making it a natural candidate for recursive reasoning -- but does introducing Mamba-2 into the recursive scaffold preserve reasoning capability? We investigate this by replacing the Transformer blocks in TRM with Mamba-2 hybrid operators while maintaining parameter parity (6.83M vs 6.86M parameters). On ARC-AGI-1, we find that the hybrid improves pass@2 (the official metric) by +2.0\% (45.88\% vs 43.88\%) and consistently outperforms at higher K values (+4.75\% at pass@100), whilst maintaining pass@1 parity. This suggests improved candidate coverage -- the model generates correct solutions more reliably -- with similar top-1 selection. Our results validate that Mamba-2 hybrid operators preserve reasoning capability within the recursive scaffold, establishing SSM-based operators as viable candidates in the recursive operator design space and taking a first step towards understanding the best mixing strategies for recursive reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑