arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.04787cs.CLcs.AI
收藏

ReSI-Math:探索可恢复能力边界以提升高效推理能力

ReSI-Math: Exploring Recoverable Capability Frontiers for Efficient Reasoning Improvement

Capital Normal University(首都师范大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学)

Yao Cui, Zhen Yang, Yuandong Wang, Yangfu Zhu, Tianyu Wo, Zhenzhou Shao

AI总结:

本研究提出ReSI-Math框架,通过探索大语言模型的可恢复能力边界,结合自评判引导进化搜索与顺序RLVR,提升了Qwen3-VL-8B在MATH500和WE-MATH上的推理性能。

摘要:

带可验证奖励的强化学习(RLVR)已成为提升大语言模型数学推理能力的有效方法,但仍存在一个基本问题:当模型无法解决某一问题时,是否真的缺乏相关底层能力?本研究表明,直接失败并不一定意味着能力不足,部分推理失败可通过探索模型自身轨迹恢复,由此揭示了可恢复能力边界:这是一个依赖策略的区域,直接解码无法可靠生成解决方案,但通过轨迹级搜索可引出解决方案。基于该观察,我们提出ReSI-Math框架,用于探索可恢复能力边界,并将恢复的能力转化为数据高效的RLVR监督信号。ReSI-Math采用自评判引导的进化搜索,在固定策略下识别可恢复失败,并在策略演化时自适应选择边界问题用于顺序RLVR。在Qwen3-VL-8B上,正确-错误轨迹交叉在621次尝试中产生272个经验证器确认的正确后代(成功率43.8%)。经过三个顺序的单次RLVR阶段后,MATH500和WE-MATH的平均得分从78.71提升至80.16。在匹配的单问题RLVR设置下,随机选择的得分仅为53.31,而边界引导的Q1得分达54.16。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has become an effective approach for improving mathematical reasoning in large language models. However, a fundamental question remains: when a model fails to solve a problem, does it truly lack the underlying capability? In this work, we reveal that direct failure does not necessarily imply inability. Some reasoning failures can be recovered by exploring the model's own trajectories, exposing a recoverable capability frontier: a policy-dependent region where solutions are not reliably generated through direct decoding but can be elicited through trajectory-level search. Based on this observation, we introduce ReSI-Math, a framework that explores recoverable capability frontiers and converts recovered capabilities into data-efficient RLVR supervision. ReSI-Math uses Self-Judge-guided evolutionary search to identify recoverable failures under a fixed policy and adaptively selects frontier problems for sequential RLVR as the policy evolves. On Qwen3-VL-8B, correct-incorrect trajectory crossover produces 272 verifier-confirmed correct offspring from 621 attempts (43.8%). After three sequential one-shot RLVR stages, the MATH500 and WE-MATH average improves from 78.71 to 80.16. Under matched single-problem RLVR, random selection reaches only 53.31 versus 54.16 for frontier-guided Q1.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑