arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01591cs.AI
收藏

ThinkTwice:联合优化大型语言模型以进行推理和自我完善

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Department of Computer Science, University of Toronto(多伦多大学计算机科学系);Coolwei AI Lab(Coolwei AI实验室)

Difan Jiao, Qianfeng Wen, Blair Yang, Zhenwei Tang, Ashton Anderson

AI总结:

ThinkTwice通过两阶段框架联合优化LLM,提升推理和自我完善性能,基于GRPO方法,在多个数学推理基准上优于现有基线。

摘要:

我们介绍了ThinkTwice,一种简单的两阶段框架,基于Group Relative Policy Optimization (GRPO),用于联合优化LLM以解决推理问题并完善答案。在每对训练步骤中,ThinkTwice首先优化模型解决推理问题,然后优化其自身解决方案的完善,使用相同的二进制正确性奖励,无需正确性信号或批评注释。在五个数学推理基准和两个模型家族(包括Qwen3-4B和Olmo3-7B)上,ThinkTwice显著提高了推理和完善性能。具体而言,在Qwen3-4B上,ThinkTwice在自我完善前比GRPO在AIME上高出5个百分点,在一次自我完善步骤后高出11.5个百分点,以pass@4衡量。对ThinkTwice训练动态的分析揭示了隐含的修正-强化课程:完善主要在训练早期纠正错误,并自然转向保留已正确解决方案,产生更修正的奖励信号。我们的工作确立了推理和自我完善联合训练作为RLVR中一种原则性且有效的方法。

英文摘要:

We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑