arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09883cs.LGcs.AI
收藏

TD-Grokking:通过训练时分解从零奖励问题中学习

TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition

Ningyuan Xi 1,2(西宁元 1,2);Hao Xu 3(许浩 3);Hongsheng Xin 3(辛红生 3);Ning Miao 1,2, †(苗宁 1,2, †)

Ningyuan Xi, Hao Xu, Hongsheng Xin, Ning Miao

AI总结:

针对强化学习在零奖励问题上无法提供优化信号的问题,提出训练时分解框架TD-Grokking,将难解问题递归分解为可验证子问题,在数学和医疗任务上优于基线方法。

摘要:

大型语言模型(LLMs)在推理任务上取得了显著进展,这主要归功于后训练范式,特别是基于可验证奖励的强化学习(RLVR)。然而,一个关键瓶颈依然存在:RLVR在极具挑战性的零奖励问题上失败,因为所有采样的推理轨迹都产生统一失败的结果,无法提供优化信号来驱动模型改进。先前解决这一限制的努力,如密集过程监督、部分奖励分配或前缀引导探索,受到固有任务约束的限制,或者未能完全赋予策略模型解决原始难解问题所需的能力。为了解决这个问题,我们提出了TD-Grokking,一个针对零奖励问题的训练时分解框架。它递归地将难解的根问题分解为自包含、可验证的子问题,形成层次树,其中可解的叶子节点提供非零奖励。在数学和医疗任务上的评估表明,TD-Grokking优于普通的GRPO以及所有基线方法。结合详细分析,这些结果证实训练时分解有效地将零奖励示例转化为可用的训练信号,从而实现一致的性能提升。我们的代码和数据集可在以下网址获取:https://this URL。

英文摘要:

Large language models (LLMs) have made remarkable progress in reasoning tasks, largely driven by post-training paradigms, especially reinforcement learning with verifiable rewards (RLVR). However, a critical bottleneck persists: RLVR fails on highly challenging zero-reward problems, where all sampled reasoning trajectories yield uniformly failed outcomes, providing no optimization signal to drive model improvement. Prior efforts to address this limitation, such as dense process supervision, partial reward assignment, or prefix-guided exploration, suffer from inherent task constraints or do not fully equip the policy model with the capabilities necessary to solve the original intractable problems. To address this, we propose TD-Grokking, a training-time decomposition framework for zero-reward problems. It recursively decomposes intractable root problems into self-contained, verifiable subproblems, forming hierarchical trees where solvable leaves provide non-zero rewards. Evaluations on mathematical and medical tasks show that TD-Grokking outperforms vanilla GRPO as well as all baseline approaches. Together with detailed analysis, these results confirm that training-time decomposition effectively converts zero-reward examples into usable training signals, enabling consistent performance gains. Our code and datasets are available at https://anonymous.4open.science/r/TD-Grokking-6567/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑