VERIFY-RL: 在数学推理中用于强化学习的可验证递归分解
VERIFY-RL: Verifiable Recursive Decomposition for Reinforcement Learning in Mathematical Reasoning
AI总结：
Verify-RL通过可验证的递归分解提升数学推理强化学习的准确性和稳定性。
摘要：
训练语言模型解决复杂数学问题受益于课程学习，逐步训练更简单的子问题。然而，现有分解方法往往具有启发性，没有保证子问题更简单，解决它们有助于父任务，或它们的关系有数学基础。我们观察到，符号微分提供了一种自然的可验证分解结构：微积分规则明确定义了如何将表达式减少为具有可证明性质的更简单的组成部分。我们引入了Verify-RL框架，其中每个父-子分解都满足三个可验证条件：严格递减的结构复杂度、解包含性和形式规则推导。与启发式方法不同，其中许多分解无效，我们的属性允许通过符号计算自动验证，实现“通过构造验证”。实验表明，消除无效分解带来了显著收益，最困难问题的准确性从32%增加到68%，总体相对改进达40%。
英文摘要：
Training language models to solve complex mathematical problems benefits from curriculum learning progressively training on simpler subproblems. However, existing decomposition methods are often heuristic, offering no guarantees that subproblems are simpler, that solving them aids the parent task, or that their relationships are mathematically grounded. We observe that symbolic differentiation provides a natural structure for verified decomposition: calculus rules explicitly define how expressions reduce to simpler components with provable properties. We introduce Verify-RL, a framework where every parent-child decomposition satisfies three verifiable conditions: strictly decreasing structural complexity, solution containment, and formal rule derivation. Unlike heuristic methods where a significant fraction of decompositions are invalid our properties admit automatic verification through symbolic computation, achieving "verification by construction" Experiments demonstrate that eliminating invalid decompositions yields sizable gains, accuracy on the hardest problems more than doubles from 32% to 68%, with a 40% relative improvement overall.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
