从不可恢复中学习：用于工具集成大语言模型推理的误差局部化策略优化
Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning
AI总结：
ELPO通过误差局部化策略优化提升工具集成大语言模型推理的性能，有效解决稀疏奖励和信用分配问题。
摘要：
工具集成推理（TIR）使LLM代理能够通过规划、工具使用和迭代修订来解决任务，但在这种设置中仅基于结果的强化学习面临稀疏、延迟奖励和弱步级信用分配的问题。在长时间 horizon 的TIR轨迹中，早期不可恢复的错误会决定成功或失败，因此关键在于局部化第一个不可恢复的步骤并利用它进行细粒度的信用分配。我们提出了误差局部化策略优化（ELPO），通过在固定滚动预算下使用二分搜索滚动树来局部化第一个不可恢复的步骤，通过分层优势归因将结果树转换为稳定的学信号，并应用误差局部化的自适应裁剪来加强关键步骤及其后缀的纠正更新。在数学、科学问答和代码执行的TIR基准测试中，ELPO在可比的采样预算下始终优于强大的代理强化学习基线，此外在Pass@K和Major@K扩展、滚动排名质量和工具调用效率方面还有额外的收益。我们的代码将在不久后公开发布。
英文摘要：
Tool-integrated reasoning (TIR) enables LLM agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
