arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.16004cs.CLcs.AI
收藏

AgentV-RL: 通过代理验证器扩展奖励建模

AgentV-RL: Scaling Reward Modeling with Agentic Verifier

College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Huazhong University of Science and Technology（华中科技大学）；The University of Hong Kong（香港大学）；ByteDance Seed（字节跳动种子）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身人工智能研究院）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）
Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Ding… 展开作者

Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Dingwei Zhu, Tao Gui, Qi Zhang, Xuanjing Huang

备注：
ACL 2026
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Agentic Verifier框架，通过多轮工具增强的反思过程提升奖励建模效果，实验显示其在并行和顺序TTS任务中表现优异，4B变体超越现有最优ORMs 25.2%。

摘要：

验证器已被证明可通过测试时扩展（TTS）增强大语言模型推理。然而，在复杂领域中验证器面临显著挑战。错误的中间推理可能导致看似合理的解决方案出现假阳性，而缺乏外部语境使验证器在计算或知识密集型任务中不可靠。为解决这些问题，我们提出Agentic Verifier框架，将奖励建模转换为多轮、工具增强的反思过程。我们引入互补的前向和后向代理：一个从前提追溯到结论，另一个将结论重新检查其底层前提。这种双向过程使解决方案评估更加全面、可靠和可解释。为促进实际部署，我们提出AgentV-RL。通过主动探索和强化学习，验证器自主地将工具使用与内部推理交织。大量实验表明，Agentic Verifier在并行和顺序TTS任务中均表现出一致的性能提升。值得注意的是，我们的4B变体超越了现有最优ORMs 25.2%，使其成为代理奖励建模的有前景范式。

英文摘要：

Verifiers have been demonstrated to enhance LLM reasoning via test-time scaling (TTS). Yet, they face significant challenges in complex domains. Error propagation from incorrect intermediate reasoning can lead to false positives for seemingly plausible solutions, while lacking external grounding makes verifiers unreliable on computation or knowledge-intensive tasks. To address these challenges, we propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process. We introduce complementary forward and backward agents: one traces solutions from premises to conclusions, while the other re-checks conclusions against their underlying premises. This bidirectional process enables a comprehensive, reliable, and interpretable assessment of solutions. To facilitate practical deployment, we propose AgentV-RL. Through proactive exploration and reinforcement learning, the verifier autonomously interleaves tool-use with internal reasoning. Extensive experiments show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS. Notably, our 4B variant surpasses state-of-the-art ORMs by 25.2%, positioning it as a promising paradigm for agentic reward modeling.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑