ToolRLA: 工具集成代理的乘法奖励分解
ToolRLA: Multiplicative Reward Decomposition for Tool-Integrated Agents
AI总结:
ToolRLA通过乘法奖励分解提升工具集成代理的任务完成率、减少错误与违规,实现高效领域特定部署。
摘要:
集成推理与API调用的工具代理在复杂任务中具有潜力,但要在高风险、领域特定的部署中对齐仍具挑战:现有强化学习方法依赖于粗略的二元奖励,无法区分工具选择错误与参数格式错误。我们提出了ToolRLA,一种针对领域特定工具代理的三阶段后训练流程(SFT -> GRPO -> DPO)。核心贡献是一种细粒度奖励函数,具有跨越四个维度的乘法正确性分解——格式有效性、工具选择、参数准确性以及合规性——该函数将领域优先级顺序作为归纳偏置编码在奖励景观中。在金融顾问助手(80多名顾问,1200多条每日查询)上部署,ToolRLA在三个月内实现了任务完成率提升47%(62%->91%)、工具调用错误减少63%(38%->14%)、合规性违规减少93%(12%->0.8%),并在子2秒延迟内完成。消融研究显示,乘法奖励设计相比加法替代方案贡献了7个百分点的改进。通用性进一步在ToolBench和API-Bank上得到验证。
英文摘要:
Tool-integrated agents that interleave reasoning with API calls are promising for complex tasks, yet aligning them for high-stakes, domain-specific deployment remains challenging: existing reinforcement learning approaches rely on coarse binary rewards that cannot distinguish tool selection errors from malformed parameters. We present ToolRLA, a three-stage post-training pipeline (SFT -> GRPO -> DPO) for domain-specific tool agents. The core contribution is a fine-grained reward function with multiplicative correctness decomposition spanning four dimensions -- format validity, tool selection, parameter accuracy, and regulatory compliance -- that encodes domain priority orderings as inductive biases in the reward landscape. Deployed on a financial advisory copilot (80+ advisors, 1,200+ daily queries), ToolRLA achieves over three months: a 47% improvement in task completion rate (62%->91%), a 63% reduction in tool invocation errors (38%->14%), and a 93% reduction in regulatory violations (12%->0.8%), within sub-2-second latency. Ablation studies show the multiplicative reward design accounts for 7 percentage points of improvement over additive alternatives. Generalization is further validated on ToolBench and API-Bank.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。