学习何时不行动：缓解智能体强化学习中的工具滥用
Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning
AI总结：
提出EAPO框架，通过引入无工具轨迹、难度感知奖励塑造和置信度感知令牌重加权，在数学和知识密集型推理任务中减少工具滥用，同时提升准确率-效率权衡。
摘要：
智能体强化学习可能引发工具滥用，即模型过度使用外部工具，即使对于内部推理可解的查询也是如此。现有方法通过统一的工具使用惩罚或硬限制来缓解此问题，这降低了工具使用频率，但可能抑制有用的工具辅助探索。我们提出EAPO，一种高效的智能体策略优化框架，学习选择性工具使用。EAPO在每个rollout组中引入无工具轨迹，应用难度感知奖励塑造以主要对较简单查询上的冗余工具调用进行惩罚，并使用置信度感知令牌重加权来改进策略学习。在九个数学和知识密集型推理基准上，EAPO在Qwen2.5-3B、Qwen2.5-7B和Llama3.1-8B上持续改善了准确率-效率权衡。与GRPO相比，EAPO的平均性能分别提高了10.45%、7.27%和9.69%，同时平均工具调用次数分别减少了18.33%、18.33%和24.59%。这些结果表明，智能体可以在不损害工具集成推理的情况下学习何时不使用工具。
英文摘要：
Agentic reinforcement learning can induce tool abuse, where models overuse external tools even for queries solvable by internal reasoning. Existing approaches mitigate this issue with uniform tool-use penalties or hard limits, which reduce tool frequency but may also suppress useful tool-assisted exploration. We propose EAPO, an Efficient Agentic Policy Optimization framework that learns selective tool use. EAPO introduces tool-free trajectories into each rollout group, applies difficulty-aware reward shaping to penalize redundant tool calls mainly on easier queries, and uses confidence-aware token reweighting to improve policy learning. Across nine mathematical and knowledge-intensive reasoning benchmarks, EAPO consistently improves the accuracy efficiency trade-off on Qwen2.5-3B, Qwen2.5-7B, and Llama3.1-8B. Compared with GRPO, EAPO improves average performance by 10.45%, 7.27%, and 9.69%, while reducing average tool calls by 18.33%, 18.33%, and 24.59%, respectively. These results show that agents can learn when not to use tools without compromising tool-integrated reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
