通过不确定性对齐强化学习探索智能体工具调用决策
Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
AI总结：
针对智能体工具调用中错误累积问题，提出TRUST方法，将不确定性量化作为排斥力融入奖励设计，并标注轻量关键轮次用于多轮轨迹统一后训练，显著提升决策质量与智能体性能。
摘要：
基于大语言模型的智能体经常做出次优的工具使用决策，包括不支持的工具调用和幻觉式的直接响应，这可能在多步交互中累积错误。现有方法主要通过推理时校正或基于决策结果和结构化检查表的粗粒度奖励信号来改善这些行为，而忽略了智能体决策的不确定性特征。我们观察到，面向决策的强化学习倾向于削弱正确和错误动作之间的不确定性分离，导致过度自信的错误和较弱的探索信号。因此，我们提出TRUST，将不确定性量化作为排斥力融入奖励设计以维持不确定性分离，并标注轻量级关键轮次注释用于多轮轨迹的统一后训练。在多种工具使用基准上的实验结果表明，TRUST在优化过程中持续提升决策质量和智能体性能，同时保持更可靠的不确定性估计。
英文摘要：
Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions. Existing approaches mainly improve these behaviors through inference-time correction or coarse-grained reward signals based on decision outcomes and structured checklists, leaving the uncertainty characteristics of agent decisions underexplored. We observe that decision-oriented reinforcement learning tends to weaken the uncertainty separation between correct and incorrect actions, resulting in overconfident mistakes and weaker exploration signals. Therefore, we propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation, and labels lightweight key-turn annotations for unified post-training of multi-turn trajectories. Experimental results across diverse tool-use benchmarks show that TRUST consistently enhances both decision quality and agent performance while maintaining more reliable uncertainty estimates during optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
