置信二元性:分析和缓解工具使用代理中的校准偏差
The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
AI总结:
本研究提出强化学习框架,通过优化任务准确性和校准,缓解工具使用代理中的校准偏差,提升其在不同领域和环境中的鲁棒性。
摘要:
基于大语言模型(LLMs)的自主代理正在迅速发展以处理多轮任务,但确保其可信度仍是一个关键挑战。信任的基础是校准,即代理表达信心的能力能够可靠地反映其实际表现。虽然校准在静态模型中已得到良好确立,但在集成工具的代理工作流程中的动态性仍处于探索阶段。在本工作中,我们系统地研究了工具使用代理中的口头校准,揭示了由工具类型驱动的根本置信二元性。具体而言,我们的初步研究发现,证据工具(如网络搜索)由于检索信息中的固有噪声,系统性地导致严重过度自信,而验证工具(如代码解释器)可以通过确定性反馈来支撑推理并缓解校准偏差。为了在不同工具类型中稳健地提高校准,我们提出了一种强化学习(RL)微调框架,通过全面的奖励设计基准,联合优化任务准确性和校准。我们证明,经过训练的代理不仅实现了更优的校准,还能在本地训练环境到嘈杂网络设置以及数学推理等不同领域中表现出鲁棒的泛化能力。我们的结果强调了为工具使用代理制定领域特定校准策略的必要性。更广泛地说,这项工作为构建能够可靠传达不确定性的自我意识代理奠定了基础,这对于高风险、现实世界部署至关重要。
英文摘要:
Autonomous agents based on large language models (LLMs) are rapidly evolving to handle multi-turn tasks, but ensuring their trustworthiness remains a critical challenge. A fundamental pillar of this trustworthiness is calibration, which refers to an agent's ability to express confidence that reliably reflects its actual performance. While calibration is well-established for static models, its dynamics in tool-integrated agentic workflows remain underexplored. In this work, we systematically investigate verbalized calibration in tool-use agents, revealing a fundamental confidence dichotomy driven by tool type. Specifically, our pilot study identifies that evidence tools (e.g., web search) systematically induce severe overconfidence due to inherent noise in retrieved information, while verification tools (e.g., code interpreters) can ground reasoning through deterministic feedback and mitigate miscalibration. To robustly improve calibration across tool types, we propose a reinforcement learning (RL) fine-tuning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. We demonstrate that our trained agents not only achieve superior calibration but also exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. Our results highlight the necessity of domain-specific calibration strategies for tool-use agents. More broadly, this work establishes a foundation for building self-aware agents that can reliably communicate uncertainty in high-stakes, real-world deployments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。