arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.04034cs.SEcs.AI
收藏

“我不知道”过滤器：提高函数调用中智能体的可靠性

The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling

University of California, Davis, Department of Computer Science（加州大学戴维斯分校计算机科学系）；Okareo, Inc.（Okareo公司）；University of California, Davis, Department of Mathematics（加州大学戴维斯分校数学系）

Stefan Broecker, Mason del Rosario, Boris Selitser, Thomas Strohmer

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究语言模型在函数调用基准测试中性能提升但因训练评估指标致幻觉问题。提出考虑错误函数调用负面结果的评估指标及可量化不确定性、去除有害函数调用的轻量级可训练过滤器，助力智能体知道何时说“我不知道”以提升可靠性。

摘要：

支撑智能体的语言模型在函数调用基准测试中的性能迅速提升。然而，这些模型训练和评估中使用的指标往往鼓励模型即使答案不确定也做出肯定声明，导致幻觉。当信任语言模型在高风险应用中使用函数调用做决策时，这种幻觉可能是灾难性的。为此，我们提出一种考虑与错误函数调用相关负面结果的智能体评估指标。此外，为在幻觉造成危害之前捕捉到它们，我们提出一种轻量级可训练过滤器，它可以量化语言模型的不确定性并去除潜在有害的函数调用。通过在不修改基础模型的情况下训练该过滤器以检测和抑制不确定的函数调用，我们展示了一条通向知道何时说“我不知道”的智能体的实用路径，我们认为这一属性对于生产可靠性至关重要。

英文摘要：

The language models that underpin agents have seen a rapid rise in performance on function calling benchmarks. However, the metrics used in the training and evaluation of these models often encourage models to make positive claims even when the answer is uncertain, leading to hallucinations. Such hallucinations can be disastrous when language models are trusted to use function calls to make decisions in high stakes applications. To that end, we propose an agent evaluation metric that takes into account the negative outcomes associated with incorrect function calls. Further, to catch hallucinations before they can cause harm, we propose a lightweight trainable filter that can quantify a language model's uncertainty and remove potentially harmful function calls. By training that filter to detect and suppress uncertain function calls without modifying the underlying model, we demonstrate a practical path toward agents that know when to say "I don't know," a property we argue is essential to production reliability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑