arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.16280cs.AI
收藏

当代理人失败时:多代理LLM系统工具调用可靠性的诊断框架

When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems

Singapore Management University(新加坡管理大学);Mastercard(万事达卡)

Donghao Huang, Gauri Malwe, Zhaoxia Wang

AI总结:

本文提出了一种多代理LLM系统工具调用可靠性的诊断框架,通过大数据分析评估程序可靠性,发现工具初始化故障是小模型的主要瓶颈,而qwen2.5:32b在性能上接近GPT-4.1。

摘要:

由大型语言模型(LLMs)驱动的多代理系统正在改变企业自动化,但系统评估方法论仍不完善。我们介绍了一个全面的诊断框架,利用大数据分析来评估智能代理系统的程序可靠性,以满足隐私敏感环境中小型企业部署的需求。我们的方法包含12类错误分类,涵盖工具初始化、参数处理、执行和结果解释中的故障模式。通过系统评估1,980个确定性测试实例,涵盖开放式模型(Qwen2.5系列、Functionary)和专有替代品(GPT-4、Claude 3.5/3.7)在多样化边缘硬件配置中的表现,我们确定了生产部署中的可操作可靠性阈值。我们的分析表明,程序可靠性,特别是工具初始化故障,是较小模型的主要瓶颈,而qwen2.5:32b实现了与GPT-4.1相同的完美性能。该框架显示,中型模型(qwen2.5:14b)在商用硬件上提供实际的准确率-效率权衡(96.6%的成功率,7.3秒延迟),使资源受限的组织能够以低成本部署智能代理。这项工作为系统评估工具增强的多代理AI系统的可靠性奠定了基础。

英文摘要:

Multi-agent systems powered by large language models (LLMs) are transforming enterprise automation, yet systematic evaluation methodologies for assessing tool-use reliability remain underdeveloped. We introduce a comprehensive diagnostic framework that leverages big data analytics to evaluate procedural reliability in intelligent agent systems, addressing critical needs for SME-centric deployment in privacy-sensitive environments. Our approach features a 12-category error taxonomy capturing failure modes across tool initialization, parameter handling, execution, and result interpretation. Through systematic evaluation of 1,980 deterministic test instances spanning both open-weight models (Qwen2.5 series, Functionary) and proprietary alternatives (GPT-4, Claude 3.5/3.7) across diverse edge hardware configurations, we identify actionable reliability thresholds for production deployment. Our analysis reveals that procedural reliability, particularly tool initialization failures, constitutes the primary bottleneck for smaller models, while qwen2.5:32b achieves flawless performance matching GPT-4.1. The framework demonstrates that mid-sized models (qwen2.5:14b) offer practical accuracy-efficiency trade-offs on commodity hardware (96.6\% success rate, 7.3 s latency), enabling cost-effective intelligent agent deployment for resource-constrained organizations. This work establishes foundational infrastructure for systematic reliability evaluation of tool-augmented multi-agent AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑