Multi-agent systems powered by large language models (LLMs) are transforming enterprise automation, yet systematic evaluation methodologies for assessing tool-use reliability remain underdeveloped. We introduce a comprehensive diagnostic framework that leverages big data analytics to evaluate procedural reliability in intelligent agent systems, addressing critical needs for SME-centric deployment in privacy-sensitive environments. Our approach features a 12-category error taxonomy capturing failure modes across tool initialization, parameter handling, execution, and result interpretation. Through systematic evaluation of 1,980 deterministic test instances spanning both open-weight models (Qwen2.5 series, Functionary) and proprietary alternatives (GPT-4, Claude 3.5/3.7) across diverse edge hardware configurations, we identify actionable reliability thresholds for production deployment. Our analysis reveals that procedural reliability, particularly tool initialization failures, constitutes the primary bottleneck for smaller models, while qwen2.5:32b achieves flawless performance matching GPT-4.1. The framework demonstrates that mid-sized models (qwen2.5:14b) offer practical accuracy-efficiency trade-offs on commodity hardware (96.6\% success rate, 7.3 s latency), enabling cost-effective intelligent agent deployment for resource-constrained organizations. This work establishes foundational infrastructure for systematic reliability evaluation of tool-augmented multi-agent AI systems.
RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University(上海智能信息处理关键实验室,计算机科学与人工智能学院,复旦大学);Artificial Intelligence, Fudan University(人工智能,复旦大学)
CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation
School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机科学学院);Department of Automation, Tsinghua University(清华大学自动化系);Department of Colorectal Medical Oncology, Zhejiang Cancer Hospital(浙江省肿瘤医院结直肠医学肿瘤科);School of Computer and Control Engineering, University of Chinese Academy of Sciences(中国科学院大学计算机与控制工程学院);School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)智能科学与工程学院)
RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agents
School of Geosciences and Info-Physics, Central South University(地质科学与信息物理学院,中南大学);School of Resources and Environment, University of Electronic Science and Technology of China(资源与环境学院,电子科技大学);School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology(地球科学与空间信息工程学院,湖南科技大学);Sanya Institute of Hunan University of Science and Technology(海南科技大学三亚研究院)
Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery
Southwest Jiaotong University, China(西南交通大学);The University of Tokyo, Japan(东京大学);RIKEN, Japan(日本理化学研究所);Westlake University, China(西湖大学);Hokkaido University, Japan(北海道大学)
The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院 Gallagher 学院);King Abdullah University of Science and Technology(国王 Abdullah 科学技术大学);Dongbei University of Finance and Economics(东北财经大学);University of Science and Technology of China(中国科学技术大学)
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);National University of Singapore(新加坡国立大学);Om AI Research(Om AI研究院);Binjiang Institute of Zhejiang University(浙江大学滨江研究院)
MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
National Chengchi University(国立中正大学);Georgetown University(乔治城大学);University of Michigan(密歇根大学);Stevens Institute of Technology(史蒂文斯理工学院);National Taiwan University of Science and Technology(台湾科技大学);Far Eastern Memorial Hospital(东方纪念医院)
HPOQuest: A Rare-Disease Diagnostic Agent Using Active Phenotype Acquisition
Technical University of Munich(慕尼黑工业大学);Helmholtz Center Munich(亥姆霍兹慕尼黑中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Munich Data Science Institute (MDSI)(慕尼黑数据科学研究所);Aalto University(阿尔托大学)