arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.14136cs.AI
收藏

超越准确率:企业级智能体AI系统的多维度评估框架

Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems

Sushant Mehta

AI总结:

针对现有智能体AI基准忽略企业级成本、可靠性等需求的问题,提出多维度评估框架CLEAR,实验证明其比仅用准确率更能准确预测智能体的生产部署表现。

摘要:

当前智能体AI基准主要评估任务完成准确率,却忽略了成本效率、可靠性、运行稳定性等关键企业需求。通过对12个主流基准的系统分析以及对前沿智能体的实证评估,我们发现三大核心局限:一是缺乏成本控制评估,导致相近精度下成本差异可达50倍;二是可靠性评估不足,智能体性能从单次运行的60%降至8次运行一致性的25%;三是缺少针对安全性、延迟、策略合规性的多维度指标。我们提出专为企业部署设计的整体评估框架CLEAR(成本、延迟、效能、保障、可靠性)。在300项企业任务上对6款主流智能体的评估表明,仅优化准确率得到的智能体,成本是性能相当的成本感知型方案的4.4-10.8倍。15名专家参与的评估证实,CLEAR对生产环境成功的预测相关性ρ=0.83,优于仅用准确率评估的ρ=0.41。

英文摘要:

Current agentic AI benchmarks predominantly evaluate task completion accuracy, while overlooking critical enterprise requirements such as cost-efficiency, reliability, and operational stability. Through systematic analysis of 12 main benchmarks and empirical evaluation of state-of-the-art agents, we identify three fundamental limitations: (1) absence of cost-controlled evaluation leading to 50x cost variations for similar precision, (2) inadequate reliability assessment where agent performance drops from 60\% (single run) to 25\% (8-run consistency), and (3) missing multidimensional metrics for security, latency, and policy compliance. We propose \textbf{CLEAR} (Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment. Evaluation of six leading agents on 300 enterprise tasks demonstrates that optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance. Expert evaluation (N=15) confirms that CLEAR better predicts production success (correlation $ρ=0.83$) compared to accuracy-only evaluation ($ρ=0.41$).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑