迈向标准、与企业相关的智能体AI基准：来自55亿token智能体AI评估的经验教训
Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations
AI总结：
针对传统LLM基准无法满足企业智能体AI评估需求的问题，提出兼顾抗污染性与智能体能力评估的KAMI v0.1基准，通过大规模实验揭示传统基准排名与实际智能体性能的差异，为企业部署提供关键见解。
摘要：
企业采用智能体AI系统需要能反映真实部署场景的可靠评估方法。传统大语言模型（LLM）基准存在训练数据污染问题，且无法衡量多步工具使用、不确定性下决策等智能体能力。我们提出面向企业的基准Kamiwaza智能体价值指数（KAMI）v0.1，它兼顾抗污染性与智能体评估。通过17万LLM测试项、35种模型配置处理超55亿token的实验，我们发现传统基准排名难以预测实际智能体性能。值得注意的是，Llama 4或Qwen 3等新一代模型在企业相关任务上并非总能优于旧版本，与传统基准趋势矛盾。我们还提出成本-性能权衡、模型特定行为模式及推理能力对token效率影响的见解，这些发现对企业部署决策至关重要。
英文摘要：
Enterprise adoption of agentic AI systems requires reliable evaluation methods that reflect real-world deployment scenarios. Traditional LLM benchmarks suffer from training data contamination and fail to measure agentic capabilities such as multi-step tool use and decision-making under uncertainty. We present the Kamiwaza Agentic Merit Index (KAMI) v0.1, an enterprise-focused benchmark that addresses both contamination resistance and agentic evaluation. Through 170,000 LLM test items processing over 5.5 billion tokens across 35 model configurations, we demonstrate that traditional benchmark rankings poorly predict practical agentic performance. Notably, newer generation models like Llama 4 or Qwen 3 do not always outperform their older generation variants on enterprise-relevant tasks, contradicting traditional benchmark trends. We also present insights on cost-performance tradeoffs, model-specific behavioral patterns, and the impact of reasoning capabilities on token efficiency -- findings critical for enterprises making deployment decisions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
