对基准进行基准测试：工具调用评估的有效性审计
Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
AI总结：
对四个主要工具调用基准家族进行系统有效性和可重复性审计，发现诸多评估者与人类意见分歧，指出当前分数可能反映评估者问题而非智能体能力，还介绍相关分类、审计工件等及新基准和系统。
摘要：
工具调用基准越来越多地用于对语言模型智能体进行排名，但其分数常被视为基本事实而未对评估者本身进行验证。我们对四个主要工具调用基准家族进行了系统的有效性和可重复性审计。发现诸多问题，结果表明当前工具调用分数可能反映评估者工件而非智能体能力。我们引入相关内容及新基准和系统。
英文摘要：
Tool-calling benchmarks are increasingly used to rank language-model agents, yet their scores are often treated as ground truth without validating the evaluators themselves. We present a systematic validity and reproducibility audit of four major tool-calling benchmark families: BFCL v4, τ2-Bench, LiveMCPBench, and MCP-Atlas. Across 496 expert-reviewed benchmark tasks, we find 92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate. The failures are not isolated annotation mistakes: deterministic benchmarks exhibit brittle state matching, trajectory lock-in, incorrect ground truths, substring-based communication failures, and reward-basis misalignment, while LLM-judge benchmarks exhibit rubric drift, hallucinated completion, answer-only scoring, and substantial run-to-run variance. In LiveMCPBench, 23 repeated evaluations of the same setup produce scores ranging from 57.9% to 76.8%, a spread of 18.9 percentage points, large enough to change leaderboard conclusions. These results show that current tool-calling scores can reflect evaluator artifacts rather than agent capability. We introduce a unified taxonomy of tool-calling evaluation failures, release trace-level audit artifacts and corrected evaluation components, and argue for decomposed metrics that separately measure tool invocation, task completion, and outcome verification. Our findings suggest that progress in tool-using agents requires benchmarks whose evaluators are themselves reproducible, auditable, and aligned with human judgments of task success. We further introduce Tool-Veritas, a configurable benchmark that combines deterministic state verification with optional qualitative judging, and Harness Lab, an open-source system for benchmark execution, trace inspection, repeated-run comparison, and evaluator debugging.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
