智能体基准在任务耗尽前就停止排名
Agent Benchmarks Stop Ranking Before They Run Out of Tasks
AI总结：
该研究发现智能体基准排名停止的问题源于测量方式，提出截断对数比率评分法，重新分析8个基准后揭示了通过/失败评分的缺陷及模型间的量化差异。
摘要：
数据科学与机器学习工程智能体在MLE-bench、ALE-Bench、SWE-fficiency和AlgoTune等基准上接受评估，这些基准的任务会产生一个量化值（Kaggle指标、竞赛得分或加速比）。当基准无法区分最优模型时，通常的解决办法是采用更难的任务版本。我们对包括SWE-bench Verified和Terminal-Bench在内的8个基准的已发布运行结果重新评分，发现这种饱和现象更多源于测量方式而非任务本身。大多数智能体基准将一次运行简化为一个通过/失败比特，这存在两方面问题：一是过于粗糙，SWE-bench Verified上排名前十的相邻智能体差异小于其测试的误差，修正这些差异对相邻智能体的重排效果不超过删除每个智能体随机六分之一通过项的效果；二是受限于参考值，ALE-Bench排名前十的智能体按竞赛得分区分，在与人类中位数评分时并列，尽管近一半的运行结果优于该人类水平。在固定任务下，提高难度只会转移问题，而分级评分可解决该问题。我们推导了一种带有时间锚点的截断对数比率评分，并将成本作为单独维度。在Terminal-Bench的重新运行中，通过/失败并列的模型在测试所丢弃的量化值上差异超过十倍。
英文摘要：
Data-science and ML-engineering agents are evaluated on benchmarks such as MLE-bench, ALE-Bench, SWE-fficiency and AlgoTune, whose tasks yield a quantity (a Kaggle metric, contest score or speedup). When one stops separating the best models, the usual remedy is a harder version. Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, we find much of this saturation lies in the measurement, not the tasks. Most agent benchmarks reduce a run to one pass/fail bit, which fails in two ways. It is too coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than its tests' errors, and correcting them reorders neighbours no more than deleting a random sixth of each agent's passes. It is capped at the reference: ALE-Bench's top ten, separated by contest score, tie when graded against the median human, although nearly half the runs beat that human. On fixed tasks, a higher bar only moves the problem; graded credit fixes it. We derive a clipped log-ratio score with dated anchors and cost as a separate axis. In a Terminal-Bench re-run, pass/fail-tied models differ over tenfold on the quantity the tests discard.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
