当判断变成噪声：LLM 裁判基准的设计缺陷如何悄然削弱有效性
When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
AI总结：
本研究指出 LLM 裁判基准存在设计缺陷导致排名噪声，提出模式一致性和心理测量效度两种诊断机制，并在 Arena-Hard Auto 上验证了严重问题，提供改进原则。
摘要：
LLM 裁判基准（LLM-judged benchmarks）越来越多地被用于评估复杂的模型行为，然而其设计引入了传统基于真实标签（ground-truth）的基准中不存在的失效模式。我们认为，如果没有严格的目标和可验证的构造，基准排名可能产生高置信度的排名，而这些排名实际上在很大程度上是噪声。我们引入了两种机制来诊断这些问题。模式一致性（Schematic adherence）量化了裁判的整体裁决中有多少是由显式评估模式解释的，当裁判偏离其自身评分标准时，揭示出无法解释的方差。心理测量效度（Psychometric validity）聚合内部一致性和区分效度信号，以量化任何基准测试运行中不可约的不确定性。将这些工具应用于 Arena-Hard Auto，我们发现流行裁判中存在严重的模式不一致和因子坍缩：例如，DeepSeek-R1-32B 的无法解释方差超过 90%，大多数标准的因子相关性高于 0.93。我们还表明，Arena-Hard Auto 使用的 ELO 式聚合坍缩并掩盖了真实的排名不确定性。我们的结果突显了削弱有效性的设计缺陷，并为构建范围更明确、可靠性感知的 LLM 裁判基准提供了可操作的原则。我们已在 https://github.com/penfever/judgment-to-noise 发布了代码和数据集。
英文摘要：
LLM-judged benchmarks are increasingly used to evaluate complex model behaviors, yet their design introduces failure modes absent in conventional ground-truth based benchmarks. We argue that without tight objectives and verifiable constructions, benchmark rankings can produce high-confidence rankings that are in fact largely noise. We introduce two mechanisms to diagnose these issues. Schematic adherence quantifies how much of a judge's overall verdict is explained by the explicit evaluation schema, revealing unexplained variance when judges deviate from their own rubric. Psychometric validity aggregates internal consistency and discriminant validity signals to quantify irreducible uncertainty in any benchmarking run. Applying these tools to Arena-Hard Auto, we find severe schema incoherence and factor collapse across popular judges: for example, unexplained variance exceeding 90 percent for DeepSeek-R1-32B and factor correlations above 0.93 for most criteria. We also show that the ELO-style aggregation used by Arena-Hard Auto collapses and masks genuine ranking uncertainty. Our results highlight design failures that undermine validity and offer actionable principles for building better-scoped, reliability-aware LLM-judged benchmarks. We released our code and dataset at https://github.com/penfever/judgment-to-noise
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
