Judge's Verdict：基于人类一致性对LLM评判能力的全面分析
Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
AI总结：
提出Judge's Verdict Benchmark两步评估法，通过相关性测试与基于z分数的人类相似性测试，对54个LLM作为评判者的能力进行分级，发现评判能力取决于训练策略而非模型规模。
摘要：
本研究引入了 Judge's Verdict Benchmark，这是一种新颖的两步方法，用于评估大型语言模型（LLMs）作为响应准确性评估任务评判者的能力。我们评估了 54 个 LLM 在对来自 RAG（检索增强生成）或 Agentic 管道的响应与真实答案进行评分时，能否复现人类判断。我们的方法从传统的相关性分析逐步推进到全面的 Cohen's Kappa 分析，以测量实际的一致性模式。该两步法包括：(1) 相关性测试，筛选出具有强一致性的评判者；随后进行 (2) 使用 z 分数的人类相似性测试，以识别两种截然不同的判断模式：模仿自然人类变化的人类相似判断（|z| < 1），以及超过典型人与人之间一致性水平的超一致判断（z > 1）。该方法表明，在 54 个测试的 LLM 中有 27 个达到了 Tier 1 性能：23 个模型表现出保留人类判断细微差别的人类相似模式，而 4 个模型展示了超一致行为，这种模式可能表明可靠性增强或对复杂判断的过度简化。通过测试 43 个开源模型（1B-405B 参数）和 11 个闭源模型（GPT、Gemini、Claude 变体），我们证明评判能力的卓越并非仅仅依赖于模型规模，而是取决于特定的训练策略。我们的主要贡献包括：(1) 证明仅靠相关性不足以评估评判者，(2) 引入基于一致性模式的“评判者图灵测试”，以及 (3) 提供一个标准化基准，将 LLM 评判者划分为不同的性能层级，以满足不同的评估需求。
英文摘要：
This research introduces the Judge's Verdict Benchmark, a novel two-step methodology to evaluate Large Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlation analysis to comprehensive Cohen's Kappa analysis that measures actual agreement patterns. The two-step approach includes: (1) a correlation test that filters judges with strong alignment, followed by (2) a human-likeness test using z-scores to identify two distinct judgment patterns: human-like judgment (|z| < 1) that mimics natural human variation, and super-consistent judgment (z > 1) that exceeds typical human-to-human agreement levels. This methodology reveals that 27 out of 54 tested LLMs achieve Tier 1 performance: 23 models exhibit human-like patterns that preserve the nuances of human judgment, while 4 models demonstrate super-consistent behavior, a pattern that could indicate either enhanced reliability or oversimplification of complex judgments. Testing 43 open-source models (1B-405B parameters) and 11 closed models (GPT, Gemini, Claude variants), we demonstrate that judge excellence is not solely dependent on model size but on specific training strategies. Our key contributions include: (1) establishing that correlation alone is insufficient for judge evaluation, (2) introducing a "Turing Test for judges" based on agreement patterns, and (3) providing a standardized benchmark for classifying LLM judges into distinct performance tiers for different evaluation needs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
