JudgmentBench: 比较评分量规与偏好评估在质量评价中的应用
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
AI总结：
本研究通过构建包含30个真实法律任务、1539个评分量规和1530对偏好判断的数据集JudgmentBench，比较了评分量规与成对比较两种评估方法，发现成对比较在恢复预期质量排序上显著优于评分量规（平均斯皮尔曼等级相关系数0.908 vs 0.150），且注释时间减少一半以上。
摘要：
当前基准测试实践中主导着两种方法论：基于评分量规的评分根据预定义标准评估项目，而比较判断则引发输出之间的成对偏好。尽管两种方法论被广泛使用，但两者之间的选择很少被论证。我们发布了JudgmentBench，一个包含30个真实法律任务的基准测试，配对了来自执业律师（包括美国主要律师事务所）的1539个评分量规和1530个成对偏好判断，这些律师具有丰富的经验。这些注释构成了高专业领域内首个公开可用的数据集，其中两种监督信号由同一专家对同一项目进行收集。使用LLM生成的三个质量级别的输出，我们提供了初步的经验比较：比较判断在恢复预期质量排序方面显著优于评分量规（平均斯皮尔曼等级相关系数为0.908 vs 0.150，估计差异=0.758 [0.494, 1.021]），同时所需的注释时间不到一半。这一模式对人类注释者和LLM自动评分器均成立。除了这一初步比较，数据集的配对结构支持更广泛的研究议程，探讨在没有可验证真实情况的领域中，如何引导、聚合专家判断并将其用作监督信号。
英文摘要：
Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
