LLM 裁判具有暗电流：LLM 作为裁判评估的心理测量数据表
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
AI总结：
提出裁判数据表协议，通过真空输入、表面变异、位置偏好等指标测量 LLM 裁判的暗电流和偏差，揭示其测量特性。
摘要：
LLM 作为裁判的系统现在常规用于开放式模型评估，其中人类偏好标注成本高、速度慢且难以复现。然而，这些裁判通常被报告为标量准确率、胜率或一致性指标。我们认为，裁判应被报告为测量仪器。我们引入了一个裁判数据表协议，该协议测量在真实真空输入下的暗电流、对相同质量表面变化的稳定交叉敏感性、位置虚假偏好、在受控质量阶梯上的目标敏感性，以及由平局指令引发的标准或操作点。方向-稳定性分解揭示，明显的 Delta0 偏好可能是稳定的表面响应或伪装的位置偏差。在一个三裁判开放权重案例研究中，Llama-3.1-8B 显示出高暗电流和呈现冲突的 Delta0 行为，Qwen2.5-14B 是真空清洁且对目标敏感，但混合了稳定和位置过度判别，而 Qwen2.5-32B 是真空清洁，具有低稳定交叉敏感性和低位置虚假偏好。严格的平局标准消除了 Qwen32B 的 Delta0 虚假偏好，但将边缘 Delta1 目标信号吸收为平局，同时保留了 Delta5 敏感性。结果表明，提示移动的是标准，而不是分辨率。我们并不声称激发这项工作的下游机制假设已得到确认；贡献是在做出下游声明之前测量测量仪器的计量协议。
英文摘要：
LLM-as-a-judge systems are now routinely used for open-ended model evaluation, where human preference annotation is costly, slow, and difficult to reproduce. Yet these judges are often reported as scalar accuracy, win-rate, or agreement devices. We argue that a judge should instead be reported as a measurement instrument. We introduce a Judge Datasheet protocol that measures dark current under true-vacuum inputs, stable cross-sensitivity to same-quality surface variation, positional false preference, target sensitivity on a controlled quality ladder, and the criterion or operating point induced by tie instructions. The direction-stability decomposition reveals that apparent Delta0 preference can be stable surface response or disguised position bias. In a three-judge open-weight case study, Llama-3.1-8B shows high dark current and presentation-conflicted Delta0 behavior, Qwen2.5-14B is vacuum-clean and target-sensitive but mixes stable and positional over-discrimination, and Qwen2.5-32B is vacuum-clean with low stable cross-sensitivity and low positional false preference. A strict tie criterion eliminates Qwen32B Delta0 false preference but absorbs marginal Delta1 target signals into ties while preserving Delta5 sensitivity. The results show that prompting moves the criterion, not the resolution. We do not claim that the downstream mechanism hypothesis that motivated this work is confirmed; the contribution is a metrological protocol for measuring the measuring device before downstream claims are made.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
