PReMISE：作为LLM评判者测量规范的政策评分标准
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
AI总结：
提出PReMISE框架，从人类偏好数据中发现政策级评分标准集，并从结构充分性、可靠性、偏好拟合和对抗鲁棒性四个维度审计评分标准，通过偏好排名选择和可靠性约束修复操作提升评判准确性并降低可被利用性。
摘要：
LLM评判者越来越多地被用于评估开放式回答，但其分数强烈依赖于条件化它们的评分标准。一个模糊的评分标准要求回答“有帮助且事实准确”可能会奖励那些编造事实或违反用户意图的精心修饰的回答。我们将可重复使用的评分标准视为测量规范：改变评分标准会改变由固定评判者产生的回答质量测量。我们引入PReMISE，一个框架，给定成对的人类偏好数据，(i) 发现一个政策级别的评分标准集，以及(ii) 在LLM评判者使用下，沿着四个维度审计任何评分标准集：结构充分性、可靠性、偏好拟合和对抗鲁棒性。在评分标准来源中，没有原始来源同时具有可靠性、偏好预测性和对抗鲁棒性；高评分者间一致性并不意味着低可被利用性。PReMISE是唯一同时在适用性、特异性和有效维度上得分非平凡的评分标准来源。我们贡献了两个针对审计的修复操作：偏好排名选择将评判者在成对回答上的准确率从65.0%提高到68.6%，与最强的评分标准发现基线竞争，并在我们的跨评判者扫描中在三个评判者中的两个上领先；可靠性约束精炼将利用性回答获得高分的比率从46.4%降低到36.0%，而评分者间一致性变化很小（α=.531→.519）。
英文摘要：
LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rubric asking for a response to be ``helpful and factual'' can reward polished answers that invent facts or violate user intent. We treat reusable rubrics as measurement specifications: changing the rubric changes the response quality measurement induced by a fixed judge. We introduce PReMISE, a framework that, given pairwise human-preference data, (i) discovers a policy-level rubric set, and (ii) audits any rubric set under LLM-judge use along four axes: structural adequacy, reliability, preference fit, and adversarial robustness. Across rubric sources no raw source is simultaneously reliable, preference-predictive, and adversarially robust; and high inter-rater agreement does not imply low exploitability. PReMISE is the only rubric source to score non-trivially on applicability, specificity, and effective dimensionality simultaneously. We contribute two audit-targeted repair operations: preference-rank selection raises judge accuracy on paired responses from $65.0\%$ to $68.6\%$, competitive with the strongest rubric-discovery baselines and leading on two of three judges in our cross-judge sweep; reliability-constrained refinement reduces the rate at which exploit responses receive high scores from $46.4\%$ to $36.0\%$ with little change in inter-judge agreement ($α{=}.531\to.519$).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
