偏爱自己：一种衡量 LLM-as-a-Judge 中自偏见的统计方法
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
AI总结：
本文提出一种统计框架，通过对比 LLM 评判者对自身与第三方模型输出的评分差异并控制基础质量，可靠量化了 LLM-as-a-Judge 中的自偏见与家族偏见，为自动评估的解读与纠偏提供了实用指导。
摘要：
大型语言模型（LLMs）可以作为评判者，对其他 LLM 的输出提供快速且可靠的评估。然而，模型可能会系统性地对自己的输出给予过高评价，这种现象被称为自偏见，它会扭曲对真实模型性能的评估。以往的研究常常将模型质量的实际差异与偏见混为一谈，或者错误地假设 LLM 和人类的评估遵循相同的评分分布。在本研究中，我们提出了一个统计框架，明确形式化了可以识别和估计自偏见的假设条件。我们的方法对 LLM-as-a-judge 分配给自身完成结果与其他模型完成结果的评分分布差异进行建模，同时考虑由独立的第三方评判者（例如人类）提供的完成结果的基础质量。我们的方法能够可靠地分离和量化自偏见，即使模型能力各不相同，也能确保将真实的性能差异与自偏见区分开来。我们在一个包含专家人类标注和九个不同 LLM 评判者判断的大型数据集（>5000 个提示-完成对）上对自偏见进行了实证分析。我们发现，GPT-4o 和 Claude 3.5 Sonnet 等模型会系统性地给自己的输出打高分。这些模型还表现出家族偏见；系统性地对同家族其他模型生成的输出给予更高评分。我们的研究结果突显了使用 LLM 评判者的潜在陷阱，并为在解读自动评估时减轻偏见提供了实用指导。
英文摘要：
Large language models (LLMs) can serve as judges that offer rapid and reliable assessments of other LLM outputs. However, models may systematically assign overly favorable ratings to their own outputs, a phenomenon known as self-bias, which can distort evaluations of true model performance. Previous studies often conflate genuine differences in model quality with bias or incorrectly assume that evaluations from LLMs and humans follow the same rating distributions. In this work, we present a statistical framework that explicitly formalizes assumptions under which self-bias can be identified and estimated. Our method models the difference in the scoring distribution that LLM-as-a-judge assigns to its own completions compared to other models, while accounting for the underlying quality of the completions provided by an independent, third-party judge (e.g., humans). Our method reliably isolates and quantifies self-bias, even when models vary in ability, ensuring that genuine performance differences are not mistaken for self-bias. We conduct an empirical analysis of self-bias on a large dataset (>5000 prompt-completion pairs) consisting of expert human annotations and judgments from nine different LLM judges. We find that some models, such as GPT-4o and Claude 3.5 Sonnet, systematically assign higher scores to their own outputs. These models also display family-bias; systematically assigning higher ratings to outputs produced by other models of the same family. Our findings highlight potential pitfalls of using LLM judges and offer practical guidance to mitigate biases when interpreting automated evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
