量化与缓解LLM评判者的自我偏好偏差
Quantifying and Mitigating Self-Preference Bias of LLM Judges
AI总结：
提出自动化框架量化LLM自我偏好偏差，并通过认知负荷分解的多维评估策略平均降低31.5%的偏差。
摘要：
LLM-as-a-Judge已成为自动评估系统中的主导方法，在模型对齐、排行榜构建、质量控制等方面发挥关键作用。然而，该方法的可扩展性和可信度可能因自我偏好偏差（SPB）而严重失真，SPB是一种定向评估偏差，即LLM在评估时系统性地偏好或排斥自身生成的输出。现有测量方法依赖昂贵的人工标注，并将生成能力与评估立场混为一谈，因此不适用于实际系统中的大规模部署。为解决此问题，我们引入了一个完全自动化的框架来量化和缓解SPB，该框架构建质量差异可忽略的等质量回答对，从而在无需人工黄金标准的情况下，从偏差倾向中统计分离出可区分性。对20个主流LLM的实证分析表明，先进能力通常与低SPB不相关，甚至负相关。为缓解此偏差，我们提出了一种基于认知负荷分解的结构化多维评估策略，平均降低SPB 31.5%。
英文摘要：
LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5\% on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
