对比解码缓解LLM-as-a-Judge的分数范围偏差
Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge
AI总结：
本文针对LLM作为评判者时的分数范围偏差问题，提出对比解码方法，通过提升与人类评判的Spearman相关性，平均提升11.7%。
摘要：
大型语言模型（LLM）常用于各种应用中的评估，但其结果的可靠性仍面临挑战。其中一项挑战是使用LLM作为评判者进行直接评估，即在没有参考的情况下分配指定范围的分数。聚焦于摘要任务，我们首先表明该挑战源于LLM评判输出与分数范围偏差相关，即LLM评判输出对预定义的分数范围高度敏感。我们还显示同一家族中的模型也存在类似偏差。然后通过对比解码缓解这种偏差，在不同分数范围内与人类评判的Spearman相关性平均提升11.7%。
英文摘要：
Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. Focusing on summarization, we first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.7% relative improvement on average in Spearman correlation with human judgments across different score ranges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
