比较陷阱：成对比较会放大LLM评估器的有偏偏好
The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators
AI总结：
针对LLM评估器在成对比较中易放大对冗长等表面属性的有偏偏好问题，提出整合逐点推理的成对评估方法PRePair，该方法在对抗性基准LLMBar上提升性能，且在标准基准MT-Bench上优于逐点评估。
摘要：
随着大型语言模型（LLMs）越来越多地被用作自然语言生成任务的评估器，确保无偏评估至关重要。然而，LLM评估器往往表现出有偏偏好，例如偏爱冗长和权威语气。我们的实证分析表明，这些偏好在成对评估中会加剧——此时LLMs直接比较两个输出，容易优先考虑表面属性。相比之下，独立评估每个输出的逐点评估不易受此类偏见影响，因为每个输出都是单独判断的。为解决成对评估的局限性，我们引入一种新颖的评估方法PRePair，它在成对框架内整合了逐点推理。PRePair有效缓解了有偏偏好，在对抗性基准（LLMBar）上提升了性能，同时在标准基准（MT-Bench）上优于逐点评估。
英文摘要：
As large language models (LLMs) are increasingly used as evaluators for natural language generation tasks, ensuring unbiased assessments is essential. However, LLM evaluators often display biased preferences, such as favoring verbosity and authoritative tones. Our empirical analysis reveals that these biases are exacerbated in pairwise evaluation, where LLMs directly compare two outputs and easily prioritize superficial attributes. In contrast, pointwise evaluation, which assesses outputs independently, is less susceptible to such bias because each output is judged in isolation. To address the limitations of the pairwise evaluation, we introduce a novel evaluation method, PRePair, which integrates pointwise reasoning within a pairwise framework. PRePair effectively alleviates biased preference, improving performance on the adversarial benchmark (LLMBar) while outperforming pointwise evaluation on the standard benchmark (MT-Bench).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
