公平性还是流畅性？对配对LLM-as-a-judge语言偏见的调查
Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge
AI总结：
本文研究了LLM-as-a-judge在配对任务中的语言偏见，发现不同语言在性能和偏好上有显著差异，且语言偏见不能仅由困惑度解释。
摘要：
近年来，大语言模型（LLM）的进步激励了LLM-as-a-judge的应用发展，即利用LLM作为裁判来决定给定上下文下某段文本的质量。然而，先前的研究表明，LLM-as-a-judge可能会对被评判文本的不同方面产生偏见，这些偏见往往与人类偏好不一致。已识别的一种偏见是语言偏见，这表明LLM-as-a-judge的决策可能因被评判文本的语言而不同。在本文中，我们研究了配对LLM-as-a-judge中的两种语言偏见类型：（1）当裁判被提示比较同语言选项时，不同语言之间的性能差异；（2）当裁判被提示比较两种不同语言的选项时，对主要语言选项的偏见。我们发现，在同语言评判中，不同语言家族之间存在显著的性能差异，欧洲语言始终优于非洲语言，且这种偏见在文化相关主题中更为明显。在跨语言评判中，我们观察到大多数模型倾向于选择英语答案，且这种偏好受答案语言的影响大于问题语言。最后，我们调查了语言偏见是否确实由先前识别的LLM-as-a-judge的低困惑度偏见引起，并发现虽然困惑度与语言偏见略有相关，但语言偏见不能仅由困惑度解释。
英文摘要：
Recent advances in Large Language Models (LLMs) have incentivized the development of LLM-as-a-judge, an application of LLMs where they are used as judges to decide the quality of a certain piece of text given a certain context. However, previous studies have demonstrated that LLM-as-a-judge can be biased towards different aspects of the judged texts, which often do not align with human preference. One of the identified biases is language bias, which indicates that the decision of LLM-as-a-judge can differ based on the language of the judged texts. In this paper, we study two types of language bias in pairwise LLM-as-a-judge: (1) performance disparity between languages when the judge is prompted to compare options from the same language, and (2) bias towards options written in major languages when the judge is prompted to compare options of two different languages. We find that for same-language judging, there exist significant performance disparities across language families, with European languages consistently outperforming African languages, and this bias is more pronounced in culturally-related subjects. For inter-language judging, we observe that most models favor English answers, and that this preference is influenced more by answer language than question language. Finally, we investigate whether language bias is in fact caused by low-perplexity bias, a previously identified bias of LLM-as-a-judge, and we find that while perplexity is slightly correlated with language bias, language bias cannot be fully explained by perplexity only.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
