法官偏爱英语吗？评估LLM作为法官时的语言切换不变性
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
AI总结：
提出Judge-LS协议，通过语言切换变体评估LLM作为自动法官的可靠性，发现语言切换导致10.7-14.4%偏好翻转，但翻译等价测试未显示系统英语偏好。
摘要：
大型语言模型（LLM）现在被广泛用作开放式指令遵循评估的自动法官。这种做法方便、可扩展，且通常比基于参考的指标更具语义意识，但它也引入了一个新的可靠性问题：法官评估的是答案的质量，还是也对比较呈现的语言做出反应？我们提出了Judge-LS，一个轻量级元评估协议，将LLMBar响应对项目转换为英语、中文和中英文语言切换变体。一个可靠的法官应在保持标签的语言转换下保持其偏好，并且当两个答案翻译等价时不应偏好某种语言。我们在完整的419项LLMBar基准上评估了四个API可访问的法官，产生了13,408个成功的成对判断。跨模型，中文和语言切换呈现相对于英语导致10.7-14.4%的偏好翻转，所有法官在英语中达到最高准确率。然而，翻译等价的平局探测并未揭示系统的英语偏好：大多数探测被判断为平局，非平局决策更常偏向中文。我们添加了置信区间、配对显著性检验和自动转换审计，并进行了敏感性分析，排除了机械标记的高风险变体。该实验不需要模型训练，仅使用API调用，并且在适度的本地硬件上可行。
英文摘要：
Large language models (LLMs) are now widely used as automatic judges for open-ended instruction-following evaluation. This practice is convenient, scalable, and often more semantically aware than reference-based metrics, but it also introduces a new reliability question: does a judge evaluate the quality of an answer, or does it also react to the language in which the comparison is presented? We propose Judge-LS, a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants. A reliable judge should preserve its preference under label-preserving language transformations and should not prefer a language when two answers are translation-equivalent. We evaluate four API-accessible judges on the full 419-item LLMBar benchmark, producing 13,408 successful pairwise judgments. Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English. However, translation-equivalent tie probes do not reveal a systematic English preference: most probes are judged as ties, and non-tie decisions more often favor Chinese. We add confidence intervals, paired significance tests, and an automatic transformation audit with a sensitivity analysis that excludes mechanically flagged high-risk variants. The experiment requires no model training, uses only API calls, and is feasible on modest local hardware.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
