当语言不一致时：自我进化的多语言LLM评判者
When Languages Disagree: Self-Evolving Multilingual LLM Judges
AI总结：
提出SEMJ方法，利用多语言评判中的跨语言不一致性进行迭代自我反思与重新评估，在多个基准上优于投票和反思基线，提升准确性和跨语言一致性。
摘要：
多语言LLM-as-a-judge被广泛用于跨语言评估模型输出，但存在跨语言不一致性问题（Fu and Liu, 2025）。现有方法通常将这种不一致性视为噪声，并通过投票或聚合来缓解。在本工作中，我们反而表明多语言不一致性可以提供互补的评估信号。我们的oracle分析发现，跨语言采样判断比单语言判断能获得更高的性能上限，表明不同语言可能包含互补的判断。受此发现启发，我们提出SEMJ，一种自我进化的多语言评判者，利用跨语言不一致性进行迭代优化。SEMJ为每个输入构建多语言变体，收集独立的判断和理由，并将不一致的输出反馈给自我反思和重新评估。在多个基准上的实验表明，SEMJ在准确性和跨语言一致性上始终优于投票和反思基线。进一步分析表明，不一致性触发了有用的重新评估，从而提高了判断质量。
英文摘要：
Multilingual LLM-as-a-judge is widely used to evaluate model outputs across languages, but suffers from cross-lingual inconsistency (Fu and Liu, 2025). Existing methods typically treat this inconsistency as noise and mitigate it through voting or aggregation. In this work, we instead show that multilingual inconsistency can provide complementary evaluation signals. Our oracle analysis finds that sampling judgments across languages yields a higher performance upper bound than single-language judging, indicating that different languages potentially include complementary judgments. Motivated by this finding, we propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement. SEMJ constructs multilingual variants of each input, collects independent judgments and rationales, and feeds inconsistent outputs back for self-reflection and re-evaluation. Experiments on multiple benchmarks show that SEMJ consistently outperforms voting and reflection baselines in both accuracy and cross-lingual consistency. Further analysis shows that inconsistency triggers useful re-evaluation, which improves judgment quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
