BabelJudge: 跨语言和智能体轨迹中LLM作为评判者可靠性的测量
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
AI总结：
提出BabelJudge基准，通过受控扰动生成已知标签的成对样本，无监督地测量评判模型的位置偏差、冗长偏差、顺序不一致和跨语言退化，发现Swahili顺序一致性接近随机。
摘要：
LLM作为评判者已成为NLP流程中可扩展评估的主流方法，但评判者本身存在系统偏差，这些偏差被原始准确率所掩盖：它们偏好放置在槽A中的回答（位置偏差），无论质量如何都偏好更长的回答（冗长偏差），并且在低资源语言中可靠性急剧下降。我们引入BabelJudge，一个开源基准和可靠性审计框架，该框架在无需人工偏好标签的情况下，测量任何评判模型的四种失败模式——位置偏差、冗长偏差、顺序不一致和跨语言退化。关键见解是通过退化进行黄金标注：从高质量参考回答开始，应用受控扰动生成一个成对项目，其黄金标签通过构造已知，从而消除标注成本。我们在英语、印地语、阿拉伯语和斯瓦希里语上评估Qwen2.5-7B-Instruct-4bit，发现我们的复合偏差惩罚可靠性得分从印地语的0.714下降到斯瓦希里语的0.550，而原始准确率（0.835 vs. 0.660）低估了这一差距。斯瓦希里语的顺序一致性降至0.480，意味着在槽位交换下评判结果接近随机——这是一种仅凭准确率无法发现的失败模式。我们进一步将框架扩展到智能体评估，通过九种轨迹级扰动（参数破坏、工具交换、幻觉调用、缺失步骤）和三个新指标：工具准确率、幻觉检测率和轨迹长度偏差。BabelJudge作为支持11个评判后端的Python包发布。代码：此 https URL
英文摘要：
LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
