多语言提示本地化用于法官-代理系统：在需求级评估中的语言与骨干敏感性
Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation
AI总结：
研究探讨了多语言提示在法官-代理系统评估中的影响，发现语言与骨干模型的交互作用显著，不同语言下不同模型表现各异，强调语言应作为评估变量。
摘要：
在代理代码基准中，评估语言通常被视为固定英文默认设置，但本文表明改变法官语言可翻转骨干模型排名。将Agent-as-a-Judge提示栈本地化到五种语言（英语、阿拉伯语、土耳其语、中文、印地语）并评估55个DevAI开发任务，涵盖三个开发者-代理框架和六个法官骨干模型，总计4950次法官运行。核心发现是骨干模型和语言存在交互作用：GPT-4o在英语中满意度最高（44.72%），Gemini在阿拉伯语（51.72%，p<0.001 vs GPT-4o）和印地语（53.22%）领先。无单一骨干模型在所有语言中占优，不同骨干模型在个体需求判断上的一致性较低（Fleiss' κ≤0.231）。受控消融实验进一步显示，本地化法官侧指令而非仅基准内容可决定性：印地语满意度在部分本地化下从42.8%降至23.2%。这些结果表明，语言应在代理基准中作为显式评估变量。完整的需求级判断和运行时统计数据已发布以供复现。
英文摘要：
Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings. We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi) and evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs. The central finding is that backbone and language interact: GPT-4o achieves the highest satisfaction in English (44.72\%), while Gemini leads in Arabic (51.72\%, $p<0.001$ vs.\ GPT-4o) and Hindi (53.22\%). No single backbone dominates across all languages, and inter-backbone agreement on individual requirement judgments is modest (Fleiss' $κ\leq 0.231$). A controlled ablation further shows that localizing judge-side instructions, not just benchmark content, can be decisive: Hindi satisfaction drops from 42.8\% to 23.2\% under partial localization. These results indicate that language should be treated as an explicit evaluation variable in agentic benchmarks. Full requirement-level judgments and runtime statistics are released for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
