量化翻译错误对多语言大语言模型评估的影响
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
AI总结：
研究机器翻译基准中的翻译错误如何影响多语言LLM评估的可靠性，通过自动错误跨度检测和准确性下降分析揭示翻译错误与评估指标之间的关联。
摘要：
机器翻译基准被广泛用于评估大语言模型（LLM）的多语言能力，然而这些基准中的翻译错误仍未得到充分探索，引发了对多语言评估可靠性和可比性的担忧。我们解决了两个实际差距：（i）来自LLM评判者的自动MQM风格错误跨度以及跨度感知的QE基线（xCOMET-XXL）与基准翻译上的专家人工跨度注释的匹配程度，以及（ii）翻译错误（相对于英文原版中的源端问题）在多大程度上解释了翻译基准上的准确性下降。我们发现，在自然发生的基准翻译上，跨度一致性并非易事，并且目标端翻译错误始终与可测量的、百分点级别的翻译准确性下降相关，即使在控制了英文正确性和源端异常之后也是如此。
英文摘要：
Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
