“Be My Cheese?”：多语言大语言模型中机器翻译的文化细微差别基准
"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs
AI总结：
本文通过大规模人工评估基准，研究多语言大语言模型在机器翻译中处理文化细微差别（如习语、双关语、节日和文化概念）的能力，发现语法准确性与文化共鸣之间存在显著差距。
摘要：
我们提出了一个大规模人工评估基准，用于评估最先进的多语言大语言模型（LLMs）在机器翻译中的文化本地化能力。现有的机器翻译基准强调词元和语法准确性，但往往忽略了实际本地化所需的语用和文化能力。基于一项涵盖20种语言87个翻译的试点研究，我们评估了7个多语言LLMs在15个目标语言上的表现，每种语言有5名母语评分员。每位评分员对全文翻译和包含文化细微语言（习语、双关语、节日和文化嵌入概念）的片段级别实例，按0-3的序数质量等级评分；片段评分还包括一个“不适用”选项，用于未翻译的片段。在全文评估中，平均整体质量适中（1.68/3）：GPT-5（2.10/3）、Claude Sonnet 4（1.97/3）和Mistral Medium 3.1（1.84/3）构成最强梯队，灾难性失败较少。片段级别结果显示明显的类别效应：节日（2.20/3）和文化概念（2.19/3）的翻译明显优于习语（1.65/3）和双关语（1.45/3），且习语最可能未被翻译。使用Krippendorff's α和Gwet's AC2评估评分者间信度，显示总体一致性中等（Krippendorff's α = 0.45），其中双关语的一致性最低。这些发现表明语法充分性与文化共鸣之间存在持续差距。据我们所知，这是第一个明确关注翻译和本地化中文化细微差别的多语言、人工标注基准。结果凸显了对文化信息训练数据、改进跨语言语用学以及支持系统性文化翻译基准评估框架的需求。
英文摘要：
We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation produced by state-of-the-art multilingual large language models (LLMs). Existing MT benchmarks emphasise token-level and grammatical accuracy, but often overlook the pragmatic and culturally grounded competencies required for real-world localisation. Building on a pilot study of 87 translations across 20 languages, we evaluate 7 multilingual LLMs across 15 target languages with 5 native-speaker raters per language. Each rater scored both full-text translations and segment-level instances of culturally nuanced language (idioms, puns, holidays, and culturally embedded concepts) on an ordinal 0-3 quality scale; segment ratings additionally included an NA option for untranslated segments. Across full-text evaluations, mean overall quality is modest (1.68/3): GPT-5 (2.10/3), Claude Sonnet 4 (1.97/3), and Mistral Medium 3.1 (1.84/3) form the strongest tier with fewer catastrophic failures. Segment-level results show sharp category effects: holidays (2.20/3) and cultural concepts (2.19/3) translate notably better than idioms (1.65/3) and puns (1.45/3), and idioms are most likely to be left untranslated. Inter-rater reliability was assessed using Krippendorff's α and Gwet's AC2, indicating moderate agreement overall (Krippendorff's α = 0.45) with the lowest agreement for puns. These findings demonstrate a persistent gap between grammatical adequacy and cultural resonance. To our knowledge, this is the first multilingual, human-annotated benchmark focused explicitly on cultural nuance in translation and localisation. The results highlight the need for culturally informed training data, improved cross-lingual pragmatics, and evaluation frameworks that support systematic benchmarking of culturally grounded translation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
