从词语到世界：评估机器翻译中跨文化文化理解的基准测试
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
AI总结：
本文提出CulT-Eval基准测试，用于评估机器翻译对文化相关表达的处理能力，发现现有指标无法有效捕捉文化语义偏差，提出新的评估指标以改进翻译质量。
摘要：
文化表达，如习语、俚语和文化特定项目（CSIs），在自然语言中普遍存在，编码了超越字面语言形式的意义。准确翻译此类表达对机器翻译系统仍具挑战性。尽管如此，现有基准测试仍碎片化，缺乏系统框架评估翻译性能。为此，我们引入CulT-Eval基准测试，包含超过7,959个精心挑选的实例，涵盖多种文化相关表达类型，配有全面的错误分类学。通过广泛评估大型语言模型和详细分析，我们发现现有自动指标无法有效捕捉系统性失败模式。因此，我们提出互补的评估指标，针对由文化引起的语义偏差进行评估。结果表明，当前模型难以保持文化相关意义，并无法捕捉翻译中至关重要的文化与语境细微差别。我们的基准和代码可在https://anonymous.4open.science/r/CulT-Eval-E75D/获取。
英文摘要：
Culture-expressions, such as idioms, slang, and culture-specific items (CSIs), are pervasive in natural language and encode meanings that go beyond literal linguistic form. Accurately translating such expressions remains challenging for machine translation systems. Despite this, existing benchmarks remain fragmented and do not provide a systematic framework for evaluating translation performance on culture-loaded expressions. To address this gap, we introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions. CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions, with a comprehensive error taxonomy covering culturally grounded expressions. Through extensive evaluation of large language models and detailed analysis, we identify recurring and systematic failure modes that are not adequately captured by existing automatic metrics. Accordingly, we propose a complementary evaluation metric that targets culturally induced meaning deviations overlooked by standard MT metrics. The results indicate that current models struggle to preserve culturally grounded meaning and to capture the cultural and contextual nuances essential for accurate translation. Our benchmark and code are available at https://anonymous.4open.science/r/CulT-Eval-E75D/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
