多语言自然语言处理中的评估实践：机器翻译能否替代人工翻译？
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?
AI总结：
针对多语言语言模型评估语言覆盖不足的问题，分析现有评估框架局限，实证研究机器翻译替代人工翻译的可行性，发现高资源语言评估代表性尚可但低资源语言能力易被高估，且简单基线也有不错表现。
摘要：
尽管多语言语言模型（MLMs）已在100多种语言上完成训练，但由于大多数语言缺乏可用的测试数据，这类模型通常仅在少数几种语言上接受评估。在评估MLM对低资源语言和未见过语言的潜力时，这一问题尤为突出。本文中，我们分析了多语言NLP现有评估框架，讨论了其局限性，并提出了若干构建更稳健可靠评估实践的方向。此外，我们通过实证研究，探究了在跨大量语言上对MLM进行大规模评估时，机器翻译在多大程度上可作为人工翻译的可靠替代方案。我们使用一款SOTA（当前最优）翻译模型，将4项任务的测试数据翻译成198种语言，并用这些数据评估三个MLM。研究表明，尽管所选的高资源测试语言子集通常能充分代表更广泛的高资源语言，但我们往往会高估MLM在低资源语言上的能力。最后我们还发现，更简单的基线模型即便未受益于大规模多语言预训练，也能取得相对不错的性能。
英文摘要：
While multilingual language models (MLMs) have been trained on 100+ languages, they are typically only evaluated across a handful of them due to a lack of available test data in most languages. This is particularly problematic when assessing MLM's potential for low-resource and unseen languages. In this paper, we present an analysis of existing evaluation frameworks in multilingual NLP, discuss their limitations, and propose several directions for more robust and reliable evaluation practices. Furthermore, we empirically study to what extent machine translation offers a {reliable alternative to human translation} for large-scale evaluation of MLMs across a wide set of languages. We use a SOTA translation model to translate test data from 4 tasks to 198 languages and use them to evaluate three MLMs. We show that while the selected subsets of high-resource test languages are generally sufficiently representative of a wider range of high-resource languages, we tend to overestimate MLMs' ability on low-resource languages. Finally, we show that simpler baselines can achieve relatively strong performance without having benefited from large-scale multilingual pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
