arXiv 2311.098282024-04-23cs.CL

AfriMTE和AfriCOMET：增强COMET以涵盖资源匮乏的非洲语言

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

University College London（伦敦大学学院）；Masakhane NLP（Masakhane自然语言处理组织）；University of Maryland（马里兰大学）；University of Oxford（牛津大学）；Unbabel（Unbabel公司）；Instituto Superior Técnico（高等技术学院）；INESC-ID（INESC-ID研究所）；ENSIAS（国家高等计算机与系统分析学院）；SADiLaR（南非洲数字语言资源中心）；Aston University（阿斯顿大学）；University of Eastern Finland（东芬兰大学）；Luleå University of Technology（吕勒奥理工大学）；Fudan University（复旦大学）；Conservatoire National des Arts et Métiers（法国国立工艺学院）；Lelapa AI（Lelapa人工智能公司）；Imperial College London（帝国理工学院）；University of Deusto（德乌斯托大学）

Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhi… 展开作者

Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Hassan Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Abdi Mohamed, Ayinde Hassan, Oluwabusayo Olufunke Awoyomi, Lama Alkhaled, Sana Al-Azzawi, Naome A. Etori, Millicent Ochieng, Clemencia Siro, Samuel Njoroge, Eric Muchiri, Wangari Kimotho, Lyse Naomi Wamba Momo, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Nasir Iro, Saheed S. Abdullahi, Stephen E. Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Raphael Ogbu, Sam Brian, Verrah Akinyi Otiende, Chinedu Emmanuel Mbonu, Sakayo Toadoum Sari, Yao Lu, Pontus Stenetorp

备注：: Accepted by NAACL 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对资源匮乏非洲语言MT评估数据缺失及COMET等指标适用性受限的问题，本文构建了13种非洲语言的高质量人类评估数据，并提出AfriCOMET评估指标，实现了0.441的Spearman相关性。

摘要：

尽管在将多语言机器翻译（MT）扩展至多种资源匮乏的非洲语言方面近期取得了进展，但准确衡量这一进展仍具挑战性，因为评估通常基于BLEU等n-gram匹配指标进行，而这些指标通常与人类判断的相关性较弱。诸如COMET等学习型指标具有更高的相关性；然而，资源匮乏语言缺乏带有人类评分的评估数据、Multidimensional Quality Metrics (MQM)等标注指南的复杂性，以及多语言编码器有限的语言覆盖范围，阻碍了它们在非洲语言上的适用性。在本文中，我们通过为13种类型多样的非洲语言创建高质量的人类评估数据（采用简化的MQM指南进行错误检测和直接评估（DA）评分）来解决这些挑战。此外，我们通过利用来自资源丰富语言的DA数据和以非洲为中心的多语言编码器（AfroXLM-R）开发了AfriCOMET：针对非洲语言的COMET评估指标，从而创建了在与人类判断的Spearman等级相关性（0.441）方面最先进的非洲语言MT评估指标。

英文摘要：

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。