arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.07673cs.CLcs.LG
收藏

面向玛雅语与西班牙语地方语之间非正式语域机器翻译的精选数据集与神经模型

Curated Datasets and Neural Models for Machine Translation of Informal Registers between Mayan and Spanish Vernaculars

Universitat d’Alacant(阿利坎特大学)

Andrés Lou, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena

AI总结:

本文构建并发布聚焦玛雅语与西班牙语日常非正式表达的 MayanV 平行语料,并训练神经翻译模型,证明通用资源难以反映真实语域。

摘要:

玛雅语族是一个拥有悠久历史、数百万使用者和巨大文化价值的语系,然而在语言资源和全球曝光度方面仍严重不足。本文开发、整理并公开发布了一组涵盖危地马拉和墨西哥南部多种玛雅语言的语料库,我们将其命名为 MayanV。这些数据集与该地区主导语言西班牙语平行,取自专注呈现非正式、日常和非特定领域语言的官方本土来源。因此,根据我们的方言计量分析,它们在语域上不同于大多数其他可用资源。此外,我们提出了神经机器翻译模型,在尽可能多的资源和玛雅语言上进行训练,并仅在我们的数据集上评估。我们观察到,资源中的西班牙语方言与更广泛使用的西班牙语书面标准之间存在词汇差异,而且除本文提出的资源外,其他资源似乎并未提升翻译性能,这表明许多此类资源可能无法准确捕捉常见的现实语言用法。MayanV 数据集可在 https://github.com/transducens/mayanv 获取。

英文摘要:

The Mayan languages comprise a language family with an ancient history, millions of speakers, and immense cultural value, that, nevertheless, remains severely underrepresented in terms of resources and global exposure. In this paper we develop, curate, and publicly release a set of corpora in several Mayan languages spoken in Guatemala and Southern Mexico, which we call MayanV. The datasets are parallel with Spanish, the dominant language of the region, and are taken from official native sources focused on representing informal, day-to-day, and non-domain-specific language. As such, and according to our dialectometric analysis, they differ in register from most other available resources. Additionally, we present neural machine translation models, trained on as many resources and Mayan languages as possible, and evaluated exclusively on our datasets. We observe lexical divergences between the dialects of Spanish in our resources and the more widespread written standard of Spanish, and that resources other than the ones we present do not seem to improve translation performance, indicating that many such resources may not accurately capture common, real-life language usage. The MayanV dataset is available at https://github.com/transducens/mayanv.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑