arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.06517cs.CL
收藏

方言很重要:伊加拉语-英语机器翻译的方言感知基线

Dialect Matters: A Dialect-Aware Baseline for Igala-English Machine Translation

University of Jos(乔斯大学);Federal University of Applied Sciences Kachia(卡奇亚联邦应用科学大学)

Etuh E., Eli. A. Jiya

AI总结:

本文针对伊加拉语无平行语料库的问题,提出首个带方言分析的伊加拉语-英语机器翻译基线,采用混合方言训练,由Helsinki-NLP/opus-mt-en-mul微调得到,为该领域建立了研究基线。

摘要:

智能系统在人类活动各领域的广泛渗透,极大提升了清晰度、工作便捷性,增进了全球通用理解,这显著推动了各类用户对智能系统的使用。智能系统的输出正确性高度依赖精准的数据集/语料库。尽管研究人员已致力于解决自然语言处理(NLP)中的低资源语言问题,但伊加拉语仍无平行语料库。本文提出伊加拉语-英语机器翻译的方言感知基线,是首批开放系统之一,也是首个开展方言变异分析的系统。本文的1137句数据集(其中899句为训练集)是首个带方言标签的数据集:Idah方言459句(占40.4%)、Ankpa方言297句(占26.1%)、Ibaji方言160句(占14.1%)、其他方言187句(占16.4%)、Ogugu方言32句(占2.8%)。同一英语句子可对应20-60种伊加拉语形式。混合方言训练的平衡BLEU值为11.66,而仅用Idah方言训练的模型在Ibaji方言上的BLEU值下降了4.7(在Idah方言上为13.2,在Ibaji方言上为8.5),证明方言公平性需要混合方言训练。该模型由Helsinki-NLP/opus-mt-en-mul(78M参数)微调而来,轮次消融实验显示在8轮时收敛,15轮时过拟合。模型的BLEU值为11.66,chrF值为46.69。分析显示899句的对数线性趋势与非洲低资源语言文献一致。本研究为伊加拉语提供了首个方言影响分析,并为后续工作建立了基线。

英文摘要:

The massive permeability of intelligent systems in every sphere of human activities have greatly enhanced clarity, ease of work, and increased general global understanding which has significantly awaken the use of intelligent systems by all users. Intelligent systems greatly rely on precise dataset/corpus for the correctness of their output. Despite efforts by researchers to address Low-resource language in the Natural Language Processing (NLP), Igala still has zero parallel corpora despite efforts by researchers. We present a dialect-aware baseline for Igala-English machine translation, among first open systems and first with dialectal variation analysis. Our dataset of 1137 sentences (899 training) is first with dialect tags: Idah 459 (40.4%), Ankpa 297 (26.1%), Ibaji 160 (14.1%), Other 187 (16.4%), Ogugu 32 (2.8%). Same English yields 20-60 Igala forms. Mixed-dialect training achieves balanced BLEU 11.66 versus Idah-only model which drops 4.7 BLEU on Ibaji (13.2 on Idah, 8.5 on Ibaji), proving dialect fairness requires mixing. Fine-tuned from Helsinki-NLP/opus-mt-en-mul (78M), epoch ablation shows convergence at 8 epochs and overfitting at 15. BLEU 11.66, chrF 46.69. Analysis shows 899 sentences on log-linear trend consistent with African low-resource literature. This work provides first dialectal impact analysis for Igala and establishes baseline for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑