构建与扩展面向印度尼西亚当地语言的低资源与代表性不足平行数据集
Constructing and Expanding Low-Resource and Underrepresented Parallel Datasets for Indonesian Local Languages
AI总结:
本文提出涵盖五种印度尼西亚当地语言的多语言平行语料库Bhinneka Korpus,详细说明数据收集过程与挑战,并用IBM Model 1进行翻译实验,验证了低资源语言翻译的可行性,为后续多语言翻译模型研究奠定基础。
摘要:
在印度尼西亚,当地语言在文化中发挥着不可或缺的作用。然而,可用的印度尼西亚语言资源在自然语言处理(NLP)领域仍属于数据有限类别。当为这些语言构建NLP模型时,这一问题变得尤为突出。为弥补这一不足,我们提出了Bhinneka Korpus,一个涵盖五种印度尼西亚当地语言的多语言平行语料库。我们的目标是提升这些资源的可获取性与利用率,扩大其在国内的覆盖范围。我们详细说明了数据集的收集过程及相关挑战。此外,由于数据限制,我们使用IBM Model 1进行了翻译任务实验。结果表明,每种语言的表现已经显示出进一步发展的良好迹象。我们讨论了词汇变异、平滑效应和跨语言变异性等挑战。我们计划使用面向低资源语言的先进NLP技术对该语料库进行评估,为多语言翻译模型铺平道路。
英文摘要:
In Indonesia, local languages play an integral role in the culture. However, the available Indonesian language resources still fall into the category of limited data in the Natural Language Processing (NLP) field. This is become problematic when build NLP model for these languages. To address this gap, we introduce Bhinneka Korpus, a multilingual parallel corpus featuring five Indonesian local languages. Our goal is to enhance access and utilization of these resources, extending their reach within the country. We explained in a detail the dataset collection process and associated challenges. Additionally, we experimented with translation task using the IBM Model 1 due to data constraints. The result showed that the performance of each language already shows good indications for further development. Challenges such as lexical variation, smoothing effects, and cross-linguistic variability are discussed. We intend to evaluate the corpus using advanced NLP techniques for low-resource languages, paving the way for multilingual translation models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。