构建低资源非洲语言语料库：以Kidawida、Kalenjin和Dholuo为例
Building low-resource African language corpora: A case study of Kidawida, Kalenjin and Dholuo
AI总结：
本文通过众包方法为三种肯尼亚低资源语言构建平行文本和语音语料库，并开放共享，以促进非洲语言的自然语言处理研究和应用。
摘要：
自然语言处理是人工智能的关键前沿领域，在公共卫生、农业、教育和商业等多个领域具有广泛应用。然而，由于缺乏大量的语言资源，许多非洲语言在这一数字化转型中代表性不足。本文针对三种资源不足的肯尼亚语言——Kidaw'ida、Kalenjin和Dholuo，开展了语言语料库开发的案例研究，旨在推动非洲社区的自然语言处理和语言学研究。我们的项目历时一年，采用选择性众包方法，从这些语言的母语者处收集文本和语音数据。数据收集包括：（1）录制对话并将所得文本翻译成斯瓦希里语，从而创建平行语料库；（2）朗读并录制书面文本以生成语音语料库。我们通过开放研究平台免费提供这些资源，即平行文本语料库存放于Zenodo，语音数据集存放于Mozilla Common Voice，从而促进持续贡献，并方便开发者训练模型和开发自然语言处理应用。该项目展示了基层语料库建设工作如何支持非洲语言融入人工智能创新。除了填补资源空白外，这些语料库对于促进语言多样性和赋能当地社区至关重要，它们使自然语言处理应用能够满足当地需求。随着肯尼亚等非洲国家日益拥抱数字化转型，开发本土语言资源对于包容性增长必不可少。我们鼓励母语者和开发者继续合作，以扩展和利用这些语料库。
英文摘要：
Natural Language Processing is a crucial frontier in artificial intelligence, with broad applications in many areas, including public health, agriculture, education, and commerce. However, due to the lack of substantial linguistic resources, many African languages remain underrepresented in this digital transformation. This paper presents a case study on the development of linguistic corpora for three under-resourced Kenyan languages, Kidaw'ida, Kalenjin, and Dholuo, with the aim of advancing natural language processing and linguistic research in African communities. Our project, which lasted one year, employed a selective crowd-sourcing methodology to collect text and speech data from native speakers of these languages. Data collection involved (1) recording conversations and translation of the resulting text into Kiswahili, thereby creating parallel corpora, and (2) reading and recording written texts to generate speech corpora. We made these resources freely accessible via open-research platforms, namely Zenodo for the parallel text corpora and Mozilla Common Voice for the speech datasets, thus facilitating ongoing contributions and access for developers to train models and develop Natural Language Processing applications. The project demonstrates how grassroots efforts in corpus building can support the inclusion of African languages in artificial intelligence innovations. In addition to filling resource gaps, these corpora are vital in promoting linguistic diversity and empowering local communities by enabling Natural Language Processing applications tailored to their needs. As African countries like Kenya increasingly embrace digital transformation, developing indigenous language resources becomes essential for inclusive growth. We encourage continued collaboration from native speakers and developers to expand and utilize these corpora.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
