Kencorpus：用于自然语言处理任务的斯瓦希里语、Dholuo语和Luhya语肯尼亚语言语料库
Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks
AI总结：
本文介绍Kencorpus，一个涵盖斯瓦希里语、Dholuo语和Luhya语的文本与语音语料库，并通过词性标注、问答、翻译及两个概念验证系统支持低资源语言NLP研究。
摘要：
非洲本土语言在自然语言处理中被归类为服务不足的语言。因此，它们面临较差的数字包容性和信息获取状况。处理这类语言的挑战一直在于如何在缺乏必要数据的情况下使用机器学习和深度学习模型。Kencorpus项目旨在通过收集并存储质量足以支持数据驱动解决方案的文本和语音数据来弥合这一差距，这些解决方案可用于多语言社区中的机器翻译、问答和转写等应用。Kencorpus数据集是一个文本和语音语料库，涵盖三种主要在肯尼亚使用的语言：斯瓦希里语、Dholuo语和Luhya语。数据收集由来自社区、学校、媒体和出版商的研究人员完成。Kencorpus的数据集包含5,594个项目——4,442份文本（560万词）和1,152个语音文件（177小时）。基于这些数据，研究人员为Dholuo语和Luhya语开发了词性标注集（分别为50,000词和93,000词）。我们为斯瓦希里语开发了7,537个问答对，并创建了一个包含13,400个句子的文本翻译集，内容从Dholuo语和Luhya语翻译为斯瓦希里语。这些数据集可用于模型训练和翻译等下游机器学习任务。我们还开发了两个概念验证系统：一个用于Kiswahili语语音转文本，另一个是用于问答任务的机器学习系统，其结果分别为18.87%的词错误率和80%的精确匹配率（EM）。这些初步结果表明Kencorpus对机器学习社区具有很大的可用性前景。Kencorpus是面向这三种低资源语言的少数公有领域语料库之一，并为类似工作（尤其是针对低资源语言的工作）奠定了学习和经验分享的基础。
英文摘要：
Indigenous African languages are categorized as under-served in Natural Language Processing. They therefore experience poor digital inclusivity and information access. The processing challenge with such languages has been how to use machine learning and deep learning models without the requisite data. The Kencorpus project intends to bridge this gap by collecting and storing text and speech data that is good enough for data-driven solutions in applications such as machine translation, question answering and transcription in multilingual communities. The Kencorpus dataset is a text and speech corpus for three languages predominantly spoken in Kenya: Swahili, Dholuo and Luhya. Data collection was done by researchers from communities, schools, media, and publishers. The Kencorpus' dataset has a collection of 5,594 items - 4,442 texts (5.6M words) and 1,152 speech files (177hrs). Based on this data, Part of Speech tagging sets for Dholuo and Luhya (50,000 and 93,000 words respectively) were developed. We developed 7,537 Question-Answer pairs for Swahili and created a text translation set of 13,400 sentences from Dholuo and Luhya into Swahili. The datasets are useful for downstream machine learning tasks such as model training and translation. We also developed two proof of concept systems: for Kiswahili speech-to-text and machine learning system for Question Answering task, with results of 18.87% word error rate and 80% Exact Match (EM) respectively. These initial results give great promise to the usability of Kencorpus to the machine learning community. Kencorpus is one of few public domain corpora for these three low resource languages and forms a basis of learning and sharing experiences for similar works especially for low resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
