Yankari：一个单语约鲁巴语数据集
Yankari: A Monolingual Yoruba Dataset
AI总结：
本文提出Yankari，一个包含51,407篇文档、超3000万词元的单语约鲁巴语数据集，通过严谨方法填补NLP资源空白，推动模型开发与语言数字化。
摘要：
本文介绍了Yankari，一个大规模的单语约鲁巴语数据集，旨在解决这一重要的西非语言在自然语言处理（NLP）资源方面的严重缺口。尽管约鲁巴语有超过3000万使用者，但在NLP研究和应用中却严重缺乏代表性。我们详细介绍了创建该数据集的方法，包括谨慎的源选择、自动化质量控制和严格的数据清洗流程。Yankari数据集包含来自13个不同来源的51,407篇文档，总计超过3000万个词元。我们的方法侧重于道德的数据收集实践，避免有问题的来源，并解决现有数据集中普遍存在的问题。我们对该数据集进行了全面的自动化评估，证明其与现有资源相比的质量。Yankari数据集代表了约鲁巴语资源的一项重大进步，为开发更准确的NLP模型提供了基础，支持比较语言学研究，并促进了约鲁巴语的数字可访问性。
英文摘要：
This paper presents Yankari, a large-scale monolingual dataset for the Yoruba language, aimed at addressing the critical gap in Natural Language Processing (NLP) resources for this important West African language. Despite being spoken by over 30 million people, Yoruba has been severely underrepresented in NLP research and applications. We detail our methodology for creating this dataset, which includes careful source selection, automated quality control, and rigorous data cleaning processes. The Yankari dataset comprises 51,407 documents from 13 diverse sources, totaling over 30 million tokens. Our approach focuses on ethical data collection practices, avoiding problematic sources and addressing issues prevalent in existing datasets. We provide thorough automated evaluations of the dataset, demonstrating its quality compared to existing resources. The Yankari dataset represents a significant advancement in Yoruba language resources, providing a foundation for developing more accurate NLP models, supporting comparative linguistic studies, and contributing to the digital accessibility of the Yoruba language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
