MC²：迈向面向中国少数民族语言的透明且具备文化意识的NLP
MC$^2$: Towards Transparent and Culturally-Aware NLP for Minority Languages in China
AI总结：
针对中国少数民族语言低资源与书写系统被忽视的问题，本文构建并开源最大同类语料库 MC²，以质量优先的采集流程提升数据准确性、多样性与模型文化意识。
摘要：
当前的大语言模型在理解低资源语言方面表现出不足，尤其是中国的少数民族语言。这一局限源于可用预训练数据的稀缺。为应对这一可获取性挑战，我们提出了 MC²，即中国少数民族语言多语言语料库（Multilingual Corpus of Minority Languages in China），它是迄今为止同类中最大的开源语料库。MC² 包含四种代表性不足的语言：藏语、维吾尔语、哈萨克语和蒙古语。值得注意的是，我们关注哈萨克语和蒙古语中较不常见的书写系统，即分别为哈萨克语阿拉伯字母和传统蒙古文字母，这些在以往的语料库构建工作中长期被忽视。认识到现有语料库中语言混杂现象的普遍存在，我们采用以质量为中心的方案来收集 MC²，在提升多样性的同时优先保证准确性。此外，我们强调关注书写系统多元性的重要性，这与所得模型的文化意识密切相关。MC² 语料库及相关模型已向社区公开。
英文摘要：
Current large language models demonstrate deficiencies in understanding low-resource languages, particularly the minority languages in China. This limitation stems from the scarcity of available pre-training data. To address this accessibility challenge, we present MC$^2$, a Multilingual Corpus of Minority Languages in China, which is the largest open-source corpus of its kind so far. MC$^2$ includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian. Notably, we focus on the less common writing systems of Kazakh and Mongolian, i.e., Kazakh Arabic script and traditional Mongolian script, respectively, which have been long neglected in previous corpus construction efforts. Recognizing the prevalence of language contamination within existing corpora, we adopt a quality-centric solution for collecting MC$^2$, prioritizing accuracy while enhancing diversity. Furthermore, we underscore the importance of attending to the multiplicity of writing systems, which is closely related to the cultural awareness of the resulting models. The MC$^2$ corpus and related models are made public to the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
