大型语言模型在非洲语言中的现状：进展与挑战
The State of Large Language Models for African Languages: Progress and Challenges
AI总结：
本文比较分析了20个语言模型对非洲语言的覆盖现状，发现超过98%的非洲语言不受支持，并指出数据缺乏、分词偏差和高计算成本等挑战，呼吁语言标准化与社区语料库开发。
摘要：
大型语言模型（LLMs）正在变革自然语言处理（NLP），但对于非洲的2,000种低资源语言而言，其益处很大程度上是缺失的。本文对非洲语言在6个LLMs、8个小型语言模型（SLMs）和6个专用小型语言模型（SSLMs）中的覆盖情况进行了比较分析。评估涵盖了语言覆盖、训练集、技术限制、文字问题以及语言建模路线图。研究确定了42种受支持的非洲语言和23个可用的公共数据集，并揭示了一个巨大差距：4种语言（阿姆哈拉语、斯瓦希里语、南非荷兰语和马达加斯加语）总是被处理，而超过98%的非洲语言不受支持。此外，审查显示仅识别出拉丁语、阿拉伯语和吉兹语文字，而20种活跃文字被忽视。主要挑战包括缺乏数据、分词偏差、极高的计算成本和评估问题。这些问题要求进行语言标准化、由社区开发语料库以及针对非洲语言的有效适应方法。
英文摘要：
Large Language Models (LLMs) are transforming Natural Language Processing (NLP), but their benefits are largely absent for Africa's 2,000 low-resource languages. This paper comparatively analyzes African language coverage across six LLMs, eight Small Language Models (SLMs), and six Specialized SLMs (SSLMs). The evaluation covers language coverage, training sets, technical limitations, script problems, and language modelling roadmaps. The work identifies 42 supported African languages and 23 available public data sets, and it shows a big gap where four languages (Amharic, Swahili, Afrikaans, and Malagasy) are always treated while there is over 98\% of unsupported African languages. Moreover, the review shows that just Latin, Arabic, and Ge'ez scripts are identified while 20 active scripts are neglected. Some of the primary challenges are lack of data, tokenization biases, computational costs being very high, and evaluation issues. These issues demand language standardization, corpus development by the community, and effective adaptation methods for African languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
