MzansiText 和 MzansiLM：一种面向南非语言的开放语料库和解码器-only 语言模型
MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages
AI总结：
本文介绍MzansiText和MzansiLM，针对南非11种官方语言中的9种低资源语言，通过任务特定微调在自然语言理解与生成任务中取得显著成果，但小规模下少量推理仍具挑战性。
摘要：
解码器-only 语言模型可通过指令微调适应多种任务，但其在小规模和低资源语言上的泛化能力尚不明确。本文聚焦南非语言，介绍MzansiText，一个经过可重复过滤流程的多语言预训练语料库，以及MzansiLM，一个1.25亿参数的语言模型。通过三种微调模式评估MzansiLM：单语言任务特定微调、多语言任务特定微调和通用多任务指令微调。单语言任务特定微调在数据到文本生成任务中表现优异，达到isiXhosa语言20.65 BLEU分数，与更大规模的编码器-解码器基线竞争。多语言任务特定微调在主题分类任务中对相关语言有益，达到isiXhosa新闻分类78.5%的宏F1分数。尽管MzansiLM能有效适应监督NLU和NLG任务，但在此模型规模下少量推理仍具挑战性，即使对于更大规模的解码器-only模型，表现接近随机。本文发布MzansiText和MzansiLM，提供可重复的解码器-only基线和南非语言在小规模下的清晰适应策略指导。
英文摘要：
Decoder-only language models can be adapted to diverse tasks through instruction finetuning, but the extent to which this generalizes at small scale for low-resource languages remains unclear. We focus on the languages of South Africa, where we are not aware of a publicly available decoder-only model that explicitly targets all eleven official written languages, nine of which are low-resource. We introduce MzansiText, a curated multilingual pretraining corpus with a reproducible filtering pipeline, and MzansiLM, a 125M-parameter language model trained from scratch. We evaluate MzansiLM on natural language understanding and generation using three adaptation regimes: monolingual task-specific finetuning, multilingual task-specific finetuning, and general multi-task instruction finetuning. Monolingual task-specific finetuning achieves strong performance on data-to-text generation, reaching 20.65 BLEU on isiXhosa and competing with encoder-decoder baselines over ten times larger. Multilingual task-specific finetuning benefits closely related languages on topic classification, achieving 78.5% macro-F1 on isiXhosa news classification. While MzansiLM adapts effectively to supervised NLU and NLG tasks, few-shot reasoning remains challenging at this model size, with performance near chance even for much larger decoder-only models. We release MzansiText and MzansiLM to provide a reproducible decoder-only baseline and clear guidance on adaptation strategies for South African languages at small scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
