小模型，大影响：面向低资源语言的小型多语言语言模型的高效语料库与基于图的适配
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages
AI总结：
本文系统比较三种参数高效适配器架构，结合GlotCC文本与ConceptNet图谱使小型多语言模型适配低资源语言，证明其可在少参数下超越全量微调和大型LLM。
摘要：
低资源语言（LRL）由于数据有限，在自然语言处理（NLP）中面临重大挑战。尽管当前最先进的大型语言模型（LLM）在处理LRL时仍显吃力，但mBERT和XLM-R等较小的多语言模型（mLM）因其能力与较低训练数据规模更匹配而展现出更大前景。本研究系统考察了用于使mLM适配LRL的参数高效适配器方法，评估了三种架构：Sequential Bottleneck、Invertible Bottleneck和Low-Rank Adaptation。利用来自GlotCC的非结构化文本以及来自ConceptNet的结构化知识，我们表明小型适配数据集（例如至多1 GB自由文本或几MB知识图谱数据）能够在内在任务（掩码语言建模）和外在任务（主题分类、情感分析和命名实体识别）上带来提升。我们发现，Sequential Bottleneck适配器在语言建模方面表现优异，而Invertible Bottleneck适配器由于更好的嵌入对齐和更多的参数量，在下游任务上略优于其他方法。基于适配器的方法在使用少得多参数的同时，达到或超过全量微调的效果；并且事实证明，对于LRL，较小的mLM比LLaMA-3、GPT-4以及基于DeepSeek-R1的蒸馏模型等庞大LLM更有效。尽管适配能够提升性能，但预训练数据规模仍是主导因素，对于预训练覆盖广泛的语言而言尤其如此。
英文摘要：
Low-resource languages (LRLs) face significant challenges in natural language processing (NLP) due to limited data. While current state-of-the-art large language models (LLMs) still struggle with LRLs, smaller multilingual models (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of their capacity to low training data sizes. This study systematically investigates parameter-efficient adapter-based methods for adapting mLMs to LRLs, evaluating three architectures: Sequential Bottleneck, Invertible Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and structured knowledge from ConceptNet, we show that small adaptation datasets (e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains in intrinsic (masked language modeling) and extrinsic tasks (topic classification, sentiment analysis, and named entity recognition). We find that Sequential Bottleneck adapters excel in language modeling, while Invertible Bottleneck adapters slightly outperform other methods on downstream tasks due to better embedding alignment and larger parameter counts. Adapter-based methods match or outperform full fine-tuning while using far fewer parameters, and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3, GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves performance, pre-training data size remains the dominant factor, especially for languages with extensive pre-training coverage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
