arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.09015cs.CLcs.LG
收藏

LLiMba:单块GPU上的萨丁语——将一个3B语言模型适应到一种消失的罗曼语

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language

Qwen2.5-3B-Instruct

Luca Ballore

AI总结:

本文提出LLiMba,通过持续预训练和监督微调,在单块24GB显卡上将Qwen2.5-3B-Instruct模型适应到萨丁语,实现低资源罗曼语的高效生成,展示了rsLoRA r256在翻译性能上的优势。

摘要:

萨丁语是一种拥有约一百万使用者的罗曼语,目前在现代自然语言处理中存在明显不足。商业服务不支持其,现有语言模型也无法可靠生成。我们提出了LLiMba,一个3B参数的萨丁语专用模型,通过持续预训练(CPT)和监督微调(SFT)在单块24GB消费级GPU上进行适应。语料库包含1150万个萨丁语词汇,涵盖LSC、Logudorese和Campidanese方言,并辅以240万个相关罗曼语文本作为反例对抗方言模糊。经过CPT后,模型在测试的萨丁语上达到6.76的困惑度,并在所有六个FLORES-200方向上优于基础模型。我们比较了五种SFT配置:全微调、LoRA r64、rsLoRA r128、rsLoRA r256和DoRA r256。rsLoRA r256在所有方向上胜出,从英语翻译萨丁语的BLEU得分为28.5,比CPT后的17.3和全微调的21.0更高。秩的消融实验显示rsLoRA r256在BLEU上优于LoRA r64,但揭示了其他变体无法察觉的失败模式,包括跨脚本泄漏。LoRA r64保留的训练内容较少,生成更多自信的虚构内容,尽管所有方法都基于训练数据中不存在的内容进行生成。DoRA r256在训练与评估之间的差距最小,但事实准确性最差。研究发现,适配器容量比LoRA变体的选择更重要,更强的正则化并不总是有益,翻译指标可以平滑地排序行为差异显著的配置。困惑度比较必须考虑字节回退的分词,这会低估非拉丁语系的指标。

英文摘要:

Sardinian, a Romance language with roughly one million speakers, has minimal presence in modern NLP. Commercial services do not support it, and current language models do not produce it reliably. We present LLiMba, a 3B parameter Sardinian-ready model adapted from Qwen2.5-3B-Instruct through continued pretraining (CPT) and supervised fine-tuning (SFT) on a single 24 GB consumer GPU. The corpus contains 11.5 million tokens of Sardinian spanning LSC, Logudorese, and Campidanese, augmented with 2.4 million tokens of related Romance text as replay against register blurring. After CPT the model reaches a perplexity of 6.76 on held out Sardinian and outperforms the base across all six FLORES-200 directions. We compare five SFT configurations under matched conditions: full fine-tuning, LoRA r64, rsLoRA r128, rsLoRA r256, and DoRA r256. rsLoRA r256 wins on every direction into Sardinian, reaching 28.5 BLEU from English against 17.3 after CPT and 21.0 with full fine-tuning. The rank ablation places r128 between LoRA r64 and rsLoRA r256 on BLEU but reveals failure modes invisible to the metric, including leakage across scripts no other variant produces. LoRA r64 retains less factual content from SFT than configurations at higher rank and produces more confident fabrications, though all methods fabricate on content absent from training. DoRA r256 yields the smallest gap between training and evaluation but the worst factual accuracy. The findings indicate that adapter capacity matters more than the choice among LoRA variants for adapting a Romance pretrained base to a low resource Romance target, that stronger regularization is not uniformly beneficial, and that translation metrics smoothly order configurations whose qualitative behavior differs categorically. Perplexity comparisons across scripts must account for byte fallback tokenization, which deflates the metric for scripts other than Latin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑