arXiv 2603.020412026-08-26cs.CLcs.AI

EstLLM：通过持续预训练和后训练增强多语言大语言模型中的爱沙尼亚能力

EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院）；Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系）；Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院）；School of Humanities, Tallinn University（塔林大学人文学院）

Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

EstLLM通过持续预训练和后训练提升多语言大语言模型中爱沙尼亚的能力，增强语言和推理表现。

摘要：

大型语言模型（LLMs）主要在以英语为中心的数据上进行训练，导致较小语言的表现不均衡。我们研究持续预训练（CPT）是否能够显著提高预训练多语言LLM中爱沙尼亚的能力，同时保持其英语和一般推理性能。使用Llama 3.1 8B作为主要基础模型，我们对混合数据进行持续预训练，增加爱沙尼亚的暴露度，同时通过英语回放和包含代码、数学和指令样数据来近似原始训练分布。随后我们应用监督微调、偏好优化和聊天向量合并，以引入稳健的指令遵循行为。在全面的爱沙尼亚基准测试中，评估显示与原始基础模型及其指令微调变体相比，语言能力、知识、推理、翻译质量和指令遵循性均有所提高，同时在英语基准测试中保持竞争性表现。这些发现表明，通过适当平衡的数据混合，CPT与后训练对齐相结合，可以显著提高预训练多语言LLM中的单语言能力。

英文摘要：

Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can improve Estonian capabilities in multilingual LLMs while preserving English and general reasoning performance. Using Llama 3.1 8B and Apertus 8B as base models, we apply CPT with Estonian-enriched multilingual replay, followed by mostly English supervised fine-tuning, preference optimization, and chat vector merging. Evaluation on Estonian benchmarks, targeted pairwise human evaluation, and an Estonian Chatbot Arena-style setup shows consistent improvements in Estonian language competence, reasoning, translation, and instruction-following. Although Apertus exhibits stronger Estonian capabilities before adaptation, the more English-centric Llama model achieves substantially larger gains after adaptation. While some English capabilities regress relative to the original instruction-tuned models, chat vector merging substantially restores English instruction-following and reasoning performance. These findings suggest that CPT with balanced multilingual replay and lightweight post-training alignment can substantially improve single-language capabilities in multilingual LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。