arXiv 2507.092052026-05-13cs.CL

从 curated 数据到可扩展模型：为藏语持续预训练密集和 MoE 大语言模型

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan

TJUNLP Lab（TJUNLP实验室）；School of Computer Science and Technology（计算机科学与技术学院）；Tianjin University（天津大学）

Lei Yang, Leiyu Pan, Bojian Xiong, Renren Jin, Shaowei Zhang, Yue Chen, Ling Shi, Jiang Zhou, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuo… 展开作者

Lei Yang, Leiyu Pan, Bojian Xiong, Renren Jin, Shaowei Zhang, Yue Chen, Ling Shi, Jiang Zhou, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuowen Han, Zhuo Chen, Yuqi Ren, Deyi Xiong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种全面的管道，通过大规模数据整理和持续预训练推进藏语语言模型。构建了72GB高质量藏语语料库，并通过平衡多语言持续预训练和多语言指令微调改进Qwen2.5-7B模型，最终扩展为50B-A10B的MoE架构，实验结果显示模型在多种任务中优于现有开源和藏语专用模型。

摘要：

大型语言模型（LLMs）在自然语言处理任务中取得了显著成功，但其性能仍然严重偏向高资源语言。藏语，尽管具有文化意义和大量说话人口，仍然严重代表性不足。在本工作中，我们提出了一种全面的管道，通过大规模数据整理和持续预训练推进藏语语言模型。我们构建了72GB的高质量藏语语料库，这是目前最大的语料库，并通过平衡多语言持续预训练与藏语、中文和英语，随后进行多语言指令微调。为了进一步高效扩展容量，我们将密集模型扩展为50B-A10B的混合专家（MoE）架构。由于缺乏标准化的藏语基准，我们通过高质量翻译和人工验证构建了多个评估数据集。实验结果表明，密集和MoE模型在各种任务中均优于现有开源和藏语专用模型。我们的工作推进了以藏语为中心的LLM研究，并为扩展LLM到其他低资源语言提供了可转移的见解。我们将发布模型权重、评估基准和详细数据处理文档。

英文摘要：

Large language models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks, yet their performance remains heavily biased toward high-resource languages. Tibetan, despite its cultural significance and large speaker population, is still substantially underrepresented. In this work, we present a comprehensive pipeline for advancing Tibetan language modeling through large-scale data curation and continual pre-training. We construct a 72 GB high-quality Tibetan corpus, the largest to date, and adapt Qwen2.5-7B through balanced multilingual continual pre-training with Tibetan, Chinese, and English, followed by multilingual instruction tuning. To further scale capacity efficiently, we extend the dense model to a 50B-A10B Mixture-of-Experts architecture. Due to the absence of standardized Tibetan benchmarks, we build multiple evaluation datasets via high-quality translation and human verification. Experimental results show that both dense and MoE models consistently outperform existing open-source and Tibetan-focused models of similar scale across diverse tasks. Our work advances Tibetan-centric LLM research and provides transferable insights for extending LLMs to other low-resource languages. We will release the model weights, evaluation benchmarks, and detailed data processing documentation in the follow-up.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。