探索构建特定语言大语言模型（LLM）的设计选择
Exploring Design Choices for Building Language-Specific LLMs
AI总结：
本研究系统探究适配现有单语、多语大语言模型构建特定语言LLM的各类设计选择的影响，发现初始性能不代表适配后效果、简单词表扩展即可提效、最优方案依语言而定，为高效构建特定语言LLM奠定基础。
摘要：
尽管大语言模型（LLM）发展迅速，但它们在绝大多数语言上的表现仍不尽如人意。本文研究通过适配单语和多语LLM来构建特定语言LLM的方法。我们针对设计选择（基础模型选择、词表扩展、持续预训练）如何影响适配后的LLM开展了系统实验，评估维度包括效率（编码等量信息所需的token数量）和下游任务性能。我们发现：（1）LLM的初始性能并不总是与适配后的最终性能相关。适配以英语为中心的模型，尽管其在低资源语言上的初始性能较差，但最终效果可能优于适配多语模型。（2）在我们研究的大多数LLM中，通过简单的词表扩展和持续预训练就能轻松提升效率。（3）最优适配方法（基础模型选择、新词表大小、训练数据、初始化策略）高度依赖具体语言，且最简单的嵌入初始化方式在各类实验设置中都表现良好。综上，本研究为通过适配现有LLM高效构建特定语言LLM奠定了基础。
英文摘要：
Despite rapid progress in large language models (LLMs), their performance on a vast majority of languages remains unsatisfactory. In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs. We conduct systematic experiments on how design choices (base model selection, vocabulary extension, and continued pretraining) impact the adapted LLM, both in terms of efficiency (how many tokens are needed to encode the same amount of information) and end task performance. We find that (1) the initial performance of LLM does not always correlate with the final performance after the adaptation. Adapting an English-centric models can yield better results than adapting multilingual models despite their worse initial performance on low-resource languages. (2) Efficiency can easily improved with simple vocabulary extension and continued pretraining in most LLMs we study, and (3) The optimal adaptation method (choice of the base model, new vocabulary size, training data, initialization strategy) is highly language-dependent, and the simplest embedding initialization works well across various experimental settings. Together, our work lays foundations on efficiently building language-specific LLMs by adapting existing LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
