重新思考大语言模型的语言适配：Chinese Mixtral案例研究
Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral
AI总结：
该研究基于Mixtral-8x7B-v0.1，通过持续预训练与指令微调构建Chinese-Mixtral系列模型，在保持英语能力的同时提升中文性能，并分析词表扩展、初始化选择及专家作用。
摘要：
Mixtral是具有代表性的稀疏专家混合（SMoE）语言模型，因其独特的模型设计和优越性能而受到广泛关注。本文基于Mixtral-8x7B-v0.1，通过进一步预训练和指令微调，提出了中文能力得到增强的Chinese-Mixtral和Chinese-Mixtral-Instruct。实验结果表明，Chinese-Mixtral和Chinese-Mixtral-Instruct成功提升了中文理解与生成性能，同时保留了原有的英语能力。随后，本文通过实证结果与分析，讨论了对大语言模型进行语言适配时的若干关键问题，包括扩展语言专属词表的必要性以及初始化模型的选择（基础模型与指令模型）。本文还展示了各专家的可视化结果，以考察其在下游任务中的重要性。相关资源已通过https://github.com/ymcui/Chinese-Mixtral公开发布。
英文摘要：
Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks. Our resources are publicly available through \url{https://github.com/ymcui/Chinese-Mixtral}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
