Aurora：通过指令微调激活Mixtral-8x7B稀疏混合专家模型的中文聊天能力
Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning
AI总结：
本文通过整合三个中文指令数据集对Mixtral-8x7B稀疏混合专家模型进行指令微调，构建了名为Aurora的模型，在C-Eval等基准上验证了其有效性与开创性。
摘要：
现有研究表明，通过利用机器生成的指令跟随数据对大型语言模型（LLMs）进行微调，能够使这些模型在不需人类编写指令的情况下，对全新任务展现出令人印象深刻的零样本能力。本文系统地研究、预处理并整合了三个中文指令跟随数据集，旨在增强Mixtral-8x7B稀疏混合专家模型的中文对话能力。通过在这个精心处理的数据集上进行指令微调，我们成功构建了名为“Aurora”的Mixtral-8x7B稀疏混合专家模型。为了评估Aurora的性能，我们使用了三个广泛认可的基准测试：C-Eval、MMLU和CMMLU。实证研究验证了应用于Mixtral-8x7B稀疏混合专家模型的指令微调的有效性。这项工作在稀疏混合专家模型上执行指令微调方面具有开创性，标志着在增强该模型架构能力方面的重大突破。我们的代码、数据和模型已在https://github.com/WangRongsheng/Aurora上公开发布。
英文摘要：
Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture. Our code, data and model are publicly available at https://github.com/WangRongsheng/Aurora
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
