迈向更优指令遵循的中文语言模型：探究训练数据与评估的影响
Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation
AI总结：
本文探究训练数据数量、质量和语言分布对指令遵循模型性能的影响，并通过扩展LLaMA词表及在34亿中文词上二次预训练，构建并公开了性能更优的中文对话模型。
摘要：
近期，大量公共努力致力于开发具有与ChatGPT类似能力的低成本模型，从而促进了开源对话模型的发展。然而，对这些模型性能的全面且深入评估仍然匮乏。在本研究中，我们考察了训练数据因素（包括数量、质量和语言分布）对模型性能的影响。我们的分析基于多个公开可用的高质量指令数据集，以及我们自建的中文多轮对话数据。我们使用包含1,000个样本、涵盖九种真实世界场景的评估集对各种模型进行评估。我们的目标是以定量分析补充人工评估，为开源聊天模型的持续发展提供有价值的见解。此外，为了提升中文领域模型的性能以及训练和推理效率，我们扩展了LLaMA——在开源模型中性能最接近GPT-3等专有语言模型的模型——的词表，并在34亿个中文词上进行了二次预训练。我们已将模型、数据以及代码全部公开。
英文摘要：
Recently, significant public efforts have been directed towards developing low-cost models with capabilities akin to ChatGPT, thereby fostering the growth of open-source conversational models. However, there remains a scarcity of comprehensive and in-depth evaluations of these models' performance. In this study, we examine the influence of training data factors, including quantity, quality, and linguistic distribution, on model performance. Our analysis is grounded in several publicly accessible, high-quality instruction datasets, as well as our own Chinese multi-turn conversations. We assess various models using a evaluation set of 1,000 samples, encompassing nine real-world scenarios. Our goal is to supplement manual evaluations with quantitative analyses, offering valuable insights for the continued advancement of open-source chat models. Furthermore, to enhance the performance and training and inference efficiency of models in the Chinese domain, we extend the vocabulary of LLaMA - the model with the closest open-source performance to proprietary language models like GPT-3 - and conduct secondary pre-training on 3.4B Chinese words. We make our model, data, as well as code publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
