面向中文 LLaMA 与 Alpaca 的高效且有效的文本编码
Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca
AI总结：
本文通过扩展 LLaMA 词表并加入中文预训练与指令微调，提出高效中文编码方法，显著提升其中文理解生成能力，并在 C-Eval 上取得有竞争力的结果。
摘要：
大型语言模型（LLMs），如 ChatGPT 和 GPT-4，已经极大地改变了自然语言处理研究，并展现出通向通用人工智能（AGI）的可喜进展。然而，训练和部署 LLMs 的高昂成本对透明、可及的学术研究构成了重大障碍。虽然社区已经开源了若干大型语言模型，例如 LLaMA，但这些模型主要关注英语语料，限制了它们在其他语言中的实用性。在本文中，我们提出了一种方法，以增强 LLaMA 理解和生成中文文本的能力及其遵循指令的能力。我们通过向 LLaMA 现有词表扩展额外的 20,000 个中文 token 来实现这一目标，从而提高其对中文的编码效率和语义理解能力。我们进一步结合使用中文数据的二次预训练，并使用中文指令数据集对模型进行微调，显著增强了模型理解和执行指令的能力。我们的实验结果表明，新提出的模型显著提升了原始 LLaMA 在理解和生成中文内容方面的熟练程度。此外，在 C-Eval 数据集上的结果在规模数倍于我们模型的模型中取得了具有竞争力的表现。我们已通过 GitHub 公开了我们的预训练模型、训练脚本和其他资源，以促进社区的开放研究。Chinese LLaMA 系列：\url{https://github.com/ymcui/Chinese-LLaMA-Alpaca} 和 Chinese Llama-2 系列：\url{https://github.com/ymcui/Chinese-LLaMA-Alpaca-2}
英文摘要：
Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca} and Chinese Llama-2 series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca-2}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
