Okapi：基于人类反馈强化学习的多语言指令微调大型语言模型
Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
AI总结：
针对现有开源大模型多语言指令微调仅依赖SFT的局限，本文提出首个基于RLHF的多语言指令微调系统Okapi，提供26种语言的数据与基准，并验证了RLHF优于SFT。
摘要：
大型语言模型（LLMs）发展的一项关键技术是指令微调，它有助于使模型的响应与人类期望保持一致，从而实现令人印象深刻的学习能力。指令微调的两种主要方法是监督微调（SFT）和基于人类反馈的强化学习（RLHF），目前这两种方法被应用于生产最佳的商业LLM（例如，ChatGPT）。为了提高LLM在研发工作中的可及性，最近也推出了各种指令微调的开源LLM，例如Alpaca、Vicuna等。然而，现有的开源LLM仅针对英语和少数流行语言进行了指令微调，从而阻碍了它们对世界上许多其他语言的影响力和可及性。在最近探索多语言LLM指令微调的少数工作中，SFT被用作多语言LLM指令微调的唯一方法。这给基于RLHF的微调LLM在多种语言中留下了显著空白，并引发了关于RLHF如何提升多语言指令微调性能的重要问题。为了克服这个问题，我们提出了Okapi，这是第一个基于RLHF进行多语言指令微调LLM的系统。Okapi引入了26种不同语言的指令和响应排序数据，以促进未来多语言LLM研究的实验和开发。我们还提供了基准数据集，以实现多语言生成式LLM的评估。我们的实验证明了，对于不同的基础模型和数据集，RLHF在多语言指令方面优于SFT的优势。我们的框架和资源已在 https://github.com/nlp-uoregon/Okapi 发布。
英文摘要：
A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at https://github.com/nlp-uoregon/Okapi.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
