RLHF能说多种语言：解锁大语言模型的多语言偏好优化
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
AI总结：
本文通过提出可扩展的多语言反馈数据生成方法并利用跨语言迁移与更大规模偏好训练数据，将LLM偏好优化对齐扩展到23种语言，并在多语言基准上超越Aya 23 8B等现有模型。
摘要：
偏好优化技术已成为训练最先进大语言模型（LLMs）的标准最终阶段。然而，尽管被广泛采用，迄今为止绝大多数工作都集中在英语和中文等“一等公民”语言上。这只覆盖了世界上的一小部分语言，同时也使得当前最先进研究的哪些方面能够迁移到多语言环境变得不明确。在这项工作中，我们进行了一项详尽的研究，以在多语言LLM对齐方面达到新的最先进水平。我们引入了一种新颖的、可扩展的方法来生成高质量的多语言反馈数据，以平衡数据覆盖范围。我们确立了跨语言迁移和增加数据集规模在偏好训练中的益处。我们的偏好训练模型在对抗Aya 23 8B（其参数级别中当前最先进的多语言LLM）时取得了54.4%的胜率，并且在对抗广泛使用的模型（如Gemma-1.1-7B-it、Llama-3-8B-Instruct、Mistral-7B-Instruct-v0.3）时取得了69.5%或更高的胜率。作为我们研究的成果，我们将对齐技术的前沿扩展到23种语言，覆盖了世界一半的人口。
英文摘要：
Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state-of-the-art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma-1.1-7B-it, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3. As a result of our study, we expand the frontier of alignment techniques to 23 languages covering half of the world's population.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
