隐式跨语言奖励用于高效多语言偏好对齐
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
AI总结：
本文提出隐式跨语言奖励方法，从英语DPO对齐模型中提取隐式奖励模型并标注跨语言偏好数据，再通过多语言DPO微调将偏好知识迁移到其他语言，在X-AlpacaEval上显著提升多语言对齐效果并减少对多语言偏好数据的需求。
摘要：
直接偏好优化（DPO）已成为使大语言模型（LLM）与人类偏好对齐的重要方法。尽管DPO在英语LLM对齐方面取得了显著进展，但多语言偏好对齐仍受到数据稀缺的制约。为解决这一问题，我们提出一种新方法，该方法通过隐式奖励从已良好对齐的英语模型中捕获学习到的偏好，并通过迭代训练将其迁移到其他语言。具体而言，我们从一个经英语DPO对齐的模型及其对应参考模型的logits中推导出隐式奖励模型。随后利用该奖励模型为跨语言指令遵循数据对标注偏好关系，使用英语指令评估多语言响应。标注后的数据随后用于多语言DPO微调，促进偏好知识从英语向其他语言迁移。对Llama3进行两轮迭代微调后，在X-AlpacaEval排行榜上所有训练语言的平均胜率（Win Rate）提升了12.72%，长度控制胜率（Length Control Win Rate）提升了5.97%。我们的研究结果表明，利用现有的英语对齐模型可以实现高效且有效的多语言偏好对齐，从而显著减少对大规模多语言偏好数据的需求。代码可在 https://github.com/ZNLP/Implicit-Cross-Lingual-Rewarding 获取。
英文摘要：
Direct Preference Optimization (DPO) has become a prominent method for aligning Large Language Models (LLMs) with human preferences. While DPO has enabled significant progress in aligning English LLMs, multilingual preference alignment is hampered by data scarcity. To address this, we propose a novel approach that $\textit{captures}$ learned preferences from well-aligned English models by implicit rewards and $\textit{transfers}$ them to other languages through iterative training. Specifically, we derive an implicit reward model from the logits of an English DPO-aligned model and its corresponding reference model. This reward model is then leveraged to annotate preference relations in cross-lingual instruction-following pairs, using English instructions to evaluate multilingual responses. The annotated data is subsequently used for multilingual DPO fine-tuning, facilitating preference knowledge transfer from English to other languages. Fine-tuning Llama3 for two iterations resulted in a 12.72% average improvement in Win Rate and a 5.97% increase in Length Control Win Rate across all training languages on the X-AlpacaEval leaderboard. Our findings demonstrate that leveraging existing English-aligned models can enable efficient and effective multilingual preference alignment, significantly reducing the need for extensive multilingual preference data. The code is available at https://github.com/ZNLP/Implicit-Cross-Lingual-Rewarding
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
