多语言对齐中奖励模型的跨语言迁移
Cross-lingual Transfer of Reward Models in Multilingual Alignment
AI总结：
本研究探讨了奖励模型从英语到多语言的跨语言迁移，发现英语奖励模型在Multilingual RewardBench上平均提升3~4%，并通过多语言对齐增强了指令跟随能力。
摘要：
基于人类反馈的强化学习（RLHF）已被证明在很大程度上受益于精确的奖励模型（RMs）。然而，近期关于奖励建模的研究偏向于英语，限制了RLHF在多语言对齐中的适用性。在本工作中，我们研究了在不同语言（主要从英语）训练的奖励模型的跨语言迁移。我们的实验结果表明，英语奖励模型具有强大的跨语言迁移能力，在Multilingual RewardBench上平均提升3~4%，超过了目标语言奖励模型。此外，我们通过表示偏移分析了奖励模型的跨语言迁移。最后，我们进行了多语言对齐，以示例说明奖励模型中的跨语言迁移如何传播到增强的多语言指令跟随能力，并对现成的奖励模型进行了广泛分析。我们发布了代码、模型和数据。
英文摘要：
Reinforcement learning with human feedback (RLHF) is shown to largely benefit from precise reward models (RMs). However, recent studies in reward modeling schemes are skewed towards English, limiting the applicability of RLHF in multilingual alignments. In this work, we investigate the cross-lingual transfer of RMs trained in diverse languages, primarily from English. Our experimental results demonstrate the strong cross-lingual transfer of English RMs, exceeding target language RMs by 3~4% average increase in Multilingual RewardBench. Furthermore, we analyze the cross-lingual transfer of RMs through the representation shifts. Finally, we perform multilingual alignment to exemplify how cross-lingual transfer in RM propagates to enhanced multilingual instruction-following capability, along with extensive analyses on off-the-shelf RMs. We release the code, model, and data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
