CroCo: 基于自生成结果的跨语言对比偏好调优
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
AI总结：
本文提出CroCo方法，利用英语偏好训练的奖励模型对多语言自生成结果进行对比偏好调优，无需语言特定偏好标注，在14种高低资源语言上提升模型性能，并避免灾难性遗忘。
摘要：
先前工作证实，通过奖励分数设置的大语言模型自生成结果之间的受控对比性，可以改善英语中的下游偏好调优。我们将此方法扩展到多种语言，并在总共14种高资源和低资源语言上，对两个模型在一系列多样化任务上进行评估。我们的核心发现是，跨语言对比偏好调优（CroCo）无需语言特定的偏好标注即可迁移。基于英语偏好（在多语言基础模型之上）训练的奖励模型，在大多数语言中产生了有用的语言内排名，并且在单语或多语设置中进行配对，在大多数设置上改进了每个模型，同时防止了监督微调的灾难性遗忘。我们观察到，这些增益需要基于策略的数据。非策略响应减少了收益，而在线偏好优化未能优于离线变体。具体来说，在结构化任务上，我们的方法在EuroLLM-9B的6/7种语言和Aya-3B的4/7种设置中匹配或超过了基础模型。在开放式生成中，两个调优模型在11种评估语言中均优于各自的基础模型。总体而言，我们展示了多语言偏好调优的有前景的方向。
英文摘要：
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, evaluated by two judges, both tuned models win 28/30 times against their respective base across 15 evaluated languages (high and low-resource). Overall, we show promising directions for multilingual preference tuning using self-generations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
