arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.18101cs.CL
收藏

多样化偏好优化

Diverse Preference Optimization

Meta

Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, Ilia Kulikov

AI总结:

针对语言模型后训练导致生成多样性下降的问题,提出多样化偏好优化(DivPO)方法,通过基于响应池多样性的偏好选择,在保持质量的同时显著提升生成多样性,并在人物属性、故事和指令遵循任务上验证了效果。

摘要:

语言模型的后训练,无论是通过强化学习、偏好优化还是监督微调,往往会使输出概率分布变得尖锐,并降低生成响应的多样性。这对于需要多样化响应的创造性生成任务来说尤其是一个问题。在这项工作中,我们引入了多样化偏好优化(DivPO),这是一种优化方法,它学习生成比标准流程更加多样化的响应,同时保持生成质量。在DivPO中,偏好对的选取首先考虑一个响应池及其多样性度量,选择更稀有但高质量的样本作为被选中的示例,而更常见但低质量的样本作为被拒绝的示例。DivPO使得人物属性多样性提高了45.6%,故事多样性提高了74.6%,同时保持了与标准基线相似的胜率。在一般指令遵循方面,与DPO相比,DivPO使多样性提高了46.2%,胜率提高了2.4%。

英文摘要:

Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and a 74.6% increase in story diversity, while maintaining similar win rates as standard baselines. On general instruction following, DivPO results in a 46.2% increase in diversity, and a 2.4% winrate improvement compared to DPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑