arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03459cs.LG
收藏

DPO能否学习多样化的人类价值观?一个理论缩放定律

Can DPO Learn Diverse Human Values? A Theoretical Scaling Law

Department of Computer Sciences University of Wisconsin-Madison(计算机科学系威斯康星大学麦迪逊分校)

Shawn Im, Sharon Li

AI总结:

本文提出理论框架,分析DPO训练下模型泛化能力随价值多样性与样本数量的缩放规律,并通过实验验证了学习广泛价值观的挑战。

摘要:

大型语言模型(LLMs)已展现出卓越的能力,但往往难以与人类偏好对齐,导致产生有害或不受欢迎的输出。偏好学习通过训练模型基于人类反馈区分偏好响应与非偏好响应,已成为确保LLMs与人类价值观对齐的关键组成部分。确保LLMs为所有人对齐的一个重要方面是考虑多样化的价值观集合。本文引入了一个新的理论框架,用于分析在直接偏好优化(DPO)训练的模型中,泛化能力如何随价值多样性和样本数量缩放。我们的框架严格评估了模型在有限梯度步数后的泛化程度,反映了现实世界LLM训练实践。通过分析每个样本相关的奖励边际及其在训练过程中的轨迹,我们提供了泛化误差的界限,展示了有效学习广泛概念或价值观的挑战。这些见解在当代LLMs上得到了实证验证,凸显了我们理论的实际相关性。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMs align with human values. An essential part of ensuring that LLMs are aligned for all people is accounting for a diverse set of values. This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity in models trained with direct preference optimization. Our framework rigorously assesses how well models generalize after a finite number of gradient steps, reflecting real-world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, we provide a bound on the generalization error that demonstrates the challenges of effectively learning a wide set of concepts or values. These insights are empirically validated on contemporary LLMs, underscoring the practical relevance of our theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑