arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.03419cs.CLcs.AI
收藏

否定负样本:通过分布性不偏好优化实现与人类负样本的对齐

Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization

Fudan University(复旦大学);Microsoft Research Asia(微软亚洲研究院);HKUST(香港科技大学);CUHK(香港中文大学)

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, Ning Gu

AI总结:

本文提出分布性不偏好优化(D$^2$O),仅利用人类标注的负样本,通过最大化不偏好回复与自生成非负回复的差异实现对齐,在保持有用性的同时减少有害性,并优于现有基线。

摘要:

大型语言模型(LLMs)已经彻底改变了人工智能的角色,但也带来了潜在的社会风险。为了引导LLMs符合人类偏好,对齐技术被引入并受到越来越多的关注。然而,现有方法严重依赖高质量的正负训练对,受到与负样本几乎无法区分的噪声正样本的困扰。鉴于近期LLMs在生成有用回复方面的熟练能力,本工作转向一个新的研究问题:我们能否仅使用人类标注的负样本实现对齐,在保持有用性的同时减少有害性?为此,我们提出了分布性不偏好优化(D$^2$O),它最大化了不偏好回复与生成的非负回复之间的差异。通过这种方式,D$^2$O有效地避开了有害信息,而无需纳入噪声正样本,同时利用自生成回复作为锚点避免了坍缩。我们证明D$^2$O可以被视为学习一个反映人类对负样本不偏好的分布性偏好模型,这在理论上是实例级DPO的上界。大量实验表明,我们的方法实现了可比的生成质量,并在产生更少有害且信息更丰富的回复方面超越了最新的强基线,同时具有更好的训练稳定性和更快的收敛速度。

英文摘要:

Large language models (LLMs) have revolutionized the role of AI, yet pose potential social risks. To steer LLMs towards human preference, alignment technologies have been introduced and gained increasing attention. Nevertheless, existing methods heavily rely on high-quality positive-negative training pairs, suffering from noisy positive responses that are barely distinguishable from negative ones. Given recent LLMs' proficiency in generating helpful responses, this work pivots towards a new research question: can we achieve alignment using solely human-annotated negative samples, preserving helpfulness while reducing harmfulness? For this purpose, we propose Distributional Dispreference Optimization (D$^2$O), which maximizes the discrepancy between dispreferred responses and the generated non-negative ones. In this way, D$^2$O effectively eschews harmful information without incorporating noisy positive samples, while avoiding collapse using self-generated responses as anchors. We demonstrate that D$^2$O can be regarded as learning a distributional preference model reflecting human dispreference against negative responses, which is theoretically an upper bound of the instance-level DPO. Extensive experiments manifest that our method achieves comparable generation quality and surpasses the latest strong baselines in producing less harmful and more informative responses with better training stability and faster convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑