arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26707cs.CLcs.CYcs.LG
收藏

价值漂移:在大语言模型训练后追踪价值对齐

Value Drifts: Tracing Value Alignment During LLM Post-Training

Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);ETH Zurich(苏黎世联邦理工学院);University of British Columbia(不列颠哥伦比亚大学);Vector Institute(向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)

Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, Siva Reddy

AI总结:

研究大语言模型训练后价值对齐问题,通过实验区分训练后算法和数据集影响,测量价值漂移,发现SFT阶段确立模型价值,后续偏好优化难重对齐,不同算法在偏好数据不变时也有不同结果,为相关选择提供见解。

摘要:

随着大语言模型在社会中发挥越来越重要的作用,它们面临着不仅要运用常识,还要与特定人类价值体系对齐的问题。因此,研究大语言模型与人类价值的对齐已成为关键研究领域。然而,先前工作大多聚焦于评估完全训练模型的对齐,忽视了模型学习表达人类价值的训练动态。本文研究了模型训练后价值对齐如何以及在哪个阶段出现。我们的分析区分了训练后算法和数据集的影响,测量了训练期间价值漂移的大小和时间。通过对不同大小的Llama-3和Qwen-3模型以及流行的监督微调(SFT)和偏好优化数据集及算法进行实验,发现SFT阶段通常确立模型价值,后续偏好优化很少重新对齐这些价值。此外,使用能控制价值操纵的合成偏好数据集,发现即使偏好数据不变,不同偏好优化算法也会导致不同价值对齐结果。我们的发现为训练后如何学习价值提供了可操作的见解,有助于指导数据整理以及偏好优化的模型和算法选择,以改善模型与人类价值的对齐。

英文摘要:

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑