Value Drifts: Tracing Value Alignment During LLM Post-Training
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);ETH Zurich(苏黎世联邦理工学院);University of British Columbia(不列颠哥伦比亚大学);Vector Institute(向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.
DART-ing Through the Drift: Dynamic Tracing of Knowledge Neurons for Adaptive Inference-Time Pruning
Department of Electrical and Computer Engineering, National University of Singapore, Singapore(电子与计算机工程系,新加坡国立大学);Department of Computer Science, Birla Institute of Technology and Science, Pilani, India(计算机科学系,比拉理工学院与科学学院,印度)
Path Drift in Large Reasoning Models:How First-Person Commitments Override Safety
The Second Affiliated Hospital, Guangdong Provincial Key Laboratory of Allergy and Clinical Immunology, Guangzhou Medical University(广东省过敏与临床免疫学重点实验室,广州医科大学第二附属医院);NLP 2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学系NLP2CT实验室)
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models
Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH, USA(电气与计算机工程系,俄亥俄州立大学,哥伦布,OH,USA);Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA(计算机科学与工程系,俄亥俄州立大学,哥伦布,OH,USA)
Tracing Stereotypes from Representation to Output in Multilingual LLMs
Saarland University(萨尔大学);Queen’s University Belfast(贝尔法斯特女王大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));University of Technology Nuremberg(纽伦堡工业大学)
Metronome: tracing variation in poetic meters via local sequence alignment
Institute of Polish Language, Polish Academy of Sciences (IJP PAN)(波兰科学院波兰语言研究所);Tilburg University(蒂尔堡大学);Institute of Czech Literature, Czech Academy of Sciences(捷克科学院捷克文学研究所)
How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders
MBZUAI(穆萨大学人工智能研究所);SOKENDAI(神户大学);NINJAL(日本国家研究所);Tohoku University(东北大学);RIKEN(日本研究机构);NII LLMC(日本信息处理学会大语言模型中心);University of Tokyo(东京大学);Nagoya Institute of Technology(名古屋技术大学)
SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement
MARS 4.0 Fellowship, Cambridge AI Safety Hub(CAISH), University of Cambridge(剑桥大学);AWS Generative AI Innovation Center, Amazon Web Services, USA(亚马逊网络服务);Google, USA(谷歌);Stanford University(斯坦福大学);Northeastern University, Seattle, WA, USA(东北大学)
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州));Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);Renmin University of China(中国人民大学);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学)