Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user's stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley-Terry and characterize when bias in human annotators' preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.
Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Qwen Large Model Application Team, Alibaba(阿里巴巴大模型应用团队);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)
Cognitive models can reveal interpretable value trade-offs in language models
Kempner Institute for Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究所);Google DeepMind(谷歌DeepMind);Department of Psychology, Harvard University(哈佛大学心理学系)
Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE;Technology Innovation Institute, Abu Dhabi, UAE;Department of Robotics, Khalifa University, Abu Dhabi, UAE