Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.
What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct
Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University of Toronto(多伦多大学);Stanford University(斯坦福大学);University of Cincinnati(克里夫兰医学中心大学);New York University(纽约大学)
Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs
Massachusetts Institute of Technology(麻省理工学院);Johns Hopkins University(约翰霍普金斯大学);University of California, Berkeley(加州大学伯克利分校);Toronto General Hospital, University Health Network(多伦多综合医院,大学健康网络);McGill University(麦吉尔大学);University of Toronto(多伦多大学);Independent Researcher(独立研究者);Rutgers University(罗格斯大学);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard T.H. Chan School of Public Health(哈佛大学陈曾熙公共卫生学院)
When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas
ETH Zürich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);Carnegie Mellon University(卡内基梅隆大学);University of Michigan(密歇根大学);Max Planck Institute for Intelligent Systems, Tübingen(图宾根人工智能研究所);University of Toronto(多伦多大学);Vector Institute(向量研究所)
An Infectious Disease Spread Simulation Based on Large Language Model Decision Making
Computer Science and Engineering Faculty of Engineering(计算机科学与工程系);The University of New South Wales(新南威尔士大学);Department of Computer Science(计算机科学系);Emory University(埃默里大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));George Mason University(乔治·马歇尔大学);The Kirby Institute Faculty of Medicine & Health(Kirby研究所医学院与健康学院);Arizona State University(亚利桑那州立大学)