Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences(处理器国家重点实验室,中国科学院计算技术研究所);Department of Computer Science, National University of Singapore(计算机科学系,新加坡国立大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences(软件智能研究中心,中国科学院软件研究所)
Large Vision-Language Models (LVLMs) have shown significant capability in vision-language understanding. However, one critical issue that persists in these models is sycophancy, where models are unduly influenced by leading or deceptive prompts, resulting in biased outputs and hallucinations. Despite the rapid development of LVLMs, evaluating and mitigating sycophancy remains largely under-explored. In this work, we fill this gap by systematically analyzing sycophancy across multiple vision-language benchmarks and propose an inference-time mitigation framework. We curate leading queries and quantify the susceptibility of state-of-the-art LVLMs to prompt-induced bias, revealing consistent performance degradation and instability across models and tasks. Our analysis further uncovers model-specific behavioral traits, such as sentiment sensitivity and prediction polarity shifts under sycophancy. To mitigate these issues, we propose a training-free, model-agnostic framework that operates entirely at inference time. Our approach first employs a query neutralizer, leveraging an language model to suppress implicit sycophantic bias in user queries. We then introduce a sycophancy-aware contrastive decoding mechanism that dynamically recalibrates token-level output distributions by contrasting responses to neutralized and leading queries. Finally, an adaptive logits refinement module further modifies the contrasted logits by integrating both a adaptive plausibility filter and query sentiment scaler, ensuring coherent and robust generation. Extensive experiments demonstrate that this framework effectively mitigates sycophancy across all evaluated models, while maintaining performance on neutral prompts. Our results suggest that sycophancy in LVLMs is a general and urgent challenge, and that inference-time strategies offer a promising path toward trustworthy multimodal reasoning.
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs
Provable Responsible AI and Data Analytics (PRADA) Lab(可证负责任人工智能与数据 analytics 实验室);King Abdullah University of Science and Technology(国王阿卜杜勒阿齐兹科学与技术大学);HKUST(香港科技大学);MBZUAI(穆罕默德·本·拉希德人工智能研究所);University of Science and Technology of China(中国科学技术大学);Kyungpook National University(庆尚国立大学)
University of Dhaka(达卡大学);Daffodil International University(达福迪国际大学);Islamic University of Technology(伊斯兰技术大学);University of Central Florida(佛罗里达中央大学);King Fahad University of Petroleum and Minerals(国王法赫德石油与矿物大学);SDAIA - KFUPM Joint research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心)
What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct
Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University of Toronto(多伦多大学);Stanford University(斯坦福大学);University of Cincinnati(克里夫兰医学中心大学);New York University(纽约大学)
Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories
University of California, Davis(加州大学戴维斯分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学);Meta AI;Adobe Research(Adobe研究)
MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation
School of Computer Science & Tech, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of AI and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
Department of Mathematics, Purdue University, West Lafayette IN, USA(普渡大学数学系);School of Mechanical Engineering, Purdue University, West Lafayette IN, USA(普渡大学机械工程学院);Massachusetts Institute of Technology, Cambridge MA, USA(麻省理工学院)
Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models
Aalto University, Espoo, Finland(艾尔沃大学,芬兰 Espoo);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China(深圳先进技术研究院,中国科学院,深圳)
Zero-Shot Vision-and-Language Navigation with Collision Mitigation in Continuous Environment
Seoul National University(首尔大学);AI Institute of Seoul National University (AIIS)(首尔大学人工智能研究院);Dept. of Computer Science and Engineering, Seoul National University(首尔大学计算机科学与工程系)
BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
FPT Software AI Center(FPT软件AI中心);Aalborg University(奥尔堡大学);Pioneer Centre for AI(先锋人工智能中心);University of Liverpool(利物浦大学);AICV Lab, University of Arkansas(AICV实验室,阿肯色大学)