Unimodal vision models are known to rely on spurious correlations, but it remains unclear to what extent Multimodal Large Language Models (MLLMs) exhibit similar biases despite language supervision. In this paper, we investigate spurious bias in MLLMs and introduce SpurLens, a pipeline that leverages GPT-4 and open-set object detectors to automatically identify spurious visual cues without human supervision. Our findings reveal that spurious correlations cause two major failure modes in MLLMs: (1) over-reliance on spurious cues for object recognition, where removing these cues reduces accuracy, and (2) object hallucination, where spurious cues amplify the hallucination by over 10x. We validate our findings in various MLLMs and datasets. Beyond diagnosing these failures, we explore potential mitigation strategies, such as prompt ensembling and reasoning-based prompting, and conduct ablation studies to examine the root causes of spurious bias in MLLMs. By exposing the persistence of spurious correlations, our study calls for more rigorous evaluation methods and mitigation strategies to enhance the reliability of MLLMs.
Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models
Department of Computer Science（计算机科学系）；School of Nursing（护理学院）；Department of Information Sciences and Technology（信息科学与技术系）；Department of Biomedical Informatics & Medical Education（生物医学信息学与医学教育系）
Token-Based Detection of Spurious Correlations in Vision Transformers
Center for Biosystems and Biotech Data Science（生物系统与生物技术数据科学中心）；Ghent University Global Campus（根特大学全球校区）；State University of New York Korea（纽约州立大学韩国分校）；Department of Mathematics（数学系）；Université de Liège（列日大学）；Department of Applied Mathematics, Computer Science and Statistics（应用数学、计算机科学与统计学系）；Ghent University（根特大学）；Department of Electronics and Information Systems（电子与信息系统系）
Causal Prompting for Implicit Sentiment Analysis with Large Language Models
School of Computing Technologies, RMIT University（RMIT大学计算机技术学院）；Institute of Innovation, Science and Sustainability, Federation University Australia（联邦大学澳大利亚创新、科学与可持续性研究所）；CSIRO’s Data61（CSIRO数据61）
Do Large Language Models Show Biases in Causal Learning?
UBA（布宜诺斯艾利斯大学）；Università degli Studi di Genova（热那亚大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；Universidad Nacional del Sur（南方国立大学）；IIIA-CSIC（西班牙国家研究委员会人工智能研究所）；Universidad Autónoma de Barcelona（巴塞罗那自治大学）；Arizona State University（亚利桑那州立大学）
Low-rank Orthogonal Subspace Intervention for Generalizable Face Forgery Detection
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（数字取证工程研究中心，教育部，南京信息科技大学）
