视觉语言模型（VLMs）在多种多模态推理任务中表现出色，但难以区分细粒度视觉属性并推理解释因果关系。上下文学习（ICL）为VLMs适应新任务提供了前景，但其效果依赖于演示示例的选择。现有检索增强方法通常依赖被动相似性检索，倾向于选择相关但非因果的示例，放大虚假关联并限制模型鲁棒性。我们引入CIRCLES（Composed Image Retrieval for Causal Learning Example Selection），一种新颖的框架，通过针对性、属性引导的合成图像检索主动构建演示集。通过整合反事实样例，CIRCLES使VLMs能够隐式推理解释属性与结果之间的因果关系，超越表面相关性，促进更稳健和扎根的推理。在四个多样化数据集上的全面实验表明，CIRCLES在多种架构上均优于现有方法，尤其在小规模模型和信息稀缺场景下表现显著。此外，CIRCLES检索更多多样且因果信息丰富的示例，提供定性洞察，揭示模型如何利用上下文演示提升推理能力。我们的代码可在https://github.com/gzxiong/CIRCLES获取。
英文摘要：
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal reasoning tasks, but they often struggle to disentangle fine-grained visual attributes and reason about underlying causal relationships. In-context learning (ICL) offers a promising avenue for VLMs to adapt to new tasks, but its effectiveness critically depends on the selection of demonstration examples. Existing retrieval-augmented approaches typically rely on passive similarity-based retrieval, which tends to select correlated but non-causal examples, amplifying spurious associations and limiting model robustness. We introduce CIRCLES (Composed Image Retrieval for Causal Learning Example Selection), a novel framework that actively constructs demonstration sets by retrieving counterfactual-style examples through targeted, attribute-guided composed image retrieval. By incorporating counterfactual-style examples, CIRCLES enables VLMs to implicitly reason about the causal relations between attributes and outcomes, moving beyond superficial correlations and fostering more robust and grounded reasoning. Comprehensive experiments on four diverse datasets demonstrate that CIRCLES consistently outperforms existing methods across multiple architectures, especially on small-scale models, with pronounced gains under information scarcity. Furthermore, CIRCLES retrieves more diverse and causally informative examples, providing qualitative insights into how models leverage in-context demonstrations for improved reasoning. Our code is available at https://github.com/gzxiong/CIRCLES.
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
The University of Manchester（曼彻斯特大学）；The University of Melbourne（墨尔本大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Edinburgh（爱丁堡大学）；University of Southern California（南加州大学）；Fudan University（复旦大学）；University of Newcastle（纽卡斯尔大学）
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
School of Intelligent Science and Technology, Nanjing University（南京大学智能科学与技术学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学医疗数据科学国家研究院）；School of Biomedical Engineering, Nanjing University（南京大学生物医学工程学院）
Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）；School of Information Science and Engineering, Ningbo University（宁波大学信息科学与工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
The Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）；Columbia University（哥伦比亚大学）；Nanyang Technological University（南洋理工大学）
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Hikvision Research Institute（海康威视研究院）
Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ide…
RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；MThreads, Inc.（MThreads公司）；Nanyang Technological University（南洋理工大学）
A$^2$Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering
Macau University of Science and Technology（澳门科技大学）；Wuhan University of Technology（武汉理工大学）；Nanjing Institute of Technology（南京工程学院）；National Tsing Hua University（国立清华大学）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Beihang University（北京航空航天大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
School of Computer Science, Peking University（北京大学计算机科学学院）；MOE Key Laboratory of Computational Linguistics, Peking University（北京大学教育部计算语言学重点实验室）；National Key Laboratory for Multimedia Information Processing, Peking University（北京大学国家多媒体信息处理重点实验室）
Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification
School of Computing and Information Technology, Great Bay University（大湾大学计算机与信息学院）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞智能与信息科技重点实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science, Nankai University（南开大学计算机学院）；School of mathematics, Sun Yat-sen University（中山大学数学学院）
Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；College of Software Engineering, Southeast University（东南大学软件学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
University of California, San Diego（加利福尼亚大学圣迭戈分校）；SRI International（SRI国际公司）
In the domain of Visual Question Answering (VQA), studies have shown improvement in users' mental model of the VQA system when they are exposed to examples of how these systems answer certain Image-Question (IQ) pairs. In this work, we show that showing controlled counterfactual image-question examples are more effective at improving the mental model of users as compared to simply showing random examples. We compare a generative approach and a retrieval-based approach to show counterfactual examples. We use recent advances in generative adversarial networks (GANs) to generate counterfactual im…
DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
Department of Automation, Tsinghua University, Beijing, China（清华大学自动化系）；Zhongguancun Academy, Beijing, China（中关村学院）；School of Software, Xinjiang University, Urumqi, China（新疆大学软件学院）；College of Materials Science and Engineering, Fuzhou University, Fuzhou, China（福州大学材料科学与工程学院）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；Beijing Qianjue Technology Co., Ltd., Beijing, China（北京千 jue 技术有限公司）
Arizona State University（亚利桑那州立大学）；University of California, Irvine（加州大学欧文分校）
Computer Vision applications often require a textual grounding module with precision, interpretability, and resilience to counterfactual inputs/queries. To achieve high grounding precision, current textual grounding methods heavily rely on large-scale training data with manual annotations at the pixel level. Such annotations are expensive to obtain and thus severely narrow the model's scope of real-world applications. Moreover, most of these methods sacrifice interpretability, generalizability, and they neglect the importance of being resilient to counterfactual inputs. To address these issues…
Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis
Texas A&M University（德克萨斯A&M大学）；University of Minnesota（明尼苏达大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Abaka AI（Abaka人工智能）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
Division of Computational Health Sciences, Department of Surgery University of Minnesota（大学医学计算科学系，外科系）；Institute for Health Informatics University of Minnesota（健康信息学研究所，明尼苏达大学）；Department of Electrical and Computer Engineering University of Minnesota（电气与计算机工程系，明尼苏达大学）
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
One of the primary challenges limiting the applicability of deep learning is its susceptibility to learning spurious correlations rather than the underlying mechanisms of the task of interest. The resulting failure to generalise cannot be addressed by simply using more data from the same distribution. We propose an auxiliary training objective that improves the generalization capabilities of neural networks by leveraging an overlooked supervisory signal found in existing datasets. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, …
Explanation of AI, as well as fairness of algorithms' decisions and the transparency of the decision model, are becoming more and more important. And it is crucial to design effective and human-friendly techniques when opening the black-box model. Counterfactual conforms to the human way of thinking and provides a human-friendly explanation, and its corresponding explanation algorithm refers to a strategic alternation of a given data point so that its model output is "counter-facted", i.e. the prediction is reverted. In this paper, we adapt counterfactual explanation over fine-grained image cl…
Attention mechanism has demonstrated great potential in fine-grained visual recognition tasks. In this paper, we present a counterfactual attention learning method to learn more effective attention based on causal inference. Unlike most existing methods that learn visual attention based on conventional likelihood, we propose to learn the attention with counterfactual causality, which provides a tool to measure the attention quality and a powerful supervisory signal to guide the learning process. Specifically, we analyze the effect of the learned visual attention on network prediction through c…
Georgia Institute of Technology（佐治亚理工学院）；Siemens Corporation（西门子公司）
In this work, we develop a technique to produce counterfactual visual explanations. Given a 'query' image $I$ for which a vision system predicts class $c$, a counterfactual visual explanation identifies how $I$ could change such that the system would output a different specified class $c'$. To do this, we select a 'distractor' image $I'$ that the system predicts as class $c'$ and identify spatial regions in $I$ and $I'$ such that replacing the identified region in $I$ with the identified region in $I'$ would push the system towards classifying $I$ as $c'$. We apply our approach to multiple ima…
Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；CentraleSupélec（巴黎中央理工-高等电力学院）
Visual counterfactual explanations identify modifications to an image that would change the prediction of a classifier. We propose a set of techniques based on generative models (VAE) and a classifier ensemble directly trained in the latent space, which all together, improve the quality of the gradient required to compute visual counterfactuals. These improvements lead to a novel classification model, Clarity, which produces realistic counterfactual explanations over all images. We also present several experiments that give insights on why these techniques lead to better quality results than t…
A visual counterfactual explanation replaces image regions in a query image with regions from a distractor image such that the system's decision on the transformed image changes to the distractor class. In this work, we present a novel framework for computing visual counterfactual explanations based on two key ideas. First, we enforce that the replaced and replacer regions contain the same semantic part, resulting in more semantically consistent explanations. Second, we use multiple distractor images in a computationally efficient way and obtain more discriminative explanations with fewer regi…
Existing studies on disease diagnostic models focus either on diagnostic model learning for performance improvement or on the visual explanation of a trained diagnostic model. We propose a novel learn-explain-reinforce (LEAR) framework that unifies diagnostic model learning, visual explanation generation (explanation unit), and trained diagnostic model reinforcement (reinforcement unit) guided by the visual explanation. For the visual explanation, we generate a counterfactual map that transforms an input sample to be identified as an intended target label. For example, a counterfactual map can…
