We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the Q-K-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT~\cite{tan2019lxmert}, which uses fewer data and less computational power, comparable to the heavier UNITER~\cite{chen2020uniter}. Code is published in \url{https://github.com/yangxuntu/catt}.
Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
Center of Statistical Research, School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu, China.（统计研究中心，统计与数据科学学院，西南财经大学，成都，中国）；Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore, Singapore（生物医学工程系，设计与工程学院，新加坡国立大学，新加坡）
Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection
Beijing University of Posts and Telecommunications（北京邮电大学）；Huawei Technologies Ltd.（华为技术有限公司）；Peking University（北京大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Deep reinforcement learning (RL) agents are becoming increasingly proficient in a range of complex control tasks. However, the agent's behavior is usually difficult to interpret due to the introduction of black-box function, making it difficult to acquire the trust of users. Although there have been some interesting interpretation methods for vision-based RL, most of them cannot uncover temporal causal information, raising questions about their reliability. To address this problem, we present a temporal-spatial causal interpretation (TSCI) model to understand the agent's long-term behavior, wh…
Interpreting Low-level Vision Models with Causal Effect Maps
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）
Attention module does not always help deep models learn causal features that are robust in any confounding context, e.g., a foreground object feature is invariant to different backgrounds. This is because the confounders trick the attention to capture spurious correlations that benefit the prediction when the training and testing data are IID (identical & independent distribution); while harm the prediction when the data are OOD (out-of-distribution). The sole fundamental solution to learn causal attention is by causal intervention, which requires additional annotations of the confounders, e.g…
Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigation
School of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Tongji Artificial Intelligence (Suzhou) Research Institute（同济大学苏州人工智能研究院）；Frontiers Science Center for Intelligent Autonomous Systems（智能自主系统前沿科学中心）；Rutgers University（罗格斯大学）
DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality
Machine Intelligence Group, Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad Campus（机器智能组，计算机科学与信息系统系，比拉理工学院，帕利尼，海得拉巴校区）
Statistical machine learning algorithms have achieved state-of-the-art results on benchmark datasets, outperforming humans in many tasks. However, the out-of-distribution data and confounder, which have an unpredictable causal relationship, significantly degrade the performance of the existing models. Causal Representation Learning (CRL) has recently been a promising direction to address the causal relationship problem in vision understanding. This survey presents recent advances in CRL in vision. Firstly, we introduce the basic concept of causal inference. Secondly, we analyze the CRL theoret…
Information Bottleneck-based Causal Attention for Multi-label Medical Image Recognition
Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR)（联合山东大学-东北大学人工智能研究中心（C-FAIR））；Shandong University（山东大学）；School of Information Science and Engineering（信息科学与工程学院）；Child Care Hospital Affiliated to Shandong First Medical University（山东第一医科大学附属儿童医院）；Case Western Reserve University（凯斯西储大学）
Learning 1D Causal Visual Representation with De-focus Attention Networks
SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
School of Computer Science and Technology, China University of Mining and Technology（中国矿业大学计算机科学与技术学院）；Key Laboratory of Image Processing and Intelligent Control (Huazhong University of Science and Technology), Ministry of Education（华中科技大学图像信息处理与智能控制教育部重点实验室）；Mine Digitization Engineering Research Center of the Ministry of Education（矿山数字化教育部工程研究中心）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Recently, a considerable number of studies in computer vision involves deep neural architectures called vision transformers. Visual processing in these models incorporates computational models that are claimed to implement attention mechanisms. Despite an increasing body of work that attempts to understand the role of attention mechanisms in vision transformers, their effect is largely unknown. Here, we asked if the attention mechanisms in vision transformers exhibit similar effects as those known in human visual attention. To answer this question, we revisited the attention formulation in the…
