Learning 1D Causal Visual Representation with De-focus Attention Networks
SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Chenxin Tao, Xizhou Zhu, Shiqian Su, Lewei Lu, Changyao Tian, Xuan Luo, Gao Huang, Hongsheng Li, Yu Qiao, Jie Zhou, Jifeng Dai
Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code is released at https://github.com/OpenGVLab/De-focus-Attention-Networks.
Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）
CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）
Structure Causal Models and LLMs Integration in Medical Visual Question Answering
School of Microelectronics, Tianjin University（天津大学微电子学院）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality
Machine Intelligence Group, Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad Campus（机器智能组，计算机科学与信息系统系，比拉理工学院，帕利尼，海得拉巴校区）
Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
School of Informatics, Xiamen University, China（厦门大学信息学院）；Xiamen Unisound Intelligence Technology Co., Ltd（厦门Unisound智能科技有限公司）；Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室（厦门大学），中华人民共和国文化和旅游部，中国）
MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understanding
Department of Computer Science, Chongqing University, China（重庆大学计算机科学系）；National Elite Institute of Engineering, Chongqing University, China（重庆大学工程精英研究院）；College of Computer Science and Technology, National University of Deffense Technology, China（国防科技大学计算机科学与技术学院）
Towards Single-Source Domain Generalized Object Detection via Causal Visual Prompts
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（multispectral information intelligent processing technology national key laboratory, school of artificial intelligence and automation, huazhong university of science and technology）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（intelligent education technology and application zhejiang key laboratory, zhejiang normal university）；Northwest Polytechnical University；Nankai University
Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planning
Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院（BIGAI））；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；University of California, Los Angeles (UCLA)（加利福尼亚大学洛杉矶分校）
Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Huazhong University of Science and Technology（华中科技大学）；National Anti-Counterfeit Engineering Research Center（国家防伪工程技术研究中心）；Nanjing University of Science and Technology（南京理工大学）
Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning For Visual Story Synthesis
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Shanghai Jiao Tong University（上海交通大学）；School of Computer Science and Engineering, Sanjiang University（三江大学计算机科学与工程学院）
CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Singapore Polytechnic（新加坡理工学院）
CIGPose: Causal Intervention Graph Neural Network for Whole-Body Pose Estimation
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Xidian University（西安电子科技大学）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络安全学院）
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
