Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Peking University（北京大学）；Tencent Inc（腾讯公司）
Models for Visual Question Answering (VQA) often rely on the spurious correlations, i.e., the language priors, that appear in the biased samples of training set, which make them brittle against the out-of-distribution (OOD) test data. Recent methods have achieved promising progress in overcoming this problem by reducing the impact of biased samples on model training. However, these models reveal a trade-off that the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data (which is dominated by the biased samples). Therefore, we propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples and explore several strategies to use the constructed positive samples for training. Instead of undermining the importance of biased samples in model training, our approach precisely exploits the biased samples for unbiased information that contributes to reasoning. The proposed method is compatible with various VQA backbones. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.
Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering
School of Information Science and Technology, Hangzhou Normal University（杭州师范大学信息科学与技术学院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；School of Computer Science and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机科学与技术学院）；Department of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术系）
Object Attribute Matters in Visual Question Answering
Jilin University（吉林大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering
Zhejiang University（浙江大学）；University of Illinois Urbana–Champaign Institute（伊利诺伊大学厄巴纳-香槟分校研究院）；Sixth Medical Center of Chinese PLA General Hospital（中国人民解放军总医院第六医学中心）
Greedy Gradient Ensemble for Robust Visual Question Answering
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Kingsoft Cloud（金山云）；Peng Cheng Laboratory（鹏城实验室）；Huawei Technologies（华为技术有限公司）
Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs
Tsinghua University（清华大学）；Pengcheng Laboratory（鹏城实验室）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Chang’an University（长安大学）
Video Question Answering Using CLIP-Guided Visual-Text Attention
University of Nottingham Ningbo China（宁波诺丁汉大学）；Nottingham Ningbo China Beacons of Excellence Research and Innovation Institute（宁波诺丁汉大学卓越灯塔研究与创新研究院）；Nanyang Technological University（南洋理工大学）
Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Department of computer science, University of Central Florida, Orlando, USA（计算机科学系，中央佛罗里达大学）；Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA（生物工程系，宾夕法尼亚大学）；Department of electrical engineering, Columbia university, New York, NY, USA（电气工程系，哥伦比亚大学）；Technical University of Applied Sciences Regensburg, Regensburg, Germany（应用科学技术大学（雷根斯堡））；Department of Surgery, University of Calgary, Calgary, Alberta, Canada（外科系，卡尔加里大学）；University College of Nabi Akram, Tabriz, Iran（纳比阿克兰大学）；School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran（电气工程学院，伊朗科学技术大学）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；SmartMore（思谋科技）；Huawei Cloud & AI（华为云与计算）
Visual Commonsense based Heterogeneous Graph Contrastive Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人创新中心）
