Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Tencent Inc(腾讯公司);Peking University(北京大学)
Visual Question Answering (VQA) models are prone to learn the shortcut solution formed by dataset biases rather than the intended solution. To evaluate the VQA models' reasoning ability beyond shortcut learning, the VQA-CP v2 dataset introduces a distribution shift between the training and test set given a question type. In this way, the model cannot use the training set shortcut (from question type to answer) to perform well on the test set. However, VQA-CP v2 only considers one type of shortcut and thus still cannot guarantee that the model relies on the intended solution rather than a solution specific to this shortcut. To overcome this limitation, we propose a new dataset that considers varying types of shortcuts by constructing different distribution shifts in multiple OOD test sets. In addition, we overcome the three troubling practices in the use of VQA-CP v2, e.g., selecting models using OOD test sets, and further standardize OOD evaluation procedure. Our benchmark provides a more rigorous and comprehensive testbed for shortcut learning in VQA. We benchmark recent methods and find that methods specifically designed for particular shortcuts fail to simultaneously generalize to our varying OOD test sets. We also systematically study the varying shortcuts and provide several valuable findings, which may promote the exploration of shortcut learning in VQA.
A Too-Good-to-be-True Prior to Reduce Shortcut Reliance
HSE University(俄罗斯高等经济大学);University College London(伦敦大学学院);Research Center Jülich(尤利希研究中心);Heinrich Heine University Düsseldorf(海因里希·海涅大学杜塞尔多夫分校);The Alan Turing Institute(阿兰·图灵研究所)
Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);LIP6(巴黎第六大学计算机科学实验室);Carney Institute for Brain Science, Brown University(布朗大学卡尼脑科学研究所);Idiap Research Institute(伊迪亚普研究所);Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation
School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)(华中科技大学与平安财产保险研究联合实验室)
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人与智能系统国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
Raidium;Université de Paris Cité, Hôpital Européen Georges Pompidou, AP-HP(巴黎西岱大学,乔治·蓬皮杜欧洲医院,AP-HP);Department of Vascular and Oncological Interventional Radiology, INSERM(血管与肿瘤介入放射学系,法国国家健康与医学研究院)
Ask4VG: Risk-Aware Question Selection for Reducing Prior-Driven Answers in Medical VQA
School of Microelectronics, Tianjin University, Tianjin 300072, China(天津大学电子工程学院,天津 300072,中国);DISI, University of Trento, Trento, Italy(特伦托大学DISI研究所,意大利特伦托);School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China(天津大学电气与信息工程学院,天津 300072,中国)
PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis
National University of Singapore(新加坡国立大学);PuzzleLogic Pte Ltd(拼图逻辑私人有限公司);Fujian Medical University Cancer Hospital & Fujian Cancer Hospital(福建医科大学附属肿瘤医院)
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Cohere Labs(Cohere实验室);Eisai Inc.(艾伯维公司);Indian Institute of Science, Bangalore(班加罗尔印度科学研究院)
InViC: Intent-aware Visual Cues for Medical Visual Question Answering
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China(集成空天地海大数据应用技术国家工程实验室,计算机科学与工程学院,西北工业大学,西安710072,中国);Westlake University(西湖大学);Southern Medical University(南方医科大学)
尽管视觉问答(VQA)取得了很大的进步,但当前的VQA模型严重依赖于问题类型及其对应的频繁答案之间的表面相关性(即,语言先验)进行预测,而无需真正理解输入。在这项工作中,我们定义的训练实例具有相同的问题类型,但不同的答案为\textit{superficially similar instances},并归因于语言先验的VQA模型对这样的实例的混乱。为了解决这个问题,我们提出了一个新的训练框架,明确鼓励VQA模型区分表面上相似的实例。具体来说,对于每个训练实例,我们首先构建一个包含其表面相似的对应物的集合。然后,我们利用提出的区分模块,以增加之间的距离的实例和它的答案空间中的同行。通过这种方式,VQA模型被迫进一步关注输入中除问题类型之外的其他部分,这有助于克服语言先验。实验结果表明,该方法在VQA-CP v2上达到了最先进的性能。代码可从\href{https://github.com/wyk-nku/Distinguishing-VQA.git}{Distinguishing-VQA}获得。
HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision
Southern University of Science and Technology, Shenzhen, China(南方科技大学,深圳,中国);College of William and Mary, Williamsburg, VA 23185, USA(威廉玛丽学院,威廉斯堡,VA 23185,美国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Greedy Gradient Ensemble for Robust Visual Question Answering
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Kingsoft Cloud(金山云);Peng Cheng Laboratory(鹏城实验室);Huawei Technologies(华为技术有限公司)
RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology
Department of Computer Science, Johns Hopkins University(约翰霍普金斯大学计算机科学系);Clinic of Radiology and Nuclear Medicine, University Hospital Basel(巴塞尔大学医院放射科与核医学科);Department of Oncology, Johns Hopkins School of Medicine(约翰霍普金斯医学院肿瘤科)
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Eisai Inc.(卫材株式会社);IISc, Bangalore(印度科学研究所班加罗尔分校);Cohere Labs Community(Cohere实验室社区)
Overcoming Language Priors with Self-supervised Learning for Visual Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Science and Technology of China(中国科学技术大学)
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Shandong University(山东大学);Shandong Computer Science Center (National Supercomputer Center in Jinan)(山东计算机科学中心(国家超级计算济南中心));Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));National University of Singapore(新加坡国立大学)
Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥国家科学中心);Anhui Polytechnic University(安徽理工大学);Hefei University of Technology(合肥工业大学);Anhui University(安徽大学);IGS, Imperial College London(帝国理工学院伦敦分校)