The various limitations of Generative AI, such as hallucinations and model failures, have made it crucial to understand the role of different modalities in Visual Language Model (VLM) predictions. Our work investigates how the integration of information from image and text modalities influences the performance and behavior of VLMs in visual question answering (VQA) and reasoning tasks. We measure this effect through answer accuracy, reasoning quality, model uncertainty, and modality relevance. We study the interplay between text and image modalities in different configurations where visual content is essential for solving the VQA task. Our contributions include (1) the Semantic Interventions (SI)-VQA dataset, (2) a benchmark study of various VLM architectures under different modality configurations, and (3) the Interactive Semantic Interventions (ISI) tool. The SI-VQA dataset serves as the foundation for the benchmark, while the ISI tool provides an interface to test and apply semantic interventions in image and text inputs, enabling more fine-grained analysis. Our results show that complementary information between modalities improves answer and reasoning quality, while contradictory information harms model performance and confidence. Image text annotations have minimal impact on accuracy and uncertainty, slightly increasing image relevance. Attention analysis confirms the dominant role of image inputs over text in VQA tasks. In this study, we evaluate state-of-the-art VLMs that allow us to extract attention coefficients for each modality. A key finding is PaliGemma's harmful overconfidence, which poses a higher risk of silent failures compared to the LLaVA models. This work sets the foundation for rigorous analysis of modality integration, supported by datasets specifically designed for this purpose.
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
New York University, New York, USA(纽约大学);Tsinghua University, Beijing, China(清华大学);Columbia University, New York, USA(哥伦比亚大学);University of Michigan, Ann Arbor, USA(密歇根大学)
DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
Department of Computer Science(计算机科学系);Engineering, Ohio State University, Ohio, US(工程系,俄亥俄州立大学,俄亥俄,美国);Department of Computer Science, University of Central Florida, Florida, US(计算机科学系,中央佛罗里达大学,佛罗里达,美国)
Revisiting Change VQA in Remote Sensing with Structured and Native Multimodal Qwen Models
Computer Engineering Department, College of Computer and Information Sciences, King Saud University(计算机工程系,计算机与信息科学学院,沙特国王大学);Applied Computer Science Department, College of Applied Computer Science, King Saud University(应用计算机科学系,应用计算机科学学院,沙特国王大学)
Enhancing Pathological VLMs with Cross-scale Reasoning
Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电气与计算机工程系);PuzzleLogic Pte Ltd(PuzzleLogic私人有限公司);Department of Pathology, Fujian Medical University Cancer Hospital & Fujian Cancer Hospital(福建医科大学附属肿瘤医院病理科暨福建省肿瘤医院)
Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answering
Computer Science Department, Morgan State University(莫尔甘州大学计算机科学系);International Organization for Migration (IOM)(国际移民组织);Electrical & Computer Engineering Department, Morgan State University(莫尔甘州大学电气与计算机工程系)
UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
Fung Institute for Engineering Leadership(工程领导力基金会);University of California, Berkeley(加州大学伯克利分校);Department of Radiology(放射科);University of California, San Francisco(加州大学旧金山分校);Division of Clinical Informatics and Digital Transformation(临床信息学与数字转型部);Department of Neurological Surgery(神经外科部)
Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(人工智能与数字经济广东实验室);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);The Hong Kong Polytechnic University(香港理工大学);Great Bay University(大鹏大学);Shenzhen University(深圳大学);Huawei(华为)
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
Computer Aided Medical Procedures (CAMP)(计算机辅助医疗程序);TU Munich, Germany(慕尼黑工业大学,德国);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Munich, Germany(慕尼黑,德国);Zhongshan Hospital, Fudan University, China(复旦大学中山医院);The University of Hong Kong, Hongkong, China(香港大学,香港,中国)
HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
Simula Metropolitan Center for Digital Engineering (SimulaMet)(Simula数字工程研究中心);Oslo Metropolitan University (OsloMet)(奥斯陆 Metropolitan 大学);Simula Research Laboratory(Simula研究实验室)
MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery(高端工程机械智能制造国家重点实验室);Department of Electrical and Computer Engineering, Sungkyunkwan University(成均馆大学电气与计算机工程系);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院)
Multi-Sourced Compositional Generalization in Visual Question Answering
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, China(北京智能信息科技重点实验室,计算机科学与技术学院,北京理工大学,中国);Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, China(广东机器感知与智能计算实验室,深圳MSU-BIT大学,中国)