Visual Question Decomposition on Multimodal Large Language Models
Munich Center for Machine Learning（慕尼黑机器学习中心）；Technical University of Munich（慕尼黑工业大学）；Amazon Web Services（亚马逊云计算服务）；LMU Munich（慕尼黑大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Oxford（牛津大学）
Question decomposition has emerged as an effective strategy for prompting Large Language Models (LLMs) to answer complex questions. However, while existing methods primarily focus on unimodal language models, the question decomposition capability of Multimodal Large Language Models (MLLMs) has yet to be explored. To this end, this paper explores visual question decomposition on MLLMs. Specifically, we introduce a systematic evaluation framework including a dataset and several evaluation criteria to assess the quality of the decomposed sub-questions, revealing that existing MLLMs struggle to produce high-quality sub-questions. To address this limitation, we propose a specific finetuning dataset, DecoVQA+, for enhancing the model's question decomposition capability. Aiming at enabling models to perform appropriate selective decomposition, we propose an efficient finetuning pipeline. The finetuning pipeline consists of our proposed dataset and a training objective for selective decomposition. Finetuned MLLMs demonstrate significant improvements in the quality of sub-questions and the policy of selective question decomposition. Additionally, the models also achieve higher accuracy with selective decomposition on VQA benchmark datasets.
Visual Explanations from Hadamard Product in Multimodal Deep Networks
Seoul National University（首尔大学）
对模型学习表征的直观解释有助于理解学习的基本原理。以前的作品中的注意力模型用于在图像或文本上使用它们的学习权重来可视化关注区域，以确认它们的预期机制。Kim et al.（2016）表明，多模态深度网络中的Hadamard产品，以视觉问答任务的联合功能而闻名，隐含地执行视觉输入的注意机制。在这项工作中，我们扩展了他们的工作，以表明多模态深度网络中的Hadamard乘积不仅适用于视觉输入，而且还适用于文本输入，同时使用所提出的基于梯度的可视化技术。阿达玛产品的注意力效果可视化视觉和文本输入通过分析两个输入和输出的阿达玛产品与所提出的方法，并与学习的视觉问答模型的注意力权重进行比较。
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；iFLYTEK AI Research（iFLYTEK人工智能研究院）；Hong Kong University of Science and Technology（香港科学与技术大学）
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University（西湖大学工程学院）；City University of Hong Kong（香港城市大学）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity
School of AI Convergence, Soongsil University（顺斯利大学人工智能融合学院）；MAUM AI Inc.（MAUM人工智能公司）；Department of Intelligent Semiconductors, Soongsil University（顺斯利大学智能半导体系）
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
Mercer Mettl（默瑟梅特尔）；IGDTUW（英迪拉·甘地德里技术大学（女子））；Google Research（谷歌研究院）；Indian Institute of Technology Guwahati（古瓦哈蒂印度理工学院）；University of Pennsylvania（宾夕法尼亚大学）
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
School of Artificial Intelligence UCAS（中国科学院大学人工智能学院）；Institute of Automation CAS（中国科学院自动化研究所）；Ant Digital Technologies Ant Group（蚂蚁数字科技蚂蚁集团）；RUC（中国人民大学）；BIT（北京理工大学）
True Multimodal In-Context Learning Needs Attention to the Visual Context
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑技术大学）；Siemens AG（西门子股份公司）；University of Science and Technology of China（中国科学技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Konrad Zuse School of Excellence in Reliable AI (relAI)（Konrad Zuse可靠性人工智能卓越学院）；University of Oxford（牛津大学）
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China（东北大学计算机科学与工程学院，沈阳 110819，中国）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China（东北大学计算机与通信工程学院，秦皇岛 066004，中国）
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Zhongguancun Academy（中关村学院）
Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；School of Cyberspace Security, Beijing University of Posts and Telecommunications（网络安全学院，北京邮电大学）
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
