近年来，多模态大语言模型（MLLMs）在视觉问答、常识推理等任务中展现出卓越能力，而视觉感知模型也在检测、分割等感知任务中取得了长足进展。然而，MLLMs主要聚焦于高层图像-文本解读，在细粒度视觉理解方面存在短板；视觉感知模型则因模型容量有限，通常面临开放世界分布偏移的问题。为应对这些挑战，我们提出了相互增强型多模态大语言模型（Mutually Reinforced Multimodal Large Language Model，MR-MLLM）——一种协同提升视觉感知与多模态理解的新型框架。首先，我们提出共享查询融合机制，将视觉模型输出的精细化视觉输入与语言模型的语言深度相协调，协同增强多模态理解与视觉感知能力。其次，我们提出感知增强的跨模态整合方法，引入视觉感知输出的新型模态信息（如目标检测边界框）以捕捉细微视觉元素，从而丰富对视觉与文本数据的理解。此外，我们还提出了创新的感知嵌入提示生成机制，将感知信息嵌入语言模型的提示中，使响应在上下文与感知层面实现对齐，以获得更准确的多模态解读。大量实验表明，MR-MLLM在各类多模态理解与视觉感知任务中表现优异，尤其是在需要处理极端情况视觉感知与细粒度语言理解的任务中优势显著。
英文摘要：
In recent years, multimodal large language models (MLLMs) have shown remarkable capabilities in tasks like visual question answering and common sense reasoning, while visual perception models have made significant strides in perception tasks, such as detection and segmentation. However, MLLMs mainly focus on high-level image-text interpretations and struggle with fine-grained visual understanding, and vision perception models usually suffer from open-world distribution shifts due to their limited model capacity. To overcome these challenges, we propose the Mutually Reinforced Multimodal Large Language Model (MR-MLLM), a novel framework that synergistically enhances visual perception and multimodal comprehension. First, a shared query fusion mechanism is proposed to harmonize detailed visual inputs from vision models with the linguistic depth of language models, enhancing multimodal comprehension and vision perception synergistically. Second, we propose the perception-enhanced cross-modal integration method, incorporating novel modalities from vision perception outputs, like object detection bounding boxes, to capture subtle visual elements, thus enriching the understanding of both visual and textual data. In addition, an innovative perception-embedded prompt generation mechanism is proposed to embed perceptual information into the language model's prompts, aligning the responses contextually and perceptually for a more accurate multimodal interpretation. Extensive experiments demonstrate MR-MLLM's superior performance in various multimodal comprehension and vision perception tasks, particularly those requiring corner case vision perception and fine-grained language comprehension.
EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology（北京理工大学多学科科学先进研究院）；School of Mechatronical Engineering, Beijing Institute of Technology（北京理工大学机电工程学院）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；Beijing Institute of Technology（北京理工大学）
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
ChartBench: A Benchmark for Complex Visual Reasoning in Charts
International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Hong Kong University of Science and Technology, Guangzhou（香港科技大学（广州））
Tianjin University（天津大学）；University of South Carolina（南卡罗来纳大学）
Referring expression comprehension (REC) aims to localize a text-related region in a given image by a referring expression in natural language. Existing methods focus on how to build convincing visual and language representations independently, which may significantly isolate visual and language information. In this paper, we argue that for REC the referring expression and the target region are semantically correlated and subject, location and relationship consistency exist between vision and language.On top of this, we propose a novel approach called MutAtt to construct mutual guidance betwee…
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
Vision-Centric Activation and Coordination for Multimodal Large Language Models
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（人工智能MoE实验室，上海交通大学）；Ant Group（蚂蚁集团）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波数字孪生研究所，东部技术研究所，宁波）
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models
Tsinghua University（清华大学）；Jilin University（吉林大学）；Northwest Minzu University（西北民族大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference
Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）；New York University（纽约大学）
What Large Language Models Bring to Text-rich VQA?
SenseTime Research（商汤科技研究院）；Nanjing University of Science and Technology（南京理工大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；ByteDance Inc.（字节跳动有限公司）
SELU: Self-Learning Embodied MLLMs in Unknown Environments
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；School of Computer Science, Peking University（北京大学计算机学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（国家人机混合增强智能重点实验室）；National Engineering Research Center for Visual Information and Applications（国家视觉信息与应用工程技术研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）；Institute of Artificial Intelligence (TeleAI)（人工智能研究院（TeleAI））；China Telecom（中国电信）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology Beijing（北京科技大学）
IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
MAIS, Institute of Automation（自动化研究所信息处理中心）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Research Center of Aerospace Information, Institute of Automation（自动化研究所航空信息研究中心）
NavBench: Probing Multimodal Large Language Models for Embodied Navigation
The University of Adelaide（阿德莱德大学）；The University of Queensland（昆士兰大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61研究中心）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Tongji University（同济大学）
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Tencent Youtu Lab（腾讯优图实验室）；Nanjing University（南京大学）
EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuning
Department of Computer Science, Western University（计算机科学系，西部大学）；Meituan Inc.（美团公司）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，东华大学）
