Endowing Large Multimodal Models (LMMs) with visual grounding capability can significantly enhance AIs' understanding of the visual world and their interaction with humans. However, existing methods typically fine-tune the parameters of LMMs to learn additional segmentation tokens and overfit grounding and segmentation datasets. Such a design would inevitably cause a catastrophic diminution in the indispensable conversational capability of general AI assistants. In this paper, we comprehensively evaluate state-of-the-art grounding LMMs across a suite of multimodal question-answering benchmarks, observing drastic performance drops that indicate vanishing general knowledge comprehension and weakened instruction following ability. To address this issue, we present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations -- a straightforward yet effective design based on the fact that word-pixel correspondences conducive to visual grounding inherently exist in the attention mechanism of well-trained LMMs. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits, which a SAM-based mask refiner can further optimise. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data, but achieves competitive performance on referring expression segmentation and panoptic narrative grounding benchmarks while completely preserving LMMs' original conversational ability. Additionally, with instruction-following ability preserved and grounding ability obtained, F-LMM can be directly applied to complex tasks like reasoning segmentation, grounded conversation generation and visual chain-of-thought reasoning. Our code can be found at https://github.com/wusize/F-LMM.
From What to Which: Decoding Modifier Grounding in Frozen MLLMs
AI for Good (AIGO)（人工智能促进美好（AIGO））；Istituto Italiano di Tecnologia（意大利技术研究院）；University of Siena（锡耶纳大学）；University of Genoa（热那亚大学）；University of Verona（维罗纳大学）
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Tianjin University（天津大学）；Carnegie Mellon University（卡内基梅隆大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Aalto University（阿尔托大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California - Merced（加州大学默塞德分校）；Linköping University（林雪平大学）；Google Research（谷歌研究院）
Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家科学中心）；Anhui Polytechnic University（安徽理工大学）；Hefei University of Technology（合肥工业大学）；Anhui University（安徽大学）；IGS, Imperial College London（帝国理工学院伦敦分校）
GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；The University of Western Australia（西澳大利亚大学）
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Communication Content Cognition（通信内容认知技术国家重点实验室）；Peng Cheng Laboratory（鹏城实验室）
DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）
CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港大学深圳研究院数据科学学院）
Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Interdisciplinary Institute of Artificial Intelligence, Xidian University（西安电子科技大学跨学科人工智能研究院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Northwest Institute of Nuclear Technology（西北核技术研究所）；School of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Pengcheng Laboratory（鹏城实验室）；Pazhou Lab (Huangpu)（琶洲实验室（黄埔））；Hainan University（海南大学）；University of California at Merced（加州大学默塞德分校）
Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）
RECIPE: Procedural Planning via Grounding in Instructional Video
Khoury College of Computer Sciences, Northeastern University, Boston（东北大学北斯托顿学院计算机科学学院）；Department of Mathematics and Computer Science, University of Catania, Italy（卡塔尼亚大学数学与计算机科学系）
