In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connector has been relatively less explored. In this study, we aim to propose a strong vision-language connector that enables MLLMs to achieve high accuracy while maintain low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to extract them. Building on these findings, we introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds compared with baseline, while simultaneously outperforming baseline methods. This highlights the effectiveness and efficiency of AcFormer. Codes are available at https://github.com/liuhaogeng/Anchor-Former.
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
AMAP, Alibaba Group(阿里云实验室,阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学)
Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Harbin Institute of Technology, Shenzhen(深圳哈尔滨工业大学);University of Science and Technology Beijing(北京科技大学)
Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
Harbin Institute of Technology(哈尔滨工业大学);Zhongguancun Academy(中关村学院);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院);Nankai University(南开大学);Shanghai Jiaotong University(上海交通大学);Zhejiang University(浙江大学)
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Great Bay University(大湾区大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));University of Oxford(牛津大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
China University of Petroleum (East China)(中国石油大学(华东));Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学);Great Wall Motor(长城汽车);Nanyang Technological University(南洋理工大学);The University of Hong Kong(香港大学)
Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院)
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);Intelligent Game and Decision Lab, Academy of Military Sciences(智能游戏与决策实验室,军事科学院)
The Price of Consistency: Exploiting Visual Anchors for Multimodal Jailbreaking in Video Generation
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Hunan University(湖南大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
Emory University(埃默里大学);University of Electronic Science and Technology of China(电子科技大学);University of Illinois Chicago(伊利诺伊大学香槟分校);The Hong Kong Polytechnic University(香港理工大学)
AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Casivision(中科视语);Weiqiao-UCAS Science and Technology Park(魏桥国科科技园)
EventGPT: Event Stream Understanding with Multimodal Large Language Models
Xidian University(西安电子科技大学);Tsinghua University(清华大学);UCAS(中国科学院大学);Peking University(北京大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)(广东省人工智能与数字经济实验室(深圳))
Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC
Xi'an Jiaotong University(西安交通大学);Queen Mary University of London(伦敦女王玛丽大学);Center for Multimodal AI(多模态人工智能中心);Digital Environment Research Institute(数字环境研究院)
CUS-GS: A Compact Unified Structured Gaussian Splatting Framework for Multimodal Scene Representation
School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机科学学院);CAD & CG, Zhejiang University(浙江大学计算机辅助设计与图形学研究所);School of Computer Science, University of Bristol(布里斯托大学计算机科学学院)
From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning
Tsinghua University(清华大学);University of South California(南加州大学);Qwen Team, Alibaba Group(通义实验室,阿里巴巴集团);University of California San Diego(南加州大学圣地亚哥分校);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学)
Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis
College of Computer and Information Science, Fujian Agriculture and Forestry University(福建农林大学计算机与信息科学学院);College of Future Technology, Fujian Agriculture and Forestry University(福建农林大学未来技术学院);Digital Fujian Institute of Agricultural Big Data, Fujian Agriculture and Forestry University(福建农业大数据数字福建研究院);Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital(福建医科大学临床肿瘤学院病理科及福建癌症医院)
InterPruner: Interactive Structured Pruning via Taylor-Implicit Criterion and Language-Prior Modulator for Multimodal Object Detection
Beijing University of Technology(北京工业大学);Southwest University(西南大学);China University of Geosciences Wuhan(中国地质大学(武汉));Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学);Yunnan Normal University(云南师范大学);Beijing Forestry University(北京林业大学);Ghent University(根特大学)