The visual projector serves as an essential bridge between the visual encoder and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs adopt a simple MLP to preserve all visual contexts via one-to-one transformation. However, the visual tokens are redundant and can be considerably increased when dealing with high-resolution images, impairing the efficiency of MLLMs significantly. Some recent works have introduced resampler or abstractor to reduce the number of resulting visual tokens. Unfortunately, they fail to capture finer details and undermine the visual reasoning capabilities of MLLMs. In this work, we propose a novel visual projector, which adopts a coarse-to-fine scheme to inject the enriched characteristics to generate the condensed visual tokens. In specific, we first interpolate the visual features as a low-resolution point query, providing the overall visual representation as the foundation. Then, we introduce a region-to-point injection module that utilizes high-resolution, multi-level region-based cues as fine-grained reference keys and values, allowing them to be fully absorbed within the corresponding local context region. This step effectively updates the coarse point query, transforming it into an enriched one for the subsequent LLM reasoning. Extensive experiments demonstrate that our approach compresses the visual tokens by 75%~89%, while achieves comparable or even better performance across diverse benchmarks with significantly higher efficiency. The source codes can be found at https://github.com/CircleRadon/TokenPacker.
Progressive Multimodal Alignment for Continual Instruction Tuning
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心);Kyoto University(京都大学);Migu Culture Technology Co.,Ltd.(咪咕文化科技有限公司);State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology(脑认知与类脑智能技术国家重点实验室)
Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector
University of California, Merced(加州大学梅尔德分校);vivo Mobile Communication Co., Ltd.(vivo移动通信有限公司);University of Queensland(昆士兰大学);UCLA(加州大学洛杉矶分校);University at Buffalo(布法罗大学)
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Translational Data Analytics Institute, The Ohio State University(转化数据分析研究所,俄亥俄州立大学);Department of Biomedical Informatics, The Ohio State University(生物医学信息学系,俄亥俄州立大学)
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University(上海交通大学图像通信与信息处理研究所);MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院(教育部人工智能重点实验室));Huawei Technologies(华为技术有限公司)
Dynamic Pyramid Network for Efficient Multimodal Large Language Model
Beihang University(北航大学);Shanghai AI Laboratory(上海人工智能实验室);KAUST(卡塔尔大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Technical University Of Denmark(丹麦技术大学);Nanjing University of Science and Technology(南京理工大学);Peking University(北京大学);Xiaohongshu Inc(小红书公司)
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Department of Computer Science, Khalifa University of Science and Technology(卡利法科技大学计算机科学系);Information Technology University(信息技术大学);KAU(卡乌大学);University of the Western Australia(西澳大学)
The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
Peking University(北京大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Xi’an Jiaotong University(西安交通大学);Kling Team, Kuaishou Technology(快手科技 Kling 团队)
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室);Wuhan AI Research, Wuhan, China(武汉人工智能研究所)
提出统一高分辨率通用模型 Griffon v2,通过轻量下采样投影器和即插即用视觉分词器实现视觉-语言共指代,在多项视觉感知任务上达到 SOTA 并超越专家模型。
TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University(湖南省级智能计算与语言信息处理重点实验室,湖南师范大学);School of Information Science and Engineering, Hunan Normal University(信息科学与工程学院,湖南师范大学);Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business(长沙人工智能社会实验室,湖南工业大学);School of Computer Science, Hunan University of Technology and Business(计算机科学学院,湖南工业大学);Department of Computer Science, Brunel University London(伦敦布鲁内尔大学计算机科学系);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信研究中心);Department of Electrical and Computer Engineering, University of Houston(电子与计算机工程系,休斯顿大学)
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
Harvard AI and Robotics Lab(哈佛人工智能与机器人实验室);Kempner Institute for the Study of Natural and Artificial Intelligence(凯普纳自然与人工智能研究 institute);Harvard University(哈佛大学)
DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs
University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学)
Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Binjiang Research Institute of Zhejiang University(浙江大学滨江研究院);Linker Technology Research Co. Ltd(领科技术研究有限公司)
LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer
Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学);The Chinese University of Hong Kong(香港中文大学)
EventGPT: Event Stream Understanding with Multimodal Large Language Models
Xidian University(西安电子科技大学);Tsinghua University(清华大学);UCAS(中国科学院大学);Peking University(北京大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)(广东省人工智能与数字经济实验室(深圳))
Exploring the Design Space of 3D MLLMs for CT Report Generation
Vector Institute for Artificial Intelligence(向量人工智能研究所);Department of Biomedical Informatics, Harvard Medical School(生物医学信息学系,哈佛医学院);Peter Munk Cardiac Centre, University Health Network(皮特·蒙克心脏中心,大学健康网络);Medical Biophysics, University of Toronto(医学生物物理系,多伦多大学);Department of Computer Science, University of Toronto(计算机科学系,多伦多大学);Department of Laboratory Medicine and Pathobiology, University of Toronto(实验室医学与病理学系,多伦多大学);AI Hub, University Health Network(人工智能中心,大学健康网络)
VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detection
University of Science and Technology of China(中国科学技术大学);Northeastern University(东北大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)