Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings \textbf{question the assumption} that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Dalian University of Technology(大连理工大学);Tsinghua University(清华大学)
针对视觉分词器与下游任务优化脱节导致的表示瓶颈,提出 ETT 端到端联合调优方法,在多模态理解与生成任务上较冻结基线提升 2-6%。
One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination
National Key Laboratory for Novel Software Technology, Nanjing University, China(南京大学新型软件技术国家重点实验室);School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院)
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Global Innovation Exchange, Tsinghua University(清华大学全球创新交流中心);Department of Automation, Tsinghua University(清华大学自动化系);School of Computer Science, Peking University(北京大学计算机科学系)
MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Arizona State University(亚利桑那州立大学);University of Washington(华盛顿大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);Duke University(杜克大学);Zoom Communications(Zoom通讯)
TRIO: Token Reduction via Inference-Objective Guidance for Efficient Vision-Language Models
School of Cyberspace Security, Northwestern Polytechnical University(网络安全学院,西北工业大学);School of Computer Science, Northwestern Polytechnical University(计算机学院,西北工业大学);Intellifusion(智融科技)
When LLaVA Meets Objects: Token Composition for Vision-Language-Models
Tuebingen AI Center/University of Tuebingen(图宾根人工智能中心/图宾根大学);Max Planck Institute for Informatics, SIC(马克斯·普朗克信息学院,SIC);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)
Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Jiangnan University(江南大学);Nanyang Technological University(南洋理工大学);Chongqing University of Posts and Telecommunications(重庆邮电大学)
FlexAttention for Efficient High-Resolution Vision-Language Models
UMass Amherst(马萨诸塞大学阿默斯特分校);Princeton University(普林斯顿大学);South China University of Technology(华南理工大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Random Wins All: Rethinking Grouping Strategies for Vision Tokens
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China(自动化研究所,中国科学院,北京);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京)
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局信息通信研究院)
The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models
School of Engineering, Institute of Science Tokyo(东京科学大学工学院);College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电气与计算机工程系)
Leveraging per Image-Token Consistency for Vision-Language Pre-training
Southern University of Science and Technology(南方科技大学);Hong Kong University of Science and Technology(香港科技大学);ByteDance(字节跳动);Peng Cheng Laboratory(鹏城实验室)