Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical. To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers. This framework adaptively combines tokens in real-time during inference, significantly reducing computational costs while maintaining high performance. CrossGET features two primary innovations: 1) Cross-Guided Matching and Ensemble. CrossGET leverages cross-modal guided token matching and ensemble to effectively utilize cross-modal information, achieving wider applicability across both modality-independent models, e.g., CLIP, and modality-dependent ones, e.g., BLIP2. 2) Complete-Graph Soft Matching. CrossGET introduces an algorithm for the token-matching mechanism, ensuring reliable matching results while facilitating parallelizability and high efficiency. Extensive experiments have been conducted on various vision-language tasks, such as image-text retrieval, visual reasoning, image captioning, and visual question answering. The performance on both classic multimodal architectures and emerging multimodal LLMs demonstrates the framework's effectiveness and versatility. The code is available at https://github.com/sdc17/CrossGET.
Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology（浙江省数字化服务技术重点实验室）；Hong Kong Polytechnic University（香港理工大学）；The University of Southern Queensland（南昆士兰大学）
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
School of Information Engineering, Chang’an University, China（长安大学信息工程学院）；School of Artificial Intelligence, Shanghai Jiaotong University, China（上海交通大学人工智能学院）
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Carnegie Mellon University（卡内基梅隆大学）；Sichuan University（四川大学）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）
Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
Carnegie Mellon University（卡内基梅隆大学）；University of Nottingham（诺丁汉大学）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University（香港理工大学）；Columbia University（哥伦比亚大学）；University of California, Berkeley（加州大学伯克利分校）
Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））
Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Sichuan University（四川大学）；Huazhong University of Science and Technology（华中科技大学）；Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学PCA实验室）；PCA Lab, School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学PCA实验室）
Accelerating Diffusion Transformers with Token-wise Feature Caching
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science & Technology of China（电子科技大学）；Sichuan University（四川大学）；National University of Defense Technology（国防科技大学）；Zhejiang University（浙江大学）
针对 Diffusion Transformer 计算成本高的问题，提出逐 token 自适应特征缓存，并按层类型与深度设置缓存比例，在 PixArt-α、OpenSora 和 DiT 上免训练实现近无损加速。
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（南方科技大学）；Beihang University（北航）；Huawei Technologies Ltd（华为技术有限公司）；Shanghai AI Lab（上海人工智能实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
College of Software Engineering, Southeast University（东南大学软件工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算机与信息系统学院）
ARC Lab（ARC实验室）；Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Tencent PCG（腾讯平台与内容事业群）；Peng Cheng Laboratory（鹏城实验室）；Shanghai AI Laboratory（上海人工智能实验室）
The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training time, especially on large-scale web datasets. An essential obstacle to training efficiency lies in the entangled prediction rate (percentage of tokens for reconstruction) and corruption rate (percentage of corrupted tokens) in masked language modeling (MLM), that is, a proper corruption rate is achieved at the cost of a large portion of output tokens being excluded from prediction loss. To accelerate the convergence of …
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；CUHK（香港中文大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Infinigence AI（智源人工智能）
HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer…
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
School of information（信息学院）；Electronics, Beijing Institute of Technology, Beijing（电子学院，北京理工大学，北京）；Department of Computer Science, City University of Hong Kong, Hong Kong（计算机科学系，香港城市大学，香港）；School of Computer Science, Wuhan University, Wuhan（计算机科学学院，武汉大学，武汉）；School of Economics（经济学院）；Management, Communication University of China, Beijing（管理学院，中国传媒大学，北京）；School of Science（科学学院）；Engineering, The Chinese University of Hong Kong (Shenzhen), Shenzhen（工程学院，香港中文大学（深圳），深圳）
