Arbitrary Ratio Feature Compression via Next Token Prediction
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institution of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心);People AI, Inc.(People AI公司);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);School of Computer Science and Mathematics, Birkbeck College, University of London(伦敦大学伯克贝克学院计算机科学与数学学院)
Feature compression is increasingly important for improving the efficiency of downstream tasks, especially in applications involving large-scale or multi-modal data. While existing methods typically rely on dedicated models for achieving specific compression ratios, they are often limited in flexibility and generalization. In particular, retraining is necessary when adapting to a new compression ratio. To address this limitation, we propose a novel and flexible Arbitrary Ratio Feature Compression (ARFC) framework, which supports any compression ratio with a single model, eliminating the need for multiple specialized models. At its core, the Arbitrary Ratio Compressor (ARC) is an auto-regressive model that performs compression via next-token prediction. This allows the compression ratio to be controlled at inference simply by adjusting the number of generated tokens. To enhance the quality of the compressed features, two key modules are introduced. The Mixture of Solutions (MoS) module refines the compressed tokens by utilizing multiple compression results (solutions), reducing uncertainty and improving robustness. The Entity Relation Graph Constraint (ERGC) is integrated into the training process to preserve semantic and structural relationships during compression. Extensive experiments on cross-modal retrieval, image classification, and image retrieval tasks across multiple datasets demonstrate that our method consistently outperforms existing approaches at various compression ratios. Notably, in some cases, it even surpasses the performance of the original, uncompressed features. These results validate the effectiveness and versatility of ARFC for practical, resource-constrained scenarios.
PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
Tsinghua University(清华大学);OpenGVLab(开放通用实验室);Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学)
Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Northeastern University(东北大学);Shanghai AI Laboratory(上海人工智能实验室);The Hong Kong University of Science and Technology(香港科技大学);MBZUAI
OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
Media Analytics and Computing Lab, Xiamen University(厦门大学媒体分析与计算实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究所);Alibaba Group(阿里巴巴集团)
Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs
Interdisciplinary Program in AI, Seoul National University(首尔国立大学人工智能跨学科项目);Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Department of Artificial Intelligence, Chung-Ang University(成均馆大学人工智能系)
Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Jiangnan University(江南大学);Nanyang Technological University(南洋理工大学);Chongqing University of Posts and Telecommunications(重庆邮电大学)
Extreme Model Compression for Edge Vision-Language Models: Sparse Temporal Token Fusion and Adaptive Neural Compression
University of Information Technology and Sciences(信息科技与科学大学);The University of Burdwan(布尔德万大学);Tarakeswar Degree College(塔拉克瓦尔学位学院);University of Missouri, Columbia(密苏里大学,哥伦比亚)
ApET: Approximation-Error Guided Token Compression for Efficient VLMs
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology(哈尔滨工业大学);Peking University(北京大学)
A Survey of Token Compression for Efficient Multimodal Large Language Models
Zhejiang University(浙江大学);Westlake University(西湖大学);Xiamen University(厦门大学);National University of Singapore(新加坡国立大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Central Florida(佛罗里达大学);Salesforce AI Research(Salesforce AI研究);Rice University(德克萨斯大学)
WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction
Shanghai Jiao Tong University(上海交通大学);WeChat Vision, Tencent Inc.(腾讯微信视觉部门);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Hong Kong University of Science and Technology(香港科学大学);Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室)