When LLaVA Meets Objects: Token Composition for Vision-Language-Models
Tuebingen AI Center/University of Tuebingen（图宾根人工智能中心/图宾根大学）；Max Planck Institute for Informatics, SIC（马克斯·普朗克信息学院，SIC）；MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）
Soumya Jahagirdar, Walid Bousselham, Anna Kukleva, Hilde Kuehne
Current autoregressive Vision Language Models (VLMs) usually rely on a large number of visual tokens to represent images, resulting in a need for more compute especially at inference time. To address this problem, we propose Mask-LLaVA, a framework that leverages different levels of visual features to create a compact yet information-rich visual representation for autoregressive VLMs. Namely, we combine mask-based object representations together with global tokens and local patch tokens. While all tokens are used during training, it shows that the resulting model can flexibly drop especially the number of mask-based object-tokens at test time, allowing to adapt the number of tokens during inference without the need to retrain the model and without a significant drop in performance. We evaluate the proposed approach on a suite of standard benchmarks showing results competitive to current token efficient methods and comparable to the original LLaVA baseline using only a fraction of visual tokens. Our analysis demonstrates that combining multi-level features enables efficient learning with fewer tokens while allowing dynamic token selection at test time for good performance.
ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）；National University of Defense Technology（国防科技大学）
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
Peking University（北京大学）；Sun Yat-sen University（中山大学）；University of Rochester（罗切斯特大学）；National University of Singapore（新加坡国立大学）；Peng Cheng Laboratory（鹏城实验室）
VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；Tencent Youtu Lab（腾讯优图实验室）
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（pathology department, 国家癌症中心/国家癌症临床研究中心/癌症医院, 中国医学科学院和北京协和医学院）
Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
Cohere Labs Community（Cohere Labs社区）；Indian Institute of Science, Bangalore（印度科学研究所班加罗尔分校）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Imperial College London（伦敦帝国学院）；Eisai Inc.（卫材公司）；EleutherAI；Georgia Institute of Technology（佐治亚理工学院）
Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Chongqing Institute of East China Normal University（华东师范大学重庆研究院）；Shanghai University of Engineering Science（上海工程技术大学）
MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Arizona State University（亚利桑那州立大学）；University of Washington（华盛顿大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Duke University（杜克大学）；Zoom Communications（Zoom通讯）
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
Shenzhen campus of Sun Yat-sen University（中山大学深圳校区）；Huawei（华为）；The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）
