The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
School of Computer Science, Peking University(北京大学计算机学院);The Chinese University of Hong Kong(香港中文大学);University of California, San Diego(加利福尼亚大学圣迭戈分校);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs
Saudi Data & AI Authority (SDAIA)(沙特数据与人工智能局);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Princess Nourah Bint Abdulrahman University (PNU)(普罗瑟·努拉·本·阿卜杜勒拉赫曼大学)
Birkbeck, University of London(伦敦大学伯贝克学院);School of Advanced Study, Institute of Philosophy, University of London(伦敦大学高等研究院哲学研究所);University of Utah(犹他大学);Tufts University(塔夫茨大学)
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Alibaba Cloud Computing(阿里云计算);Xiaohongshu Inc.(小红书公司);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Probing Cross-modal Information Hubs in Audio-Visual LLMs
Department of Electrical Engineering, Korea Advanced Institute of Science;The Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University, Seoul, Republic of Korea
Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications
School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China(信息与电子学院,北京理工大学,北京);GIC & 6GIC, Institute for Communication Systems (ICS), University of Surrey, Guildford, United Kingdom(5GIC与6GIC,通信系统研究所(ICS), Surrey大学, Guildford,英国);Centre for Wireless Communications, University of Oulu, 90014 Oulu, Finland(无线通信中心,奥卢大学, Oulu,芬兰);School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798(计算机科学与工程学院,南洋理工大学, Singapore)
Rethinking the Understanding Ability across LLMs through Mutual Information
Department of Industrial and Systems Engineering(工业与系统工程系);University of Houston(休斯敦大学);Bakar Computational Health Sciences Institute(巴卡计算健康科学研究所);UCSF
AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
Institute of Medical Technology, Peking University Health Science Center(北京大学医学部医学技术研究所);Department of Biomedical Engineering, Peking University(北京大学生物医学工程系);National Biomedical Imaging Center, Peking University(北京大学国家生物医学成像中心)
Technische Universität Darmstadt(达姆施塔特工业大学);AI Quality & Testing Hub(AI质量与测试中心);Hessian Center for Artificial Intelligence(黑森人工智能中心);National Research Center for Applied Cybersecurity ATHENE(国家应用网络安全研究中心ATHENE)
AtomDisc: An Atom-level Tokenizer that Boosts Molecular LLMs and Reveals Structure--Property Associations
The Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(人工智能动力部,香港科学与技术大学(广州));The College of Computer Science and Technology, The Nanjing University of Aeronautics and Astronautics(计算机科学与技术学院,南京航空航天大学)
OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs
Zhejiang University(浙江大学);Carnegie Mellon University(卡内基梅隆大学);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Alibaba(阿里巴巴)