arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.10994cs.CLcs.AIcs.CVcs.MM
收藏

少即是多:一种简单而有效的用于高效多模态LLM的Token缩减方法

Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Lehigh University(利哈伊大学)

Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang

AI总结:

提出TRIM方法,利用CLIP度量在多模态LLM中缩减图像Token,在12个数据集上显著降低计算开销并保持性能。

摘要:

多模态大型语言模型(MLLMs)的快速发展使其在各个领域表现出色。然而,这种进步伴随着这些模型资源消耗的大幅激增。我们通过引入一种新方法——使用CLIP度量进行Token缩减(TRIM),来解决这一紧迫问题,旨在不牺牲性能的前提下提高MLLMs的效率。受视觉问答(VQA)任务中人类注意力模式的启发,TRIM为图像Token的选择和缩减提供了新视角。TRIM方法已在12个数据集上进行了广泛测试,结果表明在保持一致性能水平的同时,计算开销显著减少。这项研究标志着高效MLLM开发的关键一步,促进了高性能模型的更大可及性和可持续性。

英文摘要:

The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑