METEOR：面向高效视觉语言模型的多编码器协同Token剪枝
METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
AI总结：
针对多编码器视觉语言模型计算开销高的问题，提出METEOR渐进式多阶段Token剪枝框架，在编码、融合、解码阶段协同剪枝冗余视觉Token，在11个基准上相比EAGLE减少76%视觉Token且性能仅降0.3%。
摘要：
视觉编码器是多模态理解的基石。CLIP这类单编码器架构在泛化至各类多模态任务时存在固有局限，而近期的多编码器融合方法为利用多个视觉编码器的互补视觉表征实现更优性能，会带来难以承受的计算开销。为解决该问题，我们提出一种渐进式剪枝框架，即Multi-Encoder collaboraTivE tOken pRuning（METEOR，多编码器协同Token剪枝），可在多编码器多模态大语言模型（MLLM）的编码、融合和解码阶段消除冗余视觉Token。在多视觉编码阶段，我们通过秩引导的协同Token分配策略丢弃每个编码器内的冗余Token。随后在多视觉融合阶段，我们融合来自不同编码器的视觉特征，同时通过协同剪枝减少跨编码器冗余。最后，我们在大语言模型（LLM）解码阶段提出一种自适应Token剪枝方法，可根据文本提示进一步丢弃无关Token，并针对特定任务需求动态调整剪枝比例。据我们所知，这是首次通过多阶段剪枝策略成功实现高效的基于多编码器的视觉语言模型的尝试。在11个基准上的大量实验证明了所提方法的有效性。与典型的多编码器MLLM EAGLE相比，METEOR减少了76%的视觉Token，平均性能仅下降0.3%。代码可在https://github.com/YuchenLiu98/METEOR获取。
英文摘要：
Vision encoders serve as the cornerstone of multimodal understanding. Single-encoder architectures like CLIP exhibit inherent constraints in generalizing across diverse multimodal tasks, while recent multi-encoder fusion methods introduce prohibitive computational overhead to achieve superior performance using complementary visual representations from multiple vision encoders. To address this, we propose a progressive pruning framework, namely Multi-Encoder collaboraTivE tOken pRuning (METEOR), that eliminates redundant visual tokens across the encoding, fusion, and decoding stages for multi-encoder MLLMs. For multi-vision encoding, we discard redundant tokens within each encoder via a rank guided collaborative token assignment strategy. Subsequently, for multi-vision fusion, we combine the visual features from different encoders while reducing cross-encoder redundancy with cooperative pruning. Finally, we propose an adaptive token pruning method in the LLM decoding stage to further discard irrelevant tokens based on the text prompts with dynamically adjusting pruning ratios for specific task demands. To our best knowledge, this is the first successful attempt that achieves an efficient multi-encoder based vision language model with multi-stage pruning strategies. Extensive experiments on 11 benchmarks demonstrate the effectiveness of our proposed approach. Compared with EAGLE, a typical multi-encoder MLLMs, METEOR reduces 76% visual tokens with only 0.3% performance drop in average. The code is available at https://github.com/YuchenLiu98/METEOR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
