频率调制的视觉修复用于Matryoshka大多模态模型
Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models
AI总结:
本文提出FMVR策略,通过频率调制修复视觉语义,提升LMMs在减少视觉token时的推理能力,并在多个基准测试中验证其有效性。
摘要:
大型多模态模型(LMMs)由于众多视觉token而难以适应变化的计算预算。先前的方法尝试在LLMs之前或内部减少视觉token的数量。然而,这些策略不可避免地导致视觉语义的损失。为了解决这些问题,我们引入了FMVR,一种即插即用且极其简单的频率调制视觉修复策略,以在减少视觉token的情况下提升LMMs的推理能力。具体来说,FMVR通过AvgPool和MaxPool将较少的视觉token的视觉表示分解为低频和高频组件。随后,这些频率通过轻量级可学习参数进行调制。来自AvgPool的高频作用作为显著性滤波器以增强显著性视觉语义,而来自MaxPool的低频则作为反显著性滤波器以加强弱视觉语义。这使得在少量视觉token主导的视觉语义得以保留,并修复稀释的视觉语义。此外,我们将FMVR注入到Matryoshka表示学习中,以学习粗到细的视觉token集合,从而在推理过程中弹性调整视觉token的数量,同时保持可比的性能。在10个图像和4个视频基准测试中,实验表明FMVR-LLaVA将LLaVA-1.5-7B的FLOPs减少89%,同时保持几乎100%的原始准确性。代码将公开。
英文摘要:
Large Multimodal Models (LMMs) struggle to adapt varying computational budgets due to numerous visual tokens. Previous methods attempted to reduce the number of visual tokens before or within LLMs. However, these strategies inevitably result in the loss of visual semantic. To address these issues, we introduce FMVR, a plug-and-play and extremely simple Frequency-Modulated Visual Restoration strategy to boost the reasoning ability of LMMs under visual token reduction. Specifically, FMVR disentangles the visual representation of fewer visual tokens into low- and high-frequency components through AvgPool and MaxPool. The derived frequencies are subsequently modulated using lightweight learnable parameters. The high-frequency from AvgPool acts as a saliency filter to enhance saliency visual semantics, while the low-frequency from MaxPool acts as an anti-saliency filter to strengthen weak visual semantics. It enables the preservation of visual semantics dominated by few visual tokens and the restoration of diluted visual semantics. Additionally, we inject FMVR into Matryoshka Representation Learning to learn coarse-to-fine visual token sets, thus enabling to elastically adjust the number of visual tokens during inference while maintaining comparable performance. Experiments across 10 image-based and 4 video-based bench marks demonstrate that FMVR-LLaVA reduce the FLOPs of LLaVA-1.5-7B by 89%, while maintaining almost 100% of the original accuracy. The code will be open.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。