arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.17430cs.CVcs.AIcs.CLcs.LG
收藏

Matryoshka 多模态模型

Matryoshka Multimodal Models

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Microsoft Research(微软研究院)

Mu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae Lee

AI总结:

受 Matryoshka 娃娃启发,提出 M3 模型,通过嵌套视觉 token 表示多粒度信息,支持推理时灵活控制视觉粒度,并揭示固定规模表示与理论上界间的性能差距。

摘要:

大型多模态模型(LMMs)如 LLaVA 在视觉语言推理中表现出色。这些模型首先将图像嵌入为固定数量的视觉 token,然后将其输入大型语言模型(LLM)。然而,这种设计在高分辨率图像和视频等密集视觉场景中会产生过多的 token,导致极大的低效。虽然存在 token 剪枝/合并方法,但它们为每张图像产生单一长度输出,无法在信息密度与效率之间灵活权衡。受 Matryoshka 娃娃概念启发,我们提出 M3:Matryoshka 多模态模型,该模型学习将视觉内容表示为嵌套的视觉 token 集合,捕获多个从粗到细粒度的信息。我们的方法为 LMMs 提供了几个独特优势:(1)在推理过程中可以显式控制每个测试实例的视觉粒度,例如根据预期内容的复杂或简单程度调整用于表示图像的 token 数量;(2)M3 提供了一个分析现有数据集所需粒度的框架,我们发现 COCO 风格的基准测试只需约 9 个视觉 token 即可获得与使用全部 576 个 token 相似的准确率;(3)我们的方法为在样本级别探索性能与视觉 token 长度之间的最佳权衡提供了基础,我们的研究揭示了 oracle 上界与当前固定规模表示之间存在较大差距。

英文摘要:

Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around ~9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑