arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.20092cs.CV
收藏

通过视觉上下文压缩实现高效的大型多模态模型

Efficient Large Multi-modal Models via Visual Context Compression

Johns Hopkins University(约翰斯·霍普金斯大学)

Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, Alan Yuille

AI总结:

针对多模态大模型中视觉 token 的冗余问题,提出 Visual Context Compressor 和分阶段训练方案 LLaVolta,通过逐步压缩视觉 token 提升训练与推理效率,且不牺牲模型在图文及视频理解任务上的性能。

摘要:

尽管大型语言模型(LLMs)中文本嵌入的压缩表示已取得显著进展,但多模态LLMs(MLLMs)中视觉 token 的压缩在很大程度上仍是一个被忽视的领域。在本研究中,我们对这些模型中视觉 token 的冗余分析和高效训练进行了研究。我们的初步实验表明,在测试阶段通过简单的平均池化消除高达 70% 的视觉 token,在 GQA 基准上的视觉问答准确率仅降低了 3%,这表明视觉上下文中存在显著的冗余。针对这一问题,我们引入了 Visual Context Compressor,它减少了视觉 token 的数量,以提高训练和推理效率而不牺牲性能。为了最小化视觉 token 压缩造成的信息损失,同时保持训练效率,我们开发了 LLaVolta,这是一种轻量且分阶段的训练方案,它结合了阶段性的视觉上下文压缩,在训练期间从重度压缩到轻度压缩逐步压缩视觉 token,在测试时不产生信息损失。大量实验表明,我们的方法在图像-语言和视频-语言理解方面均提升了 MLLMs 的性能,同时显著降低了训练成本并提高了推理效率。

英文摘要:

While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑