arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.02080cs.CVcs.CLcs.LG
收藏

EMMA:多模态大语言模型中的高效视觉对齐

EMMA: Efficient Visual Alignment in Multi-Modal LLMs

Department of Electronic and Computer Engineering, New York University(电子与计算机工程系,纽约大学)

Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami

AI总结:

本文提出轻量级跨模态模块 EMMA,通过参数增量低于 0.2% 的早期视觉-语言融合机制生成指令感知视觉表征,提升多模态 LLM 性能与抗幻觉鲁棒性。

摘要:

多模态大语言模型(MLLM)近来通过利用视觉基础模型将图像的核心概念编码为表征,展现出令人瞩目的通用能力。这些表征随后与指令结合,并由语言模型处理以生成高质量回答。尽管在增强语言组件方面取得了显著进展,但如何在语言模型内最佳融合视觉编码以实现任务特定适应性,仍然存在挑战。近期研究聚焦于通过模态适配模块改进这种融合,但其代价是模型复杂度和训练数据需求显著增加。本文提出 EMMA(Efficient Multi-Modal Adaptation,高效多模态适配),这是一个轻量级跨模态模块,旨在高效融合视觉与文本编码,为语言模型生成具备指令感知能力的视觉表征。我们的主要贡献包括:(1)一种高效的早期融合机制,以极少的新增参数(模型规模增加不到 0.2%)整合视觉和语言表征;(2)深入的可解释性分析,阐明所提方法的内部机制;(3)综合实验证明其在 MLLM 的专用和通用基准上均取得显著改进。实证结果表明,EMMA 在多项任务上将性能提升最高达 9.3%,同时显著增强了对幻觉的鲁棒性。我们的代码可在 https://github.com/SaraGhazanfari/EMMA 获取。

英文摘要:

Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations. Our code is available at https://github.com/SaraGhazanfari/EMMA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑