arXiv 2408.118132025-09-05cs.CV

SEA：面向多模态大语言模型中令牌级视觉-文本融合的监督嵌入对齐

SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；Kuaishou Technology（快手科技）

Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大语言模型现有图像级监督对齐效果不佳、小型模型性能受限的问题，提出令牌级监督嵌入对齐方法SEA，以极小开销提升跨模态理解能力，小型模型获益最显著。

摘要：

多模态大语言模型（MLLMs）通过整合视觉与文本输入展现出卓越能力，但模态对齐仍是最具挑战性的问题之一。当前MLLMs通常依赖简单的适配器架构和预训练方法，在图像级监督的引导下连接视觉编码器与大语言模型（LLM）。我们发现这种范式往往导致模态间对齐效果不佳，严重限制了LLM正确解读视觉特征并基于其进行推理的能力，对较小的语言模型而言尤为明显。这种局限会降低整体性能——在容量约束更显著、适配能力有限的小型语言模型上表现更突出。为解决这一根本局限，我们提出监督嵌入对齐（SEA），这是一种令牌级监督对齐方法，能在预训练过程中实现更精准的视觉-文本对齐。SEA仅带来极小的计算开销，同时能保留语言能力并大幅提升跨模态理解能力。我们的综合分析揭示了适配器在多模态融合中作用的关键洞见，大量实验表明SEA能持续提升不同规模模型的性能，其中小型模型获益最大（Gemma-2B的平均性能提升达7.61%）。这项工作为未来多模态系统开发更有效的对齐策略奠定了基础。

英文摘要：

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual inputs, yet modality alignment remains one of the most challenging aspects. Current MLLMs typically rely on simple adapter architectures and pretraining approaches to bridge vision encoders with large language models (LLM), guided by image-level supervision. We identify this paradigm often leads to suboptimal alignment between modalities, significantly constraining the LLM's ability to properly interpret and reason with visual features particularly for smaller language models. This limitation degrades overall performance-particularly for smaller language models where capacity constraints are more pronounced and adaptation capabilities are limited. To address this fundamental limitation, we propose Supervised Embedding Alignment (SEA), a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. SEA introduces minimal computational overhead while preserving language capabilities and substantially improving cross-modal understanding. Our comprehensive analyses reveal critical insights into the adapter's role in multimodal integration, and extensive experiments demonstrate that SEA consistently improves performance across various model sizes, with smaller models benefiting the most (average performance gain of 7.61% for Gemma-2B). This work establishes a foundation for developing more effective alignment strategies for future multimodal systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。