arXiv 2503.194042025-11-10cs.CV

LangBridge：将图像解释为语言嵌入的组合

LangBridge: Interpreting Image as a Combination of Language Embeddings

Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；PengCheng Laboratory（鹏城实验室）；Chongqing University（重庆大学）

Jiaqi Liao, Yuwei Niu, Fanqing Meng, Hao Li, Changyao Tian, Yinuo Du, Yuwen Xiong, Dianqi Li, Xizhou Zhu, Li Yuan, Jifeng Dai, Yu Cheng

备注：: The code and weights are open-sourced. Project page: https://curryx-001.github.io/LangBridge.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LangBridge适配器，将视觉token显式映射为LLM词汇嵌入的线性组合，实现可解释的视觉-语言对齐，并支持跨不同LLM的免预训练即插即用迁移，性能几乎无损失。

摘要：

近年来，大型视觉-语言模型（LVLMs）取得了显著进展，已在多种复杂的视觉-语言任务上达到人类水平的表现。遵循LLaVA范式，主流LVLMs通常采用浅层MLP，通过两阶段训练过程实现视觉-语言对齐：先进行跨模态对齐的预训练，再进行指令微调。尽管该方法已被证明有效，但MLP如何弥合模态差距的底层机制仍不为人所充分理解。虽然一些研究探讨了LLM如何处理变换后的视觉token，但很少有研究关注基础的对齐机制。此外，每当切换LLM骨干网络时，MLP适配器都需要重新训练。为解决这些局限性，我们首先研究了MLP适配器的工作原理，发现它们会逐步学习将视觉嵌入投影到由相应文本嵌入张成的子空间中。基于这一洞察，我们提出了LangBridge，这是一种新颖的适配器，能够显式地将视觉token映射为LLM词汇嵌入的线性组合。这一创新设计使得适配器能够在不同LLM之间进行免预训练迁移，同时保持性能。我们的实验结果表明，在Qwen2-0.5B上预训练的LangBridge适配器可以直接应用于更大的模型，如LLaMA3-8B或Qwen2.5-14B，同时保持具有竞争力的性能。总体而言，LangBridge通过将视觉表示锚定在LLM词汇嵌入中，实现了可解释的视觉-语言对齐，而其即插即用的设计确保了在多个LLM之间的高效复用，且几乎没有性能下降。项目页面见：https://curryx-001.github.io/LangBridge.github.io/

英文摘要：

Recent years have witnessed remarkable advances in Large Vision-Language Models (LVLMs), which have achieved human-level performance across various complex vision-language tasks. Following LLaVA's paradigm, mainstream LVLMs typically employ a shallow MLP for visual-language alignment through a two-stage training process: pretraining for cross-modal alignment followed by instruction tuning. While this approach has proven effective, the underlying mechanisms of how MLPs bridge the modality gap remain poorly understood. Although some research has explored how LLMs process transformed visual tokens, few studies have investigated the fundamental alignment mechanism. Furthermore, the MLP adapter requires retraining whenever switching LLM backbones. To address these limitations, we first investigate the working principles of MLP adapters and discover that they learn to project visual embeddings into subspaces spanned by corresponding text embeddings progressively. Based on this insight, we propose LangBridge, a novel adapter that explicitly maps visual tokens to linear combinations of LLM vocabulary embeddings. This innovative design enables pretraining-free adapter transfer across different LLMs while maintaining performance. Our experimental results demonstrate that a LangBridge adapter pre-trained on Qwen2-0.5B can be directly applied to larger models such as LLaMA3-8B or Qwen2.5-14B while maintaining competitive performance. Overall, LangBridge enables interpretable vision-language alignment by grounding visual representations in LLM vocab embedding, while its plug-and-play design ensures efficient reuse across multiple LLMs with nearly no performance degradation. See our project page at https://curryx-001.github.io/LangBridge.github.io/

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。