arXiv 2412.161582025-02-09cs.CV

HoVLE：通过整体视觉-语言嵌入释放单一体视觉-语言模型的潜力

HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）

Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, Jifeng Dai

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对单一体视觉-语言模型（VLM）微调预训练大语言模型（LLM）导致语言能力下降的问题，提出HoVLE，通过整体视觉-语言嵌入模块及多阶段训练，在保持LLM语言能力的同时提升视觉性能，接近组合式VLM水平。

摘要：

大型语言模型（LLMs）的快速发展推动了视觉-语言模型（VLMs）的进步。避免特定模态编码器的单一体VLMs为组合式VLMs提供了有前景的替代方案，但面临性能较差的挑战。大多数现有单一体VLMs需要微调预训练LLMs以获取视觉能力，这可能会降低其语言能力。为解决这一困境，本文提出了一种名为HoVLE的新型高性能单一体VLM。我们注意到，当图像嵌入与文本嵌入对齐时，LLMs已被证明能够解释图像。当前单一体VLMs的挑战实际上在于缺乏适用于视觉和语言输入的整体嵌入模块。因此，HoVLE引入了一个整体嵌入模块，将视觉和文本输入转换到共享空间，使LLMs能够以与处理文本相同的方式处理图像。此外，我们精心设计了多阶段训练策略以增强整体嵌入模块：首先训练该模块从预训练视觉编码器提取视觉特征，并从LLM提取文本嵌入，从而实现对未配对随机图像和文本标记的大规模训练；然后整个模型在多模态数据上进行下一个标记预测以对齐嵌入；最后纳入指令微调阶段。实验表明，HoVLE在各种基准上的性能接近领先的组合式模型，大幅优于之前的单一体模型。模型可在https://huggingface.co/OpenGVLab/HoVLE获取。

英文摘要：

The rapid advance of Large Language Models (LLMs) has catalyzed the development of Vision-Language Models (VLMs). Monolithic VLMs, which avoid modality-specific encoders, offer a promising alternative to the compositional ones but face the challenge of inferior performance. Most existing monolithic VLMs require tuning pre-trained LLMs to acquire vision abilities, which may degrade their language capabilities. To address this dilemma, this paper presents a novel high-performance monolithic VLM named HoVLE. We note that LLMs have been shown capable of interpreting images, when image embeddings are aligned with text embeddings. The challenge for current monolithic VLMs actually lies in the lack of a holistic embedding module for both vision and language inputs. Therefore, HoVLE introduces a holistic embedding module that converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts. Furthermore, a multi-stage training strategy is carefully designed to empower the holistic embedding module. It is first trained to distill visual features from a pre-trained vision encoder and text embeddings from the LLM, enabling large-scale training with unpaired random images and text tokens. The whole model further undergoes next-token prediction on multi-modal data to align the embeddings. Finally, an instruction-tuning stage is incorporated. Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks, outperforming previous monolithic models by a large margin. Model available at https://huggingface.co/OpenGVLab/HoVLE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。