MATE: Meet At The Embedding——连接图像与长文本
MATE: Meet At The Embedding -- Connecting Images with Long Texts
AI总结：
针对现有视觉语言模型难以处理长文本对齐的问题，本文提出MATE方法，通过替换VLM文本编码器、加入多阶段训练的投影模块，实现图像与长文本的有效连接，并构建新基准验证了其性能。
摘要：
尽管视觉语言模型（Vision Language Models, VLMs）的发展已显著提升了视觉与文本数据的对齐效果，但这类模型主要聚焦于图像与简短描述性字幕的对齐。这一关注点限制了它们处理复杂文本交互的能力，尤其是针对长字幕或文档等较长文本的场景，而这类场景尚未得到广泛探索。本文提出了Meet At The Embedding（MATE），这是一种将VLMs与大语言模型（Large Language Models, LLMs）的能力相结合的新方法，无需额外的图像-长文本对即可攻克上述挑战。具体而言，我们用擅长理解长文本的预训练LLM编码器替换了VLM的文本编码器。为弥合VLM与LLM之间的差距，MATE引入了一个采用多阶段方式训练的投影模块。该模块首先利用大量文本对，将VLM文本编码器的嵌入与LLM的嵌入进行对齐，随后被用于将图像嵌入与LLM嵌入进行紧密且无缝的对齐。我们提出了两个新的跨模态检索基准，用于评估图像与长文本（长字幕/文档）的连接任务。大量实验结果表明，MATE能够有效连接图像与长文本，挖掘出多样的语义关系。
英文摘要：
While advancements in Vision Language Models (VLMs) have significantly improved the alignment of visual and textual data, these models primarily focus on aligning images with short descriptive captions. This focus limits their ability to handle complex text interactions, particularly with longer texts such as lengthy captions or documents, which have not been extensively explored yet. In this paper, we introduce Meet At The Embedding (MATE), a novel approach that combines the capabilities of VLMs with Large Language Models (LLMs) to overcome this challenge without the need for additional image-long text pairs. Specifically, we replace the text encoder of the VLM with a pretrained LLM-based encoder that excels in understanding long texts. To bridge the gap between VLM and LLM, MATE incorporates a projection module that is trained in a multi-stage manner. It starts by aligning the embeddings from the VLM text encoder with those from the LLM using extensive text pairs. This module is then employed to seamlessly align image embeddings closely with LLM embeddings. We propose two new cross-modal retrieval benchmarks to assess the task of connecting images with long texts (lengthy captions / documents). Extensive experimental results demonstrate that MATE effectively connects images with long texts, uncovering diverse semantic relationships.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
