多模态大语言模型的语义对齐
Semantic Alignment for Multimodal Large Language Models
AI总结：
针对多图像跨模态指令中独立提取视觉token导致图像间关联信息丢失的问题，提出SAM模型，通过在视觉token提取过程中引入双向语义引导来对齐多图像语义，并在构建的MmLINK数据集上取得显著性能提升。
摘要：
针对多图像跨模态指令的多模态大语言模型（MLLMs）研究受到了越来越多的关注并取得了显著进展，特别是在涉及高度相似图像的场景中（例如变化描述）。现有的MLLMs通常在其流程中遵循两步过程：首先，为每个输入图像独立提取视觉token，然后将这些来自不同图像的视觉token在大型语言模型（LLM）的文本特征空间中进行对齐。然而，为每张图像独立提取视觉token可能会导致在第一步中不同图像被赋予不同的语义优先级，从而导致在后续LLM分析中缺乏对图像间关联信息的保留。在图像间存在显著差异的场景中（例如视觉故事生成），这一问题变得更加严重。为了应对这一挑战，我们引入了多模态大语言模型的语义对齐（SAM）。通过在视觉token提取过程中引入不同图像之间的双向语义引导，SAM旨在增强关联信息的保留以进行连贯分析，并在将不同图像的语义输入LLM之前对其进行对齐。作为测试平台，我们提出了一个名为MmLINK的大规模数据集，包含69K个样本。与大多数用于MLLMs微调的现有数据集不同，我们的MmLINK数据集包含具有显著多样性图像的多模态指令。在群组描述任务和故事生成任务上的大量实验证明了我们SAM模型的有效性，以巨大优势超越了现有最先进的方法（在CIDEr分数上，群组描述提升+37%，故事生成提升+22%）。项目页面：https://mccartney01.github.io/SAM。
英文摘要：
Research on Multi-modal Large Language Models (MLLMs) towards the multi-image cross-modal instruction has received increasing attention and made significant progress, particularly in scenarios involving closely resembling images (e.g., change captioning). Existing MLLMs typically follow a two-step process in their pipelines: first, extracting visual tokens independently for each input image, and then aligning these visual tokens from different images with the Large Language Model (LLM) in its textual feature space. However, the independent extraction of visual tokens for each image may result in different semantics being prioritized for different images in the first step, leading to a lack of preservation of linking information among images for subsequent LLM analysis. This issue becomes more serious in scenarios where significant variations exist among the images (e.g., visual storytelling). To address this challenge, we introduce Semantic Alignment for Multi-modal large language models (SAM). By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM. As the test bed, we propose a large-scale dataset named MmLINK consisting of 69K samples. Different from most existing datasets for MLLMs fine-tuning, our MmLINK dataset comprises multi-modal instructions with significantly diverse images. Extensive experiments on the group captioning task and the storytelling task prove the effectiveness of our SAM model, surpassing the state-of-the-art methods by a large margin (+37% for group captioning and +22% for storytelling on CIDEr score). Project page: https://mccartney01.github.io/SAM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
