从视觉到词汇：通过多模态大语言模型的自回归预训练建立图像与文本标记的等价性
From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs
AI总结：
针对MLLMs缺乏精确多模态对齐的问题，提出VDEP混合自回归预训练范式，利用视觉编码器后MLP的动态嵌入监督图像隐藏状态并整合图像标记，在13个基准上超越现有方法，提升多模态对齐性能。
摘要：
尽管多模态大语言模型（MLLMs）在感知任务上表现良好，但它们缺乏精确的多模态对齐，限制了性能。为应对这一挑战，我们提出视觉动态嵌入引导预训练（VDEP），一种用于MLLMs的混合自回归训练范式。该方法利用视觉编码器后MLP的动态嵌入，监督图像隐藏状态并将图像标记整合到自回归训练中。现有MLLMs主要专注于从文本输入中恢复信息，常忽视图像数据的有效处理。相比之下，本工作的关键改进是将多模态对齐重新解释为从输入数据中恢复信息的过程，特别强调重建详细的视觉特征。所提方法无需架构更改即可无缝集成到标准模型中。在13个基准上的实验表明，VDEP优于基线，超越现有方法。
英文摘要：
While MLLMs perform well on perceptual tasks, they lack precise multimodal alignment, limiting performance. To address this challenge, we propose Vision Dynamic Embedding-Guided Pretraining (VDEP), a hybrid autoregressive training paradigm for MLLMs. Utilizing dynamic embeddings from the MLP following the visual encoder, this approach supervises image hidden states and integrates image tokens into autoregressive training. Existing MLLMs primarily focused on recovering information from textual inputs, often neglecting the effective processing of image data. In contrast, the key improvement of this work is the reinterpretation of multimodal alignment as a process of recovering information from input data, with particular emphasis on reconstructing detailed visual features.The proposed method seamlessly integrates into standard models without architectural changes. Experiments on 13 benchmarks show VDEP outperforms baselines, surpassing existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
