arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01341cs.CL
收藏

AlignVLM:面向多模态文档理解的视觉与语言潜在空间桥接

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding

ServiceNow;York University(约克大学);Mila – Quebec AI Institute(魁北克人工智能研究院);École de Technologie Supérieure(魁北克高等技术学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);University of Waterloo(滑铁卢大学);CIFAR AI Chair(CIFAR人工智能 chair);Polytechnique Montréal(蒙特利尔理工学院);University of British Columbia(不列颠哥伦比亚大学)
Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathw… 展开作者

Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathwik Tejaswi Madhusudhan, Marco Pedersoli, Bang Liu, Nicolas Chapados, Yoshua Bengio, Enamul Hoque, Christopher Pal, Issam H. Laradji, David Vazquez, Perouz Taslakian, Spandana Gella, Sai Rajeswar

AI总结:

针对视觉语言模型中现有连接器缺乏归纳偏置导致跨模态错位的问题,提出AlignVLM方法,通过将视觉特征映射为LLM文本嵌入的加权平均以利用语言先验,在文档理解任务和低资源场景下取得最优性能及鲁棒表现。

摘要:

将视觉特征与语言嵌入对齐是视觉语言模型(VLMs)的一项关键挑战。此类模型的性能取决于是否有一个良好的连接器,该连接器将视觉编码器生成的视觉特征映射到与LLM共享的嵌入空间中,同时保持语义相似性。现有的连接器,如多层感知机(MLPs),缺乏归纳偏置来将视觉特征约束在LLM嵌入空间的语言结构内,这使得它们极其消耗数据且容易发生跨模态错位。在这项工作中,我们提出了一种新颖的视觉-文本对齐方法,AlignVLM,它将视觉特征映射到LLM文本嵌入的加权平均值。我们的方法利用LLM编码的语言先验,确保视觉特征被映射到LLM能够有效解释的空间区域。AlignVLM对于文档理解任务特别有效,在这些任务中视觉和文本模态高度相关。我们的大量实验表明,与先前的对齐方法相比,AlignVLM实现了最先进的性能,在文档理解任务和低资源设置下取得了更大的收益。我们提供了进一步的分析,证明了其效率和对噪声的鲁棒性。

英文摘要:

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM's embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding tasks and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑