多模态嵌入模型在实现语义相似度、信息检索和跨模态聚类等各种下游任务中一直至关重要。然而,现有的多模态嵌入模型(如 VLM2Vec、E5-V、GME)主要聚焦于自然图像,对视频和视觉文档等其他视觉形式的支持有限。这限制了它们在现实场景中的适用性,包括 AI 智能体、多模态搜索与推荐以及检索增强生成(RAG)。为弥合这一差距,我们提出了 VLM2Vec-V2,一个用于学习跨多种视觉形式嵌入的统一框架。首先,我们引入了 MMEB-V2,这是一个综合性基准,在 MMEB 的基础上扩展了五种新的任务类型:视觉文档检索、视频检索、时间定位、视频分类和视频问答——涵盖文本、图像、视频和视觉文档输入。其次,我们训练了 VLM2Vec-V2,这是一个支持文本、图像、视频和视觉文档输入的通用嵌入模型。大量实验表明,VLM2Vec-V2 不仅在新引入的视频和文档检索任务上取得了强劲性能,而且在原始图像基准上相比先前基线也有所提升。通过广泛评估,我们的研究为各种多模态嵌入模型的泛化能力提供了见解,并突出了统一嵌入学习的有效策略,为研究和现实场景中更具可扩展性和适应性的表示学习奠定了基础。
英文摘要:
Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Alibaba Group, Hangzhou, China(阿里巴巴集团)
Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Alibaba Cloud Computing(阿里云计算);Hong Kong University of Science and Technology(香港科技大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
Key Laboratory of Target Cognition and Application Technology (TCAT), AIRI, CAS(目标认知与应用技术重点实验室(TCAT),空气动力研究所,中国科学院);School of Electronic, Electrical and Communication Engineering, UCAS(电子、电气与通信工程学院,中国科学院大学);Aerospace Information Research Institute, Chinese Academy of Sciences(航天信息研究所,中国科学院);Baidu Inc.(百度公司)
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
School of Mechanical, Aerospace, and Manufacturing Engineering, University of Connecticut, Storrs, CT 06269(机械、航空航天与制造工程学院,康涅狄格大学,斯托尔斯,CT 06269);Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA(机械工程系,麻省理工学院,剑桥,MA 02139,美国)
Docopilot: Improving Multimodal Models for Document-Level Understanding
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学);Nankai University(南开大学);Fudan University(复旦大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院)
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Archimedes, Athena Research Center(阿基米德研究中心);National Technical University of Athens(希腊国家技术大学);University of Crete(克里特大学);IACM-Forth(第四研究机构(IACM-Forth))
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Zhongke Fanyu Technology Co., Ltd(中科泛语科技有限公司)
Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
Zhejiang University(浙江大学);Fudan University(复旦大学);Soochow University(苏州大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)