arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.08099cs.CVcs.AI

VidVec:解锁视频MLLM嵌入用于视频-文本检索

VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval

Issar Tzachor, Dvir Samuel, Rami Ben-Ari

AI总结:

VidVec通过利用预训练MLLM的中间层嵌入和校准头部,实现无需训练的视频-文本检索,超越现有方法,达到最佳性能。

摘要:

近期研究将生成式多模态大语言模型(MLLMs)适应为视觉任务的嵌入提取器,通常通过微调生成通用表示。然而,其在视频上的性能仍劣于视频基础模型(VFMs)。本文聚焦于利用MLLMs进行视频-文本嵌入和检索。我们首先进行系统性的逐层分析,显示中间(预训练)MLLM层已编码大量任务相关的信息。基于这一见解,我们证明结合中间层嵌入与校准的MLLM头可以无需任何训练就实现强大的零样本检索性能。在此基础上,我们引入了一种轻量级的基于文本的对齐策略,将密集视频字幕映射到短摘要,并启用任务相关的视频-文本嵌入学习,而无需视觉监督。值得注意的是,在无需微调文本之外,我们的方法在当前方法上表现优异,通常以显著优势,在常见的视频检索基准上取得最佳结果。

英文摘要:

Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑