MARS：多模态大语言模型中隐藏着哪些用于文本-视频检索的检索信号？
MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?
浏览论文内容
中文总结 AI 辅助
针对多模态大语言模型用于文本-视频检索时压缩线索导致细粒度检索受限的问题，提出MARS框架，通过多层多槽嵌入及难负样本感知的槽专业化目标，在四个基准上取得最优结果。
中文摘要 AI 辅助
文本-视频检索需要能够区分具有相似场景、动作和时间模式的视频的表征。最近，多模态大语言模型已被适配为嵌入模型，但它们通常使用最终层的单个 token 表示每个输入，这会将多样的视频-文本线索压缩为单个向量，限制了细粒度检索。为解决这一局限，我们提出 MARS，一种用于文本-视频检索的多层多槽嵌入框架。MARS 通过结合不同解码器层的隐藏状态构建多个自适应表征槽，比较对应的文本和视频槽，并聚合它们的相似度以用于检索。为更好地处理混淆候选，我们进一步引入了难负样本感知的槽专业化目标，该目标鼓励槽捕获具有区分性的匹配线索。在四个文本-视频检索基准上的实验表明，MARS 在基于直接相似度的检索和重排序设置中均取得了最先进的结果。消融研究和分析表明，多层融合、多槽以及难负样本感知的槽专业化提供了互补的增益。代码可在此 https URL 获取。
英文摘要
Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.
发表机构
- Sejong University（世宗大学）
机构由 AI 辅助整理，请以论文原文为准。
