arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.20190cs.CV
收藏

CoVR-R: 推理感知的组合视频检索

CoVR-R:Reason-Aware Composed Video Retrieval

Mohamed bin Zayed University of AI(莫扎德·本·扎耶德人工智能大学);University of Chicago(芝加哥大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Linköping University(林奈大学)

Omkar Thawakar, Dmitry Demidov, Vaishnav Potlapalli, Sai Prasanna Teja Reddy Bogireddy, Viswanatha Reddy Gajjala, Alaa Mostafa Lasheen, Rao Muhammad Anwer, Fahad Khan

AI总结:

提出一种零样本推理优先方法,利用大型多模态模型推断编辑的因果和时序后效,并构建CoVR-Reason基准评估,在隐式效应子集上显著优于强基线。

摘要:

组合视频检索(CoVR)旨在根据参考视频和文本修改找到目标视频。先前的工作假设修改文本完全指定了视觉变化,忽略了编辑产生的后效和隐含后果(例如,运动、状态转换、视角或持续时间线索)。我们认为成功的CoVR需要对这些后效进行推理。我们提出了一种推理优先的零样本方法,利用大型多模态模型(i)推断编辑所隐含的因果和时序后果,以及(ii)将得到的推理查询与候选视频对齐,无需任务特定的微调。为了评估CoVR中的推理能力,我们还提出了CoVR-Reason基准,该基准将每个(参考、编辑、目标)三元组与结构化的内部推理轨迹和具有挑战性的干扰项配对,这些干扰项需要预测后效而不是关键词匹配。实验表明,我们的零样本方法在召回率@K上优于强检索基线,并且在隐式效应子集上尤其出色。我们的自动和人工分析证实了检索结果中更高的步骤一致性和效果真实性。我们的发现表明,将推理纳入通用多模态模型可以通过明确考虑因果和时序后效来实现有效的CoVR。这减少了对任务特定监督的依赖,提高了对具有挑战性的隐式效应案例的泛化能力,并增强了检索结果的可解释性。这些结果指向了一个可扩展且原则性的可解释视频搜索框架。模型、代码和基准可在该网址获取。

英文摘要:

Composed Video Retrieval (CoVR) aims to find a target video given a reference video and a textual modification. Prior work assumes the modification text fully specifies the visual changes, overlooking after-effects and implicit consequences (e.g., motion, state transitions, viewpoint or duration cues) that emerge from the edit. We argue that successful CoVR requires reasoning about these after-effects. We introduce a reasoning-first, zero-shot approach that leverages large multimodal models to (i) infer causal and temporal consequences implied by the edit, and (ii) align the resulting reasoned queries to candidate videos without task-specific finetuning. To evaluate reasoning in CoVR, we also propose CoVR-Reason, a benchmark that pairs each (reference, edit, target) triplet with structured internal reasoning traces and challenging distractors that require predicting after-effects rather than keyword matching. Experiments show that our zero-shot method outperforms strong retrieval baselines on recall at K and particularly excels on implicit-effect subsets. Our automatic and human analysis confirm higher step consistency and effect factuality in our retrieved results. Our findings show that incorporating reasoning into general-purpose multimodal models enables effective CoVR by explicitly accounting for causal and temporal after-effects. This reduces dependence on task-specific supervision, improves generalization to challenging implicit-effect cases, and enhances interpretability of retrieval outcomes. These results point toward a scalable and principled framework for explainable video search. The model, code, and benchmark are available at https://github.com/mbzuai-oryx/CoVR-R.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑