基于因果关系干预的去噪视频矩检索
Deconfounded Video Moment Retrieval with Causal Intervention
摘要：
我们处理视频时刻检索（VMR）的任务，其目的是本地化的一个特定的时刻，在视频中根据文本查询。现有的方法主要是通过复杂的跨模态交互来建模查询和矩之间的匹配关系。尽管它们的有效性，目前的模型大多利用数据集的偏见，而忽略了视频内容，从而导致较差的泛化能力。我们认为这个问题是由VMR中隐藏的混杂因素引起的，{即，时刻的时间位置}，其虚假地使模型输入和预测相关。如何针对时间位置偏差设计鲁棒的匹配模型是至关重要的，但据我们所知，还没有研究VMR。为了填补这一研究空白，我们提出了一个基于因果关系的VMR框架，该框架构建了结构因果模型来捕捉查询和视频内容对预测的真实影响。具体来说，我们开发了一个去混淆交叉模态匹配（DCM）的方法来消除混淆的时刻位置的影响。该方法首先对矩表示进行分解，以推断视觉内容的核心特征，然后基于后门调整对分解后的多模态输入进行因果干预，迫使模型公平地考虑目标的每个可能位置。大量的实验清楚地表明，我们的方法可以实现显着改善国家的最先进的方法在准确性和泛化（代码：\color{blue}{\url{https://github.com/Xun-Yang/Causal_Video_Moment_Retrieval}}
英文摘要：
We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. Despite their effectiveness, current models mostly exploit dataset biases while ignoring the video content, thus leading to poor generalizability. We argue that the issue is caused by the hidden confounder in VMR, {i.e., temporal location of moments}, that spuriously correlates the model input and prediction. How to design robust matching models against the temporal location biases is crucial but, as far as we know, has not been studied yet for VMR. To fill the research gap, we propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction. Specifically, we develop a Deconfounded Cross-modal Matching (DCM) method to remove the confounding effects of moment location. It first disentangles moment representation to infer the core feature of visual content, and then applies causal intervention on the disentangled multimodal input based on backdoor adjustment, which forces the model to fairly incorporate each possible location of the target into consideration. Extensive experiments clearly show that our approach can achieve significant improvement over the state-of-the-art methods in terms of both accuracy and generalization (Codes: \color{blue}{\url{https://github.com/Xun-Yang/Causal_Video_Moment_Retrieval}}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
