反事实跨模态推理用于弱监督视频时刻定位
Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization
AI总结：
针对弱监督视频时刻定位中视频与语言不匹配问题，提出反事实跨模态推理方法，显式建模并抑制虚假相关性，提升对比学习效果。
摘要：
视频时刻定位旨在根据自然语言查询检索未修剪视频中的目标片段。由于目标片段的精确时间位置并不总是可得，弱监督方法近年来受到关注。然而，弱监督方法面临的最大挑战之一隐含在由粗略时间标注引起的视频与语言之间的不匹配中。为了细化视觉-语言对齐，近期工作通过重建正负视频提案之间的掩码查询来对比跨模态相似性。然而，重建可能受到未掩码与掩码部分之间潜在虚假相关性的影响，这会扭曲恢复过程并进一步降低对比学习的有效性，因为掩码词并未完全从跨模态知识中重建。在本文中，我们通过一种新颖提出的反事实跨模态推理方法发现并缓解了这种虚假相关性。具体而言，我们首先将查询重建表述为跨模态与查询知识的聚合因果效应。然后通过将反事实跨模态知识引入该聚合，显式建模了未掩码部分对重建贡献的虚假影响。最后，通过抑制掩码查询的单模态效应，我们可以纠正视频提案的重建以进行合理的对比学习。大量实验评估证明了我们提出方法的有效性。代码可在https://github.com/sLdZ0306/CCR获取。
英文摘要：
Video moment localization aims to retrieve the target segment of an untrimmed video according to the natural language query. Weakly supervised methods gains attention recently, as the precise temporal location of the target segment is not always available. However, one of the greatest challenges encountered by the weakly supervised method is implied in the mismatch between the video and language induced by the coarse temporal annotations. To refine the vision-language alignment, recent works contrast the cross-modality similarities driven by reconstructing masked queries between positive and negative video proposals. However, the reconstruction may be influenced by the latent spurious correlation between the unmasked and the masked parts, which distorts the restoring process and further degrades the efficacy of contrastive learning since the masked words are not completely reconstructed from the cross-modality knowledge. In this paper, we discover and mitigate this spurious correlation through a novel proposed counterfactual cross-modality reasoning method. Specifically, we first formulate query reconstruction as an aggregated causal effect of cross-modality and query knowledge. Then by introducing counterfactual cross-modality knowledge into this aggregation, the spurious impact of the unmasked part contributing to the reconstruction is explicitly modeled. Finally, by suppressing the unimodal effect of masked query, we can rectify the reconstructions of video proposals to perform reasonable contrastive learning. Extensive experimental evaluations demonstrate the effectiveness of our proposed method. The code is available at \href{https://github.com/sLdZ0306/CCR}{https://github.com/sLdZ0306/CCR}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
