面向视频问题定位的跨模态因果关联对齐
Cross-modal Causal Relation Alignment for Video Question Grounding
AI总结：
本文提出跨模态因果关联对齐框架CRA，通过高斯平滑定位、跨模态对齐和显式因果干预消除虚假相关性，提升视频问题定位的因果一致性与鲁棒性。
摘要：
视频问题定位（VideoQG）要求模型在回答问题的同时推断支持答案的相关视频片段。然而，现有VideoQG方法通常受到虚假跨模态相关性的影响，导致无法识别与目标问题对齐的主导视觉场景。此外，视觉-语言模型在VideoQG等具有挑战性的下游任务上表现出不忠实的泛化性能，并且缺乏鲁棒性。在本工作中，我们提出了一种新颖的VideoQG框架，称为跨模态因果关联对齐（CRA），以消除虚假相关性并提升问答与视频时间定位之间的因果一致性。我们的CRA包含三个核心组件：i）高斯平滑定位（GSG）模块，通过跨模态注意力估计时间区间，并由自适应高斯滤波器进行去噪；ii）跨模态对齐（CMA）模块，利用估计视频片段与问答特征之间的双向对比学习来增强弱监督VideoQG的性能；iii）显式因果干预（ECI）模块，用于多模态去混杂，其中包括针对视觉的前门干预和针对语言的后门干预。在两个VideoQG数据集上的大量实验表明，我们的CRA在发现视觉支撑内容以及实现鲁棒问题推理方面具有优越性。代码可在https://github.com/WissingChen/CRA-GQA获取。
英文摘要：
Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers. However, existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question. Moreover, vision-language models exhibit unfaithful generalization performance and lack robustness on challenging downstream tasks such as VideoQG. In this work, we propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding. Our CRA involves three essential components: i) Gaussian Smoothing Grounding (GSG) module for estimating the time interval via cross-modal attention, which is de-noised by an adaptive Gaussian filter, ii) Cross-Modal Alignment (CMA) enhances the performance of weakly supervised VideoQG by leveraging bidirectional contrastive learning between estimated video segments and QA features, iii) Explicit Causal Intervention (ECI) module for multimodal deconfounding, which involves front-door intervention for vision and back-door intervention for language. Extensive experiments on two VideoQG datasets demonstrate the superiority of our CRA in discovering visually grounded content and achieving robust question reasoning. Codes are available at https://github.com/WissingChen/CRA-GQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
