通过因果推理增强时序句子定位
Boosting Temporal Sentence Grounding via Causal Inference
AI总结：
针对时序句子定位任务中视频与文本查询间虚假相关性导致预测不可靠、泛化差的问题，提出因果干预与反事实推理框架，通过结构因果建模、文本因果干预及视觉反事实推理消除偏差，实验验证其优越性。
摘要：
时序句子定位（TSG）旨在识别未剪辑视频中与给定文本查询在语义上对应的相关时刻。尽管现有研究已取得重大进展，但它们往往忽视视频与文本查询之间的虚假相关性问题。这些虚假相关性源于两个主要因素：（1）文本数据中的固有偏差，例如特定动词或短语的频繁共现；（2）模型过度拟合视频内容中显著或重复模式的倾向。此类偏差会误导模型将文本线索与错误的视觉时刻相关联，导致不可靠的预测和对分布外示例的泛化能力差。为了克服这些限制，我们提出了一种新颖的TSG框架——因果干预与反事实推理（causal intervention and counterfactual reasoning），该框架利用因果推理消除虚假相关性并增强模型的鲁棒性。具体而言，我们首先从因果角度用结构因果模型（structural causal model）构建TSG任务。然后，为了解决反映文本对特定动词或短语存在偏差的未观测混杂因素，我们提出了文本因果干预，利用do-微积分（do-calculus）估计因果效应。此外，我们通过构建仅关注视频特征、排除查询和融合多模态特征的反事实场景来执行视觉反事实推理。这使我们能够通过分离和消除视频对整体效应的影响来为模型去偏。在公共数据集上的实验证明了所提出方法的优越性。代码可在https://github.com/Tangkfan/CICR获取。
英文摘要：
Temporal Sentence Grounding (TSG) aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query. Despite existing studies having made substantial progress, they often overlook the issue of spurious correlations between video and textual queries. These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content. Such biases mislead the model into associating textual cues with incorrect visual moments, resulting in unreliable predictions and poor generalization to out-of-distribution examples. To overcome these limitations, we propose a novel TSG framework, causal intervention and counterfactual reasoning that utilizes causal inference to eliminate spurious correlations and enhance the model's robustness. Specifically, we first formulate the TSG task from a causal perspective with a structural causal model. Then, to address unobserved confounders reflecting textual biases toward specific verbs or phrases, a textual causal intervention is proposed, utilizing do-calculus to estimate the causal effects. Furthermore, visual counterfactual reasoning is performed by constructing a counterfactual scenario that focuses solely on video features, excluding the query and fused multi-modal features. This allows us to debias the model by isolating and removing the influence of the video from the overall effect. Experiments on public datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/Tangkfan/CICR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
