MECD+:解锁视频推理中的事件级因果图发现
MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning
AI总结:
本文提出MECD任务与数据集,并设计基于事件格兰杰检验及因果推断的框架,用于发现长视频中多事件间的因果图,性能优于GPT-4o和VideoChat2。
摘要:
视频因果推理旨在从因果视角实现对视频的高层次理解。然而,其范围存在局限性,主要在问答范式中执行,并聚焦于包含孤立事件和基本因果关系的简短视频片段,缺乏对具有多个相互关联事件的视频进行全面、结构化的因果分析。为填补这一空白,我们引入了一个新任务和数据集,即多事件因果发现(MECD)。该任务旨在揭示长视频中按时间顺序分布的事件之间的因果关系。给定视觉片段和事件的文本描述,MECD识别这些事件之间的因果关联,以推导出全面且结构化的事件级视频因果图,解释结果事件发生的原因和方式。为应对MECD的挑战,我们设计了一种受格兰杰因果方法启发的新框架,结合高效的基于掩码的事件预测模型来执行事件格兰杰检验。该方法通过比较前提事件被掩蔽与未被掩蔽时预测的结果事件来估计因果关系。此外,我们整合了因果推断技术,如前门调整和反事实推断,以缓解MECD中因果混杂和虚假因果等挑战。同时,引入上下文链推理以进行更稳健和泛化的推理。实验验证了我们的框架在推理完整因果关系方面的有效性,分别比GPT-4o和VideoChat2高出5.77%和2.70%。进一步实验表明,因果关系图也能有助于下游视频理解任务,如视频问答和视频事件预测。
英文摘要:
Video causal reasoning aims to achieve a high-level understanding of videos from a causal perspective. However, it exhibits limitations in its scope, primarily executed in a question-answering paradigm and focusing on brief video segments containing isolated events and basic causal relations, lacking comprehensive and structured causality analysis for videos with multiple interconnected events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relations between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD identifies the causal associations between these events to derive a comprehensive and structured event-level video causal graph explaining why and how the result event occurred. To address the challenges of MECD, we devise a novel framework inspired by the Granger Causality method, incorporating an efficient mask-based event prediction model to perform an Event Granger Test. It estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to mitigate challenges in MECD like causality confounding and illusory causality. Additionally, context chain reasoning is introduced to conduct more robust and generalized reasoning. Experiments validate the effectiveness of our framework in reasoning complete causal relations, outperforming GPT-4o and VideoChat2 by 5.77% and 2.70%, respectively. Further experiments demonstrate that causal relation graphs can also contribute to downstream video understanding tasks such as video question answering and video event prediction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。