用于物理视听常识推理的解耦反事实学习
Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
AI总结：
针对物理视听常识推理中多模态特征利用不足和因果推理缺失的问题，提出即插即用的DCL，通过解耦视频静动态因素并结合反事实干预建模，显著提升基线并达到SOTA。
摘要：
本文提出一种用于物理视听常识推理的解耦反事实学习（Disentangled Counterfactual Learning，DCL）方法。该任务旨在基于视频和音频输入推断物体的物理常识，主要挑战在于如何模仿人类的推理能力。当前大多数方法未能充分利用多模态数据中的不同特征，而且模型缺乏因果推理能力，阻碍了隐式物理知识推断的进展。为解决这些问题，我们提出的DCL方法通过解耦序列编码器，在潜在空间中将视频解耦为静态（时不变）因素和动态（时变）因素；该编码器采用变分自编码器（VAE），并借助对比损失函数最大化互信息。此外，我们引入反事实学习模块，通过在反事实干预下建模不同物体之间的物理知识关系，增强模型的推理能力。所提方法是一个即插即用模块，可集成到任意基线方法中。实验表明，我们提出的方法改进了基线方法，并取得了最先进性能。源代码可在 https://github.com/Andy20178/DCL 获取。
英文摘要：
In this paper, we propose a Disentangled Counterfactual Learning~(DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed DCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. Our proposed method is a plug-and-play module that can be incorporated into any baseline. In experiments, we show that our proposed method improves baseline methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Andy20178/DCL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
