面向物理视听常识推理的鲁棒解耦反事实学习
Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning
AI总结：
针对物理视听常识推理中的模态缺失与因果推理不足问题，提出可即插即用的 RDCL 框架，通过解耦表征、反事实干预和缺失模态恢复提升基线性能。
摘要：
本文提出了一种新的鲁棒解耦反事实学习（Robust Disentangled Counterfactual Learning，RDCL）方法，用于物理视听常识推理。该任务旨在基于视频和音频输入推断物体的物理常识，主要挑战在于如何模仿人类的推理能力，即使在模态缺失的场景下也是如此。当前大多数方法未能充分利用多模态数据中的不同特征，而且模型缺乏因果推理能力，阻碍了隐式物理知识推断的进展。为解决这些问题，我们提出的 RDCL 方法通过解耦序列编码器在潜在空间中将视频解耦为静态（时不变）因素和动态（时变）因素；该编码器采用变分自编码器（VAE），并借助对比损失函数最大化互信息。此外，我们引入反事实学习模块，通过在反事实干预下建模不同物体之间的物理知识关系，增强模型的推理能力。为缓解模态数据不完整的问题，我们引入一种鲁棒的多模态学习方法，通过分解共享特征和模型专属特征来恢复缺失数据。我们提出的方法是一个即插即用模块，可集成到包括 VLM 在内的任何基线中。实验表明，我们提出的方法提高了基线方法的推理准确性和鲁棒性，并取得了最先进的性能。
英文摘要：
In this paper, we propose a new Robust Disentangled Counterfactual Learning (RDCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge being how to imitate the reasoning ability of humans, even under the scenario of missing modalities. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed RDCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. To alleviate the incomplete modality data issue, we introduce a robust multimodal learning method to recover the missing data by decomposing the shared features and model-specific features. Our proposed method is a plug-and-play module that can be incorporated into any baseline including VLMs. In experiments, we show that our proposed method improves the reasoning accuracy and robustness of baseline methods and achieves the state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
