arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.18539cs.CVcs.LG
收藏

使用非线性先验从视频中学习可解释表示

Interpretable Representation Learning from Videos using Nonlinear Priors

Visual Geometry Group(视觉几何组);University of Oxford(牛津大学)

Marian Longa, João F. Henriques

AI总结:

本文提出将 VAE 先验扩展为非线性时间 ANM,并用 GMM 线性化与稳定 KL 估计,从物理视频中学习可解释潜变量及生成反事实场景。

摘要:

学习视觉数据的可解释表示是一项重要挑战,其目的是让机器的决策能够被人类理解,并提升训练分布之外的泛化能力。为此,我们提出了一个深度学习框架,可在其中为视频指定非线性先验(例如牛顿物理学先验),使模型能够学习可解释的潜变量,并利用这些变量生成训练时未观察到的假设场景视频。我们的做法是将 Variational Auto-Encoder(VAE,变分自编码器)的先验从简单的各向同性高斯分布扩展为任意的非线性时间 Additive Noise Model(ANM,加性噪声模型),该模型能够描述大量过程(例如牛顿物理学)。我们提出了一种新颖的线性化方法,通过构建 Gaussian Mixture Model(GMM,高斯混合模型)来近似该先验,并推导了后验 GMM 与先验 GMM 之间 KL 散度的数值稳定 Monte Carlo 估计。我们在不同的真实物理视频上验证了该方法,包括单摆、弹簧上的物块、下落物体以及脉冲星(旋转中子星)。我们为每个实验指定物理先验,并表明模型学到了正确的变量。模型训练完成后,我们对其进行干预以改变不同物理变量(例如振荡振幅或加入空气阻力),从而生成此前未观察到的、物理上正确的假设场景视频。

英文摘要:

Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use these to generate videos of hypothetical scenarios not observed at training time. We do this by extending the Variational Auto-Encoder (VAE) prior from a simple isotropic Gaussian to an arbitrary nonlinear temporal Additive Noise Model (ANM), which can describe a large number of processes (e.g. Newtonian physics). We propose a novel linearization method that constructs a Gaussian Mixture Model (GMM) approximating the prior, and derive a numerically stable Monte Carlo estimate of the KL divergence between the posterior and prior GMMs. We validate the method on different real-world physics videos including a pendulum, a mass on a spring, a falling object and a pulsar (rotating neutron star). We specify a physical prior for each experiment and show that the correct variables are learned. Once a model is trained, we intervene on it to change different physical variables (such as oscillation amplitude or adding air drag) to generate physically correct videos of hypothetical scenarios that were not observed previously.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑