学习视觉数据的可解释表示是一项重要挑战,其目的是让机器的决策能够被人类理解,并提升训练分布之外的泛化能力。为此,我们提出了一个深度学习框架,可在其中为视频指定非线性先验(例如牛顿物理学先验),使模型能够学习可解释的潜变量,并利用这些变量生成训练时未观察到的假设场景视频。我们的做法是将 Variational Auto-Encoder(VAE,变分自编码器)的先验从简单的各向同性高斯分布扩展为任意的非线性时间 Additive Noise Model(ANM,加性噪声模型),该模型能够描述大量过程(例如牛顿物理学)。我们提出了一种新颖的线性化方法,通过构建 Gaussian Mixture Model(GMM,高斯混合模型)来近似该先验,并推导了后验 GMM 与先验 GMM 之间 KL 散度的数值稳定 Monte Carlo 估计。我们在不同的真实物理视频上验证了该方法,包括单摆、弹簧上的物块、下落物体以及脉冲星(旋转中子星)。我们为每个实验指定物理先验,并表明模型学到了正确的变量。模型训练完成后,我们对其进行干预以改变不同物理变量(例如振荡振幅或加入空气阻力),从而生成此前未观察到的、物理上正确的假设场景视频。
英文摘要:
Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use these to generate videos of hypothetical scenarios not observed at training time. We do this by extending the Variational Auto-Encoder (VAE) prior from a simple isotropic Gaussian to an arbitrary nonlinear temporal Additive Noise Model (ANM), which can describe a large number of processes (e.g. Newtonian physics). We propose a novel linearization method that constructs a Gaussian Mixture Model (GMM) approximating the prior, and derive a numerically stable Monte Carlo estimate of the KL divergence between the posterior and prior GMMs. We validate the method on different real-world physics videos including a pendulum, a mass on a spring, a falling object and a pulsar (rotating neutron star). We specify a physical prior for each experiment and show that the correct variables are learned. Once a model is trained, we intervene on it to change different physical variables (such as oscillation amplitude or adding air drag) to generate physically correct videos of hypothetical scenarios that were not observed previously.
Training-Free and Interpretable Hateful Video Detection via Multi-stage Adversarial Reasoning
Multimodal Intelligence Lab, Department of Computer Science, University of Exeter, United Kingdom(埃克塞特大学计算机科学系多模态智能实验室);Institute for Analytics and Data Science, University of Essex, United Kingdom(埃塞克斯大学分析与数据科学研究所)
Automated Interpretable 2D Video Extraction from 3D Echocardiography
University of California, Los Angeles(加州大学洛杉矶分校);Kaiser Permanente Division of Research(凯撒医疗集团研究部);Cedars-Sinai Medical Center(西达赛西医学中心);Stanford University(斯坦福大学)
Learning Visually Interpretable Oscillator Networks for Soft Continuum Robots from Video
Department of Advanced Interdisciplinary Studies, The University of Tokyo(东京大学先进跨学科研究系);Institute of Assembly Technology and Robotics, Leibniz University Hannover(莱比锡大学汉诺威装配技术与机器人研究所);Research Center for Advanced Science and Technology, The University of Tokyo(东京大学先进科学研究中心)
All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streams
Chair of Process and Data Science, RWTH Aachen University(过程与数据科学教授席位,亚琛工业大学);Artificial Intelligence & Machine Learning Lab, Technical University of Darmstadt(人工智能与机器学习实验室,达姆施塔特技术大学)
An interpretable approach to automating the assessment of biofouling in video footage
Centre of Excellence for Biosecurity Risk Analysis, The University of Melbourne(生物安全风险分析卓越中心,墨尔本大学);Melbourne Centre for Data Science, The University of Melbourne(墨尔本数据科学中心,墨尔本大学);Biosecurity Animal Division, Department of Agriculture, Fisheries and Forestry(农业、渔业和林业部生物安全动物司)
Low-Rank-Modulated Functa: Exploring the Latent Space of Implicit Neural Representations for Interpretable Ultrasound Video Analysis
Yale Biomedical Imaging Institute, Yale University(耶鲁大学生物医学成像研究所);Department of Emergency Medicine, Yale School of Medicine(耶鲁大学医学院急诊医学系);Dept. of Biomedical Engineering, University of Basel(巴塞尔大学生物医学工程系)
VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks
Autonomous Intelligent Systems group, University of Bonn, Germany(bonn大学自主智能系统小组);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所);Center for Robotics, University of Bonn, Germany(bonn大学机器人中心)
Interpretable and Granular Video-Based Quantification of Motor Characteristics from the Finger Tapping Test in Parkinson Disease
Department of Neurology, Center of Expertise for Parkinson and Movement Disorders, Donders Institute for Brain, Cognition and Behaviour, Radboud University Medical Center(神经科、帕金森病和运动障碍专家中心、大脑、认知与行为研究所、拉德堡德大学医学中心);Department of Artificial Intelligence, Donders Institute for Brain, Cognition and Behaviour, Radboud University(人工智能系、大脑、认知与行为研究所、拉德堡德大学)
Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding
Harvard University(哈佛大学);University of Cambridge(剑桥大学);Northwestern University(西北大学);Cold Spring Harbor Laboratory(冷泉港实验室);Columbia University(哥伦比亚大学);University of Science and Technology of China(中国科学技术大学)
Automatic Estimation of Self-Reported Pain by Interpretable Representations of Motion Dynamics
Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Centrale Lille(里尔中央理工学院);IMT Lille Douai(里尔杜埃国立高等矿业电信学院);University of Florence(佛罗伦萨大学);Carnegie Mellon University(卡内基梅隆大学)
Generative Neural Video Compression via Video Diffusion Prior
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,通信大学);School of Computer Science, Peking University(计算机学院,北京大学)