A central challenge in data-driven model discovery is the presence of hidden, or latent, variables that are not directly measured but are dynamically important. Takens' theorem provides conditions for when it is possible to augment these partial measurements with time delayed information, resulting in an attractor that is diffeomorphic to that of the original full-state system. However, the coordinate transformation back to the original attractor is typically unknown, and learning the dynamics in the embedding space has remained an open challenge for decades. Here, we design a custom deep autoencoder network to learn a coordinate transformation from the delay embedded space into a new space where it is possible to represent the dynamics in a sparse, closed form. We demonstrate this approach on the Lorenz, Rössler, and Lotka-Volterra systems, learning dynamics from a single measurement variable. As a challenging example, we learn a Lorenz analogue from a single scalar variable extracted from a video of a chaotic waterwheel experiment. The resulting modeling framework combines deep learning to uncover effective coordinates and the sparse identification of nonlinear dynamics (SINDy) for interpretable modeling. Thus, we show that it is possible to simultaneously learn a closed-form model and the associated coordinate system for partially observed dynamics.
We use autoencoders to create low-dimensional embeddings of underlying patient phenotypes that we hypothesize are a governing factor in determining how different patients will react to different interventions. We compare the performance of autoencoders that take fixed length sequences of concatenated timesteps as input with a recurrent sequence-to-sequence autoencoder. We evaluate our methods on around 35,500 patients from the latest MIMIC III dataset from Beth Israel Deaconess Hospital.
Recent progress in autoencoder-based sparse identification of nonlinear dynamics (SINDy) under $\ell_1$ constraints allows joint discoveries of governing equations and latent coordinate systems from spatio-temporal data, including simulated video frames. However, it is challenging for $\ell_1$-based sparse inference to perform correct identification for real data due to the noisy measurements and often limited sample sizes. To address the data-driven discovery of physics in the low-data and high-noise regimes, we propose Bayesian SINDy autoencoders, which incorporate a hierarchical Bayesian sp…
University of Twente（特文特大学）；University of Washington（华盛顿大学）
Complex systems manifest a small number of instabilities and bifurcations that are canonical in nature, resulting in universal pattern forming characteristics as a function of some parametric dependence. Such parametric instabilities are mathematically characterized by their universal un-foldings, or normal form dynamics, whereby a parsimonious model can be used to represent the dynamics. Although center manifold theory guarantees the existence of such low-dimensional normal forms, finding them has remained a long standing challenge. In this work, we introduce deep learning autoencoders to dis…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Illinois Institute of Technology（伊利诺伊理工学院）
With the rapid increase of valuable observational, experimental and simulating data for complex systems, great efforts are being devoted to discovering governing laws underlying the evolution of these systems. However, the existing techniques are limited to extract governing laws from data as either deterministic differential equations or stochastic differential equations with Gaussian noise. In the present work, we develop a new data-driven approach to extract stochastic dynamical systems with non-Gaussian symmetric Lévy noise, as well as Gaussian noise. First, we establish a feasible theoret…
Massachusetts Institute of Technology（麻省理工学院）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Identifying the governing equations of a nonlinear dynamical system is key to both understanding the physical features of the system and constructing an accurate model of the dynamics that generalizes well beyond the available data. We propose a machine learning framework for discovering these governing equations using only partial observations, combining an encoder for state reconstruction with a sparse symbolic model. Our tests show that this method can successfully reconstruct the full system state and identify the underlying dynamics for a variety of ODE and PDE systems.
The partial differential equation (PDE) plays a significantly important role in many fields of science and engineering. The conventional case of the derivation of PDE mainly relies on first principles and empirical observation. However, the development of machine learning technology allows us to mine potential control equations from the massive amounts of stored data in a fresh way. Although there has been considerable progress in the data-driven discovery of PDE, the extant literature mostly focuses on the improvements of discovery methods, without substantial breakthroughs in the discovery p…
With the wide adoption of mobile devices, today's location tracking systems such as satellites, cellular base stations and wireless access points are continuously producing tremendous amounts of location data of moving objects. The ability to discover moving objects that travel together, i.e., traveling companions, from their trajectories is desired by many applications such as intelligent transportation systems and location-based services. Existing algorithms are either based on pattern mining methods that define a particular pattern of traveling companions or based on representation learning…
Northeastern University（东北大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Notre Dame（圣母大学）；Renmin University of China（中国人民大学）
Nonlinear dynamics is ubiquitous in nature and commonly seen in various science and engineering disciplines. Distilling analytical expressions that govern nonlinear dynamics from limited data remains vital but challenging. To tackle this fundamental issue, we propose a novel Symbolic Physics Learner (SPL) machine to discover the mathematical structure of nonlinear dynamics. The key concept is to interpret mathematical operations and system state variables by computational rules and symbols, establish symbolic reasoning of mathematical formulas via expression trees, and employ a Monte Carlo tre…
Discovering and decoding latent mean-field structure with variational autoencoders
Department of Physics and the James Franck Institute, University of Chicago（芝加哥大学物理系及詹姆斯·弗兰克研究所）；CuspAI；AMLab, University of Amsterdam（阿姆斯特丹大学AMLab）；Leinweber Institute for Theoretical Physics（莱因韦伯理论物理研究所）
Finite Expression Methods for Discovering Physical Laws from Data
Department of Statistics and Data Science（统计与数据科学系）；Yale University（耶鲁大学）；Department of Mathematics（数学系）；University of Florida（佛罗里达大学）；Department of Computer Science（计算机科学系）；University of Maryland（马里兰大学）；College Park（学院公园）
Helmholtz AI（亥姆霍兹人工智能中心）；Technical University of Munich（慕尼黑工业大学）；Apple（苹果公司）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；DeepMind（深度思维）；OpenAI（开放人工智能）
Natural laws are often described through differential equations yet finding a differential equation that describes the governing law underlying observed data is a challenging and still mostly manual task. In this paper we make a step towards the automation of this process: we propose a transformer-based sequence-to-sequence model that recovers scalar autonomous ordinary differential equations (ODEs) in symbolic form from time-series data of a single observed solution of the ODE. Our method is efficiently scalable: after one-time pretraining on a large set of ODEs, we can infer the governing la…
Discovering Governing Equations in the Presence of Uncertainty
Ridwan Olabiyi School of Computing and Augmented Intelligence, Arizona State University（Ridwan Olabiyi 计算与增强智能学院，亚利桑那州立大学）；Han Hu Department of Mechanical Engineering, University of Arkansas（Han Hu 机械工程系，阿肯色大学）；Ashif Iquebal School of Computing and Augmented Intelligence, Arizona State University（Ashif Iquebal 计算与增强智能学院，亚利桑那州立大学）
A complete understanding of physical systems requires models that are accurate and obeys natural conservation laws. Recent trends in representation learning involve learning Lagrangian from data rather than the direct discovery of governing equations of motion. The generalization of equation discovery techniques has huge potential; however, existing Lagrangian discovery frameworks are black-box in nature. This raises a concern about the reusability of the discovered Lagrangian. In this article, we propose a novel data-driven machine-learning algorithm to automate the discovery of interpretable…
Discovering the governing laws underpinning physical and chemical phenomena is a key step towards understanding and ultimately controlling systems in science and engineering. We introduce Discovery of Dynamical Systems via Moving Horizon Optimization (DySMHO), a scalable machine learning framework for identifying governing laws in the form of differential equations from large-scale noisy experimental data sets. DySMHO consists of a novel moving horizon dynamic optimization strategy that sequentially learns the underlying governing equations from a large dictionary of basis functions. The seque…
Discovering stochastic dynamical equations from biological time series data
Indian Institute of Science（印度科学学院）；University of Konstanz（康斯坦茨大学）；Max Planck Institute of Animal Behaviour（马克斯·普朗克动物行为研究所）；Ahmedabad University（艾哈迈达巴德大学）；University of Basel（巴塞尔大学）；IIT Madras（印度马德拉斯理工学院）
With the advent of modern data collection and storage technologies, data-driven approaches have been developed for discovering the governing partial differential equations (PDE) of physical problems. However, in the extant works the model parameters in the equations are either assumed to be known or have a linear dependency. Therefore, most of the realistic physical processes cannot be identified with the current data-driven PDE discovery approaches. In this study, an innovative framework is developed that combines data-driven and data-assimilation methods for simultaneously identifying physic…
GE Research（通用电气研究中心）；University of Southern California（南加州大学）
We present a method of discovering governing differential equations from data without the need to specify a priori the terms to appear in the equation. The input to our method is a dataset (or ensemble of datasets) corresponding to a particular solution (or ensemble of particular solutions) of a differential equation. The output is a human-readable differential equation with parameters calibrated to the individual particular solutions provided. The key to our method is to learn differentiable models of the data that subsequently serve as inputs to a genetic programming algorithm in which graph…
University of Padua（帕多瓦大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
In recent years, deep generative models for graphs have been used to generate new molecules. These models have produced good results, leading to several proposals in the literature. However, these models may have troubles learning some of the complex laws governing the chemical world. In this work, we explore the usage of the histogram of atom valences to drive the generation of molecules in such models. We present Conditional Constrained Graph Variational Autoencoder (CCGVAE), a model that implements this key-idea in a state-of-the-art model, and shows improved results on several evaluation m…
Université de Paris（巴黎大学）；INSERM U1284（法国国家健康与医学研究院 U1284）；CRI（计算研究中心）；IBM Research Paris-Saclay（IBM 巴黎-萨克雷研究院）
Automated model discovery of partial differential equations (PDEs) usually considers a single experiment or dataset to infer the underlying governing equations. In practice, experiments have inherent natural variability in parameters, initial and boundary conditions that cannot be simply averaged out. We introduce a randomised adaptive group Lasso sparsity estimator to promote grouped sparsity and implement it in a deep learning based PDE discovery framework. It allows to create a learning bias that implies the a priori assumption that all experiments can be explained by the same underlying PD…
Discovering governing equations from data is critical for diverse scientific disciplines as they can provide insights into the underlying phenomenon of dynamic systems. This work presents a new representation for governing equations by designing the Mathematical Operation Network (MathONet) with a deep neural network-like hierarchical structure. Specifically, the MathONet is stacked by several layers of unary operations (e.g., sin, cos, log) and binary operations (e.g., +,-), respectively. An initialized MathONet is typically regarded as a super-graph with a redundant structure, a sub-graph of…
Redwood Center for Theoretical Neuroscience（雷德伍德理论神经科学中心）；University of California, Berkeley（加州大学伯克利分校）；Nervana Systems, Inc.（Nervana系统公司）
Deep learning has enjoyed a great deal of success because of its ability to learn useful features for tasks such as classification. But there has been less exploration in learning the factors of variation apart from the classification signal. By augmenting autoencoders with simple regularization terms during training, we demonstrate that standard deep architectures can discover and explicitly represent factors of variation beyond those relevant for categorization. We introduce a cross-covariance penalty (XCov) as a method to disentangle factors like handwriting style for digits and subject ide…
Discovering equations from data: symbolic regression in dynamical systems
Institute of Sciences Mathematics and Computation, Universidade de São Paulo-Campus de São Carlos（数学与计算科学研究院，圣保罗大学-萨o卡洛斯校区）；Statistic Department, State University of Maringá（统计系，马林加州立大学）；Department of Mathematics Applied and Statistics, Institute of Sciences Mathematics and Computation, Universidade de São Paulo-Campus de São Carlos（应用数学与统计系，数学与计算科学研究院，圣保罗大学-萨o卡洛斯校区）
We present a method for unsupervised learning of equations of motion for objects in raw and optionally distorted unlabeled video. We first train an autoencoder that maps each video frame into a low-dimensional latent space where the laws of motion are as simple as possible, by minimizing a combination of non-linearity, acceleration and prediction error. Differential equations describing the motion are then discovered using Pareto-optimal symbolic regression. We find that our pre-regression ("pregression") step is able to rediscover Cartesian coordinates of unlabeled moving objects even when th…
We present the MagNet, a neural network-based multi-agent interaction model to discover the governing dynamics and predict evolution of a complex multi-agent system from observations. We formulate a multi-agent system as a coupled non-linear network with a generic ordinary differential equation (ODE) based state evolution, and develop a neural network-based realization of its time-discretized model. MagNet is trained to discover the core dynamics of a multi-agent system from observations, and tuned on-line to learn agent-specific parameters of the dynamics to ensure accurate prediction even wh…
Discovering Ordinary Differential Equations with LLM-Based Qualitative and Quantitative Evaluation
Department of Artificial Intelligence, Chung-Ang University, Seoul, Republic of Korea（韩国首尔 Chung-Ang 大学人工智能系）；Daejin University, Pocheon, Republic of Korea（韩国坡州市 Daejin 大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Model reduction of high-dimensional dynamical systems alleviates computational burdens faced in various tasks from design optimization to model predictive control. One popular model reduction approach is based on projecting the governing equations onto a subspace spanned by basis functions obtained from the compression of a dataset of solution snapshots. However, this method is intrusive since the projection requires access to the system operators. Further, some systems may require special treatment of nonlinearities to ensure computational efficiency or additional modeling to preserve stabili…
USACE Engineer Research Development Center（美国陆军工程兵团工程师研究与发展中心）；University of Genoa（热那亚大学）；University of Padua（帕多瓦大学）
Model reduction for fluid flow simulation continues to be of great interest across a number of scientific and engineering fields. In a previous work [arXiv:2104.13962], we explored the use of Neural Ordinary Differential Equations (NODE) as a non-intrusive method for propagating the latent-space dynamics in reduced order models. Here, we investigate employing deep autoencoders for discovering the reduced basis representation, the dynamics of which are then approximated by NODE. The ability of deep autoencoders to represent the latent-space is compared to the traditional proper orthogonal decom…
Harnessing data to discover the underlying governing laws or equations that describe the behavior of complex physical systems can significantly advance our modeling, simulation and understanding of such systems in various science and engineering disciplines. This work introduces a novel physics-informed deep learning framework to discover governing partial differential equations (PDEs) from scarce and noisy data for nonlinear spatiotemporal systems. In particular, this approach seamlessly integrates the strengths of deep neural networks for rich representation learning, physics embedding, auto…
Optimizing molecular design and discovering novel chemical structures to meet certain objectives, such as quantitative estimates of the drug-likeness score (QEDs), is NP-hard due to the vast combinatorial design space of discrete molecular structures, which makes it near impossible to explore the entire search space comprehensively to exploit de novo structures with properties of interest. To address this challenge, reducing the intractable search space into a lower-dimensional latent volume helps examine molecular candidates more feasibly via inverse design. Autoencoders are suitable deep lea…
Distilling analytical models from data has the potential to advance our understanding and prediction of nonlinear dynamics. Although discovery of governing equations based on observed system states (e.g., trajectory time series) has revealed success in a wide range of nonlinear dynamics, uncovering the closed-form equations directly from raw videos still remains an open challenge. To this end, we introduce a novel end-to-end unsupervised deep learning framework to uncover the mathematical structure of equations that governs the dynamics of moving objects in videos. Such an architecture consist…
Max Planck Institute for Dynamics of Complex Technical Systems（马克斯·普朗克复杂技术系统动力学研究所）
Discovering dynamical models to describe underlying dynamical behavior is essential to draw decisive conclusions and engineering studies, e.g., optimizing a process. Experimental data availability notwithstanding has increased significantly, but interpretable and explainable models in science and engineering yet remain incomprehensible. In this work, we blend machine learning and dictionary-based learning with numerical analysis tools to discover governing differential equations from noisy and sparsely-sampled measurement data. We utilize the fact that given a dictionary containing huge candid…
Northeastern University（东北大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Renmin University of China（中国人民大学）
Distilling interpretable physical laws from videos has led to expanded interest in the computer vision community recently thanks to the advances in deep learning, but still remains a great challenge. This paper introduces an end-to-end unsupervised deep learning framework to uncover the explicit governing equations of dynamics presented by moving object(s), based on recorded videos. Instead in the pixel (spatial) coordinate system of image space, the physical law is modeled in a regressed underlying physical coordinate system where the physical states follow potential explicit governing equati…
Balance-Guided Sparse Identification of Multiscale Nonlinear PDEs with Small-coefficient Terms
State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences, Beijing 100190, China（非线性力学国家重点实验室，力学研究所，中国科学院，北京100190，中国）；School of Engineering Sciences, University of Chinese Academy of Sciences, Beijing 100049, China（中国科学院大学工程科学学院，北京100049，中国）；School of Mathematics and Statistics, Northwestern Polytechnical University, Xi'an 710072, China（西北工业大学数学与统计学院，西安710072，中国）
Joint discovery of governing partial differential equations from multi-source datasets by competitive optimization
Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology（浙江工业智能与数字孪生重点实验室，东方理工高等研究院）；Department of Electrical Engineering, Tsinghua University（清华大学电机工程系）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工高等研究院宁波数字孪生研究院）
