We describe, implement and test a novel method for training neural networks to estimate the Jacobian matrix $J$ of an unknown multivariate function $F$. The training set is constructed from finitely many pairs $(x,F(x))$ and it contains no explicit information about $J$. The loss function for backpropagation is based on linear approximations and on a nearest neighbor search in the sample data. We formally establish an upper bound on the uniform norm of the error, in operator norm, between the estimated Jacobian matrix provided by the algorithm and the actual Jacobian matrix, under natural assumptions on the function, on the training set and on the loss of the neural network during training.
The Jacobian matrix of a multivariate function contains a wealth of information about the function and it has numerous applications in science and engineering. The method given here represents a step in moving from black-box approximations of functions by neural networks to approximations that provide some structural information about the function in question.
Channel Matters: Estimating Channel Influence for Multivariate Time Series
Intitute of Information Sensing（信息感知研究所）；School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Hong Kong University of Science and Technology（香港科学与技术大学）
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；RIKEN AIP（理化学研究所人工智能项目）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；University of Helsinki（赫尔辛基大学）
Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals -- thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their maximum likelihood based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a tra…
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experi…
To gain insight into complex systems it is a key challenge to infer nonlinear causal directional relations from observational time-series data. Specifically, estimating causal relationships between interacting components in large systems with only short recordings over few temporal observations remains an important, yet unresolved problem. Here, we introduce a large-scale Nonlinear Granger Causality (lsNGC) approach for inferring directional, nonlinear, multivariate causal interactions between system components from short high-dimensional time-series recordings. By modeling interactions with n…
Generative moment matching networks (GMMNs) are introduced for generating quasi-random samples from multivariate models with any underlying copula in order to compute estimates under variance reduction. So far, quasi-random sampling for multivariate distributions required a careful design, exploiting specific properties (such as conditional distributions) of the implied parametric copula or the underlying quasi-Monte Carlo (QMC) point set, and was only tractable for a small number of models. Utilizing GMMNs allows one to construct quasi-random samples for a much larger variety of multivariate …
The local Lipschitz constant of a neural network is a useful metric with applications in robustness, generalization, and fairness evaluation. We provide novel analytic results relating the local Lipschitz constant of nonsmooth vector-valued functions to a maximization over the norm of the generalized Jacobian. We present a sufficient condition for which backpropagation always returns an element of the generalized Jacobian, and reframe the problem over this broad class of functions. We show strong inapproximability results for estimating Lipschitz constants of ReLU networks, and then formulate …
Technion - Israel Institute of Technology（以色列理工学院）；UC Berkeley（加州大学伯克利分校）
The recently proposed distributional approach to reinforcement learning (DiRL) is centered on learning the distribution of the reward-to-go, often referred to as the value distribution. In this work, we show that the distributional Bellman equation, which drives DiRL methods, is equivalent to a generative adversarial network (GAN) model. In this formulation, DiRL can be seen as learning a deep generative model of the value distribution, driven by the discrepancy between the distribution of the current value, and the distribution of the sum of current reward and next value. We use this insight …
Equation-Free Period-Aware Forecast-Error Contraction for Estimating Negative Largest Lyapunov Exponents from Short Trajectory Ensembles
Institute of Physics and Technology, Petrozavodsk State University（彼得罗扎沃茨克国立大学物理与技术学院）；School of Science and Engineering, International University of Grand-Bassam（格兰德巴萨姆国际大学科学与工程学院）；Faculty of Electronics Technology, Industrial University of Ho Chi Minh City（胡志明市工业大学电子技术学院）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Columbia University（哥伦比亚大学）
We consider the problem of estimating the common mean of independently sampled data, where samples are drawn in a possibly non-identical manner from symmetric, unimodal distributions with a common mean. This generalizes the setting of Gaussian mixture modeling, since the number of distinct mixture components may diverge with the number of observations. We propose an estimator that adapts to the level of heterogeneity in the data, achieving near-optimality in both the i.i.d. setting and some heterogeneous settings, where the fraction of ``low-noise'' points is as small as $\frac{\log n}{n}$. Ou…
