The paper uses statistical and differential geometric motivation to acquire prior information about the learning capability of an artificial neural network on a given dataset. The paper considers a broad class of neural networks with generalized architecture performing simple least square regression with stochastic gradient descent (SGD). The system characteristics at two critical epochs in the learning trajectory are analyzed. During some epochs of the training phase, the system reaches equilibrium with the generalization capability attaining a maximum. The system can also be coherent with localized, non-equilibrium states, which is characterized by the stabilization of the Hessian matrix. The paper proves that neural networks with higher generalization capability will have a slower convergence rate. The relationship between the generalization capability with the stability of the neural network has also been discussed. By correlating the principles of high-energy physics with the learning theory of neural networks, the paper establishes a variant of the Complexity-Action conjecture from an artificial neural network perspective.
University of Pennsylvania（宾夕法尼亚大学）；Wharton School（沃顿商学院）
A random forest is a popular tool for estimating probabilities in machine learning classification tasks. However, the means by which this is accomplished is unprincipled: one simply counts the fraction of trees in a forest that vote for a certain class. In this paper, we forge a connection between random forests and kernel regression. This places random forest probability estimation on more sound statistical footing. As part of our investigation, we develop a model for the proximity kernel and relate it to the geometry and sparsity of the estimation problem. We also provide intuition and recom…
CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61研究所）；Australian National University（澳大利亚国立大学）；École Polytechnique Paris Saclay（巴黎萨克雷巴黎综合理工学院）；University of Sydney（悉尼大学）；RIKEN（日本理化学研究所）；University of Tokyo（东京大学）
We introduce a family of pairwise stochastic gradient estimators for gradients of expectations, which are related to the log-derivative trick, but involve pairwise interactions between samples. The simplest example of our new estimator, dubbed the fundamental trick estimator, is shown to arise from either a) introducing and approximating an integral representation based on the fundamental theorem of calculus, or b) applying the reparameterisation trick to an implicit parameterisation under infinitesimal perturbation of the parameters. From the former perspective we generalise to a reproducing …
The University of Tokyo（东京大学）；Japan Science and Technology Agency（日本科学技术振兴机构）；RIKEN（理化学研究所）
We develop a new theoretical framework to analyze the generalization error of deep learning, and derive a new fast learning rate for two representative algorithms: empirical risk minimization and Bayesian deep learning. The series of theoretical analyses of deep learning has revealed its high expressive power and universal approximation capability. Although these analyses are highly nonparametric, existing generalization error analyses have been developed mainly in a fixed dimensional parametric model. To compensate this gap, we develop an infinite dimensional model that is based on an integra…
Statistical Inference for Optimal Transport Maps: Recent Advances and Perspectives
Department of Statistics and Data Science（统计与数据科学系）；Machine Learning Department（机器学习系）；Carnegie Mellon University（卡内基梅隆大学）；Statistics and Data Science Center（统计与数据科学中心）；Massachusetts Institute of Technology（麻省理工学院）
Transfer learning assumes classifiers of similar tasks share certain parameter structures. Unfortunately, modern classifiers uses sophisticated feature representations with huge parameter spaces which lead to costly transfer. Under the impression that changes from one classifier to another should be ``simple'', an efficient transfer learning criteria that only learns the ``differences'' is proposed in this paper. We train a \emph{posterior ratio} which turns out to minimizes the upper-bound of the target learning risk. The model of posterior ratio does not have to share the same parameter spac…
University of Science and Technology of China（中国科学技术大学）；Tianjin University（天津大学）
In the online innovation market, the fund-raising performance of the start-up project is a concerning issue for creators, investors and platforms. Unfortunately, existing studies always focus on modeling the fund-raising process after the publishment of a project but the predicting of a project attraction in the market before setting up is largely unexploited. Usually, this prediction is always with great challenges to making a comprehensive understanding of both the start-up project and market environment. To that end, in this paper, we present a focused study on this important problem from a…
Despite its impressive theory \& practical performance, Frequent Directions (\acrshort{fd}) has not been widely adopted for large-scale regression tasks. Prior work has shown randomized sketches (i) perform worse in estimating the covariance matrix of the data than \acrshort{fd}; (ii) incur high error when estimating the bias and/or variance on sketched ridge regression. We give the first constant factor relative error bounds on the bias \& variance for sketched ridge regression using \acrshort{fd}. We complement these statistical results by showing that \acrshort{fd} can be used in the optimi…
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Google Research（谷歌研究院）
We consider the robust Perspective-n-Point (PnP) problem using a hybrid approach that combines deep learning with model based algorithms. PnP is the problem of estimating the pose of a calibrated camera given a set of 3D points in the world and their corresponding 2D projections in the image. In its more challenging robust version, some of the correspondences may be mismatched and must be efficiently discarded. Classical solutions address PnP via iterative robust non-linear least squares method that exploit the problem's geometry but are either inaccurate or computationally intensive. In contr…
Recent results in coupled or temporal graphical models offer schemes for estimating the relationship structure between features when the data come from related (but distinct) longitudinal sources. A novel application of these ideas is for analyzing group-level differences, i.e., in identifying if trends of estimated objects (e.g., covariance or precision matrices) are different across disparate conditions (e.g., gender or disease). Often, poor effect sizes make detecting the differential signal over the full set of features difficult: for example, dependencies between only a subset of features…
On the Computational Capability of Graph Neural Networks: A Circuit Complexity Bound Perspective
The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at UC Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；UC Berkeley（加州大学伯克利分校）
University of Michigan（密歇根大学）；RIKEN Cluster for Pioneering Research（理化学研究所先驱研究集群）；RIKEN Center for Quantum Computing（理化学研究所量子计算中心）；The University of Michigan（密歇根大学）
Feynman path integrals provide an elegant, classically inspired representation for the quantum propagator and the quantum dynamics, through summing over a huge manifold of all possible paths. From computational and simulational perspectives, the ergodic tracking of the whole path manifold is a hard problem. Machine learning can help, in an efficient manner, to identify the relevant subspace and the intrinsic structure residing at a small fraction of the vast path manifold. In this work, we propose the Feynman path generator for quantum mechanical systems, which efficiently generates Feynman pa…
University of Queensland-IIT Delhi Academy of Research（昆士兰大学-印度理工学院德里分校研究院）；University of Queensland（昆士兰大学）
In this paper we propose a one-dimensional convolutional neural network (CNN)-based state of charge estimation algorithm for electric vehicles. The CNN is trained using two publicly available battery datasets. The influence of different types of noises on the estimation capabilities of the CNN model has been studied. Moreover, a transfer learning mechanism is proposed in order to make the developed algorithm generalize better and estimate with an acceptable accuracy when a battery with different chemical characteristics than the one used for training the model, is used. It has been observed th…
University of Oxford（牛津大学）；Massachusetts Institute of Technology (MIT)（麻省理工学院）；TU Delft（代尔夫特理工大学）；German Aerospace Center (DLR)（德国航空航天中心）；Proxima Robotics srl（普罗xima机器人有限公司）
When an epidemic spreads into a population, it is often unpractical or impossible to have a continuous monitoring of all subjects involved. As an alternative, algorithmic solutions can be used to infer the state of the whole population from a limited amount of measures. We analyze the capability of deep neural networks to solve this challenging task. Our proposed architecture is based on Graph Convolutional Neural Networks. As such it can reason on the effect of the underlying social network structure, which is recognized as the main component in the spreading of an epidemic. We test the propo…
