Randomized methods of neural network learning suffer from a problem with the generation of random parameters as they are difficult to set optimally to obtain a good projection space. The standard method draws the parameters from a fixed interval which is independent of the data scope and activation function type. This does not lead to good results in the approximation of the strongly nonlinear functions. In this work, a method which adjusts the random parameters, representing the slopes and positions of the sigmoids, to the target function features is proposed. The method randomly selects the input space regions, places the sigmoids in these regions and then adjusts the sigmoid slopes to the local fluctuations of the target function. This brings very good results in the approximation of the complex target functions when compared to the standard fixed interval method and other methods recently proposed in the literature.
In this work, a method of random parameters generation for randomized learning of a single-hidden-layer feedforward neural network is proposed. The method firstly, randomly selects the slope angles of the hidden neurons activation functions from an interval adjusted to the target function, then randomly rotates the activation functions, and finally distributes them across the input space. For complex target functions the proposed method gives better results than the approach commonly used in practice, where the random parameters are selected from the fixed interval. This is because it introduc…
This work contributes to the development of a new data-driven method (D-DM) of feedforward neural networks (FNNs) learning. This method was proposed recently as a way of improving randomized learning of FNNs by adjusting the network parameters to the target function fluctuations. The method employs logistic sigmoid activation functions for hidden nodes. In this study, we introduce other activation functions, such as bipolar sigmoid, sine function, saturating linear functions, reLU, and softplus. We derive formulas for their parameters, i.e. weights and biases. In the simulation study, we evalu…
Feedforward neural networks are widely used as universal predictive models to fit data distribution. Common gradient-based learning, however, suffers from many drawbacks making the training process ineffective and time-consuming. Alternative randomized learning does not use gradients but selects hidden node parameters randomly. This makes the training process extremely fast. However, the problem in randomized learning is how to determine the random parameters. A recently proposed method uses autoencoders for unsupervised parameter learning. This method showed superior performance on classifica…
Feedforward neural networks with random hidden nodes suffer from a problem with the generation of random weights and biases as these are difficult to set optimally to obtain a good projection space. Typically, random parameters are drawn from an interval which is fixed before or adapted during the learning process. Due to the different functions of the weights and biases, selecting them both from the same interval is not a good idea. Recently more sophisticated methods of random parameters generation have been developed, such as the data-driven method proposed in \cite{Anon19}, where the sigmo…
Politecnico di Torino（都灵理工大学）；IEIIT-CNR Torino（意大利国家研究委员会电子、电信与信息工程研究所都灵分部）；INRIA（法国国家信息与自动化研究所）；Ecole polytechnique（巴黎综合理工学院）
We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node (LSE networks) is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of networks maps to a family of subtraction-free ratios of generalized posynomials, which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main adva…
Optimization of randomized neural networks for transfer operator approximation
Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University（爱丁堡大学麦克斯韦数学科学研究所和赫里奥特-瓦特大学）；School of Mathematical & Computer Sciences, Heriot–Watt University（赫里奥特-瓦特大学数学与计算机科学学院）
Beijing Institute of Technology（北京理工大学）；Tsinghua University（清华大学）
A reliable controller is critical and essential for the execution of safe and smooth maneuvers of an autonomous vehicle.The controller must be robust to external disturbances, such as road surface, weather, and wind conditions, and so on.It also needs to deal with the internal parametric variations of vehicle sub-systems, including power-train efficiency, measurement errors, time delay,so on.Moreover, as in most production vehicles, the low-control commands for the engine, brake, and steering systems are delivered through separate electronic control units.These aforementioned factors introduce…
Federated Learning for Data-Driven Feedforward Control: A Case Study on Vehicle Lateral Dynamics
Center for Vision, Automation & Control, AIT Austrian Institute of Technology GmbH（视觉、自动化与控制中心，奥地利技术研究院）；Robert Bosch GmbH（博世集团）；Automation and Control Institute, TU Wien（自动化与控制研究所，维也纳技术大学）
The successes of deep learning, variational inference, and many other fields have been aided by specialized implementations of reverse-mode automatic differentiation (AD) to compute gradients of mega-dimensional objectives. The AD techniques underlying these tools were designed to compute exact gradients to numerical precision, but modern machine learning models are almost always trained with stochastic gradient descent. Why spend computation and memory on exact (minibatch) gradients only to use them for stochastic optimization? We develop a general framework and approach for randomized automa…
This paper presents the application of a newly developed nature-inspired metaheuristic optimization method, namely the Adaptive Wind Driven Optimization (AWDO), to the training of feedforward artificial neural networks (NN) and presents a discussion into the future research of AWDO implementation in Deep Learning (DL). Application example of digit classification with MNIST dataset reveals interesting behavior of the derivative-free AWDO method compared to steepest descent method where results and future work on the implementation of AWDO in deep neural networks are discussed.
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Conventional seismic techniques for detecting the subsurface geologic features are challenged by limited data coverage, computational inefficiency, and subjective human factors. We developed a novel data-driven geological feature detection approach based on pre-stack seismic measurements. Our detection method employs an efficient and accurate machine-learning detection approach to extract useful subsurface geologic features automatically. Specifically, our method is based on kernel ridge regression model. The conventional kernel ridge regression can be computationally prohibited because of the…
Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we sh…
In this paper, we propose a novel data-driven operator learning framework referred to as the \textit{Randomized Prior Wavelet Neural Operator} (RP-WNO). The proposed RP-WNO is an extension of the recently proposed wavelet neural operator, which boasts excellent generalizing capabilities but cannot estimate the uncertainty associated with its predictions. RP-WNO, unlike the vanilla WNO, comes with inherent uncertainty quantification module and hence, is expected to be extremely useful for scientists and engineers alike. RP-WNO utilizes randomized prior networks, which can account for prior info…
Adaptive Randomized Neural Networks for PDEs: Structure-Aware Feature Redistribution and Residual-Driven Block Enrichment
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；School of Mathematics and Statistics & State Key Laboratory of Multiphase Flow in Power Engineering, Xi’an Jiaotong University（西安交通大学数学与统计学院）
Tokyo University of Agriculture and Technology（东京农工大学）
This paper reports the features of a splashing drop on a solid surface and the temporal evolution, which are extracted through image-sequence classification using a highly interpretable feedforward neural network (FNN) with zero hidden layer. The image sequences used for training-validation and testing of the FNN show the early-stage deformation of milli-sized ethanol drops that impact a hydrophilic glass substrate with the Weber number ranges between 31-474 (splashing threshold about 173). Specific videographing conditions and digital image processing are performed to ensure the high similari…
We study the use of feedforward neural networks (FNN) to develop models of nonlinear dynamical systems from data. Emphasis is placed on predictions at long times, with limited data availability. Inspired by global stability analysis, and the observation of the strong correlation between the local error and the maximum singular value of the Jacobian of the ANN, we introduce Jacobian regularization in the loss function. This regularization suppresses the sensitivity of the prediction to the local error and is shown to improve accuracy and robustness. Comparison between the proposed approach and …
Polyhedral Geometry of Time-to-First-Spike Neural Networks
Ludwig-Maximilians-Universität München（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Max Planck Institute for Mathematics in the Sciences, Leipzig（马克斯·普朗克数学科学研究所）；University of Tromsø（特罗姆瑟大学）
A Randomized PDE Energy driven Iterative Framework for Efficient and Stable PDE Solutions
School of Traffic and Transportation Engineering, Central South University（中南大学交通与运输工程学院）；School of Civil Engineering, Central South University（中南大学土木工程学院）；National Engineering Research Center of High-speed Railway Construction Technology（高速铁路建设技术国家工程研究中心）；School of Urban Construction and Transportation, Hefei University（合肥大学城市建设与交通学院）；College of Mechanical Engineering, Zhejiang University of Technology（浙江工业大学机械工程学院）
Yale-NUS College（耶鲁-新加坡国立大学学院）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of the signal space. This paper provides a precise description of the individual affine linear representations and corresponding domain regions that the (data-driven) neural network associates to each signal of the input space. In particular, we describe atomic decompositions of th…
A data-driven analysis method known as dynamic mode decomposition (DMD) approximates the linear Koopman operator on projected space. In the spirit of Johnson-Lindenstrauss Lemma, we will use random projection to estimate the DMD modes in reduced dimensional space. In practical applications, snapshots are in high dimensional observable space and the DMD operator matrix is massive. Hence, computing DMD with the full spectrum is infeasible, so our main computational goal is estimating the eigenvalue and eigenvectors of the DMD operator in a projected domain. We will generalize the current algorit…
University of Toronto（多伦多大学）；DeepMind；Google Brain（谷歌大脑）
Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption an…
Large-scale kernel approximation is an important problem in machine learning research. Approaches using random Fourier features have become increasingly popular [Rahimi and Recht, 2007], where kernel approximation is treated as empirical mean estimation via Monte Carlo (MC) or Quasi-Monte Carlo (QMC) integration [Yang et al., 2014]. A limitation of the current approaches is that all the features receive an equal weight summing to 1. In this paper, we propose a novel shrinkage estimator from "Stein effect", which provides a data-driven weighting strategy for random features and enjoys theoretic…
Faculty of Electrical Engineering, University of West Bohemia（西波希米亚大学电气工程学院）
Timepix and Timepix3 are hybrid pixel detectors ($256\times 256$ pixels), capable of tracking ionizing particles as isolated clusters of pixels. To efficiently analyze such clusters at potentially high rates, we introduce multiple randomized pattern recognition algorithms inspired by computer vision. Offering desirable probabilistic bounds on accuracy and complexity, the presented methods are well-suited for use in real-time applications, and some may even be modified to tackle trans-dimensional problems. In Timepix detectors, which do not support data-driven acquisition, they have been shown …
Ben Gurion University of the Negev（内盖夫本·古里安大学）；Cornell University（康奈尔大学）
Directed information (DI) is a fundamental measure for the study and analysis of sequential stochastic models. In particular, when optimized over input distributions it characterizes the capacity of general communication channels. However, analytic computation of DI is typically intractable and existing optimization techniques over discrete input alphabets require knowledge of the channel model, which renders them inapplicable when only samples are available. To overcome these limitations, we propose a novel estimation-optimization framework for DI over discrete input spaces. We formulate DI o…
The University of Edinburgh（爱丁堡大学）；Rutgers University（罗格斯大学）；Baidu Inc（百度公司）
We study the problem of minimizing the average of a large number of smooth convex functions penalized with a strongly convex regularizer. We propose and analyze a novel primal-dual method (Quartz) which at every iteration samples and updates a random subset of the dual variables, chosen according to an arbitrary distribution. In contrast to typical analysis, we directly bound the decrease of the primal-dual error (in expectation), without the need to first analyze the dual error. Depending on the choice of the sampling, we obtain efficient serial, parallel and distributed variants of the metho…
Stanford University（斯坦福大学）；McGovern Institute for Brain Research, MIT（麻省理工学院麦戈文脑研究所）；MIT（麻省理工学院）；KU Leuven（鲁汶大学）；Google Brain, Google, Inc.（谷歌公司谷歌大脑）；Wu Tsai Neurosciences Institute（吴蔡神经科学研究所）
Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (…
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；University of Waterloo（滑铁卢大学）
In this paper, we investigate the joint device activity and data detection in massive machine-type communications (mMTC) with a one-phase non-coherent scheme, where data bits are embedded in the pilot sequences and the base station simultaneously detects active devices and their embedded data bits without explicit channel estimation. Due to the correlated sparsity pattern introduced by the non-coherent transmission scheme, the traditional approximate message passing (AMP) algorithm cannot achieve satisfactory performance. Therefore, we propose a deep learning (DL) modified AMP network (DL-mAMP…
Evaluating probabilistic and data-driven inference models for fiber-coupled NV-diamond temperature sensors
School of Mathematical and Statistical Sciences, Clemson University（克莱姆森大学数学与统计学院）；Department of Mathematical Sciences, George Mason University（乔治梅森大学数学科学系）
