We train an artificial neural network which distinguishes chaotic and regular dynamics of the two-dimensional Chirikov standard map. We use finite length trajectories and compare the performance with traditional numerical methods which need to evaluate the Lyapunov exponent. The neural network has superior performance for short periods with length down to 10 Lyapunov times on which the traditional Lyapunov exponent computation is far from converging. We show the robustness of the neural network to varying control parameters, in particular we train with one set of control parameters, and successfully test in a complementary set. Furthermore, we use the neural network to successfully test the dynamics of discrete maps in different dimensions, e.g. the one-dimensional logistic map and a three-dimensional discrete version of the Lorenz system. Our results demonstrate that a convolutional neural network can be used as an excellent chaos indicator.
National Institute of Advanced Studies（国家高等研究院）；PES University（PES大学）
Inspired by chaotic firing of neurons in the brain, we propose ChaosNet -- a novel chaos based artificial neural network architecture for classification tasks. ChaosNet is built using layers of neurons, each of which is a 1D chaotic map known as the Generalized Luroth Series (GLS) which has been shown in earlier works to possess very useful properties for compression, cryptography and for computing XOR and other logical operations. In this work, we design a novel learning algorithm on ChaosNet that exploits the topological transitivity property of the chaotic GLS neurons. The proposed learning…
Inspired by the well-known permutation entropy (PE), an effective image encoding scheme for chaotic time series, Triad State Space Construction (TSSC), is proposed. The TSSC image can recognize higher-order temporal patterns and identify new forbidden regions in time series motifs beyond the Bandt-Pompe probabilities. The Convolutional Neural Network (ConvNet) is widely used in image classification. The ConvNet classifier based on TSSC images (TSSC-ConvNet) are highly accurate and very robust in the chaotic signal classification.
University of Oxford（牛津大学）；United Kingdom Atomic Energy Authority（英国原子能管理局）
We use standard deep neural networks to classify univariate time series generated by discrete and continuous dynamical systems based on their chaotic or non-chaotic behaviour. Our approach to circumvent the lack of precise models for some of the most challenging real-life applications is to train different neural networks on a data set from a dynamical system with a basic or low-dimensional phase space and then use these networks to classify univariate time series of a dynamical system with more intricate or high-dimensional phase space. We illustrate this generalisation approach using the log…
A catastrophic bifurcation in non-linear dynamical systems, called crisis, often leads to their convergence to an undesirable non-chaotic state after some initial chaotic transients. Preventing such behavior has proved to be quite challenging. We demonstrate that deep Reinforcement Learning (RL) is able to restore chaos in a transiently-chaotic regime of the Lorenz system of equations. Without requiring any a priori knowledge of the underlying dynamics of the governing equations, the RL agent discovers an effective perturbation strategy for sustaining the chaotic trajectory. We analyze the age…
The theoretical explanation for deep neural network (DNN) is still an open problem. In this paper DNN is considered as a discrete-time dynamical system due to its layered structure. The complexity provided by the nonlinearity in the dynamics is analyzed in terms of topological entropy and chaos characterized by Lyapunov exponents. The properties revealed for the dynamics of DNN are applied to analyze the corresponding capabilities of classification and generalization.
National Institute of Advanced Studies（国家先进研究所）；Indian Institute of Science Campus（印度科学研究所校区）；Consciousness Studies Programme（意识研究项目）；Indian Institute of Science（印度科学研究所）；Indian Institute of Science Campus, Bengaluru, India（印度科学研究所校区，班加罗尔，印度）
There has been empirical evidence of presence of non-linearity and chaos at the level of single neurons in biological neural networks. The properties of chaotic neurons inspires us to employ them in artificial learning systems. Here, we propose a Neurochaos Learning (NL) architecture, where the neurons used to extract features from data are 1D chaotic maps. ChaosFEX+SVM, an instance of this NL architecture, is proposed as a hybrid combination of chaos and classical machine learning algorithm. We formally prove that a single layer of NL with a finite number of 1D chaotic neurons satisfies the U…
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitativ…
We forecast two different chaotic dynamics of the quasiperiodically forced logistic map using the well-known deep learning framework Long Short-Term Memory. We generate two data sets and use one in the training process and the other in the testing process. The predicted values are evaluated using the metric called Root Mean Square Error and visualized using the scatter plots. The robustness of the Long Short-Term Memory model is evaluated using the number of units in the layers of the model. We also make multi-step forecasting of the considered system. We show that the considered Long Short-Te…
Neural Network Entropy (NNetEn): Entropy-Based EEG Signal and Chaotic Time Series Classification, Python Package for NNetEn Calculation
Institute of Physics and Technology, Petrozavodsk State University（彼得罗扎沃茨克国立大学物理与技术学院）；Kuwait College of Science and Technology（科威特科学技术学院）；Vels Institute of Sciences, Technology, and Advanced Studies（维尔斯科学技术与高等研究院）；Damghan University（达姆甘大学）
Exploiting Chaotic Dynamics as Deep Neural Networks
Graduate School of Information Science and Technology, The University of Tokyo（东京大学情报理工学系研究科）；Graduate School of Informatics, Kyoto University（京都大学情报学研究科）
In this letter, as a proof of concept, we propose a deep learning-based approach to attack the chaos-based image encryption algorithm in \cite{guan2005chaos}. The proposed method first projects the chaos-based encrypted images into the low-dimensional feature space, where essential information of plain images has been largely preserved. With the low-dimensional features, a deconvolutional generator is utilized to regenerate perceptually similar decrypted images to approximate the plain images in the high-dimensional space. Compared with conventional image encryption attack algorithms, the prop…
IBM T.J. Watson Research Center（IBM T.J. Watson 研究中心）
We present a deep neural network for a model-free prediction of a chaotic dynamical system from noisy observations. The proposed deep learning model aims to predict the conditional probability distribution of a state variable. The Long Short-Term Memory network (LSTM) is employed to model the nonlinear dynamics and a softmax layer is used to approximate a probability distribution. The LSTM model is trained by minimizing a regularized cross-entropy function. The LSTM model is validated against delay-time chaotic dynamical systems, Mackey-Glass and Ikeda equations. It is shown that the present L…
Sensitivity analysis of image classification models using generalized polynomial chaos
TUM School of Engineering and Design（慕尼黑技术大学工程与设计学院）；Technical University of Munich（慕尼黑技术大学）；Digitization Department for Production（生产数字化部门）；Leipzig University of Applied Sciences（莱比锡应用科学大学）；Max Planck Institute for Human Cognitive and Brain Sciences（人类认知与脑科学马克斯·普朗克研究所）；HTW Berlin（柏林应用技术大学）；Technical University of Chemnitz（切欣茨技术大学）；Institute for Casting, Composite and Processing Technology (IGCV)（铸造、复合材料与加工技术研究所）
A Compression Based Classification Framework Using Symbolic Dynamics of Chaotic Maps
Department of Computer Science and Information Systems (CSIS)（计算机科学与信息系统系）；Adjunct Faculty, Consciousness Studies Programme, National Institute of Advanced Studies, IISc Campus, Bengaluru, 560012, Karnataka, India（意识研究项目兼职教员，国家高级研究院，IISc校区，班加罗尔，560012，卡纳塔克邦，印度）
Many complex phenomena, from weather systems to heartbeat rhythm patterns, are effectively modeled as low-dimensional dynamical systems. Such systems may behave chaotically under certain conditions, and so the ability to detect chaos based on empirical measurement is an important step in characterizing and predicting these processes. Classifying a system as chaotic usually requires estimating its largest Lyapunov exponent, which quantifies the average rate of convergence or divergence of initially close trajectories in state space, and for which a positive value is generally accepted as an ope…
Recently, a general data driven numerical framework has been developed for learning and modeling of unknown dynamical systems using fully- or partially-observed data. The method utilizes deep neural networks (DNNs) to construct a model for the flow map of the unknown system. Once an accurate DNN approximation of the flow map is constructed, it can be recursively executed to serve as an effective predictive model of the unknown system. In this paper, we apply this framework to chaotic systems, in particular the well-known Lorenz 63 and 96 systems, and critically examine the predictive performan…
National University of Defense Technology（国防科技大学）；Chinese Academy of Military Science（中国人民解放军军事科学院）
The surrogate model-based uncertainty quantification method has drawn much attention in many engineering fields. Polynomial chaos expansion (PCE) and deep learning (DL) are powerful methods for building a surrogate model. However, PCE needs to increase the expansion order to improve the accuracy of the surrogate model, which causes more labeled data to solve the expansion coefficients, and DL also requires a lot of labeled data to train the deep neural network (DNN). First of all, this paper proposes the adaptive arbitrary polynomial chaos (aPC) and proves two properties about the adaptive exp…
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学空天科学学院）；Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Polynomial chaos expansion (PCE) is a powerful surrogate model-based reliability analysis method. Generally, a PCE model with a higher expansion order is usually required to obtain an accurate surrogate model for some complex non-linear stochastic systems. However, the high-order PCE increases the number of labeled data required for solving the expansion coefficients. To alleviate this problem, this paper proposes a consistency regularization-based deep polynomial chaos neural network (Deep PCNN) method, including the low-order adaptive PCE model (the auxiliary model) and the high-order polyno…
Institute for Development and Research in Banking Technology（银行技术发展与研究所）；University of Hyderabad（海得拉巴大学）
This paper proposes to model chaos in the ATM cash withdrawal time series of a big Indian bank and forecast the withdrawals using deep learning methods. It also considers the importance of day-of-the-week and includes it as a dummy exogenous variable. We first modelled the chaos present in the withdrawal time series by reconstructing the state space of each series using the lag, and embedding dimension found using an auto-correlation function and Cao's method. This process converts the uni-variate time series into multi variate time series. The "day-of-the-week" is converted into seven feature…
In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale ($X$), intermediate ($Y$), and fast/small-scale ($Z$) processes. For training or testing, on…
DeepSPoC: A Deep Learning Based Sequential Propagation of Chaos
Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Institute of Computational Mathematics and Scientific/Engineering Computing, AMSS, Chinese Academy of Sciences（中国科学院数学与系统科学研究院计算数学与科学工程计算研究所）
Shanghai Jiao Tong University（上海交通大学）；NVIDIA Corporation（英伟达公司）；University of California（加利福尼亚大学）
Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of several widely used activation functions obtained by Hermite polynomials, and find sharp declines or even saddle points in the feature space, which stagnate the information transfer in DNNs. We then present a new activation function design based on the Hermite polynomials for be…
Deep reinforcement learning (RL) is a data-driven, model-free method capable of discovering complex control strategies for macroscopic objectives in high-dimensional systems, making its application towards flow control promising. Many systems of flow control interest possess symmetries that, when neglected, can significantly inhibit the learning and performance of a naive deep RL approach. Using a test-bed consisting of the Kuramoto-Sivashinsky Equation (KSE), equally spaced actuators, and a goal of minimizing dissipation and power cost, we demonstrate that by moving the deep RL problem to a s…
Russian Quantum Center（俄罗斯量子中心）；University of New South Wales（新南威尔士大学）；Joint Center for Quantum Information and Computer Science, NIST/University of Maryland（美国国家标准与技术研究院/马里兰大学量子信息与计算机科学联合中心）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；QuTech, Delft Technical University（代尔夫特理工大学量子技术研究所）；Steklov Mathematical Institute of Russian Academy of Sciences（俄罗斯科学院斯捷克洛夫数学研究所）；National University of Science and Technology MISIS（国立科技大学米西斯）
Understanding properties of quantum matter is an outstanding challenge in science. In this paper, we demonstrate how machine-learning methods can be successfully applied for the classification of various regimes in single-particle and many-body systems. We realize neural network algorithms that perform a classification between regular and chaotic behavior in quantum billiard models with remarkably high accuracy. We use the variational autoencoder for autosupervised classification of regular/chaotic wave functions, as well as demonstrating that variational autoencoders could be used as a tool f…
University of Edinburgh（爱丁堡大学）；University of Cambridge（剑桥大学）；Instituto de Telecomunicações（电信研究所）；Leiden University（莱顿大学）
Since its formulation by Sir Isaac Newton, the problem of solving the equations of motion for three bodies under their own gravitational force has remained practically unsolved. Currently, the solution for a given initialization can only be found by performing laborious iterative calculations that have unpredictable and potentially infinite computational cost, due to the system's chaotic nature. We show that an ensemble of solutions obtained using an arbitrarily precise numerical integrator can be used to train a deep artificial neural network (ANN) that, over a bounded time interval, provides…
Classification accuracy provided by a machine learning model depends a lot on the feature set used in the learning process. Feature Selection (FS) is an important and challenging pre-processing technique which helps to identify only the relevant features from a dataset thereby reducing the feature dimension as well as improving the classification accuracy at the same time. The binary version of Whale Optimization Algorithm (WOA) is a popular FS technique which is inspired from the foraging behavior of humpback whales. In this paper, an embedded version of WOA called Embedded Chaotic Whale Surv…
Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We analyze architectural features of Deep Neural Networks (DNNs) using the so-called Neural Tangent Kernel (NTK), which describes the training and generalization of DNNs in the infinite-width setting. In this setting, we show that for fully-connected DNNs, as the depth grows, two regimes appear: "order", where the (scaled) NTK converges to a constant, and "chaos", where it converges to a Kronecker delta. Extreme order slows down training while extreme chaos hinders generalization. Using the scaled ReLU as a nonlinearity, we end up in the ordered regime. In contrast, Layer Normalization brings …
Chaotic Map based Compression Approach to Classification
BITS Pilani K K Birla Goa Campus（BITS Pilani K K Birla 果阿校区）；National Institute of Advanced Studies（国家高级研究院）；Norwegian Institute of Science and Technology（挪威科学技术研究院）
We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these …
The introduction of unexpected system disturbances and new system dynamics does not allow guaranteed continuous system stability. In this research we present a novel approach for detecting early failure indicators of non-linear highly chaotic system and accordingly predict the best parameter calibrations to offset such instability using deep machine learning regression model. The approach proposed continuously monitors the system and controller signals. The Re-calibration of the system and controller parameters is triggered according to a set of conditions designed to maintain system stability…
School of Computer Science And Information Sciences, University of Hyderabad（海得拉巴大学计算机科学与信息科学学院）
Machine Learning (ML) has become the new contrivance in almost every field. This makes them a target of fraudsters by various adversary attacks, thereby hindering the performance of ML models. Evasion and Data-Poison-based attacks are well acclaimed, especially in finance, healthcare, etc. This motivated us to propose a novel computationally less expensive attack mechanism based on the adversarial sample generation by Variational Auto Encoder (VAE). It is well known that Wavelet Neural Network (WNN) is considered computationally efficient in solving image and audio processing, speech recogniti…
Stanford University（斯坦福大学）；Element AI（元素人工智能公司）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
In suitably initialized wide networks, small learning rates transform deep neural networks (DNNs) into neural tangent kernel (NTK) machines, whose training dynamics is well-approximated by a linear weight expansion of the network at initialization. Standard training, however, diverges from its linearization in ways that are poorly understood. We study the relationship between the training dynamics of nonlinear deep networks, the geometry of the loss landscape, and the time evolution of a data-dependent NTK. We do so through a large-scale phenomenological analysis of training, synthesizing dive…
Despite many of the most common chaotic dynamical systems being continuous in time, it is through discrete time mappings that much of the understanding of chaos is formed. Henri Poincaré first made this connection by tracking consecutive iterations of the continuous flow with a lower-dimensional, transverse subspace. The mapping that iterates the dynamics through consecutive intersections of the flow with the subspace is now referred to as a Poincaré map, and it is the primary method available for interpreting and classifying chaotic dynamics. Unfortunately, in all but the simplest systems, an…
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of h…
Making a simple model by choosing a limited number of features with the purpose of reducing the computational complexity of the algorithms involved in classification is one of the main issues in machine learning and data mining. The aim of Feature Selection (FS) is to reduce the number of redundant and irrelevant features and improve the accuracy of classification in a data set. We propose an efficient ISPSO-GLOBAL (Improved Seeding Particle Swarm Optimization GLOBAL) method which investigates the specified iterations to produce prominent features and store them in storage list. The goal is to…
Max-Planck-Institut für Physik komplexer Systeme（马克斯·普朗克复杂系统物理研究所）；Würzburg-Dresden Cluster of Excellence ct.qmat（维尔茨堡-德累斯顿卓越集群ct.qmat）；Instituut-Lorentz, Universiteit Leiden（莱顿大学洛伦兹研究所）；Nordita KTH Royal Institute of Technology and Stockholm University（瑞典皇家理工学院和斯德哥尔摩大学北欧理论物理研究所）
We explicitly construct the quantum field theory corresponding to a general class of deep neural networks encompassing both recurrent and feedforward architectures. We first consider the mean-field theory (MFT) obtained as the leading saddlepoint in the action, and derive the condition for criticality via the largest Lyapunov exponent. We then compute the loop corrections to the correlation function in a perturbative expansion in the ratio of depth $T$ to width $N$, and find a precise analogy with the well-studied $O(N)$ vector model, in which the variance of the weight initializations plays t…
Variational Kernel Design for Internal Noise: Gaussian Chaos Noise, Representation Compatibility, and Reliable Deep Learning
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS)（上海数学与交叉科学研究院）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究所）
National Defense Academy of Japan（日本防卫学院）；Research Center for Mathematical Engineering, Chiba Institute of Technology（数学工程研究中心，秋田工科大学）；International Research Center for Neuro intelligence (WPI-IRCN), The University of Tokyo（神经智能国际研究中心（WPI-IRCN），东京大学）
Linnaeus University（林奈大学）；Machine Intelligence Research Labs (MIR Labs)（机器智能研究实验室）
Deep learning is an important component of big-data analytic tools and intelligent applications, such as, self-driving cars, computer vision, speech recognition, or precision medicine. However, the training process is computationally intensive, and often requires a large amount of time if performed sequentially. Modern parallel computing systems provide the capability to reduce the required training time of deep neural networks. In this paper, we present our parallelization scheme for training convolutional neural networks (CNN) named Controlled Hogwild with Arbitrary Order of Synchronization …
