Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.
Opening the Black Box: predicting the trainability of deep neural networks with reconstruction entropy
Institute for Theoretical Physics and Astrophysics（理论物理与天体物理研究所）；Würzburg-Dresden Cluster of Excellence ct.qmat（维尔茨堡-德累斯顿卓越集群ct.qmat）；Julius-Maximilians-University Würzburg（维尔茨堡大学）；Institute for Theoretical Physics（理论物理研究所）；Utrecht University（乌得勒支大学）；Department of Information and Computing Sciences（信息与计算科学系）
Advances in Land Surface Model-based Forecasting: A comparative study of LSTM, Gradient Boosting, and Feedforward Neural Network Models as prognostic state emulators
University of Freiburg（弗莱堡大学）；European Centre for Medium-Range Weather Forecasts（欧洲中期天气预报中心）
Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group
Julius-Maximilians-Universität Würzburg（维尔茨堡大学）；Max Planck Institute for the Physics of Complex Systems（马克斯·普朗克复杂系统物理研究所）；Nordita KTH Royal Institute of Technology and Stockholm University（北欧理论物理研究所、瑞典皇家理工学院和斯德哥尔摩大学）
Normative Networks for Source Separation via Local Plasticity and Dendritic Computation
Gatsby Computational Neuroscience Unit, University College London, UK（Gatsby计算神经科学单元，伦敦大学学院，英国）；Department of Computer Science, University of Oxford, UK（计算机科学系，牛津大学，英国）；Brain Network Dynamics Unit, University of Oxford, UK（脑网络动力学单元，牛津大学，英国）；MRC Centre of Research Excellence in Restorative Neural Dynamics, UK（英国医学研究委员会修复神经动力学研究卓越中心）
