Fisher information flow in artificial neural networks
Institute for Theoretical Physics, Vienna University of Technology (TU Wien), A–1040 Vienna, Austria;School of Physics;Astronomy, University of Glasgow, Glasgow, G12 8QQ, United Kingdom;Faculty of Computer Science, Ruhr University Bochum, 44801 Bochum, Germany;Universit \'e Grenoble Alpes, CNRS, LIPhy, 38000 Grenoble, France
Maximilian Weimar, Lukas M. Rachbauer, Ilya Starshynov, Daniele Faccio, Linara Adilova, Dorian Bouchet, Stefan Rotter
The estimation of continuous parameters from measured data plays a central role in many fields of physics. A key tool in understanding and improving such estimation processes is the concept of Fisher information, which quantifies how information about unknown parameters propagates through a physical system and determines the ultimate limits of precision. With Artificial Neural Networks (ANNs) gradually becoming an integral part of many measurement systems, it is essential to understand how they process and transmit parameter-relevant information internally. Here, we present a method to monitor the flow of Fisher information through an ANN performing a parameter estimation task, tracking it from the input to the output layer. We show that optimal estimation performance corresponds to the maximal transmission of Fisher information, and that training beyond this point results in information loss due to overfitting. This provides a model-free stopping criterion for network training-eliminating the need for a separate validation dataset. To demonstrate the practical relevance of our approach, we apply it to a network trained on data from an imaging experiment, highlighting its effectiveness in a realistic physical setting.
A Fisher-Rao gradient flow for entropic mean-field min-max games
Heriot-Watt University(赫瑞瓦特大学);Maxwell Institute for Mathematical Sciences(麦克斯韦数学研究所);University of Edinburgh(爱丁堡大学);The Alan Turing Institute(阿兰·图灵研究所);Simtopia(Simtopia公司)
Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients
RWTH Aachen University(亚琛工业大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Max Planck Institute for Mathematics in the Sciences(马克斯·普朗克数学科学研究所)
Projected Wasserstein-Fisher-Rao Flows for Hawkes Background-Intensity Estimation
Facultad de Ciencias, Universidad Autónoma de México(墨西哥国立自治大学理学院);Department of Probability and Statistics, Centro de Investigación en Matemáticas A.C.(数学研究中心概率与统计系)
Fisher Decorator: Refining Flow Policy via A Local Transport Map
University College London(伦敦大学学院);Shanghai Jiao Tong University(上海交通大学);King’s College London(伦敦国王学院);Peking University(北京大学);Shanghai University of Finance and Economics(上海财经大学);Imperial College London(伦敦帝国学院)
Understanding the Feedforward Artificial Neural Network Model From the Perspective of Network Flow
School of Computer Science, Shanghai Key Laboratory of Data Science, Fudan University(复旦大学计算机科学技术学院,上海市数据科学重点实验室);School of Automation Science and Engineering, South China University of Technology(华南理工大学自动化科学与工程学院)
Category learning in deep neural networks: Information content and geometry of internal representations
Centre d’Analyse et de Mathématique Sociales (CAMS) EHESS, CNRS École des Hautes Études en Sciences Sociales(分析与数学社会中心(CAMS)EHESS,CNRS高等社会科学学院);Laboratoire de Physique de l’École Normale Supérieure (LPENS), ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité École Normale Supérieure(高等师范物理实验室(LPENS),ENS,PSL大学,CNRS,索邦大学,巴黎城市大学高等师范学院)