Dynamical loss functions shape landscape topography and improve learning in artificial neural networks
Departamento de Matemática Aplicada, ETSII, Universidad Politécnica de Madrid(应用数学系,II工学院,马德里理工大学);Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense Madrid(物质结构系,热力学与电子学系,马德里康普顿斯大学);GISC - Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense Madrid(复杂系统跨学科小组,马德里康普顿斯大学)
Dynamical loss functions are derived from standard loss functions used in supervised classification tasks, but are modified so that the contribution from each class periodically increases and decreases. These oscillations globally alter the loss landscape without affecting the global minima. In this paper, we demonstrate how to transform cross-entropy and mean squared error into dynamical loss functions. We begin by discussing the impact of increasing the size of the neural network or the learning rate on the depth and sharpness of the minima that the system explores. Building on this intuition, we propose several versions of dynamical loss functions and use a simple classification problem where we can show how they significantly improve validation accuracy for networks of varying sizes. Finally, we explore how the landscape of these dynamical loss functions evolves during training, highlighting the emergence of instabilities that may be linked to edge-of-instability minimization.
Walter Burke Institute for Theoretical Physics, California Institute of Technology, Pasadena, California 91125, USA(沃尔特·伯克理论物理研究所,加州理工学院,帕萨迪纳,加利福尼亚州91125,美国)
Testing the Quantitative Spacetime Hypothesis using Artificial Narrative Comprehension (I) : Bootstrapping Meaning from Episodic Narrative viewed as a Feature Landscape
Mapping the Landscape of Artificial Intelligence Applications against COVID-19
United Nations Global Pulse(联合国全球脉动);Institute for Data Science, Durham University(杜伦大学数据科学研究所);Mila Québec Artificial Intelligence Institute(Mila魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);NYU Stern School of Business(纽约大学斯特恩商学院);World Health Organization(世界卫生组织)
Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks
Generating gradients in the energy landscape using rectified linear type cost functions for efficiently solving 0/1 matrix factorization in Simulated Annealing
Fujitsu Limited(富士通株式会社);DXR Laboratory Inc.(DXR实验室股份有限公司);The University of Tokyo(东京大学)