arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.10690cs.LG
收藏

动态损失函数塑造损失景观地形并改善人工神经网络的学习

Dynamical loss functions shape landscape topography and improve learning in artificial neural networks

Departamento de Matemática Aplicada, ETSII, Universidad Politécnica de Madrid(应用数学系,II工学院,马德里理工大学);Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense Madrid(物质结构系,热力学与电子学系,马德里康普顿斯大学);GISC - Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense Madrid(复杂系统跨学科小组,马德里康普顿斯大学)

Eduardo Lavin Pallero, Miguel Ruiz-Garcia

AI总结:

本文提出动态损失函数,通过让各类别贡献周期性振荡来重塑损失景观,在不改变全局最小值的前提下显著提升不同规模网络的验证准确率,并揭示其与不稳定边缘最小化的关联。

摘要:

动态损失函数源自监督分类任务中使用的标准损失函数,但经过修改,使得每个类别的贡献周期性增减。这些振荡在不影响全局最小值的情况下全局改变了损失景观。在本文中,我们展示了如何将交叉熵和均方误差转换为动态损失函数。我们首先讨论增大神经网络规模或学习率对系统所探索的最小值的深度和锐度的影响。基于这一直觉,我们提出了几个版本的动态损失函数,并使用一个简单的分类问题,展示了它们如何显著提高不同规模网络的验证准确率。最后,我们探索了这些动态损失函数的景观在训练过程中如何演化,强调了可能与不稳定边缘最小化相关的不稳定性的出现。

英文摘要:

Dynamical loss functions are derived from standard loss functions used in supervised classification tasks, but are modified so that the contribution from each class periodically increases and decreases. These oscillations globally alter the loss landscape without affecting the global minima. In this paper, we demonstrate how to transform cross-entropy and mean squared error into dynamical loss functions. We begin by discussing the impact of increasing the size of the neural network or the learning rate on the depth and sharpness of the minima that the system explores. Building on this intuition, we propose several versions of dynamical loss functions and use a simple classification problem where we can show how they significantly improve validation accuracy for networks of varying sizes. Finally, we explore how the landscape of these dynamical loss functions evolves during training, highlighting the emergence of instabilities that may be linked to edge-of-instability minimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑