arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.03793cs.LGcond-mat.softstat.ML
收藏

倾斜竞争环境:机器学习的动态损失函数

Tilting the playing field: Dynamical loss functions for machine learning

University of Pennsylvania(宾夕法尼亚大学);Universidad Politécnica de Madrid(马德里理工大学);Google Inc.(谷歌公司)

Miguel Ruiz-Garcia, Ge Zhang, Samuel S. Schoenholz, Andrea J. Liu

摘要:

我们表明,学习可以通过使用损失函数,在训练过程中循环演变,强调一个类的时间。在欠参数化网络中,这种动态损失函数可以导致未能找到标准交叉熵损失的深度最小值的网络的成功训练。在过参数化网络中,动态损失函数可以导致更好的泛化。改进产生于不断变化的损失景观与系统动态的相互作用,因为它的演变,以尽量减少损失。特别是,作为损失函数振荡,不稳定性的发展形式的分叉级联,我们研究使用海森和神经切线核。在一个循环中,随着损失景观的变化,景观中的山谷变宽变深,然后变窄变高。随着景观变窄,学习率变大,网络变得不稳定,在山谷周围反弹。这一过程最终将系统推向更深更广的损失区域,其特征在于海森特征值的下降。这导致更好的正则化模型,具有更好的泛化性能。

英文摘要:

We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamical loss functions can lead to successful training for networks that fail to find a deep minima of the standard cross-entropy loss. In overparameterized networks, dynamical loss functions can lead to better generalization. Improvement arises from the interplay of the changing loss landscape with the dynamics of the system as it evolves to minimize the loss. In particular, as the loss function oscillates, instabilities develop in the form of bifurcation cascades, which we study using the Hessian and Neural Tangent Kernel. Valleys in the landscape widen and deepen, and then narrow and rise as the loss landscape changes during a cycle. As the landscape narrows, the learning rate becomes too large and the network becomes unstable and bounces around the valley. This process ultimately pushes the system into deeper and wider regions of the loss landscape and is characterized by decreasing eigenvalues of the Hessian. This results in better regularized models with improved generalization performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑