arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.10026stat.MLcs.AIcs.LG
收藏

DNN的损失面、模式连通性和快速集成

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Samsung AI Center in Moscow(三星莫斯科人工智能中心);Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);Cornell University(康奈尔大学);National Research University Higher School of Economics(国立研究型大学高等经济学院)

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson

摘要:

深度神经网络的损失函数很复杂,其几何性质还没有得到很好的理解。我们表明,这些复杂的损失函数的最优值实际上是由简单的曲线连接,在这些曲线上,训练和测试精度几乎是恒定的。我们引入了一个训练过程来发现这些模式之间的高精度路径。受这种新的几何见解的启发,我们还提出了一种新的集成方法,称为快速几何集成(FGE)。使用FGE,我们可以在训练单个模型所需的时间内训练高性能的合奏。与最近最先进的快照集成相比,我们在CIFAR-10,CIFAR-100和ImageNet上实现了更高的性能。

英文摘要:

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑