DNN的损失面、模式连通性和快速集成
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
摘要:
深度神经网络的损失函数很复杂,其几何性质还没有得到很好的理解。我们表明,这些复杂的损失函数的最优值实际上是由简单的曲线连接,在这些曲线上,训练和测试精度几乎是恒定的。我们引入了一个训练过程来发现这些模式之间的高精度路径。受这种新的几何见解的启发,我们还提出了一种新的集成方法,称为快速几何集成(FGE)。使用FGE,我们可以在训练单个模型所需的时间内训练高性能的合奏。与最近最先进的快照集成相比,我们在CIFAR-10,CIFAR-100和ImageNet上实现了更高的性能。
英文摘要:
The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。