模连接的损耗面简化法与快速合成
Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
摘要:
通过更好地理解多层网络的损失面,我们可以构建更强大和准确的训练过程。最近发现,独立训练的SGD解决方案可以沿着接近恒定训练损失的一维路径连接。在本文中,我们证明了存在模式连接单纯复形,它们形成低损耗的多维流形,连接许多独立训练的模型。受这一发现的启发,我们展示了如何有效地构建用于快速集成的单纯复形,在准确性,校准和对数据集偏移的鲁棒性方面优于独立训练的深度集成。值得注意的是,我们的方法只需要几个训练时期就可以从预先训练的解决方案开始发现低损失的单纯形。代码可在https://github.com/g-benton/loss-surface-simplexes上获得。
英文摘要:
With a better understanding of the loss surfaces for multilayer networks, we can build more robust and accurate training procedures. Recently it was discovered that independently trained SGD solutions can be connected along one-dimensional paths of near-constant training loss. In this paper, we show that there are mode-connecting simplicial complexes that form multi-dimensional manifolds of low loss, connecting many independently trained models. Inspired by this discovery, we show how to efficiently build simplicial complexes for fast ensembling, outperforming independently trained deep ensembles in accuracy, calibration, and robustness to dataset shift. Notably, our approach only requires a few training epochs to discover a low-loss simplex, starting from a pre-trained solution. Code is available at https://github.com/g-benton/loss-surface-simplexes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。