利用随机权重平均（SWA）提升数据增强效果

中文摘要 AI 辅助

学习任务的对称性已成为设计现代深度学习解决方案的重要因素，数据增强是将对称性融入通用神经网络的直接且有效方式。近期研究表明，在增强数据上训练时，无限大的深度集成模型可呈现完美对称性，但训练集成需多次重复训练过程，成本高昂。本研究将随机权重平均（SWA）作为无需重复训练的替代集成技术，通过用奥恩斯坦-乌伦贝克过程近似训练末期的随机训练轨迹来分析SWA，证明在无限宽度极限下，增强数据上的SWA可实现超出仅SWA性能提升预期的等方差增强。我们在涵盖计算机视觉、图分类且具有离散与连续对称性的众多模型上开展大量数值实验，验证了上述结果。

英文摘要

The symmetries of a learning task have become an important factor in designing modern deep learning solutions. Data augmentation is a straightforward and effective way of incorporating symmetries into a generic neural network. Recent results show that infinitely large deep ensembles show perfect symmetry when trained on augmented data. However, since training ensembles requires repeating the training process many times, this method is costly. In this work, we study stochastic weight averaging (SWA) applied to classification as an alternative ensembling technique that does not require repeated training runs. We analyze SWA by approximating the stochastic training trajectory at the end of training with an Ornstein--Uhlenbeck process. We show that in the infinite-width limit, SWA on augmented data provides an equivariance boost that goes beyond what could be expected from the performance increase due to SWA alone. We verify our results with extensive numerical experiments on numerous models spanning image and graph classification with both discrete and continuous symmetries.

发表机构

• Chalmers University of Technology and the University of Gothenburg（查尔姆斯理工大学与哥德堡大学）
• Umeå University（于默奥大学）

机构由 AI 辅助整理，请以论文原文为准。