arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.13404cs.LGmath.OC
收藏

利用参数对称性提升收敛性与泛化能力

Improving Convergence and Generalization Using Parameter Symmetries

University of California San Diego(加利福尼亚大学圣迭戈分校);Flatiron Institute(弗莱肯研究所);Northeastern University(东北大学)

Bo Zhao, Robert M. Gower, Robin Walters, Rose Yu

AI总结:

该研究探究参数对称性优化方法teleportation的作用机制,发现其不仅能加快短期优化、缩短整体收敛时间,还可通过切换不同曲率极小值提升泛化能力,且适用于多种优化算法与元学习场景,验证了对称性在优化中的应用潜力。

摘要:

在许多神经网络中,不同的参数取值可能产生相同的损失值。参数空间对称性是指改变模型参数但保持损失不变的变换。Teleportation(瞬移优化)通过应用这类变换来加速优化过程,然而该算法生效的确切机制尚未被充分厘清。本文研究表明,teleportation不仅能在短期内加快优化速度,还能整体缩短收敛所需时间。此外,瞬移至曲率不同的极小值点可提升泛化能力,这表明极小值点的曲率与泛化能力之间存在关联。最后,研究证实将teleportation整合到各类优化算法以及基于优化的元学习中,均能改善收敛效果。研究结果展现了teleportation的通用性,也证明了在优化中引入对称性的潜力。

英文摘要:

In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑