The Gaussian equivalence of generative models for learning with shallow neural networks
International School of Advanced Studies (SISSA)（国际高等研究学院（SISSA））；Duke University（杜克大学）；Université PSL（巴黎文理研究大学）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）
Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, Lenka Zdeborová
Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise independently from some simple probability distribution. Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models. This is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. We provide three strands of rigorous, analytical and numerical evidence corroborating this equivalence. First, we establish rigorous conditions for the Gaussian equivalence to hold in the case of single-layer generative models, as well as deterministic rates for convergence in distribution. Second, we leverage this equivalence to derive a closed set of equations describing the generalisation performance of two widely studied machine learning problems: two-layer neural networks trained using one-pass stochastic gradient descent, and full-batch pre-learned features or kernel methods. Finally, we perform experiments demonstrating how our theory applies to deep, pre-trained generative models. These results open a viable path to the theoretical study of machine learning models with realistic data.
New Equivalences Between Interpolation and SVMs: Kernels and Structured Features
School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电子与计算机工程系）；Institute of Mathematics, EPFL（瑞士联邦理工学院数学学院）；School of Industrial & Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程系）
Lipschitz-Guided Design of Interpolation Schedules in Generative Models
Department of Mathematics, University of California, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Machine Learning Lab, Capital Fund Management, Paris, France（Capital Fund Management机器学习实验室）；Courant Institute, New York University, NY, USA（纽约大学Courant研究所）；Now at University of Maryland, College Park, MD, USA（现在位于马里兰大学 College Park 分校）
An Online Learning Approach to Generative Adversarial Networks
ETH Zürich（苏黎世联邦理工学院）
我们考虑用生成对抗网络（GAN）训练生成模型的问题。虽然GANs可以精确地模拟复杂的分布，但由于困难的极大极小优化问题导致的不稳定性，它们很难训练。在本文中，我们将训练GANs的问题视为在零和游戏中找到混合策略。基于在线学习的思想，我们提出了一种名为Chekhov GAN 1的新训练方法。在理论方面，我们证明了我们的方法可证明收敛到半浅GAN架构的平衡，即架构中的网络是一个单层网络和发电机是任意的。在实践方面，我们开发了一个有效的启发式指导我们的理论结果，我们适用于常用的深GAN架构。在几个现实世界的任务，我们的方法表现出改进的稳定性和性能相比，标准的GAN训练。
Carnegie Mellon University（卡内基梅隆大学）；Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学研究所）；University of Southern California（南加州大学）；University of Pennsylvania（宾夕法尼亚大学）
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）；Center for Computational Mathematics, Flatiron Institute（弗莱森计算数学中心）
