Elbert Du
In this work, we showed that the Implicit Update and Predictive Methods dynamics introduced in prior work satisfy last iterate convergence to a neighborhood around the optimum in overparameterized GANs, where the size of the neighborhood shrinks with the width of the neural network. This is in contrast to prior results, which only guaranteed average iterate convergence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Generative adversarial networks (GANs) are so complex that the existing learning theories do not provide a satisfactory explanation for why GANs have great success in practice. The same situation also remains largely open for deep neural networks. To fill this gap, we introduce a Lipschitz theory to analyze generalization. We demonstrate its simplicity by analyzing generalization and consistency of overparameterized neural networks. We then use this theory to derive Lipschitz-based generalization bounds for GANs. Our bounds show that penalizing the Lipschitz constant of the GAN loss can improv…
Double Descent and Other Interpolation Phenomena in GANs
本文研究 GAN 过参数化的泛化行为，证明度量或 f-散度学习下无双重下降，并提出伪监督方法使 GAN 出现双重下降，结合过参数化可加速训练并提升泛化性能。
Pruning effectively compresses overparameterized models. Despite the success of pruning methods for discriminative models, applying them for generative models has been relatively rarely approached. This study conducts structured pruning on U-Net generators of conditional GANs. A per-layer sensitivity analysis confirms that many unnecessary filters exist in the innermost layers near the bottleneck and can be substantially pruned. Based on this observation, we prune these filters from multiple inner layers or suggest alternative architectures by completely eliminating the layers. We evaluate our…
A broad class of unsupervised deep learning methods such as Generative Adversarial Networks (GANs) involve training of overparameterized models where the number of parameters of the model exceeds a certain threshold. A large body of work in supervised learning have shown the importance of model overparameterization in the convergence of the gradient descent (GD) to globally optimal solutions. In contrast, the unsupervised setting and GANs in particular involve non-convex concave mini-max optimization problems that are often trained using Gradient Descent/Ascent (GDA). The role and benefits of …
Training generative adversarial networks (GANs) often suffers from cyclic behaviors of iterates. Based on a simple intuition that the direction of centripetal acceleration of an object moving in uniform circular motion is toward the center of the circle, we present the Simultaneous Centripetal Acceleration (SCA) method and the Alternating Centripetal Acceleration (ACA) method to alleviate the cyclic behaviors. Under suitable conditions, gradient descent methods with either SCA or ACA are shown to be linearly convergent for bilinear games. Numerical experiments are conducted by applying ACA to …
A method for escaping limit cycles in training GANs
本文提出预测向心加速算法PCAA以缓解GAN训练中的极限环问题，推导其收敛速率上下界，并结合Adam提出PCAA-Adam，在双线性博弈、多元高斯分布和CelebA数据集上验证了有效性。
Stochastic gradient descent (SGD) has been shown to generalize well in many deep learning applications. In practice, one often runs SGD with a geometrically decaying stepsize, i.e., a constant initial stepsize followed by multiple geometric stepsize decay, and uses the last iterate as the output. This kind of SGD is known to be nearly minimax optimal for classical finite-dimensional linear regression problems (Ge et al., 2019). However, a sharp analysis for the last iterate of SGD in the overparameterized setting is still open. In this paper, we provide a problem-dependent analysis on the last…
Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Are
本文证明高阶梯度正则化 GAN 的最优判别器等价于 n 维多调和 RBF 数据插值，并提出 Poly-LSGAN 与 Poly-WGAN，闭式实现显著优于任意架构。
We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs. Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games. WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics. Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs. We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynam…
Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression
本文研究过参数化线性回归的泛化问题，提出数据-算法兼容性概念，通过分析依赖数据的训练轨迹证明考虑早停迭代可在更弱限制下保证泛化。
Last Iterate Convergence of Incremental Methods and Applications in Continual Learning
本文首次在一般凸光滑和凸Lipschitz设置下建立了增量梯度法与增量近端法最后迭代的收敛性保证，其oracle复杂度界几乎匹配平均迭代的最优界，并将结果推广到加权平均和随机排列更新，同时论证了强正则化对持续学习中防止灾难性遗忘的关键作用。
Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing
本文研究过参数化非对称低秩矩阵感知问题，证明分解梯度下降通过轨迹耦合和算法正则化这两个隐式特性，从小随机初始化出发能收敛至全局最优且泛化良好的解。
Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime
研究双空间预条件化梯度下降的收敛性质，分析其在过参数化线性模型中的隐式偏置及收敛特性。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。