面向CTR预测中神经网络的、带稀疏Group Lasso的自适应优化器
Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction
AI总结：
本文提出将稀疏group lasso融入多种自适应优化器的框架并给出收敛保证，实验表明其在CTR预测中可实现高稀疏度并提升或保持模型性能。
摘要：
我们开发了一个新颖框架，将稀疏group lasso的正则化项加入深度学习中的一族自适应优化器，例如Momentum、Adagrad、Adam、AMSGrad和AdaHessian，并据此创建了一类新的优化器，分别命名为Group Momentum、Group Adagrad、Group Adam、Group AMSGrad、Group AdaHessian等。我们基于原始-对偶方法，在随机凸情形下建立了经理论证明的收敛保证。我们在三个大规模真实广告点击数据集上，使用最先进的深度学习模型评估了新优化器的正则化效果。实验结果表明，与采用幅值剪枝方法的后处理流程的原始优化器相比，在相同稀疏度水平下，模型性能可以得到显著提升。此外，与不进行幅值剪枝的情况相比，我们的方法能够实现极高的稀疏度，同时性能显著更优或极具竞争力。代码可在https://github.com/intelligent-machine-learning/tfplus/tree/main/tfplus获取。
英文摘要：
We develop a novel framework that adds the regularizers of the sparse group lasso to a family of adaptive optimizers in deep learning, such as Momentum, Adagrad, Adam, AMSGrad, AdaHessian, and create a new class of optimizers, which are named Group Momentum, Group Adagrad, Group Adam, Group AMSGrad and Group AdaHessian, etc., accordingly. We establish theoretically proven convergence guarantees in the stochastic convex settings, based on primal-dual methods. We evaluate the regularized effect of our new optimizers on three large-scale real-world ad click datasets with state-of-the-art deep learning models. The experimental results reveal that compared with the original optimizers with the post-processing procedure which uses the magnitude pruning method, the performance of the models can be significantly improved on the same sparsity level. Furthermore, in comparison to the cases without magnitude pruning, our methods can achieve extremely high sparsity with significantly better or highly competitive performance. The code is available at https://github.com/intelligent-machine-learning/tfplus/tree/main/tfplus.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
