基于Lipschitz连续损失函数和加权群L0-norm约束的稀疏训练
Sparse Training with Lipschitz Continuous Loss Functions and a Weighted Group L0-norm Constraint
摘要：
本文的动机是深度神经网络训练的结构化稀疏性。研究了一类加权群L0-范数约束，给出了这类约束的投影和法锥。使用随机平滑，我们开发的零阶和一阶算法，最小化Lipschitz连续函数约束的任何闭集，可以投影到。非渐近收敛的保证被证明在预期的两个相关的收敛标准，可以被认为是近似平稳点的算法。使用所提出的算法给出了两个进一步的方法：一个具有非渐近收敛保证的高概率，和其他渐近保证几乎必然的一个平稳点。我们特别相信，这些是第一个这样的非渐近收敛的结果，约束Lipschitz连续损失函数。
英文摘要：
This paper is motivated by structured sparsity for deep neural network training. We study a weighted group L0-norm constraint, and present the projection and normal cone of this set. Using randomized smoothing, we develop zeroth and first-order algorithms for minimizing a Lipschitz continuous function constrained by any closed set which can be projected onto. Non-asymptotic convergence guarantees are proven in expectation for the proposed algorithms for two related convergence criteria which can be considered as approximate stationary points. Two further methods are given using the proposed algorithms: one with non-asymptotic convergence guarantees in high probability, and the other with asymptotic guarantees to a stationary point almost surely. We believe in particular that these are the first such non-asymptotic convergence results for constrained Lipschitz continuous loss functions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
