spred：用 SGD 求解 $L_1$ 惩罚
spred: Solving $L_1$ Penalty with SGD
AI总结：
本文提出 spred，通过简单重参数化和 SGD 精确求解 $L_1$ 惩罚，并在基因选择与神经网络压缩任务中验证其有效性。
摘要：
我们提出通过一种简单的重参数化和直接的随机梯度下降，来最小化带有 $L_1$ 约束的一般可微目标函数。我们的方案是对以往思想的直接推广，即 $L_1$ 惩罚可能等价于一种带权重衰减的可微重参数化。我们证明，所提出的方法 spred 是 $L_1$ 的精确可微求解器，并且对于一般的非凸函数，该重参数化技巧完全是“良性”的。在实践中，我们通过以下两方面展示该方法的实用性：（1）训练稀疏神经网络以执行基因选择任务，这涉及在极高维空间中寻找相关特征；（2）神经网络压缩任务，以往将 $L_1$ 惩罚应用于该任务的尝试均未成功。从概念上讲，我们的结果弥合了深度学习中的稀疏性与传统统计学习之间的差距。
英文摘要：
We propose to minimize a generic differentiable objective with $L_1$ constraint using a simple reparametrization and straightforward stochastic gradient descent. Our proposal is the direct generalization of previous ideas that the $L_1$ penalty may be equivalent to a differentiable reparametrization with weight decay. We prove that the proposed method, \textit{spred}, is an exact differentiable solver of $L_1$ and that the reparametrization trick is completely ``benign" for a generic nonconvex function. Practically, we demonstrate the usefulness of the method in (1) training sparse neural networks to perform gene selection tasks, which involves finding relevant features in a very high dimensional space, and (2) neural network compression task, to which previous attempts at applying the $L_1$-penalty have been unsuccessful. Conceptually, our result bridges the gap between the sparsity in deep learning and conventional statistical learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
