深度权重分解：通过人工对称性的视角实现稀疏学习
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries
AI总结：
本文提出深度权重分解方法，将权重分解为多个因子并施加可微L2正则化，理论上等价于非凸稀疏正则化，并通过定制初始化和学习率策略在多种架构与数据集上优于浅层分解和剪枝方法。
摘要：
稀疏正则化技术在机器学习中已经相当成熟，但由于诸如$L_1$范数等惩罚项不可微，与随机梯度下降不兼容，其在神经网络中的应用仍然具有挑战性。一种有前景的替代方案是浅层权重分解，即将权重分解为两个因子，通过向因子添加可微的$L_2$正则化来实现$L_1$惩罚神经网络的平滑优化。在这项工作中，我们引入深度权重分解，将以往的浅层方法扩展到两个以上因子。我们从理论上建立了深度分解与非凸稀疏正则化的等价性，并分析了其对训练动态和优化的影响。由于标准训练实践带来的局限性，我们提出了一种定制的初始化方案，并确定了训练分解网络所需的重要学习率要求。我们通过在多种架构和数据集上的实验证明了深度权重分解的有效性，其表现始终优于其浅层对应方法和广泛使用的剪枝方法。
英文摘要：
Sparse regularization techniques are well-established in machine learning, yet their application in neural networks remains challenging due to the non-differentiability of penalties like the $L_1$ norm, which is incompatible with stochastic gradient descent. A promising alternative is shallow weight factorization, where weights are decomposed into two factors, allowing for smooth optimization of $L_1$-penalized neural networks by adding differentiable $L_2$ regularization to the factors. In this work, we introduce deep weight factorization, extending previous shallow approaches to more than two factors. We theoretically establish equivalence of our deep factorization with non-convex sparse regularization and analyze its impact on training dynamics and optimization. Due to the limitations posed by standard training practices, we propose a tailored initialization scheme and identify important learning rate requirements necessary for training factorized networks. We demonstrate the effectiveness of our deep weight factorization through experiments on various architectures and datasets, consistently outperforming its shallow counterpart and widely used pruning methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
