掩码、符号与学习率回溯
Masks, Signs, And Learning Rate Rewinding
AI总结：
本文通过解耦掩码学习与参数优化，揭示 LRR 借助早期符号翻转和抗符号扰动能力，更有效地识别并优化多样化稀疏掩码，并在单隐藏神经元情形下证明其优于 IMP。
摘要：
学习率回溯（Learning Rate Rewinding，LRR）已被确立为迭代幅度剪枝（Iterative Magnitude Pruning，IMP）的一种强大变体，用于在深度过参数化神经网络中寻找彩票。尽管这两种迭代剪枝方案都将结构学习与参数学习耦合在一起，但理解 LRR 如何在这两个方面表现优异，有助于我们设计更灵活的深度学习算法，从而优化多样化的稀疏架构集合。为此，我们开展实验，将掩码学习与参数优化的作用解耦，并研究二者如何从过参数化中获益。LRR 能够在早期翻转参数符号，并对符号扰动保持稳健，这似乎不仅使其在掩码识别方面更有效，也使其更擅长优化包括随机掩码在内的多样化掩码集合。为支持这一假设，我们在一个简化的单隐藏神经元设定中证明，LRR 比 IMP 在更多情况下取得成功，因为它能够逃离最初有问题的符号配置。
英文摘要：
Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks. While both iterative pruning schemes couple structure and parameter learning, understanding how LRR excels in both aspects can bring us closer to the design of more flexible deep learning algorithms that can optimize diverse sets of sparse architectures. To this end, we conduct experiments that disentangle the effect of mask learning and parameter optimization and how both benefit from overparameterization. The ability of LRR to flip parameter signs early and stay robust to sign perturbations seems to make it not only more effective in mask identification but also in optimizing diverse sets of masks, including random ones. In support of this hypothesis, we prove in a simplified single hidden neuron setting that LRR succeeds in more cases than IMP, as it can escape initially problematic sign configurations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
