最稀疏模型难以被剪枝获得：对当前剪枝能力的揭示
Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities
AI总结：
本文通过485,838次实验发现当前剪枝算法难以恢复最稀疏模型，并指出其性能差距源于过参数化下表现差、产生不连通路径及易陷入次优解，呼吁发展真正实现网络稀疏性的新剪枝方法。
摘要：
剪枝已被视为压缩大规模模型的一种有前景的方法，但其在恢复最稀疏模型方面的有效性尚未得到探索。我们开展了大规模的485,838次实验，将一系列最先进的剪枝算法应用于我们创建的名为Cubist Spiral的合成数据集。研究结果揭示了与理想稀疏网络相比存在显著性能差距，而理想稀疏网络是通过我们提出的一种新颖的组合搜索算法识别得到的。我们将这一性能差距归因于当前剪枝算法在过参数化条件下表现不佳、倾向于在网络中诱导出不连通路径，以及即使在给定最优宽度和初始化时也容易陷入次优解。考虑到我们研究中使用的网络架构和数据集较为简单，这一差距令人担忧。我们希望本研究能推动对新剪枝技术的进一步探索，以实现真正的网络稀疏性。
英文摘要：
Pruning has emerged as a promising approach for compressing large-scale models, yet its effectiveness in recovering the sparsest of models has not yet been explored. We conducted an extensive series of 485,838 experiments, applying a range of state-of-the-art pruning algorithms to a synthetic dataset we created, named the Cubist Spiral. Our findings reveal a significant gap in performance compared to ideal sparse networks, which we identified through a novel combinatorial search algorithm. We attribute this performance gap to current pruning algorithms' poor behaviour under overparameterization, their tendency to induce disconnected paths throughout the network, and their propensity to get stuck at suboptimal solutions, even when given the optimal width and initialization. This gap is concerning, given the simplicity of the network architectures and datasets used in our study. We hope that our research encourages further investigation into new pruning techniques that strive for true network sparsity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
