arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.12279cs.LGstat.ML
收藏

非结构化剪枝的损失模型研究

Revisiting Loss Modelling for Unstructured Pruning

Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Insight Centre for Data Analytics(洞察数据分析中心);Dublin City University(都柏林城市大学);Facebook AI Research(脸书人工智能研究院);Canadian Institute for Advanced Research (CIFAR)(加拿大高等研究院)

César Laurent, Camille Ballas, Thomas George, Nicolas Ballas, Pascal Vincent

摘要:

通过从深度神经网络中移除参数,非结构化修剪方法旨在减少内存占用和计算成本,同时保持预测精度。为了解决这个棘手的问题,这些方法中的许多方法使用一阶或二阶泰勒展开来对损失情况进行建模,以识别可以丢弃哪些参数。我们重新审视非结构化修剪的损失建模:我们展示了确保修剪步骤局部性的重要性。我们系统地比较了一阶和二阶泰勒展开,并实证表明,两者都可以达到类似的性能水平。最后,我们表明,更好地保持原始网络功能并不一定转移到更好的性能网络微调后,这表明,只考虑修剪的损失的影响可能不是一个足够的目标,设计良好的修剪标准。

英文摘要:

By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps. We systematically compare first and second order Taylor expansions and empirically show that both can reach similar levels of performance. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑