arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.11204cs.LGcs.AIcs.CV
收藏

层冻结和数据筛选:稀疏训练通用框架的缺失部分

Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training

Snap Inc.(Snap公司);Northeastern University(东北大学);University of Pittsburgh(匹兹堡大学)

Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, Jian Ren

摘要:

最近,稀疏训练已经成为边缘设备上高效深度学习的一个有前途的范例。目前的研究主要致力于通过进一步增加模型稀疏性来降低训练成本。然而,增加稀疏性并不总是理想的,因为它将不可避免地引入严重的精度退化在一个非常高的稀疏性水平。本文旨在探索其他可能的方向,以有效和高效地降低稀疏训练成本,同时保持准确性。为此,我们研究了两种技术,即层冻结和数据筛选。首先,层冻结方法在密集模型训练和微调方面取得了成功,但在稀疏训练领域从未采用过。然而,稀疏训练的独特特性可能会阻碍层冻结技术的结合。因此,我们分析了在稀疏训练中使用层冻结技术的可行性和潜力,发现它具有节省大量训练成本的潜力。第二,我们提出了一种数据筛选方法来进行有效的训练,通过确保在整个训练过程中只使用部分数据集,进一步降低了训练成本。我们表明,这两种技术可以很好地纳入稀疏训练算法,形成一个通用的框架,我们称之为SpFDE。我们广泛的实验表明,SpFDE可以显着降低训练成本,同时从三个维度保持准确性:权重稀疏性,层冻结和数据集筛选。

英文摘要:

Recently, sparse training has emerged as a promising paradigm for efficient deep learning on edge devices. The current research mainly devotes efforts to reducing training costs by further increasing model sparsity. However, increasing sparsity is not always ideal since it will inevitably introduce severe accuracy degradation at an extremely high sparsity level. This paper intends to explore other possible directions to effectively and efficiently reduce sparse training costs while preserving accuracy. To this end, we investigate two techniques, namely, layer freezing and data sieving. First, the layer freezing approach has shown its success in dense model training and fine-tuning, yet it has never been adopted in the sparse training domain. Nevertheless, the unique characteristics of sparse training may hinder the incorporation of layer freezing techniques. Therefore, we analyze the feasibility and potentiality of using the layer freezing technique in sparse training and find it has the potential to save considerable training costs. Second, we propose a data sieving method for dataset-efficient training, which further reduces training costs by ensuring only a partial dataset is used throughout the entire training process. We show that both techniques can be well incorporated into the sparse training algorithm to form a generic framework, which we dub SpFDE. Our extensive experiments demonstrate that SpFDE can significantly reduce training costs while preserving accuracy from three dimensions: weight sparsity, layer freezing, and dataset sieving.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑