动态稀疏训练与密集训练:图像损坏鲁棒性中的意外赢家
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
AI总结:
本文质疑密集训练在图像损坏鲁棒性上的常规优势,发现在10%至50%稀疏度下,动态稀疏训练在不增加资源成本时能始终超越密集训练,为提升深度学习鲁棒性开辟新途径。
摘要:
人们普遍认为,动态稀疏训练为人工神经网络开启了可扩展性和效率的新时代,但可能在分类任务的准确性上付出一些代价。同时,如果希望最大化人工神经网络对图像损坏的鲁棒性,密集训练被广泛认为是“事实上的”训练方法。在本文中,我们对这一常规做法提出质疑。因此,我们声称,与普遍看法相反,动态稀疏训练方法在鲁棒性准确度上能始终优于密集训练,特别是在效率方面不被视为主要目标的情况下(即稀疏度在10%至50%之间),且无需增加(甚至减少)资源成本。我们在两类数据(图像和视频)上验证了我们的主张,使用了多种传统和现代的计算机视觉深度学习架构以及三种广泛研究的动态稀疏训练算法。我们的发现揭示了动态稀疏训练一个全新且未知的益处,并为超越当前最先进水平提升深度学习鲁棒性开辟了新的可能性。
英文摘要:
It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, we claim that, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。