基于去隐层的深度神经网络自整定
Auto-tuning of Deep Neural Networks by Conflicting Layer Removal
摘要:
设计神经网络架构是一项具有挑战性的任务,并且知道模型的哪些特定层必须被调整以提高性能几乎是一个谜。在本文中,我们介绍了一种新的方法来识别降低训练模型测试精度的层。早在训练开始时就检测到分层。在最坏的情况下,我们证明了这样一个层可能会导致一个根本无法训练的网络。提供了一个理论分析,这些层的起源,导致整体网络性能较低,这是我们广泛的经验评估的补充。更准确地说,我们确定了那些降低性能的层,因为它们会产生我们称之为冲突的训练包。我们将证明,经过训练的残差网络的大约60%的层可以完全从架构中删除,而测试错误不会显着增加。我们将进一步提出一种新的神经架构搜索(NAS)算法,该算法在训练开始时识别冲突层。与更复杂的最先进架构相比,我们的自动调整算法发现的架构实现了具有竞争力的准确度值,同时大大减少了不同计算机视觉任务的内存消耗和推理时间。源代码可在https://github.com/peerdavid/conflicting-bundles上获得
英文摘要:
Designing neural network architectures is a challenging task and knowing which specific layers of a model must be adapted to improve the performance is almost a mystery. In this paper, we introduce a novel methodology to identify layers that decrease the test accuracy of trained models. Conflicting layers are detected as early as the beginning of training. In the worst-case scenario, we prove that such a layer could lead to a network that cannot be trained at all. A theoretical analysis is provided on what is the origin of those layers that result in a lower overall network performance, which is complemented by our extensive empirical evaluation. More precisely, we identified those layers that worsen the performance because they would produce what we name conflicting training bundles. We will show that around 60% of the layers of trained residual networks can be completely removed from the architecture with no significant increase in the test-error. We will further present a novel neural-architecture-search (NAS) algorithm that identifies conflicting layers at the beginning of the training. Architectures found by our auto-tuning algorithm achieve competitive accuracy values when compared against more complex state-of-the-art architectures, while drastically reducing memory consumption and inference time for different computer vision tasks. The source code is available on https://github.com/peerdavid/conflicting-bundles
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。