DyRep:动态重新参数化的自举训练
DyRep: Bootstrapping Training with Dynamic Re-parameterization
摘要:
结构重新参数化(Rep)方法在简单的VGG风格网络上实现了显著的改进。尽管流行,目前的Rep方法只是将所有操作重新参数化到一个增强的网络中,包括那些很少有助于模型性能的操作。因此,要付出的代价是操纵这些不必要的行为的昂贵的计算开销。为了消除上述警告,我们的目标是通过设计一种动态重新参数化(DyRep)方法,将Rep技术编码到动态演变网络结构的训练过程中,以最小的成本引导训练。具体地说,我们的建议自适应地找到的操作,最有助于网络中的损失,并应用代表,以提高其代表性的能力。此外,为了抑制Rep引入的噪声和冗余操作,我们设计了一种更紧凑的重新参数化的去参数化技术。在这方面,DyRep比Rep更有效,因为它平滑地演化给定的网络,而不是构建一个过度参数化的网络。实验结果证明了我们的有效性,例如,DyRep通过ImageNet上的$2.04\%$提高了ResNet-18的准确性,并在基线上减少了$22\%$运行时间。代码可从以下网址获得:https://github.com/hunto/DyRep。
英文摘要:
Structural re-parameterization (Rep) methods achieve noticeable improvements on simple VGG-style networks. Despite the prevalence, current Rep methods simply re-parameterize all operations into an augmented network, including those that rarely contribute to the model's performance. As such, the price to pay is an expensive computational overhead to manipulate these unnecessary behaviors. To eliminate the above caveats, we aim to bootstrap the training with minimal cost by devising a dynamic re-parameterization (DyRep) method, which encodes Rep technique into the training process that dynamically evolves the network structures. Concretely, our proposal adaptively finds the operations which contribute most to the loss in the network, and applies Rep to enhance their representational capacity. Besides, to suppress the noisy and redundant operations introduced by Rep, we devise a de-parameterization technique for a more compact re-parameterization. With this regard, DyRep is more efficient than Rep since it smoothly evolves the given network instead of constructing an over-parameterized network. Experimental results demonstrate our effectiveness, e.g., DyRep improves the accuracy of ResNet-18 by $2.04\%$ on ImageNet and reduces $22\%$ runtime over the baseline. Code is available at: https://github.com/hunto/DyRep.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。