iDARTS：通过节点归一化和去相关离散化改进DARTS
iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
摘要：
微分ARchiToggle搜索（DARTS）使用网络表示的连续松弛，并在GPU天内将神经架构搜索（NAS）大幅加速近数千倍。然而，DARTS的搜索过程是不稳定的，当训练时期变大时，搜索过程会严重退化，从而限制了其应用。在本文中，我们认为这种退化问题是由不同节点之间的不平衡范数和各种操作的高度相关输出引起的。然后，我们提出了一个改进版本的DARTS，即iDARTS，以处理这两个问题。在训练阶段，它引入节点归一化来保持范数平衡。在离散化阶段，基于节点的输出与去相关操作之间的相似性而不是架构参数的值来近似连续架构。在CIFAR-10和ImageNet上进行了广泛的评估，在0.2和1.9 GPU-天内，架构搜索的出错率分别为2.25%和24.7%，表明了该方法的有效性。进一步的分析还表明，iDARTS具有鲁棒性和泛化能力优于其他基于DARTS的同行。
英文摘要：
Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this paper, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25\% and 24.7\% are reported within 0.2 and 1.9 GPU-day for architecture search respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
