arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.11184cs.CV
收藏

通过随机梯度下降神经进化优化深度神经网络

Optimizing Deep Neural Networks through Neuroevolution with Stochastic Gradient Descent

Donghua University(东华大学);CSIRO(澳大利亚联邦科学与工业研究组织)

Haichao Zhang, Kuangrong Hao, Lei Gao, Bing Wei, Xuesong Tang

摘要:

深度神经网络(DNN)在计算机视觉中取得了显著的成功;然而,训练DNN以获得令人满意的性能仍然具有挑战性,并且对用于训练的优化算法的经验选择敏感。随机梯度下降(SGD)在通过调整神经网络权重以最小化DNN损失函数来训练DNN中占主导地位。作为一种替代方法,神经进化更符合进化过程,并提供了一些关键的功能,往往是在SGD,如启发式黑盒搜索策略的基础上,在神经进化的个人合作。本文提出了一种新的方法,结合了神经进化和SGD的优点,使进化搜索,并行探索,并为最佳DNN的有效探针。为了克服个体间相似权值更新的问题,提高种群多样性,提出了一种分层聚类抑制算法。我们基于四个公开的数据集在四个代表性的DNN中实现了所提出的方法。实验结果表明,在所有数据集上,采用该方法优化的四个DNN的性能均优于仅采用SGD优化的相应DNN。通过该方法优化的DNN的性能也优于最先进的深度网络。这项工作也为研究人工通用智能提供了一个有意义的尝试。

英文摘要:

Deep neural networks (DNNs) have achieved remarkable success in computer vision; however, training DNNs for satisfactory performance remains challenging and suffers from sensitivity to empirical selections of an optimization algorithm for training. Stochastic gradient descent (SGD) is dominant in training a DNN by adjusting neural network weights to minimize the DNNs loss function. As an alternative approach, neuroevolution is more in line with an evolutionary process and provides some key capabilities that are often unavailable in SGD, such as the heuristic black-box search strategy based on individual collaboration in neuroevolution. This paper proposes a novel approach that combines the merits of both neuroevolution and SGD, enabling evolutionary search, parallel exploration, and an effective probe for optimal DNNs. A hierarchical cluster-based suppression algorithm is also developed to overcome similar weight updates among individuals for improving population diversity. We implement the proposed approach in four representative DNNs based on four publicly-available datasets. Experiment results demonstrate that the four DNNs optimized by the proposed approach all outperform corresponding ones optimized by only SGD on all datasets. The performance of DNNs optimized by the proposed approach also outperforms state-of-the-art deep networks. This work also presents a meaningful attempt for pursuing artificial general intelligence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑