BayesFT:容错神经网络结构的贝叶斯优化
BayesFT: Bayesian Optimization for Fault Tolerant Neural Network Architecture
摘要:
为了在资源有限的场景中部署深度学习算法,新兴的设备电阻式随机存取存储器(ReRAM)被认为是通过模拟计算有前途的。然而,由于制造、热噪声等多因素的影响,ReRAM神经网络的权值会发生漂移,限制了ReRAM的实用性。本文提出了一种新的容错神经网络结构(BayesFT)贝叶斯优化方法。对于神经结构搜索空间的设计,而不是在整个可行的神经结构搜索空间上进行神经结构搜索,我们首先系统地探索了不同神经网络组件的权重漂移容忍度,例如dropout,归一化,层数和激活函数,其中dropout被发现能够提高神经网络对权重漂移的鲁棒性。基于我们的分析,我们提出了一个有效的搜索空间,只搜索每一层的辍学率。然后,我们使用贝叶斯优化搜索的最佳神经网络结构的重量漂移的鲁棒性。实证实验表明,我们的算法框架已经超过了国家的最先进的方法高达10倍的各种任务,如图像分类和目标检测。
英文摘要:
To deploy deep learning algorithms on resource-limited scenarios, an emerging device-resistive random access memory (ReRAM) has been regarded as promising via analog computing. However, the practicability of ReRAM is primarily limited due to the weight drifting of ReRAM neural networks due to multi-factor reasons, including manufacturing, thermal noises, and etc. In this paper, we propose a novel Bayesian optimization method for fault tolerant neural network architecture (BayesFT). For neural architecture search space design, instead of conducting neural architecture search on the whole feasible neural architecture search space, we first systematically explore the weight drifting tolerance of different neural network components, such as dropout, normalization, number of layers, and activation functions in which dropout is found to be able to improve the neural network robustness to weight drifting. Based on our analysis, we propose an efficient search space by only searching for dropout rates for each layer. Then, we use Bayesian optimization to search for the optimal neural architecture robust to weight drifting. Empirical experiments demonstrate that our algorithmic framework has outperformed the state-of-the-art methods by up to 10 times on various tasks, such as image classification and object detection.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。