arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.07856cs.LGcs.AI
收藏

学习率对深度学习模型抗噪性能的影响

Impact of Learning Rate on Noise Resistant Property of Deep Learning Models

Prairie View A&M University(普雷里维尤农工大学);Texas A&M University System(得克萨斯农工大学系统)

Omobayode Fagbohungbe, Lijun Qian

摘要:

由于其快速的计算速度和出色的能效,近年来对模拟计算的兴趣大幅增长,这对于深度学习推理的亚瓦级功率包络中的边缘和物联网设备非常重要。然而,由于模拟计算中存在的固有噪声,深度学习模型遭受的显著性能下降可能会限制其在关键任务应用中的使用。因此,有必要了解关键模型超参数选择对模型抗噪声特性的影响。这种需求至关重要,因为获得的洞察力可用于设计对模拟噪声具有鲁棒性的深度学习模型。在本文中,学习率,一个关键的设计选择,抗噪声性能的影响进行了研究。该研究通过首先使用不同的学习率训练深度学习模型来实现。此后,模型被注入模拟噪声,并且通过测量由于模拟噪声引起的性能劣化来检查所得到的模型的抗噪声特性。结果表明,存在一个最佳的学习率值,实现了模型的预测性能和模型的抗噪声性能之间的良好平衡。此外,所观察到的现象的理论依据。

英文摘要:

The interest in analog computation has grown tremendously in recent years due to its fast computation speed and excellent energy efficiency, which is very important for edge and IoT devices in the sub-watt power envelope for deep learning inferencing. However, significant performance degradation suffered by deep learning models due to the inherent noise present in the analog computation can limit their use in mission-critical applications. Hence, there is a need to understand the impact of critical model hyperparameters choice on the resulting model noise-resistant property. This need is critical as the insight obtained can be used to design deep learning models that are robust to analog noise. In this paper, the impact of the learning rate, a critical design choice, on the noise-resistant property is investigated. The study is achieved by first training deep learning models using different learning rates. Thereafter, the models are injected with analog noise and the noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property. Furthermore, the theoretical justification of the observed phenomenon is provided.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑