CPT：通过循环精度实现高效深度神经网络训练
CPT: Efficient Deep Neural Network Training via Cyclic Precision
AI总结：
本文提出循环精度训练（CPT），在训练过程中周期性地在两个边界精度之间切换，以类似动态学习率的方式提升深度神经网络训练的时间与能效，并通过实验验证其可改善泛化性能并降低训练方差。
摘要：
低精度深度神经网络（DNN）训练已获得极大关注，因为降低精度是提升 DNN 训练时间/能效最有效的手段之一。在本文中，受近期关于理解 DNN 训练的研究发现启发，我们尝试从新的视角探索低精度训练：我们推测 DNN 的精度在训练过程中可能具有与学习率类似的作用，并主张沿训练轨迹动态调整精度，以进一步提升 DNN 训练的时间/能效。具体而言，我们提出循环精度训练（CPT），在两个边界值之间循环改变精度，这两个边界值可以通过在最初几个训练轮次内进行简单的精度范围测试来确定。在五个数据集和十一个模型上的大量仿真和消融研究表明，CPT 的有效性在各种模型/任务（包括分类和语言建模）中保持一致。此外，通过实验和可视化，我们表明 CPT 有助于（1）收敛到更宽的极小值，从而获得更低的泛化误差；（2）降低训练方差，我们认为这为同时改善 DNN 训练的优化效果和效率开辟了一个新的设计维度。我们的代码可在以下地址获取：https://github.com/RICE-EIC/CPT。
英文摘要：
Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at: https://github.com/RICE-EIC/CPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
