FracTrain：在时间和空间上分数化地挤出比特节省以实现高效 DNN 训练
FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training
AI总结：
针对边缘 DNN 训练资源受限问题，提出 FracTrain，通过渐进式与输入自适应动态分数量化降低计算、能耗和延迟，并保持精度。
摘要：
深度神经网络（DNN）近期的突破推动了对具备现场学习能力的智能边缘设备的巨大需求，然而，由于边缘可用资源有限以及最先进（SOTA）DNN 所需的海量训练成本，此类系统的实际实现仍然是一项挑战。由于降低精度是提升训练时间/能效的最有效手段之一，低精度 DNN 训练受到越来越多的关注。在本文中，我们从一个正交方向展开探索：如何从最冗余的比特级别中分数化地挤出更多训练成本节省，沿训练轨迹逐步进行，并针对每个输入动态进行。具体而言，我们提出 FracTrain，它集成了（i）渐进式分数量化，逐步提高激活、权重和梯度的精度，直到最终训练阶段才达到 SOTA 静态量化 DNN 训练的精度；以及（ii）动态分数量化，以输入自适应的方式为每一层的激活和梯度分配精度，仅用于“分数化”更新层参数。大量仿真和消融研究（六个模型、四个数据集，以及包括标准训练、适配和微调在内的三种训练设置）验证了 FracTrain 在降低 DNN 训练计算成本和硬件量化能耗/延迟方面的有效性，同时达到相当或更好（-0.12%~+1.87%）的精度。例如，在 CIFAR-10 上训练 ResNet-74 时，与最佳 SOTA 基线相比，FracTrain 分别实现 77.6% 和 53.5% 的计算成本与训练延迟节省，同时达到相当（-0.07%）的精度。我们的代码可在 https://github.com/RICE-EIC/FracTrain 获取。
英文摘要：
Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendous demand for intelligent edge devices featuring on-site learning, while the practical realization of such systems remains a challenge due to the limited resources available at the edge and the required massive training costs for state-of-the-art (SOTA) DNNs. As reducing precision is one of the most effective knobs for boosting training time/energy efficiency, there has been a growing interest in low-precision DNN training. In this paper, we explore from an orthogonal direction: how to fractionally squeeze out more training cost savings from the most redundant bit level, progressively along the training trajectory and dynamically per input. Specifically, we propose FracTrain that integrates (i) progressive fractional quantization which gradually increases the precision of activations, weights, and gradients that will not reach the precision of SOTA static quantized DNN training until the final training stage, and (ii) dynamic fractional quantization which assigns precisions to both the activations and gradients of each layer in an input-adaptive manner, for only "fractionally" updating layer parameters. Extensive simulations and ablation studies (six models, four datasets, and three training settings including standard, adaptation, and fine-tuning) validate the effectiveness of FracTrain in reducing computational cost and hardware-quantified energy/latency of DNN training while achieving a comparable or better (-0.12%~+1.87%) accuracy. For example, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and 53.5% computational cost and training latency savings, respectively, compared with the best SOTA baseline, while achieving a comparable (-0.07%) accuracy. Our codes are available at: https://github.com/RICE-EIC/FracTrain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
