arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.07109cs.LGcs.AI
收藏

借助TinyPropv2推进端侧神经网络训练:动态、稀疏且高效的反向传播

Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation

Hahn-Schickard(哈恩-施卡德机构);Technical University of Munich(慕尼黑工业大学)

Marcus Rüb, Axel Sikora, Daniel Mueller-Gritschneder

AI总结:

本研究提出面向低功耗微控制器的端侧训练算法TinyPropv2,通过动态调整稀疏度的稀疏反向传播降低计算量,在多数据集上精度降幅约1%,计算量最低仅为全量训练的10%,优于其他稀疏训练方法,适用于物联网嵌入式设备。

摘要:

本研究提出TinyPropv2,这是一种针对深度神经网络端侧学习优化的创新算法,专为低功耗微控制器单元设计。TinyPropv2通过动态调整稀疏度水平改进了稀疏反向传播,包括具备选择性跳过训练步骤的能力。该特性在不大幅损失精度的前提下显著降低了计算量。我们在CIFAR-10、CIFAR100、Flower、Food、Speech Command、MNIST、HAR、DCASE2020等多样数据集上开展了全面评估,结果显示TinyPropv2的性能与全量训练方法接近,多数情况下平均精度下降仅约1%。例如,与全量训练相比,TinyPropv2的精度降幅极小,在CIFAR-10上仅为0.82%,在CIFAR100上为1.07%。在计算量方面,TinyPropv2实现了显著降低,部分场景下仅需全量训练所需计算量的10%,且性能始终优于其他稀疏训练方法。这些研究结果凸显了TinyPropv2在保持高精度的同时高效管理计算资源的能力,使其成为物联网生态中先进嵌入式设备应用的优势解决方案。

英文摘要:

This study introduces TinyPropv2, an innovative algorithm optimized for on-device learning in deep neural networks, specifically designed for low-power microcontroller units. TinyPropv2 refines sparse backpropagation by dynamically adjusting the level of sparsity, including the ability to selectively skip training steps. This feature significantly lowers computational effort without substantially compromising accuracy. Our comprehensive evaluation across diverse datasets CIFAR 10, CIFAR100, Flower, Food, Speech Command, MNIST, HAR, and DCASE2020 reveals that TinyPropv2 achieves near-parity with full training methods, with an average accuracy drop of only around 1 percent in most cases. For instance, against full training, TinyPropv2's accuracy drop is minimal, for example, only 0.82 percent on CIFAR 10 and 1.07 percent on CIFAR100. In terms of computational effort, TinyPropv2 shows a marked reduction, requiring as little as 10 percent of the computational effort needed for full training in some scenarios, and consistently outperforms other sparse training methodologies. These findings underscore TinyPropv2's capacity to efficiently manage computational resources while maintaining high accuracy, positioning it as an advantageous solution for advanced embedded device applications in the IoT ecosystem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑