TinyML深度神经网络推理中用于内存优化的融合深度平铺
Fused Depthwise Tiling for Memory Optimization in TinyML Deep Neural Network Inference
AI总结:
针对TinyML设备上DNN推理内存受限问题,提出融合深度平铺(FDT)方法,在无运行时开销下减少内存使用,适用更多网络层,并通过端到端流程自动配置,在多个模型上实现显著内存优化。
摘要:
随着TinyML的出现,深度神经网络(DNN)推理的内存优化变得高度相关,TinyML是指在微型、低功耗微控制器上部署DNN推理任务。诸如音频关键词检测或基于雷达的手势识别等应用受到此类微型设备上有限内存的严重制约,因为DNN推理需要大型中间运行时缓冲区来存储激活值和其他中间数据,从而导致高内存使用率。在本文中,我们提出了一种用于DNN内存优化的新型融合深度平铺(FDT)方法,与现有的平铺方法相比,该方法在不产生任何运行时开销的情况下减少了内存使用。FDT比专注于卷积的现有平铺方法适用于更多种类的网络层。它通过减少以前无法减少的模型的内存,显著改善了TinyML内存优化,并为使用现有方法显示出高运行时开销的模型额外提供了替代设计点。为了识别最佳平铺配置,提出了一种带有新路径发现方法的端到端流程,该方法以完全自动化的方式应用FDT和现有平铺方法,包括操作的调度和内存中缓冲区布局的规划。在七个评估模型中,FDT在现有平铺方法无法应用的情况下,通过76.2%和18.1%的比例显著减少了两个模型的内存。另外两个模型在使用现有方法时显示出显著的运行时开销,而FDT提供了无开销但内存节省减少的替代设计点。
英文摘要:
Memory optimization for deep neural network (DNN) inference gains high relevance with the emergence of TinyML, which refers to the deployment of DNN inference tasks on tiny, low-power microcontrollers. Applications such as audio keyword detection or radar-based gesture recognition are heavily constrained by the limited memory on such tiny devices because DNN inference requires large intermediate run-time buffers to store activations and other intermediate data, which leads to high memory usage. In this paper, we propose a new Fused Depthwise Tiling (FDT) method for the memory optimization of DNNs, which, compared to existing tiling methods, reduces memory usage without inducing any run time overhead. FDT applies to a larger variety of network layers than existing tiling methods that focus on convolutions. It improves TinyML memory optimization significantly by reducing memory of models where this was not possible before and additionally providing alternative design points for models that show high run time overhead with existing methods. In order to identify the best tiling configuration, an end-to-end flow with a new path discovery method is proposed, which applies FDT and existing tiling methods in a fully automated way, including the scheduling of the operations and planning of the layout of buffers in memory. Out of seven evaluated models, FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied. Two other models showed a significant run time overhead with existing methods and FDT provided alternative design points with no overhead but reduced memory savings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。