实时的DNN推理的需求分层算法
Demand Layering for Real-Time DNN Inference with Minimized Memory Usage
摘要：
当执行深度神经网络（DNN）时，其模型参数在执行之前被加载到GPU内存中，从而导致显著的GPU内存负担。有研究通过利用CPU内存作为交换设备来减少GPU内存使用。然而，这种方法不适用于大多数具有集成GPU的嵌入式系统，其中CPU和GPU共享公共存储器。在这方面，我们提出了需求分层，它采用快速固态驱动器（SSD）作为GPU的共同运行伙伴，并利用DNN的逐层执行。在我们的方法中，DNN以逐层的方式加载和执行，将内存使用量最小化到单层的数量级。此外，我们开发了一个管道架构，隐藏了由层执行的交错参数加载引起的大多数额外延迟。我们的实现显示了96.5%的内存减少，平均只有14.8%的延迟开销。此外，通过利用内存延迟权衡，可以实现接近零的延迟开销（低于1 ms），内存使用量略有增加（仍然减少88.4%），显示了需求分层的巨大潜力。
英文摘要：
When executing a deep neural network (DNN), its model parameters are loaded into GPU memory before execution, incurring a significant GPU memory burden. There are studies that reduce GPU memory usage by exploiting CPU memory as a swap device. However, this approach is not applicable in most embedded systems with integrated GPUs where CPU and GPU share a common memory. In this regard, we present Demand Layering, which employs a fast solid-state drive (SSD) as a co-running partner of a GPU and exploits the layer-by-layer execution of DNNs. In our approach, a DNN is loaded and executed in a layer-by-layer manner, minimizing the memory usage to the order of a single layer. Also, we developed a pipeline architecture that hides most additional delays caused by the interleaved parameter loadings alongside layer executions. Our implementation shows a 96.5% memory reduction with just 14.8% delay overhead on average for representative DNNs. Furthermore, by exploiting the memory-delay tradeoff, near-zero delay overhead (under 1 ms) can be achieved with a slightly increased memory usage (still an 88.4% reduction), showing the great potential of Demand Layering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
