arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.16757cs.LGcs.AIcs.DC
收藏

SwapNet:在内存预算不足的边缘AI设备上实现DNN推理的高效交换机制

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

City University of Hong Kong(香港城市大学)

Kun Wang, Jiani Cao, Zimu Zhou, Zhenjiang Li

AI总结:

针对边缘AI设备内存不足限制DNN部署的问题,本研究提出SwapNet中间件,通过分块交换并消除冗余内存操作,在超内存需求2.32-5.81倍时仍实现接近充足内存的推理延迟。

摘要:

在边缘人工智能(AI)设备上运行深度神经网络(DNN)可支撑各类自主移动计算应用。但边缘AI设备的内存预算限制了此类应用中可部署DNN的数量与复杂度。现有方案如模型压缩或云端卸载虽能降低DNN推理的内存占用,却以牺牲模型精度或自主性为代价。为规避这些缺陷,我们将DNN划分为多个块并按序换入换出,使大型DNN可在较小内存预算下运行。然而,由于边缘AI设备的DNN开发生态中存在冗余内存操作,直接在边缘AI设备上进行简单交换会产生显著延迟。为此,我们开发了SwapNet——一款面向边缘AI设备的高效DNN块交换中间件。我们系统性地消除了块交换过程中的不必要内存操作,同时保持与边缘AI设备的深度学习框架、GPU后端及硬件架构的兼容性。我们还通过一种多DNN调度方案进一步展示了SwapNet的实用性。在三类应用的11项DNN推理任务上的评估表明,即便DNN所需内存超出可用预算的2.32倍至5.81倍,SwapNet的延迟仍与内存充足时的情况几乎一致。SwapNet的设计也为未来在边缘AI设备上部署大语言模型(LLMs)提供了新颖且可行的思路。

英文摘要:

Executing deep neural networks (DNNs) on edge artificial intelligence (AI) devices enables various autonomous mobile computing applications. However, the memory budget of edge AI devices restricts the number and complexity of DNNs allowed in such applications. Existing solutions, such as model compression or cloud offloading, reduce the memory footprint of DNN inference at the cost of decreased model accuracy or autonomy. To avoid these drawbacks, we divide DNN into blocks and swap them in and out in order, such that large DNNs can execute within a small memory budget. Nevertheless, naive swapping on edge AI devices induces significant delays due to the redundant memory operations in the DNN development ecosystem for edge AI devices. To this end, we develop SwapNet, an efficient DNN block swapping middleware for edge AI devices. We systematically eliminate the unnecessary memory operations during block swapping while retaining compatible with the deep learning frameworks, GPU backends, and hardware architectures of edge AI devices. We further showcase the utility of SwapNet via a multi-DNN scheduling scheme. Evaluations on eleven DNN inference tasks in three applications demonstrate that SwapNet achieves almost the same latency as the case with sufficient memory even when DNNs demand 2.32x to 5.81x memory beyond the available budget. The design of SwapNet also provides novel and feasible insights for deploying large language models (LLMs) on edge AI devices in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑