arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.04104cs.ARcs.AIcs.LGcs.OS
收藏

面向云平台的神经处理单元硬件辅助虚拟化

Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Google(谷歌公司)

Yuqi Xue, Yiqi Liu, Lifeng Nai, Jian Huang

AI总结:

针对云平台NPU虚拟化缺乏系统抽象与架构支持的问题,提出全栈一体化框架Neu10,通过vNPU抽象、资源分配器与ISA扩展实现细粒度虚拟化,相较现有方案显著提升吞吐量、降低尾延迟并提高资源利用率。

摘要:

如今云平台已部署神经处理单元(NPU)等硬件加速器,为机器学习(ML)推理服务提供算力支持。为在保障合理服务质量的同时最大化资源利用率,一种自然的思路是对NPU进行虚拟化,以实现多租户ML服务的高效资源共享。然而,为现代云平台实现NPU虚拟化并非易事,这不仅是因为NPU硬件缺乏系统抽象支持,还因为现有架构和指令集架构(ISA)缺少对虚拟化NPU的细粒度动态算子调度的支撑。\n 我们提出了Neu10,一个一体化的NPU虚拟化框架,从软硬件全栈层面研究NPU虚拟化技术。Neu10包含三个部分:(1)名为vNPU的灵活NPU抽象,可对物理NPU(pNPU)中的异构计算单元进行细粒度虚拟化;(2)vNPU资源分配器,支持按量付费的计算模式以及灵活的vNPU到pNPU的映射,以提升资源利用率和成本效益;(3)对现代NPU架构的ISA扩展,用于为多个vNPU的细粒度张量算子调度提供便利。我们基于生产级NPU模拟器实现了Neu10。实验表明,与现有最先进的NPU共享方法相比,Neu10将ML推理服务的吞吐量提升了最高1.4×,尾延迟降低了最高4.6×,同时NPU利用率平均提升了1.2×。

英文摘要:

Cloud platforms today have been deploying hardware accelerators like neural processing units (NPUs) for powering machine learning (ML) inference services. To maximize the resource utilization while ensuring reasonable quality of service, a natural approach is to virtualize NPUs for efficient resource sharing for multi-tenant ML services. However, virtualizing NPUs for modern cloud platforms is not easy. This is not only due to the lack of system abstraction support for NPU hardware, but also due to the lack of architectural and ISA support for enabling fine-grained dynamic operator scheduling for virtualized NPUs. We present Neu10, a holistic NPU virtualization framework. We investigate virtualization techniques for NPUs across the entire software and hardware stack. Neu10 consists of (1) a flexible NPU abstraction called vNPU, which enables fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU resource allocator that enables pay-as-you-go computing model and flexible vNPU-to-pNPU mappings for improved resource utilization and cost-effectiveness; (3) an ISA extension of modern NPU architecture for facilitating fine-grained tensor operator scheduling for multiple vNPUs. We implement Neu10 based on a production-level NPU simulator. Our experiments show that Neu10 improves the throughput of ML inference services by up to 1.4$\times$ and reduces the tail latency by up to 4.6$\times$, while improving the NPU utilization by 1.2$\times$ on average, compared to state-of-the-art NPU sharing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑