arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.01611cs.DCcs.AI
收藏

基于GPU空间划分的多模型机器学习推理服务

Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning

KAIST(韩国科学技术院)

Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, Jaehyuk Huh

摘要:

随着机器学习技术被应用于越来越广泛的应用,高吞吐量的机器学习(ML)推理服务器对于在线服务应用已经变得至关重要。这种ML推理服务器提出了两个挑战:首先,它们必须为每个请求提供有限的延迟,以支持一致的服务级别目标(SLO),其次,它们可以在系统中服务于多个异构ML模型,因为某些任务涉及调用多个模型,并且合并多个模型可以提高系统利用率。针对ML推理服务器的这两个需求,提出了一种新的多模型ML推理服务器的ML推理调度框架。该论文首先表明,在SLO约束下,当前的GPU并未充分用于ML推理任务。为了最大限度地提高推理服务器的资源效率,本文提出的一个关键机制是利用硬件支持的GPU资源的空间划分。通过分区机制,创建了一个新的GPU资源抽象层,并使用可配置的GPU资源。调度器将请求分配给虚拟GPU,称为gpu-lets,具有最有效的资源量。本文还研究了当两个ML任务在GPU中并发运行时潜在干扰影响的补救措施。我们的原型实现证明,空间分区提高吞吐量的平均102.6%,同时满足SLO。

英文摘要:

As machine learning techniques are applied to a widening range of applications, high throughput machine learning (ML) inference servers have become critical for online service applications. Such ML inference servers pose two challenges: first, they must provide a bounded latency for each request to support consistent service-level objective (SLO), and second, they can serve multiple heterogeneous ML models in a system as certain tasks involve invocation of multiple models and consolidating multiple models can improve system utilization. To address the two requirements of ML inference servers, this paper proposes a new ML inference scheduling framework for multi-model ML inference servers. The paper first shows that with SLO constraints, current GPUs are not fully utilized for ML inference tasks. To maximize the resource efficiency of inference servers, a key mechanism proposed in this paper is to exploit hardware support for spatial partitioning of GPU resources. With the partitioning mechanism, a new abstraction layer of GPU resources is created with configurable GPU resources. The scheduler assigns requests to virtual GPUs, called gpu-lets, with the most effective amount of resources. The paper also investigates a remedy for potential interference effects when two ML tasks are running concurrently in a GPU. Our prototype implementation proves that spatial partitioning enhances throughput by 102.6% on average while satisfying SLOs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑