arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.08980cs.DC
收藏

借助MultiWorld实现弹性模型服务

Enabling Elastic Model Serving with MultiWorld

Myungjin Lee, Akshay Jajoo, Ramana Rao Kompella

AI总结:

针对大模型推理服务动态负载、故障影响用户且传统CCL无法弹性扩缩容的问题,提出支持工作节点级容错与在线扩缩容的MultiWorld方案,多数场景下吞吐量损失仅1.4%-4.3%。

摘要:

过去几年间,机器学习模型的参数量呈指数级增长,如今我们正见证万亿参数模型的兴起。这类大模型无法装入单块GPU,因此需要跨多块GPU甚至多台主机进行分区部署。诸如NCCL之类的高性能集合通信库(CCL)对于充分利用昂贵的GPU资源至关重要。然而,CCL并不十分适配推理场景。与训练阶段使用固定数量的GPU资源处理固定工作负载(例如输入数据集)不同,推理工作负载会随时间动态变化。服务期间的故障也会直接影响单个用户的体验。相比之下,CCL进程组中的工作节点共享同一个故障域,且进程组无法随工作负载增长而扩容。模型服务的独特特性与CCL的固有属性之间存在差距,使得大模型难以实现弹性服务。为弥合这一差距,我们提出了MultiWorld,它能够以工作节点为粒度,为模型服务提供容错能力和在线扩缩容能力。我们的评估表明,在大多数测试场景下,启用这些新功能带来的开销很小(吞吐量损失为1.4%-4.3%)。

英文摘要:

Machine learning models have been exponentially growing in terms of their parameter size over the past few years. We are now seeing the rise of trillion-parameter models. The large models cannot fit into a single GPU and thus require partitioned deployment across GPUs and even hosts. A high-performance collective communication library (CCL) such as NCCL is essential to fully utilize expensive GPU resources. However, CCL is not a great fit for inference. Unlike training for which a fixed amount of GPU resources is used for fixed workloads (e.g., input datasets), the inference workloads can change dynamically over time. Failures at the serving time can also impact individual user's experiences directly. In contrast, workers in a CCL process group share a single fault domain and the process group cannot grow as the workloads increase. The gap between the unique characteristics of model serving and CCL's nature makes it hard to serve large models elastically. To bridge the gap, we propose MultiWorld that enables fault tolerance and online scaling at the granularity of workers for model serving. Our evaluation showcases that enabling these new functionalities incurs small overheads (1.4-4.3% throughput loss) for most of the scenarios we tested.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑