Fiddler：面向混合专家模型快速推理的 CPU-GPU 协同调度
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
AI总结：
本文提出Fiddler，一个在GPU资源受限环境下的MoE模型高效推理系统，通过策略性调度CPU和GPU确定最优执行策略，在单批次推理、长预填充和束搜索等所有场景下均优于现有系统。
摘要：
具有混合专家架构的大型语言模型在各类任务上展现出了良好的性能。然而，由于模型规模庞大，在GPU内存不足的资源受限环境中运行它们具有挑战性。一些现有系统提出使用CPU资源来解决这一问题，但它们要么受困于在CPU和GPU之间频繁移动数据的巨大开销，要么未能考虑CPU和GPU的不同特性。本文提出了Fiddler，一个在GPU资源有限的情况下用于MoE模型的资源高效推理系统。Fiddler通过确定最优执行策略来策略性地利用CPU和GPU资源。我们的评估表明，与针对特定场景（如单批次推理或长预填充）进行优化的最先进系统不同，Fiddler在所有场景下均表现更佳。与不同基线相比，Fiddler在单批次推理中实现了1.26倍的加速，在长预填充处理中实现了1.30倍的加速，在束搜索推理中实现了11.57倍的加速。Fiddler的代码已在 https://github.com/efeslab/fiddler 公开。
英文摘要：
Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
