arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.03444cs.DC
收藏

LLMSched:面向复合 LLM 应用的不确定性感知负载调度

LLMSched: Uncertainty-Aware Workload Scheduling for Compound LLM Applications

Botao Zhu, Chen Chen, Xiaoyi Fan, Yifei Zhu

AI总结:

本文提出面向复合LLM应用的不确定性感知调度框架LLMSched,通过基于DAG的模型、贝叶斯网络画像与基于熵的机制量化并降低不确定性,结合JCT高效策略,将平均作业完成时间降低了14~79%。

摘要:

开发复合大型语言模型(LLM)应用正成为解决现实世界问题日益普遍的方法。在这些应用中,LLM 与各种外部模块(包括 API 甚至其他 LLM)协作,以实现复杂的智能服务。然而,我们揭示了复合 LLM 应用中固有的持续时间和结构不确定性,给 LLM 服务提供商在高效服务和调度方面带来了巨大挑战。在本文中,我们提出了 LLMSched,一种面向新兴复合 LLM 应用的不确定性感知调度框架。在 LLMSched 中,我们首先设计了一种新颖的基于 DAG 的模型来描述不确定的复合 LLM 应用。然后,我们采用贝叶斯网络对复合 LLM 应用进行全面画像,并识别出降低不确定性的阶段,同时配合基于熵的机制来量化其不确定性降低程度。结合不确定性降低策略和作业完成时间(JCT)高效方案,我们进一步提出了一种高效的调度器来降低平均 JCT。在多种代表性复合 LLM 应用上进行的仿真和测试床实验评估表明,与现有最先进的调度方案相比,LLMSched 能够将平均 JCT 降低 14~79%。

英文摘要:

Developing compound Large Language Model (LLM) applications is becoming an increasingly prevalent approach to solving real-world problems. In these applications, an LLM collaborates with various external modules, including APIs and even other LLMs, to realize complex intelligent services. However, we reveal that the intrinsic duration and structural uncertainty in compound LLM applications pose great challenges for LLM service providers in serving and scheduling them efficiently. In this paper, we propose LLMSched, an uncertainty-aware scheduling framework for emerging compound LLM applications. In LLMSched, we first design a novel DAG-based model to describe the uncertain compound LLM applications. Then, we adopt the Bayesian network to comprehensively profile compound LLM applications and identify uncertainty-reducing stages, along with an entropy-based mechanism to quantify their uncertainty reduction. Combining an uncertainty reduction strategy and a job completion time (JCT)-efficient scheme, we further propose an efficient scheduler to reduce the average JCT. Evaluation of both simulation and testbed experiments on various representative compound LLM applications shows that compared to existing state-of-the-art scheduling schemes, LLMSched can reduce the average JCT by 14~79%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑