现在别阻止我:面向 LLM 的基于嵌入的调度
Don't Stop Me Now: Embedding Based Scheduling for LLMs
AI总结:
本文提出 TRAIL,利用目标 LLM 的 token 内部嵌入训练轻量分类器预测剩余输出长度,并结合考虑内存开销的受限抢占 SRPT 策略,以降低交互式 LLM 请求完成时间。
摘要:
高效调度对于交互式大型语言模型(LLM)应用至关重要,在这类应用中,较低的请求完成时间会直接影响用户参与度。诸如最短剩余处理时间(SRPT)等基于大小的调度算法,旨在利用已知或估计的请求大小,并允许服务时间更短的新到达作业抢占当前作业,从而降低平均请求完成时间。然而,将基于大小的调度应用于 LLM 系统时会出现两个主要挑战。第一,根据提示准确预测输出长度具有挑战性,而且通常耗费大量资源,使其在许多系统中并不实用。因此,最先进的 LLM 系统默认采用先到先服务调度,这可能导致队头阻塞并降低系统效率。第二,抢占会给 LLM 系统带来额外的内存开销,因为系统必须为未完成(被抢占)的请求维护中间状态。本文提出 TRAIL,一种从目标 LLM 自身获取输出预测的方法。在生成每个输出 token 后,我们将其内部结构的嵌入回收,作为轻量级分类器的输入,该分类器预测每个正在运行请求的剩余长度。利用这些预测,我们提出一种基于预测的 SRPT 变体,其抢占受到限制,专门用于考虑 LLM 系统中的内存开销。该变体允许在请求执行早期、内存消耗较低时进行抢占,但随着请求接近完成会限制抢占,以优化资源利用率。在理论方面,我们在 M/G/1 队列模型中推导了该 SRPT 变体的闭式公式,证明了其潜在价值。在我们的系统中,我们将这一抢占策略与基于嵌入的预测方法一同实现。
英文摘要:
Efficient scheduling is crucial for interactive Large Language Model (LLM) applications, where low request completion time directly impacts user engagement. Size-based scheduling algorithms like Shortest Remaining Process Time (SRPT) aim to reduce average request completion time by leveraging known or estimated request sizes and allowing preemption by incoming jobs with shorter service times. However, two main challenges arise when applying size-based scheduling to LLM systems. First, accurately predicting output lengths from prompts is challenging and often resource-intensive, making it impractical for many systems. As a result, the state-of-the-art LLM systems default to first-come, first-served scheduling, which can lead to head-of-line blocking and reduced system efficiency. Second, preemption introduces extra memory overhead to LLM systems as they must maintain intermediate states for unfinished (preempted) requests. In this paper, we propose TRAIL, a method to obtain output predictions from the target LLM itself. After generating each output token, we recycle the embedding of its internal structure as input for a lightweight classifier that predicts the remaining length for each running request. Using these predictions, we propose a prediction-based SRPT variant with limited preemption designed to account for memory overhead in LLM systems. This variant allows preemption early in request execution when memory consumption is low but restricts preemption as requests approach completion to optimize resource utilization. On the theoretical side, we derive a closed-form formula for this SRPT variant in an M/G/1 queue model, which demonstrates its potential value. In our system, we implement this preemption policy alongside our embedding-based prediction method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。