岩石、鹅卵石和沙子:面向多模态大语言模型推理的模态感知调度
Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference
AI总结:
针对多模态大语言模型推理中资源需求差异大导致的延迟和内存问题,提出RPS-Serve调度器,通过动态优先级和老化机制实现高效资源利用,降低TTFT并提升响应性能。
摘要:
多模态大语言模型(MLLMs)如ChatGPT、Gemini和Copilot通过文本、图像和视频实现更丰富的交互。这些异构工作负载引入了视觉预处理和编码等额外推理阶段,导致延迟和内存需求增加。现有LLM服务系统优化于纯文本工作负载,在多模态场景下失效:大请求(如视频)垄断资源,导致严重头阻塞和性能下降。我们的关键见解是多模态请求的资源需求差异巨大,我们通过简单抽象:视频像岩石,图像像鹅卵石,文本像沙子。我们设计RPS-Serve,一种模态感知调度器,使沙子快速通过鹅卵石和岩石,确保交互响应性的同时避免饥饿。RPS-Serve对请求进行分类,动态优先级,并应用老化以避免饥饿。评估表明,RPS-Serve在最先进MLLMs上平均将时间到第一个token(TTFT)减少54%,对延迟敏感请求减少78.5%,相比现有系统。RPS-Serve通过模态感知调度和高效利用可用资源,为MLLMs提供LLM般的响应性。
英文摘要:
Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling rich interactions across text, images, and videos. Unlike text-only workloads, multimodal requests are inherently heterogeneous, differing by orders of magnitude in latency and memory footprint. Existing LLM serving systems, designed for homogeneous text workloads, struggle under this heterogeneity: large image and video requests monopolize GPU resources, causing severe head-of-line blocking and degrading the responsiveness of lightweight, latency-critical text queries. Recent multimodal serving systems mitigate this interference through disaggregation, isolating encoding and inference stages across GPUs. While effective at large scale, such designs introduce severe GPU underutilization in more practical, moderate-scale deployments. In response, we build RPS-Serve a modality-aware scheduling framework that leverages the extreme heterogeneity in multimodal workloads and introduces a simple abstraction: videos behave like rocks, images like pebbles, and text like sand. Then, it lets sand flow quickly through pebbles and rocks, maximizing GPU utilization while ensuring fast responsiveness for lightweight requests and controlled execution for heavier ones. To achieve this, RPS-Serve estimates the GPU admission cost of each request, classifies requests into rocks, pebbles, and sand using lightweight clustering, and applies priority-based scheduling with aging to balance performance across modalities. Evaluation across state-of-the-art MLLMs shows that RPS-Serve increases GPU utilization by 65% and reduces TTFT by up to 9.3x, on average across models, compared to disaggregation-based multi-modal systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。