发表机构
Shanghai Jiao Tong University; Ant Group（上海交通大学; 蚂蚁集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
HALO通过空间复用和SLO感知调度，解决LoRA辅助本地LLM部署中的多SLO保证问题，提高吞吐并减少违规。
AI 中文摘要
随着大型语言模型（LLMs）在如医院和政府机构等隐私敏感领域变得至关重要，本地LLM服务器提供了比公共云服务更具成本效益且更安全的替代方案。然而，这些资源受限的服务器在同时服务多个LoRA适配服务时，难以保证异构服务级别目标（SLOs）。现有的服务框架由于LoRA层的计算开销和批处理调度的刚性，遭受严重的SLO违规。为解决此问题，我们提出HALO，一种专为LoRA辅助本地LLM部署设计的调度方法。HALO引入两项关键创新：一种空间复用策略，通过划分GPU流式多处理器（SMs）来重叠基础（Base）和LoRA计算；以及一个SLO感知调度器，基于“请求级松弛”解耦请求执行。通过优先处理紧急任务并利用空闲预算进行流量整形，HALO显著缓解资源争用。我们的评估表明，与最先进的基线相比，HALO在提高吞吐量的同时最小化SLO违规。
英文摘要
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objectives (SLOs) when serving multiple LoRA-adapted services simultaneously. Existing serving frameworks suffer from severe SLO violations due to the computational overhead of LoRA layers and the rigid nature of batch scheduling. To address this, we propose HALO, a scheduling method tailored for LoRA-assisted on-premise LLM deployment. HALO introduces two key innovations: a spatial multiplexing strategy that overlaps Base and LoRA computations by partitioning GPU Streaming Multiprocessors (SMs), and an SLO-aware scheduler that decouples request execution based on "request-level slack." By prioritizing urgent tasks and utilizing idle budget for traffic shaping, HALO significantly mitigates resource contention. Our evaluation demonstrates that HALO minimizes SLO violations while improving throughput compared to state-of-the-art baselines.
Comments22 pages, 16 figures, 4 tables
