Large Language Models (LLMs), as the foundational architecture for next-generation interactive AI applications, not only power intelligent dialogue systems but also drive the evolution of embodied intelligence on edge devices, including humanoid robots, smart vehicles, and other scenarios. The applications running on these edge devices impose differentiated Service Level Objectives (SLO) requirements on LLM services, specifically manifested as distinct constraints on Time to First Token (TTFT) and Time Per Output Token (TPOT) as well as end-to-end latency. Notably, edge devices typically handle real-time tasks that are extremely sensitive to latency, such as machine control and navigation planning. However, existing scheduling service systems still prioritize maximizing output token throughput as the sole optimization objective, failing to adequately address the diversity of SLO requirements. This ultimately results in persistently high violation rates for end-to-end latency or TPOT related SLOs.
This paper proposes SLICE, an innovative scheduling solution designed for edge computing scenarios with differentiated SLO requirements. By combining a utility-maximizing request scheduling algorithm with a dynamic iterative control mechanism for generation rates, SLICE significantly improves LLM inference service SLO attainment. Experimental results demonstrate that compared to state-of-the-art solutions Orca and FastServe, SLICE achieves up to 35x higher SLO attainment and 3.4x advantage in task completion time than the other two solutions. This version is temporarily hosted anonymously for double-blind review.
Kernelet: High-Throughput GPU Kernel Executions with Dynamic Slicing and Scheduling
图形处理器（GPU）最近被广泛用作集群和云等共享环境中的加速器。在这样的共享环境中，许多内核从不同的用户提交给GPU，吞吐量是性能和总拥有成本的重要指标。尽管最近改进了对并发GPU内核执行的运行时支持，但GPU可能严重未得到充分利用，导致次优吞吐量。在本文中，我们提出了Kernelet，一个运行时系统与动态切片和调度技术，以提高吞吐量的并发内核执行GPU上。通过切片，Kernelet将GPU内核划分为多个子内核（即切片）。每个切片都具有可调的占用率，以允许与其他切片协同调度并充分利用GPU资源。我们开发了一种新的和有效的基于马尔可夫链的性能模型来指导调度决策。我们的实验结果表明，在NVIDIA Tesla C2050和GTX 680 GPU上，性能分别提高了31.1%和23.4%。
Stage-Aware Communication Scheduling for Disaggregated LLM Serving
iSING Lab, Hong Kong University of Science and Technology（香港科技大学 iSING 实验室）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）
Deep Reinforcement Learning for Multi-Resource Multi-Machine Job Scheduling
Research Center of Smart Networks and Systems, Fudan University（复旦大学智慧网络与系统研究中心）；School of Operations Research and Information Engineering, Cornell University（康奈尔大学运筹学与信息工程学院）
