In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.
Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads
Dept. of Industrial Engineering and Decision Analytics HKUST（工业工程与决策分析系香港科技大学）；Dept. of Computer Science and Technology Tsinghua University（计算机科学与技术系清华大学）；IIIS Tsinghua University（清华大学信息学院）；Huawei Hong Kong Research Center（华为香港研发中心）；School of Mathematical Sciences Peking University（北京大学数学科学学院）
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Jiao Tong University（上海交通大学）
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；The International Joint Institute of Tianjin University, Tianjin University（天津大学国际联合学院）；Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics（天津财经大学数字经济与管理学院）
