Serving disaggregated large language models (LLMs) over tens of thousands of xPU devices (GPUs or NPUs) with reliable performance faces multiple challenges. 1) Ignoring the diversity (various prefixes and tidal requests), treating all the prompts in a mixed pool is inadequate. To facilitate the similarity per scenario and minimize the inner mismatch on P/D (prefill and decoding) processing, fine-grained organization is required, dynamically adjusting P/D ratios for better performance. 2) Due to inaccurate estimation on workload (queue status or maintained connections), the global scheduler easily incurs unnecessary timeouts in prefill. 3) Block-fixed device-to-device (D2D) KVCache transfer over cluster-level RDMA (remote direct memory access) fails to achieve desired D2D utilization as expected. To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access. P/D-Serve is implemented upon Ascend and MindSpore, has been deployed over tens of thousands of NPUs for more than eight months in commercial use, and further achieves 60\%, 42\% and 46\% improvements on E2E throughput, time-to-first-token (TTFT) SLO (service level objective) and D2D transfer time. As the E2E system with optimizations, P/D-Serve achieves 6.7x increase on throughput, compared with aggregated LLMs.
Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Sensetime Research（商汤科技研究院）
InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models
Shanghai Jiao Tong University（上海交通大学）；Bytedance（字节跳动）；Singapore Management University（新加坡管理大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National University of Singapore（新加坡国立大学）
DisagFusion: Asynchronous Pipeline Parallelism and Elastic Scheduling for Disaggregated Diffusion Serving
Renmin University of China（中国人民大学）；SenseTime（商汤科技）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）；Beihang University（北京航空航天大学）
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Jiao Tong University（上海交通大学）
vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
AI Framework and Data Technology Lab, Huawei（华为人工智能框架与数据技术实验室）；The Chinese University of Hong Kong（香港中文大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Sun Yat-sen University（中山大学）
Efficient Multi-round LLM Inference over Disaggregated Serving
Shanghai Jiao Tong University（上海交通大学）；Southeast University（东南大学）；University of Cambridge（剑桥大学）；Peking University（北京大学）；OceanBase, Ant Group（蚂蚁集团 OceanBase）；Computational Social Science, Peking University (Qingdao)（北京大学（青岛）计算社会科学）
BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；State Key Lab of IOTSC, Faculty of Science and Technology, University of Macau（澳门科技大学物联网与感知系统国家重点实验室、理学院）；AIOS Team, Alibaba Group Inc（阿里巴巴集团 AIOS 团队）
Stage-Aware Communication Scheduling for Disaggregated LLM Serving
iSING Lab, Hong Kong University of Science and Technology（香港科技大学 iSING 实验室）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）
