Machine learning models have been exponentially growing in terms of their parameter size over the past few years. We are now seeing the rise of trillion-parameter models. The large models cannot fit into a single GPU and thus require partitioned deployment across GPUs and even hosts. A high-performance collective communication library (CCL) such as NCCL is essential to fully utilize expensive GPU resources. However, CCL is not a great fit for inference. Unlike training for which a fixed amount of GPU resources is used for fixed workloads (e.g., input datasets), the inference workloads can change dynamically over time. Failures at the serving time can also impact individual user's experiences directly. In contrast, workers in a CCL process group share a single fault domain and the process group cannot grow as the workloads increase. The gap between the unique characteristics of model serving and CCL's nature makes it hard to serve large models elastically. To bridge the gap, we propose MultiWorld that enables fault tolerance and online scaling at the granularity of workers for model serving. Our evaluation showcases that enabling these new functionalities incurs small overheads (1.4-4.3% throughput loss) for most of the scenarios we tested.
ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Electronic Science and Technology of China(电子科技大学);National University of Singapore(新加坡国立大学)
DisagFusion: Asynchronous Pipeline Parallelism and Elastic Scheduling for Disaggregated Diffusion Serving
Renmin University of China(中国人民大学);SenseTime(商汤科技);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学);Beihang University(北京航空航天大学)
eLLM: Elastic Memory Management Framework for Efficient LLM Serving
Shanghai Jiao Tong University(上海交通大学);Shanghai Qi Zhi Institute(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机与数据科学学院);Department of Computer Science, Purdue University(普渡大学计算机科学系)