Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；HKBU（香港 Baptist University）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Hong Kong University of Science and Technology（香港科学与技术大学）
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学院，奥布达大学）
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学校，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学系，奥布达大学）
BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models
Southwestern University of Finance and Economics（西南财经大学）；The University of Hong Kong（香港大学）；Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province（四川省人工智能与数字金融重点实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong Polytechnic University（香港理工大学）；Sun Yat-sen University（中山大学）
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.（南京大学电子科学与工程学院）；National Key Laboratory for Novel Software Technology, Nanjing University, China.（南京大学新型软件技术国家重点实验室）
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；North China Electric Power University（华北电力大学）；State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所处理器国家重点实验室）
Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures
Duke University（杜克大学）；Microsoft Research（微软研究院）
随着智能设备和物联网的无处不在的部署，用于机器学习推理的数据源越来越多地移动到网络边缘。现有的机器学习推理平台通常假设同构基础设施，并且不考虑更复杂和分层的计算基础设施，包括边缘设备、本地集线器、边缘计算中心和云计算中心。另一方面，最近的AutoML努力为异构环境的模型压缩、修剪和量化提供了可行的解决方案;对于机器学习模型，现在我们可以很容易地找到甚至生成一系列在准确性和效率之间具有不同权衡的模型。我们设计并实现了JellyBean，一个在异构基础设施上服务和优化机器学习推理工作流的系统。给定服务级别目标（例如，吞吐量、准确性），JellyBean选择符合准确性目标的最具成本效益的模型，并决定如何在不同的基础架构层中部署它们。评估表明，与最先进的模型选择和工人分配解决方案相比，JellyBean将视觉问答的总服务成本降低了58%，将NVIDIA AI城市挑战赛的车辆跟踪成本降低了36%。JellyBean还优于先前的ML服务系统（例如，Spark on the Cloud），服务成本高达5倍。
DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
