arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.14686cs.DCcs.AI
收藏

xLLM 技术报告

xLLM Technical Report

THU(清华大学);USTC(University of Science and Technology of China);BUAA(Beijing University of Aeronautics and Automation);PKU(Peking University);TJU(Tianjin University)
Tongxuan Liu, Tao Peng, Peijun Yang, Xiaoyang Zhao, Xiusheng Lu, Weizhe Huang, Zirui Liu, Xiaoyu Chen, Zhiwei Liang, Jun Xiong, Donghe Jin, Minchao Zhang, Jinro… 展开作者

Tongxuan Liu, Tao Peng, Peijun Yang, Xiaoyang Zhao, Xiusheng Lu, Weizhe Huang, Zirui Liu, Xiaoyu Chen, Zhiwei Liang, Jun Xiong, Donghe Jin, Minchao Zhang, Jinrong Guo, Yingxu Deng, Xu Zhang, Xianzhe Dong, Siqi Wang, Siyu Wu, Yu Wu, Zihan Tang, Yuting Zeng, Yanshu Wang, Jinguang Liu, Meng Kang, Menxin Li, Yunlong Wang, Yiming Liu, Xiaolong Ma, Yifan Wang, Yichen Zhang, Jinrun Yin, Keyang Zheng, Jiawei Yin, Jun Zhang, Ziyue Wang, Xiaobo Lin, Liangyu Liu, Liwei Lan, Yang Liu, Chunhua Peng, Han Liu, Songcheng Ren, Xuezhu Wang, Yunheng Shen, Yi Wang, Guyue Liu, Yitao Hu, Hui Chen, Tong Yang, Hailong Yang, Jing Li, Guiguang Ding, Ke Zhang

AI总结:

xLLM是一种针对高性能大规模企业级服务设计的高效LLM推理框架,通过解耦服务引擎架构和多层优化提升吞吐量和推理效率。

摘要:

我们介绍了xLLM,一种智能且高效的大型语言模型(LLM)推理框架,旨在实现高性能、大规模的企业级服务,具有对多种人工智能加速器的深度优化。为解决这些挑战,xLLM构建了新颖的解耦服务引擎架构。在服务层,xLLM-Service具有一个智能调度模块,能够高效处理多模态请求,并通过统一弹性调度将在线和离线任务进行协同定位,以最大化集群利用率。该模块还依赖于一种工作负载自适应的动态预填解码(PD)解耦策略和一种针对多模态输入设计的新型编码-预填解码(EPD)解耦策略。此外,它还集成了分布式架构,以提供全局KV缓存管理和强大的容错能力,以确保高可用性。在引擎层,xLLM-Engine协同优化系统和算法设计,以充分利用计算资源。这通过全面的多层执行管道优化、自适应图模式和xTensor内存管理来实现。xLLM-Engine还进一步集成了算法增强,如优化的推测解码和动态EPLB,共同服务于显著提高吞吐量和推理效率。广泛的评估表明,xLLM在性能和资源效率方面显著优于现有方法。在相同的TPOT约束下,xLLM在使用Qwen系列模型时,吞吐量达到MindIE的1.7倍,达到vLLM-Ascend的2.2倍;在使用Deepseek系列模型时,平均吞吐量达到MindIE的1.7倍。xLLM框架已在https://github.com/jd-opensource/xllm和https://github.com/jd-opensource/xllm-service上公开发布。

英文摘要:

We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimizations for diverse AI accelerators. To address these challenges, xLLM builds a novel decoupled service-engine architecture. At the service layer, xLLM-Service features an intelligent scheduling module that efficiently processes multimodal requests and co-locates online and offline tasks through unified elastic scheduling to maximize cluster utilization. This module also relies on a workload-adaptive dynamic Prefill-Decode (PD) disaggregation policy and a novel Encode-Prefill-Decode (EPD) disaggregation policy designed for multimodal inputs. Furthermore, it incorporates a distributed architecture to provide global KV Cache management and robust fault-tolerant capabilities for high availability. At the engine layer, xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources. This is achieved through comprehensive multi-layer execution pipeline optimizations, an adaptive graph mode and an xTensor memory management. xLLM-Engine also further integrates algorithmic enhancements such as optimized speculative decoding and dynamic EPLB, collectively serving to substantially boost throughput and inference efficiency. Extensive evaluations demonstrate that xLLM delivers significantly superior performance and resource efficiency. Under identical TPOT constraints, xLLM achieves throughput up to 1.7x that of MindIE and 2.2x that of vLLM-Ascend with Qwen-series models, while maintaining an average throughput of 1.7x that of MindIE with Deepseek-series models. xLLM framework is publicly available at https://github.com/jd-opensource/xllm and https://github.com/jd-opensource/xllm-service.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑