Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
ParisTech Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);UM-SJTU Joint Institute, Shanghai Jiao Tong University(上海交通大学密西根学院);Shanghai Aerospace System Engineering Institute(上海航天系统工程研究所);The University of Melbourne(墨尔本大学)
The application of Transformer-based large models has achieved numerous success in recent years. However, the exponential growth in the parameters of large models introduces formidable memory challenge for edge deployment. Prior works to address this challenge mainly focus on optimizing the model structure and adopting memory swapping methods. However, the former reduces the inference accuracy, and the latter raises the inference latency. This paper introduces PIPELOAD, a novel memory-efficient pipeline execution mechanism. It reduces memory usage by incorporating dynamic memory management and minimizes inference latency by employing parallel model loading. Based on PIPELOAD mechanism, we present Hermes, a framework optimized for large model inference on edge devices. We evaluate Hermes on Transformer-based models of different sizes. Our experiments illustrate that Hermes achieves up to 4.24 X increase in inference speed and 86.7% lower memory consumption than the state-of-the-art pipeline mechanism for BERT and ViT models, 2.58 X increase in inference speed and 90.3% lower memory consumption for GPT-style models.
Memory at Your Service: Fast Memory Allocation for Latency-critical Services
在诸如键值存储和web搜索之类的延迟关键型服务与尽力而为的批处理作业之间的协同定位和存储器共享是提高多租户数据中心系统中的存储器利用率的有吸引力的方法。然而,我们发现,非常不同的目标的工作协同定位和GNU/Linux系统堆栈可能会导致严重的性能下降的延迟关键服务的内存压力下,在多租户系统。我们通过快速内存分配和主动回收来解决延迟关键型服务的内存压力。我们发现,内存分配延迟占主导地位的整体查询延迟,特别是在内存压力。我们分析了GNU/Linux系统栈提供的默认存储管理机制,并找出了它在多租户系统中对延迟关键服务效率低下的原因。我们提出了Hermes,一个快速的内存分配机制,在用户空间,自适应地保留内存的延迟关键的服务。它建议Linux操作系统主动回收批处理作业的内存。我们在GNU C Library中实现了Hermes。实验结果表明,Hermes在微基准测试中,平均内存分配延迟和$99^{th}$百分比内存分配延迟分别降低了54.4%和62.4%.对于两个实际的延迟关键型服务,Hermes将平均和$99^{th}$百分位数尾部查询延迟降低了40.3%。与默认的Glibc、jemalloc和TCMalloc相比,Hermes在内存压力下将服务级别目标违规减少了84.3%。
OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training
Shanghai University of Finance and Economics(上海金融学院);National University of Singapore(新加坡国立大学);Shanghai Jiao Tong University(上海交通大学);Stanford University(斯坦福大学)
BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Heidelberg, Germany(海德堡大学硬件与人工智能实验室);NVIDIA Corporation, Santa Clara, CA, USA(英伟达公司)
LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices
School of Computer Science and Technology, Shandong University(山东大学计算机科学与技术学院);School of Artificial Intelligence, Shandong University(山东大学人工智能学院);School of Cryptologic Science and Engineering, Shandong University(山东大学密码学与工程学院);Shandong Provincial Key Laboratory of Computing-Network Integration, Shandong University(山东大学计算网络集成山东省重点实验室)
DisagFusion: Asynchronous Pipeline Parallelism and Elastic Scheduling for Disaggregated Diffusion Serving
Renmin University of China(中国人民大学);SenseTime(商汤科技);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学);Beihang University(北京航空航天大学)