MuxTune: 通过时空骨干多路复用实现多租户数据中心中的高效多任务LLM微调
MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing
AI总结:
MuxTune通过时空骨干多路复用实现多租户数据中心中多个PEFT任务的高效并发执行,显著提升吞吐量和内存利用率。
摘要:
参数高效微调(PEFT)被广泛应用于大型语言模型(LLM)定制化数据中心的微调API后端。服务提供商为每个PEFT任务部署单独的实例,导致显著的资源不效率,包括(1)由于小规模PEFT原生运算符导致的GPU利用率低下,以及(2)并行执行中通信延迟和数据依赖性导致的设备停滞。为了解决这些问题,本文提出MuxTune,一种能够实现多个PEFT任务高效并发执行的微调系统。关键思想是通过时空方式在独立任务之间复用骨干,以提高利用率并减少停滞。基于灵活、模块化的骨干共享,通过统一的PEFT表示,MuxTune提出了具有任务、运算符和数据层次优化的分层协同调度方案。具体而言,它通过空间和时间复用的混合方式融合任务,并在双层混合并行中协调多任务运算符执行。此外,MuxTune采用基于块的数据对齐来缓解跨任务无效令牌。实验结果表明,与三个最先进的基线相比,MuxTune在吞吐量上提高了2.33倍,在内存减少方面达到了5.29倍。
英文摘要:
Parameter-Efficient Fine-Tuning (PEFT) is widely applied as the backend of fine-tuning APIs for large language model (LLM) customization in datacenters. Service providers deploy separate instances for individual PEFT tasks, giving rise to prominent resource inefficiencies, including (1) GPU underutilization from small-scale, PEFT-native operators and (2) device stalls from communication delays and data dependencies in parallelized execution. To address these issues, this paper presents MuxTune, a fine-tuning system that enables resource-efficient concurrent execution of multiple PEFT tasks. The key idea is to multiplex the backbone across independent tasks in a spatial-temporal manner for improved utilization and reduced stalls. Building on flexible, modularized backbone sharing via unified PEFT representations, MuxTune proposes hierarchical co-scheduling scheme with task, operator, and data-level optimizations. Specifically, it fuses tasks through a hybrid of spatial and temporal multiplexing, and orchestrates multi-task operator execution in two-tiered hybrid parallelism. Additionally, MuxTune employs chunk-based data alignment to mitigate inter-task ineffective tokens. Experimental results demonstrate that MuxTune achieves up to $2.33\times$ higher throughput and $5.29\times$ memory reduction compared to three state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。