tLoRA: 一种高效的多LoRA训练方法与弹性共享超模型
tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Models
AI总结：
tLoRA通过弹性共享超模型实现多LoRA任务高效训练，提升吞吐量和GPU利用率
摘要：
随着低秩适应（LoRA）成为高效微调大型语言模型（LLMs）的标准方法，共享集群越来越多地在同一冻结的骨干网络上执行多个并发的LoRA训练任务。尽管最近的进展使能够批量处理（协同定位）多个适配器，但高效训练时协同定位异构LoRA适配器却带来了独特的挑战。任务通常在适配器秩、批次大小和资源分配上有所不同，而简单的批量处理可能会引入同步停滞、通信开销和每任务的减速，这些情况比单独执行更差。我们介绍了tLoRA，一种能够高效批量训练多个LoRA任务的框架。tLoRA将共享相同基础模型的适配器融合为一个弹性共享超模型，利用现有的分布式训练框架来推导出能够有效共享资源的并行计划。在内核层面，tLoRA采用了一个融合的LoRA内核，能够自适应地重建低秩计算瓷砖，并调度具有秩意识的纳秒批次以最大化计算与通信在适配器之间的重叠。在调度层，tLoRA引入了一个在线的、基于剩余容量的调度器，能够自适应地将任务分组以最大化集体吞吐量。使用真实世界集群跟踪数据的评估表明，tLoRA提高了训练吞吐量1.2至1.8倍，任务训练完成时间减少了2.3至5.4倍，GPU利用率提高了37%。
英文摘要：
As Low-Rank Adaptation (LoRA) becomes the standard approach for efficiently fine-tuning large language models (LLMs), shared clusters increasingly execute many concurrent LoRA training jobs over the same frozen backbone. While recent advances enable batching (co-locating) multiple adapters during serving, efficient training-time co-location of heterogeneous LoRA adapters presents unique challenges. Jobs often differ in adapter rank, batch size, and resource allocation, and naïve batching can introduce synchronization stalls, communication overheads, and per-job slowdowns that are worse than executing independently. We introduce tLoRA, a framework that enables efficient batch training of multiple LoRA jobs. tLoRA fuses adapters that share the same base model into an elastic shared super-model, exploiting existing distributed training frameworks to derive parallelism plans that share resources effectively. At the kernel level, tLoRA employs a fused LoRA kernel that adaptively reconstructs low-rank computation tiles and schedules rank-aware nano-batches to maximize overlap between computation and communication across adapters. At the scheduling layer, tLoRA incorporates an online, residual-capacity-aware scheduler that adaptively groups jobs to maximize collective throughput. Evaluations using real-world cluster traces demonstrate that tLoRA improves training throughput by 1.2--1.8x, job training completion time by 2.3--5.4x, and GPU utilization by 37%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
