Spindle：通过波前调度实现多任务大模型的高效分布式训练
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
AI总结：
针对多任务多模态大模型训练面临的系统挑战，提出Spindle系统，通过波前调度将模型执行分解为波，依次解决异构感知的工作负载并行化和依赖驱动的执行调度问题，实验显示其相比现有系统加速比高达71%。
摘要：
近年来的基础模型能够通过统一的基础模型结构和若干专用模型组件处理多任务和多数据模态。然而，由于复杂的模型架构以及不同任务和模态的异构工作负载，此类多任务（MT）多模态（MM）模型的高效训练面临重大系统挑战。本文提出Spindle，这是一种全新的训练系统，通过波前调度为MT MM模型的资源高效且高性能训练量身定制。Spindle的核心思想是将模型执行分解为波，并依次解决联合优化问题，包括异构感知的工作负载并行化和依赖驱动的执行调度。我们构建了该系统并在多种MT MM模型上进行评估，实验表明Spindle具有卓越的性能和效率，与最先进的训练系统相比加速比高达71%。
英文摘要：
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However, efficient training of such multi-task (MT) multi-modal (MM) models poses significant system challenges due to the sophisticated model architecture and the heterogeneous workloads of different tasks and modalities. In this paper, we propose Spindle, a brand new training system tailored for resource-efficient and high-performance training of MT MM models via wavefront scheduling. The key idea of Spindle is to decompose the model execution into waves and address the joint optimization problem sequentially, including both heterogeneity-aware workload parallelization and dependency-driven execution scheduling. We build our system and evaluate it on various MT MM models. Experiments demonstrate the superior performance and efficiency of Spindle, with speedup ratio up to 71% compared to state-of-the-art training systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
