Asteroid：面向异构边缘设备协同DNN训练的资源高效混合流水线并行方法
Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices
AI总结：
针对边缘端DNN训练资源受限问题，提出分布式训练系统Asteroid，通过混合流水线并行、资源感知规划及轻量容错重放机制，在异构边缘设备上实现训练加速与鲁棒性提升。
摘要：
端侧深度神经网络（DNN）训练已被公认为边缘隐私保护机器学习的关键技术。然而，密集的训练工作负载与有限的板载计算资源，给模型训练的可用性与效率带来了严峻挑战。现有研究多通过原生资源管理优化应对这些问题，而我们基于观察提出不同思路：边缘环境通常包含大量配套的可信边缘设备，其闲置资源远超单个终端的可用量。我们提出Asteroid——一种分布式边缘训练系统，可打破异构边缘设备间的资源壁垒，实现高效的模型训练加速。Asteroid采用混合流水线并行机制调度分布式训练，并搭配精心设计的并行规划方案，以在特定资源约束下最大化吞吐量。此外，我们还开发了一种兼具容错性与轻量性的流水线重放机制，以应对设备级动态变化，保障训练鲁棒性与性能稳定性。我们在异构边缘设备上基于视觉模型与语言模型实现了Asteroid，评估结果显示，其训练速度比传统并行方法最高快12.2倍，比当前最优的混合并行方法快2.1倍。此外，即便出现意外的设备退出与故障，Asteroid的训练流水线恢复速度也比基线方法快14倍，同时仍能保持相当的吞吐量。
英文摘要：
On-device Deep Neural Network (DNN) training has been recognized as crucial for privacy-preserving machine learning at the edge. However, the intensive training workload and limited onboard computing resources pose significant challenges to the availability and efficiency of model training. While existing works address these challenges through native resource management optimization, we instead leverage our observation that edge environments usually comprise a rich set of accompanying trusted edge devices with idle resources beyond a single terminal. We propose Asteroid, a distributed edge training system that breaks the resource walls across heterogeneous edge devices for efficient model training acceleration. Asteroid adopts a hybrid pipeline parallelism to orchestrate distributed training, along with a judicious parallelism planning for maximizing throughput under certain resource constraints. Furthermore, a fault-tolerant yet lightweight pipeline replay mechanism is developed to tame the device-level dynamics for training robustness and performance stability. We implement Asteroid on heterogeneous edge devices with both vision and language models, demonstrating up to 12.2x faster training than conventional parallelism methods and 2.1x faster than state-of-the-art hybrid parallelism methods through evaluations. Furthermore, Asteroid can recover training pipeline 14x faster than baseline methods while preserving comparable throughput despite unexpected device exiting and failure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
