arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.22142cs.DCcs.LG
收藏

在边缘计算中利用空闲计算资源进行基础模型训练

On Harnessing Idle Compute at the Edge for Foundation Model Training

The University of Edinburgh(爱丁堡大学);Johns Hopkins University(约翰霍普金斯大学);Cisco Research(思科研究院)

Leyang Xue, Meghana Madhyastha, Myungjin Lee, Amos Storkey, Randal Burns, Mahesh K. Marina

AI总结:

本文提出Cleave框架,通过利用边缘设备的异构计算资源,实现与云训练相媲美的性能,并在设备异构性和故障恢复方面取得显著优势。

摘要:

基础模型生态系统仍然高度集中,因为训练需要巨大的计算资源,因此主要由大型云运营商主导。利用边缘设备的空闲计算资源进行基础模型训练提供了一种更民主化的替代方案。然而,现有边缘训练方法存在不足:难以匹配云训练性能、扩展到更大模型、适应设备内存限制或保持通信开销可控。它们还无法有效处理设备异质性和设备 churn。我们引入Cleave,基于结构洞察:每个GEMM操作具有不对称的I/O模式——其输入矩阵通过下行链路发送,远大于通过上行链路返回的部分输出块——这与边缘网络中下行链路带宽超过上行链路2-10倍的特性相匹配。利用这种对齐,基于参数服务器架构的Cleave使每台设备的通信量随着更多设备加入而减少,而不是像传统TP一样保持不变。将训练分解为独立的子GEMM任务,产生一种统一的调度抽象,统一了内存限制、通信开销和设备 churn 下的容错性。我们的评估显示,Cleave在云训练性能上与云训练相媲美,并在相同设备数量下,比最先进的边缘训练方法在每批次运行时间上快4-10倍。在此共享操作范围内之外,Cleave可扩展到数千个异构设备——这是先前边缘训练系统无法操作的领域,并且在设备故障恢复方面至少快100倍。

英文摘要:

The foundation-model ecosystem remains highly centralized because training requires immense compute resources and is therefore largely limited to large cloud operators. Edge-assisted foundation model training that harnesses spare compute on edge devices offers a more democratized alternative. However, existing edge-training approaches fall short: they struggle to match cloud-training performance, scale to larger models, fit within device memory limits, or keep communication overhead manageable. They also do not handle device heterogeneity and churn satisfactorily. We introduce Cleave, built on a structural insight: each GEMM has an asymmetric I/O pattern -- its input matrices, sent over downlink, are much larger than the partial output blocks returned over uplink -- matching edge networks where downlink bandwidth exceeds uplink by 2--10x. Exploiting this alignment with a parameter-server-centric architecture, Cleave makes per-device communication \emph{decrease} as more devices join, rather than stay constant as in conventional TP. Decomposing training into independent sub-GEMM tasks yields one scheduling abstraction that unifies memory constraints, communication overhead, and fault tolerance under device churn. Our evaluation shows that Cleave achieves cloud-comparable GPU training performance and outperforms state-of-the-art edge-training methods by 4--10x in per-batch runtime at the same device counts. Beyond this shared operating range, Cleave scales to thousands of heterogeneous devices -- a regime where prior edge-training systems cannot operate -- and achieves at least 100x faster recovery from device failures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑