arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03622cs.LG
收藏

MSfusion:一种面向资源受限机器的动态模型切分方法,用于协作训练更大模型

MSfusion: A Dynamic Model Splitting Approach for Resource-Constrained Machines to Collaboratively Train Larger Models

Internet of Things Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)物联网部);School of Cyber Science and Engineering, Southeast University(东南大学网络空间科学与工程学院)

Jin Xie, Songze Li

AI总结:

MSfusion 提出一种基于双重移位模型切分与自适应重叠、对比损失的协作学习框架,使资源受限设备能够高效协作训练大模型,并展现出强可扩展性。

摘要:

训练大型模型需要大量数据以及充足的计算资源。虽然协作学习(例如联邦学习)为利用众多参与者的集体数据提供了一种有前景的范式,但对于移动设备等资源有限的参与者而言,训练大型模型仍然是一项重大挑战。我们提出了 MSfusion,一种有效且高效的协作学习框架,专门用于通过模型切分在资源受限的机器上训练更大的模型。具体来说,我们设计了一种双重移位模型切分方案,使得在每一轮训练中,每个参与者被分配一个模型参数子集,在本地数据上进行训练,并与其他参与者在公共参数上的子模型进行聚合。虽然模型切分显著降低了个体参与者的计算和通信成本,但我们在自适应模型重叠和对比损失函数上的额外新颖设计,帮助 MSfusion 在面对参与者之间模型偏移时保持训练有效性。在图像和自然语言处理任务上的大量实验表明,MSfusion 在训练大型模型时具有显著的性能和效率优势,并且具有很强的可扩展性:随着参与者数量的增加,每个参与者的计算成本显著降低。

英文摘要:

Training large models requires a large amount of data, as well as abundant computation resources. While collaborative learning (e.g., federated learning) provides a promising paradigm to harness collective data from many participants, training large models remains a major challenge for participants with limited resources like mobile devices. We introduce MSfusion, an effective and efficient collaborative learning framework, tailored for training larger models on resourceconstraint machines through model splitting. Specifically, a double shifting model splitting scheme is designed such that in each training round, each participant is assigned a subset of model parameters to train over local data, and aggregates with sub-models of other peers on common parameters. While model splitting significantly reduces the computation and communication costs of individual participants, additional novel designs on adaptive model overlapping and contrastive loss functions help MSfusion to maintain training effectiveness, against model shift across participants. Extensive experiments on image and NLP tasks illustrate significant advantages of MSfusion in performance and efficiency for training large models, and its strong scalability: computation cost of each participant reduces significantly as the number of participants increases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑