边缘场景下的灵活并行学习:通信、计算和能源成本
Flexible Parallel Learning in Edge Scenarios: Communication, Computational and Energy Cost
摘要:
传统上,分布式机器学习采用(i)不同的节点训练相同的模型(如联合学习),或者(ii)一个模型在多个节点之间分裂(如分布式随机梯度下降)。在这项工作中,我们强调了基于雾和物联网的场景通常需要结合这两种方法,我们提出了一个灵活的并行学习(FPL)框架,实现数据和模型并行。此外,我们研究了不同的方式分布和并行化的学习任务在参与节点的结果在不同的计算,通信和能源成本。我们使用最先进的深度网络架构和大规模数据集进行的实验证实,FPL可以在计算(因此是能源)成本、通信开销和学习性能之间实现出色的权衡。
英文摘要:
Traditionally, distributed machine learning takes the guise of (i) different nodes training the same model (as in federated learning), or (ii) one model being split among multiple nodes (as in distributed stochastic gradient descent). In this work, we highlight how fog- and IoT-based scenarios often require combining both approaches, and we present a framework for flexible parallel learning (FPL), achieving both data and model parallelism. Further, we investigate how different ways of distributing and parallelizing learning tasks across the participating nodes result in different computation, communication, and energy costs. Our experiments, carried out using state-of-the-art deep-network architectures and large-scale datasets, confirm that FPL allows for an excellent trade-off among computational (hence energy) cost, communication overhead, and learning performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。