面向物联网的分布式机器学习技术评价与优化
Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things
摘要：
联邦学习（FL）和分裂学习（SL）是最先进的分布式机器学习技术，用于在不访问客户端或终端设备上的原始数据的情况下实现机器学习训练。然而，它们的\emph{comparative training performance}在现实世界资源受限的物联网（IoT）设备设置下，例如，Raspberry Pi几乎没有被研究过，据我们所知，它还没有被评估和比较，这给从业者带来了不便。这项工作首先提供了FL和SL在现实世界物联网环境中的经验比较，涉及（i）异构数据分布的学习性能和（ii）设备上的执行开销。我们在这项工作中的分析表明，SL的学习性能优于FL下的不平衡的数据分布，但比FL下的极端非IID数据分布差。最近，FL和SL被组合以形成分裂式学习（SFL），以利用它们的每个优点（例如，FL的并行训练和SL的轻量级设备上计算要求）。然后，这项工作考虑FL，SL和SFL，并将它们安装在Raspberry Pi设备上以评估它们的性能，包括训练时间，通信开销，功耗和内存使用。除了评估之外，我们还应用了两个优化。首先，我们通过仔细检查服务器端混合类型模型训练的可能性来推广SFL。广义SFL合并了模型训练的顺序（依赖）和并行（独立）过程，因此对于具有大规模物联网设备的系统是有益的，特别是在服务器端操作。其次，我们提出了务实的技术，大大减少了高达四倍的SL和（广义）SFL的通信开销。
英文摘要：
Federated learning (FL) and split learning (SL) are state-of-the-art distributed machine learning techniques to enable machine learning training without accessing raw data on clients or end devices. However, their \emph{comparative training performance} under real-world resource-restricted Internet of Things (IoT) device settings, e.g., Raspberry Pi, remains barely studied, which, to our knowledge, have not yet been evaluated and compared, rendering inconvenient reference for practitioners. This work firstly provides empirical comparisons of FL and SL in real-world IoT settings regarding (i) learning performance with heterogeneous data distributions and (ii) on-device execution overhead. Our analyses in this work demonstrate that the learning performance of SL is better than FL under an imbalanced data distribution but worse than FL under an extreme non-IID data distribution. Recently, FL and SL are combined to form splitfed learning (SFL) to leverage each of their benefits (e.g., parallel training of FL and lightweight on-device computation requirement of SL). This work then considers FL, SL, and SFL, and mount them on Raspberry Pi devices to evaluate their performance, including training time, communication overhead, power consumption, and memory usage. Besides evaluations, we apply two optimizations. Firstly, we generalize SFL by carefully examining the possibility of a hybrid type of model training at the server-side. The generalized SFL merges sequential (dependent) and parallel (independent) processes of model training and is thus beneficial for a system with large-scaled IoT devices, specifically at the server-side operations. Secondly, we propose pragmatic techniques to substantially reduce the communication overhead by up to four times for the SL and (generalized) SFL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
