用于高效数据并行训练的可控周期同步
Controlled Periodic Synchronization for Efficient Data-Parallel Training
浏览论文内容
中文总结 AI 辅助
研究通信受限下分布式训练的同步频率，提出可控周期数据并行（CPDP）策略，结合梯度全规约和慢模型参数平均，在多个实验设置中，CPDP在准确率上优于DDP等，证明同步频率是分布式训练实用控制参数。
中文摘要 AI 辅助
数据并行训练依赖于跨工作节点的频繁梯度同步。标准的分布式数据并行（DDP）在每次迭代时同步梯度，在快速局域网中有效，但在地理分布式环境中对通信延迟和网络变化越来越敏感。诸如局部随机梯度下降（LocalSGD）等周期性方法降低了同步频率，但主要依赖参数平均，当工作节点轨迹发散时可能不够。本文研究将同步频率作为通信受限分布式训练的系统参数。评估了可控周期数据并行（CPDP），这是一种与PyTorch-DDP兼容的策略，它将局部更新与结合梯度全规约和慢模型（SlowMo）参数平均的协调步骤交替进行。在Grid'5000的站内集群以及跨越南锡和索菲亚的跨站点广域网部署（往返时间16.6毫秒）上进行了实验。在ResNet-50/CIFAR-100的主要固定学习率设置为0.1时，CPDP在评估配置中实现了最高的峰值测试准确率。在K = 2时，CPDP比DDP提高了2.28个百分点，但增加了额外的挂钟时间。在K = 4时，CPDP比DDP提高了2.44个百分点，同时平均挂钟时间减少了13.8%。直接分析表明，K = 4时的暴露同步时间约为DDP的一半，解释了广域网上改进的准确率-时间权衡。在ViT-S/CIFAR-100和ResNet-50/TinyImageNet上的额外实验表明，CPDP与DDP相比仍具有竞争力，并且总体上优于LocalSGD。总体而言，结果表明同步频率是通信受限分布式训练的一个实用控制参数。
英文摘要
Data-parallel training relies on frequent gradient synchronization across workers. Standard DDP synchronizes gradients at every iteration, which is effective on fast local-area networks but increasingly sensitive to communication latency and network variability in geographically distributed environments. Periodic methods such as LocalSGD reduce synchronization frequency but rely mainly on parameter averaging, which may be insufficient when worker trajectories diverge. This paper studies synchronization frequency as a systems parameter for communication-constrained distributed training. We evaluate Controlled Periodic Data Parallelism (CPDP), a PyTorch-DDP-compatible strategy that alternates local updates with a reconciliation step combining gradient AllReduce and SlowMo parameter averaging. Experiments are conducted on Grid'5000 across intra-site clusters and a cross-site WAN deployment spanning Nancy and Sophia with 16.6 ms RTT. In the main fixed learning-rate setting of 0.1 on ResNet-50/CIFAR-100, CPDP achieves the highest peak test accuracy among the evaluated configurations. At K=2, CPDP improves over DDP by 2.28 percentage points but incurs additional wall-clock time. At K=4, CPDP improves over DDP by 2.44 percentage points while reducing average wall-clock time by 13.8%. Direct profiling shows that exposed synchronization time at K=4 is roughly half that of DDP, explaining the improved WAN accuracy-time trade-off. Additional experiments on ViT-S/CIFAR-100 and ResNet-50/TinyImageNet show that CPDP remains competitive with DDP and generally improves over LocalSGD. Overall, the results show that synchronization frequency is a practical control parameter for distributed training under communication constraints.
