dPRO：一个加速分布式DNN训练的通用剖析和优化系统
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
摘要：
使用多个设备的分布式训练（例如，GPU）已被广泛用于在大型数据集上学习DNN模型。然而，大规模分布式训练的性能在实践中往往远离线性加速。考虑到分布式系统的复杂性，当出现意外的低训练速度时，识别效率低下的根本原因并进行有效的性能优化是具有挑战性的。到目前为止，还没有软件工具可以诊断性能问题并帮助加快分布式DNN训练，而训练可以使用不同的深度学习框架运行。本文提出了一个工具包dPRO，它包括：（1）一个高效的profiler，它可以收集跨多个框架的分布式DNN训练的运行时痕迹，特别是细粒度的通信痕迹，并构建全局数据流图，包括详细的通信操作，以便准确回放;（2）有效识别性能瓶颈并探索优化策略的优化器（从计算、通信和存储器方面）用于训练加速。我们在多个深度学习框架（TensorFlow，MXNet）和代表性的通信方案（AllReduce和Parameter Server）上实现了dPRO。广泛的实验表明，dPRO在大多数情况下都能预测各种设置下的分布式训练性能，误差小于5%，并找到比基线速度提高3.48倍的优化策略。
英文摘要：
Distributed training using multiple devices (e.g., GPUs) has been widely adopted for learning DNN models over large datasets. However, the performance of large-scale distributed training tends to be far from linear speed-up in practice. Given the complexity of distributed systems, it is challenging to identify the root cause(s) of inefficiency and exercise effective performance optimizations when unexpected low training speed occurs. To date, there exists no software tool which diagnoses performance issues and helps expedite distributed DNN training, while the training can be run using different deep learning frameworks. This paper proposes dPRO, a toolkit that includes: (1) an efficient profiler that collects runtime traces of distributed DNN training across multiple frameworks, especially fine-grained communication traces, and constructs global data flow graphs including detailed communication operations for accurate replay; (2) an optimizer that effectively identifies performance bottlenecks and explores optimization strategies (from computation, communication, and memory aspects) for training acceleration. We implement dPRO on multiple deep learning frameworks (TensorFlow, MXNet) and representative communication schemes (AllReduce and Parameter Server). Extensive experiments show that dPRO predicts the performance of distributed training in various settings with < 5% errors in most cases and finds optimization strategies with up to 3.48x speed-up over the baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
