VirtualFlow：将深度学习模型与底层硬件解耦
VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
摘要：
最先进的深度学习系统，如TensorFlow和PyTorch，将模型与底层硬件紧密耦合。这种耦合要求用户修改应用程序逻辑，以便跨不同的资源集运行相同的作业，从而限制了给定工作负载的硬件选择，并可能迫使用户放弃更有效的硬件配置。我们提出VirtualFlow，一个系统利用一种新的抽象称为虚拟节点处理解耦的模型从硬件。在训练或推断的每个步骤中，该批输入数据跨虚拟节点而不是硬件加速器（例如GPU和TPU）被分割。将多个虚拟节点映射到每个加速器并按顺序处理它们可以有效地对批处理进行时间切片，从而允许用户减少其工作负载的内存需求，并在小型群集上模拟大批量。使用这种技术，VirtualFlow可以实现许多新的用例，例如跨不同硬件、资源弹性和异构训练再现训练结果。在我们的评估中，我们为TensorFlow实现的VirtualFlow通过开箱即用的超参数在不同硬件上实现了强大的收敛保证，通过资源弹性将作业完成时间降低了48%，通过异构训练将吞吐量提高了42%。
英文摘要：
State-of-the-art deep learning systems such as TensorFlow and PyTorch tightly couple the model with the underlying hardware. This coupling requires the user to modify application logic in order to run the same job across a different set of resources, thereby limiting the choice of hardware for a given workload and potentially forcing the user to forgo more efficient hardware configurations. We propose VirtualFlow, a system leveraging a novel abstraction called virtual node processing to decouple the model from the hardware. In each step of training or inference, the batch of input data is split across virtual nodes instead of hardware accelerators (e.g. GPUs and TPUs). Mapping multiple virtual nodes to each accelerator and processing them sequentially effectively time slices the batch, thereby allowing users to reduce the memory requirement of their workloads and mimic large batch sizes on small clusters. Using this technique, VirtualFlow enables many new use cases, such as reproducing training results across different hardware, resource elasticity, and heterogeneous training. In our evaluation, our implementation of VirtualFlow for TensorFlow achieved strong convergence guarantees across different hardware with out-of-the-box hyperparameters, up to 48% lower job completion times with resource elasticity, and up to 42% higher throughput with heterogeneous training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
