使用自动数据流图划分支持超大型模型
Supporting Very Large Models using Automatic Dataflow Graph Partitioning
摘要:
本文介绍了Tofu,这是一个在多个GPU设备上划分非常大的DNN模型的系统,以减少每个GPU的内存占用。Tofu旨在划分细粒度张量运算符的低维图,以便透明地与MXNet等通用深度学习平台一起工作。为了自动划分每个操作符,我们建议描述一个操作符的语义在一个简单的语言,表示张量的lambda函数映射从张量坐标值。为了最佳地划分一个图中的不同操作符,豆腐使用递归搜索算法,最大限度地减少总的通信成本。我们在8-GPU机器上的实验表明,Tofu可以训练非常大的CNN和RNN模型。它还实现了25% - 400%的加速比替代方法的训练非常大的模型。
英文摘要:
This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint. Tofu is designed to partition a dataflow graph of fine-grained tensor operators in order to work transparently with a general-purpose deep learning platform like MXNet. In order to automatically partition each operator, we propose to describe the semantics of an operator in a simple language which represents tensors as lambda functions mapping from tensor coordinates to values. To optimally partition different operators in a dataflow graph, Tofu uses a recursive search algorithm that minimizes the total communication cost. Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models. It also achieves 25% - 400% speedup over alternative approaches to train very large models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。