arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.02037cs.LGcs.AIstat.ML
收藏

TFLMS:TensorFlow中基于图重写的大模型支持

TFLMS: Large Model Support in TensorFlow by Graph Rewriting

IBM Research - Tokyo(IBM东京研究院)

Tung D. Le, Haruki Imai, Yasushi Negishi, Kiyokuni Kawachiya

摘要:

虽然GPU等加速器的内存有限,但深度神经网络正变得越来越大,无法适应加速器的内存限制。我们提出了一种方法来解决这个问题,重写的计算图的神经网络,其中交换出和交换在操作插入临时存储在CPU内存中的中间结果。特别是,我们首先修改了计算图的概念,定义了一个具体的语义图中的变量。然后,我们正式展示了如何从现有的图形中导出换出和换入操作,并提出了优化图形的规则。为了实现我们的方法,我们在TensorFlow中开发了一个名为TFLMS的模块。TFLMS在TensorFlow存储库中以拉取请求的形式发布,用于为TensorFlow社区做出贡献。使用TFLMS,我们能够分别以4.7倍和2倍的批量大小训练ResNet-50和3DUnet。特别是,我们能够使用大小为$192^3$的图像来训练3DUNet进行图像分割,在没有TFLMS的情况下,仅通过将图像划分为更小的图像来完成图像分割,这影响了准确性。

英文摘要:

While accelerators such as GPUs have limited memory, deep neural networks are becoming larger and will not fit with the memory limitation of accelerators for training. We propose an approach to tackle this problem by rewriting the computational graph of a neural network, in which swap-out and swap-in operations are inserted to temporarily store intermediate results on CPU memory. In particular, we first revise the concept of a computational graph by defining a concrete semantics for variables in a graph. We then formally show how to derive swap-out and swap-in operations from an existing graph and present rules to optimize the graph. To realize our approach, we developed a module in TensorFlow, named TFLMS. TFLMS is published as a pull request in the TensorFlow repository for contributing to the TensorFlow community. With TFLMS, we were able to train ResNet-50 and 3DUnet with 4.7x and 2x larger batch size, respectively. In particular, we were able to train 3DUNet using images of size of $192^3$ for image segmentation, which, without TFLMS, had been done only by dividing the images to smaller images, which affects the accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑