arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1810.03035cs.DC
收藏

在TensorFlow中表征深度学习I/O工作负载

Characterizing Deep-Learning I/O Workloads in TensorFlow

Steven W. D. Chien, Stefano Markidis, Chaitanya Prasad Sishtla, Luis Santos, Pawel Herman, Sai Narasimhamurthy, Erwin Laure

备注：
Accepted for publication at pdsw-DISCS 2018
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

深度学习（DL）计算框架的性能依赖于数据摄取和检查点的性能。事实上，在训练过程中，大量相对较小的文件首先在CPU上加载和预处理，然后移动到加速器进行计算。此外，执行检查点和重启操作以允许DL计算框架从检查点快速重启。因此，I/O会影响DL应用程序的性能。在这项工作中，我们描述了TensorFlow的I/O性能和扩展，TensorFlow是由Google开发的开源编程框架，专门用于解决DL问题。为了测量TensorFlow I/O性能，我们首先设计了一个微基准来测量TensorFlow读取，然后使用基于AlexNet的TensorFlow迷你应用程序来测量TensorFlow中I/O和检查点的性能成本。为了提高检查点的性能，我们设计并实现了一个突发缓冲区。我们发现，在我们的基准测试环境中，增加线程数量将TensorFlow带宽最多增加2.3倍和7.8倍。tensorFlow预取器的使用导致加速器上的计算和CPU上的输入管道完全重叠，消除了I/O对整体性能的有效成本。在我们的基准测试环境中，使用突发缓冲区将检查点设置到快速的小容量存储，并将检查点异步复制到较慢的大容量存储，与直接将检查点设置到较慢的存储相比，性能提高了2.6倍。

英文摘要：

The performance of Deep-Learning (DL) computing frameworks rely on the performance of data ingestion and checkpointing. In fact, during the training, a considerable high number of relatively small files are first loaded and pre-processed on CPUs and then moved to accelerator for computation. In addition, checkpointing and restart operations are carried out to allow DL computing frameworks to restart quickly from a checkpoint. Because of this, I/O affects the performance of DL applications. In this work, we characterize the I/O performance and scaling of TensorFlow, an open-source programming framework developed by Google and specifically designed for solving DL problems. To measure TensorFlow I/O performance, we first design a micro-benchmark to measure TensorFlow reads, and then use a TensorFlow mini-application based on AlexNet to measure the performance cost of I/O and checkpointing in TensorFlow. To improve the checkpointing performance, we design and implement a burst buffer. We find that increasing the number of threads increases TensorFlow bandwidth by a maximum of 2.3x and 7.8x on our benchmark environments. The use of the tensorFlow prefetcher results in a complete overlap of computation on accelerator and input pipeline on CPU eliminating the effective cost of I/O on the overall performance. The use of a burst buffer to checkpoint to a fast small capacity storage and copy asynchronously the checkpoints to a slower large capacity storage resulted in a performance improvement of 2.6x with respect to checkpointing directly to slower storage on our benchmark environment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑