TensorFlow is a popular cloud computing framework that targets machine learning applications. It separates the specification of application logic (in a dataflow graph) from the execution of the logic. TensorFlow's native runtime executes the application with low overhead across a diverse set of hardware including CPUs, GPUs, and ASICs. Although the underlying dataflow engine supporting these features could be applied to computations beyond machine learning, certain design decisions limit this broader application, such as the inability for an application to differentiate between data items across concurrent requests.
This paper introduces Pipelined TensorFlow (PTF), a system that extends TensorFlow's semantics to provide support for a broader variety of application logic. In particular, PTF supports applications that concurrently process finite batches of data on a single instantiation. PTF adds these semantics by partitioning the dataflow graph into a pipeline of smaller graphs and tagging each data item with metadata. These smaller graphs are separated by gates: new data structures in PTF that buffer data items between graphs and interpret the metadata to apply the new semantics. PTF's pipeline architecture executes on an unmodified TensorFlow runtime, maintaining compatibility with many existing TensorFlow library functions. Our evaluation shows that the pipelining mechanism of PTF can increase the throughput of a bioinformatics application by 4$\times$ while only increasing its latency by 0.13$\times$. This results in a sustained genome alignment and sorting rate of 321 megabases/second, using the compute and I/O resources of 20 computers.
深度学习(DL)算法已成为数据分析的{ \em de facto}选择。几个DL实现-主要限于单个计算节点-如Caffe,TensorFlow,Theano和Torch已经变得容易使用。能够在大规模系统上执行的分布式DL实现对于解决由科学模拟和实验产生的大数据的计算需求变得越来越重要。然而,分布式DL实现的采用面临重大障碍:1)大多数实现要求DL分析师显著修改其代码-这是一个展示停止器,2)一些分布式DL实现面向云计算系统-这不足以在超级计算机等大规模并行系统上执行。这项工作解决了所有这些问题。我们通过在TensorFlow运行时本身中合并所需的更改来提供分布式内存DL实现。这大大降低了使用分布式TensorFlow实现的进入门槛。我们使用消息传递接口(MPI)--它提供了性能可移植性,特别是因为MPI特定的更改是从用户那里抽象出来的。最后,也可以说是最重要的一点,我们在{ \texttt http://hpc.pnl.gov/matex}上的机器学习Toolkit for Extreme Scale(MaTEX)的保护下,使我们的实现可用于更广泛的用途。我们将我们的实现称为MaTEx-TensorFlow。