arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.09147cs.LG
收藏

PARTIME:使用深度神经网络进行可扩展的并行处理

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

University of Siena(锡耶纳大学);University of Florence(佛罗伦萨大学);Université Côte d’Azur(蔚蓝海岸大学)

Enrico Meloni, Lapo Faggi, Simone Marullo, Alessandro Betti, Matteo Tiezzi, Marco Gori, Stefano Melacci

摘要:

在本文中,我们介绍了PARTIME,这是一个用Python编写并基于PyTorch的软件库,专门用于在数据随着时间的推移而连续流动时加速神经网络,用于学习和推理。现有的库被设计为利用数据级并行性,假设样本是批处理的,这在基于流数据的应用程序中并不自然满足。相应地,PARTIME在数据样本从流中变得可用时开始处理每个数据样本。PARTIME包装了实现前馈多层网络的代码,并将逐层处理分配给多个设备,如图形处理单元(GPU)。由于其基于流水线的计算方案,PARTIME允许设备并行执行计算。在推理时,这导致缩放能力理论上与设备数量成线性关系。在学习阶段,PARTIME可以利用非i.i.d.具有随时间平滑演变的样本的流式数据的性质,用于有效的梯度计算。进行实验是为了在在线学习中将PARTIME与经典的非并行神经计算进行经验比较,将操作分布在多达8个NVIDIA GPU上,显示出在设备数量上几乎呈线性的显着加速,减轻了数据传输开销的影响。

英文摘要:

In this paper, we present PARTIME, a software library written in Python and based on PyTorch, designed specifically to speed up neural networks whenever data is continuously streamed over time, for both learning and inference. Existing libraries are designed to exploit data-level parallelism, assuming that samples are batched, a condition that is not naturally met in applications that are based on streamed data. Differently, PARTIME starts processing each data sample at the time in which it becomes available from the stream. PARTIME wraps the code that implements a feed-forward multi-layer network and it distributes the layer-wise processing among multiple devices, such as Graphics Processing Units (GPUs). Thanks to its pipeline-based computational scheme, PARTIME allows the devices to perform computations in parallel. At inference time this results in scaling capabilities that are theoretically linear with respect to the number of devices. During the learning stage, PARTIME can leverage the non-i.i.d. nature of the streamed data with samples that are smoothly evolving over time for efficient gradient computations. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning, distributing operations on up to 8 NVIDIA GPUs, showing significant speedups that are almost linear in the number of devices, mitigating the impact of the data transfer overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑