arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.06468cs.DCcs.ARcs.LGcs.NE
收藏

超越记忆墙:以记忆为中心的深度学习HPC系统案例

Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning

KAIST(韩国科学技术院(KAIST))

Youngeun Kwon, Minsoo Rhu

摘要:

随着用于训练深度学习(DL)模型的模型和数据集的扩展,系统架构师面临着新的挑战,其中之一是存储器容量瓶颈,其中加速器设备内部有限的物理存储器约束了可以研究的算法。我们提出了一个以内存为中心的深度学习系统,它可以透明地扩展加速器可用的内存容量,同时还为并行训练提供快速的设备间通信。我们的建议在设备端互连内本地聚合一个内存模块池,这些内存模块与主机接口解耦,并用作透明内存容量扩展的工具。与传统系统相比,我们的建议在8个DL应用上实现了平均2.8倍的加速,并将系统范围的存储容量增加到数十TB。

英文摘要:

As the models and the datasets to train deep learning (DL) models scale, system architects are faced with new challenges, one of which is the memory capacity bottleneck, where the limited physical memory inside the accelerator device constrains the algorithm that can be studied. We propose a memory-centric deep learning system that can transparently expand the memory capacity available to the accelerators while also providing fast inter-device communication for parallel training. Our proposal aggregates a pool of memory modules locally within the device-side interconnect, which are decoupled from the host interface and function as a vehicle for transparent memory capacity expansion. Compared to conventional systems, our proposal achieves an average 2.8x speedup on eight DL applications and increases the system-wide memory capacity to tens of TBs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑