用于大规模深度学习的高性能I/O
High Performance I/O For Large Scale Deep Learning
摘要:
在千万亿次数据集上训练深度学习(DL)模型对于在语音、视频分析和对象识别等应用中实现具有竞争力的最先进性能至关重要。然而,现有的分布式文件系统并不是针对DL作业的访问模式和可用性需求而开发的。在本文中,我们描述了AIStore,一个高度可扩展的,易于部署的存储系统,和WebDataset,一个基于标准的存储格式和库,允许有效地访问非常大的数据集。我们使用图像分类工作负载并在各种后端上存储训练数据,包括本地SSD,单节点NFS和两个相同的裸机集群:HDFS和AIStore,通过实验比较系统性能。
英文摘要:
Training deep learning (DL) models on petascale datasets is essential for achieving competitive and state-of-the-art performance in applications such as speech, video analytics, and object recognition. However, existing distributed filesystems were not developed for the access patterns and usability requirements of DL jobs. In this paper, we describe AIStore, a highly scalable, easy-to-deploy storage system, and WebDataset, a standards-based storage format and library that permits efficient access to very large datasets. We compare system performance experimentally using image classification workloads and storing training data on a variety of backends, including local SSDs, single-node NFS, and two identical bare-metal clusters: HDFS and AIStore.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。