arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.02037cs.DCcs.AIcs.LG
收藏

MIT超级云数据集

The MIT Supercloud Dataset

MIT(麻省理工学院);Northeastern University(东北大学);US Air Force(美国空军)
Siddharth Samsi, Matthew L Weiss, David Bestor, Baolin Li, Michael Jones, Albert Reuther, Daniel Edelman, William Arcand, Chansup Byun, John Holodnack, Matthew … 展开作者

Siddharth Samsi, Matthew L Weiss, David Bestor, Baolin Li, Michael Jones, Albert Reuther, Daniel Edelman, William Arcand, Chansup Byun, John Holodnack, Matthew Hubbell, Jeremy Kepner, Anna Klein, Joseph McDonald, Adam Michaleas, Peter Michaleas, Lauren Milechin, Julia Mullen, Charles Yee, Benjamin Price, Andrew Prout, Antonio Rosa, Allan Vanterpool, Lindsey McEvoy, Anson Cheng, Devesh Tiwari, Vijay Gadepally

摘要:

人工智能(AI)和机器学习(ML)工作负载在传统高性能计算(HPC)中心和商业云系统中的计算工作负载中所占份额越来越大。这导致HPC集群和商业云的部署方法发生变化,并重新关注优化资源使用的方法,新AI框架的分配和部署,以及支持快速原型设计和部署的功能,如Xueyter笔记本电脑。随着这些变化,需要更好地了解群集/数据中心操作,目标是开发改进的调度策略,识别资源利用率、能源/功耗、故障预测方面的低效率,以及识别违反策略的情况。在本文中,我们介绍了MIT超级云数据集,该数据集旨在促进创新的AI/ML方法来分析大规模HPC和数据中心/云操作。我们提供来自MIT Supercloud系统的详细监控日志,其中包括按作业划分的CPU和GPU使用情况、内存使用情况、文件系统日志和物理监控数据。本文讨论了数据集的细节,收集方法,数据的可用性,并讨论了使用这些数据正在开发的潜在挑战问题。数据集和未来的挑战公告将通过https://dcc.mit.edu提供。

英文摘要:

Artificial intelligence (AI) and Machine learning (ML) workloads are an increasingly larger share of the compute workloads in traditional High-Performance Computing (HPC) centers and commercial cloud systems. This has led to changes in deployment approaches of HPC clusters and the commercial cloud, as well as a new focus on approaches to optimized resource usage, allocations and deployment of new AI frame- works, and capabilities such as Jupyter notebooks to enable rapid prototyping and deployment. With these changes, there is a need to better understand cluster/datacenter operations with the goal of developing improved scheduling policies, identifying inefficiencies in resource utilization, energy/power consumption, failure prediction, and identifying policy violations. In this paper we introduce the MIT Supercloud Dataset which aims to foster innovative AI/ML approaches to the analysis of large scale HPC and datacenter/cloud operations. We provide detailed monitoring logs from the MIT Supercloud system, which include CPU and GPU usage by jobs, memory usage, file system logs, and physical monitoring data. This paper discusses the details of the dataset, collection methodology, data availability, and discusses potential challenge problems being developed using this data. Datasets and future challenge announcements will be available via https://dcc.mit.edu.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑