MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs
Alibaba Group(阿里巴巴集团);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);Wuhan University(武汉大学);NVIDIA AI Tech Center(英伟达人工智能技术中心);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学)
Huaizheng Zhang, Yuanming Li, Wencong Xiao, Yizheng Huang, Xing Di, Jianxiong Yin, Simon See, Yong Luo, Chiew Tong Lau, Yang You
New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Peking University(北京大学);Key Lab of HCST (PKU), MOE(高可信软件技术教育部重点实验室(北京大学));University of Chinese Academy of Sciences(中国科学院大学);Institute for AI Industry Research, Tsinghua University(清华大学人工智能研究院);Tsinghua University(清华大学);University of Leeds(利兹大学);Huawei Cloud(华为云)
Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services
Tencent(腾讯);Hong Kong University of Science and Technology(香港科技大学);Xiamen University(厦门大学);Macau University of Science and Technology(澳门科技大学);Zhongguancun Lab.(中关村实验室)
Robustness of deep learning classification to adversarial input on GPUs: asynchronous parallel accumulation is a source of vulnerability
Maxeler Technologies, a Groq Company(Maxeler Technologies,Groq公司);ETH Zurich / CSCS(苏黎世联邦理工学院 / CSCS);Georgia Institute of Technology(佐治亚理工学院);Oak Ridge National Laboratory(橡树岭国家实验室)