arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.14489cs.DCcs.AI
收藏

模型架构与硬件协同设计的必要性研究

The Case for Co-Designing Model Architectures with Hardware

EleutherAI;Ohio State University(俄亥俄州立大学);NVIDIA(英伟达);Contextual AI;Oak Ridge National Lab(橡树岭国家实验室)

Quentin Anthony, Jacob Hatef, Deepak Narayanan, Stella Biderman, Stas Bekman, Junqi Yin, Aamir Shafi, Hari Subramoni, Dhabaleswar Panda

AI总结:

针对深度学习模型设计常忽略GPU架构影响的问题,本文通过分析模型形状超参数对GPU计算内核效率的影响,提出Transformer模型性能优化指南,验证了形状优化可在保精度下将吞吐量最高提升39%。

摘要:

尽管GPU负责训练绝大多数最先进的深度学习模型,但在设计新的深度学习(DL)模型时,其架构的影响往往被忽视。因此,调整深度学习模型使其更适配目标硬件,可显著提升深度学习训练与推理的运行时性能。本文提出了一套供用户最大化Transformer模型运行时性能的指导原则,这些原则是通过仔细考量控制模型形状的各类模型超参数对GPU上执行的底层计算内核效率的影响而制定的。研究发现,与参数量相近但形状未优化的模型相比,具备高效模型形状的模型在保持精度的同时,吞吐量最高可提升39%。

英文摘要:

While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new deep learning (DL) models. As a consequence, modifying a DL model to be more amenable to the target hardware can significantly improve the runtime performance of DL training and inference. In this paper, we provide a set of guidelines for users to maximize the runtime performance of their transformer models. These guidelines have been created by carefully considering the impact of various model hyperparameters controlling model shape on the efficiency of the underlying computation kernels executed on the GPU. We find the throughput of models with efficient model shapes is up to 39\% higher while preserving accuracy compared to models with a similar number of parameters but with unoptimized shapes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑