arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.07518cs.ARcs.AI
收藏

SambaNova SN40L:以数据流和专家组合扩展AI内存墙

SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts

SambaNova Systems, Inc(SambaNova系统公司)
Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang, Karen Li, Yongning Sheng, Josh… 展开作者

Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang, Karen Li, Yongning Sheng, Joshua Brot, Denis Sokolov, Apurv Vivek, Calvin Leung, Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jackson, Mark Luttrell, Manish K. Shah, Edison Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J. Brown, Kunle Olukotun

AI总结:

本文提出Samba-CoE,结合专家组合、流式数据流和三级内存系统,在SambaNova SN40L RDU上实现150专家万亿参数规模,较DGX H100和A100分别加速3.7倍和6.6倍。

摘要:

像GPT-4这样的单体大型语言模型(LLM)为现代生成式AI应用铺平了道路。然而,大规模训练、服务和维护单体LLM仍然极其昂贵且具有挑战性。现代AI加速器计算与内存比率的过度增长造成了内存墙,迫使人们需要新的方法来部署AI。专家组合(CoE)是一种替代性的模块化方法,降低了训练和服务的成本与复杂性。然而,在使用传统硬件时,这种方法面临两个关键挑战:(1)没有融合操作,较小的模型操作强度较低,这使得实现高利用率更具挑战性;(2)托管大量模型可能极其昂贵,或者在动态切换时速度缓慢。在本文中,我们描述了如何将CoE、流式数据流和三级内存系统相结合来扩展AI内存墙。我们描述了Samba-CoE,一个拥有150个专家和总计万亿参数的CoE系统。我们在SambaNova SN40L可重构数据流单元(RDU)上部署了Samba-CoE——这是一种为企业和推理训练应用共同设计的商业数据流加速器架构。该芯片引入了新的三级内存系统,包括片上分布式SRAM、封装内HBM和封装外DDR DRAM。专用的RDU间网络支持在多个插槽上进行扩展。我们在八个RDU插槽上运行的各种基准测试中,与未融合的基线相比,展示了从2倍到13倍的加速。我们表明,对于CoE推理部署,8插槽RDU节点将机器占用空间减少了高达19倍,将模型切换时间加快了15倍到31倍,并且与DGX H100相比实现了3.7倍的总体加速,与DGX A100相比实现了6.6倍的加速。

英文摘要:

Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in compute-to-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Composition of Experts (CoE) is an alternative modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them. In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a commercial dataflow accelerator architecture that has been co-designed for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2$\times$ to 13$\times$ on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19$\times$, speeds up model switching time by 15$\times$ to 31$\times$, and achieves an overall speedup of 3.7$\times$ over a DGX H100 and 6.6$\times$ over a DGX A100.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑