arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.02825cs.ARcs.DC
收藏

未来硬件上未来Transformer的计算与通信扩展对比

Computation vs. Communication Scaling for Future Transformers on Future Hardware

Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, Matthew D. Sinclair

AI总结:

本研究从多维度分析未来Transformer模型在未来硬件上的计算与通信扩展关系,通过算子模型推算降低剖析成本,发现通信占比将达40-75%且难以被隐藏,为未来高效训练大模型的系统设计提供指导。

摘要:

神经网络模型的扩展已在各类机器学习问题中带来了显著的质量提升。然而,这种扩展增加了对高效分布式训练技术的依赖。因此,与其他分布式计算场景一样,了解随着模型扩展和硬件演进,计算与通信的相对扩展关系至关重要。针对这一问题的细致研究能更好地指导未来系统的设计,使其能够高效训练未来的大型模型。\n 为此,本工作针对未来硬件上的未来Transformer模型,从算法、实证、硬件演进多维度开展了计算与通信(Comp-vs.-Comm)扩展的综合分析。首先,我们的算法分析表明,随着模型扩展,计算通常比通信更具优势。但由于内存容量的扩展速度慢于计算,这些趋势正面临压力。接下来,我们通过实证研究未来硬件上未来模型的计算与通信扩展情况,量化了这一优势。为避免在多种设置下对大量Transformer模型进行性能剖析,我们提取执行区域并使用算子模型推算成本。这使得我们能够准确研究数百种未来模型/硬件场景(误差低于15%),并将性能剖析成本降低2100倍。我们的实验表明,随着模型和硬件的演进,通信将占运行时的很大一部分(40-75%)。此外,在当今模型中被重叠计算隐藏的通信,在未来更大的模型中往往无法被隐藏。总体而言,本工作强调了通信在模型扩展过程中日益重要的作用,并讨论了有助于解决这一问题的技术和即将推出的技术。

英文摘要:

Scaling neural network models has delivered dramatic quality gains across ML problems. However, this scaling has increased the reliance on efficient distributed training techniques. Accordingly, as with other distributed computing scenarios, it is important to understand how will compute and communication scale relative to one another as models scale and hardware evolves? A careful study which answers this question can better guide the design of future systems which can efficiently train future large models. Accordingly, this work provides a comprehensive multi-axial (algorithmic, empirical, hardware evolution) analysis of compute vs. communication (Comp-vs.-Comm) scaling for future Transformer models on future hardware. First, our algorithmic analysis shows that compute generally enjoys an edge over communication as models scale. However, since memory capacity scales slower than compute, these trends are being stressed. Next, we quantify this edge by empirically studying how Comp-vs.-Comm scales for future models on future hardware. To avoid profiling numerous Transformer models across many setups, we extract execution regions and project costs using operator models. This allows a spectrum (hundreds) of future model/hardware scenarios to be accurately studied ($<$15% error), and reduces profiling costs by 2100$\times$. Our experiments show that communication will be a significant portion (40-75%) of runtime as models and hardware evolve. Moreover, communication which is hidden by overlapped computation in today's models often cannot be hidden in future, larger models. Overall, this work highlights the increasingly large role communication will play as models scale and discusses techniques and upcoming technologies that can help address it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑