揭开BERT的神秘面纱：加速器设计的启示
Demystifying BERT: Implications for Accelerator Design
摘要：
自然语言处理（NLP）中的迁移学习，如使用BERT（来自Transformer的双向编码器表示）等模型实现的，已经通过可以解决具有挑战性的语言问题的模型显着改善了语言表示。因此，这些应用正在推动未来系统的需求。因此，我们专注于BERT，最流行的NLP迁移学习算法之一，以确定其算法行为如何指导未来的加速器设计。为此，我们仔细分析了BERT训练，并确定了加速器设计中值得注意的关键算法行为。我们观察到，虽然表现为矩阵乘法的计算主导了BERT的整体运行时间，但与许多卷积神经网络一样，内存密集型计算也很突出。我们描述这些计算，迄今为止很少受到关注。此外，我们还确定计算密集型BERT计算的异构性，并讨论软件和可能的硬件机制，以进一步优化这些计算。最后，我们讨论了这些行为的影响，随着网络变得越来越大，并使用分布式训练环境，以及如何技术，如微训练和混合精度训练规模。总的来说，我们的分析确定了整体解决方案，以优化系统的BERT类模型。
英文摘要：
Transfer learning in natural language processing (NLP), as realized using models like BERT (Bi-directional Encoder Representation from Transformer), has significantly improved language representation with models that can tackle challenging language problems. Consequently, these applications are driving the requirements of future systems. Thus, we focus on BERT, one of the most popular NLP transfer learning algorithms, to identify how its algorithmic behavior can guide future accelerator design. To this end, we carefully profile BERT training and identify key algorithmic behaviors which are worthy of attention in accelerator design. We observe that while computations which manifest as matrix multiplication dominate BERT's overall runtime, as in many convolutional neural networks, memory-intensive computations also feature prominently. We characterize these computations, which have received little attention so far. Further, we also identify heterogeneity in compute-intensive BERT computations and discuss software and possible hardware mechanisms to further optimize these computations. Finally, we discuss implications of these behaviors as networks get larger and use distributed training environments, and how techniques such as micro-batching and mixed-precision training scale. Overall, our analysis identifies holistic solutions to optimize systems for BERT-like models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
