arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.03842cs.LGcs.ARstat.ML
收藏

GOBO:基于注意力的量化NLP模型,用于低延迟和节能推理

GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

University of Toronto(多伦多大学)

Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, Andreas Moshovos

摘要:

基于注意力的模型在各种自然语言理解任务中取得了显著的成功。然而,高效的执行仍然是一个挑战,这些模型是内存限制,由于其大量的参数。我们提出了GOBO,模型量化技术,压缩的绝大多数(通常为99.9%)的32位浮点参数的最先进的BERT模型及其变种到3位,同时保持其准确性。与其他量化方法不同,GOBO不需要微调或重新训练来补偿量化误差。我们提出了两个实际的硬件应用GOBO。在第一个GOBO减少了内存存储和交通,并因此推理延迟和能源消耗。这种GOBO内存压缩机制与许多架构兼容;我们使用TPU,Eyeriss和使用Tensor Core类单元的架构来演示它。其次,我们提出了一个共同设计的硬件架构,也减少了计算。独特的是,GOBO架构即使在计算期间也保持了3b中的大部分权重,这一特性:(1)使处理元件面积有效,允许我们在每单位面积上封装更多的计算能力,(2)用加法取代大多数乘法累加,以及(3)通过放大片上存储器容量来减少片外流量。

英文摘要:

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑