一种面向CPU上基于Transformer的语言模型的高效稀疏推理软件加速器
An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs
AI总结：
提出一种面向CPU的高效稀疏推理软件栈，利用Intel Deep Learning Boost优化SpMM计算，在多种Transformer模型和稀疏率下显著超越现有稀疏库与主流推理框架，实现大幅加速。
摘要：
近年来，基于Transformer的语言模型已成为自然语言处理任务的标准方法。然而，工业应用中严格的吞吐量和延迟要求限制了它们的采用。为了缓解这一差距，结构化剪枝等模型压缩技术正被用于提高推理效率。然而，大多数现有的神经网络推理运行时缺乏对结构化稀疏性的充分支持。在本文中，我们提出了一种高效的稀疏深度学习推理软件栈，适用于权重以恒定块大小进行剪枝的基于Transformer的语言模型。我们的稀疏软件加速器利用Intel Deep Learning Boost来最大化CPU上稀疏矩阵-密集矩阵乘法（通常缩写为SpMM）的性能。在5种代表性稀疏率（70%、75%、80%、85%、90%）下，我们的SpMM内核在各种GEMM形状上的性能比现有的稀疏库（oneMKL、TVM和LIBXSMM）高出一个数量级。此外，我们的SpMM内核与工业界广泛使用的优化密集库oneDNN的密集GEMM内核相比，实现了高达5倍的加速。我们将稀疏加速器应用于广泛使用的基于Transformer的语言模型，包括Bert-Mini、DistilBERT、Bert-Base和BERT-Large。在Amazon Web Services的Xeon上，在代理生产延迟约束下，与相同配置下的Neural Magic的Deepsparse相比，我们的稀疏推理软件实现了高达1.5倍的加速。我们还将该解决方案与两种基于框架的推理解决方案ONNX Runtime和PyTorch进行了比较，在延迟约束下，在Xeon上相比ONNX Runtime实现了高达37倍的加速，相比PyTorch实现了345倍的加速。所有源代码均在Github上公开：https://github.com/intel/intel-extension-for-transformers。
英文摘要：
In recent years, Transformer-based language models have become the standard approach for natural language processing tasks. However, stringent throughput and latency requirements in industrial applications are limiting their adoption. To mitigate the gap, model compression techniques such as structured pruning are being used to improve inference efficiency. However, most existing neural network inference runtimes lack adequate support for structured sparsity. In this paper, we propose an efficient sparse deep learning inference software stack for Transformer-based language models where the weights are pruned with constant block size. Our sparse software accelerator leverages Intel Deep Learning Boost to maximize the performance of sparse matrix - dense matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an order of magnitude on a wide range of GEMM shapes under 5 representative sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library widely used in industry. We apply our sparse accelerator on widely-used Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base, and BERT-Large. Our sparse inference software shows up to 1.5x speedup over Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web Services under proxy production latency constraints. We also compare our solution with two framework-based inference solutions, ONNX Runtime and PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over PyTorch on Xeon under the latency constraints. All the source code is publicly available on Github: https://github.com/intel/intel-extension-for-transformers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
