Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. In this paper, we present a comprehensive survey of the several research efforts that have been presented for the acceleration of transformer networks for Large Language Models using hardware accelerators.
The survey presents the frameworks that have been proposed and then performs a qualitative and quantitative comparison regarding the technology, the processing platform (FPGA, ASIC, In-Memory, GPU), the speedup, the energy efficiency, the performance (GOPs), and the energy efficiency (GOPs/W) of each framework. The main challenge in comparison is that every proposed scheme is implemented on a different process technology making hard a fair comparison. The main contribution of this paper is that we extrapolate the results of the performance and the energy efficiency on the same technology to make a fair comparison; one theoretical and one more practical. We implement part of the LLMs on several FPGA chips to extrapolate the results to the same process technology and then we make a fair comparison of the performance.
MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece（德摩克利特大学特拉克分校集成电路实验室，电气与计算机工程，德摩克利特大学特拉克分校（DUTH），希腊）
ChipGPT: How far are we from natural language hardware design
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, China（处理器国家重点实验室，计算技术研究所，中国科学院，中国）；University of Chinese Academy of Sciences（中国科学院大学）；School of Information Science and Technology, ShanghaiTech University, China（信息科学与技术学院，上海科技大学，中国）
QiMeng-TensorOp: Automatically Generating High-Performance Tensor Operators with Hardware Primitives
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peking University（北京大学）
KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?
Beijing Normal University（北京师范大学）；Beijing Jiaotong University（北京交通大学）；Institute of Automation, CAS（中国科学院自动化研究所）；Peking University（北京大学）；BAAI（北京智源人工智能研究院）
University of Science and Technology of China（中国科学技术大学）
Sampling is an important process in many GNN structures in order to train larger datasets with a smaller computational complexity. However, compared to other processes in GNN (such as aggregate, backward propagation), the sampling process still costs tremendous time, which limits the speed of training. To reduce the time of sampling, hardware acceleration is an ideal choice. However, state of the art GNN acceleration proposal did not specify how to accelerate the sampling process. What's more, directly accelerating traditional sampling algorithms will make the structure of the accelerator very…
With the expansion of cloud services, serious concerns about the privacy of users' data arise due to the exposure of the unencrypted data to the server during computation. Various security primitives are under investigation to preserve privacy while evaluating private data, including Fully Homomorphic Encryption (FHE), Private Set Intersection (PSI), and others. However, the prohibitive processing time of these primitives hinders their practical applications. This work proposes and implements an architecture for accelerating third-generation FHE with Amazon Web Services (AWS) cloud FPGAs, mark…
The design of heterogeneous systems that include domain specific accelerators is a challenging and time-consuming process. While taking into account area constraints, designers must decide which parts of an application to accelerate in hardware and which to leave in software. Moreover, applications in domains such as Extended Reality (XR) offer opportunities for various forms of parallel execution, including loop level, task level and pipeline parallelism. To assist the design process and expose every possible level of parallelism, we present Trireme, a fully automated tool-chain that explores…
MC$^2$A: Enabling Algorithm-Hardware Co-Design for Efficient Markov Chain Monte Carlo Acceleration
Department of Electrical Engineering - MICAS, KU Leuven, Belgium（电气工程系-MICAS，卢森堡大学）；Department of Computer Science - DTAI, KU Leuven, Belgium（计算机科学系-DTAI，卢森堡大学）；Department of Electronics and Nanoengineering, Aalto University（电子与纳米工程系，阿alto大学）
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
BERT is the most recent Transformer-based model that achieves state-of-the-art performance in various NLP tasks. In this paper, we investigate the hardware acceleration of BERT on FPGA for edge computing. To tackle the issue of huge computational complexity and memory footprint, we propose to fully quantize the BERT (FQ-BERT), including weights, activations, softmax, layer normalization, and all the intermediate results. Experiments demonstrate that the FQ-BERT can achieve 7.94x compression for weights with negligible performance loss. We then propose an accelerator tailored for the FQ-BERT an…
Modern hardware design trends have shifted towards specialized hardware acceleration for computationally intensive tasks like machine learning and computer vision. While these complex workloads can be accelerated by commercial GPUs, domain-specific hardware is far more optimal when needing to meet the stringent memory, throughput, and power constraints of mobile and embedded devices. This paper proposes and evaluates a Binary-Enabled Architecture for Neural Network Acceleration (BEANNA), a neural network hardware accelerator capable of processing both floating point and binary network layers. …
In this paper, we demonstrate how Hyperledger Fabric, one of the most popular permissioned blockchains, can benefit from network-attached acceleration. The scalability and peak performance of Fabric is primarily limited by the bottlenecks present in its block validation/commit phase. We propose Blockchain Machine, a hardware accelerator coupled with a hardware-friendly communication protocol, to act as the validator peer. It can be adapted to applications and their smart contracts, and is targeted for a server with network-attached FPGA acceleration card. The Blockchain Machine retrieves block…
School of Computing Informatics and Decision Systems Engineering at Arizona State University（亚利桑那州立大学计算、信息科学与工程学院）；Computer Science and Artificial Intelligence Laboratory at Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；School of Computer Science at University of California, Los Angeles（加利福尼亚大学洛杉矶分校计算机科学学院）；Parallel Computing Lab at Intel Labs（英特尔实验室并行计算实验室）
Machine learning (ML) models are widely used in many important domains. For efficiently processing these computational- and memory-intensive applications, tensors of these over-parameterized models are compressed by leveraging sparsity, size reduction, and quantization of tensors. Unstructured sparsity and tensors with varying dimensions yield irregular computation, communication, and memory access patterns; processing them on hardware accelerators in a conventional manner does not inherently leverage acceleration opportunities. This paper provides a comprehensive survey on the efficient execu…
A two-dimensional Ising model with nearest-neighbors ferromagnetic interactions is implemented in a Field Programmable Gate Array (FPGA) board.Extensive Monte Carlo simulations were carried out using an efficient hardware representation of individual spins and a combined global-local LFSR random number generator. Consistent results regarding the descriptive properties of magnetic systems, like energy, magnetization and susceptibility are obtained while a speed-up factor of approximately 6 times is achieved in comparison to previous FPGA-based published works and almost $10^4$ times in comparis…
Georgia Institute of Technology（佐治亚理工学院）；Nanjing University（南京大学）；Rice University（莱斯大学）
Multiplication is arguably the most cost-dominant operation in modern deep neural networks (DNNs), limiting their achievable efficiency and thus more extensive deployment in resource-constrained applications. To tackle this limitation, pioneering works have developed handcrafted multiplication-free DNNs, which require expert knowledge and time-consuming manual iteration, calling for fast development tools. To this end, we propose a Neural Architecture Search and Acceleration framework dubbed NASA, which enables automated multiplication-reduced DNN development and integrates a dedicated multipl…
Georgia Institute of Technology（佐治亚理工学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California, San Diego（加州大学圣地亚哥分校）
The data revolution is fueled by advances in machine learning, databases, and hardware design. Programmable accelerators are making their way into each of these areas independently. As such, there is a void of solutions that enables hardware acceleration at the intersection of these disjoint fields. This paper sets out to be the initial step towards a unifying solution for in-Database Acceleration of Advanced Analytics (DAnA). Deploying specialized hardware, such as FPGAs, for in-database analytics currently requires hand-designing the hardware and manually routing the data. Instead, DAnA auto…
Today, artificial neural networks are one of the major innovators pushing the progress of machine learning. This has particularly affected the development of neural network accelerating hardware. However, since most of these architectures require specialized toolchains, there is a certain amount of additional effort for developers each time they want to make use of a new deep learning accelerator. Furthermore the flexibility of the device is bound to the architecture itself, as well as to the functionality of the runtime environment. In this paper we propose a toolflow using TensorFlow as fron…
