This study presents an efficient implementation of transformer architectures in Field-Programmable Gate Arrays(FPGAs) using hls4ml. We demonstrate the strategy for implementing the multi-head attention, softmax, and normalization layer and evaluate three distinct models. Their deployment on VU13P FPGA chip achieved latency less than 2us, demonstrating the potential for real-time applications. HLS4ML compatibility with any TensorFlow-built transformer model further enhances the scalability and applicability of this work. Index Terms: FPGAs, machine learning, transformers, high energy physics, LIGO
FTRANS: Energy-Efficient Acceleration of Transformers using FPGA
University of Connecticut(康涅狄格大学);Stevens Institute of Technology(斯蒂文斯理工学院);Syracuse University(雪城大学);Microsoft Corporation(微软公司);Oak Ridge National Laboratory(橡树岭国家实验室);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)
Low Latency Edge Classification GNN for Particle Trajectory Tracking on FPGAs
National Yang Ming Chiao Tung University(国立阳明交通大学);University of California San Diego(加利福尼亚大学圣迭戈分校);University of Washington(华盛顿大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
LL-GNN: Low Latency Graph Neural Networks on FPGAs for High Energy Physics
Imperial College London(伦敦帝国理工学院);Southeast University(东南大学);AMD Adaptive and Embedded Computing Group (AECG) Labs(AMD自适应与嵌入式计算集团实验室);European Organization for Nuclear Research (CERN)(欧洲核子研究中心)
Enabling Low-Latency Machine learning on Radiation-Hard FPGAs with hls4ml
Massachusetts Institute of Technology (MIT)(麻省理工学院);Department of Experimental Physics, European Organization for Nuclear Research (CERN)(欧洲核子研究中心(CERN)实验物理系);University of Milano-Bicocca (Italy)(米兰-布雷拉大学)
Towards Tensor Network Models for Low-Latency Jet Tagging on FPGAs
Dipartimento di Fisica e Astronomia "Galileo Galilei", University of Padua(伽利略·伽利莱物理与天文学系,帕多瓦大学);PlanQC GmbH(PlanQC公司);Institute for Complex Quantum Systems, Ulm University(复杂量子系统研究所,乌尔姆大学);European Organization for Nuclear Research (CERN)(欧洲核子研究中心);Padua Quantum Technologies Research Center, University of Padua(帕多瓦量子技术研究中心,帕多瓦大学)
Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml
University of Washington(华盛顿大学);Massachusetts Institute of Technology(麻省理工学院);SLAC National Accelerator Laboratory(SLAC国家加速器实验室);European Organization for Nuclear Research (CERN)(欧洲核子研究组织(CERN))
Ultra-low latency quantum-inspired machine learning predictors implemented on FPGA
Istituto Nazionale di Fisica Nucleare (INFN)(国家核物理研究所);Dipartimendo di Fisica e Astronomia (DFA)(物理与天文学系);University of Padua(帕多瓦大学);Dipartimendo di Ingegneria dell’Informazione (DEI)(信息工程系);Dipartimendo di Ingegneria Industriale (DII)(工业工程系);Padua Quantum Technology Research Center(帕多瓦量子技术研究中心);Tensor AI Solutions GmbH(张量人工智能解决方案有限公司)
HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAs
California Institute of Technology(加利福尼亚理工学院);Imperial College London(伦敦帝国理工学院);ETH Zurich(苏黎世联邦理工学院);Institute of Physics Belgrade(贝尔格莱德物理研究所);Fermilab(费米实验室)
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Northeastern University(东北大学);Simon Fraser University(西蒙弗雷泽大学);University of California, Irvine(加州大学欧文分校);ETH Zurich(苏黎世联邦理工学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
视觉Transformer(ViT)正在出现,在计算机视觉任务中具有显著提高的准确性。然而,其复杂的体系结构和巨大的计算/存储需求迫切需要新的硬件加速器的设计方法。这项工作提出了一个FPGA感知的自动ViT加速框架的基础上提出的混合方案量化。据我们所知,这是第一个基于FPGA的ViT加速框架,探索模型量化。与最先进的ViT量化工作(算法方法,只有没有硬件加速)相比,我们的量化实现了0.47%至1.36%的高的Top-1精度在相同的位宽。与32位浮点基线FPGA加速器相比,我们的加速器的帧速率提高了约5.6倍(即,56.8 FPS vs. 10.0 FPS),DeiT-base在ImageNet数据集上的准确率下降了0.71%。
Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization
Department of Electrical and Computer Engineering, University of California, Santa Barbara(加州大学圣芭芭拉分校电子与计算机工程系);Intel Corporation(英特尔公司);Department of Computer Science, North Carolina State University(北卡罗来纳州立大学计算机科学系);School of Electrical and Computer Engineering, Georgia Institute of Technology(佐治亚理工学院电气与计算机工程学院)
A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs
Department of Electrical Engineering and Computer Science, University of Arkansas(电气工程与计算机科学系,亚拉巴马大学);Department of Computer Science and Engineering, University of South Carolina(计算机科学与工程系,南卡罗来纳大学)
A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining
University of Connecticut(康涅狄格大学);Stevens Institute of Technology(史蒂文斯理工学院);Pacific Northwest National Laboratory(太平洋西北国家实验室);George Mason University(乔治梅森大学);Lehigh University(理海大学)
Hardware-Accelerated Event-Graph Neural Networks for Low-Latency Time-Series Classification on SoC FPGA
Graduate School of Science and Technology(科学与技术研究生院);Embedded Vision Systems Group, Computer Vision Laboratory(嵌入式视觉系统组,计算机视觉实验室);AGH University of Krakow(克拉科夫AGH大学);CEA-List, Université Grenoble Alpes(CEA-List,格勒诺布尔阿尔卑斯大学)
Low latency optical-based mode tracking with machine learning deployed on FPGAs on a tokamak
Columbia University(哥伦比亚大学);Fermi National Accelerator Laboratory(费米国家加速器实验室);Lehigh University(利哈伊大学);Northwestern University(西北大学);Drexel University(德雷塞尔大学)
Fast inference of Boosted Decision Trees in FPGAs for particle physics
European Organization for Nuclear Research (CERN)(欧洲核子研究组织(CERN));Columbia University(哥伦比亚大学);University of California San Diego(加利福尼亚大学圣迭戈分校);Massachusetts Institute of Technology(麻省理工学院);Rhodes College(罗德学院);Fermi National Accelerator Laboratory(费米国家加速器实验室);Institute of Physics Belgrade(贝尔格莱德物理研究所);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
SparsePixels: Efficient Convolution for Sparse Data on FPGAs
University of Pennsylvania, USA(美国宾夕法尼亚大学);Institute of Physics Belgrade, Serbia(塞尔维亚贝尔格莱德物理研究所);Massachusetts Institute of Technology, USA(美国麻省理工学院);Institute for Artificial Intelligence and Fundamental Interactions, USA(美国人工智能与基本相互作用研究所)
Enabling Vibration-Based Gesture Recognition on Everyday Furniture via Energy-Efficient FPGA Implementation of 1D Convolutional Networks
1 Ubiquitous Computing Systems Laboratory, Nara Institute of Science;Technology, Nara, Japan 2 Intelligent Embedded Systems Laboratory, University of Duisburg-Essen, Duisburg, Germany 3 PALUNO, The Ruhr Institute for Software Technology, Essen, Germany 4 RIKEN, Center for Advanced Intelligence Project, Tokyo, Japan
Fast convolutional neural networks on FPGAs with hls4ml
European Organization for Nuclear Research (CERN)(欧洲核子研究组织);Institute of Physics Belgrade(贝尔格莱德物理研究所);Politecnico di Milano(米兰理工大学);California Institute of Technology(加州理工学院);Chalmers University of Technology(查尔姆斯理工大学);Zenseact(Zenseact公司);ICEPP, University of Tokyo(东京大学国际基本粒子物理研究所);Columbia University(哥伦比亚大学);University of California San Diego(加州大学圣地亚哥分校);Massachusetts Institute of Technology(麻省理工学院);Fermi National Accelerator Laboratory(费米国家加速器实验室);Purdue University(普渡大学);HawkEye360(鹰眼360公司);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Rhodes College(罗德学院)