Transformers are ubiquitous in Natural Language Processing (NLP) tasks, but they are difficult to be deployed on hardware due to the intensive computation. To enable low-latency inference on resource-constrained hardware platforms, we propose to design Hardware-Aware Transformers (HAT) with neural architecture search. We first construct a large design space with $\textit{arbitrary encoder-decoder attention}$ and $\textit{heterogeneous layers}$. Then we train a $\textit{SuperTransformer}$ that covers all candidates in the design space, and efficiently produces many $\textit{SubTransformers}$ with weight sharing. Finally, we perform an evolutionary search with a hardware latency constraint to find a specialized $\textit{SubTransformer}$ dedicated to run fast on the target hardware. Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device). When running WMT'14 translation task on Raspberry Pi-4, HAT can achieve $\textbf{3}\times$ speedup, $\textbf{3.7}\times$ smaller size over baseline Transformer; $\textbf{2.7}\times$ speedup, $\textbf{3.6}\times$ smaller size over Evolved Transformer with $\textbf{12,041}\times$ less search cost and no performance loss. HAT code is https://github.com/mit-han-lab/hardware-aware-transformers.git
HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence(人工智能研究院);Hefei Comprehensive National Science Center(合肥综合性国家科学中心);Huawei Technologies(华为技术有限公司);Peking University(北京大学)
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
University of Copenhagen(哥本哈根大学);CSIRO Data61(澳大利亚联邦科学与工业研究组织Data61);Athens University of Economics and Business(雅典经济与商业大学);Pioneer Centre for AI(人工智能先锋中心)
Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages
Department of Computer Science;Engineering, Indian Institute of Technology, Kharagpur, West Bengal, India;Department of Electrical Engineering , Indian Institute of Technology, Kharagpur, West Bengal, India
Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages
Universitat Politècnica de València(瓦伦西亚理工大学);Constantine 2 University(君士坦丁2大学);ESCF de Constantine(君士坦丁ESCF);Hamad Bin Khalifa University(哈马德·本·哈利法大学)
Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languages
MKSSS Cummins College of Engineering for Women(MKSSS卡明斯女子工程学院);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(RUDN大学));Joint Institute for Nuclear Research(联合核研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学)
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning
University of Connecticut(康涅狄格大学);Northeastern University(东北大学);Syracuse University(雪城大学);Microsoft Corporation(微软公司);Stevens Institute of Technology(史蒂文斯理工学院)
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Université Lumière Lyon 2(里昂第二大学);CNRS(法国国家科学研究中心);Ecole Centrale de Lyon(里昂中央理工学院);INSA Lyon(里昂国立应用科学学院);Universite Claude Bernard Lyon 1(克洛德·贝尔纳里昂第一大学);LIRIS, UMR5205(里昂信息图像与科学计算实验室,UMR5205)