In the world of deep learning, Transformer models have become very significant, leading to improvements in many areas from understanding language to recognizing images, covering a wide range of applications. Despite their success, the deployment of these models in real-time applications, particularly on edge devices, poses significant challenges due to their quadratic computational intensity and memory demands. To overcome these challenges we introduce a novel Hybrid Dynamic Pruning (HDP), an efficient algorithm-architecture co-design approach that accelerates transformers using head sparsity, block sparsity and approximation opportunities to reduce computations in attention and reduce memory access. With the observation of the huge redundancy in attention scores and attention heads, we propose a novel integer-based row-balanced block pruning to prune unimportant blocks in the attention matrix at run time, also propose integer-based head pruning to detect and prune unimportant heads at an early stage at run time. Also we propose an approximation method that reduces attention computations. To efficiently support these methods with lower latency and power efficiency, we propose a HDP co-processor architecture.
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国);School of Software, Soongsil University, Seoul, South Korea(软件学院,顺天大学,首尔,韩国)
Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因茨·赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所(BIFOLD))
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning
University of Connecticut(康涅狄格大学);Northeastern University(东北大学);Syracuse University(雪城大学);Microsoft Corporation(微软公司);Stevens Institute of Technology(史蒂文斯理工学院)
PSViT: A Methodology for Structurally Pruning Spiking Vision Transformers
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi(eBRAIN实验室,工程系,纽约大学(NYU)阿布扎赫德分校);New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)(纽约大学(NYU)阿布扎赫德分校,阿拉伯联合酋长国(UAE))
LEAP: Learnable Pruning for Transformer-based Models
Microsoft(微软);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Berkeley(加州大学伯克利分校);International Computer Science Institute(国际计算机科学研究所)
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
School of Computer Science and Technology, Guangdong University of Technology(广东工业大学计算机科学与技术学院);College of Information and Artificial Intelligence, Yangzhou University(扬州大学信息与人工智能学院)
PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi(eBRAIN实验室,工程系,纽约大学(NYU)阿布扎克分校);New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)(纽约大学(NYU)阿布扎克分校,阿拉伯联合酋长国(UAE))
TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);College of Computer Science and Technology(计算机科学与技术学院);College of Computer Science and Electronic Engineering(计算机科学与电子工程学院)
Pruning and Malicious Injection: A Retraining-Free Backdoor Attack on Transformer Models
Division of Computer Science and Engineering, Louisiana State University(计算机科学与工程系,路易斯安那州立大学);Division of Electrical and Computer Engineering, Louisiana State University(电气与计算机工程系,路易斯安那州立大学);School of Engineering and Science, Stevens Institute of Technology(工程与科学学院,史蒂文斯理工学院)
Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization
College of Science and Engineering, James Cook University(詹姆斯库克大学科学与工程学院);College of Health Care Sciences, James Cook University(詹姆斯库克大学健康护理科学学院);College of Public Health, Medical, and Vet Sciences, James Cook University(詹姆斯库克大学公共卫生、医学与兽医学科学学院)
Unveiling Optimal SDG Pathways: An Innovative Approach Leveraging Graph Pruning and Intent Graph for Effective Recommendations
College of Computer, Nanjing University of Posts and Telecommunications(南京邮电大学计算机学院);Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences(中国科学院地理科学与资源研究所);University of Chinese Academy of Sciences(中国科学院大学)
University of Electronic Science and Technology of China(电子科技大学);Clinical Hospital of Chengdu Brain Science Institute(成都脑科学医院);MOE-K Lab for NeuroInformation(神经信息MOE-K实验室);Int Institute of Brain-Apparatus Communication(脑机通讯国际研究所);School of Life Science and Technology(生命科学与技术学院)