N:M稀疏性是一种新兴的模型压缩方法，正得到越来越多加速器的支持，用于加速深度神经网络中的稀疏矩阵乘法。现有大多数N:M稀疏性方法对网络中所有层采用统一设置来压缩神经网络，或是通过考量每层的参数量启发式地确定逐层配置。然而，针对视觉Transformer（ViTs）获取逐层定制化N:M稀疏配置的方法仍十分匮乏，而ViT通常由参数量相同的Transformer块构成。为解决在支持N:M稀疏性的加速器上为ViT选择合适稀疏配置的难题，本文提出ELSA（Exploiting Layer-wise N:M Sparsity for ViTs，面向视觉Transformer的逐层N:M稀疏性利用方法）。该方法不仅考虑给定加速器支持的所有N:M稀疏等级，还兼顾预期的吞吐量提升，通过以可忽略的精度损失换取ViT模型的内存占用与推理时间降低，充分发挥支持混合稀疏性的加速器的优势。例如，该方法在ImageNet数据集上，使Swin-B和DeiT-B的浮点运算量（FLOPs）均实现了显著的2.9倍降低，而精度仅出现边际下降。本文代码将在论文录用后公开。
英文摘要：
$N{:}M$ sparsity is an emerging model compression method supported by more and more accelerators to speed up sparse matrix multiplication in deep neural networks. Most existing $N{:}M$ sparsity methods compress neural networks with a uniform setting for all layers in a network or heuristically determine the layer-wise configuration by considering the number of parameters in each layer. However, very few methods have been designed for obtaining a layer-wise customized $N{:}M$ sparse configuration for vision transformers (ViTs), which usually consist of transformer blocks involving the same number of parameters. In this work, to address the challenge of selecting suitable sparse configuration for ViTs on $N{:}M$ sparsity-supporting accelerators, we propose ELSA, Exploiting Layer-wise $N{:}M$ Sparsity for ViTs. Considering not only all $N{:}M$ sparsity levels supported by a given accelerator but also the expected throughput improvement, our methodology can reap the benefits of accelerators supporting mixed sparsity by trading off negligible accuracy loss with both memory usage and inference time reduction for ViT models. For instance, our approach achieves a noteworthy 2.9$\times$ reduction in FLOPs for both Swin-B and DeiT-B with only a marginal degradation of accuracy on ImageNet. Our code will be released upon paper acceptance.
Optimal Transport Based Unsupervised Restoration Learning Exploiting Degradation Sparsity
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, China（信息科学与电子工程学院，上海交通大学）；School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China（通信工程学院，杭州电子科技大学）
SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training
School of Microelectronics, BDBC, Beihang University（北京航空航天大学微电子学院，大数据科学与脑机智能高精尖创新中心）；School of Computer Science and Engineering, BDBC, Beihang University（北京航空航天大学计算机科学与工程学院，大数据科学与脑机智能高精尖创新中心）；Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）
Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression
Fujian Key Laboratory of Sensing and Computing for Smart City, Department of Cognitive ScienceSchool of Information Science and Engineering, Xiamen University（厦门大学信息科学与技术学院认知科学系智能城市感知与计算福建省重点实验室）；Beihang University（北京航空航天大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University at Buffalo（布法罗大学）；BestImage, Tencent Technology (Shanghai) Co.,Ltd（腾讯科技（上海）有限公司）；Peng Cheng Laboratory（鹏城实验室）
DeltaDPD: Exploiting Dynamic Temporal Sparsity in Recurrent Neural Networks for Energy-Efficient Wideband Digital Predistortion
Department of Microelectronics, Delft University of Technology（代尔夫特理工大学微电子系）；Ampleon Netherlands B.V.（阿姆佩隆荷兰公司）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University（莱顿先进计算机科学研究所（LIACS），莱顿大学）；School of Electrical and Electronic Engineering, University College Dublin（都柏林大学学院电子与电气工程学院）
Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking
Northeastern University（东北大学）；Science and Technology on Near-Surface Detection Laboratory（近地面探测技术重点实验室）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization（工业智能与系统优化国家级前沿科学中心）
Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance
University of Shanghai for Science and Technology（上海理工大学）；Institute of Photonic Chips（光子芯片研究院）；University of Zürich（苏黎世大学）；ETH Zürich（苏黎世联邦理工学院）；Institute of Neuroinformatics（神经信息学研究所）
HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）
