扩散Transformer（DiT）在图像和视频生成方面表现出色，但由于自注意力算子的二次复杂度而面临计算挑战。我们提出DiTFastAttn，一种训练后压缩方法，以缓解DiT的计算瓶颈。我们识别出DiT推理过程中注意力计算的三个关键冗余：（1）空间冗余，许多注意力头关注局部信息；（2）时间冗余，相邻步骤的注意力输出之间具有高度相似性；（3）条件冗余，条件推理和无条件推理表现出显著相似性。我们提出三种技术来减少这些冗余：（1）带残差共享的窗口注意力以减少空间冗余；（2）跨时间步的注意力共享以利用步骤之间的相似性；（3）跨CFG的注意力共享以在条件生成期间跳过冗余计算。我们将DiTFastAttn应用于图像生成任务的DiT、PixArt-Sigma以及视频生成任务的OpenSora。我们的结果表明，对于图像生成，我们的方法在高分辨率（2k x 2k）生成中最多减少76%的注意力FLOPs，并实现高达1.8倍的端到端加速。
英文摘要：
Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DiT, PixArt-Sigma for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up to 76% of the attention FLOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation.
Transformer architecture has gained popularity due to its ability to scale with large dataset. Consequently, there is a need to reduce the model size and latency, especially for on-device deployment. We focus on vision transformer proposed for image recognition task (Dosovitskiy et al., 2021), and explore the application of different compression techniques such as low rank approximation and pruning for this purpose. Specifically, we investigate a structured pruning method proposed recently in Zhu et al. (2021) and find that mostly feedforward blocks are pruned with this approach, that too, wit…
Dense Vision Transformer Compression with Few Samples
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
General Compression Framework for Efficient Transformer Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（上海智能信息处理关键实验室，计算机科学与人工智能学院，复旦大学）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；Lenovo Research（联想研究）
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Peking University（北京大学）；Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Huawei（华为）；Tongji University（同济大学）
Shenzhen International Graduate School Tsinghua University（清华大学深圳国际研究生院）；Institute of Artificial Intelligence Xiamen University（厦门大学人工智能研究院）；China Star Optoelectronics Technology（中国星电科技）；TCL Corporate Research (HK) Co Ltd.（TCL香港企业研发中心）；Department of Electrical and Electronic Engineering The University of Hong Kong（香港大学电子与电气工程系）
Spectral Compression Transformer with Line Pose Graph for Monocular 3D Human Pose Estimation
College of Sciences, Northeastern University（东北大学理学院）；Key Laboratory of Differential Equations and Their Applications, Northeastern University（东北大学差分方程及其应用重点实验室）；Department of Informatics, King’s College London（伦敦国王学院信息学院）
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；Peking University（北京大学）
We propose an end-to-end image compression and analysis model with Transformers, targeting to the cloud-based image classification application. Instead of placing an existing Transformer-based image classification model directly after an image codec, we aim to redesign the Vision Transformer (ViT) model to perform image classification from the compressed features and facilitate image compression with the long-term information from the Transformer. Specifically, we first replace the patchify stem (i.e., image splitting and embedding) of the ViT model with a lightweight image encoder modelled by…
One critical component in lossy deep image compression is the entropy model, which predicts the probability distribution of the quantized latent representation in the encoding and decoding modules. Previous works build entropy models upon convolutional neural networks which are inefficient in capturing global dependencies. In this work, we propose a novel transformer-based entropy model, termed Entroformer, to capture long-range dependencies in probability distribution estimation effectively and efficiently. Different from vision transformers in image classification, the Entroformer is highly …
Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；Shanghai Jiao Tong University（上海交通大学）
Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable compl…
As a special type of transformer, Vision Transformers (ViTs) are used to various computer vision applications (CV), such as image recognition. There are several potential problems with convolutional neural networks (CNNs) that can be solved with ViTs. For image coding tasks like compression, super-resolution, segmentation, and denoising, different variants of the ViTs are used. The purpose of this survey is to present the first application of ViTs in CV. The survey is the first of its kind on ViTs for CVs to the best of our knowledge. In the first step, we classify different CV applications wh…
CAPformer: Compression-Aware Pre-trained Transformer for Low-Light Image Enhancement
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）
HyCoT: A Transformer-Based Autoencoder for Hyperspectral Image Compression
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电气工程与计算机科学学院）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））
Under stereo settings, the performance of image JPEG artifacts removal can be further improved by exploiting the additional information provided by a second view. However, incorporating this information for stereo image JPEG artifacts removal is a huge challenge, since the existing compression artifacts make pixel-level view alignment difficult. In this paper, we propose a novel parallax transformer network (PTNet) to integrate the information from stereo image pairs for stereo image JPEG artifacts removal. Specifically, a well-designed symmetric bi-directional parallax transformer module is p…
We show how transformers can be used to vastly simplify neural video compression. Previous methods have been relying on an increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. Instead, we independently map input frames to representations and use a transformer to model their dependencies, letting it predict the distribution of future representations given the past. The resulting video compression transformer outperforms previous methods on standard video compression data sets. Experiments on synthetic data show th…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Texas A&M University（德州农工大学）；Kwai Inc.（快手公司）
Vision transformers (ViTs) have gained popularity recently. Even without customized image operators such as convolutions, ViTs can yield competitive performance when properly trained on massive data. However, the computational overhead of ViTs remains prohibitive, due to stacking multi-head self-attention modules and else. Compared to the vast literature and prevailing success in compressing convolutional neural networks, the study of Vision Transformer compression has also just emerged, and existing works focused on one or two aspects of compression. This paper proposes a unified ViT compress…
Vision transformers (ViT) have recently attracted considerable attentions, but the huge computational cost remains an issue for practical deployment. Previous ViT pruning methods tend to prune the model along one dimension solely, which may suffer from excessive reduction and lead to sub-optimal model quality. In contrast, we advocate a multi-dimensional ViT compression paradigm, and propose to harness the redundancy reduction from attention head, neuron and sequence dimensions jointly. We firstly propose a statistical dependence based pruning criterion that is generalizable to different dimen…
A Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. Both main and hyper encoders are comprised of a sequence of neural transformation units (NTUs) to analyse and aggregate important information for more compact representation of input image, while the decoders mirror the encoder-side operations to generate pixel-domain image reconstruction from the compressed bitstream. Each NTU is consist of a Swin Transformer Block (STB) and a convolutional layer (Conv) to best emb…
FlattenGPT: Depth Compression for Transformer with Layer Flattening
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Tsinghua University（清华大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics, HKISI_CAS（中国科学院香港创新研究院人工智能与机器人创新中心）
Learned image compression methods have exhibited superior rate-distortion performance than classical image compression standards. Most existing learned image compression models are based on Convolutional Neural Networks (CNNs). Despite great contributions, a main drawback of CNN based model is that its structure is not designed for capturing local redundancy, especially the non-repetitive textures, which severely affects the reconstruction quality. Therefore, how to make full use of both global structure and local texture becomes the core problem for learning-based image compression. Inspired …
ROI-based Deep Image Compression with Swin Transformers
School of Engineering Science, Simon Fraser University（西蒙菲莎大学工程科学学院）；School of Microelectronics, Xi’an Jiaotong University（西安交通大学微电子学院）；Google LLC（谷歌有限责任公司）
Efficient Token Compression for Vision Transformer with Spatial Information Preserved
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；State Key Laboratory of Software Development Environment, Institute of Artificial Intelligence, Beihang University（北京航空航天大学软件开发环境国家重点实验室人工智能研究院）；Terminus Group（特斯联集团）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）
提出Prune and Merge方法，通过集成Token修剪与合并、梯度加权注意力评分及自动识别压缩结构，在DeiT-Small等模型上实现显著加速且精度损失极小。
