Diffusion Transformer (DiT) architectures have significantly advanced Text-to-Image (T2I) generation but suffer from prohibitive computational costs and deployment barriers. To address these challenges, we propose an efficient compression framework that transforms the 60-layer dual-stream MMDiT-based Qwen-Image into lightweight models without training from scratch. Leveraging this framework, we introduce Amber-Image, a series of streamlined T2I models. We first derive Amber-Image-10B using a timestep-sensitive depth pruning strategy, where retained layers are reinitialized via local weight averaging and optimized through layer-wise distillation and full-parameter fine-tuning. Building on this, we develop Amber-Image-6B by introducing a hybrid-stream architecture that converts deep-layer dual streams into a single stream initialized from the image branch, further refined via progressive distillation and lightweight fine-tuning. Our approach reduces parameters by 70% and eliminates the need for large-scale data engineering. Notably, the entire compression and training pipeline-from the 10B to the 6B variant-requires fewer than 2,000 GPU hours, demonstrating exceptional cost-efficiency compared to training from scratch. Extensive evaluations on benchmarks like DPG-Bench and LongText-Bench show that Amber-Image achieves high-fidelity synthesis and superior text rendering, matching much larger models.
Less is More: AMBER-AFNO -- a New Benchmark for Lightweight 3D Medical Image Segmentation
Department of Physics E. Pancini, University of Naples Federico II（那不勒斯费德里科二世大学物理系）；Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II（那不勒斯费德里科二世大学电气工程与信息技术系）
AMBER -- Advanced SegFormer for Multi-Band Image Segmentation: an application to Hyperspectral Imaging
University of Naples Federico II（那不勒斯费德里科二世大学）；INFN – Section of Naples（国家核物理研究所那不勒斯分部）；Stazione Zoologica Anton Dohrn（安托万·多尔恩动物站）；INAF - Osservatorio Astronomico di Capodimonte（意大利国家天文研究所卡波迪蒙特天文台）
Transformer architecture has gained popularity due to its ability to scale with large dataset. Consequently, there is a need to reduce the model size and latency, especially for on-device deployment. We focus on vision transformer proposed for image recognition task (Dosovitskiy et al., 2021), and explore the application of different compression techniques such as low rank approximation and pruning for this purpose. Specifically, we investigate a structured pruning method proposed recently in Zhu et al. (2021) and find that mostly feedforward blocks are pruned with this approach, that too, wit…
Dense Vision Transformer Compression with Few Samples
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；Peking University（北京大学）
We propose an end-to-end image compression and analysis model with Transformers, targeting to the cloud-based image classification application. Instead of placing an existing Transformer-based image classification model directly after an image codec, we aim to redesign the Vision Transformer (ViT) model to perform image classification from the compressed features and facilitate image compression with the long-term information from the Transformer. Specifically, we first replace the patchify stem (i.e., image splitting and embedding) of the ViT model with a lightweight image encoder modelled by…
General Compression Framework for Efficient Transformer Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（上海智能信息处理关键实验室，计算机科学与人工智能学院，复旦大学）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；Lenovo Research（联想研究）
One critical component in lossy deep image compression is the entropy model, which predicts the probability distribution of the quantized latent representation in the encoding and decoding modules. Previous works build entropy models upon convolutional neural networks which are inefficient in capturing global dependencies. In this work, we propose a novel transformer-based entropy model, termed Entroformer, to capture long-range dependencies in probability distribution estimation effectively and efficiently. Different from vision transformers in image classification, the Entroformer is highly …
Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；Shanghai Jiao Tong University（上海交通大学）
Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable compl…
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
We show how transformers can be used to vastly simplify neural video compression. Previous methods have been relying on an increasing number of architectural biases and priors, including motion prediction and warping operations, resulting in complex models. Instead, we independently map input frames to representations and use a transformer to model their dependencies, letting it predict the distribution of future representations given the past. The resulting video compression transformer outperforms previous methods on standard video compression data sets. Experiments on synthetic data show th…
CAPformer: Compression-Aware Pre-trained Transformer for Low-Light Image Enhancement
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）
HyCoT: A Transformer-Based Autoencoder for Hyperspectral Image Compression
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电气工程与计算机科学学院）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Peking University（北京大学）；Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Huawei（华为）；Tongji University（同济大学）
Shenzhen International Graduate School Tsinghua University（清华大学深圳国际研究生院）；Institute of Artificial Intelligence Xiamen University（厦门大学人工智能研究院）；China Star Optoelectronics Technology（中国星电科技）；TCL Corporate Research (HK) Co Ltd.（TCL香港企业研发中心）；Department of Electrical and Electronic Engineering The University of Hong Kong（香港大学电子与电气工程系）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Texas A&M University（德州农工大学）；Kwai Inc.（快手公司）
Vision transformers (ViTs) have gained popularity recently. Even without customized image operators such as convolutions, ViTs can yield competitive performance when properly trained on massive data. However, the computational overhead of ViTs remains prohibitive, due to stacking multi-head self-attention modules and else. Compared to the vast literature and prevailing success in compressing convolutional neural networks, the study of Vision Transformer compression has also just emerged, and existing works focused on one or two aspects of compression. This paper proposes a unified ViT compress…
Vision transformers (ViT) have recently attracted considerable attentions, but the huge computational cost remains an issue for practical deployment. Previous ViT pruning methods tend to prune the model along one dimension solely, which may suffer from excessive reduction and lead to sub-optimal model quality. In contrast, we advocate a multi-dimensional ViT compression paradigm, and propose to harness the redundancy reduction from attention head, neuron and sequence dimensions jointly. We firstly propose a statistical dependence based pruning criterion that is generalizable to different dimen…
A Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. Both main and hyper encoders are comprised of a sequence of neural transformation units (NTUs) to analyse and aggregate important information for more compact representation of input image, while the decoders mirror the encoder-side operations to generate pixel-domain image reconstruction from the compressed bitstream. Each NTU is consist of a Swin Transformer Block (STB) and a convolutional layer (Conv) to best emb…
FlattenGPT: Depth Compression for Transformer with Layer Flattening
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Tsinghua University（清华大学）
Spectral Compression Transformer with Line Pose Graph for Monocular 3D Human Pose Estimation
College of Sciences, Northeastern University（东北大学理学院）；Key Laboratory of Differential Equations and Their Applications, Northeastern University（东北大学差分方程及其应用重点实验室）；Department of Informatics, King’s College London（伦敦国王学院信息学院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics, HKISI_CAS（中国科学院香港创新研究院人工智能与机器人创新中心）
Learned image compression methods have exhibited superior rate-distortion performance than classical image compression standards. Most existing learned image compression models are based on Convolutional Neural Networks (CNNs). Despite great contributions, a main drawback of CNN based model is that its structure is not designed for capturing local redundancy, especially the non-repetitive textures, which severely affects the reconstruction quality. Therefore, how to make full use of both global structure and local texture becomes the core problem for learning-based image compression. Inspired …
As a special type of transformer, Vision Transformers (ViTs) are used to various computer vision applications (CV), such as image recognition. There are several potential problems with convolutional neural networks (CNNs) that can be solved with ViTs. For image coding tasks like compression, super-resolution, segmentation, and denoising, different variants of the ViTs are used. The purpose of this survey is to present the first application of ViTs in CV. The survey is the first of its kind on ViTs for CVs to the best of our knowledge. In the first step, we classify different CV applications wh…
ROI-based Deep Image Compression with Swin Transformers
School of Engineering Science, Simon Fraser University（西蒙菲莎大学工程科学学院）；School of Microelectronics, Xi’an Jiaotong University（西安交通大学微电子学院）；Google LLC（谷歌有限责任公司）
Efficient Token Compression for Vision Transformer with Spatial Information Preserved
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；State Key Laboratory of Software Development Environment, Institute of Artificial Intelligence, Beihang University（北京航空航天大学软件开发环境国家重点实验室人工智能研究院）；Terminus Group（特斯联集团）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）
提出Prune and Merge方法，通过集成Token修剪与合并、梯度加权注意力评分及自动识别压缩结构，在DeiT-Small等模型上实现显著加速且精度损失极小。
