Recent vision architectures and self-supervised training methods enable vision models that are extremely accurate and general, but come with massive parameter and computational costs. In practical settings, such as camera traps, users have limited resources, and may fine-tune a pretrained model on (often limited) data from a small set of specific categories of interest. These users may wish to make use of modern, highly-accurate models, but are often computationally constrained. To address this, we ask: can we quickly compress large generalist models into accurate and efficient specialists? For this, we propose a simple and versatile technique called Few-Shot Task-Aware Compression (TACO). Given a large vision model that is pretrained to be accurate on a broad task, such as classification over ImageNet-22K, TACO produces a smaller model that is accurate on specialized tasks, such as classification across vehicle types or animal species. Crucially, TACO works in few-shot fashion, i.e. only a few task-specific samples are used, and the procedure has low computational overheads. We validate TACO on highly-accurate ResNet, ViT/DeiT, and ConvNeXt models, originally trained on ImageNet, LAION, or iNaturalist, which we specialize and compress to a diverse set of "downstream" subtasks. TACO can reduce the number of non-zero parameters in existing models by up to 20x relative to the original models, leading to inference speedups of up to 3$\times$, while remaining accuracy-competitive with the uncompressed models on the specialized tasks.
Dense Vision Transformer Compression with Few Samples
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization
Institute for Infocomm Research, Agency for Science, Technology and Research(新加坡科技研究局资讯通信研究院);College of Computer Science, Sichuan University(四川大学计算机学院);Nanyang Technological University(南洋理工大学)
Leveraging Compression to Construct Transferable Bitrate Ladders
Laboratory for Image and Video Engineering (LIVE), The University of Texas at Austin(图像与视频工程实验室(LIVE),德克萨斯大学奥斯汀分校);Colorado’s Laboratory for Image and Video Engineering (LIVE), University of Colorado Boulder(科罗拉多州图像与视频工程实验室(LIVE),科罗拉多大学波德分校);Meta Platforms Inc.(Meta平台公司)
One-Shot Knowledge Transfer for Scalable Person Re-Identification
School of Computer Science and Engineering, Southeast University, Nanjing, China(计算机科学与工程学院,东南大学,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用关键实验室(东南大学),教育部,中国)
Progressive Network Grafting for Few-Shot Knowledge Distillation
Zhejiang University(浙江大学);Stevens Institute of Technology(史蒂文斯理工学院);Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies(阿里巴巴-浙江大学前沿技术联合研究院)