arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.14409cs.CV
收藏

视觉模型可以通过Few-Shot任务感知压缩进行高效专业化

Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compression

Skoltech(斯科尔科沃科学技术研究院);Yandex(扬德克斯);IST Austria(奥地利科学技术研究所);Sharif UT(谢里夫理工大学);Massachusetts Institute of Technology(麻省理工学院);Neural Magic(神经魔法公司)

Denis Kuznedelev, Soroush Tabesh, Kimia Noorbakhsh, Elias Frantar, Sara Beery, Eldar Kurtic, Dan Alistarh

摘要:

最近的视觉架构和自监督训练方法使视觉模型非常准确和通用,但需要大量的参数和计算成本。在实际设置中,例如相机陷阱,用户的资源有限,并且可能会对来自一小组特定类别的数据(通常是有限的)进行预训练模型微调。这些用户可能希望使用现代的、高精度的模型,但通常在计算上受到限制。为了解决这个问题,我们问:我们能否快速地将大型通才模型压缩成准确高效的专家?为此,我们提出了一种简单而通用的技术,称为Few-Shot任务感知压缩(TACO)。给定一个大型视觉模型,该模型在广泛的任务(例如ImageNet-22 K上的分类)上进行了预训练,TACO生成了一个较小的模型,该模型在特定的任务(例如车辆类型或动物物种的分类)上是准确的。至关重要的是,TACO以Few-Shot方式工作,即仅使用少数任务特定的样本,并且该过程具有低计算开销。我们在高精度的ResNet、ViT/DeiT和ConvNeXt模型上验证TACO,这些模型最初是在ImageNet、LAION或iNaturalist上训练的,我们将其专门化并压缩到一组不同的“下游”子任务中。相对于原始模型,TACO可以将现有模型中的非零参数数量减少多达20倍,从而导致推理加速高达3 $\times$,同时在专门任务上与未压缩模型保持准确性竞争。

英文摘要:

Recent vision architectures and self-supervised training methods enable vision models that are extremely accurate and general, but come with massive parameter and computational costs. In practical settings, such as camera traps, users have limited resources, and may fine-tune a pretrained model on (often limited) data from a small set of specific categories of interest. These users may wish to make use of modern, highly-accurate models, but are often computationally constrained. To address this, we ask: can we quickly compress large generalist models into accurate and efficient specialists? For this, we propose a simple and versatile technique called Few-Shot Task-Aware Compression (TACO). Given a large vision model that is pretrained to be accurate on a broad task, such as classification over ImageNet-22K, TACO produces a smaller model that is accurate on specialized tasks, such as classification across vehicle types or animal species. Crucially, TACO works in few-shot fashion, i.e. only a few task-specific samples are used, and the procedure has low computational overheads. We validate TACO on highly-accurate ResNet, ViT/DeiT, and ConvNeXt models, originally trained on ImageNet, LAION, or iNaturalist, which we specialize and compress to a diverse set of "downstream" subtasks. TACO can reduce the number of non-zero parameters in existing models by up to 20x relative to the original models, leading to inference speedups of up to 3$\times$, while remaining accuracy-competitive with the uncompressed models on the specialized tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑