arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.02263cs.CV
收藏

面向高效任务驱动模型重编程的基础模型方法

Towards Efficient Task-Driven Model Reprogramming with Foundation Models

South China University of Technology(华南理工大学);Shanghai Jiao Tong University(上海交通大学);Hong Kong Baptist University(香港浸会大学);PengCheng Laboratory(鹏城实验室)

Shoukai Xu, Jiangchao Yao, Ran Luo, Shuhai Zhang, Zihao Lian, Mingkui Tan, Bo Han, Yaowei Wang

AI总结:

提出任务驱动模型重编程(TDMR)框架,通过代理空间投影和渐进蒸馏,将基础模型知识高效迁移至不同架构的小型下游模型,实验验证其有效性。

摘要:

视觉基础模型展现出令人印象深刻的强大能力,这得益于其极大的模型容量和广泛的训练数据。然而,在实际应用中,由于计算资源有限或效率方面的考虑,下游场景可能只支持小型模型。此外,用于预训练基础模型的数据通常是不可见的,并且与下游任务的目标数据差异很大。这给基础模型的实际应用带来了一个关键挑战:必须将基础模型的知识迁移到架构差异较大的下游任务中,且仅能使用下游目标数据。现有的迁移学习或知识蒸馏方法要么依赖于相同的模型结构,要么需要对基础模型进行微调。因此,直接引入这些方法可能不可行或效率极低。为解决这一问题,我们提出了任务驱动模型重编程(TDMR)框架。具体而言,我们重编程基础模型,将知识投影到一个代理空间中,以缓解任务不匹配和领域不一致带来的不利影响。然后,我们通过从代理空间进行渐进式蒸馏来重编程目标模型,从而高效地学习来自重编程基础模型的知识。TDMR兼容不同的预训练模型类型(CNN、Transformer或其混合)以及有限的目标数据,并以经济高效的方式促进视觉基础模型在下游任务中的广泛应用。在多个不同下游分类任务和目标模型结构上的大量实验表明,我们的方法在CNN和Transformer基础模型上均有效。

英文摘要:

Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. Moreover, the data used for pretraining foundation models are usually invisible and very different from the target data of downstream tasks. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task that has a quite different architecture with only downstream target data. Existing transfer learning or knowledge distillation methods depend on either the same model structure or finetuning of the foundation model. Thus, naively introducing these methods can be either infeasible or very inefficient. To address this, we propose a Task-Driven Model Reprogramming (TDMR) framework. Specifically, we reprogram the foundation model to project the knowledge into a proxy space, which alleviates the adverse effect of task mismatch and domain inconsistency. Then, we reprogram the target model via progressive distillation from the proxy space to efficiently learn the knowledge from the reprogrammed foundation model. TDMR is compatible with different pre-trained model types (CNN, transformer or their mix) and limited target data, and promotes the wide applications of vision foundation models to downstream tasks in a cost-effective manner. Extensive experiments on different downstream classification tasks and target model structures demonstrate the effectiveness of our methods with both CNNs and transformer foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑