arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02192cs.CV
收藏

利用视觉-语言预训练的无监督域适应

Unsupervised Domain Adaption Harnessing Vision-Language Pre-training

School of Electronic and Information Engineering, South China University of Technology(华南理工大学电子与信息学院);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, South China University of Technology(华南理工大学大数据与智能机器人教育部重点实验室)

Wenlve Zhou, Zhiheng Zhou

AI总结:

本文提出CMKD利用VLP教师模型指导目标域学习,并结合仅调整0.1%~0.5%参数的RST降低部署开销,在UDA基准上取得最优性能。

摘要:

本文解决无监督域适应(UDA)中的两个重要挑战,重点在于利用视觉-语言预训练(VLP)模型的能力。首先,UDA主要依赖于ImageNet预训练模型。然而,VLP模型在UDA中的潜力在很大程度上仍未得到探索。VLP模型的丰富表示对于增强UDA任务具有重要前景。为解决这一问题,我们提出了一种名为跨模态知识蒸馏(CMKD)的新方法,利用VLP模型作为教师模型来指导目标域中的学习过程,从而实现最先进的性能。其次,当前UDA范式涉及为每个任务训练独立模型,随着迁移任务数量增加,会导致显著的存储开销和不切实际的模型部署。为克服这一挑战,我们引入残差稀疏训练(RST),利用VLP大规模预训练所带来的优势;该技术只需对VLP模型参数进行极少调整(约0.1%~0.5%),即可达到与微调相当的性能。通过结合CMKD和RST,我们提出了一种综合解决方案,可有效利用VLP模型完成UDA任务,同时降低模型部署的存储开销。此外,CMKD可作为基线,与FixMatch等其他方法结合使用,提升UDA性能。我们提出的方法在标准基准测试上优于现有技术。代码将发布于:https://github.com/Wenlve-Zhou/VLP-UDA。

英文摘要:

This paper addresses two vital challenges in Unsupervised Domain Adaptation (UDA) with a focus on harnessing the power of Vision-Language Pre-training (VLP) models. Firstly, UDA has primarily relied on ImageNet pre-trained models. However, the potential of VLP models in UDA remains largely unexplored. The rich representation of VLP models holds significant promise for enhancing UDA tasks. To address this, we propose a novel method called Cross-Modal Knowledge Distillation (CMKD), leveraging VLP models as teacher models to guide the learning process in the target domain, resulting in state-of-the-art performance. Secondly, current UDA paradigms involve training separate models for each task, leading to significant storage overhead and impractical model deployment as the number of transfer tasks grows. To overcome this challenge, we introduce Residual Sparse Training (RST) exploiting the benefits conferred by VLP's extensive pre-training, a technique that requires minimal adjustment (approximately 0.1\%$\sim$0.5\%) of VLP model parameters to achieve performance comparable to fine-tuning. Combining CMKD and RST, we present a comprehensive solution that effectively leverages VLP models for UDA tasks while reducing storage overhead for model deployment. Furthermore, CMKD can serve as a baseline in conjunction with other methods like FixMatch, enhancing the performance of UDA. Our proposed method outperforms existing techniques on standard benchmarks. Our code will be available at: https://github.com/Wenlve-Zhou/VLP-UDA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑