arXiv 2401.078532024-04-13cs.CV

VeCAF：具备训练目标感知的视觉-语言协同主动微调

VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness

Nanjing University（南京大学）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Tsinghua University（清华大学）；Panasonic（松下电器（Panasonic））；University of California Berkeley（加州大学伯克利分校）

Rongyu Zhang, Zefan Cai, Huanrui Yang, Zidong Liu, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Baobao Chang, Yuan Du, Li Du, Shanghang Zhang

备注：: 13 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统预训练视觉模型微调训练效率低的问题，提出视觉-语言协同主动微调方法VeCAF，结合微调目标与文本语义选数据增强，在分布内外分类任务上性能领先，ImageNet上效率与准确率均优于现有方法。

摘要：

对预训练视觉模型（PVM）进行微调是学习下游视觉任务的常用技术。然而，采用随机采样数据点的传统微调流程会导致训练效率下降。为解决这一缺陷，我们提出了一种新方法——视觉-语言协同主动微调（VeCAF）。随着通过网络规模爬取或受控生成获取图像标签与自然语言标注的方式日益普及，VeCAF利用这些信息为PVM微调执行参数化数据选择。VeCAF结合微调目标来筛选重要数据点，有效引导PVM更快收敛以达到性能目标，该过程借助文本嵌入空间固有的语义丰富性来增强图像特征。此外，文本域增强的灵活性使VeCAF无需外部数据即可处理分布外场景。大量实验表明，VeCAF在分布内和分布外图像分类任务中均优于基线方法，具备领先性能与高计算效率。在ImageNet数据集上，与全量微调相比，VeCAF达到目标性能所需的训练批次最多减少3.3倍；在相同批次数量下，其准确率比当前最优的主动微调方法提升2.7%。

英文摘要：

Finetuning a pretrained vision model (PVM) is a common technique for learning downstream vision tasks. However, the conventional finetuning process with randomly sampled data points results in diminished training efficiency. To address this drawback, we propose a novel approach, Vision-language Collaborative Active Finetuning (VeCAF). With the emerging availability of labels and natural language annotations of images through web-scale crawling or controlled generation, VeCAF makes use of these information to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence to meet the performance goal. This process is assisted by the inherent semantic richness of the text embedding space which we use to augment image features. Furthermore, the flexibility of text-domain augmentation allows VeCAF to handle out-of-distribution scenarios without external data. Extensive experiments show the leading performance and high computational efficiency of VeCAF that is superior to baselines in both in-distribution and out-of-distribution image classification tasks. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.7% over the state-of-the-art active finetuning method with the same number of batches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。