arXiv 2304.014572023-06-21cs.AIcs.CVcs.LG

探索视觉语言模型在不平衡学习中的应用

Exploring Vision-Language Models for Imbalanced Learning

Peking University（北京大学）；North Carolina State University（北卡罗来纳州立大学）；Carnegie Mellon University（卡内基梅隆大学）

Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, Shikun Zhang

备注：: IJCV minor revision; 16 pages; code: https://github.com/Imbalance-VLM/Imbalance-VLM

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对视觉语言模型在不平衡数据集上性能不佳的问题，提出添加轻量级解码器并融合提示调优、微调及不平衡算法，显著提升了在ImageNet-LT等数据集上的分类准确率。

摘要：

使用对比语言-图像预训练的视觉语言模型（VLMs）在零样本分类任务中表现出了良好的性能。然而，它们在类别分布不平衡的数据集上性能相对较差，其中训练数据集中类别的分布存在偏斜，导致对少数类别的预测性能不佳。例如，CLIP在iNaturalist18数据集上仅达到了5%的准确率。我们提出为VLMs添加一个轻量级解码器，以避免因类别数量过多而导致的OOM（内存不足）问题，并捕获尾部类别的细微特征。随后，我们探索了通过提示调优、微调以及结合不平衡算法（如Focal Loss、Balanced SoftMax和Distribution Alignment）来改进VLMs。实验表明，当VLMs与解码器和不平衡方法结合使用时，其性能可以进一步提升。具体来说，我们改进的VLMs在ImageNet-LT、iNaturalist18和Places-LT上分别平均比零样本分类显著高出6.58%、69.82%和6.17%的准确率。我们进一步分析了预训练数据规模、骨干网络和训练成本的影响。我们的研究强调了在面对由海量数据预训练的VLMs时，不平衡学习算法的重要性。我们在https://github.com/Imbalance-VLM/Imbalance-VLM发布了我们的代码。

英文摘要：

Vision-Language models (VLMs) that use contrastive language-image pre-training have shown promising zero-shot classification performance. However, their performance on imbalanced dataset is relatively poor, where the distribution of classes in the training dataset is skewed, leading to poor performance in predicting minority classes. For instance, CLIP achieved only 5% accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder to VLMs to avoid OOM (out of memory) problem caused by large number of classes and capture nuanced features for tail classes. Then, we explore improvements of VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments demonstrate that the performance of VLMs can be further boosted when used with decoder and imbalanced methods. Specifically, our improved VLMs significantly outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%, and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We further analyze the influence of pre-training data size, backbones, and training cost. Our study highlights the significance of imbalanced learning algorithms in face of VLMs pre-trained by huge data. We release our code at https://github.com/Imbalance-VLM/Imbalance-VLM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。