arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16145cs.LGcs.CV
收藏

通过多项选择题改进少样本图像分类

Improved Few-Shot Image Classification Through Multiple-Choice Questions

Generative AI Innovation Center(生成式人工智能创新中心)

Dipika Khullar, Emmett Goodman, Negin Sokhandan

AI总结:

提出一种无需训练的少样本图像分类方法,利用多项选择题提取VQA模型的潜在表示构建图像嵌入,并通过少量标注样本解码,在多个基准上优于纯视觉编码器和零样本VQA基线。

摘要:

通过简单的多项选择语言提示,VQA模型可以作为零样本图像分类器运行,生成分类标签。与典型的图像编码器相比,VQA模型具有优势:通过量身定制的语言提示,VQA生成的图像嵌入可以融入最相关的视觉信息。然而,对于大多数任务,零样本VQA性能不足,原因要么是不熟悉的类别名称,要么是预训练数据与测试数据分布不相似。我们提出了一种简单的方法,仅使用少量标注样本和一个多项选择题来提升VQA在图像分类上的性能。这种少样本方法无需训练,并保持了VQA模型的动态和灵活优势。我们的方法不依赖最终的语言输出,而是使用多项选择题来提取特定于提示的潜在表示,这些表示富含相关视觉信息。这些表示被组合起来创建最终的整体图像嵌入,并通过参考由少量标注样本构建的潜在类别原型进行解码。我们证明该方法优于纯视觉编码器和零样本VQA基线,在包括MiniImageNet、Caltech-UCSD Birds和CIFAR-100在内的常见少样本任务上取得了令人印象深刻的性能。最后,我们展示了我们的方法在具有大量多样视觉属性的场景中表现尤为出色,例如不同衣物的面料、款式、纹理和视角,在这些场景中其他少样本方法表现挣扎,因为我们可以仅根据感兴趣的语义特征来定制图像表示。

英文摘要:

Through a simple multiple choice language prompt a VQA model can operate as a zero-shot image classifier, producing a classification label. Compared to typical image encoders, VQA models offer an advantage: VQA-produced image embeddings can be infused with the most relevant visual information through tailored language prompts. Nevertheless, for most tasks, zero-shot VQA performance is lacking, either because of unfamiliar category names, or dissimilar pre-training data and test data distributions. We propose a simple method to boost VQA performance for image classification using only a handful of labeled examples and a multiple-choice question. This few-shot method is training-free and maintains the dynamic and flexible advantages of the VQA model. Rather than relying on the final language output, our approach uses multiple-choice questions to extract prompt-specific latent representations, which are enriched with relevant visual information. These representations are combined to create a final overall image embedding, which is decoded via reference to latent class prototypes constructed from the few labeled examples. We demonstrate this method outperforms both pure visual encoders and zero-shot VQA baselines to achieve impressive performance on common few-shot tasks including MiniImageNet, Caltech-UCSD Birds, and CIFAR-100. Finally, we show our approach does particularly well in settings with numerous diverse visual attributes such as the fabric, article-style, texture, and view of different articles of clothing, where other few-shot approaches struggle, as we can tailor our image representations only on the semantic features of interest.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑