arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02662cs.CVcs.CLcs.LG
收藏

图像-标题编码以提升零样本泛化能力

Image-Caption Encoding for Improving Zero-Shot Generalization

University of California San Diego(加利福尼亚大学圣迭戈分校);Boston University(波士顿大学);University of Maryland(马里兰大学);MIT Lincoln Laboratory(麻省理工学院林肯实验室)

Eric Yang Yu, Christopher Liao, Sathvik Ravi, Theodoros Tsiligkaridis, Brian Kulis

AI总结:

本文提出图像-标题编码(ICE)方法,利用生成标题引导局部搜索,在评估时强制一致性,以提升零样本图像分类的分布外泛化能力,平均提升Top-1准确率0.5%,最高达3%。

摘要:

近期视觉-语言模型的进展将对比学习方法与生成方法相结合,在下游推理任务(如零样本图像分类)中达到了最先进(SOTA)水平。然而,这些模型在图像分类中一个持续存在的问题是其分布外(OOD)泛化能力。我们首先证明,当一个OOD数据点被错误分类时,正确的类别通常可以在Top-K预测类别中找到。为了将模型预测引导至Top-K预测类别中的正确类别,我们提出了图像-标题编码(ICE)方法,这是一种直接的方法,仅在评估时强制图像条件和标题条件预测之间的一致性。直观上,我们利用生成标题的独特属性来指导我们在Top-K预测类别中对正确类别标签的局部搜索。我们展示了我们的方法可以轻松与其他SOTA方法结合,平均将Top-1 OOD准确率提高0.5%,在具有挑战性的数据集上最高提高3%。我们的代码:https://github.com/Chris210634/ice

英文摘要:

Recent advances in vision-language models have combined contrastive approaches with generative methods to achieve state-of-the-art (SOTA) on downstream inference tasks like zero-shot image classification. However, a persistent issue of these models for image classification is their out-of-distribution (OOD) generalization capabilities. We first show that when an OOD data point is misclassified, the correct class can be typically found in the Top-K predicted classes. In order to steer the model prediction toward the correct class within the top predicted classes, we propose the Image-Caption Encoding (ICE) method, a straightforward approach that directly enforces consistency between the image-conditioned and caption-conditioned predictions at evaluation time only. Intuitively, we take advantage of unique properties of the generated captions to guide our local search for the correct class label within the Top-K predicted classes. We show that our method can be easily combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5% on average and up to 3% on challenging datasets. Our code: https://github.com/Chris210634/ice

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑