在基础模型预训练数据中被低估？一个单次探测器
Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe
AI总结：
本文提出一种低成本方法，通过单张标记图像预测VLFM在目标领域的零样本准确率，用于评估和优化数据标注决策。
摘要：
大规模视觉-语言基础模型（VLFMs），如CLIP，现在成为计算机视觉研究和应用的基石。VLFMs通常被适应到各种领域特定的任务中。然而，VLFM在新型、专门化或被低估的领域中的表现仍不一致。评估VLFMs通常需要标记的测试集，而这些测试集在感兴趣的狭窄领域中往往不可用，特别是来自全球南方的领域。我们通过提出一种高度数据高效的方 法来预测VLFM在目标领域上的零样本准确率，仅需每个类别一个标记的图像。我们的方法利用大型语言模型生成合理的反事实描述。通过测量VLFM区分正确描述与这些困难负样本的能力，我们工程出捕捉VLFM在共享嵌入空间中判别能力的特征。一个在这些相似度分数上训练的线性回归器能够估计VLFM在各种视觉领域上的零样本测试准确率，相关系数为0.96。我们展示了该方法在五个多样化的数据集上的表现，包括标准基准数据集和来自非洲的被低估数据集。我们的工作提供了一种低成本、可靠的探测VLFM的工具，使研究人员和实践者能够在投入大量资源之前，做出关于数据标注努力的知情决策。模型训练代码、生成的标题和反事实描述已在此处发布：https://github.com/chris-vorster/PreLabellingProbe。
英文摘要：
Large-scale Vision-Language Foundation Models (VLFMs), such as CLIP, now underpin a wide range of computer vision research and applications. VLFMs are often adapted to various domain-specific tasks. However, VLFM performance on novel, specialised, or underrepresented domains remains inconsistent. Evaluating VLFMs typically requires labelled test sets, which are often unavailable for niche domains of interest, particularly those from the Global South. We address this gap by proposing a highly data-efficient method to predict a VLFM's zero-shot accuracy on a target domain using only a single labelled image per class. Our approach uses a Large Language Model to generate plausible counterfactual descriptions of a given image. By measuring the VLFM's ability to distinguish the correct description from these hard negatives, we engineer features that capture the VLFM's discriminative power in its shared embedding space. A linear regressor trained on these similarity scores estimates the VLFM's zero-shot test accuracy across various visual domains with a Pearson-r correlation of 0.96. We demonstrate our method's performance across five diverse datasets, including standard benchmark datasets and underrepresented datasets from Africa. Our work provides a low-cost, reliable tool for probing VLFMs, enabling researchers and practitioners to make informed decisions about data annotation efforts before committing significant resources. The model training code, generated captions and counterfactuals are released here: https://github.com/chris-vorster/PreLabellingProbe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
