当视觉语言模型遇见图像分类：通过缺失标签识别进行测试集翻新
When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification
AI总结：
针对图像分类测试集中被忽视的缺失标签问题，提出REVEAL框架，整合视觉语言模型与标签整理方法，检测并修正噪声和缺失标签，显著提升6个基准测试集质量。
摘要：
图像分类基准数据集（如CIFAR、MNIST和ImageNet）是模型评估的关键工具。然而，尽管经过了清理工作，这些数据集仍然存在普遍的噪声标签，并且由于图像样本中常出现多类别共存的图像模式，往往包含缺失标签。这导致了误导性的模型比较和不公平的评估。现有的标签清理方法主要关注噪声标签，但缺失标签问题在很大程度上仍被忽视。受这些挑战的驱动，我们提出了一个名为REVEAL的综合框架，该框架整合了最先进的预训练视觉语言模型（如LLaVA、BLIP、Janus、Qwen）与先进的机器/人工标签整理方法（如Docta、Cleanlab、MTurk），以系统地解决广泛使用的图像分类测试集中的噪声标签和缺失标签检测问题。REVEAL检测潜在的噪声标签和遗漏，聚合来自各种方法的预测，并通过基于置信度的预测和共识过滤来细化标签准确性。此外，我们提供了对最先进的视觉语言模型和预训练图像分类器的深入分析，通过揭示10项观察结果，突出了它们在数据集翻新背景下的优势和局限性。我们的方法有效地从公共数据集中揭示缺失标签，并提供带有似然度的软标签结果。通过人工验证，REVEAL显著提高了6个基准测试集的质量，高度符合人类判断，使得图像分类中的比较更加准确和有意义。
英文摘要：
Image classification benchmark datasets such as CIFAR, MNIST, and ImageNet serve as critical tools for model evaluation. However, despite the cleaning efforts, these datasets still suffer from pervasive noisy labels and often contain missing labels due to the co-existing image pattern where multiple classes appear in an image sample. This results in misleading model comparisons and unfair evaluations. Existing label cleaning methods focus primarily on noisy labels, but the issue of missing labels remains largely overlooked. Motivated by these challenges, we present a comprehensive framework named REVEAL, integrating state-of-the-art pre-trained vision-language models (e.g., LLaVA, BLIP, Janus, Qwen) with advanced machine/human label curation methods (e.g., Docta, Cleanlab, MTurk), to systematically address both noisy labels and missing label detection in widely-used image classification test sets. REVEAL detects potential noisy labels and omissions, aggregates predictions from various methods, and refines label accuracy through confidence-informed predictions and consensus-based filtering. Additionally, we provide a thorough analysis of state-of-the-art vision-language models and pre-trained image classifiers, highlighting their strengths and limitations within the context of dataset renovation by revealing 10 observations. Our method effectively reveals missing labels from public datasets and provides soft-labeled results with likelihoods. Through human verifications, REVEAL significantly improves the quality of 6 benchmark test sets, highly aligning to human judgments and enabling more accurate and meaningful comparisons in image classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
