图像识别基准测试中单标签假设的影响
The Impact of the Single-Label Assumption in Image Recognition Benchmarking
AI总结：
本文研究单标签假设对图像识别基准的影响，提出可变top-$k$评估和ASMA指标，并构建PatchML数据集，发现现代DNN比标准指标显示的具有更强的多标签识别能力。
摘要：
深度神经网络（DNN）通常在每张图像具有单一正确标签的假设下进行评估。然而，诸如ImageNet等基准测试中的许多图像包含多个有效标签，这在评估协议与视觉数据的实际复杂性之间造成了不匹配。这种不匹配会因DNN预测了正确但未标注的标签而对其造成惩罚，这或许能部分解释已报告的准确率下降，例如在ImageNetV2（ImageNet的复制测试集）上被广泛引用的11%至14%的top-1准确率下降。这引发了一个问题：此类下降是反映了真实的泛化失败，还是限制性评估指标的人为假象？我们严格评估了多标签特征对报告准确率差距的影响。为了评估单标签训练模型的多标签预测能力（MLPC），我们引入了可变top-$k$评估，其中$k$与每张图像的有效标签数量相匹配。应用于315个ImageNet训练模型后，我们的分析表明，传统的top-1准确率不成比例地惩罚了有效但次要的预测。我们还提出了聚合子组模型准确率（ASMA），以更好地捕捉跨模型子组的多标签性能。结果显示MLPC存在广泛差异，部分模型能持续将多个正确标签排在更高位置。在此评估下，ImageNet与ImageNetV2之间感知到的差距显著缩小。为了进一步将多标签识别性能与上下文线索隔离，我们引入了PatchML，这是一个包含系统组合对象图块的合成数据集。PatchML表明，许多采用单标签监督训练的模型仍能识别多个对象。总而言之，这些发现突显了单标签评估的局限性，并揭示现代DNN比标准指标所暗示的具备更强的多标签能力。
英文摘要：
Deep neural networks (DNNs) are typically evaluated under the assumption that each image has a single correct label. However, many images in benchmarks like ImageNet contain multiple valid labels, creating a mismatch between evaluation protocols and the actual complexity of visual data. This mismatch can penalize DNNs for predicting correct but unannotated labels, which may partly explain reported accuracy drops, such as the widely cited 11 to 14 percent top-1 accuracy decline on ImageNetV2, a replication test set for ImageNet. This raises the question: do such drops reflect genuine generalization failures or artifacts of restrictive evaluation metrics? We rigorously assess the impact of multi-label characteristics on reported accuracy gaps. To evaluate the multi-label prediction capability (MLPC) of single-label-trained models, we introduce a variable top-$k$ evaluation, where $k$ matches the number of valid labels per image. Applied to 315 ImageNet-trained models, our analyses demonstrate that conventional top-1 accuracy disproportionately penalizes valid but secondary predictions. We also propose Aggregate Subgroup Model Accuracy (ASMA) to better capture multi-label performance across model subgroups. Our results reveal wide variability in MLPC, with some models consistently ranking multiple correct labels higher. Under this evaluation, the perceived gap between ImageNet and ImageNetV2 narrows substantially. To further isolate multi-label recognition performance from contextual cues, we introduce PatchML, a synthetic dataset containing systematically combined object patches. PatchML demonstrates that many models trained with single-label supervision nonetheless recognize multiple objects. Altogether, these findings highlight limitations in single-label evaluation and reveal that modern DNNs have stronger multi-label capabilities than standard metrics suggest.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
