是什么让 CLIP 对长尾预训练数据更鲁棒？一项旨在获得可迁移洞见的受控研究
What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights
AI总结：
本文通过受控实验揭示 CLIP 凭借动态分类代理任务隔离主导类偏差并隐式平衡学习信号，结合语言监督、数据规模与开放世界概念获得长尾鲁棒性，并将洞见迁移至监督与自监督模型。
摘要：
严重的数据不平衡自然存在于网络规模的视觉-语言数据集中。尽管如此，我们发现在此类数据上预训练的 CLIP 相比监督学习，对数据不平衡表现出显著的鲁棒性，并在学习可泛化表征方面展现出突出有效性。为探究这一发现背后的原因，我们开展受控实验以研究各种潜在因素，并揭示 CLIP 的代理任务构成了一个动态分类问题：每次训练中只有一个类别的子集出现。这隔离了来自主导类别的偏差，并隐式地平衡了学习信号。此外，CLIP 的鲁棒性和判别能力会随着更具描述性的语言监督、更大的数据规模以及更广泛的开放世界概念而提升，而这些是监督学习无法获得的。我们的研究不仅揭示了 CLIP 在数据不平衡之外仍具泛化性的机制，也为研究界提供了可迁移的洞见。这些发现在监督学习和自监督学习中均得到验证，使在不平衡数据上训练的模型能够在多种识别任务上达到 CLIP 级别的性能。代码和数据可在 https://github.com/CVMI-Lab/clip-beyond-tail 获取。
英文摘要：
Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
