CLIP的泛化性能主要源于高训练-测试相似度吗？
Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?
AI总结：
本研究通过剪枝LAION数据集复制ImageNet的训练-测试相似度，发现CLIP的高OOD性能并非仅源于高相似度，并找到可匹配原始性能的1亿子集。
摘要：
像CLIP这样的基础模型在数亿个样本上进行训练，并能轻松泛化到新任务和新输入。开箱即用，CLIP在广泛的分布外（OOD）基准测试上展现出卓越的零样本和少样本能力，先前的工作主要将其归因于当今大规模且全面的训练数据集（如LAION）。然而，对于CLIP而言，诸如分布外泛化之类的术语有多大意义是值得怀疑的，因为像LAION这样的网络规模数据集似乎很可能只是包含了许多与最初为ImageNet设计的常见OOD基准相似的样本。为了验证这一假设，我们在经过剪枝的LAION子集上重新训练CLIP，这些子集复制了ImageNet相对于常见OOD基准的训练-测试相似度。虽然我们观察到在某些基准上的性能下降，但令人惊讶的是，CLIP的整体性能仍然很高。这表明高训练-测试相似度不足以解释CLIP的OOD性能，训练数据的其他属性必定驱动CLIP学习更可泛化的表示。此外，通过剪除与OOD基准不相似的数据点，我们发现了一个LAION的1亿子集（其原始大小的四分之一），在该子集上训练CLIP可以匹配其原始的OOD性能。
英文摘要：
Foundation models like CLIP are trained on hundreds of millions of samples and effortlessly generalize to new tasks and inputs. Out of the box, CLIP shows stellar zero-shot and few-shot capabilities on a wide range of out-of-distribution (OOD) benchmarks, which prior works attribute mainly to today's large and comprehensive training dataset (like LAION). However, it is questionable how meaningful terms like out-of-distribution generalization are for CLIP as it seems likely that web-scale datasets like LAION simply contain many samples that are similar to common OOD benchmarks originally designed for ImageNet. To test this hypothesis, we retrain CLIP on pruned LAION splits that replicate ImageNet's train-test similarity with respect to common OOD benchmarks. While we observe a performance drop on some benchmarks, surprisingly, CLIP's overall performance remains high. This shows that high train-test similarity is insufficient to explain CLIP's OOD performance, and other properties of the training data must drive CLIP to learn more generalizable representations. Additionally, by pruning data points that are dissimilar to the OOD benchmarks, we uncover a 100M split of LAION ($\frac{1}{4}$th of its original size) on which CLIP can be trained to match its original OOD performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
