数据还是语言监督：是什么让CLIP优于DINO？
Data or Language Supervision: What Makes CLIP Better than DINO?
AI总结：
本研究通过受控实验对比CLIP与DINO，发现CLIP的优势源于语言监督而非数据规模，其在文本密集型任务更优，而DINO在视觉中心任务略胜，为视觉编码器设计提供见解。
摘要：
CLIP作为视觉语言模型（VLM）的视觉编码器，优于DINO等自监督模型，但尚不清楚这一优势源于CLIP的语言监督还是其更大的训练数据。为厘清这些因素，我们在受控设置下预训练CLIP和DINO——使用相同的架构、数据集和训练配置——并达到相似的ImageNet准确率。嵌入分析显示，CLIP捕获高层语义（如物体类别、文本），而DINO对颜色和风格等低层特征更敏感。当集成到VLM中并在20个VQA基准上评估时，CLIP在文本密集型任务上表现出色，而DINO在视觉中心任务上略胜一筹。语言监督的变体（如sigmoid损失、预训练语言编码器）带来的收益有限。我们的发现为视觉编码器设计及其对VLM性能的影响提供了科学见解。
英文摘要：
CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings -- using the same architecture, dataset, and training configuration -- achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
