定位vs.语义：单模态与多模态模型中的视觉表征
Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models
AI总结：
本研究通过多任务探测对比视觉-语言模型与纯视觉模型的视觉表征，发现前者擅长标签预测类任务、后者在需局部信息的密集预测任务上更优，为理解语言在视觉学习中的作用及预训练模型选型提供实证参考。
摘要：
尽管视觉-语言预训练已取得令人瞩目的进展，但这种联合学习范式是否有助于理解每种单独的模态仍不明确。在本研究中，我们通过探测广泛的任务，对现有视觉-语言模型和纯视觉模型中的视觉表征开展对比分析，旨在以细致的方式评估所学表征的质量。有趣的是，我们的实证观察表明，视觉-语言模型在物体、属性预测这类标签预测任务上表现更优，而纯视觉模型在需要更多局部信息的密集预测任务上能力更强。我们希望本研究能阐明语言在视觉学习中的作用，并为各类预训练模型提供实证指导。代码将发布于https://github.com/Lizw14/visual_probing
英文摘要：
Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing a broad range of tasks, aiming to assess the quality of the learned representations in a nuanced manner. Interestingly, our empirical observations suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models. Code will be released at https://github.com/Lizw14/visual_probing
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
