arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.08313cs.AIcs.CLcs.CV
收藏

语言是否有助于视觉模型的泛化?

Does language help generalization in vision models?

Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse(图卢兹大学人工与自然智能图卢兹研究所);CerCO, CNRS UMR5549(法国国家科学研究中心 CerCO 联合研究单位)

Benjamin Devillers, Bhavin Choksi, Romain Bielawski, Rufin VanRullen

摘要:

在多模态数据集上训练的视觉模型可以受益于大型图像标题数据集的广泛可用性。最近的模型(CLIP)被发现在zero-shot和迁移学习设置中推广得很好。这可能意味着语言或“语义基础”赋予视觉特征空间额外的泛化能力。在这里,我们系统地评估了各种多模态架构和纯视觉模型的无监督聚类,少样本学习,迁移学习和对抗鲁棒性。在每种设置中,与标准的监督视觉训练相比,多模式训练没有产生额外的泛化能力。我们的结论是,工作仍然需要语义基础,以帮助改善视觉模型。

英文摘要:

Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or "semantic grounding" confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑