探索视觉-语言组合性与识别的谱系
Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition
AI总结：
本文通过 12 个组合性基准及 23 个识别相关基准评估 274 个 CLIP 检查点，揭示 VLM 组合理解与识别准确率之间的模式和权衡，并开放评估框架。
摘要：
诸如 CLIP 之类的视觉与语言模型（VLM）已展现出卓越的零样本识别能力，但在视觉-语言组合性方面仍面临挑战，尤其是在语言理解和细粒度图文对齐方面。本文探讨了组合性与识别之间错综复杂的关系——二者是 VLM 能力的两个关键方面。我们对现有 VLM 进行了全面评估，既涵盖面向识别的预训练方法，也包括旨在提升组合性的微调方法。评估使用了 12 个组合性基准，以及用于识别的 21 个零样本分类基准和两个检索基准。在对 274 个 CLIP 模型检查点的分析中，我们揭示了组合理解与识别准确率之间出现的模式和权衡。最终，这表明需要作出战略性努力，以开发同时提升两种能力的模型，并精心构建组合性基准。我们在 https://github.com/ytaek-oh/vl_compo 开放评估框架。
英文摘要：
Vision and language models (VLMs) such as CLIP have showcased remarkable zero-shot recognition abilities yet face challenges in visio-linguistic compositionality, particularly in linguistic comprehension and fine-grained image-text alignment. This paper explores the intricate relationship between compositionality and recognition -- two pivotal aspects of VLM capability. We conduct a comprehensive evaluation of existing VLMs, covering both pre-training approaches aimed at recognition and the fine-tuning methods designed to improve compositionality. Our evaluation employs 12 benchmarks for compositionality, along with 21 zero-shot classification and two retrieval benchmarks for recognition. In our analysis from 274 CLIP model checkpoints, we reveal patterns and trade-offs that emerge between compositional understanding and recognition accuracy. Ultimately, this necessitates strategic efforts towards developing models that improve both capabilities, as well as the meticulous formulation of benchmarks for compositionality. We open our evaluation framework at https://github.com/ytaek-oh/vl_compo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
