评估 VLM 与扩散模型中的组合泛化能力
Evaluating Compositional Generalisation in VLMs and Diffusion Models
AI总结：
本文在 ZSL 与 GZSL 设置下评估 Diffusion Classifier、CLIP 和 ViLT 的属性与关系绑定能力，发现生成式扩散分类器概念绑定较强，但所有模型在关系推理上仍存在明显短板。
摘要：
自然语言语义的一个基本方面是，新的意义可以由先前已知部分的组合构成。视觉语言模型（VLM）近年来取得了显著进展，然而有证据表明它们无法执行这种组合。例如，给定一张红色立方体和蓝色圆柱体的图像，像 CLIP 这样的 VLM 很可能错误地将图像标注为红色圆柱体或蓝色立方体，这表明它将图像表示为“词袋”，未能捕捉组合语义。扩散模型最近因其令人印象深刻的生成能力而受到广泛关注，而且基于扩散模型的零样本分类器已被证明在某些组合任务上具有与 CLIP 相竞争的表现。在本工作中，我们探究生成式 Diffusion Classifier 相比判别式模型是否具有更强的组合泛化能力。我们在零样本学习（ZSL）和广义零样本学习（GZSL）两种设置下，评估了 Diffusion Classifier、CLIP 和 ViLT 三个模型将对象与属性及关系进行绑定的能力。结果表明，Diffusion Classifier 和 ViLT 在概念绑定任务上表现良好，但所有模型在关系型 GZSL 任务上都明显困难，这凸显了 VLM 在关系推理方面面临的更广泛挑战。对 CLIP 嵌入的分析表明，该困难可能源于左、右等关系概念的表示过于相似。代码和数据集可在 https://github.com/otmive/diffusion_classifier_clip 获取。
英文摘要：
A fundamental aspect of the semantics of natural language is that novel meanings can be formed from the composition of previously known parts. Vision-language models (VLMs) have made significant progress in recent years, however, there is evidence that they are unable to perform this kind of composition. For example, given an image of a red cube and a blue cylinder, a VLM such as CLIP is likely to incorrectly label the image as a red cylinder or a blue cube, indicating it represents the image as a `bag-of-words' and fails to capture compositional semantics. Diffusion models have recently gained significant attention for their impressive generative abilities, and zero-shot classifiers based on diffusion models have been shown to perform competitively with CLIP in certain compositional tasks. In this work we explore whether the generative Diffusion Classifier has improved compositional generalisation abilities compared to discriminative models. We assess three models -- Diffusion Classifier, CLIP, and ViLT -- on their ability to bind objects with attributes and relations in both zero-shot learning (ZSL) and generalised zero-shot learning (GZSL) settings. Our results show that the Diffusion Classifier and ViLT perform well at concept binding tasks, but that all models struggle significantly with the relational GZSL task, underscoring the broader challenges VLMs face with relational reasoning. Analysis of CLIP embeddings suggests that the difficulty may stem from overly similar representations of relational concepts such as left and right. Code and dataset are available at: https://github.com/otmive/diffusion_classifier_clip
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
