arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.12331cs.CV
收藏

SpaceVLM:视觉-语言模型中否定的子空间建模

SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models

MIT(麻省理工学院)

Sepehr Kazemi Ranjbar, Kumail Alhamoud, Marzyeh Ghassemi

AI总结:

针对视觉-语言模型否定理解能力弱、微调会损害零样本性能的问题,提出免训练框架SpaceVLM,将否定建模为嵌入子空间,在多任务上平均提升30%否定理解性能且保留零样本能力。

摘要:

视觉-语言模型(VLMs)在处理否定表述时存在困难。给定“检索(或生成)没有行人的街景”这类提示时,它们往往无法遵守“不”的约束。现有方法通过在大型否定数据集上微调来解决这一缺陷,但这类重训练通常会损害模型在肯定提示上的零样本性能。我们发现,CLIP这类VLM的嵌入空间可划分为语义一致的子空间。基于这一特性,我们提出一种免训练框架,将否定建模为联合嵌入空间中的一个子空间而非单个点(图1)。为匹配“A但非N”这类描述对应的图像,我们在A和N的嵌入周围构建两个球冠,通过靠近A、远离N的区域中心方向为图像打分。在检索、多项选择、文生图任务中,我们的方法相比此前方法平均将否定理解能力提升约30%。它缩小了肯定提示与否定提示间的性能差距,同时保留了微调模型无法维持的零样本性能。代码将在论文发表后公开。

英文摘要:

Vision-Language Models (VLMs) struggle with negation. Given a prompt like "retrieve (or generate) a street scene without pedestrians," they often fail to respect the "not." Existing methods address this limitation by fine-tuning on large negation datasets, but such retraining often compromises the model's zero-shot performance on affirmative prompts. We show that the embedding space of VLMs, such as CLIP, can be divided into semantically consistent subspaces. Based on this property, we propose a training-free framework that models negation as a subspace in the joint embedding space rather than a single point (Figure 1). To find the matching image for a caption such as "A but not N," we construct two spherical caps around the embeddings of A and N, and we score images by the central direction of the region that is close to A and far from N. Across retrieval, MCQ, and text-to-image tasks, our method improves negation understanding by about 30% on average over prior methods. It closes the gap between affirmative and negated prompts while preserving the zero-shot performance that fine-tuned models fail to maintain. Code will be released upon publication.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑