arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20554cs.CV
收藏

当否定是在视觉-语言模型中一个几何问题

When Negation Is a Geometry Problem in Vision-Language Models

ETRO Department, Vrije Universiteit Brussel(布鲁塞尔自由大学ETRO系);imec;Independent Researcher(独立研究员)

Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov, Nikos Deligiannis

AI总结:

本文探讨了视觉-语言模型中否定理解的几何问题,提出基于多模态大语言模型的评估框架,并通过表示工程操控CLIP模型实现否定意识。

摘要:

联合视觉-语言嵌入模型如CLIP通常在理解文本查询中的否定时表现不佳,例如无法区分

英文摘要:

Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑