Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering
Negation-Aware Test-Time Adaptation for Vision-Language Models
Department of AI Computing, Pengcheng Laboratory(人工智能计算系,鹏城实验室);School of Computer Science and Technology, Central South University(计算机科学与技术学院,中南大学);Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University(计算机科学与技术系,人工智能研究院,BNRist中心,THBI实验室,清华-博世联合人工智能中心,清华大学)
HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
ADAPT Centre Dublin City University, Ireland(爱尔兰都柏林城市大学ADAPT中心);University of East Anglia Norwich, UK(英国东英吉利大学);Queen’s University Belfast Belfast, UK(英国贝尔法斯特女王大学);University of Science Vietnam National University Ho Chi Minh City, Vietnam(越南胡志明市国家大学理科大学)
Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understanding
Department of Electrical and Computer Engineering, Inha University, Republic of Korea(电气与计算机工程系,印哈大学,大韩民国);College of Medicine, Inha University, Republic of Korea(医学学院,印哈大学,大韩民国)
University of Arkansas(阿肯色大学);University of Science, VNU-HCM(胡志明市国家大学科学大学);Aalto University(阿尔托大学);SpexAI GmbH(SpexAI有限公司);University of California, San Diego(加利福尼亚大学圣迭戈分校)
The Geometry of Representational Failures in Vision Language Models
Dipartimento di Fisica, Università di Torino(都灵大学物理系);Princeton Neuroscience Institute and AI Lab, Princeton University(普林斯顿大学神经科学研究所和AI实验室);Intesa Sanpaolo AI Research(Intesa Sanpaolo AI研究中心);Dipartimento di Scienze Matematiche, Politecnico di Torino(都灵理工学院数学科学系);Network Science Institute, Northeastern University London, UK(伦敦大学东北方大学网络科学研究所)
Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
University of Washington(华盛顿大学);University of California, Los Angeles(加州大学洛杉矶分校);Samaya AI(Samaya人工智能);Mistral AI(Mistral人工智能);Allen Institute for AI(人工智能研究所)
GeoDANO: Geometric VLM with Domain Agnostic Vision Encoder
Graduate School of Artificial Intelligence, POSTECH(POSTECH人工智能研究生院);Department of Computer Science and Engineering, POSTECH(POSTECH计算机科学与工程系);Australian National University(澳大利亚国立大学)
GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Robbyant;School of Computing, National University of Singapore(新加坡国立大学计算机学院);The Hong Kong University of Science and Technology(香港科技大学)
Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Amazon(亚马逊);Samsung Research(三星研究院);School of Computer Science and Engineering, Soongsil University(顺天大学计算机科学与工程学院);IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University(首尔国立大学IPAI、AIIS、ASRI、INMC和ISRC)
Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School(生物工程跨学科项目,首尔国立大学研究生院);Integrated Major in Innovative Medical Science, Seoul National University Graduate School(创新医学联合专业,首尔国立大学研究生院);Department of Radiology, Seoul National University Hospital(放射科,首尔国立大学医院)