Vision-Language Models (VLMs) struggle with negation. Given a prompt like "retrieve (or generate) a street scene without pedestrians," they often fail to respect the "not." Existing methods address this limitation by fine-tuning on large negation datasets, but such retraining often compromises the model's zero-shot performance on affirmative prompts. We show that the embedding space of VLMs, such as CLIP, can be divided into semantically consistent subspaces. Based on this property, we propose a training-free framework that models negation as a subspace in the joint embedding space rather than a single point (Figure 1). To find the matching image for a caption such as "A but not N," we construct two spherical caps around the embeddings of A and N, and we score images by the central direction of the region that is close to A and far from N. Across retrieval, MCQ, and text-to-image tasks, our method improves negation understanding by about 30% on average over prior methods. It closes the gap between affirmative and negated prompts while preserving the zero-shot performance that fine-tuned models fail to maintain. Code will be released upon publication.
Negation-Aware Test-Time Adaptation for Vision-Language Models
Department of AI Computing, Pengcheng Laboratory(人工智能计算系,鹏城实验室);School of Computer Science and Technology, Central South University(计算机科学与技术学院,中南大学);Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University(计算机科学与技术系,人工智能研究院,BNRist中心,THBI实验室,清华-博世联合人工智能中心,清华大学)
HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
ADAPT Centre Dublin City University, Ireland(爱尔兰都柏林城市大学ADAPT中心);University of East Anglia Norwich, UK(英国东英吉利大学);Queen’s University Belfast Belfast, UK(英国贝尔法斯特女王大学);University of Science Vietnam National University Ho Chi Minh City, Vietnam(越南胡志明市国家大学理科大学)
Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understanding
Department of Electrical and Computer Engineering, Inha University, Republic of Korea(电气与计算机工程系,印哈大学,大韩民国);College of Medicine, Inha University, Republic of Korea(医学学院,印哈大学,大韩民国)
University of Arkansas(阿肯色大学);University of Science, VNU-HCM(胡志明市国家大学科学大学);Aalto University(阿尔托大学);SpexAI GmbH(SpexAI有限公司);University of California, San Diego(加利福尼亚大学圣迭戈分校)
Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
University of Washington(华盛顿大学);University of California, Los Angeles(加州大学洛杉矶分校);Samaya AI(Samaya人工智能);Mistral AI(Mistral人工智能);Allen Institute for AI(人工智能研究所)
EchoVLM: Measurement-Grounded Multimodal Learning for Echocardiography
Georgia Institute of Technology(佐治亚理工学院);Siemens Healthineers(西门子医疗);Siemens Healthcare Limited(西门子医疗有限公司);Emory University(埃默里大学);University of Southern California(南加州大学)
Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School(生物工程跨学科项目,首尔国立大学研究生院);Integrated Major in Innovative Medical Science, Seoul National University Graduate School(创新医学联合专业,首尔国立大学研究生院);Department of Radiology, Seoul National University Hospital(放射科,首尔国立大学医院)