arXiv 2507.190642025-08-05cs.CV

面向视觉语言模型的否定感知测试时自适应

Negation-Aware Test-Time Adaptation for Vision-Language Models

Department of AI Computing, Pengcheng Laboratory（人工智能计算系，鹏城实验室）；School of Computer Science and Technology, Central South University（计算机科学与技术学院，中南大学）；Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University（计算机科学与技术系，人工智能研究院，BNRist中心，THBI实验室，清华-博世联合人工智能中心，清华大学）

Haochen Han, Alex Jinpeng Wang, Fangming Liu, Jun Zhu

备注：: This paper will be submitted to the IEEE for possible publication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言模型在否定理解上的不足，提出否定感知测试时自适应（NEAT）方法，通过调整分布参数减少分布偏移，以极低参数开销实现与最先进方法相当的性能。

摘要：

本文研究视觉语言模型（VLMs）中一个实际但较少被触及的问题，即否定理解。具体而言，许多现实应用要求模型明确识别什么是错误的或不存在的，例如，放射科医生可能搜索排除特定条件的图像。尽管VLM通过大规模训练具有令人印象深刻的迁移能力，但它们存在一个关键局限性，即无法处理否定。为应对这一挑战，现有方法将其根源归因于否定训练数据的稀缺，并提出在包含明确否定的大量数据上微调VLM。无疑，这种以数据为中心的解决方案需要大量的数据和计算资源，限制了其可持续的广泛采用。为了以低碳方式解决否定问题，我们通过经验观察发现，关键障碍在于肯定分布与否定分布之间的双重概念偏移。因此，我们提出一种否定感知测试时自适应（NEAT）方法，在推理过程中高效调整与分布相关的参数。简而言之，NEAT可以在一致语义中减少分布偏移，同时消除不相关语义中的虚假分布一致性。在多种否定理解任务上的大量实验验证了所提方法的有效性。值得注意的是，NEAT仅使用不到0.01%的可训练参数，即可达到与最先进的训练后方法相当或更优的性能。我们的代码可在 https://github.com/hhc1997/NEAT 获取。

英文摘要：

In this paper, we study a practical but less-touched problem in Vision-Language Models (VLMs), \ie, negation understanding. Specifically, many real-world applications require models to explicitly identify what is false or non-existent, \eg, radiologists may search for images that exclude specific conditions. Despite the impressive transferability of VLMs through large-scale training, they suffer from a critical limitation that fails to handle negation. To address this challenge, existing methods attribute its root cause to the scarcity of negation training data and propose to fine-tune VLMs on massive data containing explicit negation. Undoubtedly, such data-centric solutions demand substantial data and computational resources, limiting their sustainable widespread adoption. To tackle negation in a low-carbon manner, we empirically observe that the key obstacle lies in the dual-concept shifts between the affirmation and negation distributions. Therefore, we propose a Negation-Aware Test-Time Adaptation (NEAT) method to efficiently adjust distribution-related parameters during inference. In brief, NEAT can reduce distribution shift in consistent semantics while eliminating false distributional consistency in unrelated semantics. Extensive experiments on the various negation understanding tasks verify the effectiveness of the proposed method. Remarkably, with less than 0.01\% of trainable parameters, NEAT achieves comparable or superior performance to state-of-the-art post-training approaches. Our code is available at https://github.com/hhc1997/NEAT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。