arXiv 2505.155762025-08-28cs.CVcs.LG

面向视觉-语言模型组合推理的视觉扰动与自适应难负例对比学习

Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models

School of Artificial Intelligence and Software Engineering, Nanyang Normal University, Henan, China（人工智能与软件工程学院，南阳师范学院，河南）；Institute for Artificial Intelligence, Peking University, Beijing, China（人工智能研究院，北京大学，北京）；Collaborative Innovation Center of Intelligent Explosion-proof Equipment, Henan, China（智能防爆设备协同创新中心，河南）

Xin Huang, Ruibin Li, Tong Jia, Wei Zheng, Ya Wang

备注：: Accepted at the International Joint Conference on Artificial Intelligence (IJCAI 2025)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对VLM组合推理中图像负例缺失、负例难度不分及正例对齐不足的问题，提出AHNPL，通过视觉扰动难负例、多模态难负例损失和动态间隔损失提升复杂组合推理性能。

摘要：

视觉-语言模型（VLM）对于多模态任务至关重要，尤其是组合推理（CR）任务，这类任务要求区分视觉嵌入与文本嵌入之间的细粒度语义差异。然而，现有方法主要通过生成基于文本的难负例样本来微调模型，忽视了基于图像的负例样本的重要性，导致视觉编码器训练不足，并最终影响模型的整体性能。此外，负例样本通常被统一对待，未考虑其难度层级，而且正例样本的对齐也不充分，这给困难样本对的对齐带来挑战。为解决这些问题，我们提出自适应难负例扰动学习（AHNPL）。AHNPL将基于文本的难负例转换到视觉域，以生成受到语义扰动的基于图像的负例来训练模型，从而增强其整体性能。AHNPL还引入一种对比学习方法，使用多模态难负例损失来提升模型对各模态内难负例的辨别能力，并采用动态间隔损失，根据样本难度调整对比间隔，以增强对具有挑战性的样本对的区分。在三个公开数据集上的实验表明，我们的方法有效提升了VLM在复杂CR任务上的性能。源代码可在 https://github.com/nynu-BDAI/AHNPL 获取。

英文摘要：

Vision-Language Models (VLMs) are essential for multimodal tasks, especially compositional reasoning (CR) tasks, which require distinguishing fine-grained semantic differences between visual and textual embeddings. However, existing methods primarily fine-tune the model by generating text-based hard negative samples, neglecting the importance of image-based negative samples, which results in insufficient training of the visual encoder and ultimately impacts the overall performance of the model. Moreover, negative samples are typically treated uniformly, without considering their difficulty levels, and the alignment of positive samples is insufficient, which leads to challenges in aligning difficult sample pairs. To address these issues, we propose Adaptive Hard Negative Perturbation Learning (AHNPL). AHNPL translates text-based hard negatives into the visual domain to generate semantically disturbed image-based negatives for training the model, thereby enhancing its overall performance. AHNPL also introduces a contrastive learning approach using a multimodal hard negative loss to improve the model's discrimination of hard negatives within each modality and a dynamic margin loss that adjusts the contrastive margin according to sample difficulty to enhance the distinction of challenging sample pairs. Experiments on three public datasets demonstrate that our method effectively boosts VLMs' performance on complex CR tasks. The source code is available at https://github.com/nynu-BDAI/AHNPL.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。