arXiv 2506.072272025-06-08cs.CVcs.CL

一眼识别幻觉：受控视觉编辑与细粒度多模态学习

Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning

HKUST（香港科技大学）；Shanghai AI Lab（上海人工智能实验室）；Peking University（北京大学）；HKUST(GZ)（香港科技大学（广州））；Oxford University（牛津大学）；Imperial College London（伦敦帝国理工学院）

Tianyi Bai, Yuxuan Fan, Jiantao Qiu, Fupeng Sun, Jiayi Song, Junlin Han, Zichen Liu, Conghui He, Wentao Zhang, Binhang Yuan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型难以察觉细微视觉变化而产生幻觉的问题，本文构建含5万余图文对的MED微编辑数据集，并提出带特征级一致性损失的SFT框架，显著提升微编辑检测及通用视觉语言任务表现。

摘要：

多模态大语言模型（MLLMs）在视觉-语言任务上取得了强劲表现，但仍难以处理细粒度视觉差异，从而导致幻觉或遗漏语义变化。我们将此归因于训练数据和学习目标两方面的局限。为解决这些问题，我们提出一种受控数据生成流程，用于生成经过最小编辑的图像对，并配有语义对齐的图像描述。利用该流程，我们构建了 Micro Edit Dataset（MED，微编辑数据集），其中包含超过 5 万个图文对，涵盖 11 个细粒度编辑类别，包括属性、数量、位置和物体存在性变化。在 MED 的基础上，我们引入了一个带有特征级一致性损失的监督微调（SFT）框架，该损失促使模型在小幅编辑下保持稳定的视觉嵌入。我们在 Micro Edit Detection（微编辑检测）基准上评估了所提方法，该基准包含经过仔细平衡的评估图像对，旨在测试模型对相同编辑类别中细微视觉变化的敏感性。与包括 GPT-4o 在内的强基线相比，我们的方法提高了差异检测准确率并减少了幻觉。此外，该方法还在图像描述生成和视觉问答等标准视觉-语言任务上带来了一致提升。这些结果证明，将有针对性的数据与对齐目标相结合，能够有效增强 MLLMs 的细粒度视觉推理能力。

英文摘要：

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but still struggle with fine-grained visual differences, leading to hallucinations or missed semantic shifts. We attribute this to limitations in both training data and learning objectives. To address these issues, we propose a controlled data generation pipeline that produces minimally edited image pairs with semantically aligned captions. Using this pipeline, we construct the Micro Edit Dataset (MED), containing over 50K image-text pairs spanning 11 fine-grained edit categories, including attribute, count, position, and object presence changes. Building on MED, we introduce a supervised fine-tuning (SFT) framework with a feature-level consistency loss that promotes stable visual embeddings under small edits. We evaluate our approach on the Micro Edit Detection benchmark, which includes carefully balanced evaluation pairs designed to test sensitivity to subtle visual variations across the same edit categories. Our method improves difference detection accuracy and reduces hallucinations compared to strong baselines, including GPT-4o. Moreover, it yields consistent gains on standard vision-language tasks such as image captioning and visual question answering. These results demonstrate the effectiveness of combining targeted data and alignment objectives for enhancing fine-grained visual reasoning in MLLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。