arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.21546cs.CVcs.AIcs.CLcs.LG

反事实分割推理:诊断和缓解像素接地幻觉

Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucination

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Xinzhuo Li, Adheesh Juvekar, Jiaxun Zhang, Xingyou Liu, Muntasir Wahed, Kiet A. Nguyen, Yifan Shen, Tianjiao Yu, Ismini Lourentzou

AI总结:

本文提出反事实分割推理任务,通过反事实微调训练模型减少分割幻觉,实验表明其在减少幻觉和提升分割性能方面效果显著。

摘要:

分割视觉-语言模型(VLMs)在提升基于视觉的理解方面取得了显著进展,但它们仍然容易产生像素接地幻觉,生成错误对象的掩码或完全不存在对象的掩码。现有的评估几乎完全依赖于文本或标签基于的扰动,只检查预测的掩码是否与查询标签匹配。此类评估忽略了幻觉的空间足迹和严重性,因此无法揭示由视觉驱动的幻觉,这些幻觉更具挑战性且更普遍。为了解决这一差距,我们正式提出了反事实分割推理(CSR)的任务,其中模型必须在事实图像中分割参考对象,并在反事实对应物中放弃。为了支持这一任务,我们整理了HalluSegBench,这是首个大规模基准,利用受控的视觉反事实来诊断提及和推理表达的分割幻觉,并引入了新的评估指标,这些指标衡量幻觉的严重性并分离视觉和语言驱动的失败模式。我们进一步介绍了RobustSeg,一个通过反事实微调(CFT)训练的分割VLM,以学习何时分割和何时放弃。实验结果证实,RobustSeg将幻觉减少了30%,同时在FP-RefCOCO(+/g)上提高了分割性能。

英文摘要:

Segmentation Vision-Language Models (VLMs) have significantly advanced grounded visual understanding, yet they remain prone to pixel-grounding hallucinations, producing masks for incorrect objects or for objects that are entirely absent. Existing evaluations rely almost entirely on text- or label-based perturbations, which check only whether the predicted mask matches the queried label. Such evaluations overlook the spatial footprint and severity of hallucination and therefore fail to reveal vision-driven hallucinations, which are more challenging and more prevalent. To address this gap, we formalize the task of Counterfactual Segmentation Reasoning (CSR), where a model must segment the referenced object in the factual image and abstain in its counterfactual counterpart. To support this task, we curate HalluSegBench, the first large-scale benchmark to diagnose referring and reasoning expression segmentation hallucinations using controlled visual counterfactuals, alongside new evaluation metrics that measure hallucination severity and disentangle vision- and language-driven failure modes. We further introduce RobustSeg, a segmentation VLM trained with counterfactual fine-tuning (CFT) to learn when to segment and when to abstain. Experimental results confirm RobustSeg reduces hallucinations by 30%, while improving segmentation performance on FP-RefCOCO(+/g).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑