看见还是猜测：反事实正则化的图像描述
See or Guess: Counterfactually Regularized Image Captioning
AI总结：
本文提出基于因果推断的反事实正则化图像描述框架，利用总效应或自然直接效应增强模型对遮挡和编辑图像的描述能力，有效减少幻觉并提升泛化性与可移植性。
摘要：
图像描述旨在为图像中的视觉信息生成自然语言描述，是视觉-语言研究中的一项关键任务。以往模型通常通过统计拟合现有数据集，使机器的生成能力与人类智能对齐。这类方法对正常图像有效，但在图像某些部分被遮挡或编辑时，可能难以准确描述，而人类在这些情况下表现优异。这些模型表现出的弱点，包括幻觉和可解释性有限，往往会在关联模式发生偏移的场景中损害性能。本文提出一个通用的图像描述框架，利用因果推断使现有模型在干预任务中更具能力，并具备反事实可解释性。我们的方法包含两种变体，分别利用总效应或自然直接效应。将它们集成到训练过程中，使模型能够处理反事实场景，从而提升泛化能力。在多个数据集上的大量实验表明，我们的方法有效减少了幻觉，提高了模型对图像的忠实度，并在小规模和大规模图像到文本模型上均展现出很高的可移植性。代码见 https://github.com/Aman-4-Real/See-or-Guess。
英文摘要：
Image captioning, which generates natural language descriptions of the visual information in an image, is a crucial task in vision-language research. Previous models have typically addressed this task by aligning the generative capabilities of machines with human intelligence through statistical fitting of existing datasets. While effective for normal images, they may struggle to accurately describe those where certain parts of the image are obscured or edited, unlike humans who excel in such cases. These weaknesses they exhibit, including hallucinations and limited interpretability, often hinder performance in scenarios with shifted association patterns. In this paper, we present a generic image captioning framework that employs causal inference to make existing models more capable of interventional tasks, and counterfactually explainable. Our approach includes two variants leveraging either total effect or natural direct effect. Integrating them into the training process enables models to handle counterfactual scenarios, increasing their generalizability. Extensive experiments on various datasets show that our method effectively reduces hallucinations and improves the model's faithfulness to images, demonstrating high portability across both small-scale and large-scale image-to-text models. The code is available at https://github.com/Aman-4-Real/See-or-Guess.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
