通过自洽解释改进视觉定位
Improved Visual Grounding through Self-Consistent Explanations
AI总结：
本文提出用大型语言模型生成释义，并通过 SelfEQ 弱监督训练让短语与释义的视觉解释图保持一致，从而显著提升视觉-语言模型的视觉定位性能。
摘要：
经过训练以匹配图像与文本的视觉-语言模型，可以与视觉解释方法相结合，从而指向图像中特定对象的位置。我们的工作表明，通过针对自洽视觉解释进行微调，可以进一步提升这些模型的定位——即“grounding”——能力。我们提出了一种利用大型语言模型为现有文本-图像数据集增加释义的策略，以及 SelfEQ，这是一种针对释义视觉解释图的弱监督策略，用于鼓励自洽性。具体而言，对于输入文本短语，我们尝试生成一个释义，并微调模型，使该短语和释义映射到图像中的同一区域。我们假设，这既扩展了模型能够处理的词汇量，也提高了基于梯度的视觉解释方法（例如 GradCAM）所突出的对象位置质量。我们证明，相较于一个强基线方法和若干先前工作，SelfEQ 提升了在 Flickr30k、ReferIt 和 RefCOCO+ 上的性能。特别是，与其他不使用任何类型边界框标注的方法相比，我们在 Flickr30k 上获得 84.07%（绝对提升 4.69%），在 ReferIt 上获得 67.40%（绝对提升 7.68%），并在 RefCOCO+ 测试集 A 和 B 上分别获得 75.10% 和 55.49%（平均绝对提升 3.74%）。
英文摘要：
Vision-and-language models trained to match images with text can be combined with visual explanation methods to point to the locations of specific objects in an image. Our work shows that the localization --"grounding"-- abilities of these models can be further improved by finetuning for self-consistent visual explanations. We propose a strategy for augmenting existing text-image datasets with paraphrases using a large language model, and SelfEQ, a weakly-supervised strategy on visual explanation maps for paraphrases that encourages self-consistency. Specifically, for an input textual phrase, we attempt to generate a paraphrase and finetune the model so that the phrase and paraphrase map to the same region in the image. We posit that this both expands the vocabulary that the model is able to handle, and improves the quality of the object locations highlighted by gradient-based visual explanation methods (e.g. GradCAM). We demonstrate that SelfEQ improves performance on Flickr30k, ReferIt, and RefCOCO+ over a strong baseline method and several prior works. Particularly, comparing to other methods that do not use any type of box annotations, we obtain 84.07% on Flickr30k (an absolute improvement of 4.69%), 67.40% on ReferIt (an absolute improvement of 7.68%), and 75.10%, 55.49% on RefCOCO+ test sets A and B respectively (an absolute improvement of 3.74% on average).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
