使用判别性微调的跨域图像描述生成
Cross-Domain Image Captioning with Discriminative Finetuning
AI总结：
本文提出使用自监督判别性交流目标对现成神经描述生成器进行微调，以生成更具信息量的视觉描述，在跨域数据集上更接近人类参考描述，并在图像判别任务中为人类标注员提供更大帮助。
摘要：
神经描述生成器通常被训练以模仿人类生成的参考描述，而不针对任何特定的交流目标进行优化，这导致了诸如生成模糊描述等问题。在本文中，我们展示了使用自监督的判别性交流目标对现成的神经描述生成器进行微调，有助于恢复一种朴素、具有视觉描述性的语言，这种语言能提供更多关于图像内容的信息。给定一张目标图像，系统必须学会生成一种描述，使得现成的文本条件图像检索器能够在候选图像集中识别出该图像。我们使用流行的 ClipCap 描述生成器进行了实验，并使用 BLIP 复现了主要结果。在与真实人类描述的相似度方面，当非微调模型在同一描述数据集上训练和测试时，判别性微调生成的描述略微落后于非微调模型生成的描述。然而，当模型在无需进一步调优的情况下用于为域外数据集生成描述时，我们的判别性微调描述生成器生成的描述比未经微调的同一描述生成器生成的描述更接近人类参考描述。我们进一步表明，在 Conceptual Captions 数据集上，对于执行图像判别任务的人类标注员来说，判别性微调的描述比原始 ClipCap 描述或真实描述更有帮助。
英文摘要：
Neural captioners are typically trained to mimic human-generated references without optimizing for any specific communication goal, leading to problems such as the generation of vague captions. In this paper, we show that fine-tuning an out-of-the-box neural captioner with a self-supervised discriminative communication objective helps to recover a plain, visually descriptive language that is more informative about image contents. Given a target image, the system must learn to produce a description that enables an out-of-the-box text-conditioned image retriever to identify such image among a set of candidates. We experiment with the popular ClipCap captioner, also replicating the main results with BLIP. In terms of similarity to ground-truth human descriptions, the captions emerging from discriminative finetuning lag slightly behind those generated by the non-finetuned model, when the latter is trained and tested on the same caption dataset. However, when the model is used without further tuning to generate captions for out-of-domain datasets, our discriminatively-finetuned captioner generates descriptions that resemble human references more than those produced by the same captioner without finetuning. We further show that, on the Conceptual Captions dataset, discriminatively finetuned captions are more helpful than either vanilla ClipCap captions or ground-truth captions for human annotators tasked with an image discrimination task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
