面向新闻图像描述的视觉感知上下文建模
Visually-Aware Context Modeling for News Image Captioning
AI总结：
本文提出一种面向新闻图像描述的视觉感知上下文建模框架，包含人脸命名模块、基于CLIP的检索策略及CoLaM方法，在GoodNews和NYTimes800k数据集上显著超越现有方法。
摘要：
新闻图像描述旨在根据新闻文章和图像生成描述，强调文本上下文与视觉元素之间的联系。认识到新闻图像中人脸的重要性以及现有数据集中人脸-姓名共现模式，我们提出了一种人脸命名模块，以学习更好的姓名嵌入。除了可以直接与图像区域（人脸）关联的姓名外，新闻图像描述主要包含只能在文章中找到的上下文信息。我们设计了一种使用CLIP的检索策略，用于检索在语义上与图像接近的句子，模仿人类将文章与图像联系起来的思维过程。此外，为了解决描述中文章上下文与图像上下文比例不平衡的问题，我们在训练流程中引入了一种简单而有效的方法——Contrasting with Language Model backbone (CoLaM)。我们进行了广泛的实验以证明我们框架的有效性。我们在GoodNews/NYTimes800k上分别以7.97/5.80的CIDEr分数超越了之前的最先进方法（不使用外部数据）。我们的代码可在https://github.com/tingyu215/VACNIC获取。
英文摘要：
News Image Captioning aims to create captions from news articles and images, emphasizing the connection between textual context and visual elements. Recognizing the significance of human faces in news images and the face-name co-occurrence pattern in existing datasets, we propose a face-naming module for learning better name embeddings. Apart from names, which can be directly linked to an image area (faces), news image captions mostly contain context information that can only be found in the article. We design a retrieval strategy using CLIP to retrieve sentences that are semantically close to the image, mimicking human thought process of linking articles to images. Furthermore, to tackle the problem of the imbalanced proportion of article context and image context in captions, we introduce a simple yet effective method Contrasting with Language Model backbone (CoLaM) to the training pipeline. We conduct extensive experiments to demonstrate the efficacy of our framework. We out-perform the previous state-of-the-art (without external data) by 7.97/5.80 CIDEr scores on GoodNews/NYTimes800k. Our code is available at https://github.com/tingyu215/VACNIC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
