变换、对比与叙述：连贯的实体感知多图像描述生成
Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning
AI总结：
本文针对新闻文档中相邻图像描述缺乏连贯性的问题，提出基于Transformer和两种对比学习连贯机制的实体感知多图像描述生成模型，并构建DM800K数据集，实验证明其在描述质量和连贯性上均优于多个基线。
摘要：
连贯的实体感知多图像描述生成旨在为新闻文档中的相邻图像生成连贯的描述。相邻图像之间往往存在连贯关系，因为它们通常描述相同的实体或事件。这些关系对于实体感知的多图像描述生成非常重要，但在实体感知的单图像描述生成中却被忽视。现有的大部分工作集中于单图像描述生成，而多图像描述生成此前尚未被探索。因此，本文通过利用连贯关系，提出了一种连贯的实体感知多图像描述生成模型。该模型由一个基于Transformer的描述生成模型和两种基于对比学习的连贯机制组成。生成模型通过关注图像及其伴随文本来生成描述。描述-描述连贯机制旨在使图像描述中的实体也出现在相邻图像的描述中。描述-图像-文本连贯机制旨在使图像描述中的实体也出现在伴随文本中。为了评估描述之间的连贯性，本文提出了两种连贯性评估指标。构建的新数据集DM800K比现有的两个数据集GoodNews和NYT800K在每个文档中包含更多图像，更适合多图像描述生成。在三个数据集上的实验表明，所提出的描述生成模型在BLUE、Rouge、METEOR以及实体精确率和召回率得分上优于7个基线。实验还表明，根据描述实体得分、描述Rouge得分、所提出的两种连贯性评估指标以及人工评估，生成的描述比基线的描述更具连贯性。
英文摘要：
Coherent entity-aware multi-image captioning aims to generate coherent captions for neighboring images in a news document. There are coherence relationships among neighboring images because they often describe same entities or events. These relationships are important for entity-aware multi-image captioning, but are neglected in entity-aware single-image captioning. Most existing work focuses on single-image captioning, while multi-image captioning has not been explored before. Hence, this paper proposes a coherent entity-aware multi-image captioning model by making use of coherence relationships. The model consists of a Transformer-based caption generation model and two types of contrastive learning-based coherence mechanisms. The generation model generates the caption by paying attention to the image and the accompanying text. The caption-caption coherence mechanism aims to render entities in the caption of the image be also in captions of neighboring images. The caption-image-text coherence mechanism aims to render entities in the caption of the image be also in the accompanying text. To evaluate coherence between captions, two coherence evaluation metrics are proposed. The new dataset DM800K is constructed that has more images per document than two existing datasets GoodNews and NYT800K, and is more suitable for multi-image captioning. Experiments on three datasets show the proposed captioning model outperforms 7 baselines according to BLUE, Rouge, METEOR, and entity precision and recall scores. Experiments also show that the generated captions are more coherent than that of baselines according to caption entity scores, caption Rouge scores, the two proposed coherence evaluation metrics, and human evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
