视觉故事中实体的归纳与指称
Induction and Reference of Entities in a Visual Story
摘要：
在我们的日常生活中，我们被视觉解释的故事所包围。我们叙述一个故事的方式通常包括两个阶段，即形成实体的中心思维导图，然后围绕它们编织一个故事。连贯性的一个促进因素不仅是以这些实体为基础的故事，而且还使用适当的术语来提及它们以避免重复。在本文中，我们解决了这两个阶段，在看似合理的交界处引入正确的实体，并在视觉故事的背景下连贯地引用它们。中心思维导图的构建块，也被称为实体骨架，是包括名词性和共指表达的实体链。这个实体骨架也被表示在不同的抽象层次中，以组成一个概括的框架来编织故事。我们建立在一个编码器-解码器框架上，当解码的故事不符合这个实体骨架时，我们会惩罚模型。我们建立了一个强大的基线骨架知情生成，然后扩展到具有多任务的能力，除了生成故事预测的骨架。最后，我们建立在这个模型，并提出了一个glocal层次注意力模型，出席在句子（本地）和故事（全球）的水平的骨架。我们观察到，我们提出的模型在自动评估指标METEOR方面优于基线。我们执行各种分析，以评估我们执行实体骨架任务的性能，例如生成的实体的数量和多样性。我们还进行了人类评估，从中得出结论，我们的模型生成的视觉故事在82%的情况下是首选的。此外，我们表明，我们的glocal层次注意力模型提高了一致性，通过引入更多的代词所需的名词的存在。
英文摘要：
We are enveloped by stories of visual interpretations in our everyday lives. The way we narrate a story often comprises of two stages, which are, forming a central mind map of entities and then weaving a story around them. A contributing factor to coherence is not just basing the story on these entities but also, referring to them using appropriate terms to avoid repetition. In this paper, we address these two stages of introducing the right entities at seemingly reasonable junctures and also referring them coherently in the context of visual storytelling. The building blocks of the central mind map, also known as entity skeleton are entity chains including nominal and coreference expressions. This entity skeleton is also represented in different levels of abstractions to compose a generalized frame to weave the story. We build upon an encoder-decoder framework to penalize the model when the decoded story does not adhere to this entity skeleton. We establish a strong baseline for skeleton informed generation and then extend this to have the capability of multitasking by predicting the skeleton in addition to generating the story. Finally, we build upon this model and propose a glocal hierarchical attention model that attends to the skeleton both at the sentence (local) and the story (global) levels. We observe that our proposed models outperform the baseline in terms of automatic evaluation metric, METEOR. We perform various analysis targeted to evaluate the performance of our task of enforcing the entity skeleton such as the number and diversity of the entities generated. We also conduct human evaluation from which it is concluded that the visual stories generated by our model are preferred 82% of the times. In addition, we show that our glocal hierarchical attention model improves coherence by introducing more pronouns as required by the presence of nouns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
