Few-Shot视觉故事的主题适应和原型编码
Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling
摘要：
Visual Storytelling~（VIST）是根据给定的照片流讲述关于某个主题的叙事故事的任务。现有的研究集中在设计复杂的模型，这依赖于大量的人类注释的数据。然而，VIST的标注是非常昂贵的，并且由于长尾主题分布，许多主题无法在训练数据集中覆盖。在本文中，我们专注于提高推广能力的VIST模型，考虑Few-Shot设置。受人类讲故事的方式的启发，我们提出了一个主题自适应说书人来模拟主题间泛化的能力。在实践中，我们将基于梯度的Meta学习算法应用于多模态seq 2 seq模型，以赋予模型快速适应不同主题的能力。此外，我们还提出了一个原型编码结构来模拟主题内派生的能力。具体来说，我们编码和恢复的几个训练故事的文本作为参考，以指导在推理时的生成。实验结果表明，主题自适应和原型编码结构对Few-Shot模型在BLEU和METEOR度量上的性能都有很大的改善。进一步的个案研究表明，经过Few-Shot改编后的故事具有更强的相对性和表现力。
英文摘要：
Visual Storytelling~(VIST) is a task to tell a narrative story about a certain topic according to the given photo stream. The existing studies focus on designing complex models, which rely on a huge amount of human-annotated data. However, the annotation of VIST is extremely costly and many topics cannot be covered in the training dataset due to the long-tail topic distribution. In this paper, we focus on enhancing the generalization ability of the VIST model by considering the few-shot setting. Inspired by the way humans tell a story, we propose a topic adaptive storyteller to model the ability of inter-topic generalization. In practice, we apply the gradient-based meta-learning algorithm on multi-modal seq2seq models to endow the model the ability to adapt quickly from topic to topic. Besides, We further propose a prototype encoding structure to model the ability of intra-topic derivation. Specifically, we encode and restore the few training story text to serve as a reference to guide the generation at inference time. Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model on BLEU and METEOR metric. The further case study shows that the stories generated after few-shot adaptation are more relative and expressive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
