arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1903.02230cs.CL
收藏

Dixit:通过术语操作的交互式视觉故事讲述

Dixit: Interactive Visual Storytelling via Term Manipulation

Academia Sinica(中央研究院);Pennsylvania State University(宾夕法尼亚州立大学)

Chao-Chun Hsu, Yu-Hua Chen, Zi-Yuan Chen, Hsin-Yu Lin, Ting-Hao 'Kenneth' Huang, Lun-Wei Ku

摘要:

在本文中,我们介绍Diplomat,一个交互式的视觉讲故事系统,用户交互迭代组成一个简短的故事的照片序列。用户通过上传一系列照片来启动该过程。Diplomat首先从每张照片中提取描述对象的文本术语(例如,男孩、自行车)或动作(例如,睡眠),然后允许用户添加新的术语或删除现有术语。然后,Diplomat根据这些术语生成一个短篇小说。在幕后,Diplomat使用基于LSTM的模型,在图像标题数据和FrameNet上进行训练,从每个图像中提取术语,并利用Transformer解码器来组成上下文一致的故事。用户使用Diplomacy反复更改图像或术语,以创建最理想的故事。Diplomacy还允许用户手动编辑和评级的故事。拟议的程序开辟了可解释和可控的视觉讲故事的可能性,让用户了解的故事形成的理由,并在生成过程中进行干预。

英文摘要:

In this paper, we introduce Dixit, an interactive visual storytelling system that the user interacts with iteratively to compose a short story for a photo sequence. The user initiates the process by uploading a sequence of photos. Dixit first extracts text terms from each photo which describe the objects (e.g., boy, bike) or actions (e.g., sleep) in the photo, and then allows the user to add new terms or remove existing terms. Dixit then generates a short story based on these terms. Behind the scenes, Dixit uses an LSTM-based model trained on image caption data and FrameNet to distill terms from each image and utilizes a transformer decoder to compose a context-coherent story. Users change images or terms iteratively with Dixit to create the most ideal story. Dixit also allows users to manually edit and rate stories. The proposed procedure opens up possibilities for interpretable and controllable visual storytelling, allowing users to understand the story formation rationale and to intervene in the generation process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑