从图像描述到视觉叙事
From Image Captioning to Visual Storytelling
AI总结：
本研究将视觉叙事视为图像描述的超集，先通过视觉到语言模型生成图像描述，再以语言到语言方法转化为连贯叙事，缩短训练时间且易复用，还提出ideality指标模拟与基准模型的差距及类人性。
摘要：
视觉叙事是视觉与语言领域间一项极具挑战性的多模态任务，其目标是为图像流生成故事。该任务的难点在于，生成的故事既要基于图像序列，又需具备叙事性和连贯性。本研究旨在平衡这些方面，将视觉叙事视为图像描述的超集，这一方法与多数先前相关研究截然不同。具体而言，我们首先采用视觉到语言模型获取输入图像的描述，随后通过语言到语言方法将这些描述转化为连贯的叙事。我们的多方面评估表明，在统一框架下整合图像描述与叙事，对生成故事的质量有积极影响。此外，与众多先前研究相比，该方法缩短了训练时间，使我们的框架易于被感兴趣的人复用和复现。最后，我们提出了一种名为ideality的新指标/工具，可用于模拟某些结果与基准模型的差距，并将其应用于模拟视觉叙事中的类人性。
英文摘要：
Visual Storytelling is a challenging multimodal task between Vision & Language, where the purpose is to generate a story for a stream of images. Its difficulty lies on the fact that the story should be both grounded to the image sequence but also narrative and coherent. The aim of this work is to balance between these aspects, by treating Visual Storytelling as a superset of Image Captioning, an approach quite different compared to most of prior relevant studies. This means that we firstly employ a vision-to-language model for obtaining captions of the input images, and then, these captions are transformed into coherent narratives using language-to-language methods. Our multifarious evaluation shows that integrating captioning and storytelling under a unified framework, has a positive impact on the quality of the produced stories. In addition, compared to numerous previous studies, this approach accelerates training time and makes our framework readily reusable and reproducible by anyone interested. Lastly, we propose a new metric/tool, named ideality, that can be used to simulate how far some results are from an oracle model, and we apply it to emulate human-likeness in visual storytelling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
