SCO-VIST：基于社交交互常识知识的视觉叙事
SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling
AI总结：
针对现有视觉叙事模型缺乏社交常识的问题，提出SCO-VIST框架，通过融入人类行为动机与社交交互常识构建带权故事图，用Floyd-Warshall算法生成故事线，在多类评估指标上表现更优。
摘要：
视觉叙事旨在根据给定的图像序列自动生成连贯的故事。与图像描述等任务不同，视觉故事应包含事实描述、世界观以及人类社交常识，从而将分散的元素串联起来，形成连贯且有吸引力、符合人类创作风格的故事。然而，现有大多数模型在创作故事时，主要侧重于运用事实信息，并采用分类/词汇类外部知识。本文提出SCO-VIST框架，该框架将图像序列表示为包含对象与关系的图，其中融入了人类行为动机及其社交交互常识知识。随后，SCO-VIST利用这一代表情节点的图，通过基于语义和发生概率的边权重在各情节点之间搭建关联。该带权故事图采用Floyd-Warshall算法生成按事件顺序排列的故事线。自动评估与人工评估结果均显示，本文提出的框架在视觉基础、连贯性、多样性以及拟人化程度等多项指标上表现更优。
英文摘要：
Visual storytelling aims to automatically generate a coherent story based on a given image sequence. Unlike tasks like image captioning, visual stories should contain factual descriptions, worldviews, and human social commonsense to put disjointed elements together to form a coherent and engaging human-writeable story. However, most models mainly focus on applying factual information and using taxonomic/lexical external knowledge when attempting to create stories. This paper introduces SCO-VIST, a framework representing the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge. SCO-VIST then takes this graph representing plot points and creates bridges between plot points with semantic and occurrence-based edge weights. This weighted story graph produces the storyline in a sequence of events using Floyd-Warshall's algorithm. Our proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
