视频讲故事：事件的文本摘要
Video Storytelling: Textual Summaries for Events
摘要：
在计算机视觉和多媒体研究中，将视觉和自然语言连接起来是一个长期的目标。虽然早期的作品专注于生成一个单一的句子描述的视觉内容，最近的作品研究段落生成。在这项工作中，我们介绍了视频讲故事的问题，其目的是为长视频生成连贯和简洁的故事。视频叙事带来了新的挑战，主要是由于故事的多样性以及视频的长度和复杂性。我们提出了新的方法来应对这些挑战。首先，我们提出了一个用于多模态嵌入学习的上下文感知框架，在这个框架中，我们设计了一个残差双向递归神经网络来利用来自过去和未来的上下文信息。其次，我们提出了一个叙事者模型来发现潜在的故事情节。叙述者被制定为一个强化学习代理，它是通过直接优化生成的故事的文本度量来训练的。我们在视频故事数据集上评估了我们的方法，这是我们为进行研究而收集的一个新数据集。我们将我们的方法与多个最先进的基线进行比较，并表明我们的方法在定量测量和用户研究方面具有更好的性能。
英文摘要：
Bridging vision and natural language is a longstanding goal in computer vision and multimedia research. While earlier works focus on generating a single-sentence description for visual content, recent works have studied paragraph generation. In this work, we introduce the problem of video storytelling, which aims at generating coherent and succinct stories for long videos. Video storytelling introduces new challenges, mainly due to the diversity of the story and the length and complexity of the video. We propose novel methods to address the challenges. First, we propose a context-aware framework for multimodal embedding learning, where we design a Residual Bidirectional Recurrent Neural Network to leverage contextual information from past and future. Second, we propose a Narrator model to discover the underlying storyline. The Narrator is formulated as a reinforcement learning agent which is trained by directly optimizing the textual metric of the generated story. We evaluate our method on the Video Story dataset, a new dataset that we have collected to enable the study. We compare our method with multiple state-of-the-art baselines, and show that our method achieves better performance, in terms of quantitative measures and user study.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
