TruNet:基于故事保持截断的长视频短视频生成算法
TruNet: Short Videos Generation from Long Videos via Story-Preserving Truncation
摘要:
在这项工作中,我们引入了一个新的问题,命名为{ \em故事保持长视频截断},这需要一个算法来自动截断一个长时间的视频到多个短的和有吸引力的子视频,每个包含一个完整的故事。这与传统的视频亮点检测或视频摘要问题不同,因为每个子视频都需要保持连贯和完整的故事,这对于Youtube,Facebook,TikTok,Kwai等资源制作视频共享平台变得尤为重要。为了解决这个问题,我们收集并注释了一个新的大型视频截断数据集,命名为TruNet,它包含1470个视频,平均每个视频有11个短篇故事。利用新的数据集,我们进一步开发和训练了一个用于视频截断的神经架构,该架构由两个组件组成:边界感知网络(BAN)和快进长短期记忆(FF-LSTM)。我们首先使用BAN生成高质量的时间建议,共同考虑帧级的吸引力和边界。然后,我们应用FF-LSTM,它倾向于捕获帧序列之间的高阶依赖关系,以确定时间提案是否是一个连贯和完整的故事。我们表明,我们提出的框架优于现有的方法的故事保持长视频截断问题的定量措施和用户研究。该数据集可在https://ai.baidu.com/broad/download上用于公共学术研究。
英文摘要:
In this work, we introduce a new problem, named as {\em story-preserving long video truncation}, that requires an algorithm to automatically truncate a long-duration video into multiple short and attractive sub-videos with each one containing an unbroken story. This differs from traditional video highlight detection or video summarization problems in that each sub-video is required to maintain a coherent and integral story, which is becoming particularly important for resource-production video sharing platforms such as Youtube, Facebook, TikTok, Kwai, etc. To address the problem, we collect and annotate a new large video truncation dataset, named as TruNet, which contains 1470 videos with on average 11 short stories per video. With the new dataset, we further develop and train a neural architecture for video truncation that consists of two components: a Boundary Aware Network (BAN) and a Fast-Forward Long Short-Term Memory (FF-LSTM). We first use the BAN to generate high quality temporal proposals by jointly considering frame-level attractiveness and boundaryness. We then apply the FF-LSTM, which tends to capture high-order dependencies among a sequence of frames, to decide whether a temporal proposal is a coherent and integral story. We show that our proposed framework outperforms existing approaches for the story-preserving long video truncation problem in both quantitative measures and user-study. The dataset is available for public academic research usage at https://ai.baidu.com/broad/download.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。