arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.12694cs.CVcs.AI
收藏

多模态视频章节生成

Multi-modal Video Chapter Generation

Shandong University(山东大学);ByteDance(字节跳动)

Xiao Cao, Zitan Chen, Canyu Le, Lei Meng

摘要:

章节生成是目前在线视频的实用技术。章节断点使用户能够快速找到他们想要的部分并获得总结性注释。但是,没有用于此任务的公共方法和数据集。为了促进沿着这一方向的研究,我们引入了一个新的数据集,称为Chapter-Gen,它由大约10 k个用户生成的带有注释章节信息的视频组成。我们的数据汇集过程快速、可扩展,不需要任何额外的手动注释。在此基础上,我们设计了一个有效的基线,专门用于视频章节生成任务。其捕获视频的两个方面,包括视觉动态和叙述文本。它将本地和全局视频特征分别用于本地化和标题生成。为了有效地解析长视频,设计了跳过滑动窗口机制来定位潜在章节。并开发了交叉注意力多模态融合模块,用于融合局部特征生成标题。我们的实验表明,所提出的框架比现有方法取得了更好的结果,这表明即使经过微调,类似任务的方法设计也不能直接转移。代码和数据集可在https://github.com/czt117/MVCG上获得。

英文摘要:

Chapter generation becomes practical technique for online videos nowadays. The chapter breakpoints enable users to quickly find the parts they want and get the summative annotations. However, there is no public method and dataset for this task. To facilitate the research along this direction, we introduce a new dataset called Chapter-Gen, which consists of approximately 10k user-generated videos with annotated chapter information. Our data collection procedure is fast, scalable and does not require any additional manual annotation. On top of this dataset, we design an effective baseline specificlly for video chapters generation task. which captures two aspects of a video,including visual dynamics and narration text. It disentangles local and global video features for localization and title generation respectively. To parse the long video efficiently, a skip sliding window mechanism is designed to localize potential chapters. And a cross attention multi-modal fusion module is developed to aggregate local features for title generation. Our experiments demonstrate that the proposed framework achieves superior results over existing methods which illustrate that the method design for similar task cannot be transfered directly even after fine-tuning. Code and dataset are available at https://github.com/czt117/MVCG.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑