VidChapters-7M:大规模视频章节
VidChapters-7M: Video Chapters at Scale
AI总结:
本文构建含81.7万视频、700万章节的大规模 VidChapters-7M 数据集,定义章节生成及定位任务并评测视频-语言模型,验证其预训练可显著提升密集视频字幕性能。
摘要:
将长视频分割成章节使用户能够快速导航到他们感兴趣的信息。由于缺乏公开发布的数据集,这一重要课题一直未得到充分研究。为解决这一问题,我们提出了 VidChapters-7M,这是一个包含81.7万个由用户标注章节的视频、总计700万个章节的数据集。VidChapters-7M 通过抓取用户标注的章节,以可扩展的方式从在线视频中自动创建,因此无需任何额外的人工标注。我们基于这些数据引入了以下三项任务。首先,视频章节生成任务包括在时间上分割视频并为每个片段生成章节标题。为进一步剖析该问题,我们还定义了该任务的两个变体:给定真实边界的视频章节生成,要求在给定已标注视频片段的情况下生成章节标题;以及视频章节定位,要求根据章节的已标注标题在时间上定位该章节。我们针对这三项任务对简单基线和最先进的视频-语言模型进行了基准测试。我们还表明,在 VidChapters-7M 上进行预训练可很好地迁移到密集视频字幕任务,在零样本和微调设置下均如此,大幅提升了 YouCook2 和 ViTT 基准上的最先进水平。最后,我们的实验表明,下游性能随预训练数据集规模的扩大而良好扩展。我们的数据集、代码和模型可在 https://antoyang.github.io/vidchapters.html 公开获取。
英文摘要:
Segmenting long videos into chapters enables users to quickly navigate to the information of their interest. This important topic has been understudied due to the lack of publicly released datasets. To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total. VidChapters-7M is automatically created from videos online in a scalable manner by scraping user-annotated chapters and hence without any additional manual annotation. We introduce the following three tasks based on this data. First, the video chapter generation task consists of temporally segmenting the video and generating a chapter title for each segment. To further dissect the problem, we also define two variants of this task: video chapter generation given ground-truth boundaries, which requires generating a chapter title given an annotated video segment, and video chapter grounding, which requires temporally localizing a chapter given its annotated title. We benchmark both simple baselines and state-of-the-art video-language models for these three tasks. We also show that pretraining on VidChapters-7M transfers well to dense video captioning tasks in both zero-shot and finetuning settings, largely improving the state of the art on the YouCook2 and ViTT benchmarks. Finally, our experiments reveal that downstream performance scales well with the size of the pretraining dataset. Our dataset, code, and models are publicly available at https://antoyang.github.io/vidchapters.html.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。