arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.18227cs.CVcs.CL
收藏

GUIDE:面向教学视频理解的大纲引导数据集

GUIDE: A Guideline-Guided Dataset for Instructional Video Comprehension

Harbin Institute of Technology(哈尔滨工业大学);Kuaishou Technology(快手科技);Peng Cheng Laboratory(鹏城实验室)

Jiafeng Liang, Shixin Jiang, Zekun Wang, Haojie Pan, Zerui Chen, Zheng Chu, Ming Liu, Ruiji Fu, Zhongyuan Wang, Bing Qin

AI总结:

针对现有教学视频数据集缺乏任务级经验大纲的问题,提出GUIDE数据集,包含560个任务的3.5K视频,并设计三个子任务评估模型理解能力,作为更优基准。

摘要:

互联网上有大量教学视频,它们为我们完成各种任务提供了教程。现有的教学视频数据集仅关注视频层面的具体步骤,缺乏任务层面的经验性大纲,这可能导致初学者因缺乏相关经验而难以学习新任务。此外,没有大纲指导的具体步骤显得琐碎且缺乏系统性,难以提供清晰的教程。为解决这些问题,我们提出了GUIDE(大纲引导)数据集,其中包含8个与日常生活相关的领域中的560个教学任务的3.5K个视频。具体来说,我们为每个教学任务标注了一个大纲,代表了所有与任务相关视频共有的常见模式。在此基础上,我们标注了系统的具体步骤,包括其关联的大纲步骤、具体步骤描述和时间戳。我们提出的基准测试包含三个子任务,以评估模型的理解能力:(1)步骤描述生成:模型必须从视频中为具体步骤生成描述。(2)大纲总结:模型必须挖掘任务相关视频中的常见模式,并从中总结出大纲。(3)大纲引导的描述生成:模型必须在大纲的引导下为具体步骤生成描述。我们使用GUIDE评估了多种基础模型,并进行了深入分析。鉴于GUIDE的多样性和实用性,我们相信它可以作为教学视频理解的更好基准。

英文摘要:

There are substantial instructional videos on the Internet, which provide us tutorials for completing various tasks. Existing instructional video datasets only focus on specific steps at the video level, lacking experiential guidelines at the task level, which can lead to beginners struggling to learn new tasks due to the lack of relevant experience. Moreover, the specific steps without guidelines are trivial and unsystematic, making it difficult to provide a clear tutorial. To address these problems, we present the GUIDE (Guideline-Guided) dataset, which contains 3.5K videos of 560 instructional tasks in 8 domains related to our daily life. Specifically, we annotate each instructional task with a guideline, representing a common pattern shared by all task-related videos. On this basis, we annotate systematic specific steps, including their associated guideline steps, specific step descriptions and timestamps. Our proposed benchmark consists of three sub-tasks to evaluate comprehension ability of models: (1) Step Captioning: models have to generate captions for specific steps from videos. (2) Guideline Summarization: models have to mine the common pattern in task-related videos and summarize a guideline from them. (3) Guideline-Guided Captioning: models have to generate captions for specific steps under the guide of guideline. We evaluate plenty of foundation models with GUIDE and perform in-depth analysis. Given the diversity and practicality of GUIDE, we believe that it can be used as a better benchmark for instructional video comprehension.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑