arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.01823cs.CV
收藏

用于导航教学视频的绕行(Detours)方法

Detours for Navigating Instructional Videos

UT Austin(德克萨斯大学奥斯汀分校);FAIR, Meta(Meta FAIR研究院)

Kumar Ashutosh, Zihui Xue, Tushar Nagarajan, Kristen Grauman

AI总结:

本文提出视频绕行问题及VidDetours方法,通过视频-语言检索和弱监督训练,从教学视频库中查找满足用户修改需求的替代步骤,在烹饪视频数据集上召回率超越现有最佳方法35%。

摘要:

我们引入了视频绕行问题,用于导航教学视频。给定一个源视频和一个自然语言查询,该查询要求以某种方式改变当前操作视频的执行路径,目标是找到一个相关的“绕行视频”,以满足所请求的改变。为了应对这一挑战,我们提出了VidDetours,这是一种新颖的视频-语言方法,它利用视频和文本条件查询,学习从大规模操作视频库中检索目标时间段。此外,我们设计了一个基于语言的流程,利用操作视频的旁白文本来创建弱监督训练数据。我们展示了这一想法在烹饪操作视频领域的应用,用户可以从当前食谱绕行,以寻找具有替代食材、工具和技巧的步骤。在一个包含16K样本的真实标注数据集上进行验证,我们展示了我们的模型相对于现有最佳视频检索和问答方法的显著改进,召回率超过最先进水平35%。

英文摘要:

We introduce the video detours problem for navigating instructional videos. Given a source video and a natural language query asking to alter the how-to video's current path of execution in a certain way, the goal is to find a related ''detour video'' that satisfies the requested alteration. To address this challenge, we propose VidDetours, a novel video-language approach that learns to retrieve the targeted temporal segments from a large repository of how-to's using video-and-text conditioned queries. Furthermore, we devise a language-based pipeline that exploits how-to video narration text to create weakly supervised training data. We demonstrate our idea applied to the domain of how-to cooking videos, where a user can detour from their current recipe to find steps with alternate ingredients, tools, and techniques. Validating on a ground truth annotated dataset of 16K samples, we show our model's significant improvements over best available methods for video retrieval and question answering, with recall rates exceeding the state of the art by 35%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑