在3D中重建和接地讲述的教学视频
Reconstructing and grounding narrated instructional videos in 3D
摘要：
叙述式教学视频通常显示和描述类似物体的操作，例如，修理某款汽车或笔记本电脑。在这项工作中，我们的目标是重建这样的对象，并在3D中本地化相关的叙述。与实例级3D重建的标准场景相反，其中相同的对象或场景存在于所有视图中，不同教学视频中的对象在给定相同产品的不同条件和版本的情况下可能具有较大的外观变化。叙述还可以在自然语言表达中具有大的变化。我们通过三个贡献来应对这些挑战。首先，我们提出了一种方法的对应关系估计相结合的学习局部特征和稠密流。其次，我们设计了一个两步分治重建方法，其中单个视频的初始3D重建被组合成一个3D对齐图。最后，我们提出了一种无监督的方法，地面自然语言在获得的三维重建。我们证明了我们的方法在汽车维修领域的有效性。在没有人工监督的情况下，我们的方法成功地重建了不同汽车模型的发动机，并将文本描述与3D中的相应对象相关联。
英文摘要：
Narrated instructional videos often show and describe manipulations of similar objects, e.g., repairing a particular model of a car or laptop. In this work we aim to reconstruct such objects and to localize associated narrations in 3D. Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product. Narrations may also have large variation in natural language expressions. We address these challenges by three contributions. First, we propose an approach for correspondence estimation combining learnt local features and dense flow. Second, we design a two-step divide and conquer reconstruction approach where the initial 3D reconstructions of individual videos are combined into a 3D alignment graph. Finally, we propose an unsupervised approach to ground natural language in obtained 3D reconstructions. We demonstrate the effectiveness of our approach for the domain of car maintenance. Given raw instructional videos and no manual supervision, our method successfully reconstructs engines of different car models and associates textual descriptions with corresponding objects in 3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
