教学视频中的步骤差异
Step Differences in Instructional Video
AI总结：
本文提出利用HowTo100M自动生成视频对指令数据并训练视频条件语言模型，实现跨视频联合推理，在识别教学视频步骤差异和严重程度排序上达到最先进水平。
摘要：
将用户视频与参考操作教程视频进行比较，是为用户提供个性化辅助的AR/VR技术的关键需求。然而，当前基于语言的辅助方法只能回答关于单个视频的问题。我们提出一种方法：首先利用HowTo100M中已有的步骤标注和伴随叙述，自动生成大量涉及视频对的视觉指令微调数据，然后训练一个以视频为条件的语言模型，使其能够跨多个原始视频进行联合推理。我们的模型在识别视频对之间的差异以及根据差异严重程度对视频进行排序方面达到了最先进的性能，并展现出对多个视频进行通用推理的良好能力。项目页面：https://github.com/facebookresearch/stepdiff
英文摘要：
Comparing a user video to a reference how-to video is a key requirement for AR/VR technology delivering personalized assistance tailored to the user's progress. However, current approaches for language-based assistance can only answer questions about a single video. We propose an approach that first automatically generates large amounts of visual instruction tuning data involving pairs of videos from HowTo100M by leveraging existing step annotations and accompanying narrations, and then trains a video-conditioned language model to jointly reason across multiple raw videos. Our model achieves state-of-the-art performance at identifying differences between video pairs and ranking videos based on the severity of these differences, and shows promising ability to perform general reasoning over multiple videos. Project page: https://github.com/facebookresearch/stepdiff
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
