带旁白的“操作指南”视频已成为极具潜力的数据源，可用于从学习视觉表征到训练机器人策略的各类学习问题。然而这类数据噪声极大，因为旁白并不总是描述视频中展示的动作。为解决这一问题，我们提出了视觉旁白检测这一新任务，其目标是判断一段旁白是否由视频中的动作视觉呈现。我们提出了What You Say is What You Show（WYS^2）方法，该方法利用多模态线索和伪标签技术，仅需弱标注数据即可学习检测视觉旁白。我们的模型能成功检测真实场景视频中的视觉旁白，性能优于强基线模型，并且我们验证了该方法对教学视频的前沿摘要生成和时序对齐任务的提升作用。
英文摘要：
Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies. However, this data is extremely noisy, as the narrations do not always describe the actions demonstrated in the video. To address this problem we introduce the novel task of visual narration detection, which entails determining whether a narration is visually depicted by the actions in the video. We propose What You Say is What You Show (WYS^2), a method that leverages multi-modal cues and pseudo-labeling to learn to detect visual narrations with only weakly labeled data. Our model successfully detects visual narrations in in-the-wild videos, outperforming strong baselines, and we demonstrate its impact for state-of-the-art summarization and temporal alignment of instructional videos.
Reconstructing and grounding narrated instructional videos in 3D
Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；CIIRC – Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（布拉格捷克技术大学信息学、机器人与控制论研究所）；Microsoft（微软）；ETH Zurich（苏黎世联邦理工学院）；Facebook AI Research（Facebook人工智能研究院）
RECIPE: Procedural Planning via Grounding in Instructional Video
Khoury College of Computer Sciences, Northeastern University, Boston（东北大学北斯托顿学院计算机科学学院）；Department of Mathematics and Computer Science, University of Catania, Italy（卡塔尼亚大学数学与计算机科学系）
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Goethe University Frankfurt（法兰克福大学）；SIC；University of Bonn（波恩大学）；University of Oxford（牛津大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Saarland University（萨尔大学）
