arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.19953cs.CV
收藏

通过优化反事实实现运动概念的自监督学习

Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

Stanford University(斯坦福大学)

Stefan Stojanov, David Wendt, Seungwoo Kim, Rahul Venkatesh, Kevin Feigelis, Jiajun Wu, Daniel LK Yamins

AI总结:

本文提出Opt-CWM,一种无需标注数据的自监督运动估计方法,通过优化反事实探针从预训练视频模型中提取光流和遮挡信息,在真实视频上达到最先进性能。

摘要:

视频中的运动估计是一个重要的计算机视觉问题,具有许多下游应用,包括可控视频生成和机器人技术。当前的解决方案主要使用合成数据进行训练,或需要调整特定情境的启发式方法,这从根本上限制了这些模型在现实世界场景中的能力。尽管近期在大规模视频自监督学习方面取得了进展,但利用这些表示进行运动估计仍然相对未被充分探索。在这项工作中,我们开发了Opt-CWM,一种自监督技术,用于从预训练的下一帧预测模型中进行光流和遮挡估计。Opt-CWM通过学习优化反事实探针,从基础视频模型中提取运动信息,从而避免了对固定启发式方法的需求,同时可以在不受限制的视频输入上进行训练。我们在真实世界视频上实现了运动估计的最先进性能,且无需任何标注数据。

英文摘要:

Estimating motion in videos is an essential computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are primarily trained using synthetic data or require tuning of situation-specific heuristics, which inherently limits these models' capabilities in real-world contexts. Despite recent developments in large-scale self-supervised learning from videos, leveraging such representations for motion estimation remains relatively underexplored. In this work, we develop Opt-CWM, a self-supervised technique for flow and occlusion estimation from a pre-trained next-frame prediction model. Opt-CWM works by learning to optimize counterfactual probes that extract motion information from a base video model, avoiding the need for fixed heuristics while training on unrestricted video inputs. We achieve state-of-the-art performance for motion estimation on real-world videos while requiring no labeled data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑