arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.09082cs.CV
收藏

驯服生成视频模型以实现零样本光流提取

Taming generative video models for zero-shot optical flow extraction

Stanford University(斯坦福大学)

Seungwoo Kim, Khai Loong Aw, Klemen Kotar, Cristobal Eyzaguirre, Wanhee Lee, Yunong Liu, Jared Watrous, Stefan Stojanov, Juan Carlos Niebles, Jiajun Wu, Daniel L. K. Yamins

AI总结:

本文提出KL-tracing方法,通过反事实提示实现生成视频模型的零样本光流提取,无需微调即可在现实和合成数据集上竞争现有最佳模型。

摘要:

从视频中提取光流仍然是计算机视觉中的核心问题。受近年来大规模通用模型的成功启发,我们问是否可以将仅训练以预测未来帧的冻结自监督视频模型通过提示(而无需微调)来输出光流。先前尝试从视频生成器中读取深度或照明信息需要微调;这种策略对于光流来说并不合适,因为在光流中标注数据稀缺,合成数据集存在仿真到现实的差距。受反事实世界模型(CWM)范式的启发,该范式可以通过在下一帧预测器中注入一个小的追踪扰动并跟踪其传播来获得逐点对应关系,我们将其扩展到生成视频模型中以实现零样本光流提取。我们探索了几种流行的架构,并发现通过这种方式成功实现零样本光流提取受到三种模型属性的帮助:(1)对将来帧的分布预测(避免模糊或噪声输出);(2)因子化的潜在变量,将每个时空块独立处理;以及(3)随机访问解码,可以对任意子集的未来像素进行条件化。这些属性唯一地存在于最近引入的局部随机访问序列(LRAS)架构中。基于LRAS,我们提出了KL追踪:一种新的测试时间推理过程,它在第一帧中注入局部扰动,逐步推演模型,并计算扰动和未扰动预测分布之间的Kullback-Leibler散度。无需任何光流特定的微调,我们的方法在现实世界TAP-Vid DAVIS基准和合成TAP-Vid Kubric上与最先进的任务特定模型竞争。我们的结果表明,可控生成视频模型的反事实提示是高质量光流的有效替代方法,替代监督或光度损失方法。

英文摘要:

Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method is competitive with state-of-the-art, task-specific models on the real-world TAP-Vid DAVIS benchmark and the synthetic TAP-Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative to supervised or photometric-loss methods for high-quality flow.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑