arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03478cs.CVcs.LG
收藏

VEDIT:面向程序性视频表示学习的潜在预测架构

VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

UNC Chapel Hill(北卡罗来纳大学教堂山分校);Meta FAIR

Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, Koustuv Sinha

AI总结:

本文提出VEDIT,利用冻结视觉编码器的潜在嵌入和扩散Transformer,在不预训练预测模型且无需语言或ASR监督的情况下,显著提升程序性视频的长时程动作预期、步骤预测、任务分类与程序规划性能。

摘要:

程序性视频表示学习是一个活跃的研究领域,其目标是学习一个智能体,使其能够在给定当前视频输入的情况下,通常结合文本标注,预测和预报未来。先前的工作通常依赖大规模预训练视觉编码器和带有语言监督的预测模型。然而,扩展计算密集型预训练以学习带有噪声文本监督的视频片段序列的必要性和有效性尚未被先前工作充分验证。在这项工作中,我们表明,一个强大的现成冻结预训练视觉编码器,配合一个精心设计的预测模型,可以在预测和程序性规划方面实现最先进(SoTA)的性能,而无需对预测模型进行预训练,也不需要来自语言或ASR的额外监督。我们的方法不从像素空间学习表示,而是利用公开可用的视觉编码器的潜在嵌入空间。通过以观察步骤的冻结片段级嵌入为条件来预测未见步骤的动作,我们的预测模型能够通过迭代去噪学习用于预测的鲁棒表示——利用了扩散Transformer(Peebles & Xie, 2023)的最新进展。在四个数据集(NIV、CrossTask、COIN和Ego4D-v2)上共五个程序性学习任务的实证研究表明,我们的模型在长时程动作预期方面超越了强基线(Verb ED@20提升+2.6%,Noun ED@20提升+3.1%),并显著改善了步骤预测(+5.0%)、任务分类(+3.8%)和程序规划任务(成功率最高提升+2.28%,mAcc提升+3.39%,mIoU提升+0.90%)的最先进性能。

英文摘要:

Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising - leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023). Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑