arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07398cs.CVcs.AIcs.LGcs.RO
收藏

基于随机帧预测的视觉表征学习

Visual Representation Learning with Stochastic Frame Prediction

KAIST(韩国科学技术院);UC Berkeley(加州大学伯克利分校);Dyson Robot Learning Lab(戴森机器人学习实验室)

Huiwon Jang, Dongyoung Kim, Junsu Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo

AI总结:

本文提出结合随机帧预测与掩码图像建模的自监督视觉表征学习框架,以共享解码器高效学习时序和帧内稠密信息,并在视频与机器人任务上验证有效性。

摘要:

通过预测未来帧来自监督学习图像表征是一个很有前景的方向,但仍然是一项挑战。这是由帧预测的欠确定性本质造成的;单一当前帧可能产生多个潜在未来。为应对这一挑战,本文重新审视了随机视频生成这一思路,该思路学习捕捉帧预测中的不确定性,并探究其在表征学习中的有效性。具体而言,我们设计了一个框架,训练随机帧预测模型以学习帧之间的时序信息。此外,为学习每帧内部的稠密信息,我们引入了辅助的掩码图像建模目标,并采用共享解码器架构。我们发现,该架构能够以协同且计算高效的方式结合这两个目标。我们在来自视频标签传播和基于视觉的机器人学习领域的多种任务上证明了框架的有效性,例如视频分割、姿态跟踪、基于视觉的机器人运动以及操作任务。代码可在项目网页获取:https://sites.google.com/view/2024rsp。

英文摘要:

Self-supervised learning of image representations by predicting future frames is a promising direction but still remains a challenge. This is because of the under-determined nature of frame prediction; multiple potential futures can arise from a single current frame. To tackle this challenge, in this paper, we revisit the idea of stochastic video generation that learns to capture uncertainty in frame prediction and explore its effectiveness for representation learning. Specifically, we design a framework that trains a stochastic frame prediction model to learn temporal information between frames. Moreover, to learn dense information within each frame, we introduce an auxiliary masked image modeling objective along with a shared decoder architecture. We find this architecture allows for combining both objectives in a synergistic and compute-efficient manner. We demonstrate the effectiveness of our framework on a variety of tasks from video label propagation and vision-based robot learning domains, such as video segmentation, pose tracking, vision-based robotic locomotion, and manipulation tasks. Code is available on the project webpage: https://sites.google.com/view/2024rsp.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑