作为时空学习器的拼接掩码自编码器
Concatenated Masked Autoencoders as Spatial-Temporal Learner
AI总结：
本研究提出用于自监督视频表征学习的时空学习器CatMAE，通过首帧保留、后续帧高比例掩码及前后帧可见块重建机制学习场景演化，结合ViRe增强策略，在视频分割与动作识别任务上取得领先性能。
摘要：
从视频中学习表征需要理解连续运动以及帧间的视觉对应关系。本文提出拼接掩码自编码器（Concatenated Masked Autoencoders，CatMAE），作为一种用于自监督视频表征学习的时空学习器。对于输入的视频帧序列，CatMAE保持初始帧不变，同时对后续帧施加95%的高比例掩码。CatMAE的编码器负责分别为每一帧编码可见的图像块；随后，对于每个被掩码的帧，解码器利用来自前序帧和当前帧的可见图像块来重建原始图像。本文提出的方法使模型能够估计可见图像块之间的运动信息，匹配前后帧之间的对应关系，并最终学习场景的演化过程。此外，本文提出了一种新的数据增强策略Video-Reverse（ViRe，视频反转），该策略将反转后的视频帧作为模型的重建目标。这进一步促使模型利用连续运动细节和对应关系完成重建，从而增强模型的能力。与最先进的预训练方法相比，CatMAE在视频分割任务和动作识别任务上达到了领先水平。
英文摘要：
Learning representations from videos requires understanding continuous motion and visual correspondences between frames. In this paper, we introduce the Concatenated Masked Autoencoders (CatMAE) as a spatial-temporal learner for self-supervised video representation learning. For the input sequence of video frames, CatMAE keeps the initial frame unchanged while applying substantial masking (95%) to subsequent frames. The encoder in CatMAE is responsible for encoding visible patches for each frame individually; subsequently, for each masked frame, the decoder leverages visible patches from both previous and current frames to reconstruct the original image. Our proposed method enables the model to estimate the motion information between visible patches, match the correspondences between preceding and succeeding frames, and ultimately learn the evolution of scenes. Furthermore, we propose a new data augmentation strategy, Video-Reverse (ViRe), which uses reversed video frames as the model's reconstruction targets. This further encourages the model to utilize continuous motion details and correspondences to complete the reconstruction, thereby enhancing the model's capabilities. Compared to the most advanced pre-training methods, CatMAE achieves a leading level in video segmentation tasks and action recognition tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
