VideoMAC：视频掩码自编码器遇见 ConvNets
VideoMAC: Video Masked Autoencoders Meet ConvNets
AI总结：
本文提出 VideoMAC，将视频掩码自编码器与稀疏卷积 ConvNets 结合，并通过双编码器促进帧间重建一致性，在多个视频下游任务上超过基于 ViT 的方法。
摘要：
近来，掩码自编码器（MAE）等自监督学习技术的进展，极大影响了图像和视频的视觉表征学习。然而值得注意的是，现有掩码图像/视频建模中的主流方法过度依赖资源密集型的 vision transformers（ViTs）作为特征编码器。本文提出一种名为 VideoMAC 的新方法，将视频掩码自编码器与资源友好的 ConvNets 相结合。具体而言，VideoMAC 对随机采样的视频帧对采用对称掩码。为防止掩码模式消散问题，我们使用以稀疏卷积算子实现的 ConvNets 作为编码器。同时，我们提出一种简单而有效的掩码视频建模（MVM）方法，即由在线编码器和指数移动平均目标编码器组成的双编码器架构，旨在促进视频中的帧间重建一致性。此外，我们证明，VideoMAC 使经典（ResNet）/现代（ConvNeXt）卷积编码器能够利用 MVM 的优势，并在下游任务上优于基于 ViT 的方法，包括视频对象分割（提升 5.2%/6.4% J&F）、身体部位传播（提升 6.3%/3.1% mIoU）以及人体姿态跟踪（提升 10.2%/11.1% PCK@0.1）。
英文摘要：
Recently, the advancement of self-supervised learning techniques, like masked autoencoders (MAE), has greatly influenced visual representation learning for images and videos. Nevertheless, it is worth noting that the predominant approaches in existing masked image / video modeling rely excessively on resource-intensive vision transformers (ViTs) as the feature encoder. In this paper, we propose a new approach termed as \textbf{VideoMAC}, which combines video masked autoencoders with resource-friendly ConvNets. Specifically, VideoMAC employs symmetric masking on randomly sampled pairs of video frames. To prevent the issue of mask pattern dissipation, we utilize ConvNets which are implemented with sparse convolutional operators as encoders. Simultaneously, we present a simple yet effective masked video modeling (MVM) approach, a dual encoder architecture comprising an online encoder and an exponential moving average target encoder, aimed to facilitate inter-frame reconstruction consistency in videos. Additionally, we demonstrate that VideoMAC, empowering classical (ResNet) / modern (ConvNeXt) convolutional encoders to harness the benefits of MVM, outperforms ViT-based approaches on downstream tasks, including video object segmentation (+\textbf{5.2\%} / \textbf{6.4\%} $\mathcal{J}\&\mathcal{F}$), body part propagation (+\textbf{6.3\%} / \textbf{3.1\%} mIoU), and human pose tracking (+\textbf{10.2\%} / \textbf{11.1\%} PCK@0.1).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
