掩码视频自编码器（Masked Video Autoencoder, MVA）方法已展现出巨大潜力，其性能显著超越了以往的视频表示学习方法。然而，由于采用随机掩码策略，这类方法在预测无信息价值的令牌/帧时浪费了大量计算资源与内存（例如，需使用超过16个节点、配备128块NVIDIA A100 GPU的算力）。为解决这一问题，我们利用视频块之间信息密度不均的特性，提出了EVEREST——一种效率极高的视频表示学习MVA方法，该方法在预训练和微调阶段均能筛选出包含丰富运动特征的令牌，并丢弃无信息价值的令牌。我们进一步提出了一种高信息密度帧选择策略，使模型能够聚焦于信息丰富且具有因果性的帧，同时将冗余度降至最低。我们的方法大幅降低了MVA的计算与内存需求，仅需单台配备8块GPU的机器即可完成预训练和微调，且在多个基准数据集以及未经筛选的Ego4D数据集上，性能与计算和内存开销巨大的基线方法相当。我们希望本研究能为降低视频理解领域进一步研究的门槛贡献力量。
英文摘要：
Masked Video Autoencoder (MVA) approaches have demonstrated their potential by significantly outperforming previous video representation learning methods. However, they waste an excessive amount of computations and memory in predicting uninformative tokens/frames due to random masking strategies. (e.g., over 16 nodes with 128 NVIDIA A100 GPUs). To resolve this issue, we exploit the unequal information density among the patches in videos and propose EVEREST, a surprisingly efficient MVA approach for video representation learning that finds tokens containing rich motion features and discards uninformative ones during both pre-training and fine-tuning. We further present an information-intensive frame selection strategy that allows the model to focus on informative and causal frames with minimal redundancy. Our method significantly reduces the computation and memory requirements of MVA, enabling the pre-training and fine-tuning on a single machine with 8 GPUs while achieving comparable performance to computation- and memory-heavy baselines on multiple benchmarks and the uncurated Ego4D dataset. We hope that our work contributes to reducing the barrier to further research on video understanding.
Huazhong University of Science and Technology（华中科技大学）；Alibaba Group（阿里巴巴集团）
Standard approaches for video recognition usually operate on the full input videos, which is inefficient due to the widely present spatio-temporal redundancy in videos. Recent progress in masked video modelling, i.e., VideoMAE, has shown the ability of vanilla Vision Transformers (ViT) to complement spatio-temporal contexts given only limited visual contents. Inspired by this, we propose propose Masked Action Recognition (MAR), which reduces the redundant computation by discarding a proportion of patches and operating only on a part of the videos. MAR contains the following two indispensable c…
Sparse approximation is the problem to find the sparsest linear combination for a signal from a redundant dictionary, which is widely applied in signal processing and compressed sensing. In this project, I manage to implement the Orthogonal Matching Pursuit (OMP) algorithm and Sparse Representation-based Classification (SRC) algorithm, then use them to finish the task of masked image classification with majority voting. Here the experiment was token on the AR data-set, and the result shows the superiority of OMP algorithm combined with SRC algorithm over masked face image classification with a…
Nanjing University（南京大学）；Tencent AI Lab（腾讯人工智能实验室）；Shanghai AI Lab（上海人工智能实验室）
Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three i…
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Harnessing Massive Satellite Imagery with Efficient Masked Image Modeling
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Tsinghua University（清华大学）；School of Computer Science, Wuhan University（计算机科学学院，武汉大学）；Zhongguancun Academy（中关村学院）；Nankai University（南开大学）
Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis
Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University（湘潭大学教育部智能计算与信息处理重点实验室）；School of Computer Science, Xiangtan University（湘潭大学计算机科学学院）；Key Laboratory for Artificial Intelligence and International Communication, Hunan Normal University（湖南师范大学人工智能与国际传播重点实验室）
