Masked Image Modeling (MIM) has emerged as a promising approach for Self-Supervised Learning (SSL) of visual representations. However, the out-of-the-box performance of MIMs is typically inferior to competing approaches. Most users cannot afford fine-tuning due to the need for large amounts of data, high GPU consumption, and specialized user knowledge. Therefore, the practical use of MIM representations is limited. In this paper we ask what is the reason for the poor out-of-the-box performance of MIMs. Is it due to weaker features produced by MIM models, or is it due to suboptimal usage? Through detailed analysis, we show that attention in MIMs is spread almost uniformly over many patches, leading to ineffective aggregation by the [cls] token. Based on this insight, we propose Selective Aggregation to better capture the rich semantic information retained in patch tokens, which significantly improves the out-of-the-box performance of MIM.
Exploring the Coordination of Frequency and Attention in Masked Image Modeling
School of Cyber Science and Engineering, Southeast University（东南大学网络空间科学与工程学院）；Purple Mountain Laboratories（紫金山实验室）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Alibaba Group（阿里巴巴集团）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）；Zhuhai UM Science and Technology Research Institute（珠海澳大科技研究院）；Faculty of Science & Technology at University of Macau（澳门大学科技学院）
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Huazhong University of Science and Technology（华中科技大学）；Zhejiang University（浙江大学）；Beijing Institute of Technology（北京理工大学）
Vehicle-centric Perception via Multimodal Structured Pre-training
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；the School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电机工程与计算机科学学院）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Environmental Computational Science and Earth Observation Laboratory (ECEO)（环境计算科学与地球观测实验室）
Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet-B7 for Improved Gastrointestinal Abnormality Classification in Video Capsule Endoscopy
International Institute of Information Technology, Bangalore（国际信息技术学院，班加罗尔）；Indian Institute of Technology, Hyderabad（印度理工学院，海得拉巴）
An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Training
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Megvii Research（旷视研究院）；Beijing Institute of Basic Medical Sciences（北京基础医学研究所）；Nanchang Hangkong University（南昌航空大学）；Zhejiang University of Technology（浙江工业大学）
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of the Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Tencent Youtu Lab（腾讯优图实验室）
Exploring the GLIDE model for Human Action-effect Prediction
University of Leeds（利兹大学）；Luzhong Institute of Safety, Qingdao University of Science and Technology（青岛科技大学鲁中安全研究院）；Tongji University（同济大学）；Qingdao University of Science and Technology（青岛科技大学）；Shandong University（山东大学）
GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds
Zhejiang University（浙江大学）；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；COMAC Beijing Aircraft Technology Research Institute（中国商飞北京民用飞机技术研究中心）
Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints
Faculty of Engineering and IT, University of Technology Sydney（技术悉尼大学工程与信息学院）；Dept. of Comp. Sci. and Eng., Southern University of Science and Technology（南方科技大学计算机科学与工程系）
Exploring Visual Interpretability for Contrastive Language-Image Pre-training
The Hong Kong University of Science and Technology（香港科技大学）；University of Technology Sydney（悉尼科技大学）
对比图像预训练（CLIP）通过对自然语言的现成监督来学习丰富的表示。它提高了下游视觉任务的性能，包括但不限于zero-shot、长尾、分割、检索、字幕和视频。然而，CLIP的视觉可解释性研究很少，特别是对于原始特征图。为了提供其预测的视觉解释，我们提出了图像-文本相似性图（ITSM）。在此基础上，我们惊讶地发现，CLIP喜欢的背景区域比前景，并显示错误的可视化结果对人类的理解。这种现象在视觉Transformer和卷积网络中都是普遍存在的，这表明这个问题是独特的，而不是由于特定的网络。通过实验，我们发现问题出在池化部分，不适当的池化方法会导致一种称为语义转换的现象。针对这个问题，我们提出了可解释的对比图像预训练（ECLIP），它通过Masked Max Pooling来纠正可解释性。具体来说，为了避免语义转移，我们用max pooling代替了原来的注意力池，专注于自信的前景，并在训练过程中得到自由注意力的指导。在三个数据集上的实验表明，ECLIP大大提高了CLIP的可解释性，并在很大程度上超过了以前的可解释性方法。代码将在稍后发布。
Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
School of Computer Science, China University of Geosciences（中国地质大学（武汉）计算机学院）；Engineering Research Center of Natural Resource Information Management and Digital Twin Engineering Software, Ministry of Education（教育部自然资源信息管理与数字孪生工程软件工程研究中心）；Second Surveying and Mapping Institute of Hunan Province（湖南省第二测绘院）；Key Laboratory of Natural Resources Monitoring and Supervision in Southern Hilly Region, Ministry of Natural Resources（自然资源部南方山地自然资源监测与监管重点实验室）
LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
Arizona State University（亚利桑那州立大学）；Clemson University（克莱姆森大学）；LinkedIn Corporation（领英公司）；Washington University in St. Louis（圣路易斯华盛顿大学）；Morgan Stanley（摩根大通）
Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School（生物工程跨学科项目，首尔国立大学研究生院）；Integrated Major in Innovative Medical Science, Seoul National University Graduate School（创新医学科学整合专业，首尔国立大学研究生院）；Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院第一附属医院放射科）；Seoul National University College of Medicine（首尔国立大学医学院）；Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital（首尔国立大学医学院放射科，首尔国立大学医院）；Institute of Medical and Biological Engineering, Seoul National University Medical Research Center（医学与生物工程研究所，首尔国立大学医学研究所以及）；Institute of Radiation Medicine, Seoul National University Medical Research Center（放射医学研究所，首尔国立大学医学研究所以及）
UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval
Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）；Dalian University of Technology（大连理工大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Microsoft Research（微软研究院）
