Existing Masked Image Modeling methods apply fixed mask patterns to guide the self-supervised training. As those mask patterns resort to different criteria to depict image contents, sticking to a fixed pattern leads to a limited vision cues modeling capability.This paper introduces an evolved hierarchical masking method to pursue general visual cues modeling in self-supervised learning. The proposed method leverages the vision model being trained to parse the input visual cues into a hierarchy structure, which is hence adopted to generate masks accordingly. The accuracy of hierarchy is on par with the capability of the model being trained, leading to evolved mask patterns at different training stages. Initially, generated masks focus on low-level visual cues to grasp basic textures, then gradually evolve to depict higher-level cues to reinforce the learning of more complicated object semantics and contexts. Our method does not require extra pre-trained models or annotations and ensures training efficiency by evolving the training difficulty. We conduct extensive experiments on seven downstream tasks including partial-duplicate image retrieval relying on low-level details, as well as image classification and semantic segmentation that require semantic parsing capability. Experimental results demonstrate that it substantially boosts performance across these tasks. For instance, it surpasses the recent MAE by 1.1\% in imageNet-1K classification and 1.4\% in ADE20K segmentation with the same training epochs. We also align the proposed method with the current research focus on LLMs. The proposed approach bridges the gap with large-scale pre-training on semantic demanding tasks and enhances intricate detail perception in tasks requiring low-level feature recognition.
This paper studies the 3D instance segmentation problem, which has a variety of real-world applications such as robotics and augmented reality. Since the surroundings of 3D objects are of high complexity, the separating of different objects is very difficult. To address this challenging problem, we propose a novel framework to group and refine the 3D instances. In practice, we first learn an offset vector for each point and shift it to its predicted instance center. To better group these points, we propose a Hierarchical Point Grouping algorithm to merge the centrally aggregated points progres…
OMUDA: Omni-level Masking for Unsupervised Domain Adaptation in Semantic Segmentation
School of Mechanical Engineering, Chengdu University（成都大学机械工程学院）；School of Information Science and Technology, Harbin Institute of Technology（哈尔滨工业大学信息科学与技术学院）；School of Computer Science and Informatics, Cardiff University（卡的夫大学计算机科学与信息学院）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学与技术学院）
Michigan State University（密歇根州立大学）；AInnovation（旷视）
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight sharing based NAS approaches still adopt hand-designed heuristics to generate architecture candidates. As a consequence, the space of architecture candidates is constrained in a subset of all possible architectures, making the architecture search results sub-optimal. HM-NAS addresses…
HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）
Granular Ball Guided Masking: Structure-aware Data Augmentation
Chongqing Key Laboratory of Computational Intelligence（重庆计算智能重点实验室）；Key Laboratory of Cyberspace Big Data Intelligent Security（网络大数据智能安全重点实验室）；Sichuan-Chongqing Co-construction Key Laboratory of Digital Economy Intelligence（四川-重庆数字经济智能联合重点实验室）；Key Laboratory of Big Data Intelligent Computing（大数据智能计算重点实验室）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
University of Chinese Academy of Sciences（中国科学院大学）；Fudan University（复旦大学）
Recently, masked image modeling (MIM) has offered a new methodology of self-supervised pre-training of vision transformers. A key idea of efficient implementation is to discard the masked image patches (or tokens) throughout the target network (encoder), which requires the encoder to be a plain vision transformer (e.g., ViT), albeit hierarchical vision transformers (e.g., Swin Transformer) have potentially better properties in formulating vision inputs. In this paper, we offer a new design of hierarchical vision transformers named HiViT (short for Hierarchical ViT) that enjoys both high effici…
Models based on Convolutional Neural Networks (CNNs) have been proven very successful for semantic segmentation and object parsing that yield hierarchies of features. Our key insight is to build convolutional networks that take input of arbitrary size and produce object parsing output with efficient inference and learning. In this work, we focus on the task of instance segmentation and parsing which recognizes and localizes objects down to a pixel level base on deep CNN. Therefore, unlike some related work, a pixel cannot belong to multiple instances and parsing. Our model is based on a deep n…
Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis
Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University（湘潭大学教育部智能计算与信息处理重点实验室）；School of Computer Science, Xiangtan University（湘潭大学计算机科学学院）；Key Laboratory for Artificial Intelligence and International Communication, Hunan Normal University（湖南师范大学人工智能与国际传播重点实验室）
Zhejiang University（浙江大学）；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；COMAC Beijing Aircraft Technology Research Institute（中国商飞北京民用飞机技术研究中心）
Despite the tremendous progress of Masked Autoencoders (MAE) in developing vision tasks such as image and video, exploring MAE in large-scale 3D point clouds remains challenging due to the inherent irregularity. In contrast to previous 3D MAE frameworks, which either design a complex decoder to infer masked information from maintained regions or adopt sophisticated masking strategies, we instead propose a much simpler paradigm. The core idea is to apply a \textbf{G}enerative \textbf{D}ecoder for MAE (GD-MAE) to automatically merges the surrounding context to restore the masked geometric knowle…
Abdullah Gül University（阿卜杜拉·居尔大学）；Boğaziçi University（博阿齐奇大学）；University College Dublin（都柏林大学学院）
Deep-learning based computer vision models have proved themselves to be ground-breaking approaches to human activity recognition (HAR). However, most existing works are dedicated to improve the prediction accuracy through either creating new model architectures, increasing model complexity, or refining model parameters by training on larger datasets. Here, we propose an alternative idea, differing from existing work, to increase model accuracy and also to shape model predictions to align with human understandings through automatically creating higher-level summarizing labels for similar groups…
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-leve…
HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peking University（北京大学）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Harbin Institute of Technology（哈尔滨工业大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
University of California, Berkeley（加州大学伯克利分校）；Tencent America（美国腾讯公司）；Pennsylvania State University（宾夕法尼亚州立大学）
Action understanding has evolved into the era of fine granularity, as most human behaviors in real life have only minor differences. To detect these fine-grained actions accurately in a label-efficient way, we tackle the problem of weakly-supervised fine-grained temporal action detection in videos for the first time. Without the careful design to capture subtle differences between fine-grained actions, previous weakly-supervised models for general action detection cannot perform well in the fine-grained setting. We propose to model actions as the combinations of reusable atomic actions which a…
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Archimedes, Athena Research Center（阿基米德研究中心）；National Technical University of Athens（希腊国家技术大学）；University of Crete（克里特大学）；IACM-Forth（第四研究机构（IACM-Forth））
We present a novel approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry. Our approach is fully self-supervised and can hence be trained solely on real-world, incomplete scans. To achieve self-supervision, we remove frames from a given (incomplete) 3D scan in order to make it even more incomplete; self-supervision is then formulated by correlating the two levels of partialness of the same scan while masking out regions that have never been observed. Through generalization across a large training set, we can then p…
Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution
Virginia Tech（弗吉尼亚理工大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；University of California, Irvine（加利福尼亚大学欧文分校）；Tulane University（杜兰大学）；Battelle（巴特尔公司）；Duke University（杜克大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The Ohio State University（俄亥俄州立大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
