Advances in self-supervised learning are essential for enhancing feature extraction and understanding in point cloud processing. This paper introduces PMT-MAE (Point MLP-Transformer Masked Autoencoder), a novel self-supervised learning framework for point cloud classification. PMT-MAE features a dual-branch architecture that integrates Transformer and MLP components to capture rich features. The Transformer branch leverages global self-attention for intricate feature interactions, while the parallel MLP branch processes tokens through shared fully connected layers, offering a complementary feature transformation pathway. A fusion mechanism then combines these features, enhancing the model's capacity to learn comprehensive 3D representations. Guided by the sophisticated teacher model Point-M2AE, PMT-MAE employs a distillation strategy that includes feature distillation during pre-training and logit distillation during fine-tuning, ensuring effective knowledge transfer. On the ModelNet40 classification task, achieving an accuracy of 93.6\% without employing voting strategy, PMT-MAE surpasses the baseline Point-MAE (93.2\%) and the teacher Point-M2AE (93.4\%), underscoring its ability to learn discriminative 3D point cloud representations. Additionally, this framework demonstrates high efficiency, requiring only 40 epochs for both pre-training and fine-tuning. PMT-MAE's effectiveness and efficiency render it well-suited for scenarios with limited computational resources, positioning it as a promising solution for practical point cloud analysis.
Electronics and Telecommunications Research Institute (ETRI)（电子与电信研究所）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；AITRICS
Masked image modeling (MIM) has become a popular strategy for self-supervised learning~(SSL) of visual representations with Vision Transformers. A representative MIM model, the masked auto-encoder (MAE), randomly masks a subset of image patches and reconstructs the masked patches given the unmasked patches. Concurrently, many recent works in self-supervised learning utilize the student/teacher paradigm which provides the student with an additional target based on the output of a teacher composed of an exponential moving average (EMA) of previous students. Although common, relatively little is …
Contrastive Learning and Masked Image Modelling have demonstrated exceptional performance on self-supervised representation learning, where Momentum Contrast (i.e., MoCo) and Masked AutoEncoder (i.e., MAE) are the state-of-the-art, respectively. In this work, we propose MOMA to distill from pre-trained MoCo and MAE in a self-supervised manner to collaborate the knowledge from both paradigms. We introduce three different mechanisms of knowledge transfer in the propsoed MOMA framework. : (1) Distill pre-trained MoCo to MAE. (2) Distill pre-trained MAE to MoCo (3) Distill pre-trained MoCo and MAE…
Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学学院，柏林技术大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
I Have a Stream: Making Self-Supervised Learning Work on Continuous Video
Faculty of Electrical Engineering and Computing, University of Zagreb（萨格勒布大学电气工程与计算学院）；Fundamental AI Lab, University of Technology Nuremberg（纽伦堡工业大学基础人工智能实验室）
Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging
Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wake Forest University School of Medicine（维克森林大学医学院）；Virginia Tech-Wake Forest University School of Biomedical Engineering and Sciences（弗吉尼亚理工大学-维克森林大学生物医学工程与科学学院）；University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算机学院）；NIHR Biomedical Research Centre, Moorfields Eye Hospital NHS Foundation Trust（NIHR生物医学研究中心，摩尔菲尔兹眼科医院NHS基金会信托）
Evaluating self-supervised echocardiographic representations across downstream extraction strategies for left-ventricular segmentation and ejection fraction estimation
organization= Centre for AI, Data Science \& AI, BioPharma R\&D, AstraZeneca , addressline= Biomedical Campus, 1 Francis Crick Ave, Trumpington , city= Cambridge , postcode= CB2 0AA , country= UK
Point-SRA: Self-Representation Alignment for 3D Representation Learning
School of Electronics and Information, Xi’an Polytechnic University（西安理工大学电子与信息学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）；School of Computer Science, Xi’an Polytechnic University（西安理工大学计算机科学学院）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）；Electronic Information School, Wuhan University（电子信息学院，武汉大学）
A Closer Look at Self-Supervised Lightweight Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Megvii Research（旷视研究院）；Wenzhou University（温州大学）；ShanghaiTech University（上海科技大学）
Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning
Faculty of Automatic Control and Computer Science, Politehnica University of Bucharest（自动化控制与计算机科学系，布加勒斯特理工大学）；Institute of Mathematics of the Romanian Academy（罗马尼亚科学院数学研究所）
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology Shenzhen Research Institute（香港科技大学深圳研究院）
Left-ventricular ejection fraction (LVEF) is an important indicator of heart failure. Existing methods for LVEF estimation from video require large amounts of annotated data to achieve high performance, e.g. using 10,030 labeled echocardiogram videos to achieve mean absolute error (MAE) of 4.10. Labeling these videos is time-consuming however and limits potential downstream applications to other heart diseases. This paper presents the first semi-supervised approach for LVEF prediction. Unlike general video prediction tasks, LVEF prediction is specifically related to changes in the left ventric…
Non-Contact Physiological Monitoring in Pediatric Intensive Care Units via Adaptive Masking and Self-Supervised Learning
Biomedical Information Processing Laboratory, École de Technologie Supérieure, University of Quebec（生物医学信息处理实验室，高等技术学院，魁北克大学）；Pediatric Intensive Care Unit, CHU Sainte-Justine（儿童重症监护单元，圣朱斯特医院）
