In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones. We propose to distill knowledge from a trained generative model into standard image backbones that have been well engineered for specific perception tasks. We investigate two types of knowledge distillation: 1) distilling learned generative features onto target image backbones as an alternative to pretraining these backbones on large labeled datasets such as ImageNet, and 2) distilling labels obtained from generative networks with task heads onto logits of target backbones. We perform extensive analyses on multiple generative models, dense prediction benchmarks, and several pre-training regimes. We empirically find that our DreamTeacher significantly outperforms existing self-supervised representation learning approaches across the board. Unsupervised ImageNet pre-training with DreamTeacher leads to significant improvements over ImageNet classification pre-training on downstream datasets, showcasing generative models, and diffusion generative models specifically, as a promising approach to representation learning on large, diverse datasets without requiring manual annotation.
DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining
Fordham University（福特汉姆大学）；Rutgers University（罗格斯大学）；Pioneer Centre for AI（先锋人工智能中心）；Cold Spring Harbor Laboratory（冷泉港实验室）；Brown University（布朗大学）；University of Copenhagen（哥本哈根大学）
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology（澳大利亚昆士兰科技大学电气工程与机器人学院SAIVT组）；CSIRO Robotics, CSIRO（澳大利亚联邦科学与工业研究组织机器人部）
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
New York University（纽约大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland（马里兰大学）；Georgia Institute of Technology（佐治亚理工学院）；Inria（法国国家信息与自动化研究所）；Meta AI Research（元人工智能研究部门）
本文提出Battle of the Backbones (BoB)基准，跨多种计算机视觉任务比较1500+次训练运行的预训练模型，发现监督预训练CNN在多数任务表现最佳，而SSL骨干网络在公平比较中极具竞争力。
Large-scale pretraining of visual representations has led to state-of-the-art performance on a range of benchmark computer vision tasks, yet the benefits of these techniques at extreme scale in complex production systems has been relatively unexplored. We consider the case of a popular visual discovery product, where these representations are trained with multi-task learning, from use-case specific visual understanding (e.g. skin tone classification) to general representation learning for all visual content (e.g. embeddings for retrieval). In this work, we describe how we (1) generate a datase…
Proposal-Contrastive Pretraining for Object Detection from Fewer Data
Université Paris-Saclay（巴黎-萨克雷大学）；CEA（法国原子能委员会）；Université Jean Monnet Saint-Etienne（圣艾蒂安让·莫内大学）；CNRS（法国国家科学研究中心）；Laboratoire Hubert Curien UMR 5516（于贝尔·居里安实验室）；Institut Universitaire de France (IUF)（法国大学研究院）
Self-Conditioned Denoising for Atomistic Representation Learning
Department of Chemistry, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, 02139, MA, USA（麻省理工学院化学系）；Department of Materials Science and Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, 02139, MA, USA（麻省理工学院材料科学与工程系）
REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning
Shien-Ming Wu School of Intelligent Engineering（申铭武智能工程学院）；School of Future Technology（未来技术学院）；Department of Electrical and Electronic Engineering（电气与电子工程系）
MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation
German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing（德国癌症研究中心（DKFZ）海德堡，医学影像计算部）；Faculty of Mathematics and Computer Science, Heidelberg University（海德堡大学数学与计算机科学学院）；HIDSS4Health - Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany（HIDSS4Health - 欧洲信息与数据科学健康学校，卡尔斯鲁厄/海德堡，德国）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学学院）；National Center for Tumor Diseases (NCT), Heidelberg, Germany（肿瘤国家研究中心（NCT），海德堡，德国）；Helmholtz Imaging, German Cancer Research Center, Germany（海德堡成像，德国癌症研究中心，德国）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（模式分析与学习组，放射肿瘤学部，海德堡大学医院）
Vision Transformers versus convolutional neural networks for fine-grained orchid genus identification in a species-rich, data-poor flora: a controlled benchmark on the Orchidaceae of New Guinea
James Cook University（詹姆斯库克大学）；Southwest Papua Natural Resources Conservation Agency, Ministry of Forestry, Indonesia（印度尼西亚林业部西南巴布亚自然资源保护局）；National Research and Innovation Agency (BRIN), Republic of Indonesia（印度尼西亚共和国国家研究与创新局（BRIN））；Royal Botanic Gardens, Kew（英国皇家植物园邱园）；Australian Tropical Herbarium（澳大利亚热带植物标本馆）；Commonwealth Industrial and Scientific Research Organisation (CSIRO)（澳大利亚联邦科学与工业研究组织（CSIRO））；Australian National Herbarium（澳大利亚国家植物标本馆）
Most self-supervised video representation learning approaches focus on action recognition. In contrast, in this paper we focus on self-supervised video learning for movie understanding and propose a novel hierarchical self-supervised pretraining strategy that separately pretrains each level of our hierarchical movie understanding model (based on [37]). Specifically, we propose to pretrain the low-level video backbone using a contrastive learning objective, while pretrain the higher-level video contextualizer using an event mask prediction task, which enables the usage of different data sources…
STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy
organization= Institute of Space Sciences；Astronomy , addressline= University of Malta , city= Msida , postcode= MSD2080 , country= Malta；organization= School of Physics；Astronomy, Royal Observatory , addressline= University of Edinburgh , city= Edinburgh , country= United Kingdom
Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Computing, Department of Computer Science, National University of Singapore（新加坡国立大学计算机学院）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能MOE重点实验室）；College of Artificial Intelligence, Nanjing Agricultural University（南京农业大学人工智能学院）；Key Laboratory of Livestock Farming Equipment, Ministry of Agriculture and Rural Affairs, Nanjing Agricultural University（南京农业大学农业机械化关键实验室）
Mitigating Overfitting in Medical Imaging: Self-Supervised Pretraining vs. ImageNet Transfer Learning for Dermatological Diagnosis
Dpt. of Signal Theory and Communications University of Seville（信号理论与通信系塞维利亚大学）；Department of Dermatology Hospital Universitario Virgen Macarena（皮肤科部门塞维利亚大学医院）；Department of Dermatology Hospital Quirón Salud Sevilla（皮肤科部门Quirón Salud Sevilla医院）
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Department of Laboratory Medicine & Pathology, University of Washington（华盛顿大学检验医学与病理学系）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Electrical Engineering & Computer Science, Oregon State University（俄勒冈州立大学电气工程与计算机科学系）
Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
Department of Medical Physics, Memorial Sloan Kettering Cancer Center（医学物理系，纪念斯隆凯特琳癌症中心）；Department of Radiology, Memorial Sloan Kettering Cancer Center（放射学系，纪念斯隆凯特琳癌症中心）；Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center（放射肿瘤学系，纪念斯隆凯特琳癌症中心）；Department of Surgery, Memorial Sloan Kettering Cancer Center（外科系，纪念斯隆凯特琳癌症中心）；Department of Medicine, Memorial Sloan Kettering Cancer Center（医学系，纪念斯隆凯特琳癌症中心）
University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）；Harbin Institute of Technology（哈尔滨工业大学）
The target of 2D human pose estimation is to locate the keypoints of body parts from input 2D images. State-of-the-art methods for pose estimation usually construct pixel-wise heatmaps from keypoints as labels for learning convolution neural networks, which are usually initialized randomly or using classification models on ImageNet as their backbones. We note that 2D pose estimation task is highly dependent on the contextual relationship between image patches, thus we introduce a self-supervised method for pretraining 2D pose estimation networks. Specifically, we propose Heatmap-Style Jigsaw P…
