This study investigates the impact of self-supervised pretraining of 3D semantic segmentation models on a large-scale, domain-specific dataset. We introduce BRAINS-45K, a dataset of 44,756 brain MRI volumes from public sources, the largest public dataset available, and revisit a number of design choices for pretraining modern segmentation architectures by simplifying and optimizing state-of-the-art methods, and combining them with a novel augmentation strategy. The resulting AMAES framework is based on masked-image-modeling and intensity-based augmentation reversal and balances memory usage, runtime, and finetuning performance. Using the popular U-Net and the recent MedNeXt architecture as backbones, we evaluate the effect of pretraining on three challenging downstream tasks, covering single-sequence, low-resource settings, and out-of-domain generalization. The results highlight that pretraining on the proposed dataset with AMAES significantly improves segmentation performance in the majority of evaluated cases, and that it is beneficial to pretrain the model with augmentations, despite pretraing on a large-scale dataset. Code and model checkpoints for reproducing results, as well as the BRAINS-45K dataset are available at \url{https://github.com/asbjrnmunk/amaes}.
Selfie: Self-supervised Pretraining for Image Embedding
Google Brain（谷歌大脑）
我们引入了一种名为Selfie的预训练技术，它代表SELFie监督图像嵌入。Selfie概括了BERT的掩蔽语言建模的概念（Devlin等人，2019）通过利用对比预测编码损失（Oord等人，2018年）。给定输入图像中的屏蔽补丁，我们的方法学习选择正确的补丁，从同一图像中采样的其他“干扰”补丁，以填充屏蔽位置。该分类目标回避了预测目标块的精确像素值的需要。Selfie的预训练架构包括一个卷积块网络来处理补丁，然后是一个注意力池网络来总结未屏蔽补丁的内容，然后再预测屏蔽补丁。在微调过程中，我们重复使用预训练找到的卷积权重。我们在三个基准测试（CIFAR-10，ImageNet 32 x 32和ImageNet 224 x 224）上评估了Selfie，其中包含不同数量的标记数据，从5%到100%的训练集。与相同网络的标准监督训练相比，我们的预训练方法在所有设置中对ResNet-50进行了一致的改进。值得注意的是，在ImageNet 224 x 224上，每个类有60个示例（5%），我们的方法将ResNet-50的平均准确率从35.6%提高到46.7%，绝对准确率提高了11.1个点。我们的预训练方法还通过显着降低不同运行中测试准确度的标准差，提高了ResNet-50的训练稳定性，特别是在低数据状态下。
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China（高可信软件技术教育部重点实验室（北京大学））；School of Computer Science, Peking University（北京大学计算机学院）；School of AI, Peking University（北京大学人工智能学院）；University of Hong Kong（香港大学）；School of EIC, Huazhong University of Science & Technology（华中科技大学电子信息与通信学院）；Baidu（百度）；UC Berkeley（加州大学伯克利分校）
Large Scale Masked Autoencoding for Reducing Label Requirements on SAR Data
University of Cambridge（剑桥大学）；Universidad Nacional de Colombia（哥伦比亚国立大学）；Universitat de València（瓦伦西亚大学）；University of Oxford（牛津大学）；Universidad de Antioquia（安蒂奥基亚大学）；European Space Agency（欧洲空间局）
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction
College of Electronic Science and Technology, National University of Defense Technology (NUDT)（电子科学与技术学院，国防科技大学）；Beijing Institute of Remote Sensing Information（遥感信息研究所）
Tissue-Contrastive Semi-Masked Autoencoders for Segmentation Pretraining on Chest CT
Hangzhou City University（杭州城市大学）；Southwest Hospital, Army Medical University (Third Military Medical University)（陆军军医大学（第三军医大学）西南医院）；Zhejiang University（浙江大学）
