Inpainting has recently been proposed as a successful deep learning technique for unsupervised medical image model discovery. The masks used for inpainting are generally independent of the dataset and are not tailored to perform on different given classes of anatomy. In this work, we introduce a method for generating shape-aware masks for inpainting, which aims at learning the statistical shape prior. We hypothesize that although the variation of masks improves the generalizability of inpainting models, the shape of the masks should follow the topology of the organs of interest. Hence, we propose an unsupervised guided masking approach based on an off-the-shelf inpainting model and a superpixel over-segmentation algorithm to generate a wide range of shape-dependent masks. Experimental results on abdominal MR image reconstruction show the superiority of our proposed masking method over standard methods using square-shaped or dataset of irregular shape masks.
Local deformations in medical modalities are common phenomena due to a multitude of factors such as metallic implants or limited field of views in magnetic resonance imaging (MRI). Completion of the missing or distorted regions is of special interest for automatic image analysis frameworks to enhance post-processing tasks such as segmentation or classification. In this work, we propose a new generative framework for medical image inpainting, titled ipA-MedGAN. It bypasses the limitations of previous frameworks by enabling inpainting of arbitrary shaped regions without a prior localization of t…
Numerous factors could lead to partial deteriorations of medical images. For example, metallic implants will lead to localized perturbations in MRI scans. This will affect further post-processing tasks such as attenuation correction in PET/MRI or radiation therapy planning. In this work, we propose the inpainting of medical images via Generative Adversarial Networks (GANs). The proposed framework incorporates two patch-based discriminator networks with additional style and perceptual losses for the inpainting of missing information in realistically detailed and contextually consistent manner. …
Medical imaging is an important tool for the diagnosis and the evaluation of an aortic dissection (AD); a serious condition of the aorta, which could lead to a life-threatening aortic rupture. AD patients need life-long medical monitoring of the aortic enlargement and of the disease progression, subsequent to the diagnosis of the aortic dissection. Since there is a lack of 'healthy-dissected' image pairs from medical studies, the application of inpainting techniques offers an alternative source for generating them by doing a virtual regression from dissected aortae to healthy aortae; an indire…
Technical University of Munich（慕尼黑工业大学）；Sharif University of Technology（谢里夫理工大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
The need for a large amount of labeled data in the supervised setting has led recent studies to utilize self-supervised learning to pre-train deep neural networks using unlabeled data. Many self-supervised training strategies have been investigated especially for medical datasets to leverage the information available in the much fewer unlabeled data. One of the fundamental strategies in image-based self-supervision is context prediction. In this approach, a model is trained to reconstruct the contents of an arbitrary missing region of an image based on its surroundings. However, the existing m…
MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）；Shanghai Medical Image Insights（上海医学影像洞察）；Harvard University（哈佛大学）；University of Waterloo（滑铁卢大学）
HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）
Stanford University（斯坦福大学）；Wake Forest School of Medicine（维克森林医学院）
Although supervised learning has enabled high performance for image segmentation, it requires a large amount of labeled training data, which can be difficult to obtain in the medical imaging field. Self-supervised learning (SSL) methods involving pretext tasks have shown promise in overcoming this requirement by first pretraining models using unlabeled data. In this work, we evaluate the efficacy of two SSL methods (inpainting-based pretext tasks of context prediction and context restoration) for CT and MRI image segmentation in label-limited scenarios, and investigate the effect of implementa…
Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；Sir Run Run Shaw Hospital（邵逸夫医院）；Lenovo Research（联想研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang Normal University（浙江师范大学）
Clinical applicability of automated decision support systems depends on a robust, well-understood classification interpretation. Artificial neural networks while achieving class-leading scores fall short in this regard. Therefore, numerous approaches have been proposed that map a salient region of an image to a diagnostic classification. Utilizing heuristic methodology, like blurring and noise, they tend to produce diffuse, sometimes misleading results, hindering their general adoption. In this work we overcome these issues by presenting a model agnostic saliency mapping framework tailored to …
Mask of truth: model sensitivity to unexpected regions of medical images
IT University of Copenhagen（丹麦哥本哈根技术大学）；Copenhagen University Hospital, Herlev and Gentofte（哥本哈根大学医院，赫尔勒夫和琴托夫）；Radiological AI Testcenter（放射学AI测试中心）；Department of Radiology, Zealand University Hospital（放射科， Zealand大学医院）
Edinburgh Napier University（爱丁堡龙比亚大学）；University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
Several imaging applications (vessels, retina, plant roots, road networks from satellites) require the accurate segmentation of thin structures for subsequent analysis. Discontinuities (gaps) in the extracted foreground may hinder down-stream image-based analysis of biomarkers, organ structure and topology. In this paper, we propose a general post-processing technique to recover such gaps in large-scale segmentation masks. We cast this problem as a blind inpainting task, where the regions of missing lines in the segmentation masks are not known to the algorithm, which we solve with an adversar…
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sun Yat-sen University（中山大学）；Pazhou Lab（琶洲实验室）
Medical vision-and-language pre-training provides a feasible solution to extract effective vision-and-language representations from medical images and texts. However, few studies have been dedicated to this field to facilitate medical vision-and-language understanding. In this paper, we propose a self-supervised learning paradigm with multi-modal masked autoencoders (M$^3$AE), which learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts. There are three key designs to make this simple approach work. First, considering the different …
Deep learning based image denoising methods have been recently popular due to their improved performance. Traditionally, these methods are trained in a supervised manner, requiring a set of noisy input and clean target image pairs. More recently, self-supervised approaches have been proposed to learn denoising from only noisy images. These methods assume that noise across pixels is statistically independent, and the underlying image pixels show spatial correlations across neighborhoods. These methods rely on a masking approach that divides the image pixels into two disjoint sets, where one is …
Multitask Brain Tumor Inpainting with Diffusion Models: A Methodological Report
Mayo Clinic Artificial Intelligence Laboratory, Mayo Clinic（梅奥诊所人工智能实验室，梅奥诊所）；Orthopedic Surgery Artificial Intelligence Laboratory, Mayo Clinic（梅奥诊所骨科手术人工智能实验室）
KAIST（韩国科学技术院）；Asan Medical Center（峨山医学中心）；University of Ulsan College of Medicine（蔚山大学医学院）
Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performanc…
VRVis Zentrum für Virtual Reality und Visualisierung Forschungs-GmbH（VRVis虚拟现实与可视化研究有限公司）
The success of machine learning methods for computer vision tasks has driven a surge in computer assisted prediction for medicine and biology. Based on a data-driven relationship between input image and pathological classification, these predictors deliver unprecedented accuracy. Yet, the numerous approaches trying to explain the causality of this learned relationship have fallen short: time constraints, coarse, diffuse and at times misleading results, caused by the employment of heuristic techniques like Gaussian noise and blurring, have hindered their clinical adoption. In this work, we disc…
Recently, masked image modeling (MIM) has gained considerable attention due to its capacity to learn from vast amounts of unlabeled data and has been demonstrated to be effective on a wide variety of vision tasks involving natural images. Meanwhile, the potential of self-supervised learning in modeling 3D medical images is anticipated to be immense due to the high quantities of unlabeled images, and the expense and difficulty of quality labels. However, MIM's applicability to medical images remains uncertain. In this paper, we demonstrate that masked image modeling approaches can also advance …
MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（科学技术大学生物医学工程学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（科学技术大学苏州市先进研究所）；Center for Medical Imaging, Robotics, and Analytic Computing & LEarning (MIRACLE), Suzhou Institute for Advanced Research, USTC（医学影像、机器人与分析计算与学习中心）；Department of Automation, Shanghai Jiao Tong University（上海交通大学自动化系）；Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）；Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医学机器人研究所）；State Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China（科学技术大学精密与智能化学国家重点实验室）
Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）
Masked Image Modelling (MIM) has been shown to be an efficient self-supervised learning (SSL) pre-training paradigm when paired with transformer architectures and in the presence of a large amount of unlabelled natural images. The combination of the difficulties in accessing and obtaining large amounts of labeled data and the availability of unlabelled data in the medical imaging domain makes MIM an interesting approach to advance deep learning (DL) applications based on 3D medical imaging data. Nevertheless, SSL and, in particular, MIM applications with medical imaging data are rather scarce …
L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction
LaTIM UMR 1101, Inserm（法国国家健康与医学研究院UMR 1101拉蒂姆研究所）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Lariboisière Hospital, AP-HP（巴黎公立医院集团拉里布瓦西耶尔医院）；University Hospital of Brest（布雷斯特大学医院）；INSERM U1227 Lymphocytes B et Autoimmunite (LBAI)（法国国家健康与医学研究院UMR 1227 B淋巴细胞与自身免疫研究所）
From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction
Center for Artificial Intelligence, BCVI, Northwestern Medicine（西北医学BCVI人工智能中心）；Department of Electrical and Computer Engineering, Northwestern University（西北大学电气与计算机工程系）；Department of Surgery, Northwestern University（西北大学外科学系）；Xtasis Inc.（Xtasis公司）；Medtronic（美敦力公司）；Departments of Radiology and Clinical Neurosciences, University of Calgary（卡尔加里大学放射学与临床神经科学系）
Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（伊萨尔河右岸医院）；University of Edinburgh（爱丁堡大学）
Deep learning-based segmentation methods are vulnerable to unforeseen data distribution shifts during deployment, e.g. change of image appearances or contrasts caused by different scanners, unexpected imaging artifacts etc. In this paper, we present a cooperative framework for training image segmentation models and a latent space augmentation method for generating hard examples. Both contributions improve model generalization and robustness with limited data. The cooperative training framework consists of a fast-thinking network (FTN) and a slow-thinking network (STN). The FTN learns decoupled…
