In Computer Vision, Zero-Shot Learning (ZSL) aims at classifying unseen classes -- classes for which no matching training image exists. Most of ZSL works learn a cross-modal mapping between images and class labels for seen classes. However, the data distribution of seen and unseen classes might differ, causing a domain shift problem. Following this observation, transductive ZSL (T-ZSL) assumes that unseen classes and their associated images are known during training, but not their correspondence. As current T-ZSL approaches do not scale efficiently when the number of seen classes is high, we tackle this problem with a new model for T-ZSL based upon CycleGAN. Our model jointly (i) projects images on their seen class labels with a supervised objective and (ii) aligns unseen class labels and visual exemplars with adversarial and cycle-consistency objectives. We show the efficiency of our Cross-Modal CycleGAN model (CM-GAN) on the ImageNet T-ZSL task where we obtain state-of-the-art results. We further validate CM-GAN on a language grounding task, and on a new task that we propose: zero-shot sentence-to-image matching on MS COCO.
Plasma-CycleGAN: Plasma Biomarker-Guided MRI to PET Cross-modality Translation Using Conditional CycleGAN
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）；Basis Scottsdale（斯科茨代尔贝斯机构）；College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；Mayo Clinic Arizona（梅奥诊所亚利桑那州分院）
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Osaka University（大阪大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
CT is commonly used in orthopedic procedures. MRI is used along with CT to identify muscle structures and diagnose osteonecrosis due to its superior soft tissue contrast. However, MRI has poor contrast for bone structures. Clearly, it would be helpful if a corresponding CT were available, as bone boundaries are more clearly seen and CT has standardized (i.e., Hounsfield) units. Therefore, we aim at MR-to-CT synthesis. The CycleGAN was successfully applied to unpaired CT and MR images of the head, these images do not have as much variation of intensity pairs as do images in the pelvic region du…
The cycleGAN is becoming an influential method in medical image synthesis. However, due to a lack of direct constraints between input and synthetic images, the cycleGAN cannot guarantee structural consistency between these two images, and such consistency is of extreme importance in medical imaging. To overcome this, we propose a structure-constrained cycleGAN for brain MR-to-CT synthesis using unpaired data that defines an extra structure-consistency loss based on the modality independent neighborhood descriptor to constrain structural consistency. Additionally, we use a position-based select…
Recently, the cycle-consistent generative adversarial networks (CycleGAN) has been widely used for synthesis of multi-domain medical images. The domain-specific nonlinear deformations captured by CycleGAN make the synthesized images difficult to be used for some applications, for example, generating pseudo-CT for PET-MR attenuation correction. This paper presents a deformation-invariant CycleGAN (DicycleGAN) method using deformable convolutional layers and new cycle-consistency losses. Its robustness dealing with data that suffer from domain-specific nonlinear deformations has been evaluated t…
University of California, Davis（加州大学戴维斯分校）；University of San Francisco Data Institute（旧金山大学数据研究所）；Wicklow AI in Medicine Research Initiative（威克洛医学人工智能研究计划）；University of Texas, San Antonio Health（德克萨斯大学圣安东尼奥健康分校）
MUSE is a novel slide-free imaging technique for histological examination of tissues that can serve as an alternative to traditional histology. In order to bridge the gap between MUSE and traditional histology, we aim to convert MUSE images to resemble authentic hematoxylin- and eosin-stained (H&E) images. We evaluated four models: a non-machine-learning-based color-mapping unmixing-based tool, CycleGAN, DualGAN, and GANILLA. CycleGAN and GANILLA provided visually compelling results that appropriately transferred H&E style and preserved MUSE content. Based on training an automated critic on re…
a Division of Medical Informatics, Department of Biomedical Engineering, Link\" o ping University, Link\" o ping, Sweden b Division of Statistics；Machine Learning, Department of Computer；Information Science, Link\" o ping University, Link\" o ping, Sweden c Center for Medical Image Science；Visualization (CMIV), Link\" o ping University, Link\" o ping, Sweden
In medical imaging, a general problem is that it is costly and time consuming to collect high quality data from healthy and diseased subjects. Generative adversarial networks (GANs) is a deep learning method that has been developed for synthesizing data. GANs can thereby be used to generate more realistic training data, to improve classification performance of machine learning algorithms. Another application of GANs is image-to-image translations, e.g. generating magnetic resonance (MR) images from computed tomography (CT) images, which can be used to obtain multimodal datasets from a single m…
Unpaired image-to-image translation refers to learning inter-image-domain mapping in an unsupervised manner. Existing methods often learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen out-of-distribution (OOD) patterns at test time. To address this limitation, we propose a novel probabilistic method called Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed…
Department of Biomedical Engineering, Yale University（耶鲁大学生物医学工程系）；Department of Radiology & Biomedical Imaging, Yale School of Medicine（耶鲁大学医学院放射科与生物医学成像系）；Department of Electrical Engineering, Yale University（耶鲁大学电气工程系）；Department of Statistics & Data Science, Yale University（耶鲁大学统计与数据科学系）；Yale School of Medicine（耶鲁大学医学院）
Domain Adaptation (DA) has the potential to greatly help the generalization of deep learning models. However, the current literature usually assumes to transfer the knowledge from the source domain to a specific known target domain. Domain Agnostic Learning (DAL) proposes a new task of transferring knowledge from the source domain to data from multiple heterogeneous target domains. In this work, we propose the Domain-Agnostic Learning framework with Anatomy-Consistent Embedding (DALACE) that works on both domain-transfer and task-transfer to learn a disentangled representation, aiming to not o…
Ultrasound is the second most used modality in medical imaging. It is cost effective, hazardless, portable and implemented routinely in numerous clinical procedures. Nonetheless, image quality is characterized by granulated appearance, poor SNR and speckle noise. Specific for malignant tumors, the margins are blurred and indistinct. Thus, there is a great need for improving ultrasound image quality. We hypothesize that this can be achieved, using neural networks, by translation into a more realistic display which mimics an anatomical cut through the tissue. In order to achieve this goal, the p…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Stony Brook University（石溪大学）
Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale…
Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras
Laboratory for Intelligent & Safe Automobiles (LISA)（智能与安全汽车实验室（LISA））；University of California San Diego（加利福尼亚大学圣迭戈分校）；Visual Analysis and Perception Lab（视觉分析与感知实验室）；Aalborg Universitet（奥尔堡大学）
A lack of generalizability is one key limitation of deep learning based segmentation. Typically, one manually labels new training images when segmenting organs in different imaging modalities or segmenting abnormal organs from distinct disease cohorts. The manual efforts can be alleviated if one is able to reuse manual labels from one modality (e.g., MRI) to train a segmentation network for a new modality (e.g., CT). Previously, two stage methods have been proposed to use cycle generative adversarial networks (CycleGAN) to synthesize training images for a target modality. Then, these efforts t…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Siemens Healthineers（西门子医疗）；Rutgers University（罗格斯大学）
Recently image-to-image translation has attracted significant interests in the literature, starting from the successful use of the generative adversarial network (GAN), to the introduction of cyclic constraint, to extensions to multiple domains. However, in existing approaches, there is no guarantee that the mapping between two image domains is unique or one-to-one. Here we propose a self-inverse network learning approach for unpaired image-to-image translation. Building on top of CycleGAN, we learn a self-inverse function by simply augmenting the training samples by swapping inputs and output…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Incyte Corporation（Incyte公司）
A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality with…
Wuhan University（武汉大学）；Agency for Science, Technology and Research（科技研究局）
VIS-NIR face recognition remains a challenging task due to the distinction between spectral components of two modalities and insufficient paired training data. Inspired by the CycleGAN, this paper presents a method aiming to translate VIS face images into fake NIR images whose distributions are intended to approximate those of true NIR images, which is achieved by proposing a new facial feature embedded CycleGAN. Firstly, to learn the particular feature of NIR domain while preserving common facial representation between VIS and NIR domains, we employ a general facial feature extractor (FFE) to…
University of Bern（伯尔尼大学）；Inselspital（Inselspital医院（伯尔尼大学医院））；University Hospital of Fribourg（弗里堡大学医院）
Unsupervised domain adaptation (UDA) for cross-modality medical image segmentation has shown great progress by domain-invariant feature learning or image appearance translation. Adapted feature learning usually cannot detect domain shifts at the pixel level and is not able to achieve good results in dense semantic segmentation tasks. Image appearance translation, e.g. CycleGAN, translates images into different styles with good appearance, despite its population, its semantic consistency is hardly to maintain and results in poor cross-modality segmentation. In this paper, we propose intra- and …
Normandie Univ（诺曼底大学）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；INSA Rouen（鲁昂国立应用科学学院）；University of Burgundy（勃艮第大学）；UBFC（勃艮第-弗朗什孔泰大学）
Polarimetric imaging, along with deep learning, has shown improved performances on different tasks including scene analysis. However, its robustness may be questioned because of the small size of the training datasets. Though the issue could be solved by data augmentation, polarization modalities are subject to physical feasibility constraints unaddressed by classical data augmentation techniques. To address this issue, we propose to use CycleGAN, an image translation technique based on deep generative models that solely relies on unpaired data, to transfer large labeled road scene datasets to…
Generative adversarial networks (GANs) are unsupervised Deep Learning approach in the computer vision community which has gained significant attention from the last few years in identifying the internal structure of multimodal medical imaging data. The adversarial network simultaneously generates realistic medical images and corresponding annotations, which proven to be useful in many cases such as image augmentation, image registration, medical image generation, image reconstruction, and image-to-image translation. These properties bring the attention of the researcher in the field of medical…
Department of Biomedical Engineering, Yale University（耶鲁大学生物医学工程系）；Department of Radiology & Biomedical Imaging, Yale School of Medicine（耶鲁大学医学院放射科与生物医学成像系）；Department of Electrical Engineering, Yale University（耶鲁大学电气工程系）；Department of Statistics & Data Science, Yale University（耶鲁大学统计与数据科学系）；Yale School of Medicine（耶鲁大学医学院）；Yale University（耶鲁大学）
A deep learning model trained on some labeled data from a certain source domain generally performs poorly on data from different target domains due to domain shifts. Unsupervised domain adaptation methods address this problem by alleviating the domain shift between the labeled source data and the unlabeled target data. In this work, we achieve cross-modality domain adaptation, i.e. between CT and MRI images, via disentangled representations. Compared to learning a one-to-one mapping as the state-of-art CycleGAN, our model recovers a many-to-many mapping between domains to capture the complex c…
Medical University of Vienna（维也纳医科大学）；Medical University Vienna（维也纳医科大学）
Optical coherence tomography (OCT) has become the most important imaging modality in ophthalmology. A substantial amount of research has recently been devoted to the development of machine learning (ML) models for the identification and quantification of pathological features in OCT images. Among the several sources of variability the ML models have to deal with, a major factor is the acquisition device, which can limit the ML model's generalizability. In this paper, we propose to reduce the image variability across different OCT devices (Spectralis and Cirrus) by using CycleGAN, an unsupervis…
Are Virtual DES Images a Valid Alternative to the Real Ones?
Faculdade Ciências da Saúde, Universidade da Beira Interior and Unidade Local de Saúde do Oeste（健康科学学院，贝拉内里大学及西部本地卫生单位）；NOVA LINCS, Universidade da Beira Interior（NOVA LINCS，贝拉内里大学）；Escola Superior de Tecnologia da Saúde de Lisboa, Instituto Politécnico de Lisboa（里斯本健康技术学院，里斯本技术学院）
K. N. Toosi University of Technology（卡纳齐·图西科技大学）；Ferdowsi University of Mashhad（菲尔多西大学）；University of La Rioja（拉里奥哈大学）；Indian Institute of Technology Indore（印度理工学院印多尔分校）；Universidad de Granada（格拉纳达大学）
The outbreak of the corona virus disease (COVID-19) has changed the lives of most people on Earth. Given the high prevalence of this disease, its correct diagnosis in order to quarantine patients is of the utmost importance in steps of fighting this pandemic. Among the various modalities used for diagnosis, medical imaging, especially computed tomography (CT) imaging, has been the focus of many previous studies due to its accuracy and availability. In addition, automation of diagnostic methods can be of great help to physicians. In this paper, a method based on pre-trained deep neural networks…
Shenzhen University（深圳大学）；Islamic Azad University（伊斯兰阿扎德大学）
Magnetic resonance imaging (MRI) provides varying tissue contrast images of internal organs based on a strong magnetic field. Despite the non-invasive advantage of MRI in frequent imaging, the low contrast MR images in the target area make tissue segmentation a challenging problem. This paper demonstrates the potential benefits of image-to-image translation techniques to generate synthetic high tissue contrast (HTC) images. Notably, we adopt a new cycle generative adversarial network (CycleGAN) with an attention mechanism to increase the contrast within underlying tissues. The attention block,…
Zero-shot learning (ZSL) endows the computer vision system with the inferential capability to recognize instances of a new category that has never seen before. Two fundamental challenges in it are visual-semantic embedding and domain adaptation in cross-modality learning and unseen class prediction steps, respectively. To address both challenges, this paper presents two corresponding methods named Adaptive STructural Embedding (ASTE) and Self-PAsed Selective Strategy (SPASS), respectively. Specifically, ASTE formulates the visualsemantic interactions in a latent structural SVM framework to ada…
Nagoya University（名古屋大学）；Nagoya University Graduate School of Medicine（名古屋大学医学研究科）
This paper proposes a novel, unsupervised super-resolution (SR) approach for performing the SR of a clinical CT into the resolution level of a micro CT ($μ$CT). The precise non-invasive diagnosis of lung cancer typically utilizes clinical CT data. Due to the resolution limitations of clinical CT (about $0.5 \times 0.5 \times 0.5$ mm$^3$), it is difficult to obtain enough pathological information such as the invasion area at alveoli level. On the other hand, $μ$CT scanning allows the acquisition of volumes of lung specimens with much higher resolution ($50 \times 50 \times 50 μ{\rm m}^3$ or hig…
MCR-VQGAN: A Scalable and Cost-Effective Tau PET Synthesis Approach for Alzheimer's Disease Imaging
Department of Biomedical Engineering, Wake Forest University School of Medicine（生物医学工程系，威克森林大学医学院）；Department of Radiology, Wake Forest School of Medicine（放射学系，威克森林医学院）；Department of Radiology and Biomedical Imaging, Yale School of Medicine（放射学与生物医学成像系，耶鲁医学院）
Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
We address the problem of image translation between domains or modalities for which no direct paired data is available (i.e. zero-pair translation). We propose mix and match networks, based on multiple encoders and decoders aligned in such a way that other encoder-decoder pairs can be composed at test time to perform unseen image translation tasks between domains or modalities for which explicit paired samples were not seen during training. We study the impact of autoencoders, side information and losses in improving the alignment and transferability of trained pairwise translation models to u…
Philips Research Paris（飞利浦巴黎研究院）；Institut Imagine, Université Paris Cité（巴黎西岱大学想象研究所）；Université Paris Cité（巴黎西岱大学）；Hôpital Necker Enfants-Malades, APHP（巴黎公共援助医院集团内克儿童病院）；Sorbonne Université（索邦大学）；CNRS, LIP6（法国国家科学研究中心LIP6实验室）；Technology Innovation Institute（技术创新研究所）
Anatomical structures such as blood vessels in contrast-enhanced CT (ceCT) images can be challenging to segment due to the variability in contrast medium diffusion. The combined use of ceCT and contrast-free (CT) CT images can improve the segmentation performances, but at the cost of a double radiation exposure. To limit the radiation dose, generative models could be used to synthesize one modality, instead of acquiring it. The CycleGAN approach has recently attracted particular attention because it alleviates the need for paired data that are difficult to obtain. Despite the great performance…
Cross Modality Image Translation In Medical Imaging Using Generative Frameworks
Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与介入部门、放射物理、生物医学工程，乌梅大学）；Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，工程部门，罗马生物医学学院）；Vita-Salute San Raffaele University（维塔-萨拉特·桑拉法埃莱大学）；Department of Medicine, Surgery and Dentistry, University of Salerno（医学、外科和牙科部门，萨勒诺大学）；Division of Diagnostic and Interventional Neuroradiology, Department of Radiology, University Hospital Basel（诊断和介入神经放射学部门，放射学部门，巴塞尔大学医院）；Department of Pediatric Radiology, University Children’s Hospital Basel（儿科放射学部门，巴塞尔儿童医院）；Department of Life Science and Public Health, Università Cattolica del Sacro Cuore（生命科学与公共健康部门，圣心大学）；Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A·马里诺斯生物医学成像中心）；Artificial Intelligence and Translational Imaging (ATI) Lab, Department of Radiology, School of Medicine, University of Crete（人工智能与转化成像（ATI）实验室，放射学部门，医学院，克里特大学）；Division of Radiology, Department of Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institute（放射学部门，临床科学、介入和科技（CLINTEC）部门，卡罗林斯卡研究所）；Columbia University Medical Center（哥伦比亚大学医学中心）；Department of Diagnostics and intervention, Diagnostic radiology, Umeå University（诊断与介入部门，诊断放射学，乌梅大学）
Michigan State University（密歇根州立大学）；University of Michigan（密歇根大学）
In this paper, we study the problem of recognizing compositional attribute-object concepts within the zero-shot learning (ZSL) framework. We propose an episode-based cross-attention (EpiCA) network which combines merits of cross-attention mechanism and episode-based training strategy to recognize novel compositional concepts. Firstly, EpiCA bases on cross-attention to correlate concept-visual information and utilizes the gated pooling layer to build contextualized representations for both images and concepts. The updated representations are used for a more in-depth multi-modal relevance calcul…
University of Tokyo（东京大学）；Kyushu University（九州大学）；University of Tokyo Hospital（东京大学附属医院）；Hokkaido University（北海道大学）；Tokushima University（德岛大学）
In recent years, deep-learning-based image processing has emerged as a valuable tool for medical imaging owing to its high performance. However, the quality of deep-learning-based methods heavily relies on the amount of training data; the high cost of acquiring a large dataset is a limitation to their utilization in medical fields. Herein, based on deep learning, we developed a computed tomography (CT) modality conversion method requiring only a few unsupervised images. The proposed method is based on CycleGAN with several extensions tailored for CT images, which aims at preserving the structu…
Cyclic 2.5D Perceptual Loss for Cross-Modal 3D Medical Image Synthesis: T1w MRI to Tau PET
Department of Artificial Intelligence, Hanyang University, Seoul, South Korea（人工智能系，翰阳大学，首尔，韩国）；Department of Electronic Engineering, Hanyang University, Seoul, South Korea（电子工程系，翰阳大学，首尔，韩国）；Division of Computer Engineering, Hankuk University of Foreign Studies, Yongin, South Korea（计算机工程系，韩国外语大学， Yongin，韩国）；Division of AI Data Convergence, Hankuk University of Foreign Studies, Yongin, South Korea（AI数据融合系，韩国外语大学， Yongin，韩国）；Division of Language & AI, Hankuk University of Foreign Studies, Seoul, South Korea（语言与AI系，韩国外语大学，首尔，韩国）
Curvilinear Structure-preserving Unpaired Cross-domain Medical Image Translation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；Singapore Eye Research Institute, Singapore National Eye Centre, Singapore（新加坡眼科学研究所，新加坡国家眼科中心）；School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China（武汉科技大学计算机科学与技术学院）；Ophthalmology & Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore（眼科与视觉科学学术临床计划，新加坡国立大学医学院）；Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland（分子与临床眼科学研究所，瑞士巴塞尔）；School of Chemical and Biomedical Engineering, Nanyang Technological University, Singapore（南洋理工大学化学与生物医学工程学院）；Centre for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria（医学物理与生物医学工程中心，维也纳医科大学，维也纳，奥地利）；Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria（临床药理学系，维也纳医科大学，维也纳，奥地利）；Rothschild Foundation Hospital, Paris, France（罗特希尔德基金会医院，法国巴黎）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究所以（I 2 R），新加坡A*STAR）
Recent advances in deep learning for medical image segmentation demonstrate expert-level accuracy. However, in clinically realistic environments, such methods have marginal performance due to differences in image domains, including different imaging protocols, device vendors and patient populations. Here we consider the problem of domain generalization, when a model is trained once, and its performance generalizes to unseen domains. Intuitively, within a specific medical imaging modality the domain differences are smaller relative to natural images domain variability. We rethink data augmentat…
