Medical students and radiology trainees typically view thousands of images in order to "train their eye" to detect the subtle visual patterns necessary for diagnosis. Nevertheless, infrastructural and legal constraints often make it difficult to access and quickly query an abundance of images with a user-specified feature set. In this paper, we use a conditional generative adversarial network (GAN) to synthesize $1024\times1024$ pixel pelvic radiographs that can be queried with conditioning on fracture status. We demonstrate that the conditional GAN learns features that distinguish fractures from non-fractures by training a convolutional neural network exclusively on images sampled from the GAN and achieving an AUC of $>0.95$ on a held-out set of real images. We conduct additional analysis of the images sampled from the GAN and describe ongoing work to validate educational efficacy.
Faculty of Medicine, American University of Beirut（贝鲁特美国大学医学院）；American University of Beirut Medical Center（贝鲁特美国大学医学中心）；Department of Diagnostic Radiology, American University of Beirut Medical Center（贝鲁特美国大学医学中心诊断放射科）
During their formative years, radiology trainees are required to interpret hundreds of mammograms per month, with the objective of becoming apt at discerning the subtle patterns differentiating benign from malignant lesions. Unfortunately, medico-legal and technical hurdles make it difficult to access and query medical images for training. In this paper we train a generative adversarial network (GAN) to synthesize 512 x 512 high-resolution mammograms. The resulting model leads to the unsupervised separation of high-level features (e.g. the standard mammography views and the nature of the breas…
RadGame: An AI-Powered Platform for Radiology Education
Harvard Medical School（哈佛医学院）；Mass General Brigham（麻省总医院）；Maastricht University（马斯特里赫特大学）；Department of Medical Imaging, King Abdulaziz Medical City, Ministry of National Guard, Riyadh, Saudi Arabia（国王阿卜杜勒-阿齐兹医疗城医学影像科，沙特阿拉伯）；National Strategic Technology Research Institute, Seoul National University Hospital（全国战略技术研究所，首尔国立大学医院）；Saint Louis University School of Medicine（圣路易斯大学医学院）；College of Medicine, King Saud bin Abdulaziz University for Health Sciences（国王萨勒曼·本·阿卜杜勒阿齐兹大学医学院）；Tufts University School of Medicine（塔夫茨大学医学院）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）
Currently there is strong interest in data-driven approaches to medical image classification. However, medical imaging data is scarce, expensive, and fraught with legal concerns regarding patient privacy. Typical consent forms only allow for patient data to be used in medical journals or education, meaning the majority of medical data is inaccessible for general public research. We propose a novel, two-stage pipeline for generating synthetic medical images from a pair of generative adversarial networks, tested in practice on retinal fundi images. We develop a hierarchical generation process to…
Automated liver segmentation from radiology scans (CT, MRI) can improve surgery and therapy planning and follow-up assessment in addition to conventional use for diagnosis and prognosis. Although convolutional neural networks (CNNs) have become the standard image segmentation tasks, more recently this has started to change towards Transformers based architectures because Transformers are taking advantage of capturing long range dependence modeling capability in signals, so called attention mechanism. In this study, we propose a new segmentation approach using a hybrid approach combining the Tr…
National Institutes of Health（美国国立卫生研究院）；Ping An Technology Co., Ltd.（平安科技有限公司）
In this work, we exploit the unsupervised domain adaptation problem for radiology image interpretation across domains. Specifically, we study how to adapt the disease recognition model from a labeled source domain to an unlabeled target domain, so as to reduce the effort of labeling each new dataset. To address the shortcoming of cross-domain, unpaired image-to-image translation methods which typically ignore class-specific semantics, we propose a task-driven, discriminatively trained, cycle-consistent generative adversarial network, termed TUNA-Net. It is able to preserve 1) low-level details…
Friedrich-Alexander Universität Erlangen-Nürnberg（弗里德里希-亚历山大大学埃尔兰根-纽伦堡分校）；Technical University of Applied Sciences Amberg-Weiden（安贝格-魏登应用技术大学）；Siemens Healthcare GmbH（西门子医疗有限公司）
A Magnetic Resonance Imaging (MRI) exam typically consists of the acquisition of multiple MR pulse sequences, which are required for a reliable diagnosis. Each sequence can be parameterized through multiple acquisition parameters affecting MR image contrast, signal-to-noise ratio, resolution, or scan time. With the rise of generative deep learning models, approaches for the synthesis of MR images are developed to either synthesize additional MR contrasts, generate synthetic data, or augment existing data for AI training. However, current generative approaches for the synthesis of MR images are…
The biggest challenge faced by a Machine Learning Engineer is the lack of data they have, especially for 2-dimensional images. The image is processed to be trained into a Machine Learning model so that it can recognize patterns in the data and provide predictions. This research is intended to create a solution using the Cycle Generative Adversarial Networks (GANs) algorithm in overcoming the problem of lack of data. Then use Style Transfer to be able to generate a new image based on the given style. Based on the results of testing the resulting model has been carried out several improvements, …
University of Chicago Medicine（芝加哥大学医学中心）；Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院蒂施癌症研究所）；Ghost Autonomy, Inc.（Ghost Autonomy 公司）；University of Wisconsin at Madison（威斯康星大学麦迪逊分校）；University of Chicago（芝加哥大学）
Artificial intelligence methods including deep neural networks (DNN) can provide rapid molecular classification of tumors from routine histology with accuracy that matches or exceeds human pathologists. Discerning how neural networks make their predictions remains a significant challenge, but explainability tools help provide insights into what models have learned when corresponding histologic features are poorly defined. Here, we present a method for improving explainability of DNN models using synthetic histology generated by a conditional generative adversarial network (cGAN). We show that …
University of Central Florida（中佛罗里达大学）；Florida Hospital（佛罗里达医院）
Discriminating lung nodules as malignant or benign is still an underlying challenge. To address this challenge, radiologists need computer aided diagnosis (CAD) systems which can assist in learning discriminative imaging features corresponding to malignant and benign nodules. However, learning highly discriminative imaging features is an open problem. In this paper, our aim is to learn the most discriminative features pertaining to lung nodules by using an adversarial learning methodology. Specifically, we propose to use unsupervised learning with Deep Convolutional-Generative Adversarial Netw…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；The Hospital for Sick Children（病童医院）
Early detection of cancer is key to a good prognosis and requires frequent testing, especially in pediatrics. Whole-body magnetic resonance imaging (wbMRI) is an essential part of several well-established screening protocols, with screening starting in early childhood. To date, machine learning (ML) has been used on wbMRI images to stage adult cancer patients. It is not possible to use such tools in pediatrics due to the changing bone signal throughout growth, the difficulty of obtaining these images in young children due to movement and limited compliance, and the rarity of positive cases. We…
University of New South Wales（新南威尔士大学）；Princess Nourah bint Abdulrahman University（诺拉·本特·阿卜杜勒拉赫曼公主大学）
With the increasing interest in the content creation field in multiple sectors such as media, education, and entertainment, there is an increasing trend in the papers that uses AI algorithms to generate content such as images, videos, audio, and text. Generative Adversarial Networks (GANs) in one of the promising models that synthesizes data samples that are similar to real data samples. While the variations of GANs models, in general, have been covered to some extent in several survey papers, to the best of our knowledge, this is among the first survey papers that reviews the state-of-the-art…
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
Rethinking Patient Education as Multi-turn Multi-modal Interaction
VA Bedford Health Care（VA贝德福德医疗中心）；UMass Amherst（马萨诸塞大学阿默斯特分校）；UMass Lowell（马萨诸塞大学洛厄尔分校）；Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）；Yale School of Medicine（耶鲁医学院）
ECC Games SA（ECC游戏股份公司）；Kozminski University（科兹明斯基大学）
We present practical approaches of using deep learning to create and enhance level maps and textures for video games -- desktop, mobile, and web. We aim to present new possibilities for game developers and level artists. The task of designing levels and filling them with details is challenging. It is both time-consuming and takes effort to make levels rich, complex, and with a feeling of being natural. Fortunately, recent progress in deep learning provides new tools to accompany level designers and visual artists. Moreover, they offer a way to generate infinite worlds for game replayability an…
LPIXEL Inc.（LPIXEL公司）；Saitama Prefectural University（埼玉县立大学）；Okayama City General Medical Center, Okayama City Hospital（冈山市综合医疗中心 冈山市立医院）；Okayama University（冈山大学）
Convolutional Neural Networks play a key role in bone age assessment for investigating endocrinology, genetic, and growth disorders under various modalities and body regions. However, no researcher has tackled bone age progression/regression despite its valuable potential applications: bone-related disease diagnosis, clinical knowledge acquisition, and museum education. Therefore, we propose Bone Age Progression Generative Adversarial Network (BAPGAN) to progress/regress both femur/phalange X-ray images while preserving identity and realism. We exhaustively confirm the BAPGAN's clinical potent…
Multitask Brain Tumor Inpainting with Diffusion Models: A Methodological Report
Mayo Clinic Artificial Intelligence Laboratory, Mayo Clinic（梅奥诊所人工智能实验室，梅奥诊所）；Orthopedic Surgery Artificial Intelligence Laboratory, Mayo Clinic（梅奥诊所骨科手术人工智能实验室）
Ultrasound simulation based on ray tracing enables the synthesis of highly realistic images. It can provide an interactive environment for training sonographers as an educational tool. However, due to high computational demand, there is a trade-off between image quality and interactivity, potentially leading to sub-optimal results at interactive rates. In this work we introduce a deep learning approach based on adversarial training that mitigates this trade-off by improving the quality of simulated images with constant computation time. An image-to-image translation framework is utilized to tr…
Maharaja Agrasen Institute of Technology（马哈拉贾·阿格拉森技术学院）；Near East University（近东大学）；State University of Ceará（塞阿拉州立大学）；University of Fortaleza（福塔莱萨大学）
Coronavirus (COVID-19) is a viral disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The spread of COVID-19 seems to have a detrimental effect on the global economy and health. A positive chest X-ray of infected patients is a crucial step in the battle against COVID-19. Early results suggest that abnormalities exist in chest X-rays of patients suggestive of COVID-19. This has led to the introduction of a variety of deep learning systems and studies have shown that the accuracy of COVID-19 patient detection through the use of chest X-rays is strongly optimistic. Dee…
Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era
Griffith University（格里菲斯大学）；University of Bremen（不来梅大学）；Ho Chi Minh City University of Technology（胡志明市技术大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Hanoi University of Science and Technology（河内科技大学）；The University of Queensland（昆士兰大学）
Erasmus MC（伊拉斯谟医学中心）；Radboudumc（拉德堡德大学医学中心）；Donders Institute for Brain, Cognition and Behaviour（唐德斯大脑、认知与行为研究所）；Eindhoven University of Technology（埃因霍温理工大学）；Intel Corporation（英特尔公司）
Adversarial attacks are considered a potentially serious security threat for machine learning systems. Medical image analysis (MedIA) systems have recently been argued to be vulnerable to adversarial attacks due to strong financial incentives and the associated technological infrastructure. In this paper, we study previously unexplored factors affecting adversarial attack vulnerability of deep learning MedIA systems in three medical domains: ophthalmology, radiology, and pathology. We focus on adversarial black-box settings, in which the attacker does not have full access to the target model a…
Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）
Seoul National University（首尔大学）；Kookmin University（国民大学）
Accurate identification and localization of abnormalities from radiology images serve as a critical role in computer-aided diagnosis (CAD) systems. Building a highly generalizable system usually requires a large amount of data with high-quality annotations, including disease-specific global and localization information. However, in medical images, only a limited number of high-quality images and annotations are available due to annotation expenses. In this paper, we explore this problem by presenting a novel approach for disease generation in X-rays using a conditional generative adversarial l…
Tennessee Technological University（田纳西理工大学）；Department of Computer Science（计算机科学系）；Cookeville, TN, United States of America（美国田纳西州库克维尔）
Artificial intelligence (AI) has been a topic of major research for many years. Especially, with the emergence of deep neural network (DNN), these studies have been tremendously successful. Today machines are capable of making faster, more accurate decision than human. Thanks to the great development of machine learning (ML) techniques, ML have been used many different fields such as education, medicine, malware detection, autonomous car etc. In spite of having this degree of interest and much successful research, ML models are still vulnerable to adversarial attacks. Attackers can manipulate …
University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh Medical School（匹兹堡大学医学院）
We propose a BlackBox Counterfactual Explainer, designed to explain image classification models for medical applications. Classical approaches (e.g., saliency maps) that assess feature importance do not explain "how" imaging features in important anatomical regions are relevant to the classification decision. Our framework explains the decision for a target class by gradually "exaggerating" the semantic effect of the class in a query image. We adopted a Generative Adversarial Network (GAN) to generate a progressive set of perturbations to a query image, such that the classification decision ch…
Universitat de Barcelona（巴塞罗那大学）；University of Arkansas for Medical Sciences（阿肯色医科大学）
Synthetic data generated by generative models can enhance the performance and capabilities of data-hungry deep learning models in medical imaging. However, there is (1) limited availability of (synthetic) datasets and (2) generative models are complex to train, which hinders their adoption in research and clinical applications. To reduce this entry barrier, we propose medigan, a one-stop shop for pretrained generative models implemented as an open-source framework-agnostic Python library. medigan allows researchers and developers to create, increase, and domain-adapt their training data in jus…
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Petuum Inc.（Petuum 公司）
Due to the shortage of COVID-19 viral testing kits and the long waiting time, radiology imaging is used to complement the screening process and triage patients into different risk levels. Deep learning based methods have taken an active role in automatically detecting COVID-19 disease in chest x-ray images, as witnessed in many recent works in early 2020. Most of these works first train a Convolutional Neural Network (CNN) on an existing large-scale chest x-ray image dataset and then fine-tune it with a COVID-19 dataset at a much smaller scale. However, direct transfer across datasets from dif…
University of Houston（休斯顿大学）；University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）
Radiographic images offer an alternative method for the rapid screening and monitoring of Coronavirus Disease 2019 (COVID-19) patients. This approach is limited by the shortage of radiology experts who can provide a timely interpretation of these images. Motivated by this challenge, our paper proposes a novel learning architecture, called Detail-Oriented Capsule Networks (DECAPS), for the automatic diagnosis of COVID-19 from Computed Tomography (CT) scans. Our network combines the strength of Capsule Networks with several architecture improvements meant to boost classification accuracies. Firs…
Revisiting Medical Image Retrieval via Knowledge Consolidation
Bioengineering Department and Imperial-X, Imperial College（生物工程系和帝国理工学院）；Archimedes Unit, Athena Research Centre（阿基米德单元，阿提卡研究中心）；Hong Kong University of Science and Technology（香港科学大学）；Sun Yat-Sen University（孙中山大学）；CIMIM, Alan Turing Institute（CIMIM，艾伦·图灵研究所）；NIHR Manchester Biomedical Research Centre, University of Manchester（NIHR曼彻斯特生物医学研究中心，曼彻斯特大学）；National Heart and Lung Institute, Imperial College London（国家心脏和肺研究所，伦敦帝国学院）；Cardiovascular Research Centre, Royal Brompton Hospital（心血管研究中心，皇家布罗姆pton医院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
