Adversarial attacks on deep learning models for fatty liver disease classification by modification of ultrasound image reconstruction method
Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Medical University of Warsaw（华沙医科大学）
Michal Byra, Grzegorz Styczynski, Cezary Szmigielski, Piotr Kalinowski, Lukasz Michalowski, Rafal Paluszkiewicz, Bogna Ziarkiewicz-Wroblewska, Krzysztof Zieniewicz, Andrzej Nowicki
Convolutional neural networks (CNNs) have achieved remarkable success in medical image analysis tasks. In ultrasound (US) imaging, CNNs have been applied to object classification, image reconstruction and tissue characterization. However, CNNs can be vulnerable to adversarial attacks, even small perturbations applied to input data may significantly affect model performance and result in wrong output. In this work, we devise a novel adversarial attack, specific to ultrasound (US) imaging. US images are reconstructed based on radio-frequency signals. Since the appearance of US images depends on the applied image reconstruction method, we explore the possibility of fooling deep learning model by perturbing US B-mode image reconstruction method. We apply zeroth order optimization to find small perturbations of image reconstruction parameters, related to attenuation compensation and amplitude compression, which can result in wrong output. We illustrate our approach using a deep learning model developed for fatty liver disease diagnosis, where the proposed adversarial attack achieved success rate of 48%.
Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection
School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Department of Computer and Information Technology, Purdue University（普渡大学计算机与信息技术系）；School of Computer Science, Chengdu University of Information Technology（成都信息工程大学计算机学院）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
Applications of ultrasound images have expanded from fetal imaging to abdominal and cardiac diagnosis. Liver-being the largest gland in the body and responsible for metabolic activities requires to be to be diagnosed and therefore subject to utmost injury. Although, ultrasound imaging has developed into three and four dimensions providing higher amount of information; it requires highly trained medical staff due to the image complexity and dimensions it contain. Since 2D ultrasound images are still considered to be the basis of clinical treatments,computer aided automated liver diagnosis is ve…
Image-guided radiation therapy can benefit from accurate motion tracking by ultrasound imaging, in order to minimize treatment margins and radiate moving anatomical targets, e.g., due to breathing. One way to formulate this tracking problem is the automatic localization of given tracked anatomical landmarks throughout a temporal ultrasound sequence. For this, we herein propose a fully-convolutional Siamese network that learns the similarity between pairs of image regions containing the same landmark. Accordingly, it learns to localize and thus track arbitrary image features, not only predefine…
Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳市大数据研究院）
Accurate analysis of the fibrosis stage plays very important roles in follow-up of patients with chronic hepatitis B infection. In this paper, a deep learning framework is presented for automatically liver fibrosis prediction. On contrary of previous works, our approach can take use of the information provided by multiple ultrasound images. An indicator-guided learning mechanism is further proposed to ease the training of the proposed model. This follows the workflow of clinical diagnosis and make the prediction procedure interpretable. To support the training, a dataset is well-collected whic…
Volume visualization is a method that displays three-dimensional (3D) data in two-dimensional (2D) space. Using 3D datasets instead of 2D traditional images improves the visualization of anatomical structures, and volume visualization helps radiologists and surgeons to review large datasets comprehensively so that diagnosis and treatment can be enhanced. In liver surgery, blood vessel detection is important. Liver vessels have various shapes and due to the presence of noise in the ultrasound images, they can be confused with noise. Suboptimal images can sometimes lead to surgical errors where …
PAII Inc.（PAII 公司）；Chang Gung Memorial Hospital, Linkou Medical Center（长庚纪念医院林口医学中心）；Institute of Molecular and Genomic Medicine, National Health Research Institutes（国家卫生研究院分子与基因组医学研究所）；Ping An Insurance Group（平安保险集团）
Background & Aims: Hepatic steatosis is a major cause of chronic liver disease. 2D ultrasound is the most widely used non-invasive tool for screening and monitoring, but associated diagnoses are highly subjective. We developed a scalable deep learning (DL) algorithm for quantitative scoring of liver steatosis from 2D ultrasound images. Approach & Results: Using retrospectively collected multi-view ultrasound data from 3,310 patients, 19,513 studies, and 228,075 images, we trained a DL algorithm to diagnose steatosis stages (healthy, mild, moderate, or severe) from ultrasound diagnoses. Perform…
Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong（香港中文大学深圳大数据研究院）；Huazhong University of Science and Technology（华中科技大学）；Shenzhen Third People’s Hospital（深圳市第三人民医院）
With the development of radiomics, noninvasive diagnosis like ultrasound (US) imaging plays a very important role in automatic liver fibrosis diagnosis (ALFD). Due to the noisy data, expensive annotations of US images, the application of Artificial Intelligence (AI) assisting approaches encounters a bottleneck. Besides, the use of mono-modal US data limits the further improve of the classification results. In this work, we innovatively propose a multi-modal fusion network with active learning (MMFN-AL) for ALFD to exploit the information of multiple modalities, eliminate the noisy data and red…
PAII Inc.（PAII公司）；Chang Gung Memorial Hospital（长庚纪念医院）；Vanderbilt University（范德堡大学）；PingAn Technology（平安科技）
Ultrasound (US) is a critical modality for diagnosing liver fibrosis. Unfortunately, assessment is very subjective, motivating automated approaches. We introduce a principled deep convolutional neural network (CNN) workflow that incorporates several innovations. First, to avoid overfitting on non-relevant image features, we force the network to focus on a clinical region of interest (ROI), encompassing the liver parenchyma and upper border. Second, we introduce global heteroimage fusion (GHIF), which allows the CNN to fuse features from any arbitrary number of images in a study, increasing its…
Rutgers University（罗格斯大学）；Rutgers Cancer Institute of New Jersey（罗格斯大学新泽西癌症研究所）；Rutgers Robert Wood Johnson Medical School（罗格斯大学罗伯特·伍德·约翰逊医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
With the success of deep learning-based methods applied in medical image analysis, convolutional neural networks (CNNs) have been investigated for classifying liver disease from ultrasound (US) data. However, the scarcity of available large-scale labeled US data has hindered the success of CNNs for classifying liver disease from US data. In this work, we propose a novel generative adversarial network (GAN) architecture for realistic diseased and healthy liver US image synthesis. We adopt the concept of stacking to synthesize realistic liver US data. Quantitative and qualitative evaluation is p…
University of Oxford（牛津大学）；University of Jordan（约旦大学）
Assessing tumor tissue heterogeneity via ultrasound has recently been suggested for predicting early response to treatment. The ultrasound backscattering characteristics can assist in better understanding the tumor texture by highlighting local concentration and spatial arrangement of tissue scatterers. However, it is challenging to quantify the various tissue heterogeneities ranging from fine-to-coarse of the echo envelope peaks in tumor texture. Local parametric fractal features extracted via maximum likelihood estimation from five well-known statistical model families are evaluated for the …
Rutgers University（罗格斯大学）；Rutgers Robert Wood Johnson Medical School（罗格斯罗伯特·伍德·约翰逊医学院）
Non-alcoholic fatty liver disease (NAFLD) is one of the most common causes of chronic liver diseases (CLD) which can progress to liver cancer. The severity and treatment of NAFLD is determined by NAFLD Activity Scores (NAS)and liver fibrosis stage, which are usually obtained from liver biopsy. However, biopsy is invasive in nature and involves risk of procedural complications. Current methods to predict the fibrosis and NAS scores from noninvasive CT images rely heavily on either a large annotated dataset or transfer learning using pretrained networks. However, the availability of a large anno…
Computer Science Dept. Rutgers University（罗格斯大学计算机科学系）；Division of Gastroenterology and Hepatology Rutgers Robert Wood Johnson Medical School（罗格斯罗伯特·伍德·约翰逊医学院胃肠病学和肝病学分部）
Non-Alcoholic Fatty Liver Disease (NAFLD) is becoming increasingly prevalent in the world population. Without diagnosis at the right time, NAFLD can lead to non-alcoholic steatohepatitis (NASH) and subsequent liver damage. The diagnosis and treatment of NAFLD depend on the NAFLD activity score (NAS) and the liver fibrosis stage, which are usually evaluated from liver biopsies by pathologists. In this work, we propose a novel method to automatically predict NAS score and fibrosis stage from CT data that is non-invasive and inexpensive to obtain compared with liver biopsy. We also present a meth…
Deep Regression 2D-3D Ultrasound Registration for Liver Motion Correction in Focal Tumor Thermal Ablation
Robarts Research Institute（罗伯茨研究所）；Western University（西安大略大学）；School of Biomedical Engineering, Western University（西安大略大学生物医学工程学院）；Lawson Health Research Institute（劳森健康研究所）
A Novel Framework for Integrating 3D Ultrasound into Percutaneous Liver Tumour Ablation
Robarts Research Institute, Western University, London, Canada（罗巴茨研究所在西蒙大学伦敦分校）；School of Biomedical Engineering, Western University, London, Canada（生物医学工程学院，西蒙大学伦敦分校）；Department of Medical Biophysics, Western University, London, Canada（医学生物物理学系，西蒙大学伦敦分校）；Department of Electrical and Computer Engineering, Western University, London, Canada（电气与计算机工程系，西蒙大学伦敦分校）；Department of Medical Imaging, Western University, London, Canada（医学影像系，西蒙大学伦敦分校）
University of Jordan（约旦大学）；King Abdullah II School for IT（阿卜杜拉二世信息技术学院）
Effective ultrasound tissue characterization is usually hindered by complex tissue structures. The interlacing of speckle patterns complicates the correct estimation of backscatter distribution parameters. Nakagami parametric imaging based on localized shape parameter mapping can model different backscattering conditions. However, performance of the constructed Nakagami image depends on the sensitivity of the estimation method to the backscattered statistics and scale of analysis. Using a fixed focal region of interest in estimating the Nakagami parametric image would increase estimation varia…
Pakistan Institute of Engineering and Applied Sciences（巴基斯坦工程与应用科学学院）；Multan Institute of Nuclear Medicine and Radiotherapy（木尔坦核医学与放射治疗研究所）
This paper presents a novel approach for detection of liver abnormalities in an automated manner using ultrasound images. For this purpose, we have implemented a machine learning model that can not only generate labels (normal and abnormal) for a given ultrasound image but it can also detect when its prediction is likely to be incorrect. The proposed model abstains from generating the label of a test example if it is not confident about its prediction. Such behavior is commonly practiced by medical doctors who, when given insufficient information or a difficult case, can chose to carry out fur…
Università degli Studi di Genova（热那亚大学）；IRCCS San Martino Hospital（IRCCS圣马蒂诺医院）
Due to the need for quick and effective treatments for liver diseases, which are among the most common health problems in the world, staging fibrosis through non-invasive and economic methods has become of great importance. Taking inspiration from diagnostic laparoscopy, used in the past for hepatic diseases, in this paper ultrasound images of the liver are studied, focusing on a specific region of the organ where the Glisson's capsule is visible. In ultrasound images, the Glisson's capsule appears in the shape of a line which can be extracted via classical methods in literature. By making use…
Ultrasound images are one of the most widely used techniques in clinical settings to analyze and detect different organs for study or diagnoses of diseases. The dependence on subjective opinions of experts such as radiologists calls for an automatic recognition and detection system that can provide an objective analysis. Previous work done on this topic is limited and can be classified by the organ of interest. Hybrid neural networks, linear and logistic regression models, 3D reconstructed models, and various machine learning techniques have been used to solve complex problems such as detectio…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shandong University（山东大学）；Beijing University of Chinese Medicine（北京中医药大学）
Current commercial matrix transducers for 3D DCE-US do not display side-by-side B-mode and contrast-mode images when capturing volumetric data, thus leaving the operator with no position feedback during lengthy acquisitions. The purpose of this study was to investigate the use of transducer tracking to provide positioning feedback and to re-align images to improve quantification. An interventional tracking system was developed in house using an infrared camera and a 3D-printed tracking target attached to a X6-1 matrix transducer. The system displays a virtual probe on a separate screen and all…
In conventional 2D DCE-US, motion correction algorithms take advantage of accompanying side-by-side anatomical Bmode images that contain time-stable features. However, current commercial models of 3D DCE-US do not provide side-by-side Bmode images, which makes motion correction challenging. This work introduces a novel motion correction (MC) algorithm for 3D DCE-US and assesses its efficacy when handling clinical data sets. In brief, the algorithm uses a pyramidal approach whereby short temporal windows consisting of 3-6 consecutive frames are created to perform local registrations, which are …
Duke University School of Medicine（杜克大学医学院）；Duke University（杜克大学）；University of Arizona（亚利桑那大学）
Purpose: To design multi-disease classifiers for body CT scans for three different organ systems using automatically extracted labels from radiology text reports.Materials & Methods: This retrospective study included a total of 12,092 patients (mean age 57 +- 18; 6,172 women) for model development and testing (from 2012-2017). Rule-based algorithms were used to extract 19,225 disease labels from 13,667 body CT scans from 12,092 patients. Using a three-dimensional DenseVNet, three organ systems were segmented: lungs and pleura; liver and gallbladder; and kidneys and ureters. For each organ, a t…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）
Despite that deep learning has achieved state-of-the-art performance for medical image segmentation, its success relies on a large set of manually annotated images for training that are expensive to acquire. In this paper, we propose an annotation-efficient learning framework for segmentation tasks that avoids annotations of training images, where we use an improved Cycle-Consistent Generative Adversarial Network (GAN) to learn from a set of unpaired medical images and auxiliary masks obtained either from a shape model or public datasets. We first use the GAN to generate pseudo labels for our …
