We present W-Net, a novel Convolution Neural Network (CNN) framework that employs raw ultrasound waveforms from each A-scan, typically referred to as ultrasound Radio Frequency (RF) data, in addition to the gray ultrasound image to semantically segment and label tissues. Unlike prior work, we seek to label every pixel in the image, without the use of a background class. To the best of our knowledge, this is also the first deep-learning or CNN approach for segmentation that analyses ultrasound raw RF data along with the gray image. International patent(s) pending [PCT/US20/37519]. We chose subcutaneous tissue (SubQ) segmentation as our initial clinical goal since it has diverse intermixed tissues, is challenging to segment, and is an underrepresented research area. SubQ potential applications include plastic surgery, adipose stem-cell harvesting, lymphatic monitoring, and possibly detection/treatment of certain types of tumors. A custom dataset consisting of hand-labeled images by an expert clinician and trainees are used for the experimentation, currently labeled into the following categories: skin, fat, fat fascia/stroma, muscle and muscle fascia. We compared our results with U-Net and Attention U-Net. Our novel \emph{W-Net}'s RF-Waveform input and architecture increased mIoU accuracy (averaged across all tissue classes) by 4.5\% and 4.9\% compared to regular U-Net and Attention U-Net, respectively. We present analysis as to why the Muscle fascia and Fat fascia/stroma are the most difficult tissues to label. Muscle fascia in particular, the most difficult anatomic class to recognize for both humans and AI algorithms, saw mIoU improvements of 13\% and 16\% from our W-Net vs U-Net and Attention U-Net respectively.
University of Central Florida（中佛罗里达大学）；St. Louis University（圣路易斯大学）；National Institutes of Health（美国国立卫生研究院）
Quantification of adipose tissue (fat) from computed tomography (CT) scans is conducted mostly through manual or semi-automated image segmentation algorithms with limited efficacy. In this work, we propose a completely unsupervised and automatic method to identify adipose tissue, and then separate Subcutaneous Adipose Tissue (SAT) from Visceral Adipose Tissue (VAT) at the abdominal region. We offer a three-phase pipeline consisting of (1) Initial boundary estimation using gradient points, (2) boundary refinement using Geometric Median Absolute Deviation and Appearance based Local Outlier Score…
BAAF: A benchmark attention adaptive framework for medical ultrasound image segmentation tasks
College of Biomedical Engineering and Technology, Tianjin Medical University（天津医科大学生物医学工程学院）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Department of Urology, Fourth Medical Center of PLA General Hospital（中国人民解放军第四医学中心泌尿外科科）；Department of Urology, Civil Aviation General Hospital（民航总医院泌尿外科科）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）；Department of Ultrasonography, Tianjin Medical University Second Hospital（天津医科大学第二医院超声科）
Body tissue composition is a long-known biomarker with high diagnostic and prognostic value in cardiovascular, oncological and orthopaedic diseases, but also in rehabilitation medicine or drug dosage. In this study, the aim was to develop a fully automated, reproducible and quantitative 3D volumetry of body tissue composition from standard CT examinations of the abdomen in order to be able to offer such valuable biomarkers as part of routine clinical imaging. Therefore an in-house dataset of 40 CTs for training and 10 CTs for testing were fully annotated on every fifth axial slice with five di…
Applying machine learning technologies, especially deep learning, into medical image segmentation is being widely studied because of its state-of-the-art performance and results. It can be a key step to provide a reliable basis for clinical diagnosis, such as 3D reconstruction of human tissues, image-guided interventions, image analyzing and visualization. In this review article, deep-learning-based methods for ultrasound image segmentation are categorized into six main groups according to their architectures and training at first. Secondly, for each group, several current representative algor…
University of Electronic Science and Technology of China（电子科技大学）；Institute of Electronic and Information Engineering of UESTC in Guangdong（电子科技大学广东电子信息工程研究院）；Sichuan Provincial Hospital for Women and Children（四川省妇幼保健院）
The identification of nerve is difficult as structures of nerves are challenging to image and to detect in ultrasound images. Nevertheless, the nerve identification in ultrasound images is a crucial step to improve performance of regional anesthesia. In this paper, a network called Brachial Plexus Multi-instance Segmentation Network (BPMSegNet) is proposed to identify different tissues (nerves, arteries, veins, muscles) in ultrasound images. The BPMSegNet has three novel modules. The first is the spatial local contrast feature, which computes contrast features at different scales. The second o…
Indian Institute of Technology Kharagpur（印度理工学院卡哈拉格普尔分校）；NVIDIA（英伟达）
Ultrasound image compression by preserving speckle-based key information is a challenging task. In this paper, we introduce an ultrasound image compression framework with the ability to retain realism of speckle appearance despite achieving very high-density compression factors. The compressor employs a tissue segmentation method, transmitting segments along with transducer frequency, number of samples and image size as essential information required for decompression. The decompressor is based on a convolutional network trained to generate patho-realistic ultrasound images which convey essent…
University of Montreal Hospital Research Center (CRCHUM)（蒙特利尔大学医院研究中心）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nanosystems Institute（加州纳米系统研究所）
It is known that changes in the mechanical properties of tissues are associated with the onset and progression of certain diseases. Ultrasound elastography is a technique to characterize tissue stiffness using ultrasound imaging either by measuring tissue strain using quasi-static elastography or natural organ pulsation elastography, or by tracing a propagated shear wave induced by a source or a natural vibration using dynamic elastography. In recent years, deep learning has begun to emerge in ultrasound elastography research. In this review, several common deep learning frameworks in the comp…
Computational simulation of ultrasound (US) echography is essential for training sonographers. Realistic simulation of US interaction with microscopic tissue structures is often modeled by a tissue representation in the form of point scatterers, convolved with a spatially varying point spread function. This yields a realistic US B-mode speckle texture, given that a scatterer representation for a particular tissue type is readily available. This is often not the case and scatterers are nontrivial to determine. In this work we propose to estimate scatterer maps from sample US B-mode images of a …
University of Oxford（牛津大学）；University of Jordan（约旦大学）
Assessing tumor tissue heterogeneity via ultrasound has recently been suggested for predicting early response to treatment. The ultrasound backscattering characteristics can assist in better understanding the tumor texture by highlighting local concentration and spatial arrangement of tissue scatterers. However, it is challenging to quantify the various tissue heterogeneities ranging from fine-to-coarse of the echo envelope peaks in tumor texture. Local parametric fractal features extracted via maximum likelihood estimation from five well-known statistical model families are evaluated for the …
Technical University Munich（慕尼黑工业大学）；OneProjects Design Innovation GmbH（OneProjects设计创新有限责任公司）
In ultrasound imaging the appearance of homogeneous regions of tissue is subject to speckle, which for certain applications can make the detection of tissue irregularities difficult. To cope with this, it is common practice to apply speckle reduction filters to the images. Most conventional filtering techniques are fairly hand-crafted and often need to be finely tuned to the present hardware, imaging scheme and application. Learning based techniques on the other hand suffer from the need for a target image for training (in case of fully supervised techniques) or require narrow, complex physics…
Identifying visible tissue in intraoperative ultrasound: a method and application
The Hamlyn Centre（哈姆林中心）；Imperial College London（帝国理工学院伦敦分校）；Department Of Imaging（成像部）；Charing Cross Hospital（查林十字医院）；School Of Computation, Information and Technology（计算、信息与技术学校）；Technical University of Munich（慕尼黑技术大学）；Department Of Neurosurgery（神经外科部）
Ultrasound image quality has continually been improving. However, when needles or other metallic objects are operating inside the tissue, the resulting reverberation artifacts can severely corrupt the surrounding image quality. Such effects are challenging for existing computer vision algorithms for medical image analysis. Needle reverberation artifacts can be hard to identify at times and affect various pixel values to different degrees. The boundaries of such artifacts are ambiguous, leading to disagreement among human experts labeling the artifacts. We propose a weakly- and semi-supervised,…
College of Science, Liaoning Petrochemical University（辽宁石油化工大学理学院）；Industrial Mathematics Laboratory, Baikal School of BRICS, Irkutsk National Research Technical University（伊尔库茨克国立研究技术大学贝加尔金砖学院工业数学实验室）
Biodegradable elastic scaffolds have attracted more and more attention in the field of soft tissue repair and tissue engineering. These scaffolds made of porous bioelastomers support tissue ingrowth along with their own degradation. It is necessary to develop a computer-aided analyzing method based on ultrasound images to identify the degradation performance of the scaffold, not only to obviate the need to do destructive testing, but also to monitor the scaffold's degradation and tissue ingrowth over time. It is difficult using a single traditional image processing algorithm to extract continu…
Robotics Institute and Department of ECE, Carnegie Mellon University（卡内基梅隆大学机器人研究所与电子与计算机工程系）；Dept. of Pulmonary and Critical Care Medicine, Louisiana State University（路易斯安那州立大学肺病与危重症医学科）；Cosmeticsurg.net, LLC（Cosmeticsurg.net有限责任公司）
In this paper, we study the significance of the pleura and adipose tissue in lung ultrasound AI analysis. We highlight their more prominent appearance when using high-frequency linear (HFL) instead of curvilinear ultrasound probes, showing HFL reveals better pleura detail. We compare the diagnostic utility of the pleura and adipose tissue using an HFL ultrasound probe. Masking the adipose tissue during training and inference (while retaining the pleural line and Merlin's space artifacts such as A-lines and B-lines) improved the AI model's diagnostic accuracy.
Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquiring large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient…
University Hospital Tübingen（蒂宾根大学医院）；University of Stuttgart（斯图加特大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Eberhard Karls University（埃伯哈德·卡尔斯大学）；Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）；German Center for Diabetes Research（德国糖尿病研究中心）；University of Freiburg（弗赖堡大学）
Purpose: To enable fast and reliable assessment of subcutaneous and visceral adipose tissue compartments derived from whole-body MRI. Methods: Quantification and localization of different adipose tissue compartments from whole-body MR images is of high interest to examine metabolic conditions. For correct identification and phenotyping of individuals at increased risk for metabolic diseases, a reliable automatic segmentation of adipose tissue into subcutaneous and visceral adipose tissue is required. In this work we propose a 3D convolutional neural network (DCNet) to provide a robust and obje…
Simulation-based ultrasound training can be an essential educational tool. Realistic ultrasound image appearance with typical speckle texture can be modeled as convolution of a point spread function with point scatterers representing tissue microstructure. Such scatterer distribution, however, is in general not known and its estimation for a given tissue type is fundamentally an ill-posed inverse problem. In this paper, we demonstrate a convolutional neural network approach for probabilistic scatterer estimation from observed ultrasound data. We herein propose to impose a known statistical dis…
Ultrasound elastography is the method to image the elasticity of compliant tissues due to a mechanical compression applied to it. In elastography, the local strain of explored tissue is estimated by analyzing the echo signals. This is accomplished by a sonographer who uses ultrasound transducer to apply pressure on the tissue area causing displacement of the tissue. A set of data is obtained before and after the compression. This stress along the axial direction not only causes deformation in that direction but it can also cause lateral deformation in the tissue. This relative deformation is l…
In this work we propose to segment the prostate on a challenging dataset of trans-rectal ultrasound (TRUS) images using convolutional neural networks (CNNs) and statistical shape models (SSMs). TRUS is commonly used for a number of image-guided interventions on the prostate. Fast and accurate segmentation on the organ in these images is crucial to planning and fusion with other modalities such as magnetic resonance images (MRIs) . However, TRUS has limited soft tissue contrast and signal to noise ratio which makes the task of segmenting the prostate challenging and subject to inter-observer an…
Quantitative ultrasound, e.g., speed-of-sound (SoS) in tissues, provides information about tissue properties that have diagnostic value. Recent studies showed the possibility of extracting SoS information from pulse-echo ultrasound raw data (a.k.a. RF data) using deep neural networks that are fully trained on simulated data. These methods take sensor domain data, i.e., RF data, as input and train a network in an end-to-end fashion to learn the implicit mapping between the RF data domain and SoS domain. However, such networks are prone to overfitting to simulated data which results in poor perf…
Ultrasound attenuation is caused by absorption and scattering in tissue and is thus a function of tissue composition, hence its imaging offers great potential for screening and differential diagnosis. In this paper we propose a novel method that allows to reconstruct spatial attenuation distribution in tissue based on computed tomography, using reflections from a passive acoustic reflector. This requires a standard ultrasound transducer operating in pulse-echo mode, thus it can be implemented on conventional ultrasound systems with minor modifications. We use calibration with water measurement…
Technical University of Munich（慕尼黑工业大学）；Zhejiang University（浙江大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Tissue deformation in ultrasound (US) imaging leads to geometrical errors when measuring tissues due to the pressure exerted by probes. Such deformation has an even larger effect on 3D US volumes as the correct compounding is limited by the inconsistent location and geometry. This work proposes a patient-specified stiffness-based method to correct the tissue deformations in robotic 3D US acquisitions. To obtain the patient-specified model, robotic palpation is performed at sampling positions on the tissue. The contact force, US images and the probe poses of the palpation procedure are recorded…
Queen’s University（女王大学）；University of British Columbia（不列颠哥伦比亚大学）；Exact Imaging（Exact Imaging公司）
Deep learning-based analysis of high-frequency, high-resolution micro-ultrasound data shows great promise for prostate cancer detection. Previous approaches to analysis of ultrasound data largely follow a supervised learning paradigm. Ground truth labels for ultrasound images used for training deep networks often include coarse annotations generated from the histopathological analysis of tissue samples obtained via biopsy. This creates inherent limitations on the availability and quality of labeled data, posing major challenges to the success of supervised learning methods. On the other hand, …
Concordia University（康考迪亚大学）；University of Montreal（蒙特利尔大学）
Ultrasound elastography is used to estimate the mechanical properties of the tissue by monitoring its response to an internal or external force. Different levels of deformation are obtained from different tissue types depending on their mechanical properties, where stiffer tissues deform less. Given two radio frequency (RF) frames collected before and after some deformation, we estimate displacement and strain images by comparing the RF frames. The quality of the strain image is dependent on the type of motion that occurs during deformation. In-plane axial motion results in high-quality strain…
Korea University（高丽大学）；Korea Basic Science Institute（韩国基础科学研究院）；Korea Artificial Organ Center（韩国人工器官中心）；ICT Convergence Technology for Health and Safety（健康安全信息通信融合技术）；Department of Electronics and Information Engineering（电子与信息工程系）；Bioimaging Research Team（生物成像研究团队）
Purpose: To acquire the real-time interactive temperature map for aqueous and adipose tissue, the problems of long acquisition and processing time must be addressed. To overcome these major challenges, this paper proposes a cascaded convolutional neural network (CNN) framework and multi-echo gradient echo (meGRE) with a single reference variable flip angle (srVFA). Methods: To optimize the echo times for each method, MR images are acquired using a meGRE sequence; meGRE images with two flip angles (FAs) and meGRE images with a single FA are acquired during the pretreatment and treatment stages,…
Ultrasound elastography estimates the mechanical properties of the tissue from two Radio-Frequency (RF) frames collected before and after tissue deformation due to an external or internal force. This work focuses on strain imaging in quasi-static elastography, where the tissue undergoes slow deformations and strain images are estimated as a surrogate for elasticity modulus. The quality of the strain image depends heavily on the underlying deformation, and even the best strain estimation algorithms cannot estimate a good strain image if the underlying deformation is not suitable. Herein, we int…
Non-invasive detection of microvascular alterations in deep tissues in vivo provides critical information for clinical diagnosis and evaluation of a broad-spectrum of pathologies. Recently, the emergence of super-resolution ultrasound localization microscopy (ULM) offers new possibilities for clinical imaging of microvasculature at capillary level. Currently, the clinical utility of ULM on clinical ultrasound scanners is hindered by the technical limitations, such as long data acquisition time, and compromised tracking performance associated with low imaging frame-rate. Here we present an in-h…
Imperial College（帝国理工学院）；Hamlyn Centre for Robotic Surgery（哈姆林外科手术机器人中心）；Department of Surgery and Cancer（外科与癌症系）
Brain shift makes the pre-operative MRI navigation highly inaccurate hence the intraoperative modalities are adopted in surgical theatre. Due to the excellent economic and portability merits, the Ultrasound imaging is used at our collaborating hospital, Charing Cross Hospital, Imperial College London, UK. However, it is found that intraoperative diagnosis on Ultrasound images is not straightforward and consistent, even for very experienced clinical experts. Hence, there is a demand to design a Computer-aided-diagnosis system to provide a robust second opinion to help the surgeons. The proposed…
Ultrasound is the second most used modality in medical imaging. It is cost effective, hazardless, portable and implemented routinely in numerous clinical procedures. Nonetheless, image quality is characterized by granulated appearance, poor SNR and speckle noise. Specific for malignant tumors, the margins are blurred and indistinct. Thus, there is a great need for improving ultrasound image quality. We hypothesize that this can be achieved, using neural networks, by translation into a more realistic display which mimics an anatomical cut through the tissue. In order to achieve this goal, the p…
We experimentally observe the shear and secondary compression wave inside soft porous water-saturated melamine foams by high frame rate ultrasound imaging. Both wave speeds are supported by the weak frame of the foam. The first and second compression waves show opposite polarity, as predicted by Biot theory. Our experiments have direct implications for medical imaging: Melamine foams exhibit a similar microstructure as lung tissue. In the future, combined shear wave and slow compression wave imaging might provide new means of distinguishing malignant and healthy pulmonary tissue.
Ultrasound elasticity images which enable the visualization of quantitative maps of tissue stiffness can be reconstructed by solving an inverse problem. Classical model-based approaches for ultrasound elastography use deterministic finite element methods (FEMs) to incorporate the governing physical laws resulting in poor performance in noisy conditions. Moreover, these approaches utilize fixed regularizers for various tissue patterns while appropriate data-adaptive priors might be required for capturing the complex spatial elasticity distribution. In this regard, we propose a joint model-based…
University of Central Florida（中佛罗里达大学）；National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所）；Frederick National Laboratory for Cancer Research（弗雷德里克国家癌症研究实验室）
Abdominal fat quantification is critical since multiple vital organs are located within this region. Although computed tomography (CT) is a highly sensitive modality to segment body fat, it involves ionizing radiations which makes magnetic resonance imaging (MRI) a preferable alternative for this purpose. Additionally, the superior soft tissue contrast in MRI could lead to more accurate results. Yet, it is highly labor intensive to segment fat in MRI scans. In this study, we propose an algorithm based on deep learning technique(s) to automatically quantify fat tissue from MR images through a c…
For beamforming ultrasound (US) signals, typically a spatially constant speed-of-sound (SoS) is assumed to calculate delays. As SoS in tissue may vary relatively largely, this approximation may cause wavefront aberrations, thus degrading effective imaging resolution. In the literature, corrections have been proposed based on unidirectional SoS estimation or computationally-expensive a posteriori phase rectification. In this paper we demonstrate a direct delay correction approach for US beamforming, by leveraging 2D spatial SoS distribution estimates from plane-wave imaging. We show both in sim…
Displacement estimation is a critical step of virtually all Ultrasound Elastography (USE) techniques. Two main features make this task unique compared to the general optical flow problem: the high-frequency nature of ultrasound radio-frequency (RF) data and the governing laws of physics on the displacement field. Recently, the architecture of the optical flow networks has been modified to be able to use RF data. Also, semi-supervised and unsupervised techniques have been employed for USE by considering prior knowledge of displacement continuity in the form of the first- and second-derivative r…
Ultrasound Elastography aims to determine the mechanical properties of the tissue by monitoring tissue deformation due to internal or external forces. Tissue deformations are estimated from ultrasound radio frequency (RF) signals and are often referred to as time delay estimation (TDE). Given two RF frames I1 and I2, we can compute a displacement image which shows the change in the position of each sample in I1 to a new position in I2. Two important challenges in TDE include high computational complexity and the difficulty in choosing suitable RF frames. Selecting suitable frames is of high im…
Ultrasound strain imaging, which delineates mechanical properties to detect tissue abnormalities, involves estimating the time-delay between two radio-frequency (RF) frames collected before and after tissue deformation. The existing regularized optimization-based time-delay estimation (TDE) techniques suffer from at least one of the following drawbacks: 1) The regularizer is not aligned with tissue deformation physics due to taking only the first-order displacement derivative into account. 2) The L2-norm of the displacement derivatives, which oversmooths the estimated time-delay, is utilized a…
Objective: Strain elastography and shear wave elastography are two commonly used methods to quantify cervical elasticity; however, they have limitations. Strain elastography is effective in showing tissue elasticity distribution in a single image, but the absence of stress information causes difficulty in comparing the results acquired from different imaging sessions. Shear wave elastography is effective in measuring shear wave speed (an intrinsic tissue property correlated with elasticity) in relatively homogeneous tissue, such as in the liver. However, for inhomogeneous tissue in the cervix,…
Existing physical model-based imaging methods for ultrasound elasticity reconstruction utilize fixed variational regularizers that may not be appropriate for the application of interest or may not capture complex spatial prior information about the underlying tissues. On the other hand, end-to-end learning-based methods count solely on the training data, not taking advantage of the governing physical laws of the imaging system. Integrating learning-based priors with physical forward models for ultrasound elasticity imaging, we present a joint reconstruction framework which guarantees that lear…
Technion -- Israel Institute of Technology（以色列理工学院）；University of Waterloo（滑铁卢大学）
The cost-effectiveness and practical harmlessness of ultrasound imaging have made it one of the most widespread tools for medical diagnosis. Unfortunately, the beam-forming based image formation produces granular speckle noise, blurring, shading and other artifacts. To overcome these effects, the ultimate goal would be to reconstruct the tissue acoustic properties by solving a full wave propagation inverse problem. In this work, we make a step towards this goal, using Multi-Resolution Convolutional Neural Networks (CNN). As a result, we are able to reconstruct CT-quality images from the reflec…
Ultrasound Shear Wave Elastography (USWE) with conventional B-mode imaging demonstrates better performance in lesion segmentation and classification problems. In this article, we propose SHEAR-net, an end-to-end deep neural network, to reconstruct USWE images from tracked tissue displacement data at different time instants induced by a single acoustic radiation force (ARF) with 100% or 50% of the energy in conventional use. The SHEAR-net consists of a localizer called the S-net to first localize the lesion location and then uses recurrent layers to extract temporal correlations from wave patte…
