3D ultrasound (3DUS) becomes more interesting for target tracking in radiation therapy due to its capability to provide volumetric images in real-time without using ionizing radiation. It is potentially usable for tracking without using fiducials. For this, a method for learning meaningful representations would be useful to recognize anatomical structures in different time frames in representation space (r-space). In this study, 3DUS patches are reduced into a 128-dimensional r-space using conventional autoencoder, variational autoencoder and sliced-wasserstein autoencoder. In the r-space, the capability of separating different ultrasound patches as well as recognizing similar patches is investigated and compared based on a dataset of liver images. Two metrics to evaluate the tracking capability in the r-space are proposed. It is shown that ultrasound patches with different anatomical structures can be distinguished and sets of similar patches can be clustered in r-space. The results indicate that the investigated autoencoders have different levels of usability for target tracking in 3DUS.
School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）
The progress of deep convolutional neural networks has been successfully exploited in various real-time computer vision tasks such as image classification and segmentation. Owing to the development of computational units, availability of digital datasets, and improved performance of deep learning models, fully automatic and accurate tracking of tongue contours in real-time ultrasound data became practical only in recent years. Recent studies have shown that the performance of deep learning techniques is significant in the tracking of ultrasound tongue contours in real-time applications such as…
Image representations are commonly learned from class labels, which are a simplistic approximation of human image understanding. In this paper we demonstrate that transferable representations of images can be learned without manual annotations by modeling human visual attention. The basis of our analyses is a unique gaze tracking dataset of sonographers performing routine clinical fetal anomaly screenings. Models of sonographer visual attention are learned by training a convolutional neural network (CNN) to predict gaze on ultrasound video frames through visual saliency prediction or gaze-poin…
Image-guided radiation therapy can benefit from accurate motion tracking by ultrasound imaging, in order to minimize treatment margins and radiate moving anatomical targets, e.g., due to breathing. One way to formulate this tracking problem is the automatic localization of given tracked anatomical landmarks throughout a temporal ultrasound sequence. For this, we herein propose a fully-convolutional Siamese network that learns the similarity between pairs of image regions containing the same landmark. Accordingly, it learns to localize and thus track arbitrary image features, not only predefine…
University of Michigan（密歇根大学）；University of California San Diego（加州大学圣迭戈分校）
For speech research, ultrasound tongue imaging provides a non-invasive means for visualizing tongue position and movement during articulation. Extracting tongue contours from ultrasound images is a basic step in analyzing ultrasound data but this task often requires non-trivial manual annotation. This study presents an open source tool for fully automatic tracking of tongue contours in ultrasound frames using neural network based methods. We have implemented and systematically compared two convolutional neural networks, U-Net and DenseU-Net, under different conditions. Though both models can p…
Faculty of Engineering and Applied Sciences, Memorial University（纪念大学工程与应用科学学院）；Faculty of Medicine, Memorial University（纪念大学医学院）
Detection of relative changes in circulating blood volume is important to guide resuscitation and manage a variety of medical conditions including sepsis, trauma, dialysis and congestive heart failure. Recent studies have shown that estimates of circulating blood volume can be obtained from the cross-sectional area (CSA) of the internal jugular vein (IJV) from ultrasound images. However, accurate segmentation and tracking of the IJV in ultrasound imaging is a challenging task and is significantly influenced by a number of parameters such as the image quality, shape, and temporal variation. In …
One usage of medical ultrasound imaging is to visualize and characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures in applications such as visual training of a second language. Moreover, quantitative analysis of tongue motion needs the tongue dorsum contour to be extracted, tracked, and visualized. Manual tongue contour extraction is a cumbersome, subjective, and error-prone task.…
Ultrasound localization microscopy (ULM) refers to a promising medical imaging modality that systematically leverages the advantages of contrast-enhanced ultrasound (CEUS) to surpass the diffraction barrier and delineate the microvascular map. Localization and tracking of microbubbles (MBs), two significant steps of ULM, facilitate generating the vascular map and the velocity distribution, respectively. Herein, we propose a novel MB tracking technique considering temporal pairing as a bubble-set registration problem. Iterative registration is performed between the bubble sets in two consecutiv…
Detection of relative changes in circulating blood volume is important to guide resuscitation and manage variety of medical conditions including sepsis, trauma, dialysis and congestive heart failure. Recent studies have shown that estimates of circulating blood volume can be obtained from ultrasound imagery of the of the internal jugular vein (IJV). However, segmentation and tracking of the IJV is significantly influenced by speckle noise and shadowing which introduce uncertainty in the boundaries of the vessel. In this paper, we investigate the use of optical flow algorithms for segmentation …
UC San Diego（加州大学圣地亚哥分校）；General Electric Global Research（通用电气全球研究中心）
Ultrasound scanning is essential in several medical diagnostic and therapeutic applications. It is used to visualize and analyze anatomical features and structures that influence treatment plans. However, it is both labor intensive, and its effectiveness is operator dependent. Real-time accurate and robust automatic detection and tracking of anatomical structures while scanning would significantly impact diagnostic and therapeutic procedures to be consistent and efficient. In this paper, we propose a deep learning framework to automatically detect and track a specific anatomical target structu…
École polytechnique fédérale de Lausanne（洛桑联邦理工学院）；University Hospital Center (CHUV)（大学医院中心（CHUV））；University of Lausanne（洛桑大学）；Center for Biomedical Imaging (CIBM)（生物医学成像中心（CIBM））
Thanks to its capability of acquiring full-view frames at multiple kilohertz, ultrafast ultrasound imaging unlocked the analysis of rapidly changing physical phenomena in the human body, with pioneering applications such as ultrasensitive flow imaging in the cardiovascular system or shear-wave elastography. The accuracy achievable with these motion estimation techniques is strongly contingent upon two contradictory requirements: a high quality of consecutive frames and a high frame rate. Indeed, the image quality can usually be improved by increasing the number of steered ultrafast acquisition…
ImplicitCell: Resolution Cell Modeling of Joint Implicit Volume Reconstruction and Pose Refinement in Freehand 3D Ultrasound
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Tongren Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属同仁医院）
University of British Columbia（不列颠哥伦比亚大学）；Haskins Laboratories（哈斯金斯实验室）
Thousands of individuals need surgical removal of their larynx due to critical diseases every year and therefore, require an alternative form of communication to articulate speech sounds after the loss of their voice box. This work addresses the articulatory-to-acoustic mapping problem based on ultrasound (US) tongue images for the development of a silent-speech interface (SSI) that can provide them with an assistance in their daily interactions. Our approach targets automatically extracting tongue movement information by selecting an optimal feature set from US images and mapping these featur…
Myocardial Strain Drift Correction in Deep Learning Based Ultrasound Tracking
University of Sherbrooke（舍布鲁克大学）；INSA（法国国立应用科学学院）；Université Lyon 1（里昂第一大学）；CREATIS（CREATIS实验室）；Institut Universitaire de France (IUF)（法国大学研究院（IUF））；NTNU（挪威科技大学）；St. Olavs Hospital（圣奥拉夫医院）；SINTEF Digital（SINTEF数字研究所）；Levanger Hospital（莱旺厄尔医院）；Nord-Trøndelag Hospital Trust（北特伦德拉格医院信托）；Hôpital Croix-Rousse（克鲁瓦鲁斯医院）；Hospices Civils de Lyon（里昂民用医院集团）；Hôpital Lyon Sud（里昂南部医院）
University of Oulu（奥卢大学）；University College London（伦敦大学学院）
Many interventional surgical procedures rely on medical imaging to visualise and track instruments. Such imaging methods not only need to be real-time capable, but also provide accurate and robust positional information. In ultrasound applications, typically only two-dimensional data from a linear array are available, and as such obtaining accurate positional estimation in three dimensions is non-trivial. In this work, we first train a neural network, using realistic synthetic training data, to estimate the out-of-plane offset of an object with the associated axial aberration in the reconstruc…
Institute of Research and Technology b<>com（b<>com技术研究院）
Purpose. Ability to locate and track ultrasound images in the 3D operating space is of great benefit for multiple clinical applications. This is often accomplished by tracking the probe using a precise but expensive optical or electromagnetic tracking system. Our goal is to develop a simple and low cost augmented reality echography framework using a standard RGB-D Camera. Methods. A prototype system consisting of an Occipital Structure Core RGB-D camera, a specifically-designed 3D marker, and a fast point cloud registration algorithm FaVoR was developed and evaluated on an Ultrasonix ultrasoun…
Current commercial matrix transducers for 3D DCE-US do not display side-by-side B-mode and contrast-mode images when capturing volumetric data, thus leaving the operator with no position feedback during lengthy acquisitions. The purpose of this study was to investigate the use of transducer tracking to provide positioning feedback and to re-align images to improve quantification. An interventional tracking system was developed in house using an infrared camera and a 3D-printed tracking target attached to a X6-1 matrix transducer. The system displays a virtual probe on a separate screen and all…
A Unified Platform and Quality Assurance Framework for 3D Ultrasound Reconstruction with Robotic, Optical, and Electromagnetic Tracking
School of Computer Science, University of Leeds（利兹大学计算机科学学院）；School of Electronic and Electrical Engineering, University of Leeds（利兹大学电子与电气工程学院）；St James’s University Hospital, Leeds Teaching Hospitals NHS Trust（利兹教学医院，利兹教学医院国家健康服务信托基金）；Leeds Institute of Medical Research, University of Leeds（利兹医学研究所，利兹大学）
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）
Transrectal ultrasound (US) is the most commonly used imaging modality to guide prostate biopsy and its 3D volume provides even richer context information. Current methods for 3D volume reconstruction from freehand US scans require external tracking devices to provide spatial position for every frame. In this paper, we propose a deep contextual learning network (DCL-Net), which can efficiently exploit the image feature relationship between US frames and reconstruct 3D US volumes without any tracking device. The proposed DCL-Net utilizes 3D convolutions over a US video segment for feature extra…
We propose a three-dimensional (3D) multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. To the best of our knowledge, these techniques have not been combined before as a multimodal imaging technique. The system complements the internal 3D information acquired with ultrasound, with the external surface measured with the structure light technique. Moreover, the ultrasound probe's optical tracking for pose estimation was implemented based on a convolutional neural network. Experime…
The assessment of the blood volume is crucial for the management of many acute and chronic diseases. Recent studies have shown that circulating blood volume correlates with the cross-sectional area (CSA) of the internal jugular vein (IJV) estimated from ultrasound imagery. In this paper, a semi-automatic segmentation algorithm is proposed using a combination of region growing and active contour techniques to provide a fast and accurate segmentation of IJV ultrasound videos. The algorithm is applied to track and segment the IJV across a range of image qualities, shapes, and temporal variation. …
CycleULM: A unified label-free deep learning framework for ultrasound localisation microscopy
Department of Bioengineering, Faculty of Engineering, Imperial College London（生物工程系，工程学院，帝国理工学院伦敦分校）；Department of Electrical and Electronic Engineering, Faculty of Engineering, Imperial College London（电气与电子工程系，工程学院，帝国理工学院伦敦分校）；Faculty of Electrical and Computer Engineering, Escuela Superior Politécnica del Litoral (ESPOL)（电气与计算机工程学院，ESPOL高级理工大学）
VidFuncta: Towards Generalizable Neural Representations for Ultrasound Videos
Dept. of Biomedical Informatics \& Data Science, Yale University, New Haven, USA Yale Biomedical Imaging Institute, Yale University, New Haven, USA Dept. of Biomedical Engineering, University of Basel, Allschwil, Switzerland Dept. of Radiology \& Biomedical Imaging, Yale University, New Haven, USA Dept. of Biomedical Engineering, Yale University, New Haven, USA
School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）
Ultrasound imaging is safe, relatively affordable, and capable of real-time performance. One application of this technology is to visualize and to characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the noisy nature of ultrasound images with low-contrast characteristic, it might require expertise for non-expert users to recognize organ shape such as tongue surface (dorsum). To alleviate this difficulty for quantitative analysis of tongue shape and motion, tongue surface can be extracted, tracked, and visualized instead of …
Medical ultrasound technology is widely used in routine clinical applications such as disease diagnosis and treatment as well as other applications like real-time monitoring of human tongue shapes and motions as visual feedback in second language training. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures. Manual tongue segmentation is a cumbersome, subjective, and error-prone task. Furthermore, it is not a feasible solution for real-time applications. In the last few years, deep learning m…
Ultrasound Localization Microscopy (ULM) offers a cost-effective modality for microvascular imaging by using intravascular contrast agents (microbubbles). However, ULM has a fundamental trade-off between acquisition time and spatial resolution, which makes clinical translation challenging. In this paper, in order to circumvent the trade-off, we introduce a spatiotemporal filtering operation dubbed velocity filtering, which is capable of separating contrast agents into different groups based on their vector velocities thus reducing interference in the localization step, while simultaneously off…
Optoacoustic image formation is conventionally based upon ultrasound time-of-flight readings from multiple detection positions. Herein, we exploit acoustic scattering to physically encode the position of optical absorbers in the acquired signals, thus reduce the amount of data required to reconstruct an image from a single waveform. This concept is experimentally tested by including a random distribution of scatterers between the sample and an ultrasound detector array. Ultrasound transmission through a randomized scattering medium was calibrated by raster scanning a light-absorbing microparti…
