Contour-based 3d tongue motion visualization using ultrasound image sequences
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；Langevin Institute, ESPCI-ParisTech（巴黎高等物理化工学院朗之万研究所）；University of New Mexico（新墨西哥大学）；Tianjin University（天津大学）
Kele Xu, Yin Yang, Clémence Leboullenger, Pierre Roussel, Bruce Denby
This article describes a contour-based 3D tongue deformation visualization framework using B-mode ultrasound image sequences. A robust, automatic tracking algorithm characterizes tongue motion via a contour, which is then used to drive a generic 3D Finite Element Model (FEM). A novel contour-based 3D dynamic modeling method is presented. Modal reduction and modal warping techniques are applied to model the deformation of the tongue physically and efficiently. This work can be helpful in a variety of fields, such as speech production, silent speech recognition, articulation training, speech disorder study, etc.
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；SIGnal processing and MAchine learning Lab, ESPCI ParisTech, PSL Research University（巴黎高等物理化工学院信号处理与机器学习实验室，巴黎文理研究大学）；Vocal Tract Visualization Lab, University of Maryland Dental School（马里兰大学牙科学院声道可视化实验室）
Studying tongue motion during speech using ultrasound is a standard procedure, but automatic ultrasound image labelling remains a challenge, as standard tongue shape extraction methods typically require human intervention. This article presents a method based on deep neural networks to automatically extract tongue contour from ultrasound images on a speech dataset. We use a deep autoencoder trained to learn the relationship between an image and its related contour, so that the model is able to automatically reconstruct contours from the ultrasound image alone. In this paper, we use an automati…
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；University of New Mexico（新墨西哥大学）；University of Maryland Dental School（马里兰大学牙科学院）；Langevin Institute, ESPCI-ParisTech（巴黎高等物理化工学院朗之万研究所）
This article describes the development of a platform designed to visualize the 3D motion of the tongue using ultrasound image sequences. An overview of the system design is given and promising results are presented. Compared to the analysis of motion in 2D image sequences, such a system can provide additional visual information and a quantitative description of the tongue 3D motion. The platform can be useful in a variety of fields, such as speech production, articulation training, etc.
National University of Defense Technology（国防科技大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Tongji University（同济大学）
Ultrasound tongue imaging is widely used for speech production research, and it has attracted increasing attention as its potential applications seem to be evident in many different fields, such as the visual biofeedback tool for second language acquisition and silent speech interface. Unlike previous studies, here we explore the feasibility of age estimation using the ultrasound tongue image of the speakers. Motivated by the success of deep learning, this paper leverages deep learning on this task. We train a deep convolutional neural network model on the UltraSuite dataset. The deep model ac…
University of Michigan（密歇根大学）；University of California San Diego（加州大学圣迭戈分校）
For speech research, ultrasound tongue imaging provides a non-invasive means for visualizing tongue position and movement during articulation. Extracting tongue contours from ultrasound images is a basic step in analyzing ultrasound data but this task often requires non-trivial manual annotation. This study presents an open source tool for fully automatic tracking of tongue contours in ultrasound frames using neural network based methods. We have implemented and systematically compared two convolutional neural networks, U-Net and DenseU-Net, under different conditions. Though both models can p…
AlignUS: MRI-Guided Ultrasound Representation Learning for ALS Classification from Tongue Images
Unité de Recherche en Systèmes Intelligents Avancés (URSIA), Institut Supérieur du Numérique (SupNum)（高级智能系统研究单位（URSIA），数字高等学院（SupNum））；University of Basel（巴塞尔大学）
Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；University of Michigan（密歇根大学）；Columbia University in the City of New York（纽约哥伦比亚大学）；National University of Defense Technology（国防科技大学）
A challenge in speech production research is to predict future tongue movements based on a short period of past tongue movements. This study tackles speaker-dependent tongue motion prediction problem in unlabeled ultrasound videos with convolutional long short-term memory (ConvLSTM) networks. The model has been tested on two different ultrasound corpora. ConvLSTM outperforms 3-dimensional convolutional neural network (3DCNN) in predicting the 9\textsuperscript{th} frames based on 8 preceding frames, and also demonstrates good capacity to predict only the tongue contours in future frames. Furth…
School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）
The progress of deep convolutional neural networks has been successfully exploited in various real-time computer vision tasks such as image classification and segmentation. Owing to the development of computational units, availability of digital datasets, and improved performance of deep learning models, fully automatic and accurate tracking of tongue contours in real-time ultrasound data became practical only in recent years. Recent studies have shown that the performance of deep learning techniques is significant in the tracking of ultrasound tongue contours in real-time applications such as…
One usage of medical ultrasound imaging is to visualize and characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures in applications such as visual training of a second language. Moreover, quantitative analysis of tongue motion needs the tongue dorsum contour to be extracted, tracked, and visualized. Manual tongue contour extraction is a cumbersome, subjective, and error-prone task.…
UltraUNet: Real-Time Ultrasound Tongue Segmentation for Diverse Linguistic and Imaging Conditions
Biomedical Engineering Department（生物医学工程系）；Hong Kong Polytechnic University（香港理工大学）；English and Communication Department（英语与交流系）；Department of Chinese and Bilingual Studies（中文与双语研究系）；Department of Biomedical Engineering Research Institute for Smart Ageing（生物医学工程研究 institute for Smart Ageing）
The University of Tokyo（东京大学）；Southeast University（东南大学）
Speech production is a dynamic procedure, which involved multi human organs including the tongue, jaw and lips. Modeling the dynamics of the vocal tract deformation is a fundamental problem to understand the speech, which is the most common way for human daily communication. Researchers employ several sensory streams to describe the process simultaneously, which are incontrovertibly statistically related to other streams. In this paper, we address the following question: given an observable image sequences of lips, can we picture the corresponding tongue motion. We formulated this problem as t…
University of Edinburgh（爱丁堡大学）；University of Science and Technology of China（中国科学技术大学）；Queen Margaret University（玛格丽特女王大学）；Articulate Instruments Ltd.（Articulate仪器有限公司）
We present the Tongue and Lips corpus (TaL), a multi-speaker corpus of audio, ultrasound tongue imaging, and lip videos. TaL consists of two parts: TaL1 is a set of six recording sessions of one professional voice talent, a male native speaker of English; TaL80 is a set of recording sessions of 81 native speakers of English without voice talent experience. Overall, the corpus contains 24 hours of parallel ultrasound, video, and audio data, of which approximately 13.5 hours are speech. This paper describes the corpus and presents benchmark results for the tasks of speech recognition, speech syn…
School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）
Ultrasound imaging is safe, relatively affordable, and capable of real-time performance. One application of this technology is to visualize and to characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the noisy nature of ultrasound images with low-contrast characteristic, it might require expertise for non-expert users to recognize organ shape such as tongue surface (dorsum). To alleviate this difficulty for quantitative analysis of tongue shape and motion, tongue surface can be extracted, tracked, and visualized instead of …
Semantic contour detection is a challenging problem that is often met in medical imaging, of which placental image analysis is a particular example. In this paper, we investigate utero-placental interface (UPI) detection in 2D placental ultrasound images by formulating it as a semantic contour detection problem. As opposed to natural images, placental ultrasound images contain specific anatomical structures thus have unique geometry. We argue it would be beneficial for UPI detectors to incorporate global context modelling in order to reduce unwanted false positive UPI predictions. Our approach…
Ultrasound tongue imaging (UTI) provides a convenient way to visualize the vocal tract during speech production. UTI is increasingly being used for speech therapy, making it important to develop automatic methods to assist various time-consuming manual tasks currently performed by speech therapists. A key challenge is to generalize the automatic processing of ultrasound tongue images to previously unseen speakers. In this work, we investigate the classification of phonetic segments (tongue shapes) from raw ultrasound recordings under several training scenarios: speaker-dependent, multi-speaker…
Despite renewed awareness of the importance of articulation, it remains a challenge for instructors to handle the pronunciation needs of language learners. There are relatively scarce pedagogical tools for pronunciation teaching and learning. Unlike inefficient, traditional pronunciation instructions like listening and repeating, electronic visual feedback (EVF) systems such as ultrasound technology have been employed in new approaches. Recently, an ultrasound-enhanced multimodal method has been developed for visualizing tongue movements of a language learner overlaid on the face-side of the s…
Left ventricular segmentation is essential for measuring left ventricular function indices. Segmentation of one or several images requires an initial guess of the contour. It is hypothesized here that creating an initial guess by first detecting anatomical markers, would lead to correct detection of the endocardium. The first step of the algorithm presented here includes automatic detection of the mitral valve. Next, the apex is detected in the same frame. The valve is then tracked throughout the cardiac cycle. Contours passing from the apex to each valve corner are then found using a dynamic …
the Third Affiliated Hospital of Sun Yat-Sen University（中山大学附属第三医院）；School of Mathematical Sciences, Zhejiang University（浙江大学数学科学学院）；Henan Provincial People’s Hospital（河南省人民医院）
In this paper, a multi-scale framework with local region based active contour and boundary shape similarity constraint is proposed for the segmentation of levator hiatus in ultrasound images. In this paper, we proposed a multiscale active contour framework to segment levator hiatus ultrasound images by combining the local region information and boundary shape similarity constraint. In order to get more precisely initializations and reduce the computational cost, Gaussian pyramid method is used to decompose the image into coarse-to-fine scales. A localized region active contour model is firstly…
CP-UNet: Contour-based Probabilistic Model for Medical Ultrasound Images Segmentation
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Tianjin Key Laboratory of Cognitive Computing and Application（天津认知计算与应用重点实验室）；Tianjin Key Laboratory of Advanced Networking（天津先进网络重点实验室）；School of Future Technology, Tianjin University（天津大学未来技术学院）
Faculty of Engineering and Applied Sciences, Memorial University（纪念大学工程与应用科学学院）；Faculty of Medicine, Memorial University（纪念大学医学院）
Detection of relative changes in circulating blood volume is important to guide resuscitation and manage a variety of medical conditions including sepsis, trauma, dialysis and congestive heart failure. Recent studies have shown that estimates of circulating blood volume can be obtained from the cross-sectional area (CSA) of the internal jugular vein (IJV) from ultrasound images. However, accurate segmentation and tracking of the IJV in ultrasound imaging is a challenging task and is significantly influenced by a number of parameters such as the image quality, shape, and temporal variation. In …
Carnegie Mellon University（卡内基梅隆大学）；Wake Forest Institute for Regenerative Medicine（维克森林再生医学研究所）
Ultra High Frequency Ultrasound (UHFUS) enables the visualization of highly deformable small and medium vessels in the hand. Intricate vessel-based measurements, such as intimal wall thickness and vessel wall compliance, require sub-millimeter vessel tracking between B-scans. Our fast GPU-based approach combines the advantages of local phase analysis, a distance-regularized level set, and an Extended Kalman Filter (EKF), to rapidly segment and track the deforming vessel contour. We validated on 35 UHFUS sequences of vessels in the hand, and we show the transferability of the approach to 5 more…
University of Toronto（多伦多大学）；Sunnybrook Health Sciences Centre（新宁健康科学中心）；University of Waterloo（滑铁卢大学）
Recent advances in using quantitative ultrasound (QUS) methods have provided a promising framework to non-invasively and inexpensively monitor or predict the effectiveness of therapeutic cancer responses. One of the earliest steps in using QUS methods is contouring a region of interest (ROI) inside the tumour in ultrasound B-mode images. While manual segmentation is a very time-consuming and tedious task for human experts, auto-contouring is also an extremely difficult task for computers due to the poor quality of ultrasound B-mode images. However, for the purpose of cancer response prediction…
Image Retrieval with Intra-Sweep Representation Learning for Neck Ultrasound Scanning Guidance
The University of British Columbia（不列颠哥伦比亚大学）；Intuitive Surgical Inc.（直觉外科公司）；School of Biomedical Engineering, The University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）
In this paper we present a new method for motion tracking of tumors in liver ultrasound image sequences. Our algorithm has two main steps. In the first step, we apply mean shift algorithm with multiple features to estimate the center of the target in each frame. Target in the first frame is defined using an ellipse. Edge, texture, and intensity features are extracted from the first frame, and then mean shift algorithm is applied to each feature separately to find the center of ellipse related to that feature in the next frame. The center of ellipse will be the weighted average of these centers…
The best performance in Air-tissue boundary (ATB) segmentation of real-time Magnetic Resonance Imaging (rtMRI) videos in speech production is known to be achieved by a 3-dimensional convolutional neural network (3D-CNN) model. However, the evaluation of this model, as well as other ATB segmentation techniques reported in the literature, is done using Dynamic Time Warping (DTW) distance between the entire original and predicted contours. Such an evaluation measure may not capture local errors in the predicted contour. Careful analysis of predicted contours reveals errors in regions like the vel…
College of Science, Liaoning Petrochemical University（辽宁石油化工大学理学院）；Industrial Mathematics Laboratory, Baikal School of BRICS, Irkutsk National Research Technical University（伊尔库茨克国立研究技术大学贝加尔金砖学院工业数学实验室）
Biodegradable elastic scaffolds have attracted more and more attention in the field of soft tissue repair and tissue engineering. These scaffolds made of porous bioelastomers support tissue ingrowth along with their own degradation. It is necessary to develop a computer-aided analyzing method based on ultrasound images to identify the degradation performance of the scaffold, not only to obviate the need to do destructive testing, but also to monitor the scaffold's degradation and tissue ingrowth over time. It is difficult using a single traditional image processing algorithm to extract continu…
INSA Centre Val de Loire（卢瓦尔河谷中央国立应用科学学院）；Clinique Medipole Garonne（加龙河医疗 pole 诊所）
Ultrasound-Guided Regional Anesthesia (UGRA) has been gaining importance in the last few years, offering numerous advantages over alternative methods of nerve localization (neurostimulation or paraesthesia). However, nerve detection is one of the most tasks that anaesthetists can encounter in the UGRA procedure. Computer aided system that can detect automatically region of nerve, would help practitioner to concentrate more in anaesthetic delivery. In this paper we propose a new method based on deep learning combined with spatiotemporal information to robustly segment the nerve region. The prop…
Siemens Healthineers（西门子医疗）；Imperial College London（帝国理工学院）
Real-time localization of prostate gland in trans-rectal ultrasound images is a key technology that is required to automate the ultrasound guided prostate biopsy procedures. In this paper, we propose a new deep learning based approach which is aimed at localizing several prostate landmarks efficiently and robustly. We propose a multitask learning approach primarily to make the overall algorithm more contextually aware. In this approach, we not only consider the explicit learning of landmark locations, but also build-in a mechanism to learn the contour of the prostate. This multitask learning i…
An Edge-aware Prompt-enhanced SAM for Ultrasound Image Segmentation
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University（航空信息安全与可信计算重点实验室，教育部，武汉大学计算机科学与工程学院）；National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（多媒體軟體國家工程研究中心，人工智能研究院，武漢大學計算機科學學院，湖北省多媒體與網絡通信工程重點實驗室）
针对超声图像分割中 SAM 边界描绘不佳问题，提出 EP-SAM，通过图像编码器多块特征提取和边缘感知监督，生成高质量提示引导模型，在多个基准测试中表现优于现有基于 SAM 的方法。
Imperial College London（帝国理工学院）；ETH Zurich（苏黎世联邦理工学院）；Guy’s and St Thomas’ NHS Foundation（盖伊和圣托马斯国民保健服务基金会）
Measurement of head biometrics from fetal ultrasonography images is of key importance in monitoring the healthy development of fetuses. However, the accurate measurement of relevant anatomical structures is subject to large inter-observer variability in the clinic. To address this issue, an automated method utilizing Fully Convolutional Networks (FCN) is proposed to determine measurements of fetal head circumference (HC) and biparietal diameter (BPD). An FCN was trained on approximately 2000 2D ultrasound images of the head with annotations provided by 45 different sonographers during routine …
CSTR, School of Informatics, University of Edinburgh, UK（英国爱丁堡大学信息学院语音技术研究中心）
Audiovisual synchronisation is the task of determining the time offset between speech audio and a video recording of the articulators. In child speech therapy, audio and ultrasound videos of the tongue are captured using instruments which rely on hardware to synchronise the two modalities at recording time. Hardware synchronisation can fail in practice, and no mechanism exists to synchronise the signals post hoc. To address this problem, we employ a two-stream neural network which exploits the correlation between the two modalities to find the offset. We train our model on recordings from 69 s…
In this work we propose to segment the prostate on a challenging dataset of trans-rectal ultrasound (TRUS) images using convolutional neural networks (CNNs) and statistical shape models (SSMs). TRUS is commonly used for a number of image-guided interventions on the prostate. Fast and accurate segmentation on the organ in these images is crucial to planning and fusion with other modalities such as magnetic resonance images (MRIs) . However, TRUS has limited soft tissue contrast and signal to noise ratio which makes the task of segmenting the prostate challenging and subject to inter-observer an…
Nanjing University of Science and Technology（南京理工大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Nanjing University（南京大学）
Active contour models have been widely used in image segmentation, and the level set method (LSM) is the most popular approach for solving the models, via implicitly representing the contour by a level set function. However, the LSM suffers from high computational burden and numerical instability, requiring additional regularization terms or re-initialization techniques. In this paper, we use characteristic functions to implicitly represent the contours, propose a new representation to the geodesic active contours and derive an efficient algorithm termed as the iterative convolution-thresholdi…
Beihang University（北京航空航天大学）；XuanWu Hospital, Capital Medical University（首都医科大学宣武医院）；Beijing Diagnostic Center of Vascular Ultrasound（北京血管超声诊断中心）
Carotid arteries vulnerable plaques are a crucial factor in the screening of atherosclerosis by ultrasound technique. However, the plaques are contaminated by various noises such as artifact, speckle noise, and manual segmentation may be time-consuming. This paper proposes an automatic convolutional neural network (CNN) method for plaque segmentation in carotid ultrasound images using a small dataset. First, a parallel network with three independent scale decoders is utilized as our base segmentation network, pyramid dilation convolutions are used to enlarge receptive fields in the three segme…
School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）
Breast ultrasound (BUS) image segmentation plays a crucial role in a computer-aided diagnosis system, which is regarded as a useful tool to help increase the accuracy of breast cancer diagnosis. Recently, many deep learning methods have been developed for segmentation of BUS image and show some advantages compared with conventional region-, model-, and traditional learning-based methods. However, previous deep learning methods typically use skip-connection to concatenate the encoder and decoder, which might not make full fusion of coarse-to-fine features from encoder and decoder. Since the str…
ClickSAM: Fine-tuning Segment Anything Model using click prompts for ultrasound image segmentation
The Hockaday School（霍克黛学校）；Liberty High School（自由高中）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；The University of Texas Southwestern Medical Center at Dallas（德克萨斯大学达拉斯西南医学中心）
ERSR: An Ellipse-constrained pseudo-label refinement and symmetric regularization framework for semi-supervised fetal head segmentation in ultrasound images
School of Software, Northwestern Polytechnical University（软件学院，西北工业大学）；School of Computer Science and Technology, Wuhan University of Science and Technology（计算机科学与技术学院，武汉科技大学）；Department of Computer Science, School of Engineering, Shantou University（计算机科学系，汕大工程学院）；Department of Ultrasound, Peking University Shenzhen Hospital（超声科，北京大学深圳医院）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；Australian Institute of Health Innovation (AIHI), Macquarie University（澳大利亚健康创新研究所（AIHI），麦考瑞大学）；Faculty of Applied Science, Macao Polytechnic University（应用科学学院，澳门理工学院）；School of Computer Software, College of Intelligence and Computing, Tianjin University（计算机软件学院，智能计算学院，天津大学）
Multilevel Perception Boundary-guided Network for Breast Lesion Segmentation in Ultrasound Images
Guizhou University（贵州大学）；College of Computer Science and Technology, Guizhou University（贵州大学计算机科学与技术学院）；Engineering Research Center of Text Computing & Cognitive Intelligence, Ministry of Education（教育部文本计算与认知智能工程研究中心）；Key Laboratory of Intelligent Medical Image Analysis and Precise Diagnosis of Guizhou Province（贵州省智能医学影像分析与精准诊断重点实验室）；State Key Laboratory of Public Big Data（公共大数据国家重点实验室）
3D EAGAN: 3D edge-aware attention generative adversarial network for prostate segmentation in transrectal ultrasound images
Nanchang Hangkong University（南昌航空大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；The First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）
King Abdullah II School for Information Technology, University of Jordan（约旦大学阿卜杜拉二世国王信息技术学院）；Yale University（耶鲁大学）
An important aspect for an improved cardiac functional analysis is the accurate segmentation of the left ventricle (LV). A novel approach for fully-automated segmentation of the LV endocardium and epicardium contours is presented. This is mainly based on the natural physical characteristics of the LV shape structure. Both sides of the LV boundaries exhibit natural elliptical curvatures by having details on various scales, i.e. exhibiting fractal-like characteristics. The fractional Brownian motion (fBm), which is a non-stationary stochastic process, integrates well with the stochastic nature o…
University Hospitals Leuven（鲁汶大学医院）；King’s College London（伦敦国王学院）；KU Leuven（鲁汶大学）；INESC TEC - Institute for Systems and Computer Engineering, Technology and Science（INESC TEC - 系统与计算机工程、技术与科学研究所）
Automatic medical image segmentation via convolutional neural networks (CNNs) has shown promising results. However, they may not always be robust enough for clinical use. Sub-optimal segmentation would require clinician's to manually delineate the target object, causing frustration. To address this problem, a novel interactive CNN-based segmentation framework is proposed in this work. The aim is to represent the CNN segmentation contour as B-splines by utilising B-spline explicit active surfaces (BEAS). The interactive element of the framework allows the user to precisely edit the contour in r…
Medical Image Deidentification, Cleaning and Compression Using Pylogik
Northwestern University（西北大学）；Northwestern University Feinberg School of Medicine（西北大学范伯格医学院）；Northwestern Memorial Hospital（西北纪念医院）；The University of Queensland（昆士兰大学）
EchoXFlow: A Beamspace Echocardiography Dataset for Cardiac Motion, Flow, and Function
Medical Technology & E-Health, Akershus University Hospital, Norway（医疗技术与电子健康，阿克ershus大学医院，挪威）；Department of Technology Systems, University of Oslo, Norway（技术系统系，奥斯陆大学，挪威）；Department of Cardiology, Akershus University Hospital, Norway（心脏病学系，阿克ershus大学医院，挪威）
