In medical imaging, manual annotations can be expensive to acquire and sometimes infeasible to access, making conventional deep learning-based models difficult to scale. As a result, it would be beneficial if useful representations could be derived from raw data without the need for manual annotations. In this paper, we propose to address the problem of self-supervised representation learning with multi-modal ultrasound video-speech raw data. For this case, we assume that there is a high correlation between the ultrasound video and the corresponding narrative speech audio of the sonographer. In order to learn meaningful representations, the model needs to identify such correlation and at the same time understand the underlying anatomical features. We designed a framework to model the correspondence between video and audio without any kind of human annotations. Within this framework, we introduce cross-modal contrastive learning and an affinity-aware self-paced learning scheme to enhance correlation modelling. Experimental evaluations on multi-modal fetal ultrasound video and audio show that the proposed approach is able to learn strong representations and transfers well to downstream tasks of standard plane detection and eye-gaze prediction.
Shenzhen University（深圳大学）；University of Leeds（利兹大学）；KU Leuven（鲁汶大学）；Luohu People’s Hospital（罗湖人民医院）
Standard plane recognition plays an important role in prenatal ultrasound (US) screening. Automatically recognizing the standard plane along with the corresponding anatomical structures in US image can not only facilitate US image interpretation but also improve diagnostic efficiency. In this study, we build a novel multi-label learning (MLL) scheme to identify multiple standard planes and corresponding anatomical structures of fetus simultaneously. Our contribution is three-fold. First, we represent the class correlation by word embeddings to capture the fine-grained semantic and latent stati…
University of Oxford（牛津大学）；Tel-Aviv University（特拉维夫大学）
Self-supervised contrastive representation learning offers the advantage of learning meaningful visual representations from unlabeled medical datasets for transfer learning. However, applying current contrastive learning approaches to medical data without considering its domain-specific anatomical characteristics may lead to visual representations that are inconsistent in appearance and semantics. In this paper, we propose to improve visual representations of medical images via anatomy-aware contrastive learning (AWCL), which incorporates anatomy information to augment the positive/negative pa…
OpenUS: A Fully Open-Source Foundation Model for Ultrasound Image Analysis via Self-Adaptive Masked Contrastive Learning
Digital Environment Research Institute (DERI), Queen Mary University of London（女王玛丽大学数字环境研究 institute（DERI））；The William Harvey Research Institute, Queen Mary University of London（女王玛丽大学威廉·哈维研究 institute）；School of Electronic Engineering and Computer Science, Queen Mary University of London（女王玛丽大学电子工程与计算机科学学院）
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Well-annotated medical datasets enable deep neural networks (DNNs) to gain strong power in extracting lesion-related features. Building such large and well-designed medical datasets is costly due to the need for high-level expertise. Model pre-training based on ImageNet is a common practice to gain better generalization when the data amount is limited. However, it suffers from the domain gap between natural and medical images. In this work, we pre-train DNNs on ultrasound (US) domains instead of ImageNet to reduce the domain gap in medical US applications. To learn US image representations bas…
Intra-video Positive Pairs in Self-Supervised Learning for Ultrasound
Cheriton School of Computer Science（谢里顿计算机科学学院）；University of Waterloo（滑铁卢大学）；Department of Systems Design Engineering（系统设计工程系）；Division of Critical Care Medicine（重症医学科）；Western University（西安大略大学）
Fudan University（复旦大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Shenzhen Research Institute of Big Data（深圳市大数据研究院）；Southern University of Science and Technology（南方科技大学）
Most deep neural networks (DNNs) based ultrasound (US) medical image analysis models use pretrained backbones (e.g., ImageNet) for better model generalization. However, the domain gap between natural and medical images causes an inevitable performance bottleneck. To alleviate this problem, an US dataset named US-4 is constructed for direct pretraining on the same domain. It contains over 23,000 images from four US video sub-datasets. To learn robust features from US-4, we propose an US semi-supervised contrastive learning method, named USCL, for pretraining. In order to avoid high similarities…
SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis
School of Computer Science, Wuhan University, Wuhan, China（武汉大学计算机学院，武汉，中国）；Institute of Artificial Intelligence, Wuhan University, Wuhan, China（武汉大学人工智能研究院，武汉，中国）；Department of Ultrasound, Renmin Hospital of Wuhan University, Wuhan, China（武汉大学仁民医院超声科，武汉，中国）；National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan, China（武汉大学多媒体软件国家工程研究中心，武汉，中国）；Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan, China（湖北省多媒体与网络通信工程重点实验室，武汉大学，武汉，中国）
Hangzhou Dianzi University（杭州电子科技大学）；the Second Affiliated Hospital of Zhejiang University（浙江大学医学院附属第二医院）；Waseda University（早稻田大学）；Communication University of Zhejiang（浙江传媒学院）
Precise and rapid categorization of images in the B-scan ultrasound modality is vital for diagnosing ocular diseases. Nevertheless, distinguishing various diseases in ultrasound still challenges experienced ophthalmologists. Thus a novel contrastive disentangled network (CDNet) is developed in this work, aiming to tackle the fine-grained image categorization (FGIC) challenges of ocular abnormalities in ultrasound images, including intraocular tumor (IOT), retinal detachment (RD), posterior scleral staphyloma (PSS), and vitreous hemorrhage (VH). Three essential components of CDNet are the weakl…
Semi-Supervised Dual-Threshold Contrastive Learning for Ultrasound Image Classification and Segmentation
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China（深圳大学计算机科学与软件工程学院）；Department of Thyroid and Breast Surgery, Affiliated Hospital Group of Guangdong Medical University Shenzhen Baoan Central Hospital, Shenzhen, China（广东医科大学深圳宝安中心医院甲状腺乳腺外科）
Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding
Hangzhou City University（杭州城市大学）；Hong Kong Baptist University（香港浸会大学）；Zhejiang University（浙江大学）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；City University of Hong Kong（香港城市大学）
Prior-guided Hierarchical Instance-pixel Contrastive Learning for Ultrasound Speckle Noise Suppression
School of Biological Science and Medical Engineering, Southeast University（生物科学与医学工程学院，东南大学）；Department of Biomedical Engineering, The Ohio State University（生物医学工程系，奥斯特大学）；Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University（江苏省生物材料与器件重点实验室，东南大学）；State Key Laboratory of Digital Medical Engineering, Southeast University（国家数字医学工程重点实验室，东南大学）
Towards Objective Obstetric Ultrasound Assessment: Contrastive Representation Learning for Fetal Movement Detection
Department of Electrical and Computer Systems Engineering, Monash University（莫纳什大学电气与计算机系统工程系）；AIM for Health Lab, Faculty of Information Technology, Monash University（健康AIM实验室，信息技术学院，莫纳什大学）；School of Biomedical Engineering, University of Sydney（悉尼大学生物医学工程学院）；Department of Chemical & Biological Engineering, Monash University（莫纳什大学化学与生物工程系）；Department of Obstetrics & Gynaecology, Monash University（莫纳什大学妇产科系）
Shenzhen University（深圳大学）；Sun Yat-Sen Memorial Hospital of Sun Yat-Sen University（中山大学孙逸仙纪念医院）
Breast cancer is one of the leading causes of cancer deaths in women. As the primary output of breast screening, breast ultrasound (US) video contains exclusive dynamic information for cancer diagnosis. However, training models for video analysis is non-trivial as it requires a voluminous dataset which is also expensive to annotate. Furthermore, the diagnosis of breast lesion faces unique challenges such as inter-class similarity and intra-class variation. In this paper, we propose a pioneering approach that directly utilizes US videos in computer-aided breast cancer diagnosis. It leverages ma…
National University of Defense Technology（国防科技大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Tongji University（同济大学）
Ultrasound tongue imaging is widely used for speech production research, and it has attracted increasing attention as its potential applications seem to be evident in many different fields, such as the visual biofeedback tool for second language acquisition and silent speech interface. Unlike previous studies, here we explore the feasibility of age estimation using the ultrasound tongue image of the speakers. Motivated by the success of deep learning, this paper leverages deep learning on this task. We train a deep convolutional neural network model on the UltraSuite dataset. The deep model ac…
Shanghai Jiaotong University（上海交通大学）；CloudWalk Technology Co., Ltd（云从科技有限公司）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
The self-supervised ultrasound (US) video model pretraining can use a small amount of labeled data to achieve one of the most promising results on US diagnosis. However, it does not take full advantage of multi-level knowledge for learning deep neural networks (DNNs), and thus is difficult to learn transferable feature representations. This work proposes a hierarchical contrastive learning (HiCo) method to improve the transferability for the US video model pretraining. HiCo introduces both peer-level semantic alignment and cross-level semantic alignment to facilitate the interaction between di…
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；Langevin Institute, ESPCI-ParisTech（巴黎高等物理化工学院朗之万研究所）；University of New Mexico（新墨西哥大学）；Tianjin University（天津大学）
This article describes a contour-based 3D tongue deformation visualization framework using B-mode ultrasound image sequences. A robust, automatic tracking algorithm characterizes tongue motion via a contour, which is then used to drive a generic 3D Finite Element Model (FEM). A novel contour-based 3D dynamic modeling method is presented. Modal reduction and modal warping techniques are applied to model the deformation of the tongue physically and efficiently. This work can be helpful in a variety of fields, such as speech production, silent speech recognition, articulation training, speech dis…
Carnegie Mellon University（卡内基梅隆大学）；Louisiana State University（路易斯安那州立大学）；Cosmeticsurg.net, LLC（Cosmeticsurg.net有限责任公司）
With the onset of the COVID-19 pandemic, ultrasound has emerged as an effective tool for bedside monitoring of patients. Due to this, a large amount of lung ultrasound scans have been made available which can be used for AI based diagnosis and analysis. Several AI-based patient severity scoring models have been proposed that rely on scoring the appearance of the ultrasound scans. AI models are trained using ultrasound-appearance severity scores that are manually labeled based on standardized visual features. We address the challenge of labeling every ultrasound frame in the video clips. Our co…
Ultrasound tongue imaging (UTI) provides a convenient way to visualize the vocal tract during speech production. UTI is increasingly being used for speech therapy, making it important to develop automatic methods to assist various time-consuming manual tasks currently performed by speech therapists. A key challenge is to generalize the automatic processing of ultrasound tongue images to previously unseen speakers. In this work, we investigate the classification of phonetic segments (tongue shapes) from raw ultrasound recordings under several training scenarios: speaker-dependent, multi-speaker…
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；SIGnal processing and MAchine learning Lab, ESPCI ParisTech, PSL Research University（巴黎高等物理化工学院信号处理与机器学习实验室，巴黎文理研究大学）；Vocal Tract Visualization Lab, University of Maryland Dental School（马里兰大学牙科学院声道可视化实验室）
Studying tongue motion during speech using ultrasound is a standard procedure, but automatic ultrasound image labelling remains a challenge, as standard tongue shape extraction methods typically require human intervention. This article presents a method based on deep neural networks to automatically extract tongue contour from ultrasound images on a speech dataset. We use a deep autoencoder trained to learn the relationship between an image and its related contour, so that the model is able to automatically reconstruct contours from the ultrasound image alone. In this paper, we use an automati…
Ultromics Ltd.（奥特罗米克斯有限公司）；Metalynx Ltd.（梅塔林克斯有限公司）；Imperial College London（帝国理工学院）；King’s College London（伦敦国王学院）
Analysis of cardiac ultrasound images is commonly performed in routine clinical practice for quantification of cardiac function. Its increasing automation frequently employs deep learning networks that are trained to predict disease or detect image features. However, such models are extremely data-hungry and training requires labelling of many thousands of images by experienced clinicians. Here we propose the use of contrastive learning to mitigate the labelling bottleneck. We train view classification models for imbalanced cardiac ultrasound datasets and show improved performance for views/cl…
University of Edinburgh（爱丁堡大学）；University of Strathclyde（思克莱德大学）；Queen Margaret University（玛格丽特女王大学）；Articulate Instruments Ltd.（Articulate仪器有限公司）
We introduce UltraSuite, a curated repository of ultrasound and acoustic data, collected from recordings of child speech therapy sessions. This release includes three data collections, one from typically developing children and two from children with speech sound disorders. In addition, it includes a set of annotations, some manual and some automatically produced, and software tools to process, transform and visualise the data.
Pyramid Self-Contrastive Learning for Single-shot Test-time Ultrasound Image Denoising
Department of Electrical and Computer Engineering, The University of Hong Kong（香港大学电子与计算机工程系）；Department of Biomedical Engineering, Duke University（达特茅斯大学生物医学工程系）
Image Retrieval with Intra-Sweep Representation Learning for Neck Ultrasound Scanning Guidance
The University of British Columbia（不列颠哥伦比亚大学）；Intuitive Surgical Inc.（直觉外科公司）；School of Biomedical Engineering, The University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）
Indian Institute of Technology, Delhi（德里印度理工学院）；Postgraduate Institute of Medical Education and Research（医学教育与研究研究生院）
Rich temporal information and variations in viewpoints make video data an attractive choice for learning image representations using unsupervised contrastive learning (UCL) techniques. State-of-the-art (SOTA) contrastive learning techniques consider frames within a video as positives in the embedding space, whereas the frames from other videos are considered negatives. We observe that unlike multiple views of an object in natural scene videos, an Ultrasound (US) video captures different 2D slices of an organ. Hence, there is almost no similarity between the temporally distant frames of even th…
School of Biomedical Engineering, Health Center, Shenzhen University（深圳大学生物医学工程学院、健康中心）；Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University（深圳大学医学超声图像计算（MUSIC）实验室）；School of Biological Sciences and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）
Ultrasound (US) image segmentation embraced its significant improvement in deep learning era. However, the lack of sharp boundaries in US images still remains an inherent challenge for segmentation. Previous methods often resort to global context, multi-scale cues or auxiliary guidance to estimate the boundaries. It is hard for these methods to approach pixel-level learning for fine-grained boundary generating. In this paper, we propose a novel and effective framework to improve boundary estimation in US images. Our work has three highlights. First, we propose to formulate the boundary estimat…
University of Edinburgh（爱丁堡大学）；University of Science and Technology of China（中国科学技术大学）；Queen Margaret University（玛格丽特女王大学）；Articulate Instruments Ltd.（Articulate仪器有限公司）
We present the Tongue and Lips corpus (TaL), a multi-speaker corpus of audio, ultrasound tongue imaging, and lip videos. TaL consists of two parts: TaL1 is a set of six recording sessions of one professional voice talent, a male native speaker of English; TaL80 is a set of recording sessions of 81 native speakers of English without voice talent experience. Overall, the corpus contains 24 hours of parallel ultrasound, video, and audio data, of which approximately 13.5 hours are speech. This paper describes the corpus and presents benchmark results for the tasks of speech recognition, speech syn…
Multiscale Switch for Semi-Supervised and Contrastive Learning in Medical Ultrasound Image Segmentation
Department of Health Technology and Informatics, The Hong Kong Polytechnic University（健康科技与信息学系，香港理工大学）；Centre for Smart Health and School of Nursing, The Hong Kong Polytechnic University（智能健康中心及护理学院，香港理工大学）；Department of Imaging and Interventional Radiology, The Chinese University of Hong Kong（影像与介入放射科，香港中文大学）
University of Michigan（密歇根大学）；University of California San Diego（加州大学圣迭戈分校）
For speech research, ultrasound tongue imaging provides a non-invasive means for visualizing tongue position and movement during articulation. Extracting tongue contours from ultrasound images is a basic step in analyzing ultrasound data but this task often requires non-trivial manual annotation. This study presents an open source tool for fully automatic tracking of tongue contours in ultrasound frames using neural network based methods. We have implemented and systematically compared two convolutional neural networks, U-Net and DenseU-Net, under different conditions. Though both models can p…
Goal-conditioned reinforcement learning for ultrasound navigation guidance
King’s College London（伦敦国王学院）；Siemens Healthcare Limited（西门子医疗有限公司）；Siemens Healthineers（西门子医疗）；Siemens Healthineers AG（西门子医疗股份公司）；Carol Davila University of Medicine and Pharmacy Bucharest（布加勒斯特卡罗尔·达维拉医药大学）；Fortis Institute of Medical Sciences（富蒂斯医学院）；Guy’s and St Thomas’ NHS Foundation Trust（盖伊和圣托马斯国民保健信托基金）
UltraUNet: Real-Time Ultrasound Tongue Segmentation for Diverse Linguistic and Imaging Conditions
Biomedical Engineering Department（生物医学工程系）；Hong Kong Polytechnic University（香港理工大学）；English and Communication Department（英语与交流系）；Department of Chinese and Bilingual Studies（中文与双语研究系）；Department of Biomedical Engineering Research Institute for Smart Ageing（生物医学工程研究 institute for Smart Ageing）
The University of Tokyo（东京大学）；Southeast University（东南大学）
Speech production is a dynamic procedure, which involved multi human organs including the tongue, jaw and lips. Modeling the dynamics of the vocal tract deformation is a fundamental problem to understand the speech, which is the most common way for human daily communication. Researchers employ several sensory streams to describe the process simultaneously, which are incontrovertibly statistically related to other streams. In this paper, we address the following question: given an observable image sequences of lips, can we picture the corresponding tongue motion. We formulated this problem as t…
One usage of medical ultrasound imaging is to visualize and characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures in applications such as visual training of a second language. Moreover, quantitative analysis of tongue motion needs the tongue dorsum contour to be extracted, tracked, and visualized. Manual tongue contour extraction is a cumbersome, subjective, and error-prone task.…
CSTR, School of Informatics, University of Edinburgh, UK（英国爱丁堡大学信息学院语音技术研究中心）
Audiovisual synchronisation is the task of determining the time offset between speech audio and a video recording of the articulators. In child speech therapy, audio and ultrasound videos of the tongue are captured using instruments which rely on hardware to synchronise the two modalities at recording time. Hardware synchronisation can fail in practice, and no mechanism exists to synchronise the signals post hoc. To address this problem, we employ a two-stream neural network which exploits the correlation between the two modalities to find the offset. We train our model on recordings from 69 s…
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Speech emotion recognition is an important aspect of human-computer interaction. Prior work proposes various end-to-end models to improve the classification performance. However, most of them rely on the cross-entropy loss together with softmax as the supervision component, which does not explicitly encourage discriminative learning of features. In this paper, we introduce the contrastive loss function to encourage intra-class compactness and inter-class separability between learnable features. Furthermore, multiple feature selection methods and pairwise sample selection methods are evaluated.…
Using Spatio-Temporal Dual-Stream Network with Self-Supervised Learning for Lung Tumor Classification on Radial Probe Endobronchial Ultrasound Video
National Yang Ming Chiao Tung University（国立阳明交通大学）；National Taiwan University Cancer Center（台湾大学癌症中心）；National Taiwan University Hospital（台湾大学医院）；National Taiwan University Hsin-Chu Hospital（台湾大学新竹医院）
School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）
Ultrasound imaging is safe, relatively affordable, and capable of real-time performance. One application of this technology is to visualize and to characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the noisy nature of ultrasound images with low-contrast characteristic, it might require expertise for non-expert users to recognize organ shape such as tongue surface (dorsum). To alleviate this difficulty for quantitative analysis of tongue shape and motion, tongue surface can be extracted, tracked, and visualized instead of …
Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；University of Michigan（密歇根大学）；Columbia University in the City of New York（纽约哥伦比亚大学）；National University of Defense Technology（国防科技大学）
A challenge in speech production research is to predict future tongue movements based on a short period of past tongue movements. This study tackles speaker-dependent tongue motion prediction problem in unlabeled ultrasound videos with convolutional long short-term memory (ConvLSTM) networks. The model has been tested on two different ultrasound corpora. ConvLSTM outperforms 3-dimensional convolutional neural network (3DCNN) in predicting the 9\textsuperscript{th} frames based on 8 preceding frames, and also demonstrates good capacity to predict only the tongue contours in future frames. Furth…
Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation
University of Amsterdam（阿姆斯特丹大学）；Maastricht University（马斯特里赫特大学）；Radboud University（拉德堡德大学）；Meertens Institute（米滕斯研究所）；MPI for Psycholinguistics（马克斯·普朗克心理语言学研究所）
