Anatomical landmarks are a crucial prerequisite for many medical imaging tasks. Usually, the set of landmarks for a given task is predefined by experts. The landmark locations for a given image are then annotated manually or via machine learning methods trained on manual annotations. In this paper, in contrast, we present a method to automatically discover and localize anatomical landmarks in medical images. Specifically, we consider landmarks that attract the visual attention of humans, which we term visually salient landmarks. We illustrate the method for fetal neurosonographic images. First, full-length clinical fetal ultrasound scans are recorded with live sonographer gaze-tracking. Next, a convolutional neural network (CNN) is trained to predict the gaze point distribution (saliency map) of the sonographers on scan video frames. The CNN is then used to predict saliency maps of unseen fetal neurosonographic images, and the landmarks are extracted as the local maxima of these saliency maps. Finally, the landmarks are matched across images by clustering the landmark CNN features. We show that the discovered landmarks can be used within affine image registration, with average landmark alignment errors between 4.1% and 10.9% of the fetal head long axis length.
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；School of Computing, University of Utah（犹他大学计算学院）
Statistical shape analysis is a very useful tool in a wide range of medical and biological applications. However, it typically relies on the ability to produce a relatively small number of features that can capture the relevant variability in a population. State-of-the-art methods for obtaining such anatomical features rely on either extensive preprocessing or segmentation and/or significant tuning and post-processing. These shortcomings limit the widespread use of shape statistics. We propose that effective shape representations should provide sufficient information to align/register images. …
Research Institute for Signals, Systems and Computational Intelligence, sinc(i)（信号、系统与计算智能研究所）；CONICET（国家科学技术研究委员会）；Universidad Nacional del Litoral（国立滨海大学）
In this work we address the problem of landmark-based segmentation for anatomical structures. We propose HybridGNet, an encoder-decoder neural architecture which combines standard convolutions for image feature encoding, with graph convolutional neural networks to decode plausible representations of anatomical structures. We benchmark the proposed architecture considering other standard landmark and pixel-based models for anatomical segmentation in chest x-ray images, and found that HybridGNet is more robust to image occlusions. We also show that it can be used to construct landmark-based segm…
University College London（伦敦大学学院）；University of Oxford（牛津大学）
One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-t…
Recently, there is an increasing demand for automatically detecting anatomical landmarks which provide rich structural information to facilitate subsequent medical image analysis. Current methods related to this task often leverage the power of deep neural networks, while a major challenge in fine tuning such models in medical applications arises from insufficient number of labeled samples. To address this, we propose to regularize the knowledge transfer across source and target tasks through cross-task representation learning. The proposed method is demonstrated for extracting facial anatomic…
ETH Zurich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our meth…
Is Geometry Enough? An Evaluation of Landmark-Based Gaze Estimation
Department of Industrial Engineering and Mathematical Sciences, Università Politecnica delle Marche（工业工程与数学科学系，帕尔米塞大学）；Department of Science and Information Technology, Università Pegaso（科学与信息科技系，佩加索大学）
landmarker: a Toolkit for Anatomical Landmark Localization in 2D/3D Images
IDLab；Department of Electronics and Information Systems（电子与信息系统系）；Ghent University（根特大学）；imec；Biometrics Research Group（生物识别研究组）；Department of Morphology, Imaging, Orthopedics, Rehabilitation and Nutrition（解剖学、成像、骨科、康复和营养系）
INRIA Sophia Antipolis - Méditerranée（法国国家信息与自动化研究所索菲亚·安提波利斯-地中海研究中心）；Birla Institute of Technology and Science, Pilani（皮拉尼·博拉理工学院）；Université Côte d’Azur（蔚蓝海岸大学）
3D gaze estimation is about predicting the line of sight of a person in 3D space. Person-independent models for the same lack precision due to anatomical differences of subjects, whereas person-specific calibrated techniques add strict constraints on scalability. To overcome these issues, we propose a novel technique, Facial Landmark Heatmap Activated Multimodal Gaze Estimation (FLAME), as a way of combining eye anatomical information using eye landmark heatmaps to obtain precise gaze estimation without any person-specific calibration. Our evaluation demonstrates a competitive performance of a…
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡全国眼科中心）；Yong Loo Lin School of Medicine, NUS（新加坡国立大学杨潞龄医学院）
Localizing anatomical landmarks are important tasks in medical image analysis. However, the landmarks to be localized often lack prominent visual features. Their locations are elusive and easily confused with the background, and thus precise localization highly depends on the context formed by their surrounding areas. In addition, the required precision is usually higher than segmentation and object detection tasks. Therefore, localization has its unique challenges different from segmentation or detection. In this paper, we propose a zoom-in attentive network (ZIAN) for anatomical landmark loc…
3D anatomical landmarks play an important role in health research. Their automated prediction/localization thus becomes a vital task. In this paper, we introduce a deformation method for 3D anatomical landmarks prediction. It utilizes a source model with anatomical landmarks which are annotated by clinicians, and deforms this model non-rigidly to match the target model. Two constraints are introduced in the optimization, which are responsible for alignment and smoothness, respectively. Experiments are performed on our dataset and the results demonstrate the robustness of our method, and show t…
Fetal alcohol syndrome (FAS) caused by prenatal alcohol exposure can result in a series of cranio-facial anomalies, and behavioral and neurocognitive problems. Current diagnosis of FAS is typically done by identifying a set of facial characteristics, which are often obtained by manual examination. Anatomical landmark detection, which provides rich geometric information, is important to detect the presence of FAS associated facial anomalies. This imaging application is characterized by large variations in data appearance and limited availability of labeled data. Current deep learning-based heat…
LmPT: Conditional Point Transformer for Anatomical Landmark Detection on 3D Point Clouds
Mines Paris - PSL University, Centre for material sciences (MAT)（巴黎 Mines - PSL 大学材料科学中心）；Mines Paris - PSL University, Centre for mathematical morphology (CMM)（巴黎 Mines - PSL 大学数学形态学中心）；Mines Paris - PSL University, Centre for material forming (CEMEF)（巴黎 Mines - PSL 大学材料成形中心）；The National Veterinary School of Alfort (EnvA)（阿尔福国家兽医学校）
University of Central Florida（中佛罗里达大学）；Ther-AI, LLC（Ther-AI有限责任公司）；National Institute of Dental and Craniofacial Research (NIDCR), National Institutes of Health (NIH)（美国国立卫生研究院下属国立牙科与颅面研究所）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Northwestern University（西北大学）；Sheikh Zyed Children’s Hospital（谢赫·扎耶德儿童医院）；George Washington University（乔治·华盛顿大学）
Purpose: We perform anatomical landmarking for craniomaxillofacial (CMF) bones without explicitly segmenting them. Towards this, we propose a new simple yet efficient deep network architecture, called \textit{relational reasoning network (RRN)}, to accurately learn the local and the global relations among the landmarks in CMF bones; specifically, mandible, maxilla, and nasal bones. Approach: The proposed RRN works in an end-to-end manner, utilizing learned relations of the landmarks based on dense-block units. For a given few landmarks as input, RRN treats the landmarking process similar to a …
TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types
We explore techniques for eye gaze estimation using machine learning. Eye gaze estimation is a common problem for various behavior analysis and human-computer interfaces. The purpose of this work is to discuss various model types for eye gaze estimation and present the results from predicting gaze direction using eye landmarks in unconstrained settings. In unconstrained real-world settings, feature-based and model-based methods are outperformed by recent appearance-based methods due to factors like illumination changes and other visual artifacts. We discuss a learning-based method for eye regi…
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；Air Force Medical Center, PLA（中国人民解放军空军特色医学中心）
Colonoscopy is a standard imaging tool for visualizing the entire gastrointestinal (GI) tract of patients to capture lesion areas. However, it takes the clinicians excessive time to review a large number of images extracted from colonoscopy videos. Thus, automatic detection of biological anatomical landmarks within the colon is highly demanded, which can help reduce the burden of clinicians by providing guidance information for the locations of lesion areas. In this article, we propose a novel deep learning-based approach to detect biological anatomical landmarks in colonoscopy videos. First, …
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Detecting anatomical landmarks in medical images plays an essential role in understanding the anatomy and planning automated processing. In recent years, a variety of deep neural network methods have been developed to detect landmarks automatically. However, all of those methods are unary in the sense that a highly specialized network is trained for a single task say associated with a particular anatomical region. In this work, for the first time, we investigate the idea of "You Only Learn Once (YOLO)" and develop a universal anatomical landmark detection model to realize multiple landmark det…
National Institute of Dental and Craniofacial Research (NIDCR), National Institutes of Health (NIH)（美国国立卫生研究院（NIH）国立牙科与颅面研究所（NIDCR））；University of South Dakota Sanford School of Medicine（南达科他大学桑福德医学院）；University of Central Florida（中佛罗里达大学）
In this paper, we propose a novel deep learning framework for anatomy segmentation and automatic landmark- ing. Specifically, we focus on the challenging problem of mandible segmentation from cone-beam computed tomography (CBCT) scans and identification of 9 anatomical landmarks of the mandible on the geodesic space. The overall approach employs three inter-related steps. In step 1, we propose a deep neu- ral network architecture with carefully designed regularization, and network hyper-parameters to perform image segmentation without the need for data augmentation and complex post- processing…
Automatic and accurate detection of anatomical landmarks is an essential operation in medical image analysis with a multitude of applications. Recent deep learning methods have improved results by directly encoding the appearance of the captured anatomy with the likelihood maps (i.e., heatmaps). However, most current solutions overlook another essence of heatmap regression, the objective metric for regressing target heatmaps and rely on hand-crafted heuristics to set the target precision, thus being usually cumbersome and task-specific. In this paper, we propose a novel learning-to-learn frame…
Canon Medical Research Europe（佳能医疗欧洲研究中心）；Queen Elizabeth University Hospital（伊丽莎白女王大学医院）；University of Glasgow（格拉斯哥大学）
We present an efficient neural network method for locating anatomical landmarks in 3D medical CT scans, using atlas location autocontext in order to learn long-range spatial context. Location predictions are made by regression to Gaussian heatmaps, one heatmap per landmark. This system allows patchwise application of a shallow network, thus enabling multiple volumetric heatmaps to be predicted concurrently without prohibitive GPU memory requirements. Further, the system allows inter-landmark spatial relationships to be exploited using a simple overdetermined affine mapping that is robust to de…
University of Oulu（奥卢大学）；Oulu University Hospital（奥卢大学医院）；Aalto University（阿尔托大学）
This paper addresses the challenge of localization of anatomical landmarks in knee X-ray images at different stages of osteoarthritis (OA). Landmark localization can be viewed as regression problem, where the landmark position is directly predicted by using the region of interest or even full-size images leading to large memory footprint, especially in case of high resolution medical images. In this work, we propose an efficient deep neural networks framework with an hourglass architecture utilizing a soft-argmax layer to directly predict normalized coordinates of the landmark points. We provi…
The University of Hong Kong（香港大学）；ShanghaiTech University（上海科技大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；South China University of Technology（华南理工大学）
Well-annotated medical images are costly and sometimes even impossible to acquire, hindering landmark detection accuracy to some extent. Semi-supervised learning alleviates the reliance on large-scale annotated data by exploiting the unlabeled data to understand the population structure of anatomical landmarks. The global shape constraint is the inherent property of anatomical landmarks that provides valuable guidance for more consistent pseudo labelling of the unlabeled data, which is ignored in the previously semi-supervised methods. In this paper, we propose a model-agnostic shape-regulated…
Deploying the idea of long-term cumulative return, reinforcement learning has shown remarkable performance in various fields. We propose a formulation of the landmark localization in 3D medical images as a reinforcement learning problem. Whereas value-based methods have been widely used to solve similar problems, we adopt an actor-critic based direct policy search method framed in a temporal difference learning approach. Successful behavior learning is challenging in large state and/or action spaces, requiring many trials. We introduce a partial policy-based reinforcement learning to enable so…
University of Basel（巴塞尔大学）；University Hospital Basel（巴塞尔大学医院）；MIAC AG
We present an automated method for localizing an anatomical landmark in three-dimensional medical images. The method combines two recurrent neural networks in a coarse-to-fine approach: The first network determines a candidate neighborhood by analyzing the complete given image volume. The second network localizes the actual landmark precisely and accurately in the candidate neighborhood. Both networks take advantage of multi-dimensional gated recurrent units in their main layers, which allow for high model complexity with a comparatively small set of parameters. We localize the medullopontine …
Model-based eye tracking has been a dominant approach for eye gaze tracking because of its ability to generalize to different subjects, without the need of any training data and eye gaze annotations. Model-based eye tracking, however, is susceptible to eye feature detection errors, in particular for eye tracking in the wild. To address this issue, we propose a Bayesian framework for model-based eye tracking. The proposed system consists of a cascade-Bayesian Convolutional Neural Network (c-BCNN) to capture the probabilistic relationships between eye appearance and its landmarks, and a geometri…
Image-guided radiation therapy can benefit from accurate motion tracking by ultrasound imaging, in order to minimize treatment margins and radiate moving anatomical targets, e.g., due to breathing. One way to formulate this tracking problem is the automatic localization of given tracked anatomical landmarks throughout a temporal ultrasound sequence. For this, we herein propose a fully-convolutional Siamese network that learns the similarity between pairs of image regions containing the same landmark. Accordingly, it learns to localize and thus track arbitrary image features, not only predefine…
T2ID-CAS: Diffusion Model and Class Aware Sampling to Mitigate Class Imbalance in Neck Ultrasound Anatomical Landmark Detection
Department of Computer Science, University of Central Florida（计算机科学系，中央佛罗里达大学）；Department of Otolaryngology Head Neck Surgery, University of Cincinnati College of Medicine（耳鼻喉科与头颈外科系，辛辛那提大学医学院）；Department of Internal Medicine, University of Central Florida College of Medicine（内科系，中央佛罗里达大学医学院）
X-ray image guidance enables percutaneous alternatives to complex procedures. Unfortunately, the indirect view onto the anatomy in addition to projective simplification substantially increase the task-load for the surgeon. Additional 3D information such as knowledge of anatomical landmarks can benefit surgical decision making in complicated scenarios. Automatic detection of these landmarks in transmission imaging is challenging since image-domain features characteristic to a certain landmark change substantially depending on the viewing direction. Consequently and to the best of our knowledge,…
Nanjing Medical University（南京医科大学）；Guangdong Women and Children Hospital（广东省妇幼保健院）；Shenzhen University（深圳大学）
The ultrasound (US) screening of the infant hip is vital for the early diagnosis of developmental dysplasia of the hip (DDH). The US diagnosis of DDH refers to measuring alpha and beta angles that quantify hip joint development. These two angles are calculated from key anatomical landmarks and structures of the hip. However, this measurement process is not trivial for sonographers and usually requires a thorough understanding of complex anatomical structures. In this study, we propose a multi-task framework to learn the relationships among landmarks and structures jointly and automatically eva…
University Medical Center Utrecht（乌得勒支大学医学中心）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学阿姆斯特丹医学中心）；University of Twente（特文特大学）；Wageningen University（瓦赫宁根大学）
In this study, we propose a fast and accurate method to automatically localize anatomical landmarks in medical images. We employ a global-to-local localization approach using fully convolutional neural networks (FCNNs). First, a global FCNN localizes multiple landmarks through the analysis of image patches, performing regression and classification simultaneously. In regression, displacement vectors pointing from the center of image patches towards landmark locations are determined. In classification, presence of landmarks of interest in the patch is established. Global landmark locations are o…
University of Electronic Science and Technology of China（电子科技大学）
Localization of anatomical landmarks is essential for clinical diagnosis, treatment planning, and research. In this paper, we propose a novel deep network, named feature aggregation and refinement network (FARNet), for the automatic detection of anatomical landmarks. To alleviate the problem of limited training data in the medical domain, our network adopts a deep network pre-trained on natural images as the backbone network and several popular networks have been compared. Our FARNet also includes a multi-scale feature aggregation module for multi-scale feature fusion and a feature refinement …
FM-OSD: Foundation Model-Enabled One-Shot Detection of Anatomical Landmarks
The Chinese University of Hong Kong（香港中文大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Massachusetts General Brigham（麻省总医院布里格姆医疗系统）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与XR研究所）
Imperial College London（伦敦帝国理工学院）；King’s College London（伦敦国王学院）
We propose a new Patch-based Iterative Network (PIN) for fast and accurate landmark localisation in 3D medical volumes. PIN utilises a Convolutional Neural Network (CNN) to learn the spatial relationship between an image patch and anatomical landmark positions. During inference, patches are repeatedly passed to the CNN until the estimated landmark position converges to the true landmark location. PIN is computationally efficient since the inference stage only selectively samples a small number of patches in an iterative fashion rather than a dense sampling at every location in the volume. Our …
CheXmask-U: Quantifying uncertainty in landmark-based anatomical segmentation for X-ray images
Laboratory of Applied Artificial Intelligence, Institute of Computer Sciences, CONICET - Universidad de Buenos Aires（应用人工智能实验室，计算机科学研究所，CONICET - 华盛顿大学布宜诺斯艾利斯分校）；Weizmann Institute of Science（科学研究所）；APOLO Biotech（APOLO生物技术）；Research Institute for Signals, Systems and Computational Intelligence, sinc(i), CONICET - Universidad Nacional del Litoral（信号、系统和计算智能研究 institute，sinc(i)，CONICET - 沿海国立大学）
Kyoto University（京都大学）；Tohoku University（东北大学）；Tokyo Institute of Technology（东京工业大学）；Nagoya University（名古屋大学）
In video-assisted thoracoscopic surgeries, successful procedures of nodule resection are highly dependent on the precise estimation of lung deformation between the inflated lung in the computed tomography (CT) images during preoperative planning and the deflated lung in the treatment views during surgery. Lungs in the pneumothorax state during surgery have a large volume change from normal lungs, making it difficult to build a mechanical model. The purpose of this study is to develop a deformation estimation method of the 3D surface of a deflated lung from a few partial observations. To estima…
The various speech sounds of a language are obtained by varying the shape and position of the articulators surrounding the vocal tract. Analyzing their variations is crucial for understanding speech production, diagnosing speech disorders and planning therapy. Identifying key anatomical landmarks of these structures on medical images is a pre-requisite for any quantitative analysis and the rising amount of data generated in the field calls for an automatic solution. The challenge lies in the high inter- and intra-speaker variability, the mutual interaction between the articulators and the mode…
Uncertainty-Aware Mapping from 3D Keypoints to Anatomical Landmarks for Markerless Biomechanics
Department of Electrical and Information Engineering, University of Cassino and Southern Lazio（卡西诺和南拉齐奥大学电气与信息工程系）；Department of Research and Development, LUNEX（LUNEX研发部）
Mask-HybridGNet: Graph-based segmentation with emergent anatomical correspondence from pixel-level supervision
Biomedical Image Technologies, ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain（生物医学影像技术，电信工程学院，马德里理工大学，马德里，西班牙）；IHU PRISM National Precision Medicine Center in Oncology, Cancer Data Science Unit, CentraleSupélec, Gustave Roussy, INSERM, Université Paris-Saclay, Gif-sur-Yvette, France（IHU PRISM国家精准医学中心（肿瘤学），癌症数据科学单元，中央圣艾尔布兰，格斯塔夫·罗西，国家医学研究所，巴黎-萨克雷大学，法国）
Coordinated Transformer with Position \& Sample-aware Central Loss for Anatomical Landmark Detection
Case Western Reserve University（凯斯西储大学）；The Second Affiliated Hospital of Anhui Medical University（安徽医科大学第二附属医院）；Institute of Orthopaedics, Research Center for Translational Medicine, The Second Affiliated Hospital of Anhui Medical University（安徽医科大学第二附属医院骨科研究所、转化医学研究中心）；China University of Petroleum（中国石油大学）；Guangzhou Twelfth people’s Hospital（广州市第十二人民医院）；Guangzhou Occupational Disease Prevention and Treatment Hospital（广州市职业病防治院）；Guangzhou Otolaryngology-head and Neck Surgery Hospital（广州市耳鼻咽喉头颈外科医院）；West China Hospital（华西医院）；Renmin Hospital of Wuhan University（武汉大学人民医院）
Automatic Landmark-Based Segmentation of Human Subcortical Structures in MRI
École de technologie supérieure (ÉTS)（埃克塞尔技术高等学院）；Boston University School of Medicine（波士顿大学医学学院）；Signal and Smart Systems Lab (L3S)（信号与智能系统实验室）；National School of Engineering of Tunis, University of Tunis El Manar（突尼斯国家工程学院，突尼斯El Manar大学）
UAE: Universal Anatomical Embedding on Multi-modality Medical Images
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院，空天地海一体化大数据应用技术国家工程实验室）；Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Beijing United Family Hospital（北京和睦家医院）；The First Affiliated Hospital, Zhejiang University（浙江大学医学院附属第一医院）；Peking Union Medical College Hospital（北京协和医院）
Structure-Guided Self-Supervised Matching for One-Shot Medical Landmark Detection
Department of Computer Science（计算机科学系）；Renmin University of China（中国人民大学）；University of Science and Technology of China（中国科学技术大学）；School of Biomedical Engineering（生物医学工程学院）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiaotong University（上海交通大学）；Department of Orthopedics（骨科系）；Affiliated Hospital of Guizhou Medical University（贵州医科大学附属医院）；Yale University（耶鲁大学）
