Cephalometric Landmark Regression with Convolutional Neural Networks on 3D Computed Tomography Data
Institute of Information Technologies, Mathematics and Mechanics, Lobachevsky State University（罗巴切夫斯基州立大学信息技术、数学与力学研究所）；Medical University of Vienna（维也纳医科大学）
Dmitry Lachinov, Alexandra Getmanskaya, Vadim Turlapov
In this paper, we address the problem of automatic three-dimensional cephalometric analysis. Cephalometric analysis performed on lateral radiographs doesn't fully exploit the structure of 3D objects due to projection onto the lateral plane. With the development of three-dimensional imaging techniques such as CT, several analysis methods have been proposed that extend to the 3D case. The analysis based on these methods is invariant to rotations and translations and can describe difficult skull deformation, where 2D cephalometry has no use. In this paper, we provide a wide overview of existing approaches for cephalometric landmark regression. Moreover, we perform a series of experiments with state of the art 3D convolutional neural network (CNN) based methods for keypoint regression: direct regression with CNN, heatmap regression and Softargmax regression. For the first time, we extensively evaluate the described methods and demonstrate their effectiveness in the estimation of Frankfort Horizontal and cephalometric points locations for patients with severe skull deformations. We demonstrate that Heatmap and Softargmax regression models provide sufficient regression error for medical applications (less than 4 mm). Moreover, the Softargmax model achieves 1.15o inclination error for the Frankfort horizontal. For the fair comparison with the prior art, we also report results projected on the lateral plane.
The success of deep learning methods relies on the availability of a large number of datasets with annotations; however, curating such datasets is burdensome, especially for medical images. To relieve such a burden for a landmark detection task, we explore the feasibility of using only a single annotated image and propose a novel framework named Cascade Comparing to Detect (CC2D) for one-shot landmark detection. CC2D consists of two stages: 1) Self-supervised learning (CC2D-SSL) and 2) Training with pseudo-labels (CC2D-TPL). CC2D-SSL captures the consistent anatomical information in a coarse-t…
Cephalometric analysis has an important role in dentistry and especially in orthodontics as a treatment planning tool to gauge the size and special relationships of the teeth, jaws and cranium. The first step of using such analyses is localizing some important landmarks known as cephalometric landmarks on craniofacial in x-ray image. The past decade has seen a growing interest in automating this process. In this paper, a novel hybrid approach is proposed for automatic detection of cephalometric landmarks. Here, the landmarks are categorized into three main sets according to their anatomical ch…
Cephalometric Landmark Detection across Ages with Prototypical Network
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Lingang Laboratory（临港实验室）；Shanghai Linkedcare Information Technology Co., Ltd.（上海联众医疗信息技术有限公司）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University（上海交通大学医学院附属第九人民医院）；Shanghai United Imaging Intelligence Co. Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
The University of Hong Kong（香港大学）；Modontics (Hong Kong) Limited（Modontics（香港）有限公司）
Marking anatomical landmarks in cephalometric radiography is a critical operation in cephalometric analysis. Automatically and accurately locating these landmarks is a challenging issue because different landmarks require different levels of resolution and semantics. Based on this observation, we propose a novel attentive feature pyramid fusion module (AFPF) to explicitly shape high-resolution and semantically enhanced fusion features to achieve significantly higher accuracy than existing deep learning-based methods. We also combine heat maps and offset maps to perform pixel-wise regression-vo…
Department of Computing Science, University of Alberta（阿尔伯塔大学计算科学系）
CNNs, initially inspired by human vision, differ in a key way: they sample uniformly, rather than with highest density in a focal point. For very large images, this makes training untenable, as the memory and computation required for activation maps scales quadratically with the side length of an image. We propose an image pyramid based approach that extracts narrow glimpses of the of the input image and iteratively refines them to accomplish regression tasks. To assist with high-accuracy regression, we introduce a novel intermediate representation we call 'spatialized features'. Our approach …
University of Brasilia（巴西利亚大学）；University of Campinas（坎皮纳斯大学）；University of São Paulo（圣保罗大学）；University of Granada（格拉纳达大学）；Department of Federal Police（联邦警察局）
Facial landmarks are employed in many research areas such as facial recognition, craniofacial identification, age and sex estimation among the most important. In the forensic field, the focus is on the analysis of a particular set of facial landmarks, defined as cephalometric landmarks. Previous works demonstrated that the descriptive adequacy of these anatomical references for an indirect application (photo-anthropometric description) increased the marking precision of these points, contributing to a greater reliability of these analyzes. However, most of them are performed manually and all o…
Deep Learning Techniques for Automatic Lateral X-ray Cephalometric Landmark Detection: Is the Problem Solved?
Shenzhen University（深圳大学）；School of Biomedical Engineering, Shenzhen University Medical School（深圳大学医学部生物医学工程学院）；Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）
Towards Better Cephalometric Landmark Detection with Diffusion Data Generation
State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau（物联网智能城市国家重点实验室，澳门大学计算机与信息科学系）；Department of Orthopaedic Surgery, Zhongshan Hospital, Fudan University（复旦大学中山医院骨科部）；Department of Periodontology, Justus-Liebig-University of Giessen, Germany（吉森大学牙科系）；Department of Periodontics, Stomatology Hospital of Guangzhou Medical University（广州医科大学口腔医院牙科部）
University of Electronic Science and Technology of China（电子科技大学）
Localization of anatomical landmarks is essential for clinical diagnosis, treatment planning, and research. In this paper, we propose a novel deep network, named feature aggregation and refinement network (FARNet), for the automatic detection of anatomical landmarks. To alleviate the problem of limited training data in the medical domain, our network adopts a deep network pre-trained on natural images as the backbone network and several popular networks have been compared. Our FARNet also includes a multi-scale feature aggregation module for multi-scale feature fusion and a feature refinement …
Silhouette-to-Contour Registration: Aligning Intraoral Scan Models with Cephalometric Radiographs
School of AI and Advanced Computing, Xi'an Jiaotong-Liverpool University, China（人工智能与先进计算学院，西安交通大学利物浦大学，中国）；School of Electrical Engineering, Electronics and Computer Science, University of Liverpool, United Kingdom（电子工程、电子科学与计算机科学学院，利物浦大学，英国）；School of Advanced Technology, Xi'an Jiaotong-Liverpool University, China（先进技术学院，西安交通大学利物浦大学，中国）；School of Physical Sciences, University of Liverpool, Liverpool, United Kingdom（物理科学学院，利物浦大学，英国）；School of Mathematics and Physics, Xi'an Jiaotong-Liverpool University, China（数学与物理学院，西安交通大学利物浦大学，中国）；Institute of Artificial Intelligence Innovation, Zhejiang University of Technology, China（人工智能创新研究院，浙江工业大学，中国）
National University of Sciences and Technology (NUST)（国立科技大学（NUST））；Riphah International University（里法国际大学）
The accurate identification and precise localization of cephalometric landmarks enable the classification and quantification of anatomical abnormalities. The traditional way of marking cephalometric landmarks on lateral cephalograms is a monotonous and time-consuming job. Endeavours to develop automated landmark detection systems have persistently been made, however, they are inadequate for orthodontic applications due to unavailability of a reliable dataset. We proposed a new state-of-the-art dataset to facilitate the development of robust AI solutions for quantitative morphometric analysis. …
The University of Hong Kong（香港大学）；ShanghaiTech University（上海科技大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Detecting 3D landmarks on cone-beam computed tomography (CBCT) is crucial to assessing and quantifying the anatomical abnormalities in 3D cephalometric analysis. However, the current methods are time-consuming and suffer from large biases in landmark localization, leading to unreliable diagnosis results. In this work, we propose a novel Structure-Aware Long Short-Term Memory framework (SA-LSTM) for efficient and accurate 3D landmark detection. To reduce the computational burden, SA-LSTM is designed in two stages. It first locates the coarse landmarks via heatmap regression on a down-sampled CB…
A deep neural network based cephalometric landmark identification model is proposed. Two neural networks, named patch classification and point estimation, are trained by multi-scale image patches cropped from 935 Cephalograms (of Japanese young patients), whose size and orientation vary based on landmark-dependent criteria examined by orthodontists. The proposed model identifies both 22 hard and 11 soft tissue landmarks. In order to evaluate the proposed model, (i) landmark estimation accuracy by Euclidean distance error between true and estimated values, and (ii) success rate that the estimat…
Identification of 3D cephalometric landmarks that serve as proxy to the shape of human skull is the fundamental step in cephalometric analysis. Since manual landmarking from 3D computed tomography (CT) images is a cumbersome task even for the trained experts, automatic 3D landmark detection system is in a great need. Recently, automatic landmarking of 2D cephalograms using deep learning (DL) has achieved great success, but 3D landmarking for more than 80 landmarks has not yet reached a satisfactory level, because of the factors hindering machine learning such as the high dimensionality of the …
University Medical Center Utrecht（乌得勒支大学医学中心）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学阿姆斯特丹医学中心）；University of Twente（特文特大学）；Wageningen University（瓦赫宁根大学）
In this study, we propose a fast and accurate method to automatically localize anatomical landmarks in medical images. We employ a global-to-local localization approach using fully convolutional neural networks (FCNNs). First, a global FCNN localizes multiple landmarks through the analysis of image patches, performing regression and classification simultaneously. In regression, displacement vectors pointing from the center of image patches towards landmark locations are determined. In classification, presence of landmarks of interest in the patch is established. Global landmark locations are o…
Automatic and accurate detection of anatomical landmarks is an essential operation in medical image analysis with a multitude of applications. Recent deep learning methods have improved results by directly encoding the appearance of the captured anatomy with the likelihood maps (i.e., heatmaps). However, most current solutions overlook another essence of heatmap regression, the objective metric for regressing target heatmaps and rely on hand-crafted heuristics to set the target precision, thus being usually cumbersome and task-specific. In this paper, we propose a novel learning-to-learn frame…
Centrum Wiskunde & Informatica（数学与计算机科学国家研究所）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学阿姆斯特丹大学医学中心）；Delft University of Technology（代尔夫特理工大学）
Anatomical landmark correspondences in medical images can provide additional guidance information for the alignment of two images, which, in turn, is crucial for many medical applications. However, manual landmark annotation is labor-intensive. Therefore, we propose an end-to-end deep learning approach to automatically detect landmark correspondences in pairs of two-dimensional (2D) images. Our approach consists of a Siamese neural network, which is trained to identify salient locations in images as landmarks and predict matching probabilities for landmark pairs from two different images. We t…
Centrum Wiskunde & Informatica（荷兰国家数学与计算机科学研究所）；Amsterdam University Medical Centers（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；Delft University of Technology（代尔夫特理工大学）；Leiden University Medical Center（莱顿大学医学中心）
Purpose: Deformable Image Registration (DIR) can benefit from additional guidance using corresponding landmarks in the images. However, the benefits thereof are largely understudied, especially due to the lack of automatic landmark detection methods for three-dimensional (3D) medical images. Approach: We present a Deep Convolutional Neural Network (DCNN), called DCNN-Match, that learns to predict landmark correspondences in 3D images in a self-supervised manner. We trained DCNN-Match on pairs of Computed Tomography (CT) scans containing simulated deformations. We explored five variants of DCNN…
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Cephalometric tracing method is usually used in orthodontic diagnosis and treatment planning. In this paper, we propose a deep learning based framework to automatically detect anatomical landmarks in cephalometric X-ray images. We train the deep encoder-decoder for landmark detection, and combine global landmark configuration with local high-resolution feature responses. The proposed frame-work is based on 2-stage u-net, regressing the multi-channel heatmaps for land-mark detection. In this framework, we embed attention mechanism with global stage heatmaps, guiding the local stage inferring, t…
University of Brasilia（巴西利亚大学）；Federal University of Maranhao（马拉尼昂联邦大学）；KU Leuven（鲁汶大学）；National Institute of Criminalistics（国家刑事科学研究所）；Brazilian Federal Police（巴西联邦警察）
Facial analysis permits many investigations some of the most important of which are craniofacial identification, facial recognition, and age and sex estimation. In forensics, photo-anthropometry describes the study of facial growth and allows the identification of patterns in facial skull development by using a group of cephalometric landmarks to estimate anthropological information. In several areas, automation of manual procedures has achieved advantages over and similar measurement confidence as a forensic expert. This manuscript presents an approach using photo-anthropometric indexes, gene…
With the mushrooming use of computed tomography (CT) images in clinical decision making, management of CT data becomes increasingly difficult. From the patient identification perspective, using the standard DICOM tag to track patient information is challenged by issues such as misspelling, lost file, site variation, etc. In this paper, we explore the feasibility of leveraging the faces in 3D CT images as biometric features. Specifically, we propose an automatic processing pipeline that first detects facial landmarks in 3D for ROI extraction and then generates aligned 2D depth images, which are…
Self-CephaloNet: A Two-stage Novel Framework using Operational Neural Network for Cephalometric Analysis
Qatar University（卡塔尔大学）；Bangabandhu Sheikh Mujib Medical University（邦班杜·谢赫·穆吉布医科大学）；University of Regina（里贾纳大学）；Kennesaw State University（肯尼索州立大学）；Prince Sattam Bin Abdulaziz University（阿卜杜勒阿齐兹亲王大学）；Majmaah University（迈迈阿大学）
Enrichment of the NLST and NSCLC-Radiomics computed tomography collections with AI-derived annotations
Brigham and Women's Hospital（布莱根妇女医院）；Mass General Brigham（麻省总医院布里格姆医疗系统）；Harvard Medical School（哈佛医学院）；PixelMed Publishing（PixelMed出版公司）；Massachusetts General Hospital（麻省总医院）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Maastricht University（马斯特里赫特大学）
Warsaw University of Technology（华沙理工大学）；Poznan University of Medical Sciences（波兹南医科大学）；Medical Centre Leeuwarden（吕伐登医疗中心）；Tooploox
Convolutional neural networks (CNNs) are used in many areas of computer vision, such as object tracking and recognition, security, military, and biomedical image analysis. This review presents the application of convolutional neural networks in one of the fields of dentistry - orthodontics. Advances in medical imaging technologies and methods allow CNNs to be used in orthodontics to shorten the planning time of orthodontic treatment, including an automatic search of landmarks on cephalometric X-ray images, tooth segmentation on Cone-Beam Computed Tomography (CBCT) images or digital models, and…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；Princeton University（普林斯顿大学）
Recent methods in multiple landmark detection based on deep convolutional neural networks (CNNs) reach high accuracy and improve traditional clinical workflow. However, the vulnerability of CNNs to adversarial-example attacks can be easily exploited to break classification and segmentation tasks. This paper is the first to study how fragile a CNN-based model on multiple landmark detection to adversarial perturbations. Specifically, we propose a novel Adaptive Targeted Iterative FGSM (ATI-FGSM) attack against the state-of-the-art models in multiple landmark detection. The attacker can use ATI-F…
Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Institute of Mechatronic Systems（机电系统研究所）
In microsurgery, lasers have emerged as precise tools for bone ablation. A challenge is automatic control of laser bone ablation with 4D optical coherence tomography (OCT). OCT as high resolution imaging modality provides volumetric images of tissue and foresees information of bone position and orientation (pose) as well as thickness. However, existing approaches for OCT based laser ablation control rely on external tracking systems or invasively ablated artificial landmarks for tracking the pose of the OCT probe relative to the tissue. This can be superseded by estimating the scene flow cause…
GE Healthcare（通用电气医疗集团）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Laboratoire de Mathématiques et Modélisation d’Évry（埃夫里数学与建模实验室）；Université de Paris（巴黎大学）
Partial shapes correspondences is a problem that often occurs in computer vision (occlusion, evolution in time...). In medical imaging, data may come from different modalities and be acquired under different conditions which leads to variations in shapes and topologies. In this paper we use an asymmetric data dissimilarity term applicable to various geometric shapes like sets of curves or surfaces, assessing the embedding of a shape into another one without relying on correspondences. It is designed as a data attachment for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) framework, …
University of Central Florida（中佛罗里达大学）；Ther-AI, LLC（Ther-AI有限责任公司）；National Institute of Dental and Craniofacial Research (NIDCR), National Institutes of Health (NIH)（美国国立卫生研究院下属国立牙科与颅面研究所）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Northwestern University（西北大学）；Sheikh Zyed Children’s Hospital（谢赫·扎耶德儿童医院）；George Washington University（乔治·华盛顿大学）
Purpose: We perform anatomical landmarking for craniomaxillofacial (CMF) bones without explicitly segmenting them. Towards this, we propose a new simple yet efficient deep network architecture, called \textit{relational reasoning network (RRN)}, to accurately learn the local and the global relations among the landmarks in CMF bones; specifically, mandible, maxilla, and nasal bones. Approach: The proposed RRN works in an end-to-end manner, utilizing learned relations of the landmarks based on dense-block units. For a given few landmarks as input, RRN treats the landmarking process similar to a …
National Institute of Dental and Craniofacial Research (NIDCR), National Institutes of Health (NIH)（美国国立卫生研究院（NIH）国立牙科与颅面研究所（NIDCR））；University of South Dakota Sanford School of Medicine（南达科他大学桑福德医学院）；University of Central Florida（中佛罗里达大学）
In this paper, we propose a novel deep learning framework for anatomy segmentation and automatic landmark- ing. Specifically, we focus on the challenging problem of mandible segmentation from cone-beam computed tomography (CBCT) scans and identification of 9 anatomical landmarks of the mandible on the geodesic space. The overall approach employs three inter-related steps. In step 1, we propose a deep neu- ral network architecture with carefully designed regularization, and network hyper-parameters to perform image segmentation without the need for data augmentation and complex post- processing…
Seoul National University（首尔大学）；Harvard University（哈佛大学）；Seoul National University School of Dentistry（首尔大学牙医学院）；Soong-sil University（崇实大学）
Computerized registration between maxillofacial cone-beam computed tomography (CT) images and a scanned dental model is an essential prerequisite in surgical planning for dental implants or orthognathic surgery. We propose a novel method that performs fully automatic registration between a cone-beam CT image and an optically scanned model. To build a robust and automatic initial registration method, our method applies deep-pose regression neural networks in a reduced domain (i.e., 2-dimensional image). Subsequently, fine registration is performed via optimal clusters. Majority voting system ac…
Enhancing Alzheimer's Diagnosis: Leveraging Anatomical Landmarks in Graph Convolutional Neural Networks on Tetrahedral Meshes
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA（计算与增强智能学院，亚利桑那州立大学）；Amazon AGI, Redmond, WA, USA（亚马逊人工智慧部门）；Banner Alzheimer’s Institute, Phoenix, AZ, USA（贝纳尔德阿尔茨海默病研究所）
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Southeast University（东南大学）；Fudan University（复旦大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；The University of Hong Kong（香港大学）
Tomographic image reconstruction with deep learning is an emerging field, but a recent landmark study reveals that several deep reconstruction networks are unstable for computed tomography (CT) and magnetic resonance imaging (MRI). Specifically, three kinds of instabilities were reported: (1) strong image artefacts from tiny perturbations, (2) small features missing in a deeply reconstructed image, and (3) decreased imaging performance with increased input data. On the other hand, compressed sensing (CS) inspired reconstruction methods do not suffer from these instabilities because of their bu…
