Deep neural networks yield promising results in a wide range of computer vision applications, including landmark detection. A major challenge for accurate anatomical landmark detection in volumetric images such as clinical CT scans is that large-scale data often constrain the capacity of the employed neural network architecture due to GPU memory limitations, which in turn can limit the precision of the output. We propose a multi-scale, end-to-end deep learning method that achieves fast and memory-efficient landmark detection in 3D images. Our architecture consists of blocks of shift-equivariant networks, each of which performs landmark detection at a different spatial scale. These blocks are connected from coarse to fine-scale, with differentiable resampling layers, so that all levels can be trained together. We also present a noise injection strategy that increases the robustness of the model and allows us to quantify uncertainty at test time. We evaluate our method for carotid artery bifurcations detection on 263 CT volumes and achieve a better than state-of-the-art accuracy with mean Euclidean distance error of 2.81mm.
Locating semantically meaningful landmark points is a crucial component of a large number of computer vision pipelines. Because of the small number of available datasets with ground truth landmark annotations, it is important to design robust unsupervised and semi-supervised methods for landmark detection. Many of the recent unsupervised learning methods rely on the equivariance properties of landmarks to synthetic image deformations. Our work focuses on such widely used methods and sheds light on its core problem, its inability to produce equivariant intermediate convolutional features. This …
Shenzhen University（深圳大学）；Nanfang Medical University（南方医科大学）；The Chinese University of Hong Kong（香港中文大学）
Volumetric ultrasound has great potentials in promoting prenatal examinations. Automated solutions are highly desired to efficiently and effectively analyze the massive volumes. Segmentation and landmark localization are two key techniques in making the quantitative evaluation of prenatal ultrasound volumes available in clinic. However, both tasks are non-trivial when considering the poor image quality, boundary ambiguity and anatomical variations in volumetric ultrasound. In this paper, we propose an effective framework for simultaneous segmentation and landmark localization in prenatal ultra…
Given a collection of images, humans are able to discover landmarks by modeling the shared geometric structure across instances. This idea of geometric equivariance has been widely used for the unsupervised discovery of object landmark representations. In this paper, we develop a simple and effective approach by combining instance-discriminative and spatially-discriminative contrastive learning. We show that when a deep network is trained to be invariant to geometric and photometric transformations, representations emerge from its intermediate layers that are highly predictive of object landma…
IIT Kharagpur（印度理工学院克勒格布尔分校）；IIT Bombay（印度理工学院孟买分校）；IIT Madras（印度理工学院马德拉斯分校）；Philips Research Philips Healthcare（飞利浦研究 飞利浦医疗保健）
In the medical field, landmark detection in MRI plays an important role in reducing medical technician efforts in tasks like scan planning, image registration, etc. First, 88 landmarks spread across the brain anatomy in the three respective views -- sagittal, coronal, and axial are manually annotated, later guidelines from the expert clinical technicians are taken sub-anatomy-wise, for better localization of the existing landmarks, in order to identify and locate the important atlas landmarks even in oblique scans. To overcome limited data availability, we implement realistic data augmentation…
Center for Research on Intelligent Perception and Computing (CRIPAC)（智能感知与计算研究中心）；National Laboratory of Pattern Recognition (NLPR)（模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology (CEBSIT), CAS（中国科学院脑科学与智能技术卓越创新中心）
3D face shape is more expressive and viewpoint-consistent than its 2D counterpart. However, 3D facial landmark localization in a single image is challenging due to the ambiguous nature of landmarks under 3D perspective. Existing approaches typically adopt a suboptimal two-step strategy, performing 2D landmark localization followed by depth estimation. In this paper, we propose the Joint Voxel and Coordinate Regression (JVCR) method for 3D facial landmark localization, addressing it more effectively in an end-to-end fashion. First, a compact volumetric representation is proposed to encode the p…
Indian Institute of Science（印度科学学院）；Google Research（谷歌研究院）
In this work, we introduce LEAD, an approach to discover landmarks from an unannotated collection of category-specific images. Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image, which are further used to learn landmarks in a semi-supervised manner. While there have been advances in self-supervised learning of image features for instance-level tasks like classification, these methods do not ensure dense equivariant representations. The property of equivariance is of interest for dense prediction tasks like landma…
This paper proposes the use of an end-to-end Convolutional Neural Network for direct reconstruction of the 3D geometry of humans via volumetric regression. The proposed method does not require the fitting of a shape model and can be trained to work from a variety of input types, whether it be landmarks, images or segmentation masks. Additionally, non-visible parts, either self-occluded or otherwise, are still reconstructed, which is not the case with depth map regression. We present results that show that our method can handle both pose variation and detailed reconstruction given appropriate d…
Canon Medical Research Europe（佳能医疗欧洲研究中心）；Queen Elizabeth University Hospital（伊丽莎白女王大学医院）；University of Glasgow（格拉斯哥大学）
We present an efficient neural network method for locating anatomical landmarks in 3D medical CT scans, using atlas location autocontext in order to learn long-range spatial context. Location predictions are made by regression to Gaussian heatmaps, one heatmap per landmark. This system allows patchwise application of a shallow network, thus enabling multiple volumetric heatmaps to be predicted concurrently without prohibitive GPU memory requirements. Further, the system allows inter-landmark spatial relationships to be exploited using a simple overdetermined affine mapping that is robust to de…
Single-Photon Image Super-Resolution (SPISR) aims to recover a high-resolution volumetric photon counting cube from a noisy low-resolution one by computational imaging algorithms. In real-world scenarios, pairs of training samples are often expensive or impossible to obtain. By extending Equivariant Imaging (EI) to volumetric single-photon data, we propose a self-supervised learning framework for the SPISR task. Particularly, using the Poisson unbiased Kullback-Leibler risk estimator and equivariance, our method is able to learn from noisy measurements without ground truths. Comprehensive expe…
Efficient 3D affinely equivariant CNNs with adaptive fusion of augmented spherical Fourier-Bessel bases
School of Computer and Artificial Intelligence, Nanjing University of Finance and Economics（南京金融学院计算机与人工智能学院）；Interdisciplinary Center for Scientific Computing, Mannheim Institute for Intelligent Systems in Medicine, Medical Faculty Mannheim, Heidelberg University（科学计算交叉中心、医学智能系统研究所、曼海姆医学院、海德堡大学）；Computer Assisted Clinical Medicine, Mannheim Institute for Intelligent Systems in Medicine, Medical Faculty Mannheim, Heidelberg University（临床医学辅助计算机、医学智能系统研究所、曼海姆医学院、海德堡大学）；Department of Diagnostic and Interventional Radiology, University Hospital Bonn（诊断与介入放射科、波恩大学医院）；Interdisciplinary Center for Scientific Computing, Central Institute for Computer Engineering, CSZ Heidelberg Center for Model-Based AI, Data Analysis and Modeling in Medicine, Mannheim Institute for Intelligent Systems in Medicine, Medical Faculty Mannheim, Heidelberg University（科学计算交叉中心、计算机工程中央研究所、海德堡模型驱动AI中心、医学数据与建模中心、曼海姆医学智能系统研究所、曼海姆医学院、海德堡大学）
MILA-University of Montreal（蒙特利尔大学MILA研究所）；NVIDIA（英伟达）；Ecole Polytechnique of Montreal（蒙特利尔高等理工学院）；CIFAR（加拿大高等研究院）；Facebook AI Research（脸书人工智能研究院）
We present two techniques to improve landmark localization in images from partially annotated datasets. Our primary goal is to leverage the common situation where precise landmark locations are only provided for a small data subset, but where class labels for classification or regression tasks related to the landmarks are more abundantly available. First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data. A key…
Department of Statistics, University of Chicago, Chicago, USA（统计系，芝加哥大学，芝加哥，美国）；Department of Computer Science, University of Chicago, Chicago, USA（计算机科学系，芝加哥大学，芝加哥，美国）
Unitary；VGG, University of Oxford（牛津大学VGG）；University of Edinburgh（爱丁堡大学）
Equivariance to random image transformations is an effective method to learn landmarks of object categories, such as the eyes and the nose in faces, without manual supervision. However, this method does not explicitly guarantee that the learned landmarks are consistent with changes between different instances of the same object, such as different facial identities. In this paper, we develop a new perspective on the equivariance approach by noting that dense landmark detectors can be interpreted as local image descriptors equipped with invariance to intra-category variations. We then propose a …
4D Flow is an MRI sequence which allows acquisition of 3D images of the heart. The data is typically acquired volumetrically, so it must be reformatted to generate cardiac long axis and short axis views for diagnostic interpretation. These views may be generated by placing 6 landmarks: the left and right ventricle apex, and the aortic, mitral, pulmonary, and tricuspid valves. In this paper, we propose an automatic method to localize landmarks in order to compute the cardiac views. Our approach consists of first calculating a bounding box that tightly crops the heart, followed by a landmark loc…
Automated Landmark Detection for assessing hip conditions: A Cross-Modality Validation of MRI versus X-ray
MaLGa Center, DIBRIS University of Genoa（MaLGa中心，热那亚大学DIBRIS学院）；Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences University of Oxford（牛津大学风湿病学与肌肉骨骼科学系）；Department of Computer Science University of Oxford（牛津大学计算机科学系）
MIT（麻省理工学院）；Tsinghua University（清华大学）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）
The performance and diagnostic utility of magnetic resonance imaging (MRI) in pregnancy is fundamentally constrained by fetal motion. Motion of the fetus, which is unpredictable and rapid on the scale of conventional imaging times, limits the set of viable acquisition techniques to single-shot imaging with severe compromises in signal-to-noise ratio and diagnostic contrast, and frequently results in unacceptable image quality. Surprisingly little is known about the characteristics of fetal motion during MRI and here we propose and demonstrate methods that exploit a growing repository of MRI ob…
H3DE-Net: Efficient and Accurate 3D Landmark Detection in Medical Imaging
School of Computer Science USTC（计算机科学学院 中国科学技术大学）；School of Computer Science（计算机科学学院）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；Sciences College（科学学院）；School of Information Science, NENU（信息科学学院 南京师范大学）；Technical University of Munich (TUM)（慕尼黑技术大学）；Stanford University（斯坦福大学）
Reconstruction of 3D lumbar spine models from incomplete segmentations using landmark detection
University of Koblenz（科布伦茨大学）；Institute for Computational Visualistics（计算视觉研究所）；Institute for Medical Technology and Information Processing（医学技术与信息处理研究所）
The University of Nottingham（诺丁汉大学）；Kingston University（金斯顿大学）
3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the availability of multiple facial images (sometimes from the same subject) as input, and must address a number of methodological challenges such as establishing dense correspondences across large facial poses, expressions, and non-uniform illumination. In general these methods require complex and inefficient pipelines for model building and fitting. In this work, we propose to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropr…
Enhancing Alzheimer's Diagnosis: Leveraging Anatomical Landmarks in Graph Convolutional Neural Networks on Tetrahedral Meshes
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA（计算与增强智能学院，亚利桑那州立大学）；Amazon AGI, Redmond, WA, USA（亚马逊人工智慧部门）；Banner Alzheimer’s Institute, Phoenix, AZ, USA（贝纳尔德阿尔茨海默病研究所）
In this work, we propose a novel transformation for events from an event camera that is equivariant to optical flow under convolutions in the 3-D spatiotemporal domain. Events are generated by changes in the image, which are typically due to motion, either of the camera or the scene. As a result, different motions result in a different set of events. For learning based tasks based on a static scene such as classification which directly use the events, we must either rely on the learning method to learn the underlying object distinct from the motion, or to memorize all possible motions for each…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）
Deep learning networks have shown promising performance for accurate object localization in medial images, but require large amount of annotated data for supervised training, which is expensive and expertise burdensome. To address this problem, we present a one-shot framework for organ and landmark localization in volumetric medical images, which does not need any annotation during the training stage and could be employed to locate any landmarks or organs in test images given a support (reference) image during the inference stage. Our main idea comes from that tissues and organs from different…
3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs
Medical University of Vienna（维也纳医科大学）；BioMedIA, Imperial College London（伦敦帝国理工学院生物医学图像分析研究所）；Institute for AI and Informatics in Medicine, Klinikum rechts der Isar, Technical University Munich（慕尼黑工业大学伊萨尔河畔医院医学人工智能与信息学研究所）；Department of Ophthalmology, University of Basel（巴塞尔大学眼科系）
Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Institute of Mechatronic Systems（机电系统研究所）
In microsurgery, lasers have emerged as precise tools for bone ablation. A challenge is automatic control of laser bone ablation with 4D optical coherence tomography (OCT). OCT as high resolution imaging modality provides volumetric images of tissue and foresees information of bone position and orientation (pose) as well as thickness. However, existing approaches for OCT based laser ablation control rely on external tracking systems or invasively ablated artificial landmarks for tracking the pose of the OCT probe relative to the tissue. This can be superseded by estimating the scene flow cause…
Imperial College London（帝国理工学院）；MRC London Institute of Medical Sciences（MRC伦敦医学科学研究所）；National Heart and Lung Institute（国家心肺研究所）
Deep learning approaches have achieved state-of-the-art performance in cardiac magnetic resonance (CMR) image segmentation. However, most approaches have focused on learning image intensity features for segmentation, whereas the incorporation of anatomical shape priors has received less attention. In this paper, we combine a multi-task deep learning approach with atlas propagation to develop a shape-constrained bi-ventricular segmentation pipeline for short-axis CMR volumetric images. The pipeline first employs a fully convolutional network (FCN) that learns segmentation and landmark localisat…
Deep reinforcement learning augments the reinforcement learning framework and utilizes the powerful representation of deep neural networks. Recent works have demonstrated the remarkable successes of deep reinforcement learning in various domains including finance, medicine, healthcare, video games, robotics, and computer vision. In this work, we provide a detailed review of recent and state-of-the-art research advances of deep reinforcement learning in computer vision. We start with comprehending the theories of deep learning, reinforcement learning, and deep reinforcement learning. We then pr…
Queensland University of Technology（昆士兰科技大学）；Carnegie Mellon University（卡内基梅隆大学）
One challenge that remains open in 3D deep learning is how to efficiently represent 3D data to feed deep networks. Recent works have relied on volumetric or point cloud representations, but such approaches suffer from a number of issues such as computational complexity, unordered data, and lack of finer geometry. This paper demonstrates that a mesh representation (i.e. vertices and faces to form polygonal surfaces) is able to capture fine-grained geometry for 3D reconstruction tasks. A mesh however is also unstructured data similar to point clouds. We address this problem by proposing a learni…
